Core Concepts

To get the most out of Hypermass, it helps to understand how we move data.

Streams

A Stream is simply a series of files that are published sequentially (we call these payloads). Payloads are immutable; once a file is published to a stream, it cannot be altered. This ensures a reliable audit trail for every subscriber. Imagine a series of xml files containing the big social events in Sheffield city center week by week, or monthly aggregate regional health data.

The idea here is that a publisher can release these files as the data changes, and subscribers get these files in real time.

We perform checks to make sure the files are what they say they are, and to catch any nasties (XXE, XEE and so on). Publishers can also use a "schema" (like XSD) which we can also check, giving you strong guarantees about the shape of the data you are processing.

Stream Keys

Every stream has a unique Key (e.g., _WZBZ1QU).

• Uniquely identifies this stream - basically an ID.
• Use this key in the CLI to subscribe to data.
• If you own the stream, use the key to publish data too.

The "CLI"

We encourage automation though a special command line tool that we made for this purpose. See here for the design rationale and here for guidance on how to use this.

The CLI "Sync" command

The real magic happens here - the hypermass CLI handles all the complex side cases, retries and so on presenting a simple model for your programs to interact with. The hypermass-cli monitors one or more folder on your machine:

• Publishing: Drop a file in, and it's pushed to the stream, validated and distributed to all subscribers.
• Subscribing: New data from the stream appears as a file in your local folder - from here you parse and handle it as needed.