Building a client
Once the Hypermass CLI is syncing data to your local machine, the next step is to act on that data. Because Hypermass writes updates as standard files, you can build a client in any language without needing to implement complex networking or authentication logic yourself.
The "Filesystem Poller" Pattern​
The most common way to build a client is the Filesystem Poller. Your application periodically checks the contents of a specific directory filters and sorts the contents and processes them one by one in sequence from oldest to newest.
Please select your preferred language below for some sample code to get you started;
- Python - heavily used in data science
- C# .NET - Industry standard langauge for enterprise software
- Java - Industry standard langauge for enterprise software
Here's a general guide for implementing a subscription listener; General Implementation Guide
The hypermass sync command writes each payload file to a hidden folder (here: ./.hypermass/temporary) then moves it
to the subscription location. This means that the file is always "complete" by the time you read it - you don't need to
check to see whether the file is still being written to.
Handling Different Writer Types​
In your hypermass-config.yaml, you chose a writer-type. Your client should be built to match:
| Writer Type | Client Logic |
|---|---|
| file-per-payload | Your client should look for individual new files. Each file contains one update. |
| folder-with-metadata | Your client should look for new directories. Each directory contains the payload and a metadata.json file. |
Implementation Tips​
When implementing your watcher, it's worth noting;
- Sequence: always process files by modified date, earliest to latest - this ensures you process them in order
- Idempotency: each file is only written once, unless you replay it manually
- Cleanup: Since Hypermass will continually writing files as they are published, ensure your client deletes files after processing or moves them to an "archive" location.
- Error Handling: If your processing logic fails, stop processing and don't delete the original file, this gives you opportunity to fix the issue and restart the process.
- replay: It's possible to reset the subscription to any point manually by following the replay process
If you need access to payload metadata, use the folder-with-metadata writer type. This provides you with the
original publication timestamp, payload Id and other publication metadata.
You will have to modify the code in here to open both the metadata file and the payload file.
Future Features​
We're looking to add more stream management features (for example being able to trigger a replay to a specific time or specific payload by running a command). Keep an eye on the blog for feature announcements.