Message Cadence
There are technical tradeoffs to limiting the message cadence - that is, "how short is the wait between one payload and another" (where a high cadence means a short gap). As a platform Hypermass must limit both message size and cadence - this keeps engineering decisions sane, and costs grounded in reality.
On the one hand, while handling very large files, you'd expect these to be sent infrequently. On the other, small files might be expected to see frequent changes on the order of minutes or seconds between data.
Our first choice is to support "big, infrequent" changes - on the range of changes every 10 minutes upwards (to once per year, or even less frequent).
Big and Infrequent - the first pass
As a platform we made the call to aim for this type of data first. It's easier to use as a subscriber, easier to publish and a lot of real world data lives in this space.
Some examples that fit into this category;
| Frequency | Types of Data |
|---|---|
| Monthly+ | Housing stats, Council boundaries, Accessibility audits |
| Weekly | Event schedules, Commerce trends, School timetables |
| Daily / Hourly | Sports results, News summaries, Election tallies, Site uptime reports |
Small and frequent - a work in progress
We're working on how to handle high frequency updates. The primary challenge is not the technical challenge of processing fast data (although that is a very real challenge), but rather one of transmitting this in a broadly useful and approachable format.
Our current thinking is that most high frequency data (with broad public appeal) is "State" data - that is high frequency messages relating to the state of some "subjects" in a larger context.
Think of a set of train positions tracked in real time;
- some trains are moving - generating new positions
- some aren't moving - no new positions
- some aren't reporting - completely quiet, may be removed
In this example there's a conceptual "state" (at any given moment in time) of where all the trains are.
Similarly examples in this type of data; evolving news stories, aircraft/bus/tram positions, live air quality updates, etc.
Following this line of reasoning, most consumers of this type of data don't just want to start receiving messages - they want to know the state (where are all the trains right now) and the deltas after that state in real time (which ones moved).
We're considering several technical options here including just exposing raw streams of small messages, or doing something a little more helpful with a built-in concept of "Deltas", "Patches" or similar change-over-time abstractions.
Watch this space
If you're interested, please consider joining our Discord - it's a great place to interact with other users and our lead Dev tends to hang out there.