Publish statistics continuously

[ripienaar]

Proposed change

Monitoring / Management systems have needs to have a full-system view on a regular time cadence, our only supported way today is a full JSZ request that has severe penalties on the system and create a very large number of locks all at the same time.

We should create a model where asset information is published on a regular cadence, for example, we might say we need all asset INFO every 5 minutes. During this 5 minutes users are doing INFO requests and we are calculating INFO and STATE quite regularly for snapshots and other reasons. We would take this already calculated and serialised data and publish it to a well known subject, that must include ACCOUNT name in the subject, at the time it was calculated. Assets that have not had a publish on the 5 minute mark would have theirs published then.

We wouldn't publish EVERY calculated info during the 5 minute window but just ensure we publish them at least every 5 minutes.

Lots of details to work out here but this should significantly move the needle on the costs of monitoring and be a pattern we can apply to many kinds of information now reported by the Z subjects

Use case

Improve monitoring visibility at lower impact.

Contribution

No response