JSON models do not conform with current specifications

[sgvolt]

I'm currently implementing a parser for the JSON models according to the information model specification and found some issues when using the current models as a test case, because the currently provided models do not fully conform to the (admittedly also not quite yet consistent or final) information model specification:

• Models currently lack the (qua specification) required "label" field
• Models are not declared as "model" explicitly (I personally wouldn't object to that, honestly, but the specification disagrees)
• For parsers' sake, model "id" fields should not be purely numeric types (e.g. 1, 102, ...), given that the specification allows "alphanumeric characters and the underscore character", so even if only numeric IDs should be used for model IDs, they should still be stringly-typed (i.e. "1", "102", ...).
• Points' "access" fields have lowercase values (in contrast to the specification's uppercase "R" and "RW" definitions)
• Points' "mandatory" fields currently hold bool-ish strings ("true", "false") instead of the specification's "M" or "O" strings.
• Groups are missing their "type" attribute.
• some scaling factors are ascribed type "sunssf" instead of the specification's "sf"
• Points' "size" attribute is encoded as "len"

[bobfox]

Thanks for the detailed comments. As you can see it was a first approximation but needs some work. We are working through it now and should have an update shortly.

[altendky]

Is there any schema file that defines the form of the JSON?

[bobfox]

Currently, there is not. We are certainly open to seeing if there is a productive way to make use of one. We would like to get pysunpec to the point of being a good validator of information models in all of the encodings.

[altendky]

JSON Schema with a check script (there's https://github.com/Julian/jsonschema) and Travis turned on would go a long way. Or if you won't be clicking the button to enable Travis then we could put together a quick GitHub Actions for you.

[bobfox]

The latest pull request (#74) has an initial JSON schema to take a look at.

[nielsbasjes]

I'm wondering:
Why have you chosen to duplicate the specs into json files and manually maintain those?
I would have gone for a fully automated conversion between the formats instead.

In all of my projects I prefer having a single source of truth and an automated process converting between the various formats.

[wz2b]

You will continue to support the XML models as well, right? JSON is just an alternative? I have software written around the XML schema, I would prefer to not rewrite that.

[bobfox]

Sorry for the delay in response. We have refining the tools and process to try to make it as automated as possible. We now have a feature in the SVP Dashboard to convert from xml to spreadsheet, json to spreadsheet, spreadsheet to json, and spreadsheet to spreadsheet. We consider Excel (or application with .xlsx support) to be the information model editor. json files should not be edited by hand so that we can be provide a uniform json encoding. Once models are created/updated in the spreadsheet that can be cleaned up by generating a new spreadsheet from that one. Once the spreadsheet version is correct, the json version can be generated from that.
We have extended the modelling capabilities as specified in the SunSpec Device Information Model Specification which is in the new technology section on the sunspec website. In general, the intention is to deprecate xml and only use json going forward. The new modelling features are only supported in the json encoding. I am sorry for the inconvenience this may cause some in the short term but json was the more popular choice and we would like to have a single official definition,
We are trying to correct both process and content issues with the model definitions, so all comments and questions are really helpful. We have been focused on finalizing models 701-713 which provide DER support for IEEE 1557-2018 and putting processes and tools in place to help support the effort.
We are currently doing an update to the spec so the json schema should be used for model definition information.

[wz2b]

In general, the intention is to deprecate xml and only use json going forward. The new modelling features are only supported in the json encoding. I am sorry for the inconvenience this may cause some in the short term but json was the more popular choice and we would like to have a single official definition,

I agree one format should be authoritative and everything else derived from that. It sounds like this decision has already been made, so my comment here may very well be moot - which is okay, I'll live with whatever you decide.

It is my experience that JSON schema is not as robust, and that validation tools (including editors) aren't as good. The XSD you developed is very high quality. If it were me, I would consider the XML to be authoritative, and derive everything else from that - including writing one or two .XSLTs to go back and forth between the different formats. I say that knowing that, while I love xsl transformation, it's basically greek to a lot of people.

I currently use Xpaths to take all you .XML files and build Kotlin objects. I'm making this public, so you can see what I'm doing if you're interested. I don't mind at some point converting that to use the JSON instead of the XML ... it's just work (it took me a few days to get the code generator working).

One piece of feedback I would really like to add (but it's off topic here) is giving models unique names. 111, 112, 113 are all named "inverter." They have different labels, but I can't use the labels to name objects, so I've created a bunch of objects named Model_111, Model_112, etc. That is not especially convenient for users of this library.

[bobfox]

I agree that XML is a more robust solution for model definition and validation. The motivation for moving to JSON involved multiple factors. The SunSpec model definitions are fairly simple system that JSON can handle. Most companies have more developers that understand JSON than XML so it is preferred by many. We will be developing a simple HTTP based web service interface for SunSpec data exchange once we can all the 1547 content completed and frozen. It will be specified using JSON. We have migrated our file based sample devices to JSON encoding and they are easier to work with as well. So these are some of the factors that influenced the decision. At the end of the day, it seemed better to use one encoding strategy for all of the use cases.
I also agree that the models should have unique names. The original naming was rather haphazard for which I am to blame. There was a rational for the choices but it seems that it should be revisited. In general, the models are frozen, but I think this may be something that makes sense to update if there are no objections.

[wz2b]

No worries, it's obvious you've thought this through and made careful decisions. I'm on board. The next thing I'm wondering is: any thought been given to a set of COAP models bindings as well? That's going to be my next task - communicating with SunSpec compliant DERs but over the AMI network using COAP.

[andig]

To add to the initial list:

• some scale factors are -1 instead of point name

[andig]

@bobfox I'm currently trying to migrate https://github.com/andig/gosunspec from smdx to json. It is painful. Even two years later now, the JSON models still appear inconsistent. I'll try to post a diff of the prior/after results to give an indication of the amount of differences. Not necessarily every difference is an issue with the newer JSON definitions, but the diff will show that the differences are unpredictable instead of following a clear forward-improvement pattern.