Support for specifying extension keywords (properties) with an "x-" prefix

[rajpaulbagga]

Extensions to the json-schema specification, like the "javaType" keyword property, can trip up json schema validator tools that don't recognize the keywords and thus report errors. By convention, though, validator tools will ignore any keywords prefixed with "x-". Thus, any keyword extension to the specification should really be a keyword that begins with "x-".

In order to support "x-" prefixed versions of existing keyword extensions, while maintaining backward compatibility, the following change are made:

Change all references to node properties that are extensions to the specification such that they use helper methods (ExtensionsHelper.java) that will work on both the original extension name (e.g. "javaType") and also that name prefixed with 'x-' (e.g. x-javaType). The following keywords are affected:

• extends, javaType, existingJavaType, extendsJavaClass, javaInterfaces, javaEnumNames, javaEnums, javaJsonView, javaName, customPattern, customTimezone, customDatePattern, customTimePattern, customDateTimePattern, excludedFromEqualsAndHashCode

Update all message text references to any of the above to use "x-".

Duplicate existing test schema files & directories that exercise the above keywords and either duplicate the unit tests that use them (to use the new schemas) or parameterize the test suites that use them to operate on both the original test schema and the new "x-" variant.

Further test update details:

Due to a mix of tests in TypeIT.java, break out the tests from TypeIT.java that didn't use the common schema with many types to a separate test class, TypeMiscIT.java. (and the same for YamlTypeIT.java). This allowed parameterizing TypeIT.java without running some tests twice on the same input.

Update some other test schemas that use extension keywords (but don't explicitly validate them) to use the new "x-" variant as this should be the preferred variant for reasons stated above.

While here, add test variants of each of the above 'custom*Patterns' that use the x-customPattern alias (no tests had existed for the 'customPattern' alias)

[rajpaulbagga]

It is further recommended that documentation be updated to refer to extensions by their "x-" prefix as the preferred. And that future keyword extensions always are prefixed with "x-"

[joelittlejohn]

Hi Rajpaul. Which validator have you have trouble with? All validators I have used will ignore these extension properties, no matter how they are named. There's no need to name them like x-.... You can try this here:

https://www.jsonschemavalidator.net/
https://json-schema-validator.herokuapp.com/

Which validator have you used that only ignores properties when they are named like x-...?

The JSON schema specification has always allowed additional properties. The rules are stated here:

https://json-schema.org/draft/2019-09/json-schema-core.html#rfc.section.4.3.1

Extension keywords, meaning those defined outside of this document and its companions, are free to define other behaviors as well.

A JSON Schema MAY contain properties which are not schema keywords. Unknown keywords SHOULD be ignored.

If you have a validator that rejects extension properties like javaType, then you need to raise a bug against it. There is no way we'd support aliases for all these properties without a hugely compelling reason.

[JSSAggie]

json schema is a big part of open api and one of the tools that fails validate an open API spec including the json schema is speccy. I don't see a problem supporting both javaType and x-javaType.

I see the open api specification does also refer to vendor extensions with x- but does say they should not be required to use the x-. I just see a ton of vendor extensions that continue to use the x- pattern because unfortunately not everyone plays nice.

https://swagger.io/docs/specification/openapi-extensions/

[rajpaulbagga]

@joelittlejohn ,
@JSSAggie is my colleague and he has answered with the tool that is giving us trouble.

You make a valid point and thank you for bringing that portion of the spec to my attention. The trouble is the Open API connection here. Its spec (https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#properties ) has this to say:

Additional properties defined by the JSON Schema specification that are not mentioned here are strictly unsupported.

And later notes extensions should be prefixed with "x-".

If I reverted the messaging changes (error reports and docs) and made the support more silent, would you consider taking the support that way?

[handrews]

@rajpaulbagga @JSSAggie @joelittlejohn

This will be addressed in the OpenAPI Specification 3.1, which will be fully compatible with JSON Schema draft 2019-09 (or possibly 2020-03ish if we do a small update to coordinate with the release of OAS 3.1). Compatibility will include allowing extensions that are not prefixed with x- (within the Schema Object- outside of the Schema Object, OpenAPI extension rules still apply).

It will also include the new vocabulary extension process which is intended to help validators be extensible so that they can do something other than just ignore non-standard properties.

[handrews]

@rajpaulbagga but yes, in OAS 3.0.x Schema Objects you need an x- prefix.

[joelittlejohn]

Thanks @handrews, that's great!

If anyone needs to do something in the meantime they can use the changes in this PR and build a custom version of this plugin.

[philsturgeon]

I think this is coming from a confusion between JSON Schema and OpenAPI. OpenAPI v3.0 is a subset/superset/sideset, with its own rules like no additional keywords.

speccy uses oas-kit, and oas-kit is spec compliant. The OpenAPI spec says your schema cannot have arbitrary keywords in there unless they start with an x-.

If a JSON Schema validator is complaining about your x-javaType or javaType keywords, that would be really odd, and should be fixed because JSON Schema allows arbitrary keywords.

If an OpenAPI validator is complaining about x-javaType keyword then that is also really odd, because it's part of the spec. If it is complaining about javaType then no fixed required, it is working as expected.

The divergence being fixed in v3.1 is music to my ears and will hopefully resolve all this confusion in all tooling everywhere once we're caught up.

[JSSAggie]

If a JSON Schema validator is complaining about your x-javaType or javaType keywords, that would be really odd, and should be fixed because JSON Schema allows arbitrary keywords.

So this tool requires you to use javaType but when we validate the same jsonschema with OpenAPI v3.0 it fails to validate because it does not like javaType. So now we are in a catch22

[philsturgeon]

Yes, you cannot use JSON Schema and OpenAPI tooling interchangeably. This is why we worked so hard on OAI/OpenAPI-Specification#1977 to solve that problem.

[JSSAggie]

So lets say that OAI/OpenAPI-Specification#1977 moves forward and it looks like it has. Where would the tooling be changed to be compliant? Would OpenAPI allow javaType thus this project can keep using it? It sounds like this is the path forward? If so, does it make sense to start working on that? Just trying to put our efforts in the right place.

[philsturgeon]

Would OpenAPI allow javaType

Yup!

OpenAPI v3.1-RC1 will hopefully be out soon, so upgrade all your tooling to OpenAPI v3.1 and JSON Schema 2019-09 as soon as you can.

[handrews]

Yeah, we (JSON Schema project) intend 2019-09 to be a solid base to build on- the next draft will just be some minor tidying based on feedback, and if possible timed to align with OAS 3.1 (since they are giving us some of the most detailed feedback right now).

So it's worth investing in 2019-09. If there are any backwards-incompatible changes in the next draft, it would only be because we messed some new feature up substantially and need to fix it before OAS 3.1 can adopt it. There's no substitute for production, so the more people work on implementing 2019-09 and get feedback to us, the better the draft for OAS 3.1 will be.

@philsturgeon and I put in a lot of effort over the past year with the OpenAPI TSC folks to ensure that we'll all be back in alignment with OAS 3.1. Both projects are committed to that goal.