Mailchimp Catalog has unfollowed $refs

[cgardens]

Expected Behavior

• The mailchimp catalog should be fully resolved and not include any jsonschema $ref tags

Current Behavior

• Not all of the $ref tags in the schema are followed.
  e.g. in the Lists stream we have things like this.

          "delivery_status": {
            "$ref": "https://us1.api.mailchimp.com/schema/3.0/Definitions/DeliveryStatus.json"
          },
          "_links": {
            "$ref": "https://us1.api.mailchimp.com/schema/3.0/Definitions/HATEOASLinks.json"
          }

• We can either copy a static copy of the catalog into the code base with all of the $refs resolved or we can continued to pull from mailchimp but follow the $ref tags.

This causes a problem right now, because we started enforcing that field names only have alphanumeric characters and underscores so the $ in $ref breaks things. But even without enforcing this, we should not leave $refs in the catalog.

[eugene-kulak]

Hi, @cgardens, could you please help me to understand what exactly the limitation here with $ref? Assuming, that we can modify the rule for field names.

[cgardens]

Yeah, for sure! We expect the json that is returned by the discover method to only contain keys that are alphanumeric or _. So special symbols like "$" and "/" are not allowed.

In JsonSchema $ref is used in a schema declaration to point to a schema that an object depends on from another file. So instead of leaving the reference to the other file in the object that is returned by discover, we need the object itself to be included.

I'll do a simplified example.
parent.json:

  {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Parent",
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "children": {
        "items":{
          "$ref": "child.json#"
        }
      }
    }
  }

And then child.json:

  {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Child",
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      }
    }
  }

If this schema were being returned by discover, we would need it to be returned as follows with the reference to child replaced by the actual schema itself:

  {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Parent",
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "children": {
        "items": {
          "$schema": "http://json-schema.org/draft-07/schema#",
          "title": "Child",
          "type": "object",
          "properties": {
            "name": {
              "type": "string"
            }
          }
        }
      }
    }
  }

Does that help?

[eugene-kulak]

@cgardens Thanks, yes, now it is clear.
I was asking because python implementation of json schema validator follows references without any problem, but I understand this could be an issue if we reference local file in docker container and then validate it from host machine or another container.