Tooling plans and technical proposal

[fmvilas]

What?

I wrote a draft of a technical proposal for tooling and a tentative roadmap, as well as a strategy to succeed.

Please, review and leave your comments.

[MikeRalphson]

LGTM, comments are for discussion, not necessarily to make changes.

[MikeRalphson]

What is the skill level for Go within the contributor-base?

I like the idea of a core implementation with wrappers, and Go may be a good choice. There are JSON-Schema implementations (http://json-schema.org/implementations.html#validator-go) which may help us.

Has, however, Rustlang been considered? It can be made provably safe, doesn't have Go's package -dependency-on-github problems and generates significantly smaller code (Hello World is 2.0Mb in Go, 1.2Mb stripped, in Rust 251Kb/187Kb)?

https://doc.rust-lang.org/1.2.0/book/rust-inside-other-languages.html

To be fair, Rust may lack popular JSON Schema validators.

Either language would be a learning opportunity (!) for me.

[MikeRalphson]

Actually, there are several Rust crates which look interesting...

https://github.com/search?q=rust+json+schema
https://github.com/search?q=rust+yaml
https://github.com/search?q=rust+avro
https://github.com/search?q=rust+protobuf

[rmelian]

I just started learning Golang a couple of days ago. I think the main reason for using Golang is the capability to generate C shared libraries: many languages have a way to call native C libraries.

[fmvilas]

My experience with Go is short. I think Go is a good choice for performance reasons and it's also having an increasing adoption at companies, as well as a great community.

Not saying with this that Rust is not a great language. Probably is, I don't know 😄. And that's precisely my concern, that I haven't heard much of it yet.

[derberg]

good thing about go is that we use it in Kyma as the only allowed backend language, for many reasons, but of course one is that it is efficient and native for k8s. This means we can support you here. We need asyncapi validator inside Kyma so we can always make sure valid spec is coming in and during registration tell the operator that he has invalid spec. So we had it on the roadmap anyway, I can speed it up for march, do it for 1.2 version first and then support 2.0. This way we give more than just asyncapi-react but also the validator with the go expertise that might help ya with further development.

[MikeRalphson]

Thanks @fmvilas @rmelian @derberg one thing I wanted to mention in yesterday's call, but we ran out of time, is that I'm happy to Go with Go 😄 but I feel raising the issue of an alternative helped to clarify and explain and document the decision-making process behind the choice of backend language, and I think that's a good thing.

[fmvilas]

haha! I hope we don't get rusty going with Go.

(Someone stop this bad jokes hell, please)

[MikeRalphson]

Is there a link to the 'playground'? Does it allow creation / modification of JSON/YAML AsyncAPI definitions like the current editor?

[rmelian]

here you go https://www.asyncapi.com/asyncapi-react/ @derberg can better answer that :)

[rmelian]

@MikeRalphson I repeat here the same question but for you :) What is your skill level with Rust??

[MikeRalphson]

Total beginner. I made a few hacks to the Google Gnostic project, so I've actually done more with Go so far, but I like the sound of Rust.

[derberg]

so far you can use it to write YAML/JSON based asyncapi spec and see how it is rendered, you can also fetch external resource. For now you manually have to click refresh button but because we improved error handling I'm not sure it is still needed, this can be improved. I'm super ok with exposing it as official editor, but we probably would have to discuss details first, like what needs to change in the playground? for example if it is supposed to be an official editor, would you like to hide the Theme and Configuration tabs that are specific for the asyncapi-react component only?

[fmvilas]

@derberg Let's discuss the requirements later. I just wanted to communicate that it's on the roadmap.

[MikeRalphson]

@rmelian yes, Rust also has this property.

[rmelian]

@MikeRalphson allow me to spent 5 mins reading about RUST :)

[MikeRalphson]

Have 10!

[rmelian]

I read here https://matthias-endler.de/2017/go-vs-rust/ Rust has a steeper curve than Golang

[MikeRalphson]

@rmelian interesting article. It twice mentions C interoperability and FFI as strengths of Rust and never mentions the subject for Go...

From here:

Rust was designed to interoperate with C. Go can as well, but gives up a lot to achieve this goal, and it’s not really its focus.

I'm not sure yet what is meant by "gives up a lot"...

Rust would also give us a way into compiling to web-assembly for browser builds.

[rmelian]

I feel we would be more productive with Golang than with Rust, as @MikeRalphson has some basics, also @fmvilas. In my case, I already started learning it so If go with Rust(I only know the name) it will take us a lot more time to do something real with it.

[fmvilas]

I see Rust as an option to keep an eye on for future implementations. I'm not worried so much about the capabilities of the language we choose. Almost every language offers what we want to achieve. However, if we want to invite people to contribute to AsyncAPI, Rust will make it harder. Not because of the language, but because the community and adoption are not there yet. I'm conscious this is a chicken-egg problem: if nobody uses Rust, the community will not grow and it will make others decide not to choose it.

That said, the parser is such a key piece in the tooling I don't feel comfortable trying something that's not widely adopted. However, there will be other pieces of software where we may want to use Rust, like schema format parsers. Nothing prevents us from starting with them and eventually migrating the parser to Rust if that's what we want.

[derberg]

good thing about go is that we use it in Kyma as the only allowed backend language, for many reasons, but of course one is that it is efficient and native for k8s. This means we can support you here. We need asyncapi validator inside Kyma so we can always make sure valid spec is coming in and during registration tell the operator that he has invalid spec. So we had it on the roadmap anyway, I can speed it up for march, do it for 1.2 version first and then support 2.0. This way we give more than just asyncapi-react but also the validator with the go expertise that might help ya with further development.

[derberg]

What do you mean here? is it about parsing asyncapi JSON input for example into Avro?
shouldn't validation be separated from parsing, I mean that these are actually 2 different packages?

[derberg]

or am I missing some basic understanding here? for me validation of spec against schema is always spec as input document, schema as the schema that document must be validated against and that is it. So I see it in the way that you pass the document, from the document we know if it is following 1.2 or 2 and validate against https://github.com/asyncapi/asyncapi/blob/master/versions/1.2.0/schema.json for example. Like in this super simple library https://github.com/xeipuuv/gojsonschema

[fmvilas]

Yes, but since we're introducing different schema/message formats like Avro and Protobuf, a JSON Schema validator is not enough. Check out this issue for context: #113.

[derberg]

ok, so if I get it right asyncapi org would maintain asyncapi schema in different formats? deliver for example json schema and then also transform and officially support other formats? sorry for dumb questions but I want to make sure I fully get it

[fmvilas]

Not the whole document, just the message payloads and headers. The following is an example using protobuf.

asyncapi.yaml

asyncapi: '2.0.0'
info:
  title: 'My API'
  version: '1.0.0'

channels:
  user/created:
    subscribe:
      headers:
        application/x-protobuf:
          $ref: 'path/to/user-created.proto#Headers'
      payload:
        application/x-protobuf:
          $ref: 'path/to/user-created.proto#UserCreated'

user-created.proto

message Headers {
  string my-app-header = 1;
}

message UserCreated {
  int32 id = 1;
  string name = 2;
  string email = 3;
}

Notice the whole document remains the same. It's just the message (publish/subscribe) part that changes. The asyncapi.yaml example above would be equivalent to this one:

asyncapi: '2.0.0'
info:
  title: 'My API'
  version: '1.0.0'

channels:
  user/created:
    subscribe:
      headers:
        application/schema+json;version=draft-07:
          type: object
          properties:
            'my-app-header':
              type: string
      payload:
        application/schema+json;version=draft-07:
          type: object
          properties:
            id:
              type: integer
            name:
              type: string
            email:
              type: string

Does it make sense?

[derberg]

I guess you mean here that you want to have ready out of the box npm package for example that already wraps the validator for example and you can easily use it in node app?

[fmvilas]

Yes, a simple package that imports the native binding and provides the same interface, so users don't have to do it. Just a basic package.

[derberg]

instead of writing something from scratch wouldn't it make sense to reuse some tools like https://yeoman.io for example (sample generator https://github.com/cbornet/generator-jhipster-swagger-cli). Yeoman looks like dead project but maybe because there is no need to add anything new and projects is completed because generators are added by community constantly https://yeoman.io/generators/

if not yeoman, maybe something else is there

[fmvilas]

Agree yeoman is cool for CLI, but we should also offer non-CLI code generation. @rmelian pointed out to reusing some parts of Swagger Codegen. We can actually use both as they are totally different things.

[derberg]

ok, so you mean to support also dynamic code generation, makes sense, I think it is common for node.js apps. I suggest to make it clear in the description

[fmvilas]

I didn't go forward in the description because the requirements are still not clear. Only the parser requirements are already defined. The rest are there for "road mapping" purposes. We'll get into implementation details afterwards.

[derberg]

yeap, pretty clear it must happen, I just need to understand what changes are coming so we can properly address it

[fmvilas]

I'll work with you on that.

[derberg]

so far you can use it to write YAML/JSON based asyncapi spec and see how it is rendered, you can also fetch external resource. For now you manually have to click refresh button but because we improved error handling I'm not sure it is still needed, this can be improved. I'm super ok with exposing it as official editor, but we probably would have to discuss details first, like what needs to change in the playground? for example if it is supposed to be an official editor, would you like to hide the Theme and Configuration tabs that are specific for the asyncapi-react component only?

[derberg]

not sure how validation would work here, if user can put anything as x- this is fine to parse and and not fail validation if those extensions exist, but how would we validate the data itself?

[fmvilas]

We don't perform validation but instead, we let users provide our parser with extra validation.

Example:

Imagine we have an extension for Twitter:

info:
  x-twitter: @fmvilas 

const twitterPaserAndValidator = require('twitter-parser-and-validator');
const parsedAndValidatedDocument = asyncapiParser.parse(asyncapiDocument, [twitterPaserAndValidator]);

Here the twitter-parser-and-validator will not be developed by us. This package will take care of validating that the x-twitter value actually looks like @username.

However, if no twitter-parser-and-validator is provided, it shouldn't fail.

Does it make sense?

[derberg]

now I got it, make a lot of sense, thanks

[derberg]

ok, so if I get it right asyncapi org would maintain asyncapi schema in different formats? deliver for example json schema and then also transform and officially support other formats? sorry for dumb questions but I want to make sure I fully get it

[derberg]

ok, so you mean to support also dynamic code generation, makes sense, I think it is common for node.js apps. I suggest to make it clear in the description

[derberg]

now I got it, make a lot of sense, thanks

[asyncapi-bot]

🎉 This PR is included in version 1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[derberg]

It is actually not included in the release. This was an unintentional release caused by the release automation initial run. 1.0 version of the spec was released a long time ago. The release mentioned by the bot was removed.