Auto-generate more of the AST representation

[dcreager]

In #15544 we added a script to auto-generate large parts of the Rust data model that we use to store the AST of parsed Python code. That script consumes a TOML file, which describes all of the possible syntax nodes (e.g., StmtIf, ExprBinOp) and any groups those nodes belong to (e.g. Stmt, Expr). The details of each syntax node are still defined manually in Rust.

We could go further with auto-generation, with existing art that we could build on. rust-analyzer uses ungrammar, while Python itself uses ASDL (asdl, parser, codegen). This would eliminate even more tedious hand-written Rust code — not just the struct/enum definitions themselves, but even things like the visit_source_order methods for each syntax node. It would also allow us to experiment more easily with other internal representations for the parsed AST — such as using IndexVec to store the syntax node content (as alluded to in #12419 (comment)).

[Glyphack]

Hey, I'd like to help with this and the linked issue.
First I went through the original PR and tried resolving one of the comments to see how the code generation works(#16144).

I'll look into other files and see what other stuff can we auto generate with the current information.

I want to continue with the implementation, I appreciate your help with a few questions:

1. Is the goal use ASDL for generating the AST node structs and enums? If I'm not wrong by using ASDL we don't need the ast.toml anymore. So should we still keep ast.toml?
2. Is the plan to use ungrammar only for generating visit_source_order for nodes? Or the mention of ungrammar is for something else?

Update: I decided to extend the toml file you created. I just borrowed some names from ASDL to make it possible to generate AST nodes.