Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script parser #4613

Open
wants to merge 39 commits into
base: master
Choose a base branch
from
Open

Script parser #4613

wants to merge 39 commits into from

Conversation

bentsherman
Copy link
Member

@bentsherman bentsherman commented Dec 21, 2023

Implements the strict syntax for Nextflow scripts, using the shared "compiler" module from the language server.

To use the strict syntax, simply set NXF_ENABLE_STRICT_SYNTAX=true in your environment when running nextflow.

This approach allows us to control the parsing process -- including the syntax and detecting syntax errors -- while still leveraging the Groovy compiler for execution. In other words, we can define whatever grammar we want, as long as we can "compile" it into a Groovy AST. If you look at ScriptToGroovyVisitor, you'll see that it converts processes / workflows / includes into the same Groovy AST produced by NextflowDSLImpl.

NOTE: While this PR uses Jitpack to load the shared module, this dependency should be inverted before the release of 25.04. That is, eventually the compiler module should reside in Nextflow and the language server should consume it.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
…cess inputs/outputs

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Copy link

netlify bot commented Dec 21, 2023

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 9052d80
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/67b8f0b65f9e340008bddc94

@bentsherman
Copy link
Member Author

Some sweets-infused holiday thoughts... right now I am just producing the same AST expected by the runtime to keep this PR as simple as possible. But, like I said, we can produce whatever Groovy AST we want, so we could produce Groovy code that more effectively enables new features like static types, default arguments, etc.

The main example I'm thinking of is the annotation API (see nextflow-io/rnaseq-nf#24). I originally designed it as user-facing code, but it could also be an intermediate representation that is produced by the parser. If we "compile" the process and workflow definitions to actual function definitions, then we can more easily leverage the Groovy type checking.

This is just an example. We may not need the annotation API exactly, but it would be good to explore alternative AST representations, perhaps in a second iteration.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman
Copy link
Member Author

Another aside... GraalVM implements an AST model for every language that it supports. Here is the Graal Python AST source code.

So we could also have the parser produce a Graal/Python AST and thereby allow the pipeline code to use Python semantics instead of Groovy semantics.

We would need to design a DSL syntax for processes and workflows that would make sense with Python. Likely it would look more like Snakemake. Using native Python syntax (i.e. functions with decorators) is also an option but would likely be more verbose. We would still need to implement our own IDE tooling, but centered around Python syntax instead of Groovy syntax.

The point is, if we rely on the semantics (and compiler backend) of an existing language, it doesn't have to be Groovy. It could easily be any language supported by GraalVM.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman bentsherman changed the title Nextflow parser Formal grammar and parser Apr 29, 2024
@bentsherman bentsherman changed the title Formal grammar and parser Script parser Jul 25, 2024
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Copy link
Member

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, most the comments overlaps with the config parser PR

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman
Copy link
Member Author

Success! Tests are passing 😄

As a bonus, you can use nextflow inspect to check for errors without running:

NXF_ENABLE_STRICT_SYNTAX=true nextflow inspect <pipeline>

Anyway, let's merge the config parser before this one.

@bentsherman bentsherman marked this pull request as ready for review February 21, 2025 23:17
@bentsherman bentsherman requested a review from a team as a code owner February 21, 2025 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants