feat(tools): add import path check in docs

[schwarzkopfb]

Add tool to check import paths in JSDoc comments and readme files across std to help standardize them as suggested in #2772. Enforcing that all imports in examples are from https://deno.land/std@$STD_VERSION/*

[iuioiua]

Nice! I'm not a maintainer, but a few concerns:

1. Is the CI step you were supposed to add be deno task lint:doc-imports instead of deno task lint:deprecations?
2. Could walk() from std/fs replace your walk()?
3. IMO, regex is fairly complex. Is there a simpler alternative?

[schwarzkopfb]

@iuioiua Thanks for the great suggestions!

1. that was a dumb mistake, thanks for catching it
2. you're absolutely right, std's built-in walk() is perfectly fine and makes the code much cleaner
3. I'll spend some time to think/experiment on how could that be simplified, probably there is room to cut off some parts of that

[schwarzkopfb]

@iuioiua I came up with some changes based on your comments. WDYT?

[cjihrig]

This looks like something that will be nice to have. My only concerns are:

• The complexity of the regular expressions if we need to do any maintenance on them. I'm not sure how much we can do about that though.
• Since there is nothing to report in the codebase right now, I guess we just have to take your word for it that it works based on your screenshots 😄

[timreichen]

• The complexity of the regular expressions if we need to do any maintenance on them. I'm not sure how much we can do about that though.

I think the solution is to use a proper ast tree walker. That would also be great to establish for future tools.

The typescript module works for creating and walking ast trees, but there is also deno_ast, which is a rust mod but could (and imo should) be made runnable in deno scripts, like deno_doc is.

So the task could be done like so:
deno_doc: get jsdoc comments -> deno_ast: walk code from comments, find and verify imports, log warnings

[iuioiua]

@iuioiua I came up with some changes based on your comments. WDYT?

Looking better! However, I'm still wondering whether countChecked is needed...

[schwarzkopfb]

@timreichen I implemented the AST-based approach using the typescript module. I think it's good for a first iteration until denoland/deno_ast#120 gets resolved. Later we can upgrade to the Rust-based, more performant AST generation subsystem.

@cjihrig I added some tests so we have a better proof than just my words for that it's working. 😀 On windows it's failing for some weird, file path related issue. I'll investigate that soon, but until then FIXED I'd appreciate any feedback on the implementation.

@iuioiua The counter is only used to display a message on exit which informs the user about that something happened, but no error raised in mora handy way than just exiting with 0 or a non-zero exit code. Just like deno fmt or deno lint does. Also can be useful when someone adjusts the EXCLUDE_PATHS and can easily see a stat on how many files included in the scan.

[iuioiua]

Also can be useful when someone adjusts the EXCLUDE_PATHS and can easily see a stat on how many files included in the scan.

In this case, seeing a change in the number of files justifies the existence of the variable, IMO. Well explained.

[iuioiua]

Would it perhaps be better to use Deno.spawn over Deno.run? That's where things are moving towards.

[schwarzkopfb]

Would it perhaps be better to use Deno.spawn over Deno.run? That's where things are moving towards.

Great suggestion, makes it a bit less messy, applied. Thanks!

[schwarzkopfb]

@timreichen FYI: I also made an experiment which is using deno_doc for jsDoc parsing that produced surprising results. It is orders of magnitude slower even with a smaller set of files to work with. And it breaks if I run the scan on the whole std repo. So for now I stick with the regexp for identifying comments and extracting code blocks from it and then using the AST generated by the TypeScript compiler to check import statements.

[kt3k]

LGTM