fix(docs): fix mdx loader cache invalidation bug on versions changes #10934
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ [V2]
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
Size Change: +307 B (0%) Total Size: 11.1 MB ℹ️ View Unchanged
|
⚡️ Lighthouse report for the deploy preview of this PR
|
|
Size Change: +297 B (0%) Total Size: 11.5 MB ℹ️ View Unchanged
|
|
Unfortunately this fix prevents Rspack persistent cache from optimizing any MDX file of the docs plugin, so I'm temporarily disabling it (making it opt-in with secret env var) in #10931 We'll re-enable this later once Rspack fixes its bug web-infra-dev/rspack#9413 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
We have a bundler cache invalidation problem, that I discovered while working on #10931
The following works:
But the following fails with broken link errors:
This is because our docs plugin has:
This option affects the URLs of the versions/docs through an env variable, which in turn affect the MDX loader
resolveMarkdownLink.Depending on env variables,
@site/docs/my-doc.mdmight resolve to:Problem, the MDX loader cache entries are not invalidated when this happens, leading to "reusing" stale cache hits.
Our MDX loader uses
this.cacheable(true)(default) but is non-deterministic and the same input doesn't always lead to the same output (depending on plugin/loader options).https://webpack.js.org/api/loaders/#thiscacheable
Most options are passed through
docusaurus.config.js, and this file is declared as a WebpackbuildDependency: changing it will invalidate the whole cache.However, it remains possible to change plugin options without modifying
docusaurus.config.js. For example, when using env variables, the config file is not updated and the site cache isn't invalidated. And there are even more ways to have a stale cache that doesn't invalidate properly, such as cutting a new docs version.Unfortunately, Webpack loader API is not super convenient to help us invalidate its cached entries.
I thought entries would get invalidated automatically on options change, but it doesn't look to be the case. Also, there's no
cache.versionon the loader like there is globally, and we need to write a file to disk and add it tothis.addDependency(file)so that our MDX files are evicted from the cache.The solution I'm adding is to add a new mdx-loader
dependenciesobject to trigger cache invalidations, and pass it a__mdx-loader-dependency.jsonfile that will contain anything that should lead to cache invalidation (using plugin options and versions).This solves the problem on the docs plugin. The persistent cache still works, when options/versions don't change, but invalidates properly. The repro case above now works without error:
I'm not 100% sure this is the perfect solution, but at least it solves the problem above. This PR is mostly here to document the problem and get it out of my head so that we can come back to it later.
We'll figure out a way to generalize this solution to our other plugins, and how to help third-party plugin authors ensure the cache gets invalidated properly too.
There are likely many other edge cases to consider:
siteConfig.markdownoptions with env variablesfs.readFileindocusaurus.config.jsand use the file content to alter the Docusaurus site configMost likely, we need to expose APIs to the end user to be able to control cache invalidation on their own, because it's impossible to cover all cases automatically.
Fortunately,
docusaurus clearremains a good workaroundTest Plan
local, not easy to cover in CI 😅
Test links
https://deploy-preview-10934--docusaurus-2.netlify.app/
https://deploy-preview-10934--docusaurus-2.netlify.app/docs/api/docusaurus-config/#future
Related issues/PRs
#10931