[Bug]: Test runner doesn't find stories when using custom function

[adrianbruntonsagecom]

Describe the bug

I'm using a custom function to define the StoriesEntry array that defines where stories are to be found. However, the storybook test runner doesn't appear to be working alongside this as I receive an error when running the test runner:

> test-storybook --failOnConsole --url http://localhost:6006

[test-storybook] Could not find stories in main.js in ".storybook".
If you are using a mono-repository, please run the test-runner only against your sub-package, which contains a .storybook folder with "stories" defined in main.js.
You can change the config directory by using --config-dir <path-to-dir>

Error: Could not find stories in main.js in ".storybook".
If you are using a mono-repository, please run the test-runner only against your sub-package, which contains a .storybook folder with "stories" defined in main.js.
You can change the config directory by using --config-dir <path-to-dir>
    at getStorybookMain (C:\Projects\node_modules\@storybook\test-runner\dist\test-storybook.js:24451:11)
    at getStorybookMetadata (C:\Projects\node_modules\@storybook\test-runner\dist\test-storybook.js:24464:17)
    at main (C:\Projects\node_modules\@storybook\test-runner\dist\test-storybook.js:25010:47)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Reproduction link

https://stackblitz.com/edit/github-y1neym5w

Reproduction steps

I've not been able to run the test runner against a stackblitz URL, but the above code should be the replication.

1. Download the code above
2. Revert the comment in main.ts so the standard stories glob is used
3. Run npm run storybook
4. Storybook starts and you'll see the usual storybook structure
5. Run npx test-storybook --url http://localhost:6006/
6. Tests pass ✅
7. Stop storybook server
8. Revert the commenting change from Add support for custom webpack config #2 above so this is used with the function I've pasted below: stories: async () => [...(await getStoryEntries())],
9. Run npm run storybook
10. Storybook starts and the stories will be beneath a "fake/directory" path
11. Run npx test-storybook --url http://localhost:6006/
12. Tests fail with error below ❌

> test-storybook --failOnConsole --url http://localhost:6006

[test-storybook] Could not find stories in main.js in ".storybook".
If you are using a mono-repository, please run the test-runner only against your sub-package, which contains a .storybook folder with "stories" defined in main.js.
You can change the config directory by using --config-dir <path-to-dir>

Error: Could not find stories in main.js in ".storybook".
If you are using a mono-repository, please run the test-runner only against your sub-package, which contains a .storybook folder with "stories" defined in main.js.
You can change the config directory by using --config-dir <path-to-dir>
    at getStorybookMain (C:\Projects\node_modules\@storybook\test-runner\dist\test-storybook.js:24451:11)
    at getStorybookMetadata (C:\Projects\node_modules\@storybook\test-runner\dist\test-storybook.js:24464:17)
    at main (C:\Projects\node_modules\@storybook\test-runner\dist\test-storybook.js:25010:47)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

This is the function used which is a simpler version of what I'm actually intending to use:

async function getStoryEntries() {
  // Do an async thing - my real code uses the value of this
  await fsAsync.readdir(path.resolve(__dirname));

  return [
    {
      directory: '../src/stories',
      titlePrefix: 'fake/directory/',
    },
  ];
}

System

From the replication above:
Storybook Environment Info:

  System:
    OS: Windows 11 10.0.22631
    CPU: (8) x64 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
  Binaries:
    Node: 18.20.5 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.22 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 10.8.2 - C:\Program Files\nodejs\npm.CMD <----- active
  Browsers:
    Edge: Chromium (127.0.2651.74)
  npmPackages:
    @storybook/addon-docs: ^8.6.0-beta.4 => 8.6.0-beta.4
    @storybook/addon-essentials: ^8.6.0-beta.4 => 8.6.0-beta.4
    @storybook/addon-interactions: ^8.6.0-beta.4 => 8.6.0-beta.4
    @storybook/addon-onboarding: ^8.6.0-beta.4 => 8.6.0-beta.4
    @storybook/angular: ^8.6.0-beta.4 => 8.6.0-beta.4
    @storybook/blocks: ^8.6.0-beta.4 => 8.6.0-beta.4
    @storybook/test: ^8.6.0-beta.4 => 8.6.0-beta.4
    @storybook/test-runner: ^0.21.0 => 0.21.2
    storybook: ^8.6.0-beta.4 => 8.6.0-beta.4

Or from my codebase:
Storybook Environment Info:

  System:
    OS: Windows 11 10.0.22631
    CPU: (8) x64 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
  Binaries:
    Node: 18.20.5 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.22 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 10.8.2 - C:\Program Files\nodejs\npm.CMD <----- active
  Browsers:
    Edge: Chromium (127.0.2651.74)
  npmPackages:
    @storybook/addon-actions: ^8.5.3 => 8.5.3
    @storybook/addon-essentials: ^8.5.3 => 8.5.3
    @storybook/addon-interactions: ^8.5.3 => 8.5.3
    @storybook/addon-links: ^8.5.3 => 8.5.3
    @storybook/angular: ^8.5.3 => 8.5.3
    @storybook/test: ^8.0.8 => 8.5.3
    @storybook/test-runner: ^0.21.0 => 0.21.0
    storybook: ^8.5.3 => 8.5.3

Additional context

The purpose of the story function is to flatten the directory structure displayed so that whilst it is similar to the file structure, it has fewer levels.

[greptile-apps]

** Disclaimer** This information might be inaccurate, due to it being generated automatically
The test runner currently only supports static story configurations in main.js/ts. To support dynamic story configurations: 1. Modify @storybook/test-runner/src/getStorybookMetadata.ts to handle async story configurations: ts export async function getStorybookMain() { const main = await loadMainConfig(); const stories = typeof main.stories === 'function' ? await main.stories() : main.stories; return { ...main, stories }; } 2. As a workaround until this is implemented, you can write the dynamic story paths to a temporary main.ts that the test runner can read: ts // .storybook/main.ts const storiesForTestRunner = await getStoryEntries(); await fs.writeFile( '.storybook/main.test.ts', `export default { stories: ${JSON.stringify(storiesForTestRunner)} }` ); Then run the test runner with: test-storybook --config-dir .storybook/main.test.ts

About Greptile

This response provides a starting point for your research, not a precise solution.

Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

Edit Issue Bot Settings · Greptile

[adrianbruntonsagecom]

** Disclaimer** This information might be inaccurate, due to it being generated automatically The test runner currently only supports static story configurations in main.js/ts. To support dynamic story configurations: 1. Modify @storybook/test-runner/src/getStorybookMetadata.ts to handle async story configurations: ts export async function getStorybookMain() { const main = await loadMainConfig(); const stories = typeof main.stories === 'function' ? await main.stories() : main.stories; return { ...main, stories }; } 2. As a workaround until this is implemented, you can write the dynamic story paths to a temporary main.ts that the test runner can read: ts // .storybook/main.ts const storiesForTestRunner = await getStoryEntries(); await fs.writeFile( '.storybook/main.test.ts', `export default { stories: ${JSON.stringify(storiesForTestRunner)} }` ); Then run the test runner with: test-storybook --config-dir .storybook/main.test.ts

About Greptile

This response provides a starting point for your research, not a precise solution.

Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

Edit Issue Bot Settings · Greptile

This looks useful. I'll try this and report back. However I'll keep this open as I still think it's a bug and couldn't find anything similar to it already logged.

EDIT: No, this doesn't appear to work although I'm not certain why.

I've made my function non-async so it can sit on a top level:

function getStoryEntries() {
  // Do an async thing - my real code uses the value of this
  fs.readdirSync(path.resolve(__dirname));

  return [
    {
      directory: "../src/stories",
      titlePrefix: "fake/directory/",
    },
  ];
}

I've added this:

const storiesForTestRunner = getStoryEntries();
fs.writeFileSync(
  ".storybook/main.test.ts",
  `export default { stories: ${JSON.stringify(storiesForTestRunner)} }`
);

And whilst that file is created, when I run npx test-storybook --url http://localhost:6006/ --config-dir .storybook/main.test.ts it doesn't seem to find the config.

$ npx test-storybook --url http://localhost:6006/ --config-dir .storybook/main.test.ts
[test-storybook] Could not load main.js in .storybook/main.test.ts. Is the ".storybook/main.test.ts" config directory correct? You can change it by using --config-dir <path-to-dir>

Error: Could not load main.js in .storybook/main.test.ts. Is the ".storybook/main.test.ts" config directory correct? You can change it by using --config-dir <path-to-dir>
    at getStorybookMain (C:\Users\adrian.brunton\Downloads\github-y1neym5w\node_modules\@storybook\test-runner\dist\test-storybook.js:24448:11)
    at getStorybookMetadata (C:\Users\adrian.brunton\Downloads\github-y1neym5w\node_modules\@storybook\test-runner\dist\test-storybook.js:24464:17)     
    at main (C:\Users\adrian.brunton\Downloads\github-y1neym5w\node_modules\@storybook\test-runner\dist\test-storybook.js:25010:47)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

EDIT 2: After more debugging, I think this issue is config-dir needs to be a directory rather than a file and it must find a main.ts file within it. After moving some files around, I think I have a working solution. Again, I'll keep this open for it to be fixed without a workaround.