Adds support for running native binaries without JS intermediaries

[arcanis]

What's the problem this PR addresses?

Yarn currently cannot run native binaries without going through a JS jumper script. Other package managers don't have this restriction, and it makes the yarn run overhead worse on some scripts for little reasons - or doesn't work at all when packages don't use jumper scripts.

Fixes #882

How did you fix it?

Two mechanisms are used:

• First we check for the binary extension
• Then we check its magic number

If one of the two match a predetermined list, the binary is spawned without going through Node. This ensures that this logic is called only when the binary is truly a native binary, and will not affect the behaviour of other existing scripts.

Checklist

• I have read the Contributing Guide.

• I have set the packages that need to be released for my changes to be effective.

• I will check that all automated PR checks pass before the PR gets reviewed.

[merceyz]

Presumably we'll need to do this for the following locations as well:

berry/packages/yarnpkg-core/sources/scriptUtils.ts

Lines 491 to 495 in ca47575

	await Promise.all(
	Array.from(await getWorkspaceAccessibleBinaries(workspace), ([binaryName, [, binaryPath]]) =>
	makePathWrapper(binFolder, toFilename(binaryName), process.execPath, [binaryPath]),
	),
	);

berry/packages/yarnpkg-core/sources/scriptUtils.ts

Lines 544 to 548 in ca47575

	await Promise.all(
	Array.from(await getPackageAccessibleBinaries(locator, {project}), ([binaryName, [, binaryPath]]) =>
	makePathWrapper(binFolder, toFilename(binaryName), process.execPath, [binaryPath]),
	),
	);

berry/packages/yarnpkg-core/sources/scriptUtils.ts

Lines 741 to 745 in ca47575

	await Promise.all(
	Array.from(packageAccessibleBinaries!, ([binaryName, [, binaryPath]]) =>
	makePathWrapper(env.BERRY_BIN_FOLDER as PortablePath, toFilename(binaryName), process.execPath, [binaryPath]),
	),
	);

[merceyz]

There are some issues that this probably fixes (for example #882), could you check and link to them so they're closed?

[belgattitude]

There are some issues that this probably fixes (for example #882), could you check and link to them so they're closed?

Not sure about this PR, but from the title it might fix some turborepo hacks: https://github.com/vercel/turbo/blob/main/packages/turbo/install.js#L190

So it might also relate to

• [Feature] Improve performance of yarn run #2575
• Restrict running NPM scripts with monorepo tools #5255

[jmatsushita]

I've tried yarn set version 4.0.0-rc.47 with expo@9.0.0-beta.0 in a monorepo but it doesn't fix it.

> yarn workspace client expo info                                                                                                       .../node_modules/expo/bin/cli:3
# Forward all arguments to the local CLI (@expo/cli).
^

SyntaxError: Invalid or unexpected token
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1176:20)
    at Module._compile (node:internal/modules/cjs/loader:1218:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47

Node.js v18.16.0

[arcanis]

This fix is only about native binaries. From your logs, expo is a shellscripts, which aren't allowed in bin at the moment for the reasons outlined in #882 (comment).

[MorisR]

Will the changes be also added to yarn 3?
we're still stuck on node v16 and yarn 4 requires node v18

[merceyz]

Yes, I've backported it now so it will be in the next release.