[Bug] Some bin scripts not working in CI environment with nodeLinker: node-modules

[bhishp]

Thank you for your work on yarn and on-going improvements. Caveat, I am just upgrading to yarn 2 so this may be down to some naïvety on my part. Thank you in advance for your help.

• I'd be willing to implement a fix

Describe the bug

I am doing a migration from yarn1 to yarn2 for my workspaced monorepo. I have upgraded the version to yarn-berry and still only using nodeLinked: node-modules. The problem I'm having is that some of my existing scripts that execute packages from the bin directory do not work when they are in a CI environment, however they do work locally.

Example script

"lint": "eslint --ext js,ts,tsx src --max-warnings=0"

Locally this executes as expected, on CI the output is: command not found: eslint. Note, this has also happened with a script that uses ts-node, so is not exclusive to eslint.

Workaround

I have verified that the eslint package is actually being installed on the CI environment and it seems I can get around this by invoking the bin directly, as such:

"lint": "../../../node_modules/.bin/eslint --ext js,ts,tsx src"

This is a sub-optimal solution for a few reasons but proves the lint package is there in bin and able to be executed. It feels like yarn is unable to resolve to the bin directory for some reason, maybe because of workspaces?

Only some packages

As I mentioned, this doesn't seem to happen for everything, we also have a prettier script that runs which is working fine:

"pretty-check": "prettier --config ./.prettierrc --list-different './**/*.{js,json,ts,tsx,md,yml}'"

My understanding is that bin scripts like this need to be treated differently PnP linker is being used, but I would think there is no impact if still using node-modules?

To Reproduce

Minimum reproduction repo available here: https://github.com/bhishp/yarn-2-workflow-bin-repro

I fresh-installed yarn 2 with a workspace config and two create-react-app packages and implemented some of the scripts mentioned above. Locally all the scripts work but once in CI environment the problems occur.

Failing build with lint scripts: https://github.com/bhishp/yarn-2-workflow-bin-repro/runs/1776495972?check_suite_focus=true

Failing build with ts-node script (though explicitly added as a dependency): https://github.com/bhishp/yarn-2-workflow-bin-repro/runs/1776516171?check_suite_focus=true

Failing build with prettier script (though explicitly added as a dependency): https://github.com/bhishp/yarn-2-workflow-bin-repro/runs/1776471730

☝🏽 This prettier script is working in my actual project but failing in this reproduction

Screenshots

Workflow output of failed script, command not found: eslint:

Environment if relevant (please complete the following information):

• OS: GitHub workflows runs-on: ubuntu-latest
• Node version: 14.15.1
• Yarn version: 2.4.0

Additional context

This monorepo consists of 4 packages, 3 of which are CreateReactApp projects, mentioned because CRA bundles eslint (though this issue happens with other bins too).

yarnrc.yml for reference:

nodeLinker: node-modules

npmRegistries:
  //npm.pkg.github.com:
    npmAlwaysAuth: false

npmRegistryServer: "https://registry.yarnpkg.com"

npmScopes:
  org:
    npmPublishRegistry: "https://npm.pkg.github.com"
    npmRegistryServer: "https://npm.pkg.github.com"

plugins:
  - path: .yarn/plugins/@yarnpkg/plugin-workspace-tools.cjs
    spec: "@yarnpkg/plugin-workspace-tools"

yarnPath: ".yarn/releases/yarn-berry.cjs"

root-level package.json:

{
  "private": true,
  "workspaces": [
    "packages/ui",
    "packages/*/ui"
  ],
  "resolutions": {
    "webpack": "4.44.2"
  },
  "devDependencies": {
    "cypress": "6.2.1"
  }
}

[andreialecu]

In your repro, your package.json does not seem to have a dependency on eslint. Yarn 2 is more strict about which scripts it runs.

Is there any reason why you don't add a dependency to eslint?

[bhishp]

I avoided it because eslint is bundled within react-scripts (CRA/create react app) and CRA is opinionated about this and some other libraries - expecting developers to rely on their bundled version of eslint rather than adding the dependency itself.

I thought yarn2 was only more strict when it comes to using PnP? The migration guide only suggests using @yarnpkg/doctor once switching over to PnP.

ESLint specifically is discussed in this issue: #8 and has a specific mention for CRA:

This problem only happens when the ESLint configuration is part of a package you depend on. For example react-scripts, created by create-react-app. In this instance, the plugins aren't listed in the application package.json, meaning that the fallback cannot kick in and do its job.

But, again, my understanding was that this is only relevant for PnP?

Additionally, the build shows that ts-node and prettier are both failing, even though they are explicitly listed as dependencies.

[andreialecu]

Additionally, the build shows that ts-node and prettier are both failing, even though they are explicitly listed as dependencies.

They're listed as dependencies in your root, but they should be dependencies in each workspace as well.

Alternatively you can define the script in the root package.json and use the $INIT_CWD env variable to make it work on your target workspace. See: https://yarnpkg.com/getting-started/qa#how-to-share-scripts-between-workspaces

[arcanis]

I thought yarn2 was only more strict when it comes to using PnP? The migration guide only suggests using @yarnpkg/doctor once switching over to PnP.

Yes, with one caveat: the environment created by yarn run. When we run a script (or binary), we create a temporary folder that references all the bin from your dependencies, and add it to the PATH. Consequently, if you don't list the package as a dependency, then its bins won't be added to the PATH.

The node_modules/.bin folder is pretty much unused by Yarn, and is only there for compatibility purpose with the tools that expect it to exist.

With that being said, if you list prettier as a dependency and it's not available, then something is wrong. I'm not aware of any problem in this part of the code though, so I'd tend to think perhaps the cwd is incorrect when executing the command (you need to be in the workspace that defines prettier as dependency).

[bhishp]

Thank you both.

TLDR Resolution for reference.

In a workspaced environment - even if using nodeLinker: node-modules - make sure to list any and all dependencies that your scripts rely on for each workspace. Alternatively, create shared scripts (say in the root package.json), where these dependencies are listed and re-use them from within your workspaces, as described here: https://yarnpkg.com/getting-started/qa#how-to-share-scripts-between-workspaces

Rest..

@andreialecu adding the dependency to the specific workspace fixed the issue, as did using a shared script - an example of shared script included below for reference:

In root package.json

"scripts": {
    "g:lint": "cd $INIT_CWD && eslint --ext js,ts,tsx src",
    "g:pretty-check": "cd $INIT_CWD && prettier --list-different './**/*.{js,json,ts,tsx,md,yml}'"
  },
"devDependencies": {
  "eslint": "^7.18.0",
  "prettier": "^2.2.1",
  "ts-node": "^9.1.1"
}

In pkg1 package.json

  "scripts": {
    "lint": "yarn g:lint",
    "pretty-check": "yarn g:pretty-check"
  },

(green build: https://github.com/bhishp/yarn-2-workflow-bin-repro/runs/1777056188)

@arcanis thank you for the explanation, is it worth adding something to the migration guide for this? Anything I can help with?

Closing this issue, thanks for your help

[bhishp]

Yes, with one caveat: the environment created by yarn run. When we run a script (or binary), we create a temporary folder that references all the bin from your dependencies, and add it to the PATH. Consequently, if you don't list the package as a dependency, then its bins won't be added to the PATH.

With regards to this, I am only seeing the behaviour in a CI environment, is there any obvious reason why these commands work locally but not in CI? Especially given that the yarn-berry runner is checked-in to the repo

[andreialecu]

With regards to this, I am only seeing the behaviour in a CI environment, is there any obvious reason why these commands work locally but not in CI?

Do you have eslint installed globally on your machine perhaps?

[bhishp]

No global installs

[Knagis]

@bhishp Perhaps you have some version export PATH="./node_modules/.bin:$PATH" in your shell profile?

[iDVB]

/
/frontend

We also have this issue and not sure what the fix should be.
We are including gatsby-cli as a dependency in the workspace frontend however still getting
sh: gatsby: command not found
when in the root I run yarn deploy

root scripts look like this...

  "scripts": {
    "test": "sst test",
    "start": "sst start",
    "build": "sst build",
    "deploy": "sst deploy",
    "remove": "sst remove",
    "build:frontend": "ls",
  },

SST uses AWS CDK to run the yarn build command inside the workspace

UPDATE ---

Something that works

I tested with my own simple script ...
test.js

const execSync = require("child_process").execSync

execSync('yarn build', {
  cwd: 'frontend',
  stdio: "inherit",
  env: {
    ...process.env,
  },
});

and adding to package.json scripts

  "scripts": {
    "test": "sst test",
    "start": "sst start",
    "build": "sst build",
    "deploy": "sst deploy",
    "remove": "sst remove",
    "build:frontend": "ls",
    "dvb": "node test.js"
  },

then running yarn dan and it works fine.

Work around

Found out by also adding the gatsby-cli dependency to the root package.json vs the workspace, it now works.
Something regarding these bin files has broken things in Yarn2

[zyf0330]

I meet this problem too.
I have a monorepo, and have two packages A and B. Both two have one same dependency D inside their package.json, and not in root package.json. D provides a script named S.
I use yarn workspaces focus A root to install dependency. And I run yarn run S inside package A, it gives command not found: S. I do this inside docker build process, so environment difference should be excluded.
This problem just appears in CI jenkins, not at my local machine. It's wired.