Buildpack fails installing dependencies with yarn 2 (on IBM Cloud)

[hedphuqz]

Hi lovely people. I just wanted to check if perhaps i'm doing something very wrong here, or perhaps if yarn 2 isn't supported yet, as it seems we cannot deploy cloud foundry apps to IBM cloud using yarn 2. I couldn't find anything on the internet about this so I'm hoping one of you can illuminate this for me. Thanks in advance :-)

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

{
   "name": "IBM Cloud",
   "build": "v13.2.0",
   "support": "http://ibm.biz/bluemix-supportinfo",
   "version": 0,
   "description": "IBM Bluemix",
   "authorization_endpoint": "https://iam.cloud.ibm.com/cloudfoundry/login/eu-central",
   "token_endpoint": "https://uaa.eu-de.cf.cloud.ibm.com",
   "min_cli_version": null,
   "min_recommended_cli_version": null,
   "app_ssh_endpoint": "ssh.eu-de.cf.cloud.ibm.com:2222",
   "app_ssh_host_key_fingerprint": "c7:1f:89:2a:62:3b:78:a9:08:c9:33:81:fb:39:26:da",
   "app_ssh_oauth_client": "ssh-proxy",
   "doppler_logging_endpoint": "wss://doppler.eu-de.cf.cloud.ibm.com:443",
   "api_version": "2.148.0",
   "osbapi_version": "2.15",
   "user": "cac7af48-df6c-4003-8a3c-a9fd8794516b"
}

Invoking 'cf version'...

cf version 6.46.1+4934877ec.2019-08-23

What version of the buildpack you are using?
IBM SDK for Node.js Buildpack v4.3-20200606-1928

Based on Cloud Foundry Node.js Buildpack 1.7.20

If you were attempting to accomplish a task, what was it you were attempting to do?
Tried to deploy a cloud foundry app using yarn 2 (2.0.0-rc.36)

What did you expect to happen?
Expected the app to detect the yarn binary and use it to install dependencies

What was the actual behavior?
It's unclear whether the application actually used the provided binary or not in the.yarn directory, but nevertheless it failed to install the dependencies with the following message:

**ERROR** Unable to build dependencies: exit status 1

additionally, in the deploy logs it quotes the installation of a yarn 1 version, but then says it actually installed yarn 2

-----> Installing yarn 1.22.4

       Copy [/tmp/buildpacks/3230f8f5e1e3f597172baf178fe57dd6/dependencies/d9f0c5147bcb9ee798f041978c157822/yarn-1.22.4-any-stack-24ca2294.tgz]

       Installed yarn 2.0.0-rc.36

We also tried vendoring dependencies, this also didn't work.

Can you provide a sample app?

I am happy to provide a sample app if required.

Full log output from the deploy phase:

Downloading dotnet-core_v2_7-20200615-1457...

Downloading python_buildpack...

Downloading staticfile_buildpack...

Downloading java_buildpack...

Downloading xpages_buildpack...

Downloaded xpages_buildpack

Downloading ruby_buildpack...

Downloaded python_buildpack

Downloading nodejs_buildpack...

Downloaded dotnet-core_v2_7-20200615-1457

Downloading go_buildpack...

Downloaded java_buildpack

Downloading liberty-for-java_v3_45-20200601-1056...

Downloaded staticfile_buildpack

Downloading r_buildpack...

Downloaded ruby_buildpack

Downloading liberty-for-java_v3_44-20200430-1451...

Downloaded go_buildpack

Downloading liberty-for-java...

Downloaded nodejs_buildpack

Downloading swift_buildpack_v2_0_18-20190303-1915...

Downloaded liberty-for-java_v3_45-20200601-1056

Downloading sdk-for-nodejs...

Downloaded liberty-for-java_v3_44-20200430-1451

Downloading sdk-for-nodejs_v4_2_1-20200420-1206...

Downloaded swift_buildpack_v2_0_18-20190303-1915

Downloading php_buildpack...

Downloaded liberty-for-java

Downloading dotnet-core...

Downloaded sdk-for-nodejs

Downloading sdk-for-nodejs_v4_3-20200606-1928...

Downloaded r_buildpack

Downloading binary_buildpack...

Downloaded sdk-for-nodejs_v4_2_1-20200420-1206

Downloading swift_buildpack...

Downloaded dotnet-core

Downloading dotnet-core_v2_6-20200424-1047...

Downloaded binary_buildpack

Downloading swift_buildpack_v2_0_20-20190401-2122...

Downloaded sdk-for-nodejs_v4_3-20200606-1928

Downloading swift_buildpack_cflinuxfs3_v2_1_0-20190404-1206...

Downloaded php_buildpack

Downloading nginx_buildpack...

Downloaded swift_buildpack

Downloaded dotnet-core_v2_6-20200424-1047

Downloaded swift_buildpack_v2_0_20-20190401-2122

Downloaded nginx_buildpack

Downloaded swift_buildpack_cflinuxfs3_v2_1_0-20190404-1206

Cell 7d34ba58-a543-4fb8-829b-e5aec88f8a1a creating container for instance bebe21cd-24bd-49c4-8bdd-d66d7aaaf1e9

Cell 7d34ba58-a543-4fb8-829b-e5aec88f8a1a successfully created container for instance bebe21cd-24bd-49c4-8bdd-d66d7aaaf1e9

Downloading app package...

Downloaded app package (202.9M)

-----> IBM SDK for Node.js Buildpack v4.3-20200606-1928

       Based on Cloud Foundry Node.js Buildpack 1.7.20

       

-----> Installing binaries

       engines.node (package.json): 12.12.0

       engines.npm (package.json): unspecified (use default)

       **WARNING** Node version in .nvmrc ignored in favor of 'engines' field in package.json

       Attempting to install: 12.12.0

-----> Installing node 12.12.0

       Download [https://s3pository.heroku.com/node/v12.12.0/node-v12.12.0-linux-x64.tar.gz]

       Using default npm version: 6.11.3

-----> Installing yarn 1.22.4

       Copy [/tmp/buildpacks/3230f8f5e1e3f597172baf178fe57dd6/dependencies/d9f0c5147bcb9ee798f041978c157822/yarn-1.22.4-any-stack-24ca2294.tgz]

       Installed yarn 2.0.0-rc.36

-----> Creating runtime environment

       NODE_ENV=production

       NODE_HOME=/tmp/contents801366798/deps/0/node

       NODE_MODULES_CACHE=true

       NODE_VERBOSE=false

       NPM_CONFIG_LOGLEVEL=error

       NPM_CONFIG_PRODUCTION=true

-----> Building dependencies

       Installing node modules (yarn.lock)

       Running yarn in online mode

       To run yarn in offline mode, see: https://yarnpkg.com/blog/2016/11/24/offline-mirror

       **ERROR** Unable to build dependencies: exit status 1

Failed to compile droplet: Failed to run all supply scripts: exit status 14

Exit status 223

Cell 7d34ba58-a543-4fb8-829b-e5aec88f8a1a stopping instance bebe21cd-24bd-49c4-8bdd-d66d7aaaf1e9

Cell 7d34ba58-a543-4fb8-829b-e5aec88f8a1a destroying container for instance bebe21cd-24bd-49c4-8bdd-d66d7aaaf1e9

FAILED

Error restarting application: BuildpackCompileFailed

Please confirm where necessary:

• I have included a log output
• My log includes an error message
• I have included steps for reproduction

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/173527981

The labels on this github issue will be updated when the story is started.

[kvedurmu]

Hi @hedphuqz. The CF NodeJS Buildpack only supports Yarn 1 apps. You can see the list of dependencies in the manifest here.

We can definitely look into supporting Yarn 2 in the Paketo Buildpack. I've filed this issue to look into this. Closing for now, but feel free to open back up if you have any questions.

[abh1kg]

@kvedurmu : While yarn v2 is not packaged with the buildpack itself, the yarn berry installation works in a "trampoline" mode where if yarn > 1.22.5 is on the path, the vendored yarn cli should work. I was hoping this would help but I face a problem during staging while downloading the yarn version that comes with the buildpack:

-----> Installing yarn 1.22.10
          Download [https://buildpacks.cloudfoundry.org/dependencies/yarn/yarn_1.22.10_linux_noarch_any-stack_0057c1c9.tgz]
BuildpackCompileFailed - App staging failed in the buildpack compile phase

[killerfurbel]

Actually, the "trampoline" mode does not work. If you place a suitable .yarnrc.yml and the .yarn directory with yarn2 release in the folder and deploy it, yarn2 is initialized properly ("Installed yarn 2.2.2" for example), BUT:

The buildpack calls yarn with some hardcoded parameters: https://github.com/cloudfoundry/nodejs-buildpack/blob/master/src/nodejs/yarn/yarn.go#L63

If you try calling yarn that way, you will always get a "Unsupported option name ("--pure-lockfile")" error. For Example, this option is now called "--immutable". So the only option is adjusting the buildpack, verifying the yarn version and then changing the install commands a bit. The whole offline vs. online topic can be ignored with yarn2 as far as I know, since you can either set the "enableGlobalCache" to false and also push the .yarn/cache directory along with the application, or yarn will try to download the packages anyway.

So if anybody wants to try: Save you time, this buildpack won't help you with yarn2 (currently).

EDIT: This fork might enable at least some yarn2 functionality: fourcube@7c1bde6 --> if node_modules exist, don't run yarn install or npm install at all...

[kristian]

Just for everyone still struggling with this: I do still think the build pack should support Yarn 2+ / Berry natively, but I found a workaround and made it work even for this version of the build pack. Here is how I achieved running a Yarn Berry application on CF, taking a few shortcuts:

• Use Berry's Plug'n'Play linker (https://yarnpkg.com/features/pnp)

• What it will generate is a .yarn/cache folder containing all dependencies (as ZIPs) and a .pnp.cjs (and if you are using ESM like I do, a .pnp.loader.mjs) file, used for dependency loading.

• Create a .cfignore file with the following content:

  yarn.lock
  .yarn/*
  !.yarn/cache
  !.yarn/unplugged
  !.yarn/install-state.gz
  .yarnrc.yml
  

• This way the build pack will think it is a NPM package (because yarn.lock is not found), all other Yarn files (except the.yarn/cache) folder are not needed.

• In your package.json define a "start" script, like so:

  "start": "node --require ./.pnp.cjs server.js"

  or

  "start": "node --loader ./.pnp.loader.mjs --require ./.pnp.cjs server.js"

  In case you are using the ESM loader. This will hook into Nodes module loading and consume dependencies from the Yarn cache instead.

• Last but not least, we have to "tell" CF to not to install any dependencies, so clear out the dependencies sections of your package.json before deploying. This step requires a little "build magic" (or I am sure this could also be achieved by some clever package.json-preinstall-hook magic?!)

And that's it. Push your application, the NodeJS buildpack thinks it is a normal NPM app and start your PNP app right away 👍