yarn runtime vs development dependencies

[jonahgraham]

This may not be an issue for the tool, but rather the CQ process?

At the moment to develop in yarn it takes hundreds of development time packages that are not part of the runtime distribution. For example, the cdt-gdb-adapter has ~500 items listed in yarn.deps when run on cdt-gdb-adapter code base. However, if you only consider the runtime dependencies the list is just 5 items.

One idea of how to automate is this:

yarn init -y
yarn add cdt-gdb-adapter
node <path to>licesnes/yarn/index.js  | grep -v 'cdt-gdb-adapter' > yarn.deps

[waynebeaton]

Build time dependencies are still dependencies (i.e., "Build and Test Dependencies"). They're "works with", though, so we have considerable latitude with regard to licenses.

The worst case scenario is that a project team would maintain a dependency list (or, alternatively, an exclude list that's processed against a generated dependency list). I'm thinking that we may have no other choice for technologies that don't provide an easy means of generating a dependency list directly from the build (e.g. make files).

[jmini]

I am not a Yarn expert, but my understanding is that yarn.lock is just fixing the version of the dependency. So this is not enough to use only this file as source for the generating the report.

To know the type of dependency (build dependency vs runtime dependency) you need the package.json (and you need to traverse the complete dependency tree, using yarn.lock to know how a given dependency was resolved and then use the package.json of this dependency).

[sbernard31]

I'm not a yarn expert too. 😅

But I also understand that the kind of dependencies are in package.json file. (see yarn2 doc or older but maybe clearer yarn1 doc)

{
  "dependencies": {
    "webpack": "^5.0.0"
  },
  "optionalDependencies": {
    "fsevents": "^5.0.0"
  },
  "devDependencies": {
    "webpack": "^5.0.0"
  },
  "peerDependencies": {
    "react": "*",
    "react-dom": "*"
  }
}

The yarn.lock file just fix version. (see yarn1 doc)

So ideally, I guess that @jmini is right and both files are need to get the information we want.

I don't know if there is a way to query about this kind of information with yarn command, I will search a little bit 🤔

[sbernard31]

It seems that yarn list --prod list only "dependencies" (so I understand this is what we want to check, correct ?)
(Tested with yarn 1.22.10)
(Source : yarnpkg/yarn#4542.)

In my case, this returns 20 lines instead of 3573 (without --prod).
The result looks like :

yarn list v1.22.10
├─ @fontsource/[email protected]
├─ @mdi/[email protected]
├─ [email protected]
│  └─ follow-redirects@^1.10.0
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
│  └─ moment@^2.19.2
├─ [email protected]
├─ [email protected]
├─ [email protected]
│  └─ event-source-polyfill@^1.0.22
├─ [email protected]
├─ [email protected]
└─ [email protected]
Done in 0.99s.

If we want something more easy to parse there is something like : yarn list --prod --no-progress --json

click here to see the result.

{
  "type": "tree",
  "data": {
    "type": "list",
    "trees": [
      {
        "name": "@fontsource/[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "@mdi/[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [
          {
            "name": "follow-redirects@^1.10.0",
            "color": "dim",
            "shadow": true
          }
        ],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [
          {
            "name": "moment@^2.19.2",
            "color": "dim",
            "shadow": true
          }
        ],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [
          {
            "name": "event-source-polyfill@^1.0.22",
            "color": "dim",
            "shadow": true
          }
        ],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": "bold",
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": null,
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": null,
        "depth": 0
      },
      {
        "name": "[email protected]",
        "children": [],
        "hint": null,
        "color": null,
        "depth": 0
      }
    ]
  }
}

[sbernard31]

In case we need to write some javascript based on yarn API.
The code of the yarn list command is available here.

I also see it exists a yarn audit command which is able to query on a specific group of dependencies.
(See https://classic.yarnpkg.com/en/docs/cli/audit#toc-yarn-add)
So maybe the code of this command could be inspiring too.

[jonahgraham]

@sbernard31 - I wrote the contents of https://github.com/eclipse/dash-licenses/tree/master/yarn when I needed it almost 2 years ago, I have no doubt that there are updates and maybe better solutions now available. Please feel free to provide a PR.

(so I understand this is what we want to check, correct ?)

That is what needs checking for full license compatibility, but keep in mind as @waynebeaton said in #4 (comment) that all the hundreds to thousands of deps need handling, just that they are works-with instead. So having the tool spit out two lists would be useful. See https://www.eclipse.org/projects/handbook/#ip-third-party-test for more info.

[sbernard31]

That is what needs checking for full license compatibility, but keep in mind as @waynebeaton said in #4 (comment) that all the hundreds to thousands of deps need handling, just that they are works-with instead

This part confused me a little because it's not clear to me what should be done exactly with "work_with".
Reading the ip-third-part-test link you provide, I understand that CQ is needed anyway (but this will be a lighter check by IP team ?)

Maybe it's not true for all projects, but for me I understand that my yarn devDependencies are pretty much like maven plugin. This is just some tools which help to build/validate the project. And I never consider to create CQ for the maven-plugin I'm using. Is it something which should be theoricaly done ?

[sbernard31]

@paul-marechal, @marcdumais-work I see that you opened some issues about using dash-licences with yarn project.

I would be like to know your opinion about this issue ?

[marcdumais-work]

Hi @sbernard31,

For the Theia project, we have an unusual number of dependencies: over 1000. When we were doing the license checks manually, using clearlydefined, we would filter-down the list to only production/runtime dependencies, to keep it somewhat manageable.

Now that dash-licenses does almost all the work (1), we no longer distinguish and let the tool deal with them all. Once in a while, some license issue is found with a dependency, then we can check how it's used, and if it's a devDependency, we can often get a "workswith" approval, to use it for build & tests.

(1): I encourage you to use this amazingly useful feature, that essentially opens the equivalent of a CQ for you for each dependency that needs it: https://github.com/eclipse/dash-licenses#automatic-ip-team-review-requests. See one of these automatically opened IP team review request, here, for example: https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/1648

[sbernard31]

Another option would be to use tooling to generate SBOM.
E.g. :

• https://github.com/anchore/syft
• https://aquasecurity.github.io/trivy/v0.48/docs/supply-chain/sbom/
• https://github.com/CycloneDX/cyclonedx-node-yarn (no release yet, not sure the project is still alive Is this still active? CycloneDX/cyclonedx-node-yarn#10, 📣 looking for contributors CycloneDX/cyclonedx-node-yarn#12)

Spoiler alert for now, AFAIK none of this project support to create a SBOM without devDependencies for yarn but if you think it makes sense to follow that way. I can try to open issue. (see #4 (comment))

And then give that SBOM to dash-licences : #191

[sbernard31]

Spoiler alert for now, AFAIK none of this project support to create a SBOM without devDependencies for yarn but if you think it makes sense to follow that way. I can try to open issue.

I was wrong using trivy this is possible to generate SBOM with or without devDependencies :

Without dev dependencies :

trivy fs ./path/to/your/yarn/or/npm/project --format cyclonedx --output sbom-without-dev-dep.json

With dev dependencies :

trivy fs ./path/to/your/yarn/or/npm/project --format cyclonedx --output sbom-with-dev-dep.json --include-dev-deps