Third Party Content in NOTICES.md

[viralmodi]

Theia "Third Party Content" section in NOTICES.md mentions - "This project leverages the following third party content."
I am planning to use this framework at our work and it would be of great help if someone can shed some light on this statement. Are all packages mentioned after that third party dependencies required for building Theia OR they are just mentioned since they were referred while writing source code for Theia framework OR they will be used if included in package.json of a Theia project or something else?

[marcdumais-work]

HI @viralmodi ,

First, that file is for the most part auto-generated and had not been updated for a while. Your question prompted me to do so - here's the PR in case you want a more current picture:
#8957

Second, that file covers the whole Eclipse Foundation project, not just this repository here. So some contained items may not specifically apply, here (cloning and building this repo).
https://projects.eclipse.org/projects/ecd.theia/developer

Are all packages mentioned after that third party dependencies required for building Theia OR they are just mentioned since they were referred while writing source code for Theia framework OR they will be used if included in package.json of a Theia project or something else

Good question. I have no direct information about the inner-working of the Eclipse foundation Legal Documentation Generator, but from its output I can deduce some things.

• items that mention copying (e.g. "code copied from") refer to code that was copied from another open-source project and integrated in ours. Most often it will be MIT-licensed code, most often from the vscode project.
• If there's no mention of copying, the components are probably runtime dependencies, that we consume as-is, from public npm mostly (so included in a package.json somewhere). The list is too short, so I think it's only a subset of our runtime dependencies. I notice that some components, that have required more analysis / permissions, are mentioned, but so are others I do not remember anything about. From memory, the main repo once you run yarn has around 1500 npm dependencies, of which maybe ~1000 are runtime dependencies (recursive list).

I think I have somewhere a couple of shell command you can use to extract the approximate list of npm dependencies - would that be of interest to you?

[viralmodi]

Thank you so much @marcdumais-work .
Yes, I would be interested in all those dependencies. We have a legal system in place at our work which requies us to know that before we use this project.
Just to clarify, the ~1000 runtime dependencies that you are talking about - are they all dependencies for everyone who uses even the most vanilla Theia framework version? I mean to ask if I were to create a package.json and not include anything in dependencies section where we mention Theia extensions and also keep the Theia plugins section empty, would it still have more than 1000 runtime dependencies?
Also, I am only interested in Theia framework dependencies and not entire Eclipse Foundation project dependencies. Are these > 1000 dependencies that you mentioned, covering Theia only or entire Eclipse Foundation project?

[marcdumais-work]

Hi @viralmodi

Yes, I would be interested in all those dependencies. We have a legal system in place at our work which requies us to know that before we use this project.

I guess we have similar at the Eclipse Foundation and my employer. AFAIK there is no canonical way to extract npm dependencies - it really depends on each project. Your company might have tools already to scan source code repositories that you could use. Else I can offer a simple but hackish way to get an idea of the dependencies but I can't guarantee the results are 100% accurate or that it will cover everything.

The technique relies on parsing the output of yarn list, extracting dependencies while removing formatting, before sorting and filtering duplicate entries.

Note that this will not report dependencies for the vscode extensions listed in package.json and fetched under plugins after building the example app. These would need to be separately analysed.

cd <theia repo>
# if you want only production dependencies set this variable
# else build and test dependencies will also be included:
export NODE_ENV=production  

git clean -ffdx
yarn install 
yarn list > list.txt

# filter-out ASCII trees 
sed -E -i 's/[│└─├─ ]+(.*)/\1/g' list.txt

# a bit of manicuring on entries - sometimes `yarn list` entries have requested 
# version rather than a specific one. Remove these entries - the corresponding 
# specific version should be present too
# e.g. `component@^version` or `component@~version`. 
sed -E -i 's/^.*\^.*//g' list.txt
sed -E -i 's/^.*\~.*//g' list.txt
sed -E -i 's/^.*\*.*//g' list.txt

# sort and filter duplicates
sort list.txt > list.sorted
uniq list.sorted > list.uniq

Doing that with the current master branch, I get ~1700 runtime dependencies but there are duplicates - same npm package but different version. Last time I checked, ~1000 were unique packages.

Just to clarify, the ~1000 runtime dependencies that you are talking about - are they all dependencies for everyone who uses even the most vanilla Theia framework version? I mean to ask if I were to create a package.json and not include anything in dependencies section where we mention Theia extensions and also keep the Theia plugins section empty, would it still have more than 1000 runtime dependencies?

That would not be a Theia application - you'd need to include at least @theia/core to have a null Theia app (interfaces, little implementation for anything). If you want your Theia app to e.g. support vscode extensions (and any programming language), you'll need to add @theia/plugin-ext, which recursively pulls many/most of the extensions from this repo.

cd /tmp && mkdir test && cd test
npm init -y
npm add @theia/core 
npm add @theia/plugin-ext

yarn install --production
yarn list > list.txt
sed -E -i 's/[│└─├─ ]+(.*)/\1/g' list.txt
sed -E -i 's/^.*\^.*//g' list.txt
sed -E -i 's/^.*\~.*//g' list.txt
sed -E -i 's/^.*\*.*//g' list.txt
sort list.txt > list.sorted
uniq list.sorted > list.uniq
wc -l list.uniq

When I run the above and get ~500 production dependencies (including some duplicated dependencies versions). Anything you add it can only go up. If I only have @theia/core I get 300 dependencies.

Also, I am only interested in Theia framework dependencies and not entire Eclipse Foundation project dependencies. Are these > 1000 dependencies that you mentioned, covering Theia only or entire Eclipse Foundation project?

It's only this repo here, so arguably a subset of the Eclipse Theia project. e.g. we have a few other repos under eclipse-theia org and under eclipse. Most Theia framework extensions are here though. TL;DR: only Eclipse Theia

[marcdumais-work]

I should add: with the above you get the "expanded", recursive list of runtime npm dependencies for this mono-repo / yarn workspace. It's possible that you care more or only about direct runtime dependencies, and not their runtime dependencies, and so on. Getting that "direct runtime dependencies" list is more convoluted, but IIRC the number was over a hundred.

[viralmodi]

Thanks @marcdumais-work for that elaborate answer. That was quite helpful. I have some more questions for you below:

I should add: with the above you get the "expanded", recursive list of runtime npm dependencies for this mono-repo / yarn workspace. It's possible that you care more or only about direct runtime dependencies

I am looking for an "expanded" recursive list as well as a separate direct list (3P (third party) and 3P+4P (forth party) dependencies). Can you tell me what is the convoluted way to directly find just the direct 3P runtime dependencies as well?

Note that this will not report dependencies for the vscode extensions listed in package.json and fetched under plugins after building the example app. These would need to be separately analysed.

How can I get the 3P+4P dependencies for packages mentioned in Theia plugins (VSCode extensions + any Theia plugins) section?

Lastly, can you please share the results of the 1000 dependencies, 500 dependencies and 300 dependencies that you found using your script?

[marcdumais-work]

Hi @viralmodi - you're welcome.

I am looking for an "expanded" recursive list as well as a separate direct list (3P (third party) and 3P+4P (forth party) dependencies). Can you tell me what is the convoluted way to directly find just the direct 3P runtime dependencies as well?

There may be a better way, but the one time I did it I wrote a quick JS script that opens, in turn, every package.json of the monorepo and extract the dependencies.

How can I get the 3P+4P dependencies for packages mentioned in Theia plugins (VSCode extensions + any Theia plugins) section?

Not sure there's a generic way. The vscode extensions (Theia plugins) mentioned in package.json are just direct download URLs, that could be hosted as simple files, e.g. on a GH release page. Those links that use open-vsx.org to fetch extensions, it might be possible to use the open-vsx API to fetch each extension's package,json and build a list of dependencies that way. See:
https://open-vsx.org/swagger-ui/#/registry-api/getFileUsingGET

Lastly, can you please share the results of the 1000 dependencies, 500 dependencies and 300 dependencies that you found using your script?

I did not keep the files, but I just quickly recreated the one for recursive runtime dependencies (~1700, including some with some duplicated versions):
list.uniq.tar.gz

[marcdumais-work]

Another less granular view of the repo's dependencies. It looks like the main list only considers the root package.json, but the second one, from yarn.lock, should contain everything:
https://github.com/eclipse-theia/theia/network/dependencies

update: the other package.json seem to be covered, if one scrolls down. Interesting but this info is not easy to play -with