build: switch to pnpm

[koonpeng]

What's new

The main motivation for this change is due to the limitations and problems of pipenv, switching to poetry allows us to support multiple python versions without managing different branches (although ci is not set up to do so yet). Turns out poetry also suffers from inconistent lock files when mixing different python versions, this may be because we are mixing system dependencies.

Since we are switching to poetry, I figured might as well try out pnpm, it is one of the most promising alternative to npm/lerna/other monorepo tools we looked at in the past. We ended up with npm because it is the default and we figured it will make it easier for new people to get started, however the limitations of npm causes us to write custom scripts to run tasks with topogical ordering, install a subset of workspaces and packaging typescript build outputs correctly. pnpm has all that built in and "just works", on top of that pnpm enforces correctness in the dependencies (it doesn't allow packages to use undeclared dependencies). It even has a built in command to install nodejs, however, it can only do so globally for now.

This PR also removes reporting, they don't work without fixing all the undeclared dependencies. Also, this is base off #624.

Self-checks

• I have prototyped this new feature (if necessary) on Figma
• I'm familiar with and follow this Typescript guideline
• I added unit-tests for new components
• I tried testing edge cases
• I tested the behavior of the components that interact with the backend, with an e2e test

Discussion

[aaronchongth]

Apologies for the delayed review, I finally got to try it out in 22.04, and found that I had to manually install venv, with sudo apt install python3.10-venv. Should this be added into the prerequisite step?

aaron@nasilemak:~/workspaces/rmf_main/without-venv-rmf-web$ source ../ws/install/setup.bash
aaron@nasilemak:~/workspaces/rmf_main/without-venv-rmf-web$ pnpm install
Scope: all 10 workspace projects
Lockfile is up to date, resolution step is skipped
Packages: +2443
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /home/aaron/.local/share/pnpm/store/v3
  Virtual store is at:             node_modules/.pnpm
Progress: resolved 2443, reused 2355, downloaded 4, added 2443, done

devDependencies:
+ @typescript-eslint/eslint-plugin 4.33.0
+ @typescript-eslint/parser 4.33.0
+ eslint 7.32.0
+ eslint-plugin-react 7.30.1
+ eslint-plugin-react-hooks 4.6.0
+ husky 8.0.1
+ lint-staged 10.5.4
+ prettier 2.7.1
+ pyright 1.1.257
+ typescript 4.4.4

poetry-install install$ ../scripts/poetry-install.sh
│ /home/aaron/workspaces/rmf_main/without-venv-rmf-web/poetry-install
│ The virtual environment was not created successfully because ensurepip is not
│ available.  On Debian/Ubuntu systems, you need to install the python3-venv
│ package using the following command.
│     apt install python3.10-venv
│ You may need to use sudo with that command.  After installing the python3-venv
│ package, recreate your virtual environment.
│ Failing command: ['/home/aaron/workspaces/rmf_main/without-venv-rmf-web/.venv/bin/python3', '-Im',
└─ Failed in 122ms
. prepare$ husky install
│ husky - Git hooks installed
└─ Done in 105ms
 ELIFECYCLE  Command failed with exit code 1.
 WARN   Local package.json exists, but node_modules missing, did you mean to install?
aaron@nasilemak:~/workspaces/rmf_main/without-venv-rmf-web$

[aaronchongth]

It also looks like due to python-poetry/poetry#6334, using the latest poetry, the dependency install step fails as it dislikes the version 1.1build1, in one of the packages.

Do you think it is a good idea to stick with version 1.1.15? It works on that version.

curl -sSL https://install.python-poetry.org | python3 - --version 1.1.15

[koonpeng]

It doesn't look like poetry will change the behavior, I think we can use a fixed version to make it work.

[aaronchongth]

Other than the version of poetry, everything else is working well. Thanks for removing the reporting CI too!

[aaronchongth]

I found that I needed to manually compile rmf-auth and rmf-models, even though the initial pnpm install seems to have built them, https://gist.github.com/aaronchongth/c34bad4a4a12b88fc064852c7a0a3737.

Edit: Here is the gist of the launch saying rmf-auth is not found, https://gist.github.com/aaronchongth/c2d59f31aaec31905b3e47a09d7c14a5.

After running pnpm install -w --filter for rmf-auth and rmf-models everything runs fine,

aaron@excellence2:~/workspaces/rmf_main/rmf-web$ pnpm install -w --filter rmf-auth
Scope: 2 of 10 workspace projects
Lockfile is up to date, resolution step is skipped
. prepare$ husky install
│ husky - Git hooks installed
└─ Done in 71ms
aaron@excellence2:~/workspaces/rmf_main/rmf-web$ pnpm install -w --filter rmf-models
Scope: 2 of 10 workspace projects
Lockfile is up to date, resolution step is skipped
. prepare$ husky install
│ husky - Git hooks installed
└─ Done in 65ms

Do you experience this on your end? But otherwise everything is working well

[koonpeng]

That should not happen. Did you try cleaning up all node_modules before pnpm install? Might be because of old npm residue files.

[aaronchongth]

Hmm, now I'm wondering if there is something wrong with my setup 😅 You've been able to launch the dashboard from a fresh build without additional pnpm install calls?

This is the gist of a fresh build and run. I basically had to

pnpm install
cd packages/dashboard
npm run start
# failed, rmf-auth not found
cd ../../
pnpm install -w --filter rmf-auth
cd packages/dashboard
npm run start
# failed, rmf-models not found
cd ../../
pnpm install -w --filter rmf-models
cd packages/dashboard
npm run start
# now it works

[koonpeng]

npm run start

you need to use pnpm

[aaronchongth]

Apologies for the delay, I've found what the issue was regarding pnpm start not being able to find rmf-auth and rmf-models. There are still a number of cases where file:../package_name is being used. Unfortunately I am not very familiar with the difference 🤔 , I'll leave it to your discretion on where it should be used.

Once these are addressed, I think we are good to merge!

[aaronchongth]

This step is needed again.

[koonpeng]

pipenv is now automatically bootstrapped in the install script.

if [ ! -f .venv/pyvenv.cfg ]; then
  python3 -m venv --system-site-packages --prompt rmf-web .venv
  .venv/bin/pip install 'pipenv==2022.9.8'
fi

[aaronchongth]

Thanks that all looks good now!

[xiyuoh]

Hello! Just tested on Ubuntu 22.04, i got this error message after running pnpm install:

pipenv-install install$ ../scripts/pipenv-install.sh
│ /home/xiyu/web_ws/rmf-web/pipenv-install
│ The virtual environment was not created successfully because ensurepip is not
│ available.  On Debian/Ubuntu systems, you need to install the python3-venv
│ package using the following command.
│     apt install python3.10-venv
│ You may need to use sudo with that command.  After installing the python3-venv
│ package, recreate your virtual environment.
│ Failing command: ['/home/xiyu/web_ws/rmf-web/.venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']
└─ Failed in 108ms

After manually running sudo apt install python3-venv, everything else worked fine.

[koonpeng]

I would say that is normal, venv is part of python standard library but some distros like debian may choose to package it differently. Are you on a container or a fresh ubuntu2204 install? If you are on a fresh install, can you make a PR to add that command in the README?