[Magic CLI]: magic/max does not support usage in google colab notebooks

[guidorice]

Background

https://colab.research.google.com is widely used in data science and AI because of it's pay-as-you go model using a variety of TPU or NVIDIA GPU's. A common use case is going to be to install MAX and python packages into the notebook environment on colab. Couple of other key points:

• Colab does not seem to support switching notebook kernels via a menu item. This makes it fundamentally different than jupyter notebook which is self hosted or run locally.
• Every tutorial in existence for colab notebooks seems to start off with a notebook cell with the shell command ! pip install my-packages ... into the existing python env. Many scientific and AI packages are pre-installed, for example fastai, torch and transformers.

Reproduce steps

Go to https://docs.modular.com/max/python/get-started and follow the steps, but in a colab notebook. I am using Colab Pro, which also has a terminal window:

$ curl -ssL https://magic.modular.com | bash
$ magic init roberta-project && cd roberta-project
$ magic add max "numpy<2.0"
$ magic add --pypi "torch==2.2.2" "transformers==4.40.1"
$ magic shell
$ python3 -c 'from max import engine; print(engine.__version__)'
24.5.0
$ python --version
Python 3.12.5

^ as you can see max python package is installed, and python 3.12 as well.

The problem is, all the packages are in a virtualenv, however this is not something the notebook can use!

To confirm, create a notebook cell and execute it (from the get-started doc):

from pathlib import Path

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

from max import engine

ModuleNotFoundError: No module named 'max'

The reason the other imports succeeded is because torch and transformers is pre-installed on colab instances.

Related issues

• [Feature Request] Add support for Google Colab #184
• I assume the same issue exists in Kaggle notebooks, but I have not verified it: https://www.kaggle.com/docs/notebooks#the-notebooks-environment

Suggestion

Enhance magic to support installing into the system python, something like:

magic add --global max

[guidorice]

I discovered there is a magic global subcommand, but I did not get any fixes from it: https://docs.modular.com/magic/commands#magic-global