[RFC] Revive cabal sandboxes (UX)

[hasufell]

Motivation

At ZuriHac 2024 we (@andreabedini, myself and others) talked with @ivanperez-keera and @fdedden about the user experience of using GHC and cabal for people who are not Haskell developers and are not interested in Haskell on its own, but rather use it to:

• build a package
• use a framework/library that requires Haskell (e.g. copilot)

These users are not interested in:

• nix
• Haskell
• cabal
• content addressable storage
• declarative environment configuration

The cabal v2 commands are assuming a lot of these things UX-wise.

We are well aware that there are ways to utilize store location and environment files to achieve something similar. But this misses a consistent UX experience.

Imperative workflow

We propose here an imperative workflow of managing haskell packages.

1. the user asks cabal to install a package
2. cabal makes this package visible to
   • ghc/ghci
   • cabal itself
3. building a project (and similar operations) will re-use the already installed packages

This follows the workflow of 90% of Linux distributions. It usually disallows having multiple versions of the same library.

A sandbox workflow

The sandbox workflow is an extension of the imperative workflow. A sandbox is scoped to a specific directory (and its subdirectories). Installing packages inside the sandbox only makes them available within that directory. A sandbox can be conveniently created and conveniently destroyed. A user would rather destroy a sandbox and start from scratch rather than try to fix it. This is outlined in the following workflow:

1. user clones e.g. copilot
2. user creates a sandbox inside the copilot repo
3. user installs the necessary Haskell libraries and the copilot library
4. user uses a Makefile to link to copilot and other libraries (compare with bluespec, which does the same)
5. user upgrades the project
6. user destroys the sandbox
7. repeats from step 2

A similar workflow can be imagined with an actual cabal package that is developed ad-hoc iteratively.

Users who want maximum caching and declarative configuration can use the existing -v2 interface instead.

Implementation details

We have considered the following:

1. utilize -v1 to implement sandboxes (probably similar to the old implementation)
2. utilize -v2 to implement sandboxes

v2 based sandbox implementation

A possible implementation would be:

1. cabal sandbox init would e.g. augment cabal.project.local with several configurations
   • write-ghc-environment-files: always
   • store-dir: .cabal-sandbox (this doesn't seem to be supported yet through the cabal project file)
   • package-env: . (seems to be not supported either through cabal project file)
2. cabal install would then
   • use the sandbox store dir
   • use the current dir as package env and write the configuration file
   • also add constraints: <pkg-name> installed to cabal.project.local
3. cabal sandbox destroy would
   • delete the .cabal-sandbox directory
   • delete the ghc environment file
   • delete cabal.project.local

This would ensure consistency between directly invoking ghc and using cabal to actually build packages.

It would also allow to move the created constraints to cabal.project file instead (for committing them to git). We could also imagine slight variants of this idea where we utilize yet another project file with a different suffix if cabal.project.local is deemed inappropriate.

Alternative v2 implementation without utilizing cabal.project files

An alternative implementation could create a sandbox config file that then triggers sandbox behavior in cabal (through cabal checking for its existence). This is partly implemented here: hasufell#3
But I consider it more controversial. However it is more opaque.

Related discussions and prior art

• [UX] Current mechanism to install several libraries and expose them to the user? #9581
• Separate cabal install --lib into own command (cabal env) #6481

[mpickering]

Perhaps a design like python venv is the most suitable where you have to explicitly enter the environment.

https://docs.python.org/3/library/venv.html

I agree I have also been in situations where I really know what I am doing and just need cabal to create a package database with certain packages in. Something which is very difficult to achieve currently (and seems to line up with the motivation for this feature). If anyone is interested then this quite ugly code is in cabal-testsuite/main/cabal-tests.hs.

Implementation wish I think it should be implemented somewhat directly if possible, features such as cabal run -- foo.hs don't do a great job of isolating the user from the implementation of the feature.

[ivanperez-keera]

Thank you @hasufell for submitting this. This is great!

A possible addition to step 2 of the your proposed workflow would be that, when the user installs a library, all of its dependencies could be made available/visible by default to ghc/ghci. That way, the user doesn't need to explicitly install them to import modules from dependencies.

[ivanperez-keera]

Following up on this hoping that we can move it forward. What are the next steps?

[hasufell]

Following up on this hoping that we can move it forward. What are the next steps?

I think the main thing we are "stuck" on is to decide whether we want a "thick" or "thin" abstraction.

A thin abstraction would boil down to just creating a cabal.project.local (or similarly named) that emulates sandbox behavior through local store dir, env files etc. and then rely on the v2 commands to just do the right thing.

But it wouldn't guarantee configuration consistency. A user might unknowingly break their sandbox, e.g. because they don't know that the store-dir setting affects it.

@mpickering opined about this on the Matrix channel too.

If we want a thick abstraction, we will likely end up with an implementation that uses the old v1 style stuff again.

If we try to be pragmatic, I think a thin abstraction is more likely to have a straight forward prototype.

[ivanperez-keera]

Is there a natural transition path where we start with a thin abstraction as a first step, and progressively modify the implementation towards the thick abstraction without affecting the user experience?

[hasufell]

So the thin abstraction could theoretically live outside of cabal.

We just need to add support for certain configurations to cabal.project.

Then we can write a new executable that creates the appropriate config and utilize #9063 to have nicer integration.

If the experiment fails in terms of UX, we can continue the discussion about a thick abstraction.

The downside of this approach is that it's external and probably not shipped with cabal itself.

[ivanperez-keera]

I'd be open to having this externally as an experiment, although for this to survive each cabal release, I think it should be part of the features of cabal. Otherwise, there's a very high risk that things will change on Cabal's side in a way that no longer allows for this. Please correct me if I'm wrong about this.

Hopefully, it's possible to make it very simple, so that the maintenance burden is minimal and is not seen as an issue to make it part of the official cabal.

EDIT: Either way, let's try a quick prototype and see where we are.

[andreabedini]

Unfortunately you cannot set store-dir from cabal.project. I haven't really explored what we can be done with the external commands yet. At ZuriHac I had started a separate CLI (which was the quickest way at that moment).

[hasufell]

Unfortunately you cannot set store-dir from cabal.project.

Yes, that is described in the top post. Same goes for package-env in the context of cabal install. Those things will require proper cabal patches.

[andreabedini]

Yes, that is described in the top post. Same goes for package-env in the context of cabal install. Those things will require proper cabal patches.

But you can from the global config! brb, need to hack

[ivanperez-keera]

Ok. So how to move forward on this? What's the next immediate step?

I'm not well-versed enough in the current way cabal manages package DBs and such to fully follow the specific changes that would be needed.

[hasufell]

Ok. So how to move forward on this? What's the next immediate step?

I'm not well-versed enough in the current way cabal manages package DBs and such to fully follow the specific changes that would be needed.

This is the next step: #10184

[sgraf812]

FWIW, I think one of the benefits of venv is that you can point it to an arbitrary file path, acting as a "name" for the sandbox. IIUC, this file path does not need to point to a python project; it could live anywhere in the file system and be shared among multiple projects that want to see the same, consistent set of packages of the sandbox.

The proposed "thin" solution appears to want to hardcode the sandbox file path to /project/path/.cabal-sandbox. That's fine to get the ball rolling for an MVP, I suppose. But I would hope that changing the sandbox file path is not a feature too difficult to implement for the "thin" solution as well (it's just a CLI flag to change the custom store-dir: entry, I suppose), and would go a long way toward feature parity with a "thick" solution such as venv's. (If it wasn't clear, I don't really know enough about the Cabal impl to draw the line between "thin" and "thick".)

[hasufell]

FWIW, I think one of the benefits of venv is that you can point it to an arbitrary file path, acting as a "name" for the sandbox. IIUC, this file path does not need to point to a python project; it could live anywhere in the file system and be shared among multiple projects that want to see the same, consistent set of packages of the sandbox.

There are two variants of the "thin" (v2-based) solution:

• one that depends on a new sandbox file, triggering different behavior in cabal
• one that relies on the existing cabal.project mechanisms

We're probably going to pursue the latter, because it can be implemented outside of cabal-install much easier.

With that approach, a sandbox configuration is simply a file or code snippet following the cabal.project syntax.

Since cabal 3.8, the cabal.project syntax allows imports: https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#conditionals-and-imports

That means we can easily support the use case of "shared sandbox file". Your cabal.project in your haskell project would likely just have a line such as import: /var/cache/sandboxes/cabal.sandbox.

[mpickering]

pipx is another python project which seems relevant to this discussion - https://github.com/pypa/pipx