-
Notifications
You must be signed in to change notification settings - Fork 26
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add Conda Environment Management and Locator Support (#232)
* Refactor environment management into an abstracted interface. Separate the logic from the python builder and environment builder. Create a conda locator (folder locator) and a conda environment manager * Refactor python process spawn into static utils * refactor locators. get conda data from runtime settings * Add code to create environment from yml * Setup conda in ci * Allow overriding the req * Use CONDA * Include stderr in exceptions * Add error to exception message * minimise deps * Fix the linux path from the env var * clean up conda in GHA * Shell execute environment creation * Create the environment in CI * Don't try and create environments for now. * add docs updates * add environment page to docs * clarify calling for conda * Move the logger to the constructor of the environment management classes * Update src/CSnakes.Runtime/Locators/CondaLocator.cs Co-authored-by: Aaron Powell <[email protected]> * use file-scoped namespace * Fix merge foo * Fix logger resolver. --------- Co-authored-by: Aaron Powell <[email protected]>
- Loading branch information
1 parent
b871b33
commit 0754331
Showing
32 changed files
with
597 additions
and
133 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Environment and Package Management | ||
|
|
||
| CSnakes comes with support for executing Python within a virtual environment and the specification of dependencies. | ||
|
|
||
| There are two main package management solutions for Python, `pip` and `conda`. `pip` is the default package manager for Python and is included with the Python installation. `conda` is a package manager that is included with the Anaconda distribution of Python. Both package managers can be used to install packages and manage dependencies. | ||
|
|
||
| There are various ways to create "virtual" environments in Python, where the dependencies are isolated from the system Python installation. The most common way is to use the `venv` module that is included with Python. The `venv` module is used to create virtual environments and manage dependencies. | ||
|
|
||
| Virtual Environment creation and package management are separate concerns in Python, but some tools (like conda) combine them into a single workflow. CSnakes separates these concerns to give you more flexibility in managing your Python environments. | ||
|
|
||
| ## Virtual Environments with `venv` | ||
|
|
||
| Use the `.WithVirtualEnvironment(path)` method to specify the path to the virtual environment. | ||
|
|
||
| You can also optionally use the `.WithPipInstaller()` method to install packages listed in a `requirements.txt` file in the virtual environment. If you don't use this method, you need to install the packages manually before running the application. | ||
|
|
||
| ```csharp | ||
| ... | ||
| services | ||
| .WithPython() | ||
| .WithVirtualEnvironment(Path.Join(home, ".venv")) | ||
| // Python locators | ||
| .WithPipInstaller(); // Optional - installs packages listed in requirements.txt on startup | ||
| ``` | ||
|
|
||
| ### Disabling automatic environment creation | ||
|
|
||
| ## Virtual Environments with `conda` | ||
|
|
||
| To use the `conda` package manager, you need to specify the path to the `conda` executable and the name of the environment you want to use: | ||
|
|
||
| 1. Add the `FromConda()` extension method the host builder. | ||
| 1. Use the `.WithCondaEnvironment(name)` method to specify the name of the environment you want to use. | ||
|
|
||
| ```csharp | ||
| ... | ||
| services | ||
| .WithPython() | ||
| .FromConda(condaBinPath) | ||
| .WithCondaEnvironment("name_of_environment"); | ||
| ``` | ||
|
|
||
| The Conda Environment manager doesn't currently support automatic creation of environments or installing packages from an `environment.yml` file, so you need to create the environment and install the packages manually before running the application, by using `conda env create -n name_of_environment -f environment.yml` | ||
|
|
||
| ## Installing dependencies with `pip` | ||
|
|
||
| If you want to install dependencies using `pip`, you can use the `.WithPipInstaller()` method. This method will install the packages listed in a `requirements.txt` file in the virtual environment. | ||
|
|
||
| ```csharp | ||
| ... | ||
| services | ||
| .WithPython() | ||
| .WithVirtualEnvironment(Path.Join(home, ".venv")) | ||
| .WithPipInstaller(); // Optional - installs packages listed in requirements.txt on startup | ||
| ``` | ||
|
|
||
| `.WithPipInstaller()` takes an optional argument that specifies the path to the `requirements.txt` file. If you don't specify a path, it will look for a `requirements.txt` file in the virtual environment directory. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 41 additions & 0 deletions
41
src/CSnakes.Runtime/EnvironmentManagement/CondaEnvironmentManagement.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| using CSnakes.Runtime.Locators; | ||
| using Microsoft.Extensions.Logging; | ||
|
|
||
| namespace CSnakes.Runtime.EnvironmentManagement; | ||
| #pragma warning disable CS9113 // Parameter is unread. There for future use. | ||
| internal class CondaEnvironmentManagement(ILogger logger, string name, bool ensureExists, CondaLocator conda, string? environmentSpecPath) : IEnvironmentManagement | ||
| #pragma warning restore CS9113 // Parameter is unread. | ||
| { | ||
| ILogger IEnvironmentManagement.Logger => logger; | ||
|
|
||
| public void EnsureEnvironment(PythonLocationMetadata pythonLocation) | ||
| { | ||
| if (!ensureExists) | ||
| return; | ||
|
|
||
|
|
||
| var fullPath = Path.GetFullPath(GetPath()); | ||
| if (!Directory.Exists(fullPath)) | ||
| { | ||
| logger.LogError("Cannot find conda environment at {fullPath}.", fullPath); | ||
| // TODO: Automate the creation of the conda environments. | ||
| //var result = conda.ExecuteCondaShellCommand($"env create -n {name} -f {environmentSpecPath}"); | ||
| //if (!result) | ||
| //{ | ||
| // logger.LogError("Failed to create conda environment."); | ||
| // throw new InvalidOperationException("Could not create conda environment"); | ||
| //} | ||
| } | ||
| else | ||
| { | ||
| logger.LogDebug("Conda environment already exists at {fullPath}", fullPath); | ||
| // TODO: Check if the environment is up to date | ||
| } | ||
| } | ||
|
|
||
| public string GetPath() | ||
| { | ||
| // TODO: Conda environments are not always in the same location. Resolve the path correctly. | ||
| return Path.Combine(conda.CondaHome, "envs", name); | ||
| } | ||
| } |
25 changes: 25 additions & 0 deletions
25
src/CSnakes.Runtime/EnvironmentManagement/IEnvironmentManagement.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| using CSnakes.Runtime.Locators; | ||
| using Microsoft.Extensions.Logging; | ||
| using System.Runtime.InteropServices; | ||
|
|
||
| namespace CSnakes.Runtime.EnvironmentManagement; | ||
| public interface IEnvironmentManagement | ||
| { | ||
| ILogger Logger { get; } | ||
|
|
||
| public string GetPath(); | ||
| public virtual string GetExtraPackagePath(PythonLocationMetadata location) { | ||
| var envLibPath = string.Empty; | ||
| if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) | ||
| envLibPath = Path.Combine(GetPath(), "Lib", "site-packages"); | ||
| else | ||
| { | ||
| string suffix = location.FreeThreaded ? "t" : ""; | ||
| envLibPath = Path.Combine(GetPath(), "lib", $"python{location.Version.Major}.{location.Version.Minor}{suffix}", "site-packages"); | ||
| } | ||
| Logger.LogDebug("Adding environment site-packages to extra paths: {VenvLibPath}", envLibPath); | ||
| return envLibPath; | ||
| } | ||
| public void EnsureEnvironment(PythonLocationMetadata pythonLocation); | ||
|
|
||
| } |
46 changes: 46 additions & 0 deletions
46
src/CSnakes.Runtime/EnvironmentManagement/VenvEnvironmentManagement.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| using CSnakes.Runtime.Locators; | ||
| using Microsoft.Extensions.Logging; | ||
|
|
||
| namespace CSnakes.Runtime.EnvironmentManagement; | ||
| internal class VenvEnvironmentManagement(ILogger logger, string path, bool ensureExists) : IEnvironmentManagement | ||
| { | ||
| ILogger IEnvironmentManagement.Logger => logger; | ||
|
|
||
| public void EnsureEnvironment(PythonLocationMetadata pythonLocation) | ||
| { | ||
| if (!ensureExists) | ||
| return; | ||
|
|
||
| if (string.IsNullOrEmpty(path)) | ||
| { | ||
| logger.LogError("Virtual environment location is not set but it was requested to be created."); | ||
| throw new ArgumentNullException(nameof(path), "Virtual environment location is not set."); | ||
| } | ||
| var fullPath = Path.GetFullPath(path); | ||
| if (!Directory.Exists(path)) | ||
| { | ||
| logger.LogInformation("Creating virtual environment at {VirtualEnvPath} using {PythonBinaryPath}", fullPath, pythonLocation.PythonBinaryPath); | ||
| var (process1, _, _) = ProcessUtils.ExecutePythonCommand(logger, pythonLocation, $"-VV"); | ||
| var (process2, _, error) = ProcessUtils.ExecutePythonCommand(logger, pythonLocation, $"-m venv {fullPath}"); | ||
|
|
||
| if (process1.ExitCode != 0 || process2.ExitCode != 0) | ||
| { | ||
| logger.LogError("Failed to create virtual environment."); | ||
| process1.Dispose(); | ||
| process2.Dispose(); | ||
| throw new InvalidOperationException($"Could not create virtual environment. {error}"); | ||
| } | ||
| process1.Dispose(); | ||
| process2.Dispose(); | ||
| } | ||
| else | ||
| { | ||
| logger.LogDebug("Virtual environment already exists at {VirtualEnvPath}", fullPath); | ||
| } | ||
| } | ||
|
|
||
| public string GetPath() | ||
| { | ||
| return path; | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.