Skip to content

v0.232.0
Compare
Choose a tag to compare
@wjcunningham7 wjcunningham7 released this 10 Jan 18:54
· 35 commits to develop since this release
v0.232.0
a928f47
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

[0.232.0] - 2024-01-10

Authors

Added

  • check for /bin/bash AND /bin/sh (in that order) to execute bash leptons
  • Programmatic equivalents of CLI commands covalent start and covalent stop
  • Documentation and test cases for database triggers.
  • Added the __pow__ method to the Electron class
  • New Runner and executor API to bypass server-side memory when running tasks.
  • Added qelectron_db as an asset to be transferred from executor's machine to covalent server
  • New methods to qelectron_utils, replacing the old ones
  • Covalent deploy CLI tool added - allows provisioning infras directly from covalent
  • Added a py.typed file to support type-checking
  • Corrected support from distributed Hamiltonian expval calculations
  • Exposed qelectron db in sdk result object
  • UI changes added for qelectrons and fix for related config file corruption
  • UI fix regarding Qelectron not showing up
  • Performance optimisation of UI for large Qelectrons
  • File transfer strategy for GCP storage
  • Add CLI status for zombie, stopped process.
  • Fix for double locking file in configurations.
  • Introduced new data access layer
  • Introduced Shutil file transfer strategy for local file transfers
  • File transfer strategy for Azure blob storage
  • executor property to Electron class, allowing updation of executor after electron function definition
  • Added ability to hide post-processing electrons on the UI.
  • Added prettify of names for the graph screen on the UI.
  • Ability to specify a workdir for local and dask executors along with create_unique_workdir option for each electron / node.
  • Heartbeat file is created and updated when the server is running
  • Added SQLiteTrigger class to the triggers module.
  • Added the CloudResourceManager class
  • A new tutorial for a dynamic quantum chemistry workflow
  • Tutorial for hybrid neural network using Covalent, AWSBatch, and Qiskit Runtime.
  • Environment variable that users can set to specify the location where Covalent can find their defined executors.
  • Task group id in Electrons.
  • Reconstruct postprocessing method.
  • Postprocessing as electrons.
  • Postprocessing class in postprocessing.py module for all the different postprocessing helper methods and algorithms.
  • disable_run option added to enable "saving-only" option on covalent server and not executing the workflow
  • register_triggers, stop_triggers functions added to LocalDispatcher class
  • triggers parameter to the lattice metadata
  • BaseTrigger, DirTrigger, TimeTrigger classes added available to be assigned to any lattice enabling the triggers feature
  • TriggerLoader class added enabling loading of any kind of triggers including user defined ones without requiring installation
  • CLI options to start covalent server in triggers only, and no triggers mode
  • is_pending option added during redispatch to resume execution of a previously "saved-only", i.e pending workflow
  • API routes added for Triggers server
  • Added feature to support cancelling workflow dispatches
  • Updating/adding new tests to improve code coverage
  • API endpoint for redispatching.
  • Unit and functional tests for redispatching.
  • TransportGraphOps class for diffing operations on transport graphs.
  • Added make derived dispatch method.
  • Apply electron updates method to _TransportGraph.
  • Workflow re-dispatching functionality.
  • Added dispatcher_addr argument to ct.get_result similar to ct.dispatch so that it doesn't always fallback to using the default configured address
  • Front-end pending unit tests for the GUI.
  • Adding support for PostgresQL DB backend
  • Added check for COVALENT_DATABASE_URL, if exists connect sqlalchemy engine using that
  • Adding COVALENT_DATABASE_USER and COVALENT_DATABASE_PASSWORD environment variables
  • Adding COVALENT_DATABASE_HOSTNAME and COVALENT_DATABASE_PORT environment variables for easy configuration
  • Added TaskRuntimeError exception for executor plugin implementations to signal to Covalent that a task raised an
    unhandled exception while running in the executor backend.
  • Added environment variable for a remote database backend
  • Added support for mysql and postgresql
  • Support for transferring the contents of folders to and from S3 buckets using the file transfer module.
  • RTD for User Interface
  • Minor GUI fixes
  • Ability to use terminal on the GUI.

Changed

  • Terraform output to use scrolling buffer.
  • Terraform output handling to show errors.
  • Bumped up babel/traverse version to 7.23.2
  • Changed the axios version on the webapp side.
  • Changed the azurebatch.rst banner from default covalent jpg to azure batch's svg file
  • [Significant Changes] Improving memory management part 1/3
  • Removed strict version pins on lmdbm, mpire, orjson, and pennylane
  • Changed license to Apache
  • Improved error handling in generate_docs.py
  • [Significant Changes] Migrated core server-side code to new data access layer.
  • Changed the way UI was accessing the qelectron database to access it directly from the mdb file in object store
  • Update version of browserverify-sign
  • Limiting cloudpickle version to less than 3.0 for now
  • Raised the minimum version of Pydantic from 1.10.1 to 2.1.1 in requirements.txt
  • Electron DAL to use Covalent server's data instead of QServer's data.
  • Renamed QCluster's selector_serialized attribute so it gets propagated to the qserver.
  • Removed orm_mode = True in covalent_ui/api/v1/models/dispatch_model.py as it is deprecated in Pydantic 2
  • Removed the upper limit from dask and distributed packages' versions until we find a version which is incompatible with Covalent.
  • When the server is stopped, any workflows in a non-terminal state are first cancelled
  • Pinned sqlalchemy version with upper limit <2.0.0.
  • Added rich support to cli for better printing statements.
  • Performed minor modifications and rearrangementsto fix the broken tests in the nightly workflow.
  • Reverting nightly frequency back to midnight basis
  • Temporarily changing the nightly frequency to every hour
  • Error messages are propagated to stdout when the server is not started. These changes are applied to dispatch, redispatch, and get_result.
  • Implementation of TransportableObject property method to be backwards compatible with version 0.209.1.
  • Moved TransportableObject from transport.py to a new file transportable_object.py.
  • Postprocessing logic.
  • Sublattice logic. Sublattices are now treated as electrons. Once the transport graph has been built, the status get changed to DISPATCHING at which point it is executed as another workflow.
  • Modified ct.get_result to allow for status only requests
  • Enhanced the Dockerfile to include builds from various sources and a differentiation between SDK and Server builds
  • Location of function to load result from the database now moved to load module in covalent_dispatcher/_db folde.
  • Changed decode-uri-component package version on webapp yarn-lock file.
  • Changed json5 package version on webapp yarn-lock file.
  • Reverted nightly frequency back to once a day
  • Updated requirements.txt to include pyscopg2
  • Refactored execution.py into loosely coupled modular pieces
  • Updated directory like default environment variable paths to avoid creating redundant nested directories when self-hosting
  • Docs for Covalent's Slurm plugin updated with explanation for optional srun parameters.
  • Electron errors are segregated by type; task runtime errors are
    stored in stderr while the error attribute of a node is reserved
    for exceptions raised by Covalent itself.
  • When tasks fail in a workflow, the Lattice ErrorCard in the UI summarizes the failed tasks.
  • Width of lattice name column on dispatch list GUI.
  • Optimzing larger graphs for better performance.
  • Running migrations automatically if none have run in the past (fresh installs, after purging)
  • Single line call to join instead of a for loop
  • Updated black, mirrors-prettier, and detect-secrets in pre-commit hooks
  • Made root_dispatch_id nullable to circumvent migration issues with sqlite in certain platforms

Removed

  • Removed no longer needed methods in qelectron_utils
  • Removed dispatch-id from generate_node_result method
  • Removed unused module covalent._data_store
  • Stress test files of cpu and sublattice stress tests removed from functional tests suite.
  • Duplicate mocks for UI_SRVDIR
  • Duplicate _get_cancel_requested method from covalent_dispatcher/_core/runner.py
  • Postprocessing from runners.
  • References to specific IBMQ hub/group/project in tutorial 5
  • Slurm executor reference from qaoa tutorial since it's not compatible with conda env at the moment.

Fixed

  • Lattice-default metadata attributes are now applied correctly
  • Sublattices are built using workflow_executor
  • Added covalent version attribute to Remote Executors
  • Removed unassigned variable names
  • Contributing guidelines steps for installing for the first time
  • Updated gitignore to ignore yarn files and folders for latest version of yarn
  • Fixed the bug that caused ValueError error when using KEYWORD_ONLY parameter in electron func
  • Changed code at line 218 in covalent/_shared_files/utils.py
  • Fixed usage of deprecated pydantic validation methods
  • Fixed qelectron_db retrieval in result object
  • Fixed editability of Qelectron on settings page - UI changes
  • Certain pydantic v2 related updates
  • Fixed lattice's metadata propagation to electron's metadata in case no metadata was provided to the electron
  • Formatted executor block under Qelectron job details to handle any class-type values
  • Fixed test-cases to handle latest pydantic version changes
  • Rsync command fixed to recursively copy files when using SSH
  • Removed accidentally added migrations build files
  • Updated migration script to add a default value for qelectron_data_exists in the electrons table since it cannot be nullable
  • Reduced server memory consumption during workflow processing
  • Fixed dispatcher address not showing when covalent server starts.
  • Fixed the failing tests in the nightly workflow.
  • Using filelock package now for platform independent file locking of config file. This should fix the failing tests as well as improve compatibility with Windows.
  • When stopping the server, we send the proper SIGINT signal to uvicorn instead of SIGKILL which allows the second part of the FastAPI lifespan to execute properly.
  • Fixed the outstanding incompatibities between front-end data layer and a postgres database
  • Reverted file-lock changes
  • Fixed dispatches list UI api caused by pydantic config.
  • Fixed graph API.
  • Fixed UI backend unit test case.
  • Executor and workflow executor data dictionaries are passed to sublattices
  • Read in README.md with encoding="utf-8" in setup.py
  • Resolving correct python executable
  • Error handling for random URLs/random dispatchId entered on the GUI
  • Fixed support for dynamically assigning executor to Electron class
  • Fixed nightly by activating environment before running tests in tests.yml
  • only pin aiohttp downwards to fix install on Python 3.11 #1654
  • DB path creation now takes place at import time so that the CLI commands don't fail
  • Raise error on dispatching a non-lattice
  • Helpful message when trying to dispatch a workflow when covalent server is not available
  • Open UI preview of transport graph when lattice.draw() is invoked and print URL of the same
  • Defer creation of server specific config entries until covalent is started
  • Functional tests on CI
  • Move dask worker space to covalent cache instead of site-packages
  • Result status comparison
  • Raise error on extra args/kwargs
  • Fixed redispatching and trigger server address passing in base trigger
  • Doubling of nodes that are added to the transport graph.
  • Ensure postprocessing node end time is added as the workflow end time.
  • Functional tests
  • Custom executor how to guide, and its loading mechanism in covalent server.
  • Broken postprocessing unit test.
  • UI crashing if time values are null
  • No longer adding "http://" every time a dispatcher address is provided in local dispatcher class in order to use the provided address exactly
  • Redispatch bug involving copying reusable nodes from old transport graph to new transport graph.
  • Pennylane tutorial notebook.
  • Redispatch bug.
  • Braket pip installation instructions.
  • Removing the entrypoint for SDK-only install
  • Updating client requirements to match server versions
  • Load plugins only when COVALENT_PLUGIN_LOAD environment variable has been set to a Truthy value.
  • MNIST tutorial now shows non-Null outputs and the classifier training log image has been updated.
  • Minor changes to tutorials: autoencoder, quantum and classical svm, ensemble classification, iris classification with Pennylane, quantum chemistry, DNN tutorial, qaoa, spacetime tutorial etc.
  • The range of networkx versions in requirements.txt weren't compatible with each other, thus it is pinned to 2.8.6 now
  • SDK-only sdist and installation should now work as expected, not packaging the server
  • Fixed naming of collection nodes (was breaking postprocessing)
  • Restored compatibility with stable release of AWS executors
  • Optimization of logs on the GUI for large log file sizes.
  • Fixed UI pagination not working for more than 11 pages
  • Runtime field counting down for select running dispatches
  • Removed merge conflict symbols in changelog
  • Build graph now sets all unset lattice constraints from defaults
  • Fixed all failing functional tests
  • Fixed local executor tests on MacOS by adding ProcessPoolExecutor
  • Respecting specified AWS profile & region in remote executed S3 file transfers, defaulting to env vars of execution backend
  • Electrons will inherit the lattice executors.
  • Sublattices inherit the parent lattice executor.
  • When several electrons are running concurrently, their stdout and stderr are stored in the correct graph nodes.
  • Electron errors now appear in the Electron ErrorCard when one clicks on a failed task in the UI.
  • When an electron raises an exception during execution, the local and dask executors now try to recover any output that was already
    written.
  • Fixed functional tests.
  • Added requirements-client.txt to MANIFEST file
  • Respecting specified AWS profile & region in remote executed S3 file transfers, defaulting to env vars of execution backend
  • Fixed local executor tests on MacOS (second attempt)
  • The initialize_results_dir method attempts to use an environment variable instead of the results directory in the payload
  • Modified certain sqlalchemy commands for postgres compatibility
  • Removed references to results_dir in the payload
  • Folder transfer unit test.
  • Folder transfer download bug
  • Result objects now print correctly when nodes fail
  • Enabling logging by default
  • Removed debugging output
  • Clarify cli output when covalent db migrate needs to be run
  • Rendering newlines in ErrorCard on the UI for displaying error stacktraces
  • VERSION incrementing logic in changelog
  • Fixed v11 migration to use render as batch to make DROP operations compatible with sqlite
  • covalent restart honors the sdk.no_cluster setting
  • Re-applying default executor fix post config file reunification
  • Exceptions when instantiating executors are handled
  • Covalent start now waits for the server to settle before returning

Tests

  • Temporarily skipping the sqlite and database trigger functional tests
  • Updated tests to accommodate the new qelectron fixes
  • Added new tests for the Database class and qelectron_utils
  • Covalent deploy CLI tool tests.
  • Skipping functional tests for azure blob storage and gcp storage how to guides since they require credentials to run.
  • Added testcases for GUI backend.
  • Changed the method for startup and shutdown events for pytest to work with fastapi version 0.93.0
  • Fixed test cases to adapt changes to SQLAlchemy version 1.4.49
  • Ignored remote file transfer how-to functional tests.
  • Skipping a UI backend test for now
  • Fixed test_decorated_function test case in functional tests
  • Added tests for the CloudResourceManager class
  • Re-enable test_run_workflow_does_not_deserialize
  • Updated QA stress test execution time baseline.
  • Move QA scripts from QA repo to Covalent functional tests.
  • Updated and added tests to account for all of the above triggers related changes
  • Updated _get_result_from_dispatcher test to verify whether using a link directly works or not

Docs

  • Added federated learning showcase code
  • Updated tutorial for redispatching workflows with Streamlit
  • Fix autodoc for SSH, Slurm, AWS Braket, AWS Lambda, AWS EC2, AWS Batch, Google Batch
  • Updated documentation links in README
  • Added tutorial for redispatching workflows with Streamlit
  • Added documentation for Azure Blob Storage file transfers
  • Added documentation for Google Cloud Storage file transfers
  • Enhanced the quickstart with a set of commonly used features
  • Removed duplicate "stop server" warning in the First Experimemnt page
  • Fixed typo in quickstart
  • Fix autodoc for SSH, Slurm, AWS Braket, AWS Lambda, AWS EC2, AWS Batch, Google Batch
  • Updated documentation links in README
  • Updating and reorganizing RTD API documentation
  • Adding example links in API documentation
  • Fix Lattice docstring in RTD.
  • Added a missing , to the Slurm docs.
  • Fix typo in GCP Batch executor RTD.
  • Add steps for setting up GUI for local development in contribution guidelines.
  • Fix docstring for set_config
  • Redispatch feature page in Read the Docs.
  • Clarify installation instructions for SLURM plugin in Read the Docs (x2).
  • Fix waiting order of electrons in docs inside snippet for adding a dependency when inputs and outputs are independent.
  • Expose GCP Batch executor RTD.
  • Add GCP Batch executor image in RTD.
  • Updated Feature documentation for Triggers with an example and minor corrections
  • Update requirements file for the tutorials: 1_QuantumMachineLearning/pennylane_kernel/source.ipynb and machine_learning/dnn_comparison.ipynb.
  • Add macOS 13 (Ventura) to compatibility list.
  • Fixed broken links and typos in the documentation.
  • Updated How-to documents.
  • Port of Pennylane's Univariate QVR tutorial using Covalent to this repo.
  • Adding troubleshooting guide to RTD's
  • Added a note to First Experiment offering initial intro to executors.
  • Adding Google Batch executor plugin RTD
  • Adding cancellation RTD text files
  • Added documentation for "Triggers" and a separate section for similar "Features"
  • Tutorial guidelines
  • Redispatch API section.
  • Add how to for redispatch.
  • Mention redispatch in the concepts section.
  • Update AWS Lambda executor RTD with steps to extend the base executor image for installing custom packages
  • Updated self-deployment (server deployment).
  • Added IBMQ tutorial
  • Published Self-Deployment Guide
  • Revised UI reference. Added Settings page documentation.
  • Added broken UI links in README
  • Updated compatibility matrix
  • Adding Deployment section for self-hosting guide
  • Rewrote Concepts section in docs
  • Split Concepts into API, server, and UI sections
  • Added new examples and graphics for Concepts
  • Added DNN tutorial
  • Updated AWS Plugins install instructions
  • Updated AWS Plugins documentation (minor fixes)
  • Rewrote intro material in README.
  • Changed "Citation" in the README.
  • Renamed "Release Notes" to "What's New?" in the README. Updated What's New with a description of the newest GUI functionality.
  • Added "Quick Start" guide.
  • Updated and reorganized doc landing page.
  • Rewrote "Getting Started" page.
  • Broke out "Installing from Source" instructions to separate page.
  • Corrected some API class names in headers.
  • Added an executors-and-UI graphic.
  • Adding Deployment section for self-hosting guide
  • Rewrote intro material in README.
  • Changed "Citation" in the README.
  • Renamed "Release Notes" to "What's New?" in the README. Updated What's New with a description of the newest GUI functionality.
  • Updated AWS Lambda executor docs to address conflict with using public ecr registries
  • Fixed missing RTD content under API section for covalent, cli, leptons, deps, data transfer
  • Updated RTD with details about the new AWS lambda executor interface

Operations

  • Ignore custom executor plugin in how-to's when running test_deploy_status CLI test.
  • Change the strict version pin on pennylane from ==0.33.1 to >=0.31.1,<0.33.0
  • Updated black , prettier, pycln package versions in pre-commit config
  • Changed actions/checkout@v3 to actions/checkout@v4 in CI
  • Dependabot update to npm in changelog action
  • Update tough-cookie to 4.1.3 version
  • Added rich support to cli for better printing statements.
  • Changed semver from 5.7.1 to 5.7.2 in package.json
  • Updated word-wrap to 1.2.4 version
  • Updated the nightly workflow's schedule
  • Temporarily making the nightly workflow more frequent to test the fix for the failing tests
  • Fixed failing tests
  • Respecting node version as specified in .nvmrc file for tests workflow
  • Bumped versions in pre-commit config
  • Added prettier for markdown files.
  • Reduce the number of pinned version numbers in the setup.py, requirements.txt, and requirements-client.txt
  • Updated the wci.yml file with new features
  • Bumped pre-commit versions
  • Temporarily running nightly hourly to test whether the fix worked
  • Reverted to daily frequency for nightly
  • Update the PR template.
  • Nightly frequency set to midnight EST
  • Removed "already released" check from stable changelog action
  • Updating nightly frequency (temp)
  • Lowering number of jest workers as an attempt to fix flaky UI functional tests
  • Added exception for nightly to pass if conda release fails
  • Updated pre-commit hook versions
  • Updated codecov upload steps in tests workflow to fail if upload to codecov fails
  • Added job in nightly to trigger base executor image builds after a Covalent pre-release
  • Fixed stable-changelog action removed .catch added .on('error')
  • Removed AWS base executor deployment from release.yml
  • Removed experimental tests from nightly test matrix (will be brought back but in different workflow)
  • Updated release workflow to continue if release tag already exists
  • Reverted nightly schedule back to daily at 4:00am
  • Added Alejandro to PAUL_BLART group to allow trigerring of releases
  • Updated hotfix logic to run on a merge to a release branch
  • CodeQL workflow uses a test matrix to scan all repos in the Covalent ecosystem
  • Updated all CI Slack alerts to all go to the #covalent-ci channel
  • Removed PAUL_BLART check on build sdist step in release.yml
  • Consolidated pre & stable build into one step in release.yml
  • Fixing the validate distribution step given changes in -rc0 suffix to version
  • updated hotfix logic to run on a merge to a release branch
  • Fixing js github actions dist by re-building from develop
  • Fixing syntax in describe action & compiled action manually
Assets 2
Loading