v3.0.0

Due to a number of breaking changes, the version has been bumped to 3.0.0. However, the vast majority
of typical use patterns in typical user code will not require changes. Please review the specifics
below.

Catalog

• The tags attributes on Catalog objects can now contain up to 32 elements, each up to 1000 characters long.
  But why would you even want to go there?
• Breaking Change: Derived bands, never supported in the AWS environment and catalog products, have been
  removed.
• The new Blob.delete_many method may be used to delete large numbers of blobs efficiently.
• The Blob.get_or_create method didn't allow supplying storage_type, namespace, or name parameters.
  Now it works as expected, either returning a saved Blob from the Catalog, or an unsaved blob that
  you can use to upload and save its data.
• Image methods ndarray and download no longer pass the image's default geocontext geometry as a cutline.
  This is to avoid problems when trying to raster a complete single image in its native CRS and resolution
  where imperfect geometries (due to a simplistic projection to EPSG:4326) can cause some boundary pixels
  to be masked. When passing in an explicit GeoContext to these methods, consider whether any cutline
  geometry is required or not, to avoid these issues.

Compute

• Function and Job objects now have a new environment attribute which can be used to define environment
  variables for the jobs when they are run.
• Breaking Change: The Function.map method previously had no bound on how many jobs could be created at one time.
  This led to operational problems with very large numbers of jobs. Now it submits jobs in batches (up to 1000
  jobs per batch) to avoid request timeouts, and is more robust on retryable errors so that duplicate jobs are not
  submitted accidently. There is still no bound on how many jobs you may create with a single call to Function.map.
  Additionally, since it is possible that some jobs may be successfully submitted, and others not, the return
  value, while still behaving as a list of Jobs, is now a JobBulkCreateResult object which has a is_success
  and an error property which can be used to determine if all submissions were successful, what errors may
  have occurred, and what jobs have actually been created. Only if the first batch fails hard will the method
  raise an exception.
• The Job.statistics member is now typed as a JobStatistics object.
• The efficiency of deleting many jobs at once has been significantly improved using Function.delete and
  Function.delete_jobs. It is still possible to encounter request timeouts with very large numbers of jobs;
  workarounds are now documented in the API documentation for the Function.delete_jobs method.
• The ComputeClient.check_credentials method has been added, so that the client can determine if valid
  user credentials have already been registered with the Compute service.

Vector

• The Vector client library, previously available as the descarteslabs-vector package on PyPI, has
  now been integrated into the Descartes Labs Python Client (this package). It should no longer be
  installed separately.
• Visualization support (ipyleaflet.Map) is enabled when ipyleaflet is available. It is not
  installed by default, but can be installed manually, or by installing the descarteslabs python
  client with the viz extra (e.g. pip install descarteslabs[viz]). Note that in order to be
  compatible with jupyterlab notebooks, the visualize() method no longer returns the layer, it
  just adds it to the supplied map.
• The Vector package now has a VectorClient API client, with the usual support for get_default_client()
  and set_default_client(). Most constructors and methods now accept an optional client= parameter
  if you need to use something other than the default client.
• Configuration is now accomplished using the standard descarteslabs.config package. In particular,
  the vector_url setting is used to specify the default Vector host. The VECTOR_API_HOST environment
  variable is no longer consulted.
• Vector client methods now raise standard descarteslabs.exceptions Exception classes rather than
  the descarteslabs.vector.vector_exceptions classes of the old client.
• The is_spatial= parameter previously accepted by many methods and functions is now deprecated
  and ignored. It is not required because existing type information always determines if an operation
  is spatial or not. Warnings will be generated if it is used.
• Be advised that feature upload and download (query) do not currently support or impose any limits,
  and thus allow operations so large and slow that timeouts or other failures may occur. A future
  version will implement limits and batching, so that large operations can be supported reliably.
  Until then, the user may wish to implement their own batching were possible to avoid encountering
  network limits and timeouts.

General

• The old client version v1.12.1 is reaching end of life and will longer be supported as of February 2024.
  You can expect the version to stop working at any point after that as legacy backend support is turned off.
• Breaking Change: The deprecated Scenes client API has been removed.
• Breaking Change: The deprecated Metadata client API has been removed.
• The minimum required version of urllib3 has been bumped to 1.26.18 to address a security vulnerability.
• The minimum required version of shapely has been bumped to 2.0.0 to address thread safety issues.
• Python 3.7, formerly deprecated, is no longer supported.
• Python 3.12 is not yet officially supported due to the lack of support from blosc. However, if you
  are able to provide a functional blosc on your own, then 3.12 should work.
• Urllib3 2.X is now supported.
• Geopandas, Pydantic, and PyArrow have been added as core dependencies to support the Vector client.
• For those users of the clear_client_state function (not common), the bands cache for the Catalog client
  is now cleared also.

(note that these release notes are duplicated from the non-public v3.0.0rc0)

v3.0.0rc0

Due to a number of breaking changes, the version has been bumped to 3.0.0. However, the vast majority
of typical use patterns in typical user code will not require changes. Please review the specifics
below.

Catalog

• The tags attributes on Catalog objects can now contain up to 32 elements, each up to 1000 characters long.
  But why would you even want to go there?
• Breaking Change: Derived bands, never supported in the AWS environment and catalog products, have been
  removed.
• The new Blob.delete_many method may be used to delete large numbers of blobs efficiently.
• The Blob.get_or_create method didn't allow supplying storage_type, namespace, or name parameters.
  Now it works as expected, either returning a saved Blob from the Catalog, or an unsaved blob that
  you can use to upload and save its data.
• Image methods ndarray and download no longer pass the image's default geocontext geometry as a cutline.
  This is to avoid problems when trying to raster a complete single image in its native CRS and resolution
  where imperfect geometries (due to a simplistic projection to EPSG:4326) can cause some boundary pixels
  to be masked. When passing in an explicit GeoContext to these methods, consider whether any cutline
  geometry is required or not, to avoid these issues.

Compute

• Function and Job objects now have a new environment attribute which can be used to define environment
  variables for the jobs when they are run.
• Breaking Change: The Function.map method previously had no bound on how many jobs could be created at one time.
  This led to operational problems with very large numbers of jobs. Now it submits jobs in batches (up to 1000
  jobs per batch) to avoid request timeouts, and is more robust on retryable errors so that duplicate jobs are not
  submitted accidently. There is still no bound on how many jobs you may create with a single call to Function.map.
  Additionally, since it is possible that some jobs may be successfully submitted, and others not, the return
  value, while still behaving as a list of Jobs, is now a JobBulkCreateResult object which has a is_success
  and an error property which can be used to determine if all submissions were successful, what errors may
  have occurred, and what jobs have actually been created. Only if the first batch fails hard will the method
  raise an exception.
• The Job.statistics member is now typed as a JobStatistics object.
• The efficiency of deleting many jobs at once has been significantly improved using Function.delete and
  Function.delete_jobs. It is still possible to encounter request timeouts with very large numbers of jobs;
  workarounds are now documented in the API documentation for the Function.delete_jobs method.
• The ComputeClient.check_credentials method has been added, so that the client can determine if valid
  user credentials have already been registered with the Compute service.

Vector

• The Vector client library, previously available as the descarteslabs-vector package on PyPI, has
  now been integrated into the Descartes Labs Python Client (this package). It should no longer be
  installed separately.
• Visualization support (ipyleaflet.Map) is enabled when ipyleaflet is available. It is not
  installed by default, but can be installed manually, or by installing the descarteslabs python
  client with the viz extra (e.g. pip install descarteslabs[viz]). Note that in order to be
  compatible with jupyterlab notebooks, the visualize() method no longer returns the layer, it
  just adds it to the supplied map.
• The Vector package now has a VectorClient API client, with the usual support for get_default_client()
  and set_default_client(). Most constructors and methods now accept an optional client= parameter
  if you need to use something other than the default client.
• Configuration is now accomplished using the standard descarteslabs.config package. In particular,
  the vector_url setting is used to specify the default Vector host. The VECTOR_API_HOST environment
  variable is no longer consulted.
• Vector client methods now raise standard descarteslabs.exceptions Exception classes rather than
  the descarteslabs.vector.vector_exceptions classes of the old client.
• The is_spatial= parameter previously accepted by many methods and functions is now deprecated
  and ignored. It is not required because existing type information always determines if an operation
  is spatial or not. Warnings will be generated if it is used.
• Be advised that feature upload and download (query) do not currently support or impose any limits,
  and thus allow operations so large and slow that timeouts or other failures may occur. A future
  version will implement limits and batching, so that large operations can be supported reliably.
  Until then, the user may wish to implement their own batching were possible to avoid encountering
  network limits and timeouts.

General

• The old client version v1.12.1 is reaching end of life and will longer be supported as of February 2024.
  You can expect the version to stop working at any point after that as legacy backend support is turned off.
• Breaking Change: The deprecated Scenes client API has been removed.
• Breaking Change: The deprecated Metadata client API has been removed.
• The minimum required version of urllib3 has been bumped to 1.26.18 to address a security vulnerability.
• The minimum required version of shapely has been bumped to 2.0.0 to address thread safety issues.
• Python 3.7, formerly deprecated, is no longer supported.
• Python 3.12 is not yet officially supported due to the lack of support from blosc. However, if you
  are able to provide a functional blosc on your own, then 3.12 should work.
• Urllib3 2.X is now supported.
• Geopandas, Pydantic, and PyArrow have been added as core dependencies to support the Vector client.
• For those users of the clear_client_state function (not common), the bands cache for the Catalog client
  is now cleared also.

v2.1.2

Compute

• Function.delete_jobs was failing to implement the delete_results parameter, so job result blobs
  were not being deleted. This has been fixed.
• Add delete_results parameter to Function.delete for consistency.
• Job.statistics field added which contains statistics (CPU, memory, and network utilization) for the
  job. This can be used to determine the minimal resources necessary for the Function after some
  representative runs.

v2.1.1

Compute

• Filtering on datetime attributes (such as Function.created_at) didn't previously work with anything
  but datetime instances. Now it also handles iso format strings and unix timestamps (int or float).

v2.1.0

General

• Following our lifecycle policy, client versions v1.11.0 and earlier are no longer supported. They may
  cease to work with the Platform at any time.

Catalog

• The Catalog Blob class now has a get_data() method which can be used to retrieve the blob
  data directly given the id, without having to first retrieve the Blob metadata.

Compute

• Breaking Change The status values for Function and Job objects have changed, to provide a
  better experience managing the flow of jobs. Please see the updated Compute guide for a full explanation.
  Because of the required changes to the back end, older clients (i.e. v2.0.3) are supported in a
  best effort manner. Upgrading to this new client release is strongly advised for all users of the
  Compute service.

• Breaking Change The base images for Compute have been put on a diet. They are now themselves built
  from "slim" Python images, and they no longer include the wide variety of extra Python packages that were
  formerly included (e.g. TensorFlow, SciKit Learn, PyTorch). This has reduced the base image size by
  an order of magnitude, making function build times and job startup overhead commensurately faster.
  Any functions which require such additional packages can add them in as needed via the requirements=
  parameter. While doing so will increase image size, it will generally still be much smaller and faster
  than the prior "Everything and the kitchen sink" approach. Existing Functions with older images will continue
  to work as always, but any newly minted `Function`` using the new client will be using one of the new
  slim images.

• Base images are now available for Python3.10 and Python3.11, in addition to Python3.8 and Python3.9.

• Job results and logs are now integrated with Catalog Storage, so that results and logs can be
  searched and retrieved directly using the Catalog client as well as using the methods in the Compute
  client. Results are organized under storage_type=StorageType.COMPUTE, while logs are organized under
  storage_type=StorageType.LOGS.

• The new ComputeResult class can be used to wrap results from a Function, allowing the user to
  specify additional attributes for the result which will be stored in the Catalog Blob metadata for
  the result. This allows the function to specify properties such as geometry, description,
  expires, extra_attributes, writers and readers for the result Blob. The use of
  ComputeResult is not required.

• A Job can now be assigned arbitrary tags (strings), and searched based on them.

• A Job can now be retried on errors, and jobs track error reasons, exit codes, and execution counts.

• Function and Job objects can now be filtered by class attributes (ex.
  Job.search().filter(Job.status == JobStatus.PENDING).collect()).

• The Job.cancel() method can now be used to cancel the execution of a job which is currently
  pending or running. Pending jobs will immediately transition to JobStatus.CANCELED status,
  while running jobs will pass through JobStatus.CANCEL (waiting for the cancelation to be
  signaled to the execution engine), JobStatus.CANCELING (waiting for the execution to terminate),
  and JobStatus.CANCELED (once the job is no longer executing). Cancelation of running jobs is
  not guaranteed; a job may terminate successfully, or with a failure or timeout, before it can
  be canceled.

• The Job.result() method will raise an exception if the job does not have a status of
  JobStatus.SUCCESS. If Job.result() yields an None value, this means that there was no
  result (i.e. the execution returned a None).

• The Job.result_blob() method will return the Catalog Storage Blob holding the result, if any.

• The Job.delete() method will delete any job logs, but will not delete the job result unless
  the delete_results parameter is supplied.

• The Function object now has attributes namespace and owner.

• The Function.wait_for_completion() and new Function.as_completed() methods provide a richer
  set of functionality for waiting on and handling job completion.

• The Function.build_log() method now returns the log contents as a string, rather than printing
  the log contents.

• The Job.log() method now returns the log contents as a list of strings, rather than printing the log
  contents. Because logs can be unbounded in size, there's also a new Job.iter_log() method which returns
  an iterator over the log lines.

• The requirements= parameter to Function objects now supports more pip magic, allowing the use
  of special pip controls such as -f. Also parsing of package versions has been loosened to allow
  some more unusual version designators.

• Changes to the Function.map() method, with the parameter name change of iterargs changed to kwargs
  (the old name is still honored but deprecated), corrected documentation, and enhancements to support more
  general iterators and mappings, allowing for a more functional programming style.

• The compute package was restructured to make all the useful and relevant classes available at the top level.

Utils

• Property filters can now be deserialized as well as serialized.

v2.1.0rc1

[2.1.0] - 2023-09-21

General

• Following our lifecycle policy, client versions v1.11.0 and earlier are no longer supported. They may
  cease to work with the Platform at any time.

Catalog

• The Catalog Blob class now has a get_data() method which can be used to retrieve the blob
  data directly given the id, without having to first retrieve the Blob metadata.

Compute

• Breaking Change The status values for Function and Job objects have changed, to provide a
  better experience managing the flow of jobs. Please see the updated Compute guide for a full explanation.
  Because of the required changes to the back end, older clients (i.e. v2.0.3) are supported in a
  best effort manner. Upgrading to this new client release is strongly advised for all users of the
  Compute service.

• Breaking Change The base images for Compute have been put on a diet. They are now themselves built
  from "slim" Python images, and they no longer include the wide variety of extra Python packages that were
  formerly included (e.g. TensorFlow, SciKit Learn, PyTorch). This has reduced the base image size by
  an order of magnitude, making function build times and job startup overhead commensurately faster.
  Any functions which require such additional packages can add them in as needed via the requirements=
  parameter. While doing so will increase image size, it will generally still be much better than the prior
  "Everything and the kitchen sink" approach. Existing Functions with older images will continue
  to work as always, but any newly minted `Function`` using the new client will be using one of the new
  slim images.

• Base images are now available for Python3.10 and Python3.11, in addition to Python3.8 and Python3.9.

• Job results and logs are now integrated with Catalog Storage, so that results and logs can be
  searched and retrieved directly using the Catalog client as well as using the methods in the Compute
  client.

• The new ComputeResult class can be used to wrap results from a Function, allowing the user to
  specify additional attributes for the result which will be stored in the Catalog Blob metadata for
  the result. This allows the function to specify properties such as geometry, description,
  expires and extra_attributes for the result Blob. The use of ComputeResult is not required.

• A Job can now be assigned arbitrary tags (strings), and searched based on them.

• A Job can now be retried on errors, and jobs track error reasons, exit codes, and execution counts.

• Function and Job objects can now be filtered by class attributes (ex.
  Job.search().filter(Job.status == JobStatus.PENDING).collect()).

• The Job.cancel() method can now be used to cancel the execution of a job which is currently
  pending or running. Pending jobs will immediately transition to JobStatus.CANCELED status,
  while running jobs will pass through JobStatus.CANCEL (waiting for the cancelation to be
  signaled to the execution engine), JobStatus.CANCELING (waiting for the execution to terminate),
  and JobStatus.CANCELED (once the job is no longer executing). Cancelation of running jobs is
  not guaranteed; a job may terminate successfully, or with a failure or timeout, before it can
  be canceled.

• The Job.result() method will raise an exception if the job does not have a status of
  JobStatus.SUCCESS. If Job.result() yields an None value, this means that there was no
  result (i.e. the execution returned a None).

• The Job.result_blob() will return the Catalog Storage Blob holding the result, if any.

• The Function object now has attributes namespace and owner.

• The Function.wait_for_completion() and new Function.as_completed() methods provide a richer
  set of functionality for waiting on and handling job completion.

• The Function.build_log() method now returns the log contents as a string, rather than printing
  the log contents.

• The Job.log() method now returns the log contents as a list of strings, rather than printing the log
  contents. Because logs can be unbounded in size, there's also a new Job.iter_log() method which returns
  an iterator over the log lines.

• The requirements= parameter to Function objects now supports more pip magic, allowing the use
  of special pip controls such as -f. Also parsing of package versions has been loosened to allow
  some more unusual version designators.

• Changes to the Function.map() method, with the parameter name change of iterargs changed to kwargs
  (the old name is still honored but deprecated), corrected documentation, and enhancements to support more
  general iterators and mappings, allowing for a more functional programming style.

• The compute package was restructured to make all the useful and relevant classes available at the top level.

Utils

• Property filters can now be deserialized as well as serialized.

v2.1.0rc0

General

• Following our lifecycle policy, client versions v1.11.0 and earlier are no longer supported. They may
  cease to work with the Platform at any time.

Catalog

• The Catalog Blob class now has a get_data() method which can be used to retrieve the blob
  data directly given the id, without having to first retrieve the Blob metadata.

Compute

• Breaking Change The base images for Compute have been put on a diet. They are now themselves built
  from "slim" Python images, and they no longer include the wide variety of extra Python packages that were
  formerly included (e.g. TensorFlow, SciKit Learn, PyTorch). This has reduced the base image size by
  an order of magnitude, making function build times and job startup overhead commensurately faster.
  Any functions which require such additional packages can add them in as needed via the requirements=
  parameter. While doing so will increase image size, it will generally still be much better than the prior
  "Everything and the kitchen sink" approach. Existing Functions with older images will continue
  to work as always, but any newly minted `Function`` using the new client will be using one of the new
  slim images.

• Base images are now available for Python3.10 and Python3.11, in addition to Python3.8 and Python3.9.

• Job results and logs are now integrated with Catalog Storage, so that results and logs can be
  searched and retrieved directly using the Catalog client as well as using the methods in the Compute
  client.

• The new ComputeResult class can be used to wrap results from a Function, allowing the user to
  specify additional attributes for the result which will be stored in the Catalog Blob metadata for
  the result. This allows the function to specify properties such as geometry, description,
  expires and extra_attributes for the result Blob. The use of ComputeResult is not required.

• A Job can now be assigned arbitrary tags (strings), and searched based on them.

• A Job can now be retried on errors, and jobs track error reasons, exit codes, and execution counts.

• Function and Job objects can now be filtered by class attributes (ex. Job.search().filter(Job.status == JobStatus.PENDING).collect()).

• The requirements= parameter to Function objects now supports more pip magic, allowing the use
  of special pip controls such as -f. Also parsing of package versions has been loosened to allow
  some more unusual version designators.

• Changes to the Function.map() method, with the parameter name change of iterargs changed to kwargs
  (the old name is still honored but deprecated), corrected documentation, and enhancements to support more
  general iterators and mappings, allowing for a more functional programming style.

• The compute package was restructured to make all the useful and relevant classes available at the top level.

Utils

• Property filters can now be deserialized as well as serialized.

v2.0.3

Compute

• Allow deletion of Function objects.
  • Deleting a Function will deleted all associated Jobs.
• Allow deletion of Job objects.
  • Deleting a Job will delete all associated resources (logs, results, etc).
• Added attribute filter to Function and Job objects.
  • Attributes marked filterable=True can be used to filter objects on the compute backend api.
  • Minor optimization to Job.iter_results which now uses backend filters to load successful jobs.
• Function bundling has been enhanced.
  • New include_modules and include_data parameters allow for multiple other modules, non-code data files, etc to be added to the code bundle.
  • The requirements parameter has been improved to allow a user to pass a path to their own requirements.txt file instead of a list of strings.

v2.0.2

Catalog

• Allow data type int32 in geotiff downloads.
• BlobCollection now importable from descarteslabs.catalog.

Documentation

• Added API documentation for dynamic compute and vector

v2.0.1

Raster

• Due to recent changes in urllib3, rastering operations were failing to retry certain errors which ought to be retried, causing more failures to propagate to the user than was desirable. This is now fixed.