Skip to content

ESA-PhiLab/phidown

BranchesTags

Repository files navigation

Phi-Down Logo

GitHub pull requests GitHub last commit GitHub code size Python 3.9+ Documentation Status DOI

Φ-Down: Your Gateway to Copernicus Data website

🌐 Connect with Φ-Lab & the Author

Overview

Φ-Down is a Python library that simplifies accessing Earth Observation data from the Copernicus Data Space Ecosystem using AWS services and dataset tools. It provides a streamlined interface for searching, filtering, and downloading satellite imagery from various Sentinel missions and contributing datasets.

Whether you're a researcher, developer, or data scientist, Φ-Down makes it easy to incorporate satellite data into your workflows without dealing with complex APIs or authentication processes.

Feature Description
Example Notebook Open in Colab
Python Support Python 3.9+
PyPi PyPI Version
License License: GPL v3
Documentation Documentation Status
Contributions PRs Welcome

🚀 What's Next & Feature Status

  • ✔️ Done: Replace s3 with s5cmd for data transfers
  • ✔️ Done: Add a visualisation tool
  • ✔️ Done: Add a S-1 reference in docs.
  • ✔️ Done: Removed old boto3 dep.
  • Coming soon: Implement a count option for search results more than 1000
  • Coming soon: Add a parallel download executor
  • Coming soon: Improve docs for ALL missions.

Features

  • Authenticate with the Copernicus Data Space Ecosystem.
  • Search for Sentinel products using the OData API with various filters query_by_filter (collection, product type, date, AOI, cloud cover, etc.).
  • Search for products by exact name using query_by_name.
  • Download Sentinel products using the S3 protocol.

Here's a quick example of how to use Φ-Down to search and download Sentinel data:

from phidown.search import CopernicusDataSearcher
# You can modify these parameters as needed

searcher = CopernicusDataSearcher()
# Configure the search parameters
searcher.query_by_filter(
    collection_name='SENTINEL-1',
    product_type='SLC',
    orbit_direction=None,
    cloud_cover_threshold=None,
    aoi_wkt=None,
    start_date='2023-05-03T00:00:00',
    end_date='2024-05-03T04:00:00',
    top=1000,
    attributes={'swathIdentifier': 'S1'} # Swath identifier Stripmap: S1, ..., S6
)

df = searcher.execute_query()
print(f"Number of results: {len(df)}")
# Display the first few rows of the DataFrame
searcher.display_results(top_n=15)

For more advanced use cases, including searching with geographical filters and batch downloading, see the usage notebook.

📚 Documentation

Comprehensive documentation is available at: https://esa-philab.github.io/phidown

Credentials

Important: If your credentials are not deleted automatically, you must manually revoke them here:

🔗 Manage S3 Credentials: https://eodata-s3keysmanager.dataspace.copernicus.eu/panel/s3-credentials

Stay secure — always clean up your credentials!

Supported Missions

The following collections are currently available:

Copernicus Sentinel Mission
  • SENTINEL-1
  • SENTINEL-2
  • SENTINEL-3
  • SENTINEL-5P
  • SENTINEL-6
  • SENTINEL-1-RTC (Sentinel-1 Radiometric Terrain Corrected)
Complementary data
  • GLOBAL-MOSAICS (Sentinel-1 and Sentinel-2 Global Mosaics)
  • SMOS (Soil Moisture and Ocean Salinity)
  • ENVISAT (ENVISAT- Medium Resolution Imaging Spectrometer - MERIS)
  • LANDSAT-5
  • LANDSAT-7
  • LANDSAT-8
  • COP-DEM (Copernicus DEM)
  • TERRAAQUA (Terra MODIS and Aqua MODIS)
  • S2GLC (S2GLC 2017)
Copernicus Contributing Missions (CCM)

1. Historical ESA Missions

  • ERS-1 / ERS-2 – Radar and atmospheric data (1991–2011)
  • Envisat – Multispectral and radar data for land, ocean, and atmospheric monitoring (2002–2012)

2. ESA Earth Explorer Missions

  • SMOS – Soil moisture and ocean salinity measurements
  • CryoSat-2 – Ice thickness and elevation monitoring

3. European National and Commercial Missions

  • SPOT (4/5) – High-resolution optical imagery
  • TerraSAR-X / TanDEM-X – High-resolution SAR data
  • COSMO-SkyMed – SAR imagery for various applications
  • Pléiades – Very high-resolution optical data
  • RapidEye – Multispectral imagery for vegetation monitoring
  • DMC Constellation – Disaster monitoring with optical imagery

4. International Missions

  • Landsat (NASA/USGS) – Long-term optical data archives
  • Radarsat-2 (Canada) – SAR data for land and marine applications
  • GOSAT (Japan) – Greenhouse gas monitoring
  • GHGSat – High-resolution greenhouse gas measurements

5. Emerging European Missions

Includes nine under-contract missions to deliver multispectral, hyperspectral, thermal infrared, and atmospheric composition data. These missions will be progressively integrated as part of the Copernicus data offering.

More info

Prerequisites

  • Python >= 3.9

Installation

Option 1: Install from PyPI (Recommended)

PyPI

You can install it directly using pip:

pip install phidown
Build from source

Install with PDM (Using Lock File)

We recommend using PDM (Python Dependency Manager) for managing dependencies, especially for development or ensuring exact versions.

Step 1: Install PDM

If you don't already have PDM installed, install it via pip:

pip install pdm

Step 2: Install Dependencies Using PDM

Clone the repository and install dependencies using the pdm.lock file:

git clone https://github.com/ESA-PhiLab/phidown.git
cd phidown
pdm install

Step 3: Install All Dependencies (Including Optional)

To install all dependencies, including optional ones for development or specific features:

pdm install -G :all

Option 3: Build from Source (Using pip)

If you prefer using pip without PDM, you can clone the repository and install directly:

git clone https://github.com/ESA-PhiLab/phidown.git
cd phidown
pip install .

This will install the package and its dependencies as defined in the pyproject.toml file.

Usage

Configure Credentials

To authenticate with the Copernicus Data Space Ecosystem, you need to create a .s5cfg file containing your credentials.

Log to https://eodata-s3keysmanager.dataspace.copernicus.eu/panel/s3-credentials, create your keys them follow these steps:

  1. Create a file named .s5cfg in the root directory of the project.
  2. Add the following content to the file, replacing aws_access_key_id and aws_secret_access_key with your actual credentials:
# filepath: the current working directory where phidown is launched
[default]
aws_access_key_id = your_access_key
aws_secret_access_key = access_key_secret
aws_region = eu-central-1
host_base = eodata.dataspace.copernicus.eu
host_bucket = eodata.dataspace.copernicus.eu
use_https = true
check_ssl_certificate = true
  1. Save the file.

💡 Alternative: Pass Credentials w CLI

Instead of using a .s5cfg file, you can pass your S3 credentials directly when running the script. Use the following command:

pdm run python phidown/downloader.py --username <your_username> --password <your_password> --eo_product_name <eo_product_name>

Replace your_username and your_password with your actual S3 credentials.
Replace eo_product_name with your actual product name you want to download.

The script will:

  1. Authenticate with the Copernicus Data Space Ecosystem using S3 credentials.
  2. Search for Sentinel products within the specified AOI and date range.
  3. Download the first matching product using S3.

Example Notebook

For a detailed example of how to use Φ-Down, refer to the how_to_start.ipynb notebook or Open in Colab. It provides step-by-step instructions and practical use cases to help you get started quickly.

⚠️ Search Optimization Tips⚠️

Crucial for the search performance is specifying the collection name. Example:
Collection/Name eq 'SENTINEL-3'.

An additional efficient way to accelerate the query performance is limiting the query by acquisition dates, e.g.:
ContentDate/Start gt 2022-05-03T00:00:00 and ContentDate/Start lt 2022-05-21T00:00:00.

When searching for products and adding a wide range of dates to the query, e.g., from 2017 to 2023, we recommend splitting the query into individual years, e.g., from January 1, 2023, to December 31, 2023.

Notes

  • Credentials: S3 credentials are required and must be configured in the .s5cfg file. Do not share this file publicly.
  • Virtual Environment: PDM manages a dedicated virtual environment for the project.
  • Faster Setup: Using pdm.lock improves reproducibility and setup speed.

Troubleshooting

  • Ensure you're using Python 3.9 or higher.
  • Reinstall dependencies using pdm install.
  • Check logs and error messages for further insights.
  • If credentials are not deleted from the S3 server, you may encounter a 403 error. To resolve this, log in to the Copernicus Data Space Ecosystem, navigate to the S3 Credentials Manager, and manually delete any unused or expired credentials.

Contributing

We welcome contributions to Φ-Down! Here's how you can get involved:

  1. Fork the Repository: Click the "Fork" button at the top of this repository to create your own copy.
  2. Clone Your Fork: Clone your forked repository to your local machine:
git clone https://github.com/ESA-PhiLab/phidown.git
  1. Create a Branch: Create a new branch for your feature or bug fix:
git checkout -b feature/your-feature-name
  1. Make Changes: Implement your changes and commit them with a descriptive message:
git commit -m "Add feature: your-feature-name"
  1. Push Changes: Push your branch to your forked repository:
git push origin feature/your-feature-name
  1. Submit a Pull Request: Open a pull request to the main repository and describe your changes.

Guidelines

  • Follow the existing code style and structure.
  • Write clear and concise commit messages.
  • Ensure your code passes all tests and linting checks.
  • Update documentation if your changes affect usage.

Community Support

If you encounter any issues or have questions, feel free to open an issue in the repository or join the discussion in the Discussions section.

We appreciate your feedback and contributions to make Φ-Down even better!

Connect with the Author

Have questions, suggestions, or just want to connect? Reach out to the author:

Dr. Roberto Del Prete

LinkedIn Email Google Scholar

Citing Φ-Down

If Φ-Down has contributed to your research, project, or publication, we kindly ask that you cite it. Your citation helps support the tool's development and recognition within the scientific community! 🙏

Please use the following citation:

Del Prete, R. (2025). phidown: A Python Tool for Streamlined Data Downloading from CDSE. Zenodo. doi: 10.5281/zenodo.15332053.

BibTeX Entry:

@misc{delprete2025phidown,
  author       = {Del Prete, Roberto},
  title        = {phidown: A Python Tool for Streamlined Data Downloading from CDSE},
  year         = {2025},
  month        = {May},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.15332053},
  url          = {https://doi.org/10.5281/zenodo.15332053}
}

License

Φ-Down is proudly licensed under the GNU Lesser General Public License version v3.0 (LGPLv3).

License: LGPL v3

This license ensures that the software remains free and open-source, allowing you to:

  • Use the software freely for any purpose.
  • Share the software with others.
  • Modify the software and share your modifications under the same license terms.

For full details, please refer to the LGPLv3 License Text.

About

Manage, search, and download Earth Observation data with Φ-down from Copernicus missions with ease and efficiency.

esa-philab.github.io/phidown/

Topics

aws data-science python3 sentinel remote-sensing esa earth-observation copernicus-products european-space-agency

Resources

Readme

License

LGPL-3.0 license
Activity
Custom properties

Stars

30 stars

Watchers

1 watching

Forks

1 fork
Report repository

Releases

2 tags

Languages