Collector for WFP Hungermap Datasets

This script connects to the WFP HungerMap API and extracts data creating a dataset for each country in HDX. It makes around 500 reads to WFP's API and 100 read/writes (API calls) to HDX in a half hour period. It is run every day.

Development

Environment

Development is currently done using Python 3.13. We recommend using a virtual environment such as venv:

    python -m venv venv
    source venv/bin/activate

In your virtual environment, install all packages for development by running:

    pip install -r requirements.txt

Installing and running

For the script to run, you will need to have a file called .hdx_configuration.yaml in your home directory containing your HDX key, e.g.:

hdx_key: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
hdx_read_only: false
hdx_site: prod

You will also need to supply the universal .useragents.yaml file in your home directory as specified in the parameter user_agent_config_yaml passed to facade in run.py. The collector reads the key hdx-scraper-wfp-hungermap as specified in the parameter user_agent_lookup.

Alternatively, you can set up environment variables: USER_AGENT, HDX_KEY, HDX_SITE, EXTRA_PARAMS, TEMP_DIR, and LOG_FILE_ONLY.

To install and run, execute:

    pip install .
    python -m hdx.scraper.wfp-hungermap

Pre-commit

Be sure to install pre-commit, which is run every time you make a git commit:

    pip install pre-commit
    pre-commit install

With pre-commit, all code is formatted according to ruff guidelines.

To check if your changes pass pre-commit without committing, run:

    pre-commit run --all-files

Testing

Ensure you have the required packages to run the tests:

    pip install -r requirements-test.txt

To run the tests and view coverage, execute:

    pytest -c --cov hdx

Packages

uv is used for package management. If you’ve introduced a new package to the source code (i.e. anywhere in src/), please add it to the project.dependencies section of pyproject.toml with any known version constraints.

To add packages required only for testing, add them to the test section under [project.optional-dependencies].

Any changes to the dependencies will be automatically reflected in requirements.txt and requirements-test.txt with pre-commit, but you can re-generate the files without committing by executing:

    pre-commit run pip-compile --all-files

Project

Hatch is used for project management. The project can be built using:

    hatch build

Linting and syntax checking can be run with:

    hatch fmt --check

Tests can be executed using:

    hatch test