Public Ethics Data Downloader

This repository contains scripts used to download ethics data for the City of Oakland.

We currently support retrieving the 2018-2019 FPPC Form 700 Statement of Economic Interests. The data is downloaded from Netfile and uploaded to a data warehouse.

Usage

The logic to download and parse filings is normally deployed to Google Cloud as cloud functions.

The logic can also be run locally as scripts in the scripts directory. download_form_700_data.py will download all filings to scripts/filings. parse_local_data.py will extract data from the downloaded files to a SQLite database.

These scripts can be run with a command like the one below:

python -m scripts.download_form_700_data

Development

We use pipenv to manage environments and requirements, so install that first.

1. Install the requirements for this project:

   make requirements
   

2. Run code quality inspection:

   make quality
   

3. Run tests:

   make test
   

Deployment

This code is deployed to Google Cloud as functions responsible for (a) downloading filings and (b) performing ETL. The functions can all be deployed by running ./deploy.sh.