A tool for wrangling IRS data on 527s
Poindexter does the following:
- Downloads and extracts the IRS' bulk Political Organization Filing and Disclosure data file
- Cleans this file of database errors, errant DOS and UNIX line endings (there are both), and other cruft
- Repairs lines broken by unsupported characters in the IRS' database dump
- Logs all the weirdness it encounters and repairs
- Writes the results into a series of CSVs, one for each table described in the IRS data documentation here
Poindexter comes complete with the sql statements necessary to make the corresponding tables in a Postgres database.
To download the bulk data:
./prep_files.sh
To generate the flatfiles into a directory called 'csvs' -- which should exist in the working directory -- using default settings:
./run_this.py & tail -f filemaker.log
From there, you're on your own; SQL scripts are included in sql/ that will create tables
in Postgresql one could populate from the flatfiles with a COPY FROM command.
Poindexter should log an error when it encounters a row it can't handle.