mccelections is a Django application that can be used for any of the following election result-related tasks:
- ingesting AP election results using the Elex API wrapper
- ingesting results that have been hand-keyed or scraped into a Google Sheet (Google Sheets can parse HTML, XML and other structured data formats simply by setting up the proper formula)
- writing custom Python scrapers to ingest results data
- hand-keying results data directly into the Django admin
- automatically calculating vote percent and precinct reporting percents for hand-keyed, Google Sheet or scraped results
- recording and replaying results data, such as from AP tests
- automatically simulating manual data changes
- storing AP, scraped, manual or any combination of data in a unified format
- outputting this data as a RESTful API
- sending updates to Slack
Thanks to
- Digital Ocean's How To Serve Django Applications with uWSGI and Nginx on Ubuntu 14.04
- @panuta's How to setup Django server with virtualenv on Ubuntu Server 12.04; for reference, here's my fork of that
- @jeremyjbowers all of his advice along the way.
- Python 2.7.x
- OS X or Ubuntu 14
- Homebrew
- Associated Press API key (for ingestion)
- Amazon S3 bucket (for baking to static)
If you're using an S3 bucket, then you need to add these to your .bash_profile (Mac) or .bashrc (Ubuntu) file with their respective values:
S3_ACCESS_KEY=''
S3_SECRET_KEY=''
S3_BUCKET=''
Install pip
easy_install pip
Install virtualenv
pip install virtualenv virtualenvwrapper
Install postgresql
brew install postgresql
I recommend setting up both test and prod environments.
If using a pem key with AWS
ssh -i <filepath>/<filename>.pem <username>@<host.com>
If whitelisted via AWS, then
ssh <username>@<host.com>
Check for package updates
sudo apt-get update
Upgrade those packages
sudo apt-get upgrade
Install Python dependencies
sudo apt-get install python-pip python-dev python-setuptools
Install virtualenv and virtualenvwrapper
sudo pip install virtualenv virtualenvwrapper
Install postgres
sudo apt-get install postgresql python-psycopg2 libpq-dev
If you just need the postgres client (e.g. connecting to database via Amazon RDS), then try this (h/t Postgres docs)
sudo apt-get install postgresql-client
For Slackbot, if you want to avoid SSL warnings, via this SO thread:
sudo apt-get install libffi-dev libssl-dev
These are included in requirements.txt, so you can remove them if you're not using the Slackbot
pip install pyopenssl ndg-httpsclient pyasn1
Add the following to your .bash_profile for a Mac or .bashrc for Ubuntu:
AP API key
echo "export AP_API_KEY=<API_KEY_HERE>" >> ~/.bash_profile
Elex recording type
echo "export ELEX_RECORDING=flat" >> ~/.bash_profile
Elex recording directory
echo "export ELEX_RECORDING_DIR=/tmp/ap-elex-data/" >> ~/.bash_profile
Feel free to change this path based on your particular preference or server set up. Just be sure all other paths below (search for /home/ubuntu/) reflect where your project lives.
echo "export SAVER_DIR=/home/ubuntu/mccelections/electionsproject/snapshots" >> ~/.bash_profile
Create an Elex recording directory, for example
mkdir -p /tmp/ap-elex-data/
Create the environments directory, if you don't have one already. You can name this anything you like, as long as it's consistent with future related steps.
mkdir -p ~/Envs
Set the workon home, which is a shortcut with virtualenvwrapper (.bash_profile for Mac, .bashrc for Ubuntu)
echo "export WORKON_HOME=~/Envs" >> ~/.bash_profile
Source the virtualenvwrapper file
echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bash_profile
Source your bash profile for the changes to take effect in that terminal tab
source ~/.bash_profile
On Ubuntu
sudo apt-get install git
On OS X, follow these instructions
https://git-scm.com/book/en/v2/Getting-Started-Installing-Git#_installing_on_mac
Execute this command in your terminal
ssh-keygen -t rsa -b 4096 -C "<YOUR_EMAIL_ADDRESS>"
Enter this as the file, for example
/home/ubuntu/.ssh/id_rsa
Enter a secret passphrase
On Mac, copy it to clipboard with
pbcopy < ~/.ssh/id_rsa.pub
On other platforms like Ubuntu, you can use a text editor like vim to read and copy the file contents
vim ~/.ssh/id_rsa.pub
Or run this to output it to the terminal
cat ~/.ssh/id_rsa.pub
Then highlight, copy and paste it as a new SSH key in GitHub
Make sure you're in the correct directory, such as root
cd ~
Clone the repo
git clone [email protected]:mcclatchy/mccelections.git
Change into that directory
cd ~/mccelections
Using a handy command that comes with virtualenvwrapper
mkvirtualenv mccelections
This will install all the Python packages used by mccelections. If you don't plan to use certain features (e.g. Slack integration), feel free to remove them before installing.
pip install -r requirements.txt
Create a private settings file for each environment. NOTE: These files are excluded from version control in the .gitignore file, but it's important to confirm before comittting.
touch ~/mccelections/electionsproject/electionsproject/settings_private.py
Add the following settings to each environment's file with the correct values in place of the <VALUE> listed for each variable:
SECRET_KEY = "<SECRET_KEY>"
## Slack token to send messages to related account
SLACK_TOKEN = "<SLACK_TOKEN>"
## TEST
db_engine = "django.db.backends.postgresql_psycopg2"
db_name = "<NAME>"
db_user = "<USER>"
db_password = "<PASSWORD>"
db_host = "<HOST>"
db_port = "<PORT>"
In settings_test.py and settings_prod.py, you must add the related domains for the API base URL
MCC_API_BASE_URL
And Django allowed hosts
ALLOWED_HOSTS
If you're using a server, copy the server-specific settings to main settings file
cp electionsproject/settings_test.py electionsproject/settings.py
or
cp electionsproject/settings_prod.py electionsproject/settings.py
If you're using localhost, make sure the database settings are correct. If you haven't already created a postgres database and user, you need to do that. For example, on a Mac
https://updatemycode.com/2016/11/23/installing-postgresql-on-mac-os-x/
Check the settings file and verify everything is correct
Make sure you're in the app directory before running any management command, such as the initial database migration
cd ~/mccelections/electionsproject/results
Then migrate the database
./manage.py migrate
Now create a Django superuser
./manage.py createsuperuser
Collect static files
./manage.py collectstatic
Enter 'yes' to overwrite
Only necessary if you plan to use the bash scripts
chmod u+x bashscripts/*
If you're on localhost, start the Django development server
./manage.py runserver
Make sure you see this
Django version 1.9, using settings 'elections.settings'
Starting development server at http://127.0.0.1:8000/
If you're on an Ubuntu server, you'll need to install and configure the server
sudo pip install uwsgi
sudo mkdir -p /etc/uwsgi/sites
cd /etc/uwsgi/sites
sudo vim electionsproject.ini
Update your virtualenv name below for <YOUR_ENV> and then paste this block
[uwsgi]
project = electionsproject
base = /home/ubuntu/mccelections/electionsproject
chdir = %(base)/%(project)
home = /home/ubuntu/Envs/<YOUR ENV>
module = %(project).wsgi:application
master = true
processes = 5
socket = /tmp/%(project).sock
chmod-socket = 664
uid = www-data
gid = www-data
vacuum = true
harakiri = 120
max-requests = 50
die-on-term
catch-exceptions
Save and close
Edit the uwsgi configuration
sudo vim /etc/init/uwsgi.conf
Add the following
description "uWSGI application server in Emperor mode"
start on runlevel [2345]
stop on runlevel [!2345]
setuid ubuntu
setgid www-data
exec /usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
Save and close
Install nginx
sudo apt-get install nginx
Edit the configuration
sudo vim /etc/nginx/sites-available/electionsproject
Paste the following and be sure to replace <hostname> with your host. If you want all domains to be able to access the API data (a.k.a. CORS, cross-orgin requests), use '*' there -- as it is currently. If you want to specify a domain, add it there instead. You can also update the cache expiration time as desired.
server {
listen 80;
server_name <hostname>;
access_log /var/log/nginx/electionsproject_access.log;
error_log /var/log/nginx/electionsproject_error.log;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/ubuntu/mccelections/electionsproject;
}
location / {
add_header 'Access-Control-Allow-Origin' '*';
add_header Cache-Control "public";
expires 1h;
include uwsgi_params;
uwsgi_pass unix:/tmp/electionsproject.sock;
}
location /admin {
include uwsgi_params;
uwsgi_pass unix:/tmp/electionsproject.sock;
}
location /api/v1/resultlive {
add_header 'Access-Control-Allow-Origin' '*';
add_header Cache-Control "public";
expires 1m;
include uwsgi_params;
uwsgi_pass unix:/tmp/electionsproject.sock;
}
location /v1/resultlive {
add_header 'Access-Control-Allow-Origin' '*';
add_header Cache-Control "public";
expires 1m;
include uwsgi_params;
uwsgi_pass unix:/tmp/electionsproject.sock;
}
location /api/v1/resultmanual {
add_header 'Access-Control-Allow-Origin' '*';
add_header Cache-Control "public";
expires 1m;
include uwsgi_params;
uwsgi_pass unix:/tmp/electionsproject.sock;
}
location /v1/resultmanual {
add_header 'Access-Control-Allow-Origin' '*';
add_header Cache-Control "public";
expires 1m;
include uwsgi_params;
uwsgi_pass unix:/tmp/electionsproject.sock;
}
}
Create a symlink for the nginx configuration
sudo ln -s /etc/nginx/sites-available/electionsproject /etc/nginx/sites-enabled
Make sure the symlink worked
ll /etc/nginx/sites-enabled/
Check the configuration (make sure it says "OK" and doesn't throw any errors)
sudo service nginx configtest
Restart nginx
sudo service nginx restart
Start uwsgi
sudo service uwsgi start
Or, if uwsgi was already running
sudo service uwsgi restart
If you've made a model or admin change, ran the migration and still aren't seeing the change reflected in the admin on a server, you can try runnning restarting uwsgi. If you still aren't seeing the change, try rebooting the server.
sudo reboot
Check the url
domain.com/admin
On a server, log in
ssh <username>@<host.com>
On a local machine, open your terminal application
Activate the virtualenv and change to that directory
workon mccelections && cd code/mccelections/electionsproject/
You can also set up an alias for this in your .bash_profile (Mac) or .bashrc (Ubunutu)
echo "alias mccelections='workon mccelections && cd code/mccelections/electionsproject/'" >> ~/.bash_profile
Then to have it take effect
source ~/.bash_profile
Test by running this
mccelections
If you're not pushing any changes back to the repo, checkout any edited files to remove the updates
git checkout -- .
Grab the latest files off of GitHub, such as from the master branch. If you're on the test server and want to pull a test branch, specific that branch instead.
git pull origin master
Enter your secret passphrase
Make migrations based on the new changes (because migrations are listed in gitignore)
./manage.py makemigrations
Migrate the database
./manage.py migrate
You should be all set to start add or ingesting data! To test the main URLs:
/admin- add/update/delete data/api/v1/<model>/?format=json- API url pattern
For more information on how to use the app as a developer, visit the developer instructions.
Questions? Suggestions? Submit a pull request or contact: greglinch [at] gmail [dot] com