Using Convolutional Neural Networks to Predict Survey Estimates and other Outcome Measures

Stage 0 - Settings and Data Preparation

Files:

push.sh - shell script to push local data to HPC (script args define dir on HPC and settings JSON to use)
settings_builder.py - class to manage settings for different scripts
settings/ - folder containing settings JSONs for jobs
scripts/ - folder containing one off scripts for preparing survey data and anything else not a core utility used in following stages

Notes:

Stage 1 - Training CNN

Files:

s1_jobscripts - jobscript
main.py - primary script called by jobscript
data_prep.py - class/functions to generate sample data for training CNN
create_grid.py - class to generate point grid
runscript.py - core class/functions for running PyTorch CNN
load_data.py - classes for reading Landsat imagery for CNN training data and extracting NTL imagery values for training data labels
load_survey_data.py - class to load survey data for use with CNN predictions
resnet.py - modified ResNet class (based on PyTorch ResNet class)
vgg.py - modified VGG class (based on PyTorch VGG class) - NOT FUNCTION

Notes:

Stage 2 - Training Secondary Models

Files:

s2_jobscript - jobscript
second_stage_model.py - primary script called by jobscript
model_prep.py - functions and classes for building second stage models
merge_outputs.py - script to merge second stage model metrics

Stage 3 - Generating Predictive Surface

Files:

build_surface_grid.py

S3A - Creating Point Grid

S3B - Generating CNN Features for Surface Grid

S3C - Generating Secondary Model Values for Surface Grid

S4 - Building Raster Surface

Files:

s4_main.py -

Notes:

x

Usage Notes

Stage 1 Training

Stage 1 Predict

Stage 2 Train and Predict

running with full set of cnn features (or more; ntl, etc.) will make stage 2 take significantly longer. If testing, start with PCA instead of full feature set.
when using MLP Classifier model, single variable models (e.g., only NTL) will not converge

Stage 3

Stage 4