-
Notifications
You must be signed in to change notification settings - Fork 3
natgaertner/candidate_classifier
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Two functions exist in this repository: fetching searching for candidates using the Google Custom Search API, and classifying the results as candidate web presence or not. get_fb_links.py, get_twitter_links.py, and get_web_links.py are all very similar. They import a csv of candidate names with some associated information (state, electoral district, known url for the given web presence type, if any), make calls to the search API for a few candidate-related search terms (mainly candidate name + state) and clean and combine the results of the searches. If known urls are provided, the relation of the found urls to the known urls is given (Child, Equal, Parent, and None) The results are written out into another csv with each result associated with the unique id for each candidate. classify_fb_combined.py, classify_twitter_combined.py, and classify_web_combined.py take these results and attempt to use machine learning techniques for classifying text to label the found pages as candidate related or not. The set of candidates with known urls forms the training and test set. The classifiers used were from the scikit learn python library (http://scikit-learn.org/0.11/index.html). I based my usage of them on example code here: http://scikit-learn.org/0.11/auto_examples/document_classification_20newsgroups.html . I made two significant modifications to this code, other than obviously pulling data from my search text collection rather than their sample database: A custom feature identifier was used that could identify some custom multi-word features and some features that would be specific to each entry (for example if the candidate's name appeared in the text, if the candidate's office name appeared in the text.) The scikit learn default was simply using single word features. It has an n-gram analyzer as well, but I didn't immediately see where that could be used for multi-word features. The custom features were added to the single word default analyzer, so all single word features (minus stop words) were included as well. The addition of custom feature detection was probably the most significant improvement I achieved over just running the classifiers as-is. Second, I combined the results of the classifiers into a bayesian probability estimate for the class of the page. I was using as my input both the discrete classification returned by each classifier and the score returned for each class by each classifier. This almost certainly led to some inaccuracy in my conditional independence assumptions for my input variables, but it did give me a really nice (artificially) sharp logistic shaped function for making class choices. The Bayesian formula used was: P(class | classifications, score) = P(class)*prod(P(classification|class) for each classifier)*P(score|class)/sum(P(class)*prod(P(classification|class) for each classifier)*P(score|class) for each class)
About
Some code to loook at candidate websites and classify them
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published