Merge pull request #47 from titipata/fix-fetch-mins

Function to fetch event from MINS, update data
titipata · Sep 20, 2019 · d1c9a79 · d1c9a79
2 parents 1699f2c + 5163b96
commit d1c9a79
Show file tree

Hide file tree

Showing 10 changed files with 1,164 additions and 2,496 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,5 +1,9 @@
 # Created by https://www.gitignore.io/api/macos,reactnative
 
+# loaded data and notebooks
+.vscode/
+grobid-*/
+
 ### macOS ###
 *.DS_Store
 .AppleDouble

diff --git a/backend/README.md b/backend/README.md
@@ -1,9 +1,18 @@
 # Flask back-end for Penn event
 
+## Fetch events
+
+`fetch_events.py` contains functions to fetch Penn events. We can run GROBID and fetch to update data in `data/events.json` file 
+as follows
+
+```sh
+bash serve_grobid.sh
+python fetch_events.py
+```
 
 ## Running CRON to fetch event daily
 
-We use CRON to constantly fetch Penn events. Change `username` in `cron_fetch_events.py`
+We use CRON to make `fetch_events.py` constantly fetch Penn events. Change `username` in `cron_fetch_events.py`
 then run cron job by using the following script
 
 ```sh

diff --git a/backend/data/events.csv b/backend/data/events.csv
diff --git a/backend/data/events.json b/backend/data/events.json
diff --git a/backend/data/events_dt.csv b/backend/data/events_dt.csv
diff --git a/backend/data/events_vector.json b/backend/data/events_vector.json
diff --git a/backend/events.json b/backend/events.json
diff --git a/backend/fetch_events.py b/backend/fetch_events.py
diff --git a/backend/requirements.txt b/backend/requirements.txt
@@ -9,3 +9,4 @@ lxml
 spacy
 requests
 uwsgi
+nltk
diff --git a/backend/serve_grobid.sh b/backend/serve_grobid.sh
@@ -0,0 +1,10 @@
+# download GROBID if directory does not exist
+if [ ! -d grobid-0.5.5 ]; then
+  wget https://github.com/kermitt2/grobid/archive/0.5.5.zip
+  unzip 0.5.5.zip
+  rm 0.5.5.zip
+fi
+
+# run GROBID
+cd grobid-0.5.5
+./gradlew run
-Original file line number
+Diff line change
@@ Expand Up / @@ -9,3 +9,4 @@ lxml @@
     spacy
     requests
     uwsgi
+    nltk