Skip to content

Latest commit

 

History

History

kerala

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
joblist_2011
joblist_2011
 
 
joblist_2012
joblist_2012
 
 
joblist_2013
joblist_2013
 
 
joblist_2014
joblist_2014
 
 
joblist_2015
joblist_2015
 
 
joblist_2016
joblist_2016
 
 
kerala-out-2011
kerala-out-2011
 
 
kerala-out-2012
kerala-out-2012
 
 
kerala-out-2013
kerala-out-2013
 
 
kerala-out-2014
kerala-out-2014
 
 
kerala-out-2015
kerala-out-2015
 
 
kerala-out-2016
kerala-out-2016
 
 
notebooks
notebooks
 
 
README.md
README.md
 
 
kerala_2011.py
kerala_2011.py
 
 
kerala_2011.qsub
kerala_2011.qsub
 
 
kerala_2012.py
kerala_2012.py
 
 
kerala_2012.qsub
kerala_2012.qsub
 
 
kerala_2013.py
kerala_2013.py
 
 
kerala_2013.qsub
kerala_2013.qsub
 
 
kerala_2014.py
kerala_2014.py
 
 
kerala_2014.qsub
kerala_2014.qsub
 
 
kerala_2015.py
kerala_2015.py
 
 
kerala_2015.qsub
kerala_2015.qsub
 
 
kerala_2016.py
kerala_2016.py
 
 
kerala_2016.qsub
kerala_2016.qsub
 
 
kerala_mpi.py
kerala_mpi.py
 
 
main.py
main.py
 
 
parse_kerala_eng.ipynb
parse_kerala_eng.ipynb
 
 

README.md

How to run the script to process each year on EDU cluster server

To process the data concurrent on EDU cluster server we will split PDF files into 280 directories. Please note we have quota for 300 CPU cores and 1TB memory limit so we will submit only 280 jobs with 1 CPU core and 3.5G of RAM each.

  1. For each year follow the steps in the notebooks to prepare and setup the stuffs to process PDF files on cluster server

  2. Submit job to process on cluster server

qsub kerala_2016.qsub
  1. Check the status of running job
showq