Major refactoring and customization (#15)

* Added skip logic if a variable is not found. Reads from DRC training dataset

* Added logging

* Major refactoring

* Renamed HFC.py into main.py

* Updated gitignore

* Changed ID04 into ID04LABEL

* Added documentation to classes

* Revert "Changed ID04 into ID04LABEL"

This reverts commit 02b2807.

* Clean Unused Files

* Fixed indentation error

* Refactored MasterSheet

* LCS

* added lcs_en

* Refactored main.py and added main config file to specify indicators

* Testing All Indicators

* Switched Off LCS-EN

* Changed Non-Existant Module Log from Error to Warning

* Added Warnings count

* removed report generator from root directoy

* Urban/Rural Context

* DataFrameCustomizer

* Added Timing Checks

* Adusted time_thresholds

* timing variable names

* Date Parsing

* No Error raised if reports folder does not exist

* Added variables for DRC

* Refactored logging configuration and updated ROADMAP

* Refactored main

* Re-added loading of mastersheet report to DataBridges

* Refactored main

---------

Co-authored-by: Ali Assi <[email protected]>

.gitignore

@@ -173,16 +173,16 @@ EGCO_Sudanese_24_HFC.do
 HFC_FCS
 *.ipynb
 
+
 # data
 *.xlsx
 *.csv
 
 # other files
-ROADMAP.md
 NOTES.md
-
+*.log
+data_processing_log.txt
 
 # config files 
 data_bridges_api_config.yaml
-.env
-*.log
+data_processing_log.txt

ROADMAP.md

@@ -1,55 +1,43 @@
 # High Frenquency Checks for WFP Food Security Assessments - ROADMAP
 
-## v0.1.0 (Enumerator Training)
+## v0.1.0 Stable
 
-### Indicator checks
+
+### Checks
 - [X] FCS
 - [X] rCSI
 - [X] HDDS
 - [X] Demographics
 - [X] Food Expenditures
 - [X] Non-food Expenditures
 - [X] LCS
-
-### Data connector
-- [X] Get data from DataBridges
-- [ ] Parse data as int 
-
-### Visualization
-- [ ] Output master report in DB
-- [ ] Output master report in Tableau
-
-## v.0.2.0
-### Indicator checks
 - [ ] HHS
-
-### Enumerator performance
+- [ ] Average daily completion by enumerator
+-	Flag invalid survey with interview less than 5 minutes
+-	Flag short survey (<45 minutes)
+-	Flag long survey (>150 minutes)
 - [ ] Total survey completed
 - [ ] Total survey completed by admin region (see sample size)
-- [ ] Average daily completion by enumerator
-- [ ] Survey duration too short
-- [ ] Survey duration too long
-- [ ] Flag invalid survey with interview less than 5 minutes
-- [ ] Flag short survey 
-- [ ] Flag long survey 
+- [ ] Enumerator completion (overall number by day and scatter hours of the day if real-time submission is possible)
 
 ### Logbook 
-- [ ] Create issue log in Excel
+- [ ] Create issue log with format ```uuid | issue | action | old.value | new.value```
+
+### Data connector
+- [X] Get data from DataBridges
 
 ### Deployment
 - [ ] Run code on a schedule from HPC
+- [ ] Output report in Tableau
+
+## v.2.0
+- [ ] Configuration in settings
+- [ ] Revamp output report in Tableau
+- [ ] 
 
-## Future releases (1.0.0)
-- [ ] Additional checks
-- [ ] Make config.py something usable with no knowledge of Python (CSV input)
-- [ ] Optimize output (e.g. Excel vs CSV vs other format)
-- [ ] Dashboard
-- [ ] Optimize code (e.g. not running everything on all the dataset, but just on the data that have been collected the day before)
-- [ ] Endpoint through Core Engine
 
+## Future releases
 
-## Issues
-- Running wrong CO 
-- Full vs Base
-- Where to export the report
-  - DB -> Tableau
+- [X] Make config.py something usable with no knowledge of Python (CSV input)
+- [ ] Make the output less ugly
+- [ ] Optimize code (e.g. not running everything on all the dataset, but just on the data that have been collected the day before)