This repository has been archived by the owner on Mar 19, 2024. It is now read-only.
Releases: CDLUC3/counter-processor
Releases · CDLUC3/counter-processor
Fixing missing import statement
Speed optimizations, some code cleanup, Python 3.7.x
- Python 3.7+ is now required because some Sqlite3 library changes are only available in that version
of python or later. These are changes that allow backing up in-memory to on-disk databases and the reverse. - This adds some optimizations such as adding an index on country and reducing some unique value lengths
(used for deduplicating similar requests) by subsitituing a hash for a longer string. - Now loads database from disk into memory on startup and saves to disk again after new items are added.
- This speeds up data insertion and stat generation speed at the expense of using some memory.
- If the sqlite database gets too large then storing in memory on normal hardware may not be feasible.
However, at that scale a number of kinds of re-architecting may be required, even if not storing in
memory.
- Refactoring of
config.py
to make it look more like a standard Python object, though it is a singleton. - Rudimentary tests added.
- Examples:
python3 -m unittest test/test_main.py
andpython3 -m unittest test/config/test_config.py
- Examples:
Small validation changes of input files, output report-datasets
- Added basic validation that log line is correct length/format
- changing report_datasets to report-datasets (dash instead of underscore)
Updating to output size data to resulting report
-
The resulting report now contain size data in the output.
-
Updated library requirements
-
Gitignored an editor file
An initial working version of the counter-processor
This works and will submit reports for MDC/Counter from log files. It doesn't currently output size information.
I'd like to (non-manual) testing and validation. Tests were started but I didn't get very far with them yet.
I have manually validated some small test logs to be sure the output appears correct.
We have been running this script on a nightly basis and it seems to be quite stable for us.