Skip to content
This repository has been archived by the owner on Mar 19, 2024. It is now read-only.

Releases: CDLUC3/counter-processor

Fixing missing import statement

23 Nov 19:14
Compare
Choose a tag to compare

import statement was missing from the auto uploading functionality which is not always used.

Speed optimizations, some code cleanup, Python 3.7.x

17 Nov 22:27
32d540b
Compare
Choose a tag to compare
  • Python 3.7+ is now required because some Sqlite3 library changes are only available in that version
    of python or later. These are changes that allow backing up in-memory to on-disk databases and the reverse.
  • This adds some optimizations such as adding an index on country and reducing some unique value lengths
    (used for deduplicating similar requests) by subsitituing a hash for a longer string.
  • Now loads database from disk into memory on startup and saves to disk again after new items are added.
    • This speeds up data insertion and stat generation speed at the expense of using some memory.
    • If the sqlite database gets too large then storing in memory on normal hardware may not be feasible.
      However, at that scale a number of kinds of re-architecting may be required, even if not storing in
      memory.
  • Refactoring of config.py to make it look more like a standard Python object, though it is a singleton.
  • Rudimentary tests added.
    • Examples: python3 -m unittest test/test_main.py and python3 -m unittest test/config/test_config.py

Small validation changes of input files, output report-datasets

01 Nov 17:03
Compare
Choose a tag to compare
  • Added basic validation that log line is correct length/format
  • changing report_datasets to report-datasets (dash instead of underscore)

Updating to output size data to resulting report

30 Jan 20:30
f6c6eb3
Compare
Choose a tag to compare
  • The resulting report now contain size data in the output.

  • Updated library requirements

  • Gitignored an editor file

An initial working version of the counter-processor

30 Jan 20:34
Compare
Choose a tag to compare

This works and will submit reports for MDC/Counter from log files. It doesn't currently output size information.

I'd like to (non-manual) testing and validation. Tests were started but I didn't get very far with them yet.

I have manually validated some small test logs to be sure the output appears correct.

We have been running this script on a nightly basis and it seems to be quite stable for us.