Suspected memory leak when processing large datasets with append=false

[nordam]

Describe the bug

Processing a large dataset (300000 + images), and after about 16 hours, the memory use is now up to around 25 GB. Running with append=false in the toml file, and --num_chuncks 4 on the command line, pyopia version 2.5.5

[arsalanmstn]

Can we try "gc.collect()" from garbage collection module at the end of the "def process_file_list(file_list, c)"??? So, it would release the unused memory (I used it for streaming mode of Vimba_python). The code can be modified as:

import gc
def process_file_list(file_list, c):
for ...:
try:
....
gc.collect()

[nepstad]

Can we try "gc.collect()" from garbage collection module at the end of the "def process_file_list(file_list, c)"??? So, it would release the unused memory (I used it for streaming mode of Vimba_python). The code can be modified as:

import gc def process_file_list(file_list, c): for ...: try: .... gc.collect()

Could you try this out and report back if you saw a decrease in memory use for long running processes? If so, please open a pull request with your changes.