udf memory profiling

[jdries]

Current setting of memory overhead is 12GB, which is a bit much for operational production.

Looking up memory consumption shows very clear peaks when presto is invoked. The actual max usage is even higher, because prometheus only records memory at specific instants.

[GriffinBabe]

@jdries I'm creating a branch for this issue and provide you with a script to locally test presto

[kvantricht]

meanwhile already also made a PR to expose batch_size parameter for presto inference #59

[GriffinBabe]

@jdries I pushed to this branch a script to test presto locally. To make it work I had to go through quite some dependency issues.

1. Create a new conda environment with python3.10
2. Clone GFMAP and go to the main branch, run pip3 install --editable .
3. Clone worldcereal-classification, go to this branch, run pip3 install --editable .
4. Install the latest presto wheel file with it's dependencies: pip3 install https://artifactory.vgt.vito.be/artifactory/auxdata-public/worldcereal/dependencies/presto_worldcereal-0.1.1-py3-none-any.whl

You should be good to go