Uploaded flatfile and max size

[rizac]

Currently I am struggling little bit to implement file upload.

I am talking here about the API, i.e. users coding in their notebook (form upload is another story I can deal with).

To upload a file with a Python request, I thought we just needed some adjustments to the already provided code snippets. It turns out it's not the case (a starting discussion here: https://stackoverflow.com/a/27050525. Links here are just for ma as reminder). We cannot expect users to code 30-40 lines of code for each file upload. Possible workarounds are:

1. Document that users should install the requests module. installation overhead, but apparently then it's 2-3 lines of code: https://stackoverflow.com/a/22567429

2. Cleanup as much as possible existing examples (as said, 30-40 lines of code. See links below) and create a code snippet to be copied/pasted by users. No installation overhead but honestly the least attractive option: it's harder to maintain for us and scary/impractical for notebook users
   - https://gist.github.com/rhoit/9573c40feaeb3cf44b4a8544dc0ae2a1
   - https://stackoverflow.com/a/681182
   - https://gist.github.com/rhoit/9573c40feaeb3cf44b4a8544dc0ae2a1 (simplified version of
   - https://www.matteomattei.com/http-https-get-and-post-requests-with-python-including-file-upload/

3. create an egsim_client.py module that wraps the code in 2 and exposes simple functions as in 1. The module needs only yo be downloaded in the user project or folder and not necessarily installed. Still harder to maintain for us

@g-weatherill what is your opinion here? Also, I am googling a little bit it on uploaded file size. we can think of a maximum size of 1.5-2Mb. Still very heuristic and not tested, but let's stick to these numbers: it would be between 1/10 and 1/5 of the ESM flatfile i.e. roughly between 2000 and 4000 records. Is that covering most use cases, nobody will ever use so "few" rows and we are in troubles? what else?

Thanks

[g-weatherill]

@rizac Personally, I prefer option 1. I think it is reasonable to assume that the users who have the level of knowledge to implement the API workflow for this can cope with the installation of the requests module, and 2 - 3 lines of code is sufficient to show that a feature is reasonably clean and well supported. Otherwise the client option could would but I think that could end up being more complex for the user than just running the process to install requests (which they only have to do once) and then running the simple code snippet as needed.

[g-weatherill]

Regarding the uploaded file size, setting a maximum of 2000 - 4000 records seems a little limiting. Would a maximum size of say 10 - 15 Mb be possible, in order to push that up to maybe 10,000 to 15,000 records or so. I'm hoping the typical use case would be on the order of only a few thousand, but I think it would be good if a larger file could be supported - even if the process takes longer (which perhaps you could warn the user about)