Best way to ingest documents over network?

[pags]

I have a bunch of documents in an S3 bucket that I'd like to get into my indexing pipeline. Naively I could download the documents to local disk and then run run_batch with file_paths, but that seems wasteful. Is there some way to stream data directly from network to Haystack without touching disk?

[JasperLS]

Hi Pags,
Assuming you are running Haystack in a docker wiht API, you might run a script on AWS to upload files to haystack via REST API.

Generally, this seems like a feature many users might be interested in. Would you ming opening a Proposal documenting your use-case. Here is the process for creating a new proposal: https://github.com/deepset-ai/haystack/tree/main/proposals?