[Feature]: Implement POST /v1/files

[cbh778899]

Contact Details(optional)

No response

What feature are you requesting?

openai reference: create files

[NguyenNguyen205]

Hi, can I ask more about the details of this task ?

• What file type is accepted for uploading to the endpoint ?
• What is the file size limit ?
• Where the file will be store after uploading, is it in the volumes folder, or in an entirely new folder ?
• The json format of request and response will be exactly the same as the OpenAI reference am I correct ?
• How the files will be used ? Can it be reference in the rag-completions endpoint like a dataset ?

I'm still quite new to open source contributions, so I'm still not sure about a lot of things

[cbh778899]

We don't got too many requests on this functionalities, you can actually do what you want. But for your direction, here are some answers:

1. Since we haven't got multimodal models, currently the only acceptable file is the same as the url attribute for /v1/embedding/dataset route.
2. If you want to try multipart upload, no size limitation is needed, otherwise for your convenience just give it a small limitation like 10mb.
3. Just create a new folder for the file upload, you might want to bind that volume on local machine inside docker compose file.
4. That would be better, but we can add some customized setups but give full documentation about what is different and how to use that
5. There are 2 options from my side, the first one is to add extra attribute to /v1/embedding/dataset to use uploaded file as dataset input. The second one is to apply the file content as prompt and give it to the LLM, in that case add attribute to the rag inference route will help. Be careful if you choose the second option, CPU inference is not pretty good at long context, if the content is too long the response speed will be extremely slow.

[NguyenNguyen205]

Got it, thanks a lot for the direction

[NguyenNguyen205]

Can I ask a few more questions relating to this problem ?

• I'm currently testing with multipart upload, though this require extra dependency, so I install the mutler dependency to handle multipart upload, is this allow ?
• For the file metadata, can I store it in the lance DB, or I should put store all of them in a simple json file ?

I've uploaded the file successfully, but there are still some validation and processing needed for the uploaded file

[cbh778899]

1. Yes, you can do that as long as it works as expected. Make sure your document is clear enough (e.g. Content-Type header must be multipart/form-data)
2. Use LanceDB is a better idea, just follow the existed codes to create a table you need.

[NguyenNguyen205]

Got it, thank you

[NguyenNguyen205]

Hi, I just make the pull request for the POST file API, as well as another GET all file API for easier debugging. Can you help me to check it out.
Besides that, for the RAG use case, from my understanding, the current embedding/dataset requires the loaded dataset to follow a pre-defined structure, and the loaded dataset must also have embedding datas already. Since the uploaded file through the /v1/files can have a variety of json format, I'm not sure how to embed these into the define structure for RAG operation yet, so I haven't done it yet.
Currently I'm testing with the pokemon file. Here is the following link: https://github.com/fanzeyi/pokemon.json/blob/master/pokedex.json

[cbh778899]

Hi, thanks for your contribution! The dataset should follows the specific format currently, which will be stored into database currently. To letting it accept any json format, perhaps the code need to be altered to use only one general column and store string into that, and calculate the embeddings of that column. The problem is that the embedding engine is using NLP, so this kind of embedding might result in a lower quality, so this need to be investigate. Besides that, any data in the required format that has no embedding column will be caucluated automatically in current code, so no worries about that.