Retrieval + generation eval: parse, chunk and embed dataset

[jacopo-chevallard]

Given an ingestion workflow/configuration, we should parse, chunk and embed the reference dataset and store the results into a database for successive retrieval.

Open questions:

• create a temporary (supabase) instance and use it for evaluation purposes, so when the evaluation is finished, the instance can be deleted? Or shall we persist the results for successive inspection, for instance to understand why a certain evaluation run provided bad results?

[linear]

CORE-348 Retrieval + generation eval: parse, chunk and embed dataset