Use arrow datasets for intermediates #221
Labels
efficiency
Making docetl operations run faster
good first engineering issue
Engineering-focused issue for newcomers
JSON is a bit bulky. It's easy to view, but for intermediates, especially when using the UI, it does not make sense to store and query the intermediates as JSON.
We should have an option in a docetl config to store the intermediates as arrow datasets. We will then have to query them in the UI accordingly.
The text was updated successfully, but these errors were encountered: