Added LookUp error handling during encoding detection.#412

Open

ggcr wants to merge 17 commits intoNVIDIA:mainfrom ggcr:main

+55-13

Commits on Dec 6, 2024

Added LookUp error handling during encoding detection.
ggcr
committed

Commits on Dec 20, 2024

Removed the unstructured dependencies, we will use local-inference mode installed via make
ggcr
committed
Append dependencies again, we indeed need them...
ggcr
committed

Commits on Dec 28, 2024

Pandas .to_json is a possible bottleneck that was causing the OOM issues and a memory leak that made the program end. Fixed by using to_dict and then writting each record instead of the big dum…
ggcr
committed
Import json
ggcr
committed
Remove checking for filename, it is implictly done anyways
ggcr
committed

Commits on Dec 30, 2024

Included datasets dependency for HF datasets integration
ggcr
committed