Investigate the possibility of removing dataframe field from Document

[anakin87]

I've been thinking about dropping the Dataframe field in Haystack Document dataclass for a few reasons:

• Users are already using text representations (CSV, Markdown) that LLMs handle great - even for originally tabular data
• Pandas DataFrame creates serialization headaches. (e.g. in Hayhooks)
• Pandas is a heavy dependency that complicates things, especially in serverless environments like Lambda. We could make it optional.
• Supporting dataframes across different Document Stores requires complex workarounds.

I will reach out to internal and external users to validate my assumptions.
We should also investigate how impactful this change would be.

[anakin87]

@EdAbati, the author of dataframes-haystack confirmed that this idea makes sense to him.
@sjrl too.

[anakin87]

Possible plan

• Gather more feedback on the idea from the community with a GitHub discussion/Discord announcement (📢 Potential removal of `dataframe` field in Haystack `Document` - We need your feedback! #8688)
• Deprecate dataframe field and ExtractedTableAnswer (in 2.10 release?)
• Remove dataframe support from Document Stores
  • Remove dataframe support from InMemoryDocumentStore (after 2.10)
  • Remove dataframe support from Document Stores in core-integrations ¹
• refactor AzureOCRDocumentConverter to remove internal usage of dataframe (this can be done at any time).
• Remove dataframe field and ExtractedTableAnswer (in 2.11 release?)
• Make pandas dependency optional (in 2.11 release?)

Footnotes

1. This may be best addressed after deprecating dataframe, but could also be done earlier. Support across integrations is highly varied and often inconsistent, making this the most significant task. ↩

[anakin87]

Investigation done.
I created another issue to keep track of the plan: #8738.