Is this generally a dependency of non-pandas dataframe libraries now? Or could this change introduce a regression for e.g. polars users who are currently leveraging the dataframe interchange protocol?

Thanks for your review!

Polars doesn't depend on PyArrow, but polars.DataFrame.to_pandas always requires PyArrow. So, in practice, anyone working with both dataframe libraries may well already have PyArrow already installed

To avoid requiring PyArrow for the cases when it's not necessary, one way could be to do something like:

• try using the interchange protocol
• if it raises, then fall back to the PyCapsule Interface (which currently requires PyArrow)

This has the upside of not requiring PyArrow in some cases, but the downside of hiding issues where the interchange protocol silently produces invalid results

It may be possible to do this PyCapsule Interface conversion in the future without PyArrow but with something lighter instead, like arro3 by @kylebarron (who I'm ccing in case he has comments too)

What would be your preference?

Some polars users may not have pyarrow installed. If seaborn needs to get pandas data, the only production-ready way to do Arrow -> pandas that I know of is using pyarrow.

As Marco mentions I'm working on arro3, which is a minimal library for Arrow in Python, but Pandas interop is not a primary concern, and it's not production-ready today.

FWIW pandas 3.x is going to strongly incentivize users to install PyArrow, although it stops short of outright requiring it. In theory, the only people that shouldn't have PyArrow installed are those that operate in space/resource constrained environments, probably in headless environments like AWS Lambda where seaborn won't be used

Of course up to you how much you want to support non-PyArrow configurations, but the dataframe interchange protocol is relatively buggy and gets very little support, so you may find it easier altogether to force users towards PyArrow

cuDF have said that they will deprecate the interchange format: rapidsai/cudf#17282

Plotly have stopped using it, so Seaborn is the only library left using it

At this point, I think there's a greater risk in keeping it - I don't want to force anything here of course, just making sure you're aware

Could you please clarify what you mean by "if the infra isn't there yet"?

The Arrow C Interface already has quite widespread adoption and I'm not aware of edge cases in its implementations. @WillAyd wrote about switching his Pantab project over to it in Leveraging the Arrow C Data Interface, and noted

Almost immediately my issues went away [...] I felt more confident in the implementation and had to deal with less memory corruption / crashes than before. And, perhaps most importantly, I saved a lot of time.

That was nearly a year ago, and given that he's now suggesting it here in Plotly, I'd say that his experience has stayed just as positive

Regarding PyArrow dependency, I'll also note that polars.DataFrame.to_pandas also requires PyArrow, so any Polars user (such as myself) would already have needed PyArrow installed if they were converting to pandas via the Polars official method

Basically my threshold is "do I need to think about it at all". I'm just not interested in the minutia of competing Python dataframe libraries or the various attempts to make them work better together. The previous approach was sold as a simple protocol that always works, but it turns out that wasn't the case. Maybe this new way is better, the problem is I have no real way to say for sure without spending a lot of time learning about something that doesn't interest me.

Shall I close and leave you to remove cross-dataframe compatibility altogether?

The problem is then I get issues bugging me about Polars, so I have to think about it anyway :D

😄 that's understandable

I'm aware that you said that using Narwhals was a complete non-starter, but just to showcase that as a possibility:

import narwhals.stable.v1 as nw
from narwhals.stable.v1.typing import IntoDataFrame
import polars as pl
import pandas as pd

def convert_dataframe_to_pandas(data: IntoDataFrame) -> pd.DataFrame:
    return nw.from_native(data).to_pandas()

and then leave it up to Narwhals to convert to pandas in the best way for each input library

Altair, Plotly, and Vegafusion are using it as required dependency now, and Bokeh have a PR in progress to do the same

For completeness: the way the other libraries are using Narwhals is by making the whole logic dataframe-agnostic. In Plotly this resulted in 2-3x better performance for many plots involving group-bys (compared with converting all inputs to pandas), but I understand that you may not be interested in that

Nice, TIL