from_pandas memory leak when reading df with python dicts

[Seminko]

Describe the bug, including details regarding any error messages, version, and platform.

I’m using snowflake-connector to import data to snowflake.
Snowflake-connector converts pandas DF into parquet using pyarrow and imports that parquet into SF.
I’m routinely importing 1M row dataframes with no issues in my async script. But one batch out of cca 50 runs into memory issues – pyarrow throws: ‚malloc of size n failed‘.

I was able to identify a subset of the data that gives me trouble, even specific column.
The column in question holds python dicts (parsed URLs).
There are no obvious bad types, non-string keys or missing expected keys. No outliers in length or structure, yet it still fails.
When I do table = pa.Table.from_pandas(df) the memory starts to ballon until it reaches almost 100% and then it fails. Or if it doesn’t fail it gobbles up at least 22GB of RAM which can’t be right.

I tried identifying offending rows by processing the df in batches with the idea of narrowing it down by reducing batch sizes. The strange thing is that when I split the df into batches of 10k rows, all of them go through without an issue, but when importing the whole dataframe (80k rows) it fails every time. Hence I don’t believe there’s something wrong with specific rows that would prevent the conversion to parquet.
Again, let me reiterate that I routinely import batches of 1M rows where the dict column format is identical.

I tested this with pyarrow v18.0 as well as the latest stable v20.0.

pyarrow_memory_leak_bug_report_df.pkl can be downloaded here

import pandas as pd
import pyarrow as pa

df = pd.read_pickle("pyarrow_memory_leak_bug_report_df.pkl")

"This gobbles the whole ram"
table = pa.Table.from_pandas(df)

EDIT:
Seems the issue is with this line: https://github.com/apache/arrow/blob/924394732dbf8dbc1f332b15204fef0258abc267/python/pyarrow/pandas_compat.py#L619C13-L619C76
This is where it starts to eat up all the memory.

Component(s)

Python

[Seminko]

Nevermind, I set up a schema and it works properly now.

[Seminko]

Maybe it would be a good idea to investigate this after all. I mean, even without a schema it should work, shouldn't it.

[GujaLomsadze]

TL;DR: Not a memory leak. pyarrow infers dict -> struct, and a nested dict here has
near-unique keys (73,106 distinct / 74,270 rows), so it builds a struct with ~73k
fields, which is O(rows x keys), about 5.4B cells / ~22GB. The same data as a map
does the full dataset in 0.03s / 231MB.

It's not pickle or pandas-specific. (3-line repro
below). Question at the end: guardrail, docs, or working-as-intended?

import pyarrow as pa
data = [{f"key_{i}": "x"} for i in range(10_000)]  # 10k dicts, each a unique key
pa.array(data)   # -> struct<10000 fields>; bump to 100k and it eats all your RAM

struct_fields == N exactly: every distinct key becomes a field.

Full script to reproduce this (click to expand)

"""
Reproduce apache/arrow #47133: dict-with-unique-keys -> struct explosion.
"""
import time
import resource
import pyarrow as pa
import pandas as pd


def peak_rss_mb():
    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024


def make_dicts(n):
    # each row = a dict with a single UNIQUE key -> n distinct keys total
    return [{f"key_{i}": "x"} for i in range(n)]


print("pyarrow", pa.__version__, "| pandas", pd.__version__)
print("=" * 64)

# --- 1. Minimal repro: plain pyarrow, NO pandas ---
print("\n[1] no pandas: pa.array(list_of_dicts)")
data = make_dicts(10_000)
arr = pa.array(data)
print("    struct fields :", len(arr.type))                 # 10000
print("    first 3 names :", [arr.type.field(i).name for i in range(3)])
print("    nbytes        :", round(arr.nbytes / 1e6, 1), "MB")

# --- 2. Same thing via pandas from_pandas (how the report hit it) ---
print("\n[2] pandas: pa.Table.from_pandas(df)")
df = pd.DataFrame({"col": data})
tbl = pa.Table.from_pandas(df)
print("    struct fields :", len(tbl.schema.field("col").type))
print("    nbytes        :", round(tbl.column("col").nbytes / 1e6, 1), "MB")

# --- 3a. Scaling WITHOUT pandas (pa.array) ---
print("\n[3a] scaling, no pandas (pa.array) -- doubling N ~= 4x nbytes")
print(f"     {'N':>6} {'fields':>7} {'nbytes(MB)':>11} {'time(s)':>8}")
for n in (2000, 4000, 8000, 16000):
    d = make_dicts(n)
    t = time.time()
    a = pa.array(d)
    dt = time.time() - t
    print(f"     {n:>6} {len(a.type):>7} {a.nbytes / 1e6:>11.1f} {dt:>8.2f}")

# --- 3b. Scaling WITH pandas (from_pandas) ---
print("\n[3b] scaling, pandas (from_pandas) -- same quadratic curve")
print(f"     {'N':>6} {'fields':>7} {'nbytes(MB)':>11} {'time(s)':>8}")
for n in (2000, 4000, 8000, 16000):
    d = make_dicts(n)
    df_n = pd.DataFrame({"col": d})
    t = time.time()
    t_n = pa.Table.from_pandas(df_n)
    dt = time.time() - t
    nb = t_n.column("col").nbytes / 1e6
    print(f"     {n:>6} {len(t_n.schema.field('col').type):>7} {nb:>11.1f} {dt:>8.2f}")

# --- 4. The fix: it's really a map, not a struct (both paths) ---
print("\n[4] correct type: map<string, string> (cheap, no explosion)")
m_nopd = pa.array(data, type=pa.map_(pa.string(), pa.string()))
m_pd = pa.Table.from_pandas(
    df, schema=pa.schema([("col", pa.map_(pa.string(), pa.string()))])
)
print("    no pandas nbytes :", round(m_nopd.nbytes / 1e6, 2), "MB")
print("    pandas    nbytes :", round(m_pd.column("col").nbytes / 1e6, 2), "MB")
print("    (vs struct's ~400 MB for the same 10k rows)")

print("\n" + "=" * 64)
print("peak RSS this process:", round(peak_rss_mb(), 1), "MB")

---

@Seminko got the pickle, thanks! Drive link still works lol. Reproduced and root-caused -it's not a leak, it's schema
inference.

The url_parts dicts contain a nested queries dict whose keys are effectively unique across
rows (73,106 distinct keys in 74,270 rows). PyArrow infers dict -> struct, and a struct's
fields are the union of all keys, so queries becomes a struct with ~73k fields. Materializing
that over 74k rows is O(rows x fields), about 5.4B cells, roughly 22GB (almost all my RAM
haha). Batching to 10k "works" only because each batch sees far fewer distinct keys.

IMO the right Arrow type for open-ended keys is a map, not a struct. Converting the same
column as map<string, list<string>> does the full 74,270 rows in 0.03s / 231MB.

It's also not pickle- or pandas-specific. Pickle was just how the report shipped the data. The
real trigger is any dict column with lots of distinct keys (JSON/API payloads, scraped or
parsed URLs, NoSQL dumps, and so on). The 3-line repro above uses plain pa.array, no pandas
and no pickle. from_pandas just forwards to this same inference, so a guardrail at the
inference layer would cover all of these paths.

So: not a leak!! inference just builds a pathological struct. A few ways this could go, happy
to do whichever you prefer:

1. Guardrail: when an inferred struct exceeds some field count, raise a clear error pointing
   at pa.map_() instead of silently consuming entire RAM.
2. Docs: a short note on dict -> struct vs map, and when to use each.
3. Both.
4. ... something else, haha

Or if you'd rather treat this as working-as-intended, let me know the
preferred direction and I'll pick it up.
Way I look at it is that, at least it should drop some warning th

cc @AlenkaF @jorisvandenbossche - this is your area (from_pandas inference).