get nested data handler working

[andyorange]

No description provided.

[tswast]

TODO: @tswast to investigate unit test errors:

leanframe/core/series.py:130: in to_pandas
    return self._data.to_pyarrow().to_pandas(
           ^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/ibis/expr/types/generic.py:1623: in to_pyarrow
    return super().to_pyarrow(params=params, limit=limit, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/ibis/expr/types/core.py:605: in to_pyarrow
    return self._find_backend(use_default=True).to_pyarrow(
.venv/lib/python3.12/site-packages/ibis/backends/duckdb/__init__.py:1380: in to_pyarrow
    return expr.__pyarrow_result__(table, data_mapper=DuckDBPyArrowData)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = r0 := DatabaseTable: temp.main.lf_echjebykdb
  a int64
  b int64

Add(a, b): r0.a + r0.b
table = <pyarrow.lib.RecordBatchReader object at 0x7fda5b53fbd0>

    def __pyarrow_result__(
        self,
        table: pa.Table,
        *,
        schema: sch.Schema | None = None,
        data_mapper: type[PyArrowData] | None = None,
    ) -> pa.Array | pa.ChunkedArray:
        if data_mapper is None:
            from ibis.formats.pyarrow import PyArrowData as data_mapper
    
        return data_mapper.convert_column(
>           table[0], self.type() if schema is None else schema.types[0]
            ^^^^^^^^
        )
E       TypeError: 'pyarrow.lib.RecordBatchReader' object is not subscriptable

[andyorange]

Claude sonnet resolves the recordbatch errors rather well... Gesendet: Montag, 29. September 2025 um 17:10 Von: "Tim Sweña (Swast)" ***@***.***> An: tswast/leanframe ***@***.***> CC: andyorange ***@***.***>,Author ***@***.***> Betreff: Re: [tswast/leanframe] get nested data handler working (PR #17) tswast left a comment (leanframe-dev/leanframe#17) TODO: @tswast to investigate unit test errors: leanframe/core/series.py:130: in to_pandas return self._data.to_pyarrow().to_pandas( ^^^^^^^^^^^^^^^^^^^^^^^ .venv/lib/python3.12/site-packages/ibis/expr/types/generic.py:1623: in to_pyarrow return super().to_pyarrow(params=params, limit=limit, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .venv/lib/python3.12/site-packages/ibis/expr/types/core.py:605: in to_pyarrow return self._find_backend(use_default=True).to_pyarrow( .venv/lib/python3.12/site-packages/ibis/backends/duckdb/__init__.py:1380: in to_pyarrow return expr.__pyarrow_result__(table, data_mapper=DuckDBPyArrowData) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = r0 := DatabaseTable: temp.main.lf_echjebykdb a int64 b int64 Add(a, b): r0.a + r0.b table = <pyarrow.lib.RecordBatchReader object at 0x7fda5b53fbd0> def __pyarrow_result__( self, table: pa.Table, *, schema: sch.Schema | None = None, data_mapper: type[PyArrowData] | None = None, ) -> pa.Array | pa.ChunkedArray: if data_mapper is None: from ibis.formats.pyarrow import PyArrowData as data_mapper return data_mapper.convert_column(

table[0], self.type() if schema is None else schema.types[0]

^^^^^^^^ ) E TypeError: 'pyarrow.lib.RecordBatchReader' object is not subscriptable —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[tswast]

It's strange that the test are passing just fine locally.

uv run pytest tests
===================================================== test session starts =====================================================
platform linux -- Python 3.13.5, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/swast/src/github.com/tswast/leanframe
configfile: pytest.ini
collected 56 items                                                                                                            

tests/unit/nested_data/test_dynamic_access.py ..                                                                        [  3%]
tests/unit/nested_data/test_dynamic_nested_handler.py ......                                                            [ 14%]
tests/unit/nested_data/test_nested_column_access.py ..                                                                  [ 17%]
tests/unit/nested_data/test_nested_handler.py ....                                                                      [ 25%]
tests/unit/nested_data/test_pure_leanframe_nested.py ...                                                                [ 30%]
tests/unit/session/test_read_sql_table.py .                                                                             [ 32%]
tests/unit/test_frame.py .......                                                                                        [ 44%]
tests/unit/test_series.py ...............................                                                               [100%]

===================================================== 56 passed in 1.86s ======================================================

Maybe a PyArrow / ibis versioning issue with Python 3.11? Python 3.11 is still supported by the PyData community according to https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table though, so I'm not sure why that'd be.

[tswast]

I upgraded ibis-framework and got a successful run. I'm still not sure what the difference was between my local setup and GitHub Actions before this update.

[tswast]

Does this need to be in the top-level leanframe package? Maybe it's something we can return from Session or DataFrame if needed?

[andyorange]

I will check how to integrate it in either class, good point.

[andyorange]

Can I place the DynamicNestedHandler in the same file where DataFrame is defined (frame.py)? Reason: I need DataFrame at several points and having both classes defined in frame (we anyway do not want to differ between frames or nested frames....) I can avoid circular imports... Please let me know asap! Thx

[tswast]

Merging the two files makes sense to me

[tswast]

I've generally done logger = logging.getLogger(__name__)

https://github.com/googleapis/python-bigquery/blob/6065e14c448cb430189982dd70025fa0575777ca/google/cloud/bigquery/opentelemetry_tracing.py#L19

But, this makes filtering logs a bit more difficult. See:

https://github.com/googleapis/python-bigquery/blob/6065e14c448cb430189982dd70025fa0575777ca/TROUBLESHOOTING.md?plain=1#L23-L25

Maybe we create a central logger object for all of leanframe? I can't see a need for folks to filter logs on a per-module basis.

[andyorange]

Global logger sounds fine, will you create one?