[MNT] narwhals migration by solegalli · Pull Request #965 · feature-engine/feature_engine

solegalli · 2026-07-21T20:40:09Z

No description provided.

ojassharma7 · 2026-07-28T22:22:31Z

Hi @solegalli — I'd like to help with the narwhals migration.

If it is still free, I can take feature_engine/scaling first (small surface: mainly MeanNormalisationScaler) as a single-module PR, following the dataframe_checks pattern from #966.

Please let me know if that module is already spoken for — happy to pick another (e.g. a simpler preprocessing piece) instead.

solegalli · 2026-07-29T13:54:17Z

That is actually a good one to start with. The tests should pass with pandas. I am not sure they will pass with polars because we need to change the functions that select variables, on which I am working on right now and will soon make a PR.

ojassharma7 · 2026-07-29T15:06:16Z

Started on scaling as discussed — opened a PR against this branch: will link here once created (see latest open PR from @ojassharma7 titled migrate scaling module to narwhals).

Pandas tests for the module pass locally. As you said, polars may still need your variable-selection updates.

ojassharma7 · 2026-07-29T15:06:22Z

Scaling PR: #979

FBruzzesi

Hi @solegalli - Following up on my discord comment. I added a few comments, mostly focusing on the changes in feature_engine/dataframe_checks.py - I hope you find them helpful

I noticed that a lot of tests were refactored as well: if you want to test the same behavior for many dataframes, I would reference what I did for fairlearn (see their conftest file), namely create fixture dataframe constructor for all the dataframe types you want to test. Ideally I would like to move that into narwhals as well (see narwhals-dev/narwhals#3552), but that's still work-in-progress and under discussion 🙏🏼

FBruzzesi · 2026-08-23T20:17:09Z

-    elif isinstance(y, pd.DataFrame):
-        if y.isnull().any().any():
+        if nw_y.dtype.is_numeric():
+            if not np.isfinite(nw_y.to_numpy()).all():


Suggested change

if not np.isfinite(nw_y.to_numpy()).all():

if not nw_y.is_finite().all():

(see Series.is_finite())

Hi @FBruzzesi , thanks for the suggestion. It seems that using numpy is faster than using narwhals both for pandas and polars (mostly so for pandas). Is this a known issue?

For the polars case we run its native functionality polars.Series.is_finite. I am surprised that's faster than numpy, at least at scale

For pandas-like, we do (s > float("-inf")) & (s < float("inf")). IIRC that's to avoid using numpy with non-numpy backed series (e.g. pyarrow backed series, cudf series that live in the GPU, etc). If the delta is large at scale, we can take a look for a refactor with performance in mind.

For context: in general we tend to use the native dataframe libraries API/functionalities. pandas is a special kid as we need to do quite some gymnastic for null vs nan's, its datatype system, its multiple backends, etc..

So please keep reporting these kind of performance issues - we aim to keep overhead at the minimum

Thanks for replying so quickly. These are the values I've got (on pandas and polars, 200k rows × 20 cols):

Check pandas polars
null check (multi-col) narwhals-native 1.2x slower narwhals-native 4x slower
inf check (multi-col) narwhals-native 2.4x slower narwhals-native 1.3x slower
is_finite (single series) narwhals-native 10x slower ~same

is_finite is the same for polars, the inf and null checks make it a bit slower respect to numpy.

For pandas I just opened a PR to use numpy/cupy/pyarrow.compute native functionalities directly: see narwhals-dev/narwhals#3874

For polars, I cannot tell why numpy is faster than their native implementation - If interested, you can double check with them either in discord or in their repo

FBruzzesi · 2026-08-23T20:18:11Z

+
+    if nwd.is_into_dataframe(y):
+        nw_y = nw.from_native(y, eager_only=True)
+        if nw_y.select(nw.all().is_null().any()).to_numpy().any():


You can avoid casting to numpy:

Suggested change

if nw_y.select(nw.all().is_null().any()).to_numpy().any():

if nw_y.select(nw.any_horizontal(nw.all().is_null().any())).item():

FBruzzesi · 2026-08-23T20:20:11Z

+        "`missing_values='ignore'` when initialising this transformer."
+    )
+    nw_X = nw.from_native(X, eager_only=True)
+    if nw_X.select(nw.col(variables).is_null().any()).to_numpy().any():


Similar as above, you can use any_horizontal

* address review feedback on dataframe_checks.py Follow-up to FBruzzesi's review on PR #965: - Clarify docstrings for check_X, check_y, check_X_y in terms of which dataframe libraries are safe to pass in (pandas, polars, PyArrow, modin, cuDF), instead of narwhals-specific "eager" terminology. - Use narwhals' IntoDataFrameT instead of IntoDataFrame for check_X and check_X_y, since both return the same concrete dataframe type they receive. - Fix a null/NaN detection bug: in polars, is_null() does not catch an explicit float("nan") value (only None counts as null), so check_y and _check_contains_na could silently miss NaNs in polars data. Now also check is_nan() for numeric columns/series, keeping numpy for the finite/inf checks since it benchmarks as fast or faster there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * inline null/nan checks to match FBruzzesi's suggested one-liner Collapses the has_na/has_null/has_nan accumulator variables into a single short-circuiting if-condition, as suggested in review. This also avoids an unnecessary is_nan() call when is_null() already found a null value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * speed up numeric column detection in _check_contains_na The schema-based list comprehension rebuilt narwhals' full column schema on every single-column access, making it scale roughly quadratically with column count on pandas (benchmarked up to ~500x slower than necessary at 200 columns). Switch to the pandas fast-path / narwhals-selector pattern already used in variable_handling (find_numerical_variables, check_numerical_variables) for the same "which of these columns are numeric" problem. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* update dataframe checks * update dataframe checks take 2 * update dataframe checks take 3 * update docstrings * refactor dataframe checks * fix mypy error * add missing type hints * add missing matching error syntax * finalise tests for df checks'

The project already requires scikit-learn>=1.7.0 (pyproject.toml, tox.ini, .circleci/config.yml), so the sklearn<=1.6 branches of every check_estimator/tags conditional were dead code. This removes them, keeping only the >=1.6 branch (the one using check_estimator(expected_failed_checks=...)): - feature_engine/tags.py: collapse the sklearn_version > 1.6 check in _return_tags(), the shared helper used across ~20 estimator classes. - 11 tests/**/test_check_estimator_*.py files: collapse each if/else on sklearn_version vs 1.6, drop the now-unused sklearn/ parse_version imports and sklearn_version variables. - tests/test_prediction/test_check_estimator_prediction.py: this file had no >=1.6 branch, only the dead <1.6 one (its own TODO already flagged this). Removing it leaves the prediction module with no test_check_estimator_from_sklearn coverage - a pre-existing gap, not introduced by this change, left as a follow-up. - tests/test_creation/test_geo_features.py: __sklearn_tags__ always exists at sklearn>=1.7, so drop the hasattr() guard around it. - tests/test_wrappers/test_sklearn_wrapper.py: also collapse the _OneHotEncoder() test helper's sparse/sparse_output branch (sklearn <1.2 compat, dead for the same reason). The separate KBinsDiscretizer(quantile_method=...) branch (sklearn<1.7) is intentionally left as-is - different threshold, out of scope here. - tests/check_estimators_with_parametrize_tests.py: delete entirely. A standalone, non-CI reference file documenting the pre-1.6 parametrize_with_checks() call signature. _more_tags()/__sklearn_tags__() method definitions are untouched: _more_tags() is feature_engine's own internal metadata/xfail-checks store (read by tests/estimator_checks/*.py), not a legacy sklearn shim, and __sklearn_tags__() is the current sklearn API. Verified: identical test suite pass/fail counts before and after (2010 passed, 114 failed - all 114 are pre-existing narwhals-migration WIP failures unrelated to this change), flake8 and mypy clean (the one remaining mypy error is pre-existing in datetime_subtraction.py, unrelated to this PR).

* address review feedback on dataframe_checks.py Follow-up to FBruzzesi's review on PR #965: - Clarify docstrings for check_X, check_y, check_X_y in terms of which dataframe libraries are safe to pass in (pandas, polars, PyArrow, modin, cuDF), instead of narwhals-specific "eager" terminology. - Use narwhals' IntoDataFrameT instead of IntoDataFrame for check_X and check_X_y, since both return the same concrete dataframe type they receive. - Fix a null/NaN detection bug: in polars, is_null() does not catch an explicit float("nan") value (only None counts as null), so check_y and _check_contains_na could silently miss NaNs in polars data. Now also check is_nan() for numeric columns/series, keeping numpy for the finite/inf checks since it benchmarks as fast or faster there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * inline null/nan checks to match FBruzzesi's suggested one-liner Collapses the has_na/has_null/has_nan accumulator variables into a single short-circuiting if-condition, as suggested in review. This also avoids an unnecessary is_nan() call when is_null() already found a null value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * speed up numeric column detection in _check_contains_na The schema-based list comprehension rebuilt narwhals' full column schema on every single-column access, making it scale roughly quadratically with column count on pandas (benchmarked up to ~500x slower than necessary at 200 columns). Switch to the pandas fast-path / narwhals-selector pattern already used in variable_handling (find_numerical_variables, check_numerical_variables) for the same "which of these columns are numeric" problem. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* update variable handling module for narwahls * creating own datetime parser * Improve readability of narwhals date/type-check helpers, add missing tests Replace the double-parse-with-disagreeing-defaults trick in _looks_like_date_string with a direct call to dateutil's parser()._parse(), which exposes which date/time fields were actually found in a string without needing to approximate it - this also drops the now-unneeded sentinel default datetimes and the defensive str() coercion at its call site. Make truthiness checks and compound boolean returns explicit throughout the module, and restore the pre-narwhals function names that PR #978 had prefixed with _nw_ for no continuing reason. Rename test_fe_type_checks.py to test_variable_type_checks.py to match the module it tests, add docstrings, and add coverage for _looks_like_date_string and _is_categories_num, the two functions that previously had no direct tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Replace per-column schema access with bulk narwhals selectors for speed nw_X.schema is not cached - every access re-derives the full schema from the underlying native dataframe, so checking dtype-based conditions (is_numeric(), native Date/Datetime, categorical/enum/string) one column at a time inside a loop was quadratic instead of linear. Replace each such loop with a single nw_df.select(<selector>).columns call converted to a set, then a plain membership test per column - confirmed old vs new give identical results, and measured 8x-120x speedups depending on backend and column count. Also use by_dtype(Date, Datetime) to bulk-detect native datetime columns in one pass, only falling back to the expensive per-value _is_categorical_and_is_datetime check for columns that aren't already known to be numeric or natively datetime. Drop the now-unused _is_date_or_datetime import from both files. Simplify _looks_like_date_string's comment to link directly to the pandas source it mirrors, and instantiate dateutil's parser() per call instead of reusing a module-level instance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor find variables: * refactor check_variables * final refactor of find and check variables * finalise migration of variable handling module * update user guide * Trim backend-difference notes from docs, revert datetime.py out of scope Removes the trailing pandas/polars note blocks from the check/find categorical and datetime variable docs, keeping them focused on the walkthrough. Reverts feature_engine/datetime/datetime.py to main - the DatetimeFeatures index-datetime fix needed there for the narwhals migration belongs in a separate datetime-module PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@FBruzzesi

* Migrate creation/mixins shared base classes to narwhals, remove all pandas imports BaseCreation, BaseNumericalTransformer, and mixins.py (TransformXyMixin, FitFromDictMixin, GetFeatureNamesOutMixin) are used by every transformer in the creation module, so their remaining pandas-only code blocked a polars-only install regardless of which transformer was migrated. Adds pandas fast paths (benchmarked ~2-11x) alongside narwhals-generic branches, replaces y.loc[X.index] row alignment in TransformXyMixin with a narwhals with_row_index()-based mechanism for non-pandas backends, and adds test_base_creation.py plus polars coverage for transform_x_y. * Apply suggestion from @FBruzzesi * Fix test_get_feature_names_out_mixin.py after to_list() removal, add process rules to AGENTS.md The 48 failures here were pre-existing (unrelated to the to_list() fix, confirmed identical before/after): check_X no longer accepts raw numpy arrays, and most of this file's tests fit() on df_vartypes.to_numpy() or feed a raw-array-outputting sklearn transformer upstream. Fixes: - array-input tests converted to set feature_names_in_/n_features_in_ directly, since that's the only way left to reach the mixin's x0/x1/... naming branch (fit() rejects arrays outright now). - SimpleImputer/PolynomialFeatures steps get .set_output(transform="pandas") so they hand a dataframe to the next pipeline step instead of an array - this is also the fix any real user chaining sklearn + feature-engine transformers in a Pipeline now needs. - pure Mock-only tests (no sklearn transformer involved) parametrized over pandas and polars. Also adds two AGENTS.md rules: run a changed function/class's tests and resolve any failures, and keep user-guide docs in sync with new transformer functionality. * Remove dead array-input branch from GetFeatureNamesOutMixin This branch handled feature_names_in_ == ["x0", "x1", ...], the naming sklearn gives an estimator fit on a raw array. check_X no longer accepts arrays (dataframe-only input, per AGENTS.md), so fit() can never produce that pattern anymore - the branch, its indices=True path in _remove_feature_names, and get_support(indices=True) were all unreachable. It was also a latent correctness gap: a dataframe with columns genuinely named x0..xn would have hit this branch and skipped the usual input_features-must-match-feature_names_in_ validation. Verified via git history (#519, 2022) this was built for the old array-accepting check_X; confirmed no other code in the library still generates x0/x1/... names. Removed the branch, its now-single-path _remove_feature_names, and the tests that existed only to reach it - replaced by tests/test_base_transformers/test_get_feature_names_out_mixin.py's remaining pandas+polars dataframe coverage, which already exercises the same validation/renaming logic through the one reachable path.

* Migrate CyclicalFeatures to narwhals, add polars support fit(): unified across backends via .to_numpy().max(axis=0) instead of pandas' .max().to_dict() (~1.55x faster for pandas, ~1.28x for polars, benchmarked). .tolist() keeps the returned dict's values as plain Python int/float, matching the old .to_dict() dtype. transform(): kept as two branches rather than one narwhals-only path - benchmarked running narwhals expressions against a pandas-backed frame and it was consistently 1.24x-2.06x slower than the pandas-native loop across variable counts and row counts, worse at small scale. The pandas branch is therefore left as the original, unmodified loop (an earlier numpy-vectorized version of it was only a 1.0x-1.4x gain, not worth it once the branches stay separate anyway). The narwhals branch uses column expressions, the only approach that stayed competitive with pandas-native as variable count grows (a numpy-array round-trip loses to expressions on polars once there is more than 1 variable). Verified no legacy numpy-array-input code remains in this file or its base classes. Tests rewritten to parametrize pandas and polars via make_df; error-matching tightened per AGENTS.md except where the message legitimately differs by backend. Docstring and user-guide example gained a polars walkthrough per the new AGENTS.md doc-sync rule. * unify pandas/polars branches * Fix style/docs failures on top of the pandas/polars branch unification Style: removed the now-unused narwhals.dependencies import (flake8 F401) left over from dropping the is_pandas_dataframe branch. Also fixed 7 pre-existing flake8 issues (line length, unused variable) in test_get_feature_names_out_mixin.py that predate this branch. Docs: docs/user_guide/creation/CyclicalFeatures.rst's polars output block was under `.. code:: python`, and Sphinx's Pygments highlighter can't lex the box-drawing table as Python (misc.highlighting_failure), which -W promotes to a build error. Switched to `.. code:: text`, matching the convention already used elsewhere (PowerTransformer.rst, MeanImputer.rst) for output-only blocks. Pre-existing bug in my own doc addition, unrelated to the branch unification. Two correctness issues surfaced by testing the unification: - max_values_ lost its .tolist() call, so it held numpy scalars (np.int64) instead of plain Python int/float - restored. - narwhals' .select([]) collapses row count to 0 (not just columns), so routing pandas through the narwhals numpy path broke return_empty=True (empty variables_) with a "zero-size array to reduction operation maximum" error. Guarded for it explicitly, since return_empty=True is a real, designed-for case, not a hypothetical.

@solegalli

* Migrate GeoDistanceFeatures to narwhals, add polars support Six pandas-specific spots split into a pandas-native branch and a narwhals-generic branch, each decision benchmarked at 10k-50k rows and 0/1/6 extra columns (not assumed): - missing-columns check, feature_names_in_ extraction: narwhals-on-pandas is 13-22x slower (pure metadata overhead, row-count independent) - kept the pandas fast path established in Pass 1. - coordinate range validation: 6-8.6x slower on narwhals-on-pandas - new narwhals branch added (previously crashed outright on polars), pandas branch untouched. - numpy extraction of the 4 coordinate columns: 5-9x slower via narwhals on pandas; for the narwhals branch itself, .get_column().to_numpy() per column beats .select().to_numpy() by 5-7x on polars, so that's what it uses. - assign new column + optional drop: 1.7-2.9x slower on narwhals-on-pandas, consistent with the bar CyclicalFeatures used to keep branches separate. - column reorder is the one exception - narwhals-on-pandas is actually ~35% *faster* here at 10k rows - but stays a two-branch split per an explicit decision to keep the narwhals-everywhere pattern consistent with Pass 1/2, rather than special-case one operation. Verified end-to-end (not just isolated snippets): pandas output identical to the pre-migration code, polars value-identical to pandas, ~2% pandas speed delta (noise) at 10k rows/1 extra column, both backends' fit() error paths (missing columns, out-of-range coordinates) raise the same messages. Also fixed a pre-existing, unrelated inaccuracy in the class docstring's Examples section - the documented pandas output didn't match what the current (pre-migration) code actually produces. The same drift exists in the user guide's Python-implementation number tables (haversine, euclidean, manhattan, miles) but fixing those throughout is out of scope for this pass - flagged separately. Tests parametrized pandas+polars where a dataframe is involved; pure __init__/tag-validation tests (no dataframe) left as-is, already using match= throughout. * Apply suggestion from @solegalli * Fix stale example output throughout GeoDistanceFeatures user guide Every numeric output table in the "Python implementation" section (haversine, euclidean, manhattan, miles) had drifted from what the code actually produces - confirmed by running each documented example directly and comparing. Some differences are rounding-level, but euclidean trip 4 (1720.18 documented vs 1898.82 actual) and manhattan trip 2 (4684.16 vs 4266.82) are real gaps, and the pipeline predictions example was the furthest off: documented as the training targets exactly ([100, 150, 80, 200]), actual output is [116.67, 120.75, 88.48, 204.10]. Pre-existing, unrelated to the narwhals migration - verified the old, unmigrated code produces the same "actual" numbers used here.

* Migrate MathFeatures to narwhals, add polars support The numpy-reducer fast path (sum/mean/std/var/min/max/prod/median) is unified into a single narwhals-based code path rather than split by backend: benchmarked narwhals-on-pandas vs pandas-native at 10k rows/3 reducers and found only a 1.01x-1.27x difference, well under the bar that kept CyclicalFeatures/GeoDistanceFeatures split (1.7x+). Value extraction for the fast path stays a small pandas/narwhals split though - narwhals' select() doesn't accept integer column names the way pandas' own indexing does, and int-named variables is a real, tested, pandas-only feature (polars requires string columns). The custom-callable/uncommon-aggregation fallback can't be unified at all - narwhals has no row-wise apply. Pandas keeps .agg(func, axis=1); polars uses its native map_rows(), which passes each row as a plain tuple rather than a Series, so callables relying on Series methods (row.max()) need max(row) instead to work on both backends. Documented this explicitly. A non-callable func (e.g. an uncommon pandas aggregation string like "sem") now raises NotImplementedError for polars input rather than failing obscurely, since there's no way to resolve a pandas-specific aggregation name without pandas itself. Also fixed a real bug: the module-level `_PANDAS_LT_3 = int(pd.__version__...)` constant required pandas importable just to import this module at all, breaking every creation transformer for a polars-only install. Replaced with a lazy check using narwhals.dependencies.get_pandas() (returns the already-imported module without importing it), computed only once we already know X is pandas-backed. User guide had three separate pre-existing inaccuracies, unrelated to this migration (confirmed against the old, unmigrated code): a get_feature_names_out example listed 'amin_Age_Marks'/'amax_Age_Marks' for a transformer that was never passed np.min/np.max - it uses plain "min"/"max" strings, which have always produced "min_Age_Marks"/"max_Age_Marks"; and a std column's values matched pre-pandas-3 semantics (ddof=1) for a np.std example that runs under ddof=0 in the installed pandas 3.x, already reflected in this repo's own tests. Fixed both while verifying every table for the new "With polars" section. * Rewrite MathFeatures tests to run the same test against both backends Previously: the original pandas-only tests were left untouched and new, separate polars-only tests were added alongside them for the same behavior. That's not what dataframe-agnostic means - same input in, same values out, checked by the same test. Rewrote every test that touches a dataframe to build it via make_df and parametrize over [pd.DataFrame, pl.DataFrame], replacing pd.testing.assert_frame_equal with a cross-backend assert_df_equal (nw.from_native(...).to_dict() + a per column approx compare, handling None-vs-NaN as the same "missing" value on both sides). The one deliberately un-unified case: an uncommon aggregation string like "sem" succeeds on pandas (routes through its native .agg()) but raises NotImplementedError on polars (no way to resolve an arbitrary pandas-specific string without pandas) - that's a real, documented asymmetry, not an oversight, so it's one parametrized test with an explicit if/else on the expected outcome rather than two separate tests pretending it's the same behavior. Two genuinely pandas-only tests stay pandas-only, with a comment saying why: integer column names (polars requires string columns) and pandas' nullable Int64 dtype (no polars equivalent). Custom-callable fallback tests merged into one using max()/min()/sum() built-ins, which work identically whether the callable receives a pandas Series (pandas' agg(axis=1)) or a plain tuple (polars' map_rows) - no need for Series-specific vs tuple-specific callables in separate tests. Picked up narwhals.dependencies.is_pandas_dataframe(X) is True -> nwd.is_pandas_dataframe(X) and the _pandas_lt_3() -> _pandas_version() rename from upstream changes to the class file. * fix: correct _pandas_version() return type hint from bool to int The function returns int(pandas_version.split(".")[0]) and is used as _pandas_version() < 3, but its signature still said -> bool, failing type checking. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate RelativeFeatures to narwhals+numpy, add polars support Replaces the 8 near-identical _add/_sub/_mul/_div/_truediv/_floordiv/_mod/ _pow pandas methods (~90 lines) with a single numpy-ufunc-driven transform(), per request. Benchmarked at 10k rows, 3 variables, 2 references: the numpy version is not just "minimal loss" but actually faster than the current pandas .div(..., axis=0) approach (552.6us vs 637.0us, 0.87x) - so this is a single unified narwhals+numpy code path, no pandas/polars branch at all (re-verified against the final committed code: 635.9us pandas, down from 800.7us before this change; 262.4us polars, previously unsupported). One correctness fix during implementation: extracting all `variables` as one batched 2D array via select().to_numpy() upcasts every column to a common dtype, silently turning an int column's subtraction result into float and failing 3 existing tests. Fixed by extracting each variable as its own 1D array instead, preserving each column's own dtype promotion independently - matches pandas' per-column .sub()/.div()/etc. semantics, still a single vectorized numpy op per column (no Python-level row loop). Also matched a subtler pandas behavior: floordiv/mod on integer input stay integer-typed, and assigning a float fill_value at zero-denominator positions needs the result array explicitly widened to float first (numpy arrays don't auto-promote dtype on assignment the way pandas' DataFrame column assignment does) - verified this reproduces pandas' output exactly, including for negative numbers (floor-division sign conventions matched NumPy's floor_divide/mod exactly across int/float/negative cases, so no other adjustment was needed there). User guide's example tables verified accurate already (including the Age_pow_Age int64-overflow values, which are genuine hardware overflow behavior, not a doc error - confirmed identical between pandas and polars). Added "With polars" sections to docstring and user guide. * test: merge pandas/polars tests for RelativeFeatures into single parametrized suite Same treatment as the MathFeatures test rewrite: one test per behavior, parametrized over make_df=[pd.DataFrame, pl.DataFrame], checking identical values come out for identical input instead of separate pandas-only and polars-only test functions. Deletes the redundant separately-added polars section, keeps its 3 genuinely-new cases (mixed dtype preservation, float fill_value dtype widening, drop_original column list), and converts the pandas-specific .loc-based zero-fill assertion to a narwhals-based one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate DecisionTreeFeatures to narwhals, add polars support Follows the same pandas-native / narwhals-generic split established for GeoDistanceFeatures (this transformer also reimplements fit()/transform() directly, not via BaseCreation): .columns extraction, column reorder, prediction-column assignment, and drop_original all split by backend, consistent with every other operation in this module that's been benchmarked as a real (not minimal) loss when routed through narwhals on pandas. Confirmed empirically before designing: sklearn's DecisionTreeRegressor/ Classifier and GridSearchCV accept a polars DataFrame directly for both fit() and predict()/predict_proba(), so the actual tree training/inference calls are unchanged - only the surrounding column selection, extraction, and reassembly needed migrating. Fixed a pre-existing bug found while rewriting the exact code path it lived in: single-feature combos with an integer column name (e.g. DecisionTreeFeatures(features_to_combine=1) on a dataframe with columns 0, 1, ...) crashed, since the original `isinstance(features, str)` check missed the int case and fell through to plain X[features] indexing, which returns a 1D Series rather than the 2D input sklearn requires. Widened to isinstance(features, (str, int)); verified the same single-feature narwhals path (get_column().to_frame()) already handles both cleanly. Regression, binary classification, and multiclass classification paths all verified to produce identical predictions between pandas and polars input. return_empty=True + polars remains untestable here too (same nw.col([]) bug in dataframe_checks.py found during CyclicalFeatures, still tabled) - this is the second transformer it blocks. docs/user_guide/creation/DecisionTreeFeatures.rst is large (511 lines) and built around actual cross-validated tree fitting on the real California housing dataset across many sections - re-verified the cheap, deterministic parts (the raw data table) but did not re-run every tree-fitting example given the cost of repeated grid-search CV fits; unlike the other three creation-module docs this pass touched, the rest of this file's numbers are unverified. Added a self-contained "With polars" section using simple synthetic data instead, fully verified. * Apply suggestion from @solegalli * Apply suggestion from @solegalli * docs: clarify is True/is False and cross-backend test conventions in AGENTS.md Two rules made explicit based on recent work: the is True/is False comparison is for flow control only, not variable assignment (per Sole's own simplification of is_pandas = nwd.is_pandas_dataframe(X) is True to just nwd.is_pandas_dataframe(X) in decision_tree_features.py); and dataframe-agnostic transformers get one parametrized test per behavior covering both pandas and polars, never separate per-backend tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat: add n_jobs for parallel tree training, merge tests to single cross-backend suite Adds an n_jobs parameter to DecisionTreeFeatures that parallelizes tree training across feature combinations via joblib, using threads rather than processes since fitting a decision tree releases the GIL for the bulk of its computation - threads avoid the overhead of copying the whole dataframe to worker processes. Defaults to None (sequential), preserving current behavior. Benchmarked on the committed transformer (5000 rows, 10 vars, features_to_combine=3, 8-point param_grid, 175 trees): 12.17s sequential vs 5.15s at n_jobs=-1, ~2.4x. On small workloads (a handful of feature combinations, the shape of the existing unit tests) parallelizing is a net loss - thread-dispatch overhead outweighs the gain - which is why the default stays sequential. Parallelizing transform()'s predict loop the same way was also benchmarked and found to have no benefit (predict is too cheap per call), so only fit()'s tree training is parallelized. Correctness verified: identical trees/predictions regardless of n_jobs. Also rewrites test_decision_tree_features.py to the single cross-backend-parametrized-test convention used elsewhere in this migration: one test per behavior over make_df=[pd.DataFrame, pl.DataFrame], deleting the separately-added polars-only section that duplicated coverage already present once the original tests are parametrized. Adds n_jobs correctness coverage (parallel vs sequential training gives identical output, both backends). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: avoid pandas fragmentation warning in DecisionTreeFeatures.transform transform() assigned one new tree-prediction column at a time (X[col_name] = preds), which triggers pandas' "DataFrame is highly fragmented" PerformanceWarning once there are enough feature combinations - confirmed with 10 vars/features_to_combine=3 (175 new columns). .assign(**kwargs) does NOT fix this: it inserts columns one at a time internally too, same warning. The actual fix is building all new columns into one DataFrame and joining once (single insertion). Verified: output is byte-identical to the old behavior (pd.testing.assert_frame_equal on a 3000-row/9-var/129-tree case), drop_original still works, and a new regression test confirms the warning is gone (and fails against the old code, confirming it actually catches the regression). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * shorten docstring --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Pure elementwise math (1 / x), so followed the same precedent as ArcsinTransformer (same module, same shape of problem): extract the transform columns to a single numpy array via narwhals' to_numpy(), apply the division once, reassign via nw.new_series + with_columns. Benchmarked narwhals-on-pandas vs the old pandas-native .loc-assignment across 10k-100k rows and 1-10 columns: narwhals-on-pandas was 2-3x *faster* than the old code (0.34x-0.45x of old runtime), narwhals-on- polars faster still - a stronger case for merging into one path than even ArcsinTransformer's parity/faster numbers, so no pandas/polars branch was added. The zero-denominator check (raises ValueError "Some variables contain the value zero...") is preserved exactly in both fit() and transform(), just computed via a numpy comparison on the extracted values instead of a pandas boolean mask. inverse_transform() is unchanged - it still just calls transform(), since 1/(1/x) = x. Rewrote test_reciprocal_transformer.py to one parametrized test per behavior over pandas/polars input (previously pandas-only, relying on the global df_vartypes/df_na fixtures - replaced with local dict data, same pattern as test_arcsin_transformer.py, so both backends build from the same source). Added a verified "With polars" section to the docs; left the pre-existing Ames-housing walkthrough untouched (no network access in this environment to re-verify fetch_openml output, and it wasn't modified by this migration). Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Pure elementwise math (arcsin(sqrt(x))), so followed the MathFeatures/ RelativeFeatures precedent: extract the transform columns to a single numpy array via narwhals' to_numpy(), apply np.arcsin(np.sqrt(...)) once, reassign via nw.new_series + with_columns. Benchmarked against the old pandas-native .loc assignment across 10k-100k rows and 1-10 columns: narwhals-on-pandas was consistently at parity or faster (0.4x-1.05x of old runtime, never a regression), so merged into one narwhals-generic path with no pandas/polars branch - same decision MathFeatures/RelativeFeatures landed on for the same shape of problem. fit() and transform() both extract the same numpy array for the range check (values must be in [0, 1]) and reuse it directly for the transform in transform(), avoiding a second backend round-trip. inverse_transform() follows the same pattern. Rewrote test_arcsin_transformer.py to one parametrized test per behavior over pandas/polars input (previously pandas-only, relying on the global df_vartypes/df_na fixtures - replaced with local dict data so both backends can build from the same source). Added a verified "With polars" section to the docs. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Same elementwise-math shape as ArcsinTransformer: extract the transform columns to one numpy array via narwhals' to_numpy(), apply np.arcsinh((x - loc) / scale) once, reassign via nw.new_series + with_columns. Benchmarked against the old pandas-native .loc assignment across 10k-100k rows and 1-10 columns: narwhals-on-pandas was consistently faster than the old code (0.48x-0.83x of old runtime), so merged into one narwhals-generic path with no backend branch. Found a pre-existing stale docstring while verifying output against the old code: the class docstring's example table (arcsinh of np.random.randn(100) * 1000 with seed 42) printed values that don't match what either the old or new code actually produces (e.g. 7.516076 vs the real 6.901163 for the first row) - confirmed by running the old (pre-migration) code directly, so this predates the migration. Fixed the docstring numbers to the verified real output. The docs/user_guide/transformation/ArcSinhTransformer.rst walkthrough's printed tables were re-run and already matched exactly, so those were left as-is; added a verified "With polars" section to both the docstring and the user guide. Rewrote test_arcsinh.py to parametrize every behavior over pandas and polars input (previously pandas-only). Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate OneHotEncoder to narwhals, add polars support Uses narwhals' to_dummies() for the actual expansion rather than a manual numpy/dict loop, since it's a real vectorized one-hot op on both backends. Handles two edge cases to_dummies() doesn't cover directly: a fixed-length prefix placeholder ("__ohe_tmp__") swapped back out by slicing rather than by using the real column name, since to_dummies() only prefixes with the Series name when it's truthy - a falsy real name (e.g. an int column literally named 0) would otherwise silently drop the prefix; and learned categories absent from (or present-but-unlearned in) a given transform batch, filled with an explicit all-0 column so unseen categories are encoded as 0 across the board, matching the pre-narwhals behavior exactly. fit()'s value_counts()/unique() calls and transform()'s reassembly are a single unified narwhals path - no pandas/polars split needed, verified directly on both backends (identical dummy columns/values for identical input). Rewrote tests/test_encoding/test_onehot_encoder.py to the single cross-backend-parametrized-test convention: local dict fixtures (dropping the pandas-only global df_enc_big/df_enc_numeric/df_enc_binary fixtures) parametrized over make_df in [pd.DataFrame, pl.DataFrame], with narwhals- based column/sum assertions replacing pd.testing.assert_frame_equal. test_variables_cast_as_category stays pandas-only (pandas category dtype has no polars equivalent under test there). Verified: 43/43 own tests, full encoding suite 340 passed/17 pre-existing failures (matches the narwhals-encoding-base baseline exactly), flake8 and mypy clean, sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning), no pandas import in this file itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt OneHotEncoder to narwhals-returning check_X Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables, _check_contains_na and _get_feature_names_in (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the now-redundant nw.from_native(X) round-trips in fit() and transform(); they reuse the narwhals frame returned by check_X / _check_transform_input_and_state. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in OneHotEncoder tests Replace the file-local data dicts and _columns/_colsum helpers with the shared test structure: make_df and data_enc* fixtures, isinstance(X, make_df) plus to_dict() checks (keeping the column-order assertions, which are part of this encoder's output contract), and pytest.raises(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group OneHotEncoder init tests, match errors, shorten comments Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Match the fixed get_feature_names_out error message Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor ohe * fix code style --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate RareLabelEncoder to narwhals, add polars support fit() replaces pandas .unique()/.value_counts(normalize=True) with narwhals Series.n_unique() (for the cardinality check - matches pandas' plain unique() length, which counts a null as its own category, unlike pandas' nunique() which drops it) and drop_nulls().value_counts(sort=True, normalize=True) (same drop_nulls()/sort=True reasoning as CountEncoder.fit(): narwhals' value_counts() has no dropna param and narwhals' own value_counts default is unsorted). transform() doesn't reuse CategoricalMethodsMixin._encode() (that's a dict-based numeric remap; this encoder keeps frequent categories as-is and only replaces the rest), so it's rewritten from pandas' .loc[~isin(...), feature] = replace_with onto nw.when(<Series>).then(<Series>).otherwise(nw.lit(replace_with)).alias( feature). Passing Series from get_column() (not nw.col()) into when/then/otherwise keeps this working for pandas integer column names, same as base_encoder.py's precedent. A pandas Categorical column still needs its own add_categories(replace_with) step before assignment - kept as a small is_pandas-gated block (structural, like base_encoder.py's existing reorder branches), since narwhals has no cross-backend equivalent and polars has no matching restriction. Unlike the old pandas-only code, no manual object-dtype fixup is needed before assignment for the ignore_format + numeric-variable + string replace_with case: narwhals resolves the common dtype itself (object in pandas, cast-to-string in polars). Benchmarked pandas-native vs narwhals-on-pandas vs narwhals-on-polars at 10k/50k/100k rows x 1/2/10 columns x 5/50 categories, warmed up. First pass (zip_with(col, new_series_filled_with_replace_with)) averaged 2.41x pandas-native at 50k-100k rows - most of that cost was constructing a full same-length replacement Series every transform() call (~2.5ms of a ~4.8ms transform at 100k rows, confirmed by isolating just the Series construction). Switched to nw.when(keep).then(col) .otherwise(nw.lit(replace_with)), which lets the backend broadcast the scalar instead of materialising a parallel array: dropped the average to 1.60x, converging to 1.12x-1.54x at 100k rows/10 columns, the "realistic size" range. narwhals-on-polars is faster than pandas-native throughout (0.7x-1.5x, mostly <1x at 50k+ rows). Merged into a single narwhals path per the established decision rule - no pandas/polars performance split - the remaining overhead is fixed per-call cost, not scaling cost, and stays under a few ms in absolute terms even at the largest sizes tested. Rewrote test_rare_label_encoder.py to one parametrized test per behaviour over @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]), replacing the shared pandas-only module-level fixtures (df_enc_big, df_enc_big_na, df_enc_numeric, from tests/conftest.py, still used by other encoder test files) with local dict constants both backends can build from, per the CountEncoder precedent. Kept test_when_varnames_are_numbers and the three category-dtype tests pandas-only (integer column names and pandas Categorical dtype are backend-specific per AGENTS.md). Split test_max_n_categories_with_numeric_var into a pandas-only version (the existing str()-workaround test, unchanged) plus a new polars-only version documenting the real, expected behavioural difference: polars can't hold mixed int/str values in one column the way pandas' object dtype does, so a numeric variable with a string replace_with casts the whole column to string instead of leaving frequent numeric categories as numbers. Verified: tests/test_encoding/test_rare_label_encoder.py - 39 passed (up from 29, from parametrizing over both backends); full tests/test_encoding suite - 336 passed, 17 pre-existing failures with identical test IDs confirmed against the unmodified base_encoder.py baseline (numpy-array-input rejection checks plus 3 MeanEncoder inverse_transform failures from mean_encoding.py's still-unmigrated fit() - predate this change, reproduced identically on the unmodified rare_label.py too). flake8 clean on feature_engine and tests. mypy clean. Module imports with pandas blocked (loaded standalone, same technique as the base_encoder.py migration, since sibling encoder files in this package still import pandas at module level). sphinx -W build clean (only the pre-existing linkcode_resolve warning, confirmed identical against the unmodified baseline). Verified every doc example in RareLabelEncoder.rst against actual output; fixed a pre-existing, unrelated value_counts() Series-name drift ("Name: var_A" -> "Name: count", a pandas version difference, not caused by this migration) while touching that page, and added a verified "With polars" section to both the class docstring and the user guide (the polars value_counts() example needed an explicit .sort() - unlike pandas, its groupby-based value_counts() order isn't stable run to run). The Titanic-dataset section of the user guide could not be re-verified against live output in this sandboxed environment (SSL cert verification blocks urllib by default here, though curl succeeds) and was left untouched; a workaround (unverified SSL context) showed matching encoder_dict_/transform output, with only an unrelated .unique() repr-formatting difference from a newer pandas version. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt RareLabelEncoder to narwhals-returning check_X Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables, _check_na and _check_contains_na (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the redundant nw.from_native(X) round-trips in fit() and transform(). In transform(), detect the pandas Categorical fix-up path via nw_X.implementation .is_pandas() and run it on a copy so the user's dataframe is not mutated. Drop the now-unused narwhals.dependencies import. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in RareLabelEncoder tests Replace the file-local data dicts and the _to_pandas helper - whose polars to_pandas() call needs pyarrow, which is not a dependency, so the polars cases failed - with the shared test structure: make_df and data_enc_big* / data_enc_numeric fixtures, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group RareLabelEncoder init tests, match errors, shorten comments Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix typo in RareLabelEncoder replace_with error Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * minor refactor to tests --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate StringSimilarityEncoder to narwhals, add polars support fit() rebuilds encoder_dict_ with narwhals cast(nw.String)/value_counts, matching the CountEncoder/RareLabelEncoder convention. cast() preserves nulls as null on both pandas and polars (verified empirically), unlike pandas' own astype(str) which stringifies NaN to "nan" - this lets "impute" mode fill_null("") directly and "ignore" mode drop_nulls() before casting, replacing the old "nan"/"<NA>" text-sentinel workaround with a real null check (col.is_null()) that can't collide with a genuine category literally named "nan" or "<NA>" (both edge cases stay covered by test_string_dtype_with_literal_nan_strings). transform()'s per-row difflib.SequenceMatcher similarity has no vectorised narwhals equivalent, so it's computed once per unique value via numpy broadcasting (np.unique's inverse index fans the small per-unique-value matrix back out to all rows) and reassembled with nw.new_series()/with_columns(), same pattern DecisionTreeFeatures uses for externally-computed new columns. Benchmarked a pandas-specific fast path (X.join(dict-of-columns), as DecisionTreeFeatures uses) against the unified narwhals with_columns() here across 10k-100k rows x 1-10 columns x 5-50 categories: assembly overhead ranges 0.9x-6.25x depending on shape, but the difflib computation itself dominates wall time by 1-3 orders of magnitude in every realistic scenario (e.g. 30ms difflib vs <1ms assembly overhead at 100k rows/20 categories) - even the worst synthetic case (500 output columns) only costs ~10ms extra out of an already tens-of-ms-to-seconds transform. Went with the unified/merged implementation: no is_pandas split, one code path for both backends. Rewrote tests as single parametrized cases over @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]), keeping only the pandas-NA-sentinel tests (np.nan/pd.NA/None, StringDtype) pandas-only since polars has no equivalent multi-sentinel behavior to exercise. All doc examples (including the Titanic worked example) re-verified against actual output; added a "With polars" section. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt StringSimilarityEncoder to narwhals-returning check_X Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables and _check_contains_na (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the redundant nw.from_native(X) round-trips in fit() and transform(). The empty-variables short-circuit in transform() now returns nw_X.to_native() so callers still get a native frame. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in StringSimilarityEncoder tests Replace the file-local data dicts and _to_pandas/_columns helpers with the shared test structure: make_df and data_enc* fixtures, isinstance(X, make_df) plus to_dict() checks, and pytest.raises(match=re.escape(msg)). Tests of pandas-specific NA sentinels and the nullable string dtype stay pandas-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Check missing_values type, group StringSimilarityEncoder init tests, match errors Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Match the fixed get_feature_names_out error message Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor enc dict at the end --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate BaseDiscretiser to narwhals, add polars support Shared base for ArbitraryDiscretiser, EqualFrequencyDiscretiser, EqualWidthDiscretiser and GeometricWidthDiscretiser (not DecisionTreeDiscretiser, which extends a different base). Only transform() needed migrating - _fit_setup(), _get_feature_names_in() and _check_transform_input_and_state() are inherited unchanged from BaseNumericalTransformer, already fully narwhals-migrated. transform()'s only pandas dependency was pd.cut, applied per column to sort values into the bins already fixed by fit() (binner_dict_). Replaced it with a plain numpy implementation: pandas.cut is itself built on bins.searchsorted() internally (verified against pandas 3.0's _bins_to_cuts source), so np.searchsorted + the same include_lowest index-1 special case reproduces its bin-index logic exactly, with no per-backend branch needed - values come from nw_X.get_column(feature).to_numpy() regardless of backend, and results are re-attached via nw.new_series()/with_columns(), so the same code path runs for pandas and polars. Benchmarked old pd.cut vs the new numpy+narwhals path at 10k/50k/100k rows x 1/2/10 columns: - return_boundaries=False (bin codes): narwhals-on-pandas lands at ~1.0-1.2x of pandas-native at realistic sizes (50k-100k rows, the ~1.9x seen only at the smallest 10k-row/1-col case is fixed per-call overhead, sub-millisecond either way) - minimal loss, merged into a single path, no is_pandas split. narwhals-on-polars is ~1.0-1.3x *faster* than pandas-native at every size tested. - return_boundaries=True (interval-label strings): the numpy path is 12-20x faster than pd.cut on pandas itself (e.g. 100k rows x 10 cols: 647ms old vs 40ms new) - pd.cut's Categorical/IntervalIndex machinery has heavy per-call overhead that np.searchsorted plus plain string formatting avoids entirely. polars is ~1.2x faster still than the new pandas path. Given both branches favour or are at parity with a single numpy-driven path, there was no case for a pandas fast-path split here. return_boundaries=True's interval-label formatting ("(lower, upper]" text, e.g. "(-0.001, 20.0]") replicates pandas.cut's _round_frac/_infer_precision/lowest-edge-adjustment algorithm in pure numpy so it works identically on both backends - verified against real pd.cut(...).astype(str) output across positive/negative/duplicate- inducing/inf-edge bins, and against the California housing dataset used in the existing test. return_object=True now builds a nw.Object column (narwhals' cross-backend equivalent of pandas' "O" dtype, already used by variable_handling for categorical-column detection) instead of a pandas-only astype("O") call. Verified: tests/test_discretisation full suite unchanged (109 passed, 5 pre-existing failures in test_check_estimator_discretisers.py - sklearn's check_estimator feeds raw numpy arrays, which check_X() has rejected since the narwhals migration's dataframe-only contract; reproduced identically on the unmodified file). Manually diffed transform() output against real pd.cut() across ~10 edge cases (NaN, out-of-range values on both ends, negative bins, exact-edge values, precision auto-widening, single bin) plus the three sibling discretisers' documented doctest examples (EqualWidthDiscretiser, ArbitraryDiscretiser, EqualFrequencyDiscretiser value_counts()) - all numerically identical to old pd.cut output; the "Name: x" vs "Name: count" and bare-fit()-repr mismatches those doctests already show are a pre-existing pandas-3.0 doc-staleness issue unrelated to this migration (reproduced on the unmodified files too). flake8 and mypy clean. Module imports with pandas blocked (loaded standalone, since sibling discretiser files in this package are not yet migrated and still import pandas at their own module level). sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). test_base_discretizer.py's test_transform is now parametrized over pd.DataFrame/pl.DataFrame per AGENTS.md - its MockClassFit hard-codes binner_dict_ rather than actually fitting, so it needed no pandas-only logic to begin with. The other four discretisers' own test files stay pandas-only for now: their fit() methods still call pd.cut/pd.qcut directly and aren't migrated by this branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add shared discretiser test data fixtures data_california, data_normal_dist, data_vartypes and data_na, shared by the discretiser tests, as fixtures returning plain dicts built with make_df(data). Missing values are written as None. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in BaseDiscretiser tests Build the California housing input from the data_california fixture on the backend under test (instead of converting a pandas frame), and check isinstance(X, make_df) plus to_dict() contents. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate ArbitraryDiscretiser to narwhals, add polars support fit() needed no changes: it already delegates entirely to the already-migrated FitFromDictMixin._fit_from_dict(). The pandas dependency was in transform()'s post-hoc NaN-introduced check, which used X[...].isnull().sum().sum() / .columns / .any() / .tolist() - pandas-only calls that broke outright on polars input coming back from the now-migrated BaseDiscretiser.transform(). Replaced it with a narwhals-based per-column check, branched on return_boundaries rather than dtype: labels (return_boundaries=True) use None for missing values, which narwhals' is_null() detects correctly on both backends. Codes (return_boundaries=False) are numeric, so a numpy float cast + np.isnan is used instead of is_null()/is_nan() directly. That numeric-cast branch isn't just style - narwhals' is_null() (and polars' own null semantics) do NOT see a boxed np.nan sitting inside a polars Object-dtype column (return_object=True's output dtype): verified with a direct repro, is_null().any() returns False on a polars Object series holding all-NaN values, silently swallowing the warning/error this method exists to raise. is_nan() isn't usable there either - narwhals raises "is_nan only supported for numeric dtype, not Object". The numpy-float-cast approach sidesteps both issues and was confirmed to raise/warn correctly across all pandas/polars x return_object x return_boundaries combinations. Benchmarked old (pandas-only) vs new (narwhals) transform() at 10k/50k/100k rows x 1/2/10 cols on pandas input: return_object=False lands at parity (0.9-1.05x, within noise); return_object=True is 1.15-1.3x slower (e.g. 100k rows x 10 cols: 36.2ms old vs 44.9ms new) since the per-variable numpy float-cast replaces one vectorized pandas isnull().sum().sum() call. This falls within the "minimal loss" band used to decide against a pandas/polars split elsewhere in this migration, so a single narwhals-driven path was kept - no is_pandas branch was added. narwhals-on-polars is faster than narwhals-on-pandas at every size tested, consistent with the base branch's own findings. Verified: tests/test_discretisation full suite (114 passed, same 5 pre-existing check_estimator failures as the unmodified base branch - reproduced there too, predates this change). Rewrote test_arbitrary_discretiser.py per AGENTS.md: one parametrized test per behavior over pd.DataFrame/pl.DataFrame (previously pandas-only), switched pytest.raises()/pytest.warns() to the match= form instead of capturing and asserting on the record. flake8 and mypy clean. Module imports with pandas blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Verified the existing docstring/rst examples against real output before touching: the "Name: x" vs "Name: count" and bare-fit()-repr doctest mismatches are the same pre-existing pandas-3.0 doc-staleness noted in the base branch commit (reproduced on the unmodified file too) - left alone, out of scope here. Added a "With polars" example to both the class docstring and ArbitraryDiscretiser.rst, output verified against a real run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in ArbitraryDiscretiser tests Build the California housing input from the data_california fixture on the backend under test, check isinstance(X, make_df) plus to_dict() contents, and use the make_df fixture and pytest.raises(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group init tests and match errors in ArbitraryDiscretiser and BaseDiscretiser tests, check errors type Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate EqualWidthDiscretiser to narwhals, add polars support fit()'s only pandas dependency was pd.cut(bins=int, retbins=True, duplicates="drop"), used purely to compute equal-width bin edges from each variable's min/max (the discretised codes themselves come from transform(), already migrated to numpy searchsorted on the prior base_discretiser branch). Replaced it with _equal_width_edges(): a plain numpy np.linspace(min, max, bins+1), reproducing pandas.cut's own edge computation exactly - verified against pandas 3.0's _nbins_to_bins/_bins_to_cuts source, including the mn==mx 0.1%-range widening for constant columns and the duplicates="drop" collapse for degenerate float edges. fit() now pulls all variables' values in one nw.from_native(X).select(variables_).to_numpy() call (min/max per column via axis=0), instead of one get_column() round-trip per variable, following the pattern already used in CyclicalFeatures.fit(). Benchmarked old pandas-native (pd.cut per column) vs the new narwhals+numpy fit() at 10k/50k/100k rows x 1/2/10 columns: - narwhals-on-pandas is *faster* than the old pd.cut path everywhere except the smallest 10k-row/1-col case (2.58x slower there, but sub-millisecond either way - fixed per-call overhead). At realistic sizes (50k-100k rows) it's 2-6x faster; at 100k rows x 10 cols, 19.3ms (old) vs 3.0ms (new). - narwhals-on-polars is faster still at every size (e.g. 100k x 10: 2.9ms). Given the new path is a speedup rather than a loss on pandas, there was no case for a pandas fast-path split (is_pandas branch) - fit() is a single numpy-driven code path for every backend. Verified binner_dict_ output is numerically identical to the old pd.cut-based fit() across 53 diff cases (random/int/negative values, constant columns at zero/positive/negative, tiny near-duplicate float ranges, two-point and single-value arrays, bins=1) - zero mismatches. Also verified full fit_transform() end-to-end against the class docstring's documented value_counts() output (pre-existing "Name: x" vs "Name: count" pandas-3.0 staleness noted in the base branch is unrelated to this migration) and confirmed the module fit()/transform() round-trip works on polars with pandas import blocked at the interpreter level. tests/test_discretisation/test_equal_width_discretiser.py: converted to one parametrized test per behavior over @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) per AGENTS.md, replacing the pandas-only tests. Also fixed two vacuous assertions in the original numeric-output test (generator expressions that were checking truthiness of an always-empty filtered sequence, so they passed regardless of correctness) with real value comparisons against pd.cut ground truth, and added a dedicated constant-column case exercising the new mn==mx widening branch that pd.cut used to handle internally. docs/user_guide/discretisation/EqualWidthDiscretiser.rst: verified every existing example (binner_dict_, transformed head, dtypes, return_boundaries output) against real output - all matched, no changes needed to those values. Fixed a pre-existing copy-paste bug (predates this migration) where the "Return bin boundaries" code example set up an EqualFrequencyDiscretiser instead of EqualWidthDiscretiser. Updated the "under the hood" description that referenced pandas.cut specifically, and added a "With polars" section with a verified worked example. Verified: tests/test_discretisation full suite - 116 passed, same 5 pre-existing failures as the unmodified baseline (check_estimator feeds raw numpy arrays, rejected by check_X() since the narwhals migration's dataframe-only contract predates this branch). flake8 and mypy clean. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning, confirmed identical on the unmodified baseline). Module imports and runs fit_transform() on polars input with pandas blocked at the builtins.__import__ level. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in EqualWidthDiscretiser tests Build inputs from the data_normal_dist / data_vartypes / data_na fixtures on the backend under test instead of converting pandas frames (which needs pyarrow for polars, so the polars cases failed), check isinstance(X, make_df) plus to_dict() contents, and use pytest.raises(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group init tests and match errors in EqualWidthDiscretiser tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Move _equal_width_edges into EqualWidthDiscretiser as a private method Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…rt (#1039) * Migrate EqualFrequencyDiscretiser.fit() to narwhals, add polars support fit()'s only pandas dependency was pd.qcut(duplicates="drop"), used to compute quantile-based bin edges per variable. Replaced it with np.quantile() on each column's narwhals-extracted numpy array, plus np.unique() to sort and drop duplicate edges - reproducing qcut's duplicates="drop" behaviour without any per-backend branch, since values come from nw_X.get_column(var).to_numpy() regardless of backend. Getting a bit-exact match (not just numerically close) took two fixes verified against pandas 3.0's pandas.core.reshape.tile.qcut source: - pandas masks out NaN before calling np.quantile(values, qs, method="linear") itself, rather than using np.nanquantile - the two are not always bit-identical. Here this distinction is moot in practice: _fit_setup() already rejects NaN in variables_, so no masking is needed - values reaching the loop are already NaN-free. - qcut nudges each quantile that isn't exactly representable in base 2 up via np.nextafter (np.linspace(0, 1, q+1) then np.putmask(quantiles, q*quantiles != np.arange(q+1), nextafter(quantiles, 1))), rounding up rather than to nearest. Skipping this shifted bin edges by ~1e-13 versus real pd.qcut output and broke an existing exact-equality test. With both applied, verified bit-exact (np.array_equal) against real pd.qcut(retbins=True) across large random floats, many-duplicate-value data, all-identical-value data, negative floats, and n<q data. Benchmarked old pd.qcut vs the new numpy+narwhals path at 10k/50k/100k rows x 1/2/10 columns: the new path is consistently faster than the old pandas-native code on BOTH backends (narwhals-on-pandas lands at 0.19x-0.47x of old pd.qcut's time, narwhals-on-polars at 0.12x-0.46x, both converging to roughly 2x faster at realistic 50k-100k row sizes). A narwhals-native quantile-expression alternative was also benchmarked (one nw.col(var).quantile(qi) expr per quantile point, batched into a single select()) - fast on polars but 2-3x *slower* than old pd.qcut on pandas, since narwhals translates each expr to a separate Series.quantile call there. Given the numpy path beats old pandas on both backends, there was no case for a pandas fast-path split. Verified: tests/test_discretisation full suite unchanged (114 passed, 5 pre-existing failures in test_check_estimator_discretisers.py, reproduced identically on the unmodified branch tip - sklearn's check_estimator feeds raw numpy arrays, rejected since the narwhals migration's dataframe-only contract). flake8 and mypy clean. Module imports with pandas blocked (loaded standalone, since sibling discretiser files in this package aren't migrated yet). sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). test_equal_frequency_discretiser.py rewritten per AGENTS.md: each behaviour is now one test parametrized over @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) rather than pandas-only. docs/user_guide/discretisation/EqualFrequencyDiscretiser.rst: verified every code example against real current output. The `disc.binner_dict_` printout had two stale float digits (8099.200000000003 -> ...004, 1601.6000000000001 -> ...004, 1717.6999999999998 -> 1717.7000000000003) - reproduced identically with the OLD pd.qcut-based fit() on the same dataset/pandas version, so this predates the migration and is a doc-staleness issue, not a regression. Also corrected the "uses pandas.qcut() under the hood" line and added a "With polars" section with a verified worked example. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in EqualFrequencyDiscretiser tests Build inputs from the data_normal_dist / data_vartypes / data_na fixtures on the backend under test instead of converting pandas fixtures (which needs pyarrow for polars, so the polars cases failed), check isinstance(X, make_df) plus to_dict() contents, and use pytest.raises(match=re.escape(msg)). The check that every bin code is present was vacuous and now compares the exact set of codes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group init tests and match errors in EqualFrequencyDiscretiser tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…rt (#1041) * Migrate GeometricWidthDiscretiser.fit() to narwhals, add polars support fit()'s only pandas dependency was X[var].min()/.max() to compute the geometric progression's min/max anchors - everything downstream (the np.power/np.r_/np.sort bin-edge math) was already plain numpy and needed no changes. Replaced the pandas indexing with nw.from_native(X, eager_only=True).get_column(var).min()/.max(), which returns a numpy/python float scalar on both backends and feeds np.power identically either way. Benchmarked old pandas-native fit() vs the new narwhals-on-pandas and narwhals-on-polars paths at 10k/50k/100k rows x 1/2/10 columns (200 iterations each, min/max dominate cost either way since bin-edge math is O(bins) not O(n)): - narwhals-on-pandas: 1.0-1.3x of pandas-native at realistic sizes (50k-100k rows); the 1.8x seen only at the smallest 10k-row/1-col case is sub-millisecond fixed per-call overhead. Minimal loss - merged into a single narwhals path, no is_pandas split. - narwhals-on-polars: ~0.35-0.7x of pandas-native (i.e. 1.4-2.8x *faster*), consistent with the sibling BaseDiscretiser.transform() migration finding polars faster at every size tested. Verified: diffed new fit() bin edges against the old pandas implementation across edge cases (skewed/normal/negative-and-positive distributions, two-point range, and the min==max degenerate case) on both backends - numerically identical (exact equality, not just close). Cross-checked full fit_transform() (both return_object and return_boundaries combinations) between pandas and polars inputs - identical output values. Manually reran the GeometricWidthDiscretiser user guide's house_prices worked example (binner_dict_ and interval width numbers) against real output to confirm the docs still match current behaviour (the precision example there was already fixed in #986, prior to this branch) before adding a new "With polars" section with verified output. tests/test_discretisation/test_geometric_width_discretiser.py: the dataframe-touching tests are now parametrized over pd.DataFrame/pl.DataFrame per AGENTS.md, replacing the pandas-only df_normal_dist/df_na/df_vartypes fixtures with local dicts so the same input produces and asserts the same output on both backends (bin edges, transform values via narwhals-agnostic extraction, dtype checks, and NA-error cases). Init-only param-validation tests are unchanged since they never touch a dataframe. flake8 and mypy clean. Module imports with pandas blocked (loaded standalone, since sibling discretiser files in this package aren't migrated yet and still import pandas at their own module level). sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Full tests/test_discretisation suite: 114 passed, same 5 pre-existing failures as the unmodified base branch (test_check_estimator_discretisers.py - sklearn's check_estimator feeds raw numpy arrays, rejected by check_X()'s dataframe-only contract since the narwhals migration; unrelated to this change). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in GeometricWidthDiscretiser tests Replace the file-local _normal_dist_data/_get_column_values/_get_column_dtype helpers with the data_normal_dist fixture, make_df, isinstance(X, make_df) plus to_dict() checks, missing values written as None, and pytest.raises(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Match errors and name init test in GeometricWidthDiscretiser tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate DecisionTreeDiscretiser to narwhals, add polars support DecisionTreeDiscretiser now accepts pandas or polars input via narwhals, extending BaseNumericalTransformer directly (independent of the BaseDiscretiser migration). Never imports pandas; confirmed the module loads with pandas import blocked. Merge (single narwhals codepath, one is_pandas branch only at the final column reassembly) over split (branching at every column-selection call site): benchmarked at 10k/50k/100k rows x 1/2/10 cols, full fit+transform time is dominated by GridSearchCV tree training (10-1500ms) vs plumbing (~0.05-1.4ms per call, <1% of total even where a hand-branched pandas path was ~2x faster on the isolated plumbing microbenchmark). Merge also avoids the one-column-at-a-time write pattern that caused pandas fragmentation warnings in the DecisionTreeFeatures migration. Added optional n_jobs (default None = sequential, unchanged behaviour), parallelizing the per-variable tree fits with joblib threads, mirroring DecisionTreeFeatures. Benchmarked: net loss on small workloads (2 vars, small grid: 0.6-0.8x), real win once there's enough work (2-50 vars with a larger grid: 1.4-2.3x). Verified n_jobs=2 produces identical trees and predictions to n_jobs=None. Bug found and fixed (introduced by the base-transformer narwhals migration, not present pre-migration): check_X used to always copy its pandas input; the narwhals-based check_X no longer does, so the old transform()'s in-place `X[feature] = ...` assignments would have mutated the caller's original dataframe. Rewrote transform() to batch every replacement column and apply them in one non-mutating `.assign()` (pandas) / `.with_columns()` (polars) call instead, which also sidesteps polars' immutability and avoids per-column pandas fragmentation. Reimplemented pandas.cut's binning (bin_number/boundaries outputs) without importing pandas: np.digitize for bin assignment, and a from-scratch port of pandas' internal `_round_frac`/`_infer_precision` label-rounding algorithm (rounds each edge, bumping precision globally if that would collide two edges) so boundary labels are byte-for-byte identical to the old pd.cut output. Verified against pandas.cut directly across 500 randomized threshold/precision/value trials with zero mismatches, in addition to the existing hardcoded-value tests passing unmodified. Tests rewritten to one parametrized test per behavior over make_df in [pd.DataFrame, pl.DataFrame], replacing the pandas-only df_normal_dist/df_discretise fixtures with local data dicts (matching the DecisionTreeFeatures precedent, since those shared fixtures are still pandas-only). Fixed test_non_fitted_error, which was instantiating EqualWidthDiscretiser instead of DecisionTreeDiscretiser (a pre-existing copy-paste bug, confirmed present on main before this migration). tests/test_discretisation full suite: 123 passed (was 108 pre-migration, +15 from parametrization), same 5 pre-existing check_estimator failures (numpy-array input rejected by narwhals check_X, unrelated to this file, confirmed identical on the pre-migration baseline). flake8 and mypy clean. sphinx -W build produces only the pre-existing linkcode_resolve warning (confirmed identical on baseline). Docs: added "With polars" and "Training trees in parallel" sections, verified against real output (network available this session, so the existing fetch_openml house-prices example was re-run and confirmed still accurate). The two `binner_dict_` boundary/bin_number code blocks now display floats as plain numbers as before; current numpy's list repr actually renders them as np.float64(...), a numpy-version-only cosmetic drift present across the whole docs tree and not caused by this migration, left as-is and noted here instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in DecisionTreeDiscretiser tests Replace the file-local _normal_dist_data/_discretise_data/_unique_sorted helpers with the shared test structure: data_normal_dist fixture, y built with make_series on the backend under test, isinstance(X, make_df) plus to_dict() checks, and pytest.raises(match=re.escape(msg)). Add a test passing the target as a list and as a numpy array, which must give the same result as a Series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group init tests and match errors in DecisionTreeDiscretiser tests, check bin_output type Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Move DecisionTreeDiscretiser helper functions into the class as private methods Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Simplify the n_jobs note in the DecisionTreeDiscretiser user guide Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use the scikit-learn wording for n_jobs in DecisionTreeDiscretiser Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Call is_pandas_dataframe in the condition instead of storing it Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Shorten comment in DecisionTreeDiscretiser.transform Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Reorder DecisionTreeDiscretiser tests to the test file convention Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…hDiscretiser (#1058) * Support integer column names in DecisionTreeDiscretiser and EqualWidthDiscretiser Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Keep the pandas fast path in EqualWidthDiscretiser.fit Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate DecisionTreeEncoder to narwhals, add polars support Replaces the old sklearn Pipeline(OrdinalEncoder, DecisionTreeDiscretiser) composition with a direct narwhals-based fit: each variable's categories are ordinal-encoded via a dict built from either a target-mean group_by (encoding_method="ordered") or plain unique-value enumeration ("arbitrary"), a decision tree is trained on the ordinal codes, and predictions are made only on the (few) unique codes rather than the full column, since the tree's output for a category depends only on its code - identical result, far less prediction work for a low-cardinality variable. The "ordered" path sorts by (mean, category) rather than mean alone, matching the tie-break fix applied to the sibling OrdinalEncoder/ MeanEncoder migrations this session, since group_by's own row order isn't guaranteed to match across backends for tied means. Added n_jobs (default None, sequential, unchanged behavior), parallelizing tree training across variables via joblib threads, following the same pattern as DecisionTreeFeatures/DecisionTreeDiscretiser. Verified: 56/56 own tests, full encoding suite 345 passed/17 pre-existing failures (matches the narwhals-encoding-base baseline exactly), flake8 and mypy clean, sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning), no pandas import in this file itself (the package-level import chain still needs pandas only because sibling encoders on this branch aren't migrated yet, expected given the per-encoder parallel-branch strategy). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt DecisionTreeEncoder to narwhals-returning check_X check_X_y now returns a narwhals frame, so bind that to nw_X and keep the original native X for _check_or_select_variables, _check_contains_na and _get_feature_names_in (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the redundant nw.from_native(X) in fit(); the parallel _fit_one_variable calls reuse nw_X from check_X_y. In transform(), bind _check_transform_input_and_state to nw_X, keep native X for _check_contains_na, and pass nw_X to _encode (which now expects narwhals). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in DecisionTreeEncoder tests Replace the file-local _to_backend/_assert_values helpers with the shared test structure: make_df and data_enc* fixtures, y built with make_series on the backend under test, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Add a test passing the target as a list and as a numpy array, which take a different code path than a Series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use add_target_to_X in DecisionTreeEncoder, check encoding_method type, tidy tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Shorten n_jobs docstring in DecisionTreeEncoder Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Build DecisionTreeEncoder on OrdinalEncoder and DecisionTreeDiscretiser Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Test DecisionTreeEncoder with integer column names Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…ion transformers (#1057) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate scaling module (MeanNormalisationScaler) to narwhals, add polars support Redone from scratch off the current narwhals-migration HEAD rather than rebased forward from #979: that branch predates the dataframe_checks rewrite (#989), the variable_handling rewrite (#978), and the creation base rewrite (#990), so the delta had grown too large to carry forward safely for a module this small. fit() replaces the pandas .mean()/.max()/.min() reductions with a single narwhals+numpy path: wrap via nw.from_native, extract the variables as one batched array (nw_X.select(variables_).to_numpy()), reduce with numpy. transform()/inverse_transform() extract each variable as its own 1D array via get_column().to_numpy(), do the elementwise (x - mean) / range (or the inverse) in numpy, and write each back via nw.new_series(same_name, ...) + with_columns() -- same-named series replace the existing column in place, same as polars, rather than adding a new one the way RelativeFeatures/ MathFeatures do for their derived columns. Benchmarked narwhals-expression vs. narwhals+numpy for both fit and transform, at 100/10k/200k rows and 3/20 variables, both backends, before choosing: numpy wins by 2x-73x at small/medium scale on both pandas and polars, and even at 200k rows/polars where narwhals-expr pulls ahead it's only by ~2x, well inside the range this migration has been treating as "not worth a backend split" (CyclicalFeatures/ GeoDistanceFeatures used ~1.7x+ as the bar for splitting; nothing here gets close). One unified path, no pandas/polars branch, matching RelativeFeatures' precedent. return_empty=True guarded explicitly (mean_/range_ default to {} when variables_ is empty) -- narwhals' select([]) collapses row count too, so .to_numpy() on it would reduce over zero rows, not zero columns. Same fix CyclicalFeatures needed for the same reason. Docstring and user-guide numbers were wrong before this PR touched them, found while verifying rather than assumed: the docstring's five example values were literally the raw pre-normalization np.random.seed(42) draws, never the actual transform() output, and the user guide's inverse_transform table showed Age as a bare int (20, 21, ...) when both the pre-migration and post-migration code have always produced float64 there (multiplying by a float range always promotes the dtype, confirmed by running the pre-migration code directly). Fixed both, added a "With polars" section per AGENTS.md's doc-sync rule. Tests rewritten to the single-parametrized-over-both-backends convention (make_df=[pd.DataFrame, pl.DataFrame]) rather than kept pandas-only; all prior coverage preserved, including both class names (MeanNormalisationScaler and the deprecated MeanNormalizationScaler alias) and the deferred-attribute-assignment regression test. Verified: full test suite run twice, once against this branch and once against the unmodified narwhals-migration HEAD (via git stash) -- identical 68 pre-existing, unrelated failures in both runs (none in scaling; confirmed by diffing the two failure lists directly, not just comparing counts), 2273 -> 2287 passed (the +14 is exactly this file's new parametrized test count minus its old one). flake8 and mypy clean. * Use shared backend test fixtures and helpers in MeanNormalisationScaler tests Replace the file-local assert_df_equal/_none_to_nan helpers and parametrize decorators with the shared test structure: make_df fixture, isinstance(X, make_df) plus to_dict() checks (pytest.approx for floats), and pytest.raises(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Match errors, drop init asserts and rename a test in MeanNormalisationScaler tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Soledad Galli <solegalli@protonmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

…ds (#1053) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

…intainers (#1055) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Six methods head their return section `Return` instead of `Returns`. numpydoc only recognises `Returns`, so it warns "Unknown section Return" and drops the content: the documented return value does not appear in the rendered docs at all. The rest of the package uses `Returns` in 97 places, so these six are outliers. `_transform` in base_predictor.py additionally had its whole docstring body indented one space further than the entry beneath it, which produced a second numpydoc warning about the underline length; normalised it to match every other docstring in the file. `DropMissingData.return_na_data` had no return section at all - its return value was documented under Parameters as `X_na`, leaving the actual parameter `X` undocumented. Moved `X_na` to Returns and documented `X` using the same wording as `transform()` directly above it.

#1033) * Migrate BaseOutlier and WinsorizerBase to narwhals, add polars support Shared base for all outlier transformers (ArbitraryOutlierCapper extends BaseOutlier directly; Winsoriser/OutlierTrimmer extend WinsorizerBase): column reorder + NA/Inf checks in _check_transform_input_and_state(), the fold-limit estimation in WinsorizerBase.fit() (gaussian/iqr/mad/ quantiles), and the capping step in BaseOutlier._transform() are now dataframe-agnostic. Capping (np.clip against per-column bounds) was benchmarked three ways at 10k/50k/100k rows x 1/2/10 columns: pandas-native .clip() loop vs. a single narwhals with_columns(nw.col(v).clip(lo, hi) for v in ...) vs. grouping columns by which bound(s) apply and running up to 3 vectorized numpy calls (np.clip/minimum/maximum) via to_numpy()/new_series(), mirroring ReciprocalTransformer's numpy-acceleration pattern. narwhals-generic alone was already close to parity (0.95-1.49x pandas-native - minimal loss, mergeable per the imputation-base precedent), but the numpy-grouped version was faster still: 0.16-0.82x of pandas-native on the homogeneous case (single tail, all columns share the same bound - the common Winsoriser/ OutlierTrimmer case) and 0.42-1.52x on mixed-coverage dicts (the ArbitraryOutlierCapper case, up to 3 groups). Adopted the numpy-grouped version as the single merged code path for both backends. A first numpy attempt used a blanket -inf/inf sentinel for the missing side per column (like RelativeFeatures-style bound arrays) - that's a correctness bug, not just a style choice: mixing an int64 numpy array with a float -inf/inf bound upcasts the whole column to float64 even when the real, present bound is an int (e.g. ArbitraryOutlierCapper's own docstring example, `max_capping_dict=dict(x1=8)`, expects int64 out). Grouping columns into "both bounds" / "right only" / "left only" buckets and calling np.clip/minimum/maximum with only the bounds that actually exist avoids ever introducing an inf, so dtype promotion matches pandas .clip() exactly - verified byte-for-byte against the old pandas-only implementation across all 4 capping methods x 3 tails, plus the int-dtype and mixed-dict-coverage cases. Also found and fixed a real bug introduced while migrating fit(): plain np.mean/np.std/np.quantile/np.median propagate NaN, unlike pandas' mean/std/quantile/median which skip NaN by default. With missing_values="ignore" and NaN present, this silently produced NaN caps instead of the caps computed from non-null data. Fixed by using the nan-aware numpy variants (np.nanmean/nanstd/nanquantile/nanmedian). Caught by tests/test_outliers/test_winsorizer.py::test_transformer_ignores_na_in_df, which predates this migration but exercises exactly this path. variables/feature names can be int or str; passing a plain list to narwhals' .select() only works for string columns, so every .select() call here uses nw.col(*variables) instead - .select(list_of_ints) raises InvalidIntoExprError. Verified: tests/test_outliers full suite - 83 passed, 3 pre-existing failures in test_check_estimator_outliers.py (sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract; identical failure set before and after this change). flake8 and mypy clean on the file. Module imports and runs fit/_transform end-to-end on polars with pandas import fully blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). All 4 capping-method x tail combinations and the Winsoriser/OutlierTrimmer/ArbitraryOutlierCapper docstring examples produce byte-identical output to the pre-migration code (checked exact numeric values and dtypes). Not migrated here (belongs to the 3 follow-on transformer branches): ArbitraryOutlierCapper.fit()/transform(), Winsoriser's add_indicators branch (pd.concat), and OutlierTrimmer.transform() (its own .le/.ge/.loc row-filtering, which doesn't go through BaseOutlier._transform at all) all still import pandas directly. Existing tests in tests/test_outliers were left pandas-only rather than parametrized over polars, since they exercise those still-pandas-only subclasses, not BaseOutlier/ WinsorizerBase directly - parametrizing them now would fail on reasons unrelated to this file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt BaseOutlier and WinsorizerBase to narwhals-returning check_X check_X now returns a narwhals frame (#1019). The outlier base classes rebound X = check_X(X) and then used it as a native frame (X.columns, X[self.feature_names_in_], nw.from_native(X)), which broke every outlier transformer on narwhals-migration. Mirror the imputation and encoding modules instead: - fit(): bind nw_X = check_X(X), keep passing the native X to the variable and NA/inf checks, compute on nw_X, and set feature_names_in_ and n_features_in_ from it. - _check_transform_input_and_state(): return the narwhals frame, reordered to the train set columns. - _transform(): compute on that frame and return the native frame. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add shared outlier test data fixtures data_normal_dist and data_na, shared by the OutlierTrimmer and Winsoriser tests, as fixtures returning plain dicts built with make_df(data). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add tests for the outlier base classes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Give variables without variation infinite caps instead of raising an error Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate Winsoriser/Winsorizer to narwhals, add polars support Removed the module-level `import pandas as pd` and `import numpy as np`; X type hints now use narwhals' IntoDataFrame. WinsorizerBase.fit/transform (shared base) were already migrated on origin/narwhals-outliers-base; this change covers the Winsoriser-specific piece: transform()'s add_indicators path, which compares the capped output against the original input to build per-tail boolean flag columns and previously only worked on pandas. Benchmarked the add_indicators comparison+concat step at 10k/50k/100k rows x 1/2/10 columns: pandas-native (boolean comparison + pd.concat) is up to ~3x faster than the narwhals with_columns equivalent on pandas input, and the loss grows with column count (1 col: narwhals-on-pandas was actually faster; 10 cols: ~2-3x slower). That crosses the "keep pandas fast path" threshold, so transform() splits on `nwd.is_pandas_dataframe`, matching MissingIndicator's precedent for its own indicator-building step: pandas keeps its existing comparison+concat logic (now obtaining the `pd` module via `nw.from_native(...).__native_namespace__()` instead of importing it), and a new narwhals with_columns path (per-column Series comparison, cast to Float64) covers polars and other backends. Preserved the Winsoriser/Winsorizer deprecation exactly as-is: Winsoriser is the current public name (renamed to the British spelling in #967); Winsorizer is a deprecated subclass that raises the same FutureWarning on __init__ and will be removed in 2.1.0. Note this is the reverse of what one might guess from the class names alone. Tests: converted tests/test_outliers/test_winsorizer.py from pandas-only fixtures (df_normal_dist, df_vartypes, df_na) to local dicts parametrized over `make_df` in [pd.DataFrame, pl.DataFrame], asserting identical capping values, indicator columns, and get_feature_names_out() on both backends for the same input. Missing-value dicts use None instead of np.nan in string columns, since polars' DataFrame constructor rejects a float NaN mixed into a string column. A helper filters both pandas' NaN and polars' None representations of a missing value when comparing outputs cross-backend. Docs: verified every doc example in docs/user_guide/outliers/Winsoriser.rst against actual output (network access to fetch_openml's house_prices dataset was available; outputs matched exactly, no changes needed) and added a "With polars" section covering add_indicators, matching the pattern used in other migrated user guides. Added a verified "With polars" example to the class docstring. Verified: tests/test_outliers/test_winsorizer.py 93 passed. Full tests/test_outliers suite: 123 passed / 3 pre-existing failures in test_check_estimator_outliers.py (confirmed identical against a baseline run of origin/narwhals-outliers-base: 83 passed / same 3 failures - sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract; predates this change). flake8 and mypy clean. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning, confirmed present on the base branch too). Confirmed winsorizer.py and base_outlier.py import successfully and a full polars fit_transform (including add_indicators) runs correctly with pandas' own import blocked at the builtins level. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt Winsoriser indicators to narwhals-returning check_X With add_indicators=True, transform() compared the capped output against check_X(X), which is now a narwhals frame, so the pandas path mixed pandas and narwhals objects (broadcast errors, wrong indicators). Compare against the user's native X instead; _transform() already validates it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in Winsoriser tests Replace the file-local data dicts and _col/_cols/_shape/_drop_missing helpers with the shared test structure: make_df and data_normal_dist / data_na fixtures, isinstance(X, make_df) plus to_dict() checks, and pytest.raises(match=...). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Call is_pandas_dataframe in the condition instead of storing it in Winsorizer Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Document and test infinite caps for variables without variation Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Align Winsoriser and its tests with the repo conventions Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…orm checks (#1060) _fit_setup, _fit_from_dict and _check_transform_input_and_state now return the narwhals frame from check_X, so the numerical transformers no longer wrap X a second time. Also fixes pandas integer column names in the transformation transformers, CyclicalFeatures and MeanNormalisationScaler, and the inf check on polars when there are no variables. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate ArbitraryOutlierCapper to narwhals, add polars support fit() only builds dicts from user input and validates variables/dtypes via check_numerical_variables (already narwhals-generic) - no numeric computation, so nothing to branch on there. The only pandas-specific lines were the feature_names_in_ assignment (X.columns.to_list(), a pandas-Index method), replaced with the same is_pandas-guarded pattern WinsorizerBase.fit() already uses (list(X.columns) for pandas, nw.from_native(X).columns - already list[str] - otherwise). transform() was already dataframe-agnostic via BaseOutlier._transform(); only its type hints changed (pd.DataFrame -> IntoDataFrame). Benchmarked fit+transform end-to-end at 10k/50k/100k rows x 1/2/10 columns: pandas-native (pre-migration) vs the migrated code on pandas were within noise of each other (~0.9-1.1x), and polars ran 2-4x faster than pandas on both. No pandas/polars branch needed - merged single path, consistent with the is_pandas-only-for-.columns precedent already set in WinsorizerBase. Confirmed the module needs zero pandas: reloaded artbitrary.py in isolation with sys.modules["pandas"] = None (simulating an uninstalled pandas) and ran fit/transform end-to-end on a polars frame - works, and int64 stays int64 for a same-dtype capping dict (the class docstring's own x1 example). Found, while doing so, a real dtype-preservation bug in the already- merged BaseOutlier._transform() (base_outlier.py, commit 71bf7cf on this branch's base) that predates this migration and is not introduced here: when a capping-dict spans columns of different dtypes that land in the same bound-group (e.g. max_capping_dict={"age": 50, "fare": 200} with age int64 and fare float64 - both "right_only"), the group's columns are stacked into one 2D array via to_numpy() before np.clip, which forces a common dtype and upcasts age to float64. The pre- narwhals code (verified against 71bf7cf^) clipped each column independently (X[feature] = X[feature].clip(...)), so int columns never picked up a neighboring float column's dtype. Confirmed this reproduces identically on both pandas and polars (same merged code path) and is untouched by this commit - it lives in base_outlier.py, shared with Winsoriser/OutlierTrimmer, out of this file's scope. Flagged separately rather than fixed here. Rewrote test_arbitrary_capper.py to one parametrized test per behavior over pd.DataFrame/pl.DataFrame (previously pandas-only), using nw.from_native(...).to_dict(as_series=False) for backend-agnostic assertions in place of pd.testing.assert_frame_equal, following the same pattern used for ReciprocalTransformer/ArcsinTransformer. Added a verified "With polars" section to the docs (float dtypes throughout, to sidestep the dtype-upcast issue above rather than put an unexplained surprise in a user-facing example); left the pre-existing pandas Titanic walkthrough untouched - no network access in this environment to re-verify the fetch_openml/CSV-backed output. Verified: tests/test_outliers full suite - 88 passed (up from 83, all 5 new instances are the added polars parametrizations), same 3 pre-existing check_estimator failures as the pre-migration baseline (numpy-array input, unrelated to this change). flake8 and mypy clean. sphinx -W build clean (only the pre-existing linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt ArbitraryOutlierCapper to narwhals-returning check_X Bind the narwhals frame returned by check_X and set feature_names_in_ and n_features_in_ from it, instead of treating the check_X result as a native frame, mirroring the imputation and encoding modules. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in ArbitraryOutlierCapper tests Replace the file-local _to_dict helper and parametrize decorators with the shared test structure: make_df fixture, isinstance(X, make_df) plus to_dict() checks, missing values written as None, and pytest.raises(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Align ArbitraryOutlierCapper and its tests with the repo conventions Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Give the expected caps explicitly in the ArbitraryOutlierCapper capping test Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate OutlierTrimmer to narwhals, add polars support transform() now filters rows via a single narwhals .filter() call built from a combined boolean expression (AND of each variable's right/left cap conditions), instead of a pandas .loc masking loop. Benchmarked against pandas-native and a numpy boolean-mask extraction at 10k/50k/ 100k rows x 1/2/10 columns: the narwhals filter is within 1.4-1.75x of pandas-native at 10k rows (sub-millisecond absolute difference) and becomes faster than pandas-native from 50k rows up (0.58x-0.97x), so a single merged code path (no is_pandas branching) is the right call here - unlike BaseOutlier's elementwise capping, which benefits from numpy grouping, row-filtering is exactly what narwhals .filter() already pushes down to the native backend efficiently. Also fixes a latent bug in TransformXyMixin.transform_x_y() (_base_transformers/mixins.py): the non-pandas branch added a row-index marker column via with_row_index() and passed it straight to self.transform(), but never widened feature_names_in_/ n_features_in_ to account for it. Any transform() that validates column count (BaseOutlier._check_transform_input_and_state, via _check_X_matches_training_df) then raised a ValueError on the extra column. This was latent because no narwhals-migrated class on this branch previously combined TransformXyMixin with a column-count- checking transform() on a non-pandas backend - OutlierTrimmer is the first. The fix (guarded widen/restore of feature_names_in_ around the transform() call) is carried over verbatim from the same fix already applied to this file on branch narwhals-drop-missing-data (commit fd99caf), which hadn't been merged into this branch yet. Tests rewritten to one parametrized test per behavior over make_df in [pd.DataFrame, pl.DataFrame], plus a new test asserting that caps on two different variables combine with AND (each variable drops a distinct row) - a code path the old sequential-loop version exercised implicitly but no test isolated directly. Docs verified against live output: the class docstring's pandas examples were already accurate; the user guide's Titanic-based numbers had drifted from the current openml dataset (predates this migration, e.g. the IQR section's age max was already wrong against the old pandas-loop transform()) and are corrected here, plus a "With polars" section is added. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt OutlierTrimmer to narwhals-returning check_X _check_transform_input_and_state() now returns the narwhals frame, so filter it directly instead of re-wrapping it with nw.from_native(). TransformXyMixin.transform_x_y() (rebased onto #1024) widens n_features_in_ for the row-index tag column; also widen feature_names_in_, since BaseOutlier reorders X to feature_names_in_ and would otherwise drop the tag column on non-pandas backends. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in OutlierTrimmer tests Replace the file-local data dicts and _cols/_to_list/_make_series helpers with the shared test structure: make_df and data_normal_dist / data_na fixtures, y built with make_series, isinstance(X, make_df) plus to_dict() checks, and pytest.raises(match=...). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Document and test infinite caps for variables without variation Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Align OutlierTrimmer and its tests with the repo conventions Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Keep transform_x_y row alignment in OutlierTrimmer instead of the shared mixin Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

# Conflicts: # tests/test_creation/test_check_estimator_creation.py # tests/test_datetime/test_datetime_features.py # tests/test_discretisation/test_check_estimator_discretisers.py # tests/test_encoding/test_check_estimator_encoders.py # tests/test_imputation/test_check_estimator_imputers.py # tests/test_outliers/test_check_estimator_outliers.py # tests/test_preprocessing/test_check_estimator_preprocessing.py # tests/test_selection/test_check_estimator_selectors.py # tests/test_time_series/test_forecasting/test_check_estimator_forecasting.py # tests/test_transformation/test_check_estimator_transformers.py # tests/test_wrappers/test_check_estimator_wrappers.py

Merge main into narwhals-migration

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate MatchVariables to narwhals, add polars support fit() and transform() take any narwhals-supported dataframe and return the same backend. pandas adds, drops and reorders the columns with a single reindex (faster than the previous drop/setitem/select and than narwhals at 10k-500k rows); other backends use with_columns + select. With match_dtypes, pandas keeps its dtypes and astype, since narwhals dtypes don't hold the categories of pandas categoricals. Other backends store narwhals dtypes and cast with narwhals, turning values outside Enum categories into nulls and parsing strings into dates, as pandas does. With polars, np.nan fill values add null Float64 columns and integer fill values add Int64 columns. The verbose messages list the variables in a fixed order: training order for added variables, input order for dropped ones. Init error messages now end with "Got {param} instead.". Rewrite the tests to the make_df conventions and add a polars example to the docstring and the user guide, refreshing its outputs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Keep the dtypes learned by MatchVariables private and allow missing data in integer and boolean variables Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Mark polars output blocks in the user guide as text Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate MatchCategories to narwhals, add polars support MatchCategories now accepts pandas and polars dataframes. With pandas it keeps casting to the category dtype; with polars it casts to Enum with the categories learned in fit. Unseen categories become missing values in both. Also fixes the warning and error message when several integer-named pandas columns get missing values (it raised a TypeError). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Mark polars output blocks in the user guide as text Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Remove the transformer printout after fit() from the MatchCategories docstring Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate TextFeatures to narwhals, add polars support TextFeatures now accepts pandas, polars and other dataframes supported by narwhals, and returns the same type it receives. All features are defined once in terms of a few text statistics, computed with pandas string methods and Python loops for pandas, polars string methods for polars, and narwhals expressions for other backends. Pandas outputs are identical to before and the transform is about 3x faster. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Exclude spaces from avg_word_length and use the shared get_feature_names_out in TextFeatures Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

This was referenced Jul 26, 2026

Request to use polars dataframe package directly #809

Open

add missing_only=True to all imputers to use in combination with variables=None #974

Closed

ojassharma7 mentioned this pull request Jul 29, 2026

[MNT] migrate scaling module to narwhals #979

Closed

2 tasks

solegalli force-pushed the narwhals-migration branch 3 times, most recently from 8fe8359 to ea95750 Compare July 31, 2026 12:30

FBruzzesi reviewed Aug 23, 2026

View reviewed changes

solegalli mentioned this pull request Aug 24, 2026

address review feedback on dataframe_checks.py #989

Merged

solegalli and others added 6 commits August 24, 2026 12:12

start narwhals migration

fbbf160

fix doc test

7f33552

solegalli force-pushed the narwhals-migration branch from 0ef7acb to ff5d212 Compare August 24, 2026 10:14

solegalli and others added 6 commits August 24, 2026 17:00

FBruzzesi mentioned this pull request Aug 24, 2026

perf: Speedup pandas-like is_finite, sqrt, kurtosis and skew narwhals-dev/narwhals#3874

Merged

7 tasks

solegalli and others added 3 commits August 25, 2026 20:20

solegalli and others added 30 commits September 15, 2026 18:45

Shorten n_jobs docstring in DecisionTreeFeatures (#1049)

e2e3ff3

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Note typical data sizes per backend for benchmarks in AGENTS.md (#1052)

9a875cf

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Call is_pandas_dataframe in conditions instead of storing it in creat…

491f945

…ion transformers (#1057) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Move discretiser helper functions into their classes as private metho…

4241811

…ds (#1053) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Note in AGENTS.md that user-facing docs are written for users, not ma…

62cbea8

…intainers (#1055) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Note in AGENTS.md to call boolean checks directly in conditions (#1056)

be430f6

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Merge pull request #1062 from feature-engine/merge-main-into-narwhals

7c22dfa

Merge main into narwhals-migration

Speed up the missing-value check (#1072)

b0ec8b5

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

	if not np.isfinite(nw_y.to_numpy()).all():
	if not nw_y.is_finite().all():

	if nw_y.select(nw.all().is_null().any()).to_numpy().any():
	if nw_y.select(nw.any_horizontal(nw.all().is_null().any())).item():

Conversation

solegalli commented Jul 21, 2026

Uh oh!

ojassharma7 commented Jul 28, 2026

Uh oh!

solegalli commented Jul 29, 2026

Uh oh!

ojassharma7 commented Jul 29, 2026

Uh oh!

ojassharma7 commented Jul 29, 2026

Uh oh!

FBruzzesi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants