Is your feature request related to a problem or challenge?
OptimizeProjections recomputes projection schemas more than necessary.
In rewrite_projection_given_requirements (datafusion/optimizer/src/optimize_projections/mod.rs), a projection that has been pruned to its required columns is rebuilt with Projection::try_new(exprs_used, input). try_new recomputes the schema via projection_schema → Expr::to_field for every expression, and column resolution (DFSchema::field_from_column) is an O(M) linear scan over the input schema (there is no name→index map). So schema construction is O(exprs × schema_width) per projection, per optimizer pass.
This is most visible on plans with many wide col AS col alias projections: an alias is not a bare Column, so is_projection_unnecessary returns false and the projection is kept, its schema recomputed on every pass — even though the result schema is just a column subset of the input schema that is already available.
Describe the solution you'd like
Build the pruned schema by slicing the existing input schema at the (sorted, deduped) required indices and pass it via Projection::try_new_with_schema, instead of recomputing it from scratch. This mirrors the schema reuse already done in merge_consecutive_projections. It reduces the per-projection cost from O(exprs × width) to O(k) and is behavior-preserving.
Describe alternatives you've considered
Keeping try_new but adding a name→index map to DFSchema would also help, but is a larger change; slicing the already-available schema is local and sufficient here.
Additional context
A separate, tiny improvement in the same area: in datafusion/expr/src/expr_schema.rs, Expr::to_field for the Expr::Alias branch resolves the inner expression twice — once via expr.metadata(schema)? (which internally calls to_field) and again via expr.to_field(schema). These can be collapsed into a single to_field call, extracting both the field and its metadata from it.
I have a local patch for the first item (with a unit test) and am happy to open a PR.
Is your feature request related to a problem or challenge?
OptimizeProjectionsrecomputes projection schemas more than necessary.In
rewrite_projection_given_requirements(datafusion/optimizer/src/optimize_projections/mod.rs), a projection that has been pruned to its required columns is rebuilt withProjection::try_new(exprs_used, input).try_newrecomputes the schema viaprojection_schema→Expr::to_fieldfor every expression, and column resolution (DFSchema::field_from_column) is an O(M) linear scan over the input schema (there is no name→index map). So schema construction is O(exprs × schema_width) per projection, per optimizer pass.This is most visible on plans with many wide
col AS colalias projections: an alias is not a bareColumn, sois_projection_unnecessaryreturns false and the projection is kept, its schema recomputed on every pass — even though the result schema is just a column subset of the input schema that is already available.Describe the solution you'd like
Build the pruned schema by slicing the existing input schema at the (sorted, deduped) required indices and pass it via
Projection::try_new_with_schema, instead of recomputing it from scratch. This mirrors the schema reuse already done inmerge_consecutive_projections. It reduces the per-projection cost from O(exprs × width) to O(k) and is behavior-preserving.Describe alternatives you've considered
Keeping
try_newbut adding a name→index map toDFSchemawould also help, but is a larger change; slicing the already-available schema is local and sufficient here.Additional context
A separate, tiny improvement in the same area: in
datafusion/expr/src/expr_schema.rs,Expr::to_fieldfor theExpr::Aliasbranch resolves the inner expression twice — once viaexpr.metadata(schema)?(which internally callsto_field) and again viaexpr.to_field(schema). These can be collapsed into a singleto_fieldcall, extracting both the field and its metadata from it.I have a local patch for the first item (with a unit test) and am happy to open a PR.