Skip to content

feat: allow collection builtins to iterate maps - #988

Open
sawy3r wants to merge 1 commit into
expr-lang:masterfrom
sawy3r:map-collection-builtins
Open

sawy3r wants to merge 1 commit into
expr-lang:masterfrom
sawy3r:map-collection-builtins

Conversation

@sawy3r

@sawy3r sawy3r commented Sep 17, 2026

Copy link
Copy Markdown

What

all, any, none, one, count, sum, filter, map, find, findLast and reduce used to accept only arrays, and min, max, mean and median rejected map arguments too. That made groupBy() a dead end: it returns a map, and nothing downstream could consume it, so a pipeline like

events | groupBy(.Author) | filter(len(#) > 1)

couldn't be written — it failed at compile time with a typed environment, or with a reflection error at runtime without one.

These builtins now also accept a map with string keys (including a map declared with a loose key type that holds string keys at evaluation time, such as the ones groupBy() produces), and iterate its entries in ascending key order, comparing keys as strings byte by byte.

Semantics

  • Inside a predicate, # holds the entry's value and #index holds its key. #index was previously only available in map and reduce; it's now available in all, any, none, one, count, sum, find and findLast too. For arrays, #index keeps holding the element's position, so the two collection kinds stay symmetric.
  • filter applied to a map returns a map holding the entries whose predicate holds; map returns a map with the same keys and transformed values. Both compose, and the result keeps behaving like a map downstream — indexing, len, keys, sum, the in operator.
  • The remaining builtins return the same shape they return for arrays: all/any/none/one report on the values, count/sum work as they do for arrays, find/findLast return the first/last matching value in key order, and reduce folds the values in key order.
  • An empty or nil map behaves the way an empty array does.
  • min, max, mean and median aggregate a map's values, and a map argument mixes freely with the array and scalar arguments they already take.
  • findIndex, findLastIndex, groupBy and sortBy keep rejecting maps — positions and reordering don't translate to map entries.
  • Maps with non-string keys keep being rejected: at compile time when the environment types make that visible, otherwise at runtime.
  • This works whether or not the expression is compiled against a typed environment. With one, the checker accepts the new forms and carries sensible types through them — the key type for #index, the value type for #, and a map type for filter/map results.
  • Array behavior is unchanged everywhere.

Examples

prices | filter(# > 2)                       // map with entries > 2
prices | map(# * 10)                         // map, same keys, doubled... x10 values
events | groupBy(.Author) | filter(len(#) > 1) // the motivating example from #575
tags | reduce(#acc + #index, "")             // fold over keys in order

Implementation

  • vm: a loop Scope can now iterate a map's sorted keys instead of an array; three new opcodes (OpPointerIndex, OpMarkKey, OpCollect) replace the array-only OpGetIndex/OpArray pair so map/filter can build either an array or a map result depending on what they're iterating.
  • checker: isIterable/indexNature helpers let the predicate builtins accept arrays or string-keyed maps and give #index the right type; filter/map carry a map type through when the input is a map.
  • builtin: min/max/mean/median unwrap a map argument into its values before aggregating (a small reflection-based helper handles both map[string]any and other map value types), rejecting non-string keys.

Tests

  • test/issues/575/issue_test.go — end-to-end coverage: filter/map over maps, all/any/none/one/count/sum, find/findLast and reduce in key order, #index as key vs. position, empty/nil maps, nested map+array predicates, composition, the groupBy pipeline from the issue, min/max/mean/median, behavior with and without a typed environment, and that non-string-keyed maps and the positional builtins (findIndex, findLastIndex, groupBy, sortBy) still reject maps.
  • A handful of map cases folded into builtin.TestBuiltin (untyped) and checker.TestCheck_types (typed, including a type-flow check that # really is typed as the map's value type and not any).
  • go test ./..., go test -race ., go test -tags=expr_debug -run=TestDebugger -v ./vm, gofmt -l . and go vet ./... all pass.

Closes #575.

🤖 Generated with Claude Code

all, any, none, one, count, sum, filter, map, find, findLast and reduce
previously accepted only arrays, and min, max, mean and median rejected
map arguments too. That made groupBy() a dead end: it returns a map, and
nothing downstream could consume it, so a pipeline like

  events | groupBy(.Author) | filter(len(#) > 1)

could not be written.

These builtins now also accept a map with string keys (including a map
declared with a loose key type that holds string keys at evaluation
time, such as the ones groupBy produces), and iterate its entries in
ascending key order, comparing keys as strings byte by byte. Inside a
predicate, # holds the entry's value and #index holds its key; #index
was previously only available in map and reduce, and now becomes
available in all, any, none, one, count, sum, find and findLast too.
For arrays #index keeps holding the element's position, so the two
collection kinds stay symmetric.

filter applied to a map returns a map holding the entries whose
predicate holds, and map returns a map with the same keys and
transformed values, so both compose and their results keep behaving
like a map downstream (indexing, len, keys, sum, the in operator).
The other builtins return the same shape they return for arrays: all,
any, none and one report on the values, count and sum work as they do
for arrays, find and findLast return the first and last matching value
in key order, and reduce folds the values in key order. An empty or
nil map behaves the way an empty array does. min, max, mean and median
aggregate a map's values and mix freely with array and scalar
arguments.

findIndex, findLastIndex, groupBy and sortBy keep rejecting maps,
since positions and reordering don't translate to map entries. Maps
with non-string keys keep being rejected, at compile time when the
environment types make that visible, otherwise at runtime.

This works whether or not the expression is compiled against a typed
environment; with one, the type checker accepts the new forms and
carries sensible types through them (the key type for #index, the
value type for #, and a map for filter/map results). Array behavior is
unchanged.

Adds test/issues/575/issue_test.go covering the new behavior end to
end, plus a handful of cases folded into the existing builtin and
checker test tables, and documents map support in the affected
builtins.

Closes expr-lang#575.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow map() and filter() builtins on map values

1 participant