Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advanced transformation examples #3433

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions docs/src/man/working_with_dataframes.md
Original file line number Diff line number Diff line change
Expand Up @@ -830,6 +830,72 @@ julia> combine(df, names(df) .=> sum, names(df) .=> prod)
If you would prefer the result to have the same number of rows as the source
data frame, use `select` instead of `combine`.

Note that a `DataFrame` can store values of any type as its columns, for example
below we show how one can store a `Tuple`:

```
julia> df2 = combine(df, All() .=> extrema)
1×2 DataFrame
Row │ A_extrema B_extrema
│ Tuple… Tuple…
─────┼───────────────────────
1 │ (1, 4) (1.0, 4.0)
```

Later you might want to expand the tuples into separate columns storing the computed
minima and maxima. This can be achieved by passing multiple columns for the output.
In the example below we show how this can be done in combination with a function
so that we can generate target column names conditional on source column names:
bkamins marked this conversation as resolved.
Show resolved Hide resolved

```
julia> combine(df2, All() .=> identity .=> [c -> first(c) .* ["_min", "_max"]])
bkamins marked this conversation as resolved.
Show resolved Hide resolved
1×4 DataFrame
Row │ A_min A_max B_min B_max
│ Int64 Int64 Float64 Float64
─────┼────────────────────────────────
1 │ 1 4 1.0 4.0
```

Note that in this example we needed to pass `identity` explicitly as otherwise the
functions generated with `c -> first(c) .* ["_min", "_max"]` would be treated as transformations
and not as rules for target column names generation.
bkamins marked this conversation as resolved.
Show resolved Hide resolved

You might want to perform the transformation of the source data frame into the result
we have just shown in one step. This can be achieved with the following expression:

```
julia> combine(df, All() .=> Ref∘extrema .=> [c -> c .* ["_min", "_max"]])
bkamins marked this conversation as resolved.
Show resolved Hide resolved
1×4 DataFrame
Row │ A_min A_max B_min B_max
│ Int64 Int64 Float64 Float64
─────┼────────────────────────────────
1 │ 1 4 1.0 4.0
```

Note that in this case we needed to add a `Ref` call in the `Ref∘extrema` operation specification.
The reason why this is needed is that instead `combine` iterates the contents of the value returned
by the operation specification function and tries to expand it, which in our case is a tuple of numbers,
so one gets an error:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The reason why this is needed is that instead `combine` iterates the contents of the value returned
by the operation specification function and tries to expand it, which in our case is a tuple of numbers,
so one gets an error:
Without `Ref`, `combine` iterates the contents of the value returned by the operation specification function, which in our case is a tuple of numbers, and tries to expand it, so one gets an error:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remind me why this isn't allowed? AFAICT it would be possible to destructure the tuple and use its values to fill the columns? BTW, is "expand" the right word here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason is related to the issue we recently discussed with the Tables.jl conflict against Julia 1.11. DataFrames.jl does not recognize Tuple as a valid input so it sends it to Tables.jl columntable function. So essentially - the error is because we fall-back to Tables.jl in cases that we do not explicitly handle as special cases (like vectors, matrices, DataFrames etc.).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your suggestion is applied in the commit I pushed (I just had to re-word it a bit).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks. But then, when combine iterates, it expects each value to represent a column, not a row, and my suggestion is incorrect, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just have written something more precise as your suggestion was not precise enough (I think). What is done is I think best visible here:

julia> df = DataFrame(x=1:2)
2×1 DataFrame
 Row │ x
     │ Int64
─────┼───────
   1 │     1
   2 │     2

julia> combine(df, :x => (x -> ((a=x,), (a=2x,), (a=3x,))) => AsTable)
3×1 DataFrame
 Row │ a
     │ Array…
─────┼────────
   1 │ [1, 2]
   2 │ [2, 4]
   3 │ [3, 6]

The returned tuple ((a=x,), (a=2x,), (a=3x,)) has three elements so it produces three rows. Things work, because each element is a NamedTuple which provides column names that can be used to generate column names with AsTable.

Is it clearer now?


```
julia> combine(df, names(df) .=> extrema .=> [c -> c .* ["_min", "_max"]])
bkamins marked this conversation as resolved.
Show resolved Hide resolved
ERROR: ArgumentError: 'Tuple{Int64, Int64}' iterates 'Int64' values,
which doesn't satisfy the Tables.jl `AbstractRow` interface
```

Note that we used `Ref` as it is a container that is typically used in DataFrames.jl when one
wants to store one value, however, in general it could be another iterator. Here is an example
when the tuple returned by `extrema` is wrapped in a `Tuple`, producing the same result:

```
julia> combine(df, names(df) .=> tuple∘extrema .=> [c -> c .* ["_min", "_max"]])
1×4 DataFrame
Row │ A_min A_max B_min B_max
│ Int64 Int64 Float64 Float64
─────┼────────────────────────────────
1 │ 1 4 1.0 4.0
```
bkamins marked this conversation as resolved.
Show resolved Hide resolved

## Handling of Columns Stored in a `DataFrame`

Functions that transform a `DataFrame` to produce a
Expand Down
Loading