Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Another attempt at an astable flag #298

Merged
merged 29 commits into from
Sep 24, 2021
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
a8701c8
initial attempt
pdeffebach Sep 14, 2021
9b997a6
finally working
pdeffebach Sep 15, 2021
d639560
start adding tests
pdeffebach Sep 15, 2021
b77e8ca
more tests
pdeffebach Sep 16, 2021
3cdf0d5
more tests
pdeffebach Sep 16, 2021
b878fbb
add docstring
pdeffebach Sep 16, 2021
2344a2e
tests pass
pdeffebach Sep 16, 2021
6557def
add ByRow in docstring
pdeffebach Sep 16, 2021
6002def
add type annotation
pdeffebach Sep 21, 2021
08a1c4b
better docs
pdeffebach Sep 21, 2021
581b2cf
more docs fixes
pdeffebach Sep 21, 2021
7cc8947
update index.md
pdeffebach Sep 21, 2021
0eca67d
Apply suggestions from code review
pdeffebach Sep 21, 2021
a4ab9a6
Merge branch 'astable_2' of https://github.com/pdeffebach/DataFramesM…
pdeffebach Sep 21, 2021
ab9bae4
clean named tuple creation
pdeffebach Sep 22, 2021
495f08a
add example with string
pdeffebach Sep 22, 2021
01cb5e7
grouping tests
pdeffebach Sep 22, 2021
01fb3b7
Update src/macros.jl
pdeffebach Sep 22, 2021
915191c
changes
pdeffebach Sep 23, 2021
a331fc2
Merge branch 'astable_2' of https://github.com/pdeffebach/DataFramesM…
pdeffebach Sep 23, 2021
2ce4d9e
fix some errors
pdeffebach Sep 23, 2021
57b4051
add macro check
pdeffebach Sep 23, 2021
da7674d
add errors for bad flag combo
pdeffebach Sep 23, 2021
285e3ac
better grouping tests
pdeffebach Sep 23, 2021
713eaf0
Update src/parsing_astable.jl
pdeffebach Sep 23, 2021
4e01c4a
add snipper to transform, select, combine, by
pdeffebach Sep 23, 2021
09c692a
add mutating tests
pdeffebach Sep 23, 2021
ae26da8
get rid of debugging printin
pdeffebach Sep 24, 2021
a7fd1a2
Apply suggestions from code review
pdeffebach Sep 24, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 42 additions & 13 deletions src/macros.jl
Original file line number Diff line number Diff line change
Expand Up @@ -282,11 +282,10 @@ macro byrow(args...)
throw(ArgumentError("@byrow is deprecated outside of DataFramesMeta macros."))
end


"""
@passmissing(args...)

Propograte missing values inside DataFramesMeta.jl macros.
Propagrate missing values inside DataFramesMeta.jl macros.
pdeffebach marked this conversation as resolved.
Show resolved Hide resolved


`@passmissing` is not a "real" Julia macro but rather serves as a "flag"
Expand Down Expand Up @@ -350,18 +349,24 @@ macro passmissing(args...)
throw(ArgumentError("@passmissing only works inside DataFramesMeta macros."))
end

global astable_docstring_snippet = """
pdeffebach marked this conversation as resolved.
Show resolved Hide resolved
Transformations can also use the macro-flag `@astable` for creating multiple
pdeffebach marked this conversation as resolved.
Show resolved Hide resolved
new columns at once and letting transformations share the same name-space.
See `? @astable` for more details.
"""

"""
pdeffebach marked this conversation as resolved.
Show resolved Hide resolved
@astable(args...)

Return a `NamedTuple` from a single transformation inside the DataFramesMeta.jl
macros, `@select`, `@transform`, and their mutating and row-wise equivalents.

`@astable` acts on a single block. It works through all top-level expressions
and collects all such expressions of the form `:y = ...`, i.e. assignments to a
`Symbol`, which is a syntax error outside of DataFramesMeta.jl macros. At the end of the
expression, all assignments are collected into a `NamedTuple` to be used
with the `AsTable` destination in the DataFrames.jl transformation
mini-language.
and collects all such expressions of the form `:y = ...` or `$(DOLLAR)y = ...`, i.e. assignments to a
`Symbol` or an escaped column identifier, which is a syntax error outside of
DataFramesMeta.jl macros. At the end of the expression, all assignments are collected
into a `NamedTuple` to be used with the `AsTable` destination in the DataFrames.jl
transformation mini-language.

Concretely, the expressions

Expand Down Expand Up @@ -423,12 +428,22 @@ transformations. `@astable` solves this problem
:new_col_2 = :new_col_1 + :z
end

Column assignment in `@astable` follows the same rules as
column assignment more generally. Construct a new column
from a string by escaping it with `$DOLLAR`, which can be a
`Symbol` or an `AbstractString`. References to existing
columns may be a `Symbol`, `AbstractString`, or an
integer.
Column assignment in `@astable` follows similar rules as
column assignment in other DataFramesMeta.jl macros. The left-
-hand-side of a column assignment can be either a `Symbol` or any
expression which evaluates to a `Symbol` or `AbstractString`. For example
`:y = ...`, and `$(DOLLAR)y = ...` are both valid ways of assigning a new column.
However unlike other DataFramesMeta.jl macros, multi-column assignments via
`AsTable` are disallowed. The following will fail.

```
@transform df @astable begin
$AsTable = :x
end
```

References to existing columns also follow the same
rules as other DataFramesMeta.jl macros.

### Examples

Expand Down Expand Up @@ -462,6 +477,8 @@ julia> @by df :a @astable begin
1 │ 1 5 6
2 │ 2 70 80

julia> new_col = "New Column";

julia> @rtransform df @astable begin
f_a = first(:a)
$(DOLLAR)new_col = :a + :b + f_a
bkamins marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -1229,6 +1246,8 @@ transformations by row, `@transform` allows `@byrow` at the
beginning of a block of transformations (i.e. `@byrow begin... end`).
All transformations in the block will operate by row.

$astable_docstring_snippet

### Examples

```jldoctest
Expand Down Expand Up @@ -1365,6 +1384,8 @@ transform!ations by row, `@transform!` allows `@byrow` at the
beginning of a block of transform!ations (i.e. `@byrow begin... end`).
All transform!ations in the block will operate by row.

$astable_docstring_snippet

### Examples

```jldoctest
Expand Down Expand Up @@ -1477,6 +1498,8 @@ transformations by row, `@select` allows `@byrow` at the
beginning of a block of selectations (i.e. `@byrow begin... end`).
All transformations in the block will operate by row.

$astable_docstring_snippet

### Examples

```jldoctest
Expand Down Expand Up @@ -1597,6 +1620,8 @@ transformations by row, `@select!` allows `@byrow` at the
beginning of a block of select!ations (i.e. `@byrow begin... end`).
All transformations in the block will operate by row.

$astable_docstring_snippet

### Examples

```jldoctest
Expand Down Expand Up @@ -1713,6 +1738,8 @@ and
@combine(df, :mx = mean(:x), :sx = std(:x))
```

$astable_docstring_snippet

### Examples

```julia
Expand Down Expand Up @@ -1829,6 +1856,8 @@ and
@by(df, :g, mx = mean(:x), sx = std(:x))
```

$astable_docstring_snippet

### Examples

```julia
Expand Down
6 changes: 5 additions & 1 deletion src/parsing.jl
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,6 @@ function extract_macro_flags(ex::Expr, exprflags = deepcopy(DEFAULT_FLAGS))
return (ex, exprflags)
end
end

return (ex, exprflags)
end

Expand All @@ -127,6 +126,9 @@ function check_macro_flags_consistency(exprflags)
if !exprflags[BYROW_SYM][]
s = "The `@passmissing` flag is currently only allowed with the `@byrow` flag"
throw(ArgumentError(s))
elseif exprflags[ASTABLE_SYM][]
s = "The `@passmissing` flag is currently not allowed with the `@astable` flag"
throw(ArgumentError(s))
end
end
end
Expand Down Expand Up @@ -224,6 +226,8 @@ function get_source_fun(function_expr; exprflags = deepcopy(DEFAULT_FLAGS))
end
end

println(MacroTools.prettify(fun))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
println(MacroTools.prettify(fun))

return source, fun
end

Expand Down
7 changes: 3 additions & 4 deletions src/parsing_astable.jl
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,8 @@ end
block(ex) = isexpr(ex, :block) ? ex : :($ex;)

sym_or_str_to_sym(x::Union{AbstractString, Symbol}) = Symbol(x)
function sym_or_str_to_sym(x)
e = "New columns created inside @astable must be Symbols or AbstractStrings"
throw(ArgumentError(e))
end
sym_or_str_to_sym(x) =
throw(ArgumentError("New columns created inside @astable must be Symbols or AbstractStrings"))

function get_source_fun_astable(ex; exprflags = deepcopy(DEFAULT_FLAGS))
inputs_to_function = Dict{Any, Symbol}()
Expand Down Expand Up @@ -88,6 +86,7 @@ function get_source_fun_astable(ex; exprflags = deepcopy(DEFAULT_FLAGS))
inputargs = Expr(:tuple, values(inputs_to_function)...)
nt_iterator = (:(DataFramesMeta.sym_or_str_to_sym($k) => $v) for (k, v) in lhs_assignments)
nt_expr = Expr(:tuple, Expr(:parameters, nt_iterator...))

body = Expr(:block, Expr(:block, exprs...), nt_expr)

fun = quote
Expand Down
67 changes: 57 additions & 10 deletions test/astable_flag.jl
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ const ≅ = isequal

d = @rtransform df @astable begin
:x = 1

nothing
nalimilan marked this conversation as resolved.
Show resolved Hide resolved
end

Expand Down Expand Up @@ -42,10 +41,6 @@ const ≅ = isequal
@test d == DataFrame(x = 5)
end

@testset "@astable with just assignments, mutating" begin
# After finalizing above testset
end

@testset "@astable with strings" begin
df = DataFrame(a = 1, b = 2)

Expand Down Expand Up @@ -120,6 +115,49 @@ end
@test d == DataFrame(x = 1, z = 6)
end

@testset "@astable with mutation" begin
df = DataFrame(a = 1, b = 2)

df2 = copy(df)
d = @rtransform! df2 @astable begin
:x = 1
nothing
end

@test d == DataFrame(a = 1, b = 2, x = 1)
@test d === df2

df2 = copy(df)
d = @rselect! df2 @astable begin
:x = 1
y = 100
nothing
end

@test d == DataFrame(x = 1)
@test d === df2

df2 = copy(df)
d = @transform! df2 @astable begin
:x = [5]
y = 100
nothing
end

@test d == DataFrame(a = 1, b = 2, x = 5)
@test d === df2

df2 = copy(df)
d = @select! df2 @astable begin
:x = [5]
y = 100
nothing
end

@test d == DataFrame(x = 5)
@test d === df2
end

@testset "grouping astable flag" begin
df = DataFrame(a = [1, 1, 2, 2], b = [5, 6, 7, 8])

Expand All @@ -131,34 +169,43 @@ end
:b_max = ex[2]
end

@test sort(d.b_min) == [5, 7]
res_sorted = DataFrame(a = [1, 2], b_min = [5, 7], b_max = [6, 8])

@test sort(d, :b_min) == res_sorted

d = @combine gd @astable begin
ex = extrema(:b)
$"b_min" = ex[1]
$"b_max" = ex[2]
end

@test sort(d.b_min) == [5, 7]
@test sort(d, :b_min) == res_sorted

d = @by df :a @astable begin
ex = extrema(:b)
:b_min = ex[1]
:b_max = ex[2]
end

@test sort(d.b_min) == [5, 7]
@test sort(d, :b_min) == res_sorted

d = @by df :a @astable begin
ex = extrema(:b)
$"b_min" = ex[1]
$"b_max" = ex[2]
end

@test sort(d.b_min) == [5, 7]
@test sort(d, :b_min) == res_sorted
end


@testset "errors with passmissing" begin
@eval df = DataFrame(y = 1)
@test_throws LoadError @eval @transform df @passmising @byrow @astable :x = 2
@test_throws LoadError @eval @transform df @byrow @astable @passmissing :x = 2
@test_throws LoadError @eval @transform df @astable @passmissing @byrow :x = 2
@test_throws LoadError @eval @rtransform df @astable @passmissing :x = 2
@test_throws LoadError @eval @rtransform df @passmissing @astable :x = 2
end

@testset "bad assignments" begin
@eval df = DataFrame(y = 1)
Expand Down