Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Strange cut for imbalanced distributions #382

Closed
bkamins opened this issue Feb 25, 2022 · 2 comments · Fixed by #410
Closed

Strange cut for imbalanced distributions #382

bkamins opened this issue Feb 25, 2022 · 2 comments · Fixed by #410

Comments

@bkamins
Copy link
Member

bkamins commented Feb 25, 2022

I find this result strange:

julia> cut([fill(1, 10); 4], 2)
11-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"

julia> cut([fill(1, 10); 4], 3, allowempty=true)
11-element CategoricalArray{String,1,UInt32}:
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
 "Q2: [1.0, 4.0]"
@bkamins
Copy link
Member Author

bkamins commented Feb 25, 2022 

julia> cut([fill(1, 5); fill(4,5)], 2)
10-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 2.5)"
 "Q1: [1.0, 2.5)"
 "Q1: [1.0, 2.5)"
 "Q1: [1.0, 2.5)"
 "Q1: [1.0, 2.5)"
 "Q2: [2.5, 4.0]"
 "Q2: [2.5, 4.0]"
 "Q2: [2.5, 4.0]"
 "Q2: [2.5, 4.0]"
 "Q2: [2.5, 4.0]"

julia> cut([fill(1, 5); fill(4,5)], 3)
10-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"
 "Q1: [1.0, 4.0]"

@nalimilan nalimilan mentioned this issue Dec 27, 2024
Merged
@nalimilan
Copy link
Member

nalimilan commented Dec 27, 2024 •
edited
Loading

#410 fixes this, but not necessarily in the most convenient/intuitive way:

julia> cut([fill(1, 10); 4], 2)
ERROR: ArgumentError: cannot compute 2 quantiles: `quantile` returned only 1 group(s) due to duplicated values in `x`. Pass `allowempty=true` to allow empty quantiles or choose a lower value for `ngroups`.
Stacktrace:
 [1] cut(x::Vector{Int64}, ngroups::Int64; labels::typeof(CategoricalArrays.quantile_formatter), allowempty::Bool)
   @ CategoricalArrays ~/.julia/dev/CategoricalArrays/src/extras.jl:262
 [2] cut(x::Vector{Int64}, ngroups::Int64)
   @ CategoricalArrays ~/.julia/dev/CategoricalArrays/src/extras.jl:251
 [3] top-level scope
   @ REPL[272]:1

julia> cut([fill(1, 10); 4], 3, allowempty=true)
11-element CategoricalArray{String,1,UInt32}:
 "Q3: [1.0, 4.0]"
 "Q3: [1.0, 4.0]"
 "Q3: [1.0, 4.0]"
 ⋮
 "Q3: [1.0, 4.0]"
 "Q3: [1.0, 4.0]"
 "Q3: [1.0, 4.0]"

julia> levels(ans)
3-element Vector{String}:
 "Q1: (1.0, 1.0)"
 "Q2: (1.0, 1.0)"
 "Q3: [1.0, 4.0]"

It would probably be more useful to have classes be [1, 1] and (1, 4] in the first case and [1, 1], (1, 4), [4, 4] in the second one, but this would go against the rule that intervals are closed on the right except the last one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix corner cases of cut with duplicated breaks JuliaData/CategoricalArrays.jl
2 participants
@nalimilan @bkamins