Resizing sparse vectors

[gdalle]

Hi there!

I was trying to use sparse columns in a DataFrame and I found out that some methods were missing for the SparseVector type, such as deleteat!. Here's a MWE:

julia> using DataFrames, DataFramesMeta, SparseArrays

julia> df = DataFrame(x = sparse([0, 1, 0, 1]))
4×1 DataFrame
 Row │ x     
     │ Int64 
─────┼───────
   1 │     0
   2 │     1
   3 │     0
   4 │     1

julia> @subset!(df, :x .== 1)
ERROR: MethodError: no method matching deleteat!(::SparseVector{Int64, Int64}, ::Vector{Int64})
Closest candidates are:
  deleteat!(::Vector{T} where T, ::AbstractVector{T} where T) at array.jl:1381
  deleteat!(::Vector{T} where T, ::Any) at array.jl:1380
  deleteat!(::BitVector, ::Any) at bitarray.jl:981
  ...
Stacktrace:
 [1] _delete!_helper(df::DataFrame, drop::Vector{Int64})
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:999
 [2] delete!(df::DataFrame, inds::Vector{Int64})
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:976
 [3] subset!(df::DataFrame, args::Any; skipmissing::Bool)
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/abstractdataframe/subset.jl:292
 [4] top-level scope
   @ ~/.julia/packages/DataFramesMeta/yzaoq/src/macros.jl:830

Is there a specific reason why this doesn't exist, or is it just a case of "no one has gotten around to it yet"? In that scenario, I wouldn't mind taking a crack at it if the difficulty is reasonable enough!

cc @pdeffebach @bkamins following our discussion in JuliaData/CategoricalArrays.jl#374

[fredrikekre]

SparseVector is fixed size due to being immutable and https://github.com/JuliaLang/julia/blob/b1cf46ce699505a25d85340b588f60335f619ab4/stdlib/SparseArrays/src/sparsevector.jl#L18.

[nalimilan]

Is there any particular advantage to n being fixed and to sparse vectors being immutable?

[bkamins]

Is there any particular advantage to n being fixed and to sparse vectors being immutable?

probably bounds checking is a bit faster.

[fredrikekre]

n needs to be stored (it is not just the length of the vector of indices).

[bkamins]

Yes - it has to be stored, but it could be made mutable.

[gdalle]

However common knowledge has it that mutable structs are slower. I haven't managed to find an official source for this, but maybe this could be worked around by storing n as a reference? The idea is that we don't need (or want) to make both vectors in the SparseVector struct mutable as well

[vtjnash]

Common knowledge sounds wrong then

[KristofferC]

Common knowledge sounds wrong then

Ref JuliaLang/julia#9755

[gdalle]

So could we keep the struct immutable and wrap n in a Ref{Ti} instead? How much of a performance overhead should we expect then? And how devastating would it be for the sparse codebase 😅 ?

[gdalle]

See also JuliaLang/julia#15668

[mind6]

sounds like you need a SparseCateogrical specificly for use in dataframes, where resizing is needed over immutability.