Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve column read speed by removing type instability #33

Merged
merged 3 commits into from
Apr 3, 2024

Conversation

asinghvi17
Copy link
Contributor

@asinghvi17 asinghvi17 commented Apr 3, 2024

This improves the speed of a column re-materialization which has to be done using Tables.jl from 160 ms to 10 ms.

Benchmark:

using Shapefile, GeometryOps, DataFrames, BenchmarkTools


shp_file = "/Users/anshul/Downloads/ne_10m_admin_0_countries (1)/ne_10m_admin_0_countries.shp"
table = Shapefile.Table(shp_file)
df = DataFrame(table) # convert everything including the DBFTable to a DataFrame

_scale_by_5(x) = x .* 5
@benchmark GeometryOps.transform($_scale_by_5, $table) # 160 ms before, 13 ms after
@benchmark GeometryOps.transform($_scale_by_5, $df)    # 7 ms

src/DBFTables.jl Outdated Show resolved Hide resolved
src/DBFTables.jl Outdated Show resolved Hide resolved
@rafaqz
Copy link
Collaborator

rafaqz commented Apr 3, 2024

@asinghvi17 maybe post your benchmarks here for reviewers? I cant actually merge this.

@visr @joshday this is an order of magnitude improvement on column read speed, lets get it merged and bumped

Co-authored-by: Rafael Schouten <[email protected]>
@asinghvi17 asinghvi17 changed the title explicitly type the column returned by getproperty to avoid column un… Improve column read speed by removing type instability Apr 3, 2024
@visr visr merged commit eabfd9e into JuliaData:main Apr 3, 2024
4 of 5 checks passed
@asinghvi17 asinghvi17 deleted the patch-1 branch April 3, 2024 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants