Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement box filtering #202

Merged
merged 11 commits into from
Feb 6, 2024
Merged

Implement box filtering #202

merged 11 commits into from
Feb 6, 2024

Conversation

camposandro
Copy link
Contributor

Implements box filtering for range of right ascension (ra) and/or declination (dec). If both are provided we construct a spherical polygon and use the preexisting polygon search. This feature allows to select the partitions of a subregion of the catalog sky, reducing the number of partition files read and therefore improving runtime performance.

Needed for astronomy-commons/lsdb#90.

  • My PR includes a link to the issue that I am addressing

Code Quality

  • I have read the Contribution Guide
  • My code follows the code style of this project
  • My code builds (or compiles) cleanly without any errors or warnings
  • My code contains relevant comments and necessary documentation

@camposandro camposandro self-assigned this Jan 31, 2024
@camposandro camposandro changed the title Sandro/add radec filter Implement box filtering Jan 31, 2024
Copy link

github-actions bot commented Jan 31, 2024

Before [fb575dd] After [fd7498a] Ratio Benchmark (Parameter)
650±10ms 657±9ms 1.01 benchmarks.Suite.time_pixel_tree_creation
96.7±1ms 97.7±0.5ms 1.01 benchmarks.time_test_cone_filter_multiple_order
1.15±0.01s 1.15±0.03s 1 benchmarks.MetadataSuite.time_load_partition_info_order7
1.87±0.01s 1.86±0.02s 1 benchmarks.MetadataSuite.time_load_partition_join_info
126±3ms 125±2ms 0.99 benchmarks.time_test_alignment_even_sky
305±7ms 294±10ms 0.96 benchmarks.MetadataSuite.time_load_partition_info_order6

Click here to view all benchmarks.

Copy link

codecov bot commented Jan 31, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (fb575dd) 100.00% compared to head (10945ca) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #202   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           54        55    +1     
  Lines         1848      1923   +75     
=========================================
+ Hits          1848      1923   +75     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@camposandro camposandro marked this pull request as ready for review February 3, 2024 02:06
src/hipscat/pixel_math/box_filter.py Outdated Show resolved Hide resolved
src/hipscat/pixel_math/box_filter.py Outdated Show resolved Hide resolved
Copy link
Contributor

@hombit hombit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels and works great.

One note about performance: selecting large range, in any of ra or dec, for norder=10 takes around a minute on my machine. I believe it is the same order of magnitude as other actions on trillion-row scale catalogs, so maybe it is not a big issue for now.

@smcguire-cmu
Copy link
Contributor

It feels and works great.

One note about performance: selecting large range, in any of ra or dec, for norder=10 takes around a minute on my machine. I believe it is the same order of magnitude as other actions on trillion-row scale catalogs, so maybe it is not a big issue for now.

I'd be interested in seeing where that time is. My guess would be creating the pixel trees for large areas at order 10. Not needed now, but I wonder if we convert our large order 10 area to a MOC if that would make it quicker by reducing the number of pixels in the tree. We could also improve the pixel tree performance ofc (or rewrite it in rust) but that might be more difficult.

@camposandro camposandro merged commit 4e253a5 into main Feb 6, 2024
12 checks passed
@camposandro camposandro deleted the sandro/add-radec-filter branch February 6, 2024 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants