Skip to content

Change to Anderson-Darling for testing. - #11

Merged
mjlarson merged 8 commits into
mainfrom
mlarson/anderson-darling
Oct 9, 2026
Merged

mjlarson merged 8 commits into
mainfrom
mlarson/anderson-darling

Conversation

@mjlarson

@mjlarson mjlarson commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Using a chi2 fit to the CDF has always been a little gross and ugly. The chi2 isn't a great measure and it can lead to some parts of the distribution being overly important in the fit. A better choice would be to use something actually designed to run on CDFs instead. Here, I swap out the chi2 for an Anderson-Darling test, which seems to significantly improve the fits for the previously-worst bins. Most bins aren't affected here since they were fitting well anyway. Bias tests aren't showing much impact either, probably because most bins aren't changing.

Here's the worst bins from DNNCascades-ICEMAN along with their best-fits under chi2 (red) and AD (blue).

image (198)

And the same for NorthernTracks v5.4
king_fit_ad_vs_chi2_NTv5 4_IC79

Note: this is branched off of mlarson/sigsub in #10 , so I have to merge that one first.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.04651% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.90%. Comparing base (981e816) to head (2f8ee83).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
kingmaker/utils.py 0.00% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #11      +/-   ##
==========================================
+ Coverage   78.87%   78.90%   +0.02%     
==========================================
  Files           6        6              
  Lines         923      948      +25     
==========================================
+ Hits          728      748      +20     
- Misses        195      200       +5     
Flag Coverage Δ
test_basic 14.66% <11.62%> (+0.03%) ⬆️
test_fitting 37.76% <41.86%> (-0.38%) ⬇️
test_king_pdf 29.11% <55.81%> (+1.16%) ⬆️
test_template_smeared_king_pdf 25.84% <11.62%> (-0.27%) ⬇️
test_utils 16.24% <11.62%> (-0.01%) ⬇️
test_wrapper 38.81% <41.86%> (+0.35%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@mjlarson
mjlarson marked this pull request as ready for review October 9, 2026 16:10
Copilot AI balanced review requested due to automatic review settings October 9, 2026 16:10
@mjlarson
mjlarson merged commit fe396ec into main Oct 9, 2026
11 checks passed
@mjlarson
mjlarson deleted the mlarson/anderson-darling branch October 9, 2026 16:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The scientific fitting objective and numerical interpolation changes warrant final domain-expert validation.

0 open findings

What changed in this PR

Replaces CDF chi-square fitting with an Anderson–Darling objective while incorporating RA-averaged signal-subtraction improvements from #10.

Changes:

  • Adds the Anderson–Darling objective and analytical gradient.
  • Improves RA-averaged PDF grids, validation, and interpolation.
  • Expands numerical tests and updates signal-subtraction documentation.
File Description
kingmaker/​fitting.py Implements Anderson–Darling fitting.
kingmaker/​pdf.py Improves marginalized PDF grids and validation.
kingmaker/​utils.py Computes RA averages using adaptive grids.
kingmaker/​wrapper.py Clarifies RA-averaged output semantics.
tests/​test_fitting.py Tests the objective and gradient.
tests/​test_king_pdf.py Tests normalization, accuracy, and clamping.
README.md Updates signal-subtraction terminology.
docs/​signal_subtraction.rst Documents RA averaging and grid behavior.
docs/​examples.rst Updates examples and terminology.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants