Roadmap

@jturner314

In terms of functionality, the mid-term end goal is to achieve feature parity with the statistics routine in numpy (here) and Julia StatsBase (here).

For the next version:

Order statistics:
- partialord version for quantiles methods;
Histograms:
- merge method;

For version 0.2.0:

Order statistics:
- optimized computations of multiple quantiles if requested all at once (Bulk quantiles #26) ;
- argmin / argmax (Implement argmin argmax #30);
Summary statistics:
- harmonic mean (Means #20);
- geometric mean (Means #20);
- higher order central moments (Central moments #23);
- standardized moments (they include kurtosis and skewness) (Central moments #23);
Histograms:
- Fix error handling (Issue: Reduce cases where histogram strategies panic #16 - PR: Histogram error handling #25 )
Entropy:
- Feature parity with StatsBase.jl (Entropy #24)

For version 0.1.0:

max / nanmax (@jturner314)
min / nanmin (@jturner314)
quantile / nanquantile (it includes percentile / nanpercentile as a special case) (@LukeMathWalker & @jturner314)
correlation-methods:
- cov (@LukeMathWalker) - ~~One last fix to be made (Remove 'static bound from type A in CorrelationExt.cov #3)~~ [On hold for now]
- corrcoef (@LukeMathWalker - Pearson correlation #5)
histogram-methods (@LukeMathWalker - Histogram (revisited) #9)

@jturner314

With respect to mean, average, std: var is implemented in the main ndarray crate - would it make sense to port it here?

I think it makes sense for ndarray-stats to provide *_skipnan variants (or whatever you want to call them) of those methods. However, it would make sense to add std_axis to ndarray since ndarray already has var_axis.

For methods that are already in ndarray, we could duplicate these methods as a trait in ndarray-stats for people who want to write generic code (where the implementations just call the instance methods). I'm ambivalent on this.

I'll slowly start working on this next week and then I'll get serious the week afterwards. Could you please give me commit/PR permissions to the repository @jturner314?

Okay, that sounds good. I've given you push access. Alternatively, if you'd like to have your repo be the main one instead of this one, that would be fine with me.

@jturner314

Once #9 gets merged I think we are in a good position to officially release version 0.1.0 on crates.io - what do you think? @jturner314

I agree.

By the way, I recently came across Julia's StatsBase.jl library. It's a good source of ideas in addition to NumPy/SciPy.

@jturner314

Added a bunch of tests to #9 and merged 🎉 It feels like ages since I started to work on it 😅 Your contribution was extremely helpful to get it in the shape it is right now, thanks a lot @jturner314!

What do we need to do in order to release on crates.io?
I am going to open a small PR to add crate-level documentation - a couple of lines, nothing major.

Yay! 🎉 That was a big job; great work.

What do we need to do in order to release on crates.io?

Ideally, we'd eliminate the [patch.crates-io] section from the Cargo.toml before we can release on crates.io. (This might even be required, I'm not sure.) #11 removed the patch for noisy_float, but a new version of ndarray will need to be released for us to remove its patch. It would be nice to merge a couple more ndarray PRs before release; I'll take a look.

It would also be good to merge #12 and #13 before releasing.

Merged #12 and #13 - looking around it seems we can publish with [patch.crates-io] section in Cargo.toml, but I agree it is much nicer to point to ndarray 0.12.1 as a dependency instead of a revision on master.

Let's wait for that release and then we are good to go.

@LukeMathWalker

ndarray-stats 0.1.0 is now on crates.io. 🎉 Thanks for all your hard work @LukeMathWalker!

💯 💯 I think it's safe to say it would have never got there without your help 😛
I'll drop a post on r/rust as well 👍

@jturner314

I have drafted a tentative roadmap with the features I'd like to add in the next release - please edit it with your comments and suggestions @jturner314

The roadmap looks good to me. I'm not familiar with the applications of higher order central moments (I'd usually use a histogram instead), but I don't mind adding them if people find them useful.

By the way, I invited you as an owner for the ndarray-stats crate, but I just realized that crates.io may not have sent the invitation if you haven't logged in before. Please let me know if you need me to re-send it.

Somehow I didn't receive an email notification, but the invite was on my dashboard - accepted it!

The main objective in that area is getting kurtosis and skewness, and given the kind of computation required to achieve that it makes sense to also roll out higher order central moments I'd say :)

Hey mate, argmin / argmax looks like simple enough to look into, do you have any suggestions of where to start?

Great job on 0.2.0 everyone!

I am not sure what we should be covering next in terms of major new functionality

A couple of ideas from StatsBase.jl:

Deviation functions
Weighted calculations (mean/std/etc.)

We could also add statistical models (e.g. linear regression), but that might be best put in a separate crate.

Well done! cheers!

A couple of ideas from StatsBase.jl:

Deviation functions

Weighted calculations (mean/std/etc.)

Unless any of you have made a start on these, I'd be interested in having a go at either, or contributing. I'll try to spend some time in the next couple of days looking at what is involved with the Deviation functions.

❓ Does anyone have any implementation suggestions other than just trying to port from StatsBase.jl?

If anyone wants to collaborate on the code then let me know.

Ok I made a start: #41

Any advice for choosing traits bounds for the A element types? Is it ok to use Copy or do we need to support any types that would be Clone?

@munckymagik

I'd say to use clone @munckymagik

@LukeMathWalker

@LukeMathWalker thanks. What led you to that decision? Is there a particular data type you've seen used in ndarrays that would need this? If so I'm thinking I might use it in the test fixtures to make sure all methods have the same bounds.

@munckymagik

I see it as a tradeoff between convenience and generality - I am not personally aware of any "popular" numerical type that is not Copy, but the cost of weakening it to Clone is so low that I see it as safe future-proofing @munckymagik

I wanted to code a simple weighted_mean for myself then contribute it

pub fn weighted_mean<A, S>(data: &ArrayBase<S, Ix1>, weights: &[A]) -> A
where
    S: Data<Elem = A>,
    A: Float,
{
    data.iter().zip(weights).fold(A::zero(), |acc, (&d, &w)| acc + d * w)
}

but I realize that it's too simple. This code is only useful for 1D arrays, or flattened matrices/images, etc. I can change the Ix1 for a D: Dimension, so that we don't need to flatten anything. It's still a one-liner though and it doesn't offer any "axis" feature, like Numpy. I think we need 2 functions here, because they won't return the same type.

n-d data with n-d weight, returns a number
axis mode: n-d data with n-d weight, returns (n-1)-d array.

What do you guys had in mind?

@nilgoyette

I think we need 2 functions here, because they won't return the same type.

n-d data with n-d weight, returns a number

axis mode: n-d data with n-d weight, returns (n-1)-d array.

What do you guys had in mind?

I think it makes perfect sense to have two functions. @nilgoyette
It is also consistent with the rest of the API: we have mean and mean_axis, var and var_axis, etc. 👍

I was thinking of picking up the histogram merge method.
I am relatively new to rust and ndarray. With this exercise, I want to pick up ndarray, rust and also start contributing to ndarray-* libraries.
What do you guys think? Or are there more high level good first issue in other ndarray-* libraries?

@aeroaks

@aeroaks I'd say go for it 💯

You could raise a draft PR if you get something working and want some early feedback.

I would like to implement something like scipy.stats.binned_statistic_dd based on ndarray_stats::histogram::Histogram, allowing to caluclate running means, variances, sums, max/min value in each bin.
Would that be of interest?

@RolfStierle

It does! @RolfStierle

According to https://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram_bin_edges.html#numpy.histogram_bin_edges

Some of bins building strategies are not implemented by rust-ndarray now:

doane
scott
stone

Thanks very much for sharing the good work. Would it be possible to add univariate, bivariate and multivariate kernel density estimation functions? Thanks.

Navigation Menu

Repository navigation

Roadmap #1

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Roadmap #1

Description

Activity

jturner314 commented on Sep 16, 2018

LukeMathWalker commented on Nov 8, 2018

jturner314 commented on Nov 11, 2018

LukeMathWalker commented on Nov 18, 2018

jturner314 commented on Nov 18, 2018

LukeMathWalker commented on Nov 19, 2018

jturner314 commented on Nov 21, 2018

LukeMathWalker commented on Nov 21, 2018

LukeMathWalker commented on Nov 24, 2018

jturner314 commented on Nov 27, 2018

LukeMathWalker commented on Nov 29, 2018

phungleson commented on Mar 9, 2019

12 remaining items

jturner314 commented on Apr 13, 2019

phungleson commented on Apr 15, 2019

munckymagik commented on Apr 17, 2019

munckymagik commented on Apr 17, 2019

LukeMathWalker commented on Apr 18, 2019

munckymagik commented on Apr 18, 2019

LukeMathWalker commented on Apr 18, 2019

nilgoyette commented on Sep 13, 2019

LukeMathWalker commented on Sep 14, 2019

aeroaks commented on Nov 18, 2019

munckymagik commented on Nov 20, 2019

RolfStierle commented on Apr 5, 2020

LukeMathWalker commented on Apr 5, 2020

lebensterben commented on Apr 15, 2020

humphreylee commented on Sep 21, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions