Skip to content
This repository has been archived by the owner on Aug 27, 2019. It is now read-only.

Latest commit

 

History

History
141 lines (103 loc) · 11.7 KB

README.md

File metadata and controls

141 lines (103 loc) · 11.7 KB
Raw

Regressions.jl

[MacOS/Linux] Coverage Documentation
Build Status codecov.io TBA

This is a convenience package gathering functionalities to solve a number of generalised linear regression/classification problems which, inherently, correspond to an optimisation problem of the form

L(y, Xθ) + P(θ)

where L is a loss function and P is a penalty function (both of those can be scaled or composed). Additional regression/classification methods which do not directly correspond to this formulation may be added in the future.

The core aims of this package are:

  • make these regressions models "easy to call" and callable in a unified way,
  • interface with MLJ.jl,
  • focus on performance including in "big data" settings exploiting packages such as Optim.jl, IterativeSolvers.jl,
  • use a "machine learning" perspective, i.e.: focus essentially on prediction, hyper-parameters should be obtained via a data-driven procedure such as cross-validation.

Implemented

Regressors Formulation¹ Available solvers Comments
OLS & Ridge L2Loss + 0/L2 Analytical² or CG³
Lasso & Elastic-Net L2Loss + 0/L2 + L1 (F)ISTA⁴
Robust 0/L2 RobustLoss⁵ + 0/L2 Newton, NewtonCG, LBFGS, IWLS-CG⁶ no scale⁷
  1. "0" stands for no penalty
  2. Analytical means the solution is computed in "one shot" using the \ solver,
  3. CG = conjugate gradient
  4. (Accelerated) Proximal Gradient Descent
  5. Huber, Andrews, Bisquare, Logistic, Fair and Talwar weighing functions available
  6. Iteratively re-Weighted Least Squares where each system is solved iteratively via CG
  7. In other packages such as Scikit-Learn, a scale factor is estimated along with the parameters, this is a bit ad-hoc and corresponds more to a statistical perspective, further it does not work well with penalties; we recommend using cross-validation to set the parameter of the Huber Loss. (TODO: document)
Classifiers Formulation Available solvers Comments
Logistic 0/L2 LogisticLoss + 0/L2 Newton, Newton-CG, LBFGS yᵢ∈{±1}
Logistic L1/EN LogisticLoss + 0/L2 + L1 (F)ISTA yᵢ∈{±1}
Multinomial 0/L2 MultinomialLoss + 0/L2 Newton-CG, LBFGS yᵢ∈{1,...,c}
Multinomial L1/EN MultinomialLoss + 0/L2 + L1 ISTA, FISTA yᵢ∈{1,...,c}

Unless otherwise specified:

  • Newton-like solvers use Hager-Zhang line search (default in Optim.jl)
  • ISTA, FISTA solvers use backtracking line search and a shrinkage factor of β=0.8

Current limitations

  • The models are built and tested assuming n > p; if this doesn't hold, tricks should be employed to speed up computations; these have not been implemented yet.
  • Stochastic solvers that would be appropriate for huge models have not yet been implemented.
  • "Meta" functionalities such as One-vs-All or Cross-Validation are left to other packages such as MLJ.

Possible future models

WIP

  • Quantile reg with ADMM, IWLS, (? IP, Frisch Newton)
  • LAD with ADMM

Future

Model Formulation Comments
Huber L1/ElasticNet HuberLosss + No/L2 + L1 ⭒
Group Lasso L2Loss + ∑L1 over groups ⭒
Adaptive Lasso L2Loss + weighted L1 ⭒ A
LAD L1Loss People seem to use a simplex algorithm (Barrodale and Roberts), prox like ADMM should be ok too G, or F
SCAD L2Loss + SCAD A, B, C
MCP L2Loss + MCP A
OMP L2Loss + L0Loss D
SGD Classifiers *Loss + No/L2/L1 and OVA SkL
  • (⭒) should be added soon

Other regression models

There are a number of other regression models that may be included in this package in the longer term but may not directly correspond to the paradigm Loss+Penalty introduced earlier.

In some cases it will make more sense to just use GLM.jl.

Sklearn's list: https://scikit-learn.org/stable/supervised_learning.html#supervised-learning

Model Note Link(s)
LARS --
Quantile Regression -- Yang et al, 2013, QuantileRegression.jl
L∞ approx (Logsumexp) -- slides
Passive Agressive -- Crammer et al, 2006 SkL
Orthogonal Matching Pursuit -- SkL
Least Median of Squares -- Rousseeuw, 1984
RANSAC, Theil-Sen Robust reg Overview RANSAC, SkL, SkL, More Ransac
Ordinal regression need to figure out how they work E
Count regression need to figure out how they work R
Robust M estimators -- F
Perceptron, MIRA classifier Sklearn just does OVA with binary in SGDClassif H
Robust PTS and LTS -- PTS LTS

What about other packages

While the functionalities in this package overlap with a number of existing packages, the hope is that this package will offer a general entry point for all of them in a way that won't require too much thinking from an end user (similar to how someone would use the tools from sklearn.linear_model). If you're looking for specific functionalities/algorithms, it's probably a good idea to look at one of the packages below:

There's also GLM.jl which is more geared towards statistical analysis for reasonably-sized datasets and does (as far as I'm aware) lack a few key functionalities for ML such as penalised regressions or multinomial regression.

References

Dev notes