Fixed the implementation of the Additive Attention

Simple minor release fixing the implementation of the additive attention (see #110 )

Self-Supervised Pre-Training for Tabular models

There are a number of changes and new features in this release, here is a summary:

1. Refactored the code related to the 3 forms of training in the library:

   • Supervised Training (via the Trainer class)
   • Self-Supervised pre-training: we have implemented two methods or routines for self-supervised pre-training. These are:
     • Encoder-Decoder Pre-Training (via the EncoderDecoderTrainer class): this is inspired by the TabNet paper
     • Constrastive-Denoising Pre-Training (via de ConstrastiveDenoising class): this is inspired by the SAINT paper
   • Bayesian or Probabilistic Training (via the BayesianTrainer: this is inspired by the paper Weight Uncertainty in Neural Networks
     

   Just as a reminder, the current deep learning models for tabular data available in the library are:

   • Wide
   • TabMlp
   • TabResNet
   • TabNet
   • TabTransformer
   • FTTransformer
   • SAINT
   • TabFastformer
   • TabPerceiver
   • BayesianWide
   • BayesianTabMlp

2. The text related component has now 3 available models, all based on RNNs. There are reasons for that although the integration with the Hugginface Transformer library is the next step in the development of the library. The 3 models available are:

   • BasicRNN
   • AttentiveRNN
   • StackedAttentiveRNN

   The last two are based on Hierarchical Attention Networks for Document Classification. See the docs for details

3. The image related component is now fully integrated with the latest torchvision release, with a new Multi-Weight Support API. Currently, the model variants supported by our library are:

   • resnet
   • shufflenet
   • resnext
   • wide_resnet
   • regnet
   • densenet
   • mobilenet
   • mnasnet
   • efficientnet
   • squeezenet

Move docs to mkdocs

Simply Update all documentation

Probabilistic Models and Label/Feature Distribution smoothing

This release fixes some minor bugs but mainly brings a couple of new functionalities:

1. New experimental Attentive models, namely: ContextAttentionMLP and SelfAttentionMLP.
2. 2 Probabilistic models based on Bayes by Backprop (BBP) as described in Weight Uncertainty in Neural Networks, namely: BayesianTabMlp and BayesianWide.
3. Label and Feature Distribution Smoothing (LDS and FDS) for Deep Imbalanced Regression (DIR) as described in Delving into Deep Imbalanced Regression
4. Better integration with torchvision for the deepimage component of a WideDeep model
5. 3 Available models for the deeptext component of a WideDeep model. Namely: BasicRNN, AttentiveRNN and StackedAttentiveRNN

Transformers without categorical data

This minor release simply fixes issue #53 related to the fact that SAINT, the FT-Transformer and the TabFasformer failed when the input data had no categorical columns

v1.0.9: The TabFormer Family Grows

Functionalities:

• Added a new functionality called Tab2Vec that given a trained model and a fitted Tabular Preprocessor it will return an input dataframe transformed into embeddings

TabFormers: Increased the Tabformer (Transformers for Tabular Data) family

• Added a proper implementation of the FT-Transformer with Linear Attention (as introduced in the Linformer paper)
• Added a TabFastFormer model, an adaptation of the FastFormer for Tabular Data
• Added a TabPerceiver model, an adaptation of the Perceiver for Tabular Data

Docs

• Refined the docs to make them cleared and fix a few typos

SAINT and the FT-Transformer

The two main additions to the library are:

• SAINT from SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training and
• FT-Transformer from Revisiting Deep Learning Models for Tabular Data

In addition

• New DataLoader for imbalanced dataset. See here.
• Integration with torchmetrics.

Tabnet and v1 ready

This release represents a major step forward for the library in terms of functionalities and flexibility:

1. Ported TabNet from the fantastic implementation of the guys at dreamquark-ai.
2. Callbacks are now more flexible and save more information.
3. The save method in the Trainer is more flexible and transparent
4. The library has extensively been tested via experiments against LightGBM (see here)

v0.4.8: WideDeep with the TabTransformer

This release represents an almost-complete refactor of the previous version and I consider the code in this version well tested and production-ready. The main reason why this release is not v1 is because I want to use it with a few more datasets, but at the same time I want the version to be public to see if others use it. Also, I want the changes from the last Beta and v1 to be not too significant.

This version is not backwards compatible (at all).

These are some of the structural changes:

• Building of the model and training the model and now completely decoupled
• Added the TabTransformer as a potential deeptabular component
• Renamed many of the parameters so that they are consistent between models
• Added the possibility of customising almost every single component: model component, losses, metrics and callbacks
• Added R2 metrics for regression problems

v0.4.7: individual components can run independently and image treatment replicates that of Pytorch

The treatment of the image datasets in WideDeepDataset replicates that of Pytorch. In particular this source code:

if isinstance(pic, np.ndarray):
    # handle numpy array
    if pic.ndim == 2:
        pic = pic[:, :, None]

In addition, I have added the possibility of using each of the model components in isolation and independently. This is, one could now use the wide, deepdense (either DeepDense or DeepDenseResnet), deeptext and deepimage independently.