feature_steering_example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "from regression_dataset import make_regression_dataset\n",
    "from regression_network import RegressionNetwork\n",
    "\n",
    "# Deterministic execution.\n",
    "CUDA_LAUNCH_BLOCKING = 1\n",
    "seed = 42\n",
    "torch.manual_seed(seed)\n",
    "np.random.seed(seed)\n",
    "torch.backends.cudnn.benchmark = False\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example for the Application of the Feature Steering Method Presented in “Beyond Debiasing: Actively Steering Feature Selection via Loss Regularization”\n",
    "\n",
    "This jupyter notebook provides an example for our method for the redundant regression dataset presented in our paper.\n",
    "\n",
    "You can choose to generate feature attributions with the feature attribution method provided by Reimers et al. based on both **contextual decomposition** and **conditional mutual information**. Additionally, you can choose other hyperparameters such as the weight factor $\\lambda$ and the norm that is applied (L1 / L2 norm).\n",
    "\n",
    "<span style=\"color:red\">\n",
    "\n",
    "If you are only interested in the actual implementation of our method, take a look at `feat_steering_loss(...)` in `regression_network.py`, where the feature steering part of the loss is calculated.\n",
    "\n",
    "</span>\n",
    "\n",
    "## Dataset\n",
    "We create a small regression dataset with redundant variables as described in our paper. That is, the created dataset has 9 input variables with a redundancy of 3 variables. In total, we generate 2000 samples, of which 1400 are used for training.\n",
    "\n",
    "*Note:* In the evaluations for our paper we not only generate one, but rather 9 datasets with different seeds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configuration of the datasets.\n",
    "high_dim_transform = True\n",
    "normalize = True  # Should only be set if HIGH_DIM_TRANSFORM\n",
    "n_informative_low_dim = 6\n",
    "n_high_dim = 9\n",
    "n_train, n_test, n_validation = 1400, 300, 300\n",
    "n_uninformative_low_dim = 0\n",
    "dataset_seed = 42\n",
    "batch_size = 100\n",
    "n_datasets = 9\n",
    "\n",
    "noise_on_output = 0.0\n",
    "noise_on_high_dim_snrdb = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create and load the regression dataset.\n",
    "train_dataloader, test_dataloader, validation_dataloader, _ = make_regression_dataset(\n",
    "    high_dim_transform=high_dim_transform,\n",
    "    n_features_low_dim=n_informative_low_dim,\n",
    "    n_uninformative_low_dim=n_uninformative_low_dim,\n",
    "    n_high_dim=n_high_dim,\n",
    "    noise_on_high_dim_snrdb=noise_on_high_dim_snrdb,\n",
    "    noise_on_output=noise_on_output,\n",
    "    n_train=n_train,\n",
    "    n_test=n_test,\n",
    "    n_validation=n_validation,\n",
    "    normalize=normalize,\n",
    "    seed=dataset_seed,\n",
    "    batch_size=batch_size,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network\n",
    "We follow the paper and create a network with a single hidden layer of size 9 and input size 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Network architecture.\n",
    "input_size = n_high_dim\n",
    "hidden_dim_size = n_high_dim\n",
    "n_hidden_layers = 1\n",
    "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n",
    "\n",
    "# Create Network.\n",
    "mlp = RegressionNetwork(\n",
    "    input_shape=input_size,\n",
    "    n_hidden_layers=n_hidden_layers,\n",
    "    hidden_dim_size=hidden_dim_size,\n",
    "    device=device,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating the network, we can train it with the *feature steering loss*.\n",
    "\n",
    "Recall from the paper that our method to steer feature usage is implemented via loss regularization. Let $D$ refer to the set of features that should be discouraged and $E$ to the set of features that should be encouraged. With $c_i$ being a measure of the influence of feature $i$ on the model's prediction process, $\\lambda \\in \\mathbb{R}_{\\ge 0}$ as a weight factor and $\\mathcal{L}$ as the standard maximum-likelihood loss for network parameters $\\theta$, our model is trained with the following loss function:\n",
    "\n",
    "$$ \\mathcal{L}'(\\theta) = \\mathcal{L}(\\theta) + \\lambda \\left( \\sum_{i \\in D} || c_i || - \\sum_{i \\in E} || c_i || \\right) .$$\n",
    "For $|| \\cdot ||$, we consider the L1 and L2 norms.\n",
    "\n",
    "**Parameters:**\n",
    "\n",
    "Our implementation allows you to choose several *hyperparameters* for the feature steering process. You can adapt the following aspects of the calculation of the loss function:\n",
    "\n",
    "* The feature attributions $c_i$ are generated based on the feature attribution method proposed by Reimers et al. For this, the attribution modes `cmi` for feature attribution based on the (transformed) conditional mutual information and `contextual_decomposition` for feature attribution performed with contextual decomposition are available.\n",
    "* Feature steering can be performed with feature attributions weighted with L1 norm (`loss_l1`) and L2 norm (`loss_l2`). That is, this modifies the norm applied for $|| \\cdot ||$.\n",
    "* The indices of the features that shall be encouraged or discouraged (defining $D$ and $E$) are passed as lists.\n",
    "* The weight factor $\\lambda$ is specified as `lambda`.\n",
    "\n",
    "<span style=\"color:red\">\n",
    "\n",
    "**Implementation:**\n",
    "\n",
    "If you want to take a closer look at the implementation of this feature steering loss, take a look at the function `feat_steering_loss(...)` in `regression_network.py`. Here, you can find the calculation of the feature steering part of the loss - which is what you need to add to your own network in order to apply our method.\n",
    "\n",
    "</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training configuration.\n",
    "learning_rate = 0.01\n",
    "epochs = 90\n",
    "feat_steering_config = {\n",
    "    \"attrib_mode\": \"cmi\", # contextual_decomposition\n",
    "    \"steering_mode\": \"loss_l2\", # loss_l1, none\n",
    "    \"encourage\": [0, 1, 2],\n",
    "    \"discourage\": [],\n",
    "    \"lambda\": 100.0, # Adapt accordingly for CMI / CD\n",
    "}\n",
    "\n",
    "# Train the network.\n",
    "mlp.train(train_dataloader, feat_steering_config, epochs, learning_rate)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "featuresteering",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}