examples/jupyter_notebook/tutorial_log_clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Tutorial: Log Clustering using LogAI\n",
    "\n",
    "This is an example to show how to use LogAI to conduct log clustering analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Load Data\n",
    "\n",
    "You can use `OpensetDataLoader` to load a sample open log dataset. Here we use HDFS dataset from\n",
    "[LogHub](https://zenodo.org/record/3227177#.Y1M3LezML0o) as an example."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import os\n",
    "from logai.dataloader.openset_data_loader import OpenSetDataLoader, OpenSetDataLoaderConfig\n",
    "\n",
    "#File Configuration\n",
    "filepath = \"../datasets/HDFS_2000.log\"\n",
    "filepath = os.path.join(\"..\", \"datasets\", \"HDFS_2000.log\")\n",
    "\n",
    "dataset_name = \"HDFS\"\n",
    "data_loader = OpenSetDataLoader(\n",
    "    OpenSetDataLoaderConfig(\n",
    "        dataset_name=dataset_name,\n",
    "        filepath=filepath)\n",
    ")\n",
    "\n",
    "logrecord = data_loader.load_data()\n",
    "\n",
    "logrecord.to_dataframe().head(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Preprocess\n",
    "\n",
    "In preprocessing step user can retrieve and replace any regex strings and clean the raw loglines. This\n",
    "can be very useful to improve information extraction of the unstructured part of logs,\n",
    " as well as generate more structured attributes with domain knowledge.\n",
    "\n",
    "Here in the example, we use the below regex to retrieve Block IDs, IP addresses and filepaths."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "\n",
    "from logai.preprocess.preprocessor import PreprocessorConfig, Preprocessor\n",
    "from logai.utils import constants\n",
    "\n",
    "loglines = logrecord.body[constants.LOGLINE_NAME]\n",
    "attributes = logrecord.attributes\n",
    "\n",
    "preprocessor_config = PreprocessorConfig(\n",
    "    custom_replace_list=[\n",
    "        [r\"(?<=blk_)[-\\d]+\", \"<block_id>\"],\n",
    "        [r\"\\d+\\.\\d+\\.\\d+\\.\\d+\", \"<IP>\"],\n",
    "        [r\"(/[-\\w]+)+\", \"<file_path>\"],\n",
    "    ]\n",
    ")\n",
    "\n",
    "preprocessor = Preprocessor(preprocessor_config)\n",
    "\n",
    "clean_logs, custom_patterns = preprocessor.clean_log(\n",
    "    loglines\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Parsing\n",
    "\n",
    "After preprocessing, we call auto-parsing algorithms to automatically parse the cleaned logs.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.log_parser import LogParser, LogParserConfig\n",
    "from logai.algorithms.parsing_algo.drain import DrainParams\n",
    "\n",
    "# parsing\n",
    "parsing_algo_params = DrainParams(\n",
    "    sim_th=0.5, depth=5\n",
    ")\n",
    "\n",
    "log_parser_config = LogParserConfig(\n",
    "    parsing_algorithm=\"drain\",\n",
    "    parsing_algo_params=parsing_algo_params\n",
    ")\n",
    "\n",
    "parser = LogParser(log_parser_config)\n",
    "parsed_result = parser.parse(clean_logs)\n",
    "\n",
    "parsed_loglines = parsed_result['parsed_logline']\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Information Extraction\n",
    "\n",
    "### Vectorization for unstructured loglines\n",
    "\n",
    "Here we use `word2vec` to vectorize unstructured part of the logs. The output will be a list of\n",
    "numeric vectors that representing the semantic features of these log templates."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.log_vectorizer import VectorizerConfig, LogVectorizer\n",
    "\n",
    "vectorizer_config = VectorizerConfig(\n",
    "    algo_name = \"word2vec\"\n",
    ")\n",
    "\n",
    "vectorizer = LogVectorizer(\n",
    "    vectorizer_config\n",
    ")\n",
    "\n",
    "# Train vectorizer\n",
    "vectorizer.fit(parsed_loglines)\n",
    "\n",
    "# Transform the loglines into features\n",
    "log_vectors = vectorizer.transform(parsed_loglines)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Categorical Encoding for log attributes\n",
    "\n",
    "We also do categorical encoding for log attributes to convert the strings into numerical representations."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.categorical_encoder import CategoricalEncoderConfig, CategoricalEncoder\n",
    "\n",
    "encoder_config = CategoricalEncoderConfig(name=\"label_encoder\")\n",
    "\n",
    "encoder = CategoricalEncoder(encoder_config)\n",
    "\n",
    "attributes_encoded = encoder.fit_transform(attributes)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Feature Extraction\n",
    "\n",
    "Then we extract and concate the semantic features for both the unstructured and structured part of logs."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.feature_extractor import FeatureExtractorConfig, FeatureExtractor\n",
    "\n",
    "timestamps = logrecord.timestamp['timestamp']\n",
    "\n",
    "config = FeatureExtractorConfig(\n",
    "    max_feature_len=100\n",
    ")\n",
    "\n",
    "feature_extractor = FeatureExtractor(config)\n",
    "\n",
    "_, feature_vector = feature_extractor.convert_to_feature_vector(log_vectors, attributes_encoded, timestamps)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Clustering\n",
    "\n",
    "Here we use K-Means clustering algorithm as an example. WE set the number of clusters to 7 in\n",
    "K-Means algorithm parameter configuration."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.algorithms.clustering_algo.kmeans import KMeansParams\n",
    "from logai.analysis.clustering import ClusteringConfig, Clustering\n",
    "\n",
    "clustering_config = ClusteringConfig(\n",
    "    algo_name='kmeans',\n",
    "    algo_params=KMeansParams(\n",
    "        n_clusters=7\n",
    "    )\n",
    ")\n",
    "\n",
    "log_clustering = Clustering(clustering_config)\n",
    "\n",
    "log_clustering.fit(feature_vector)\n",
    "\n",
    "cluster_id = log_clustering.predict(feature_vector).astype(str).rename('cluster_id')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Check clustering results.\n",
    "logrecord.to_dataframe().join(cluster_id).head(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}