forked from Azure/Azure-Sentinel-Notebooks
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request Azure#24 from Azure/getting-started-fix
Getting started fix
- Loading branch information
Showing
2 changed files
with
275 additions
and
408 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,275 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "metadata": { | ||
| "collapsed": true | ||
| }, | ||
| "cell_type": "markdown", | ||
| "source": "# Get Started with KQL and Notebooks\n\n__Notebook Version:__ 1.1<br>\n__Python Version:__ Python 3.6 (including Python 3.6 - AzureML)<br>\n__Required Packages:__ Kqlmagic 0.1.90<br>\n__Platforms Supported:__<br>\n - Azure Notebooks Free Compute\n - Azure Notebooks DSVM\n__Data Source Required:__<br>\n - Log Analytics - At least one table with data from the last 30 days.\n \n### Description\nThe notebook provides you basic knowledge to using Kusto Query Language (KQL) in Azure Sentinel Notebooks.\n\n\n<font color=red>When you switch between Azure Notebooks Free Compute and Data Science Virtual Machine (DSVM), you may need to select Python version: please select Python 3.6 for Free Compute, and Python 3.6 - AzureML for DSVM.</font>" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "## Other resources\n\nThis notebook provides a very simple introduction into what can be done with KQL and Notebooks. There are many more things that notebooks can do.\nFor a more comprehensive guide to getting starting with Azure Sentinel and Notebooks please refer to the following resources:\n\n[Azure Notebooks Documentation](https://docs.microsoft.com/en-us/azure/sentinel/notebooks)\n\nThe majority of the pre-built Azure Sentinel Notebooks use the a Python library we have created to support notebook usage called [msticpy](https://github.com/Microsoft/msticpy). Documentation on this library can be found [here](https://msticpy.readthedocs.io/en/latest/) and in addition the [Configuration guide notebook](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/ConfiguringNotebookEnvironment.ipynb) provides useful support.\n\n\nFor more details on configuring your Azure Notebooks Project review this notebook:\n[AzureNotebooks-Configure Python Version](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/HowTos/AzureNotebooks-ConfigurePythonVersion.ipynb)\n\n\nFor help troubleshooting problems with notebooks use this notebook:\n[Troubleshooting Azure Sentinel Notebooks](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/TroubleShootingNotebooks.ipynb)\n\nThis [blog](https://techcommunity.microsoft.com/t5/azure-sentinel/security-investigation-with-azure-sentinel-and-jupyter-notebooks/ba-p/432921) provides a great introduction to Notebooks for security investigaiton" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "### Installation and Imports\nAt the start of any notebook we need to make sure that we have the requisite pacakges installed and imported into our notebook environment. This is a Python based notebook so we can use [pip](https://pypi.org/project/pip/) to install the packages needed. If you are using the free tier of Azure Notebooks you may need to install these packages every time, however if you use a local Jupyter server or DSVM in Azure Notebooks you should only need to do this installation once." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true, | ||
| "scrolled": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "!pip install Azure-Sentinel-Utilities --upgrade \n!pip install msticpy --upgrade ", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "We have also created some utilities that can help you check that you have the required packages installed and the correct version of Python for your notebook enabled. For more details on this checker please refer to https://github.com/Azure/Azure-Sentinel/tree/master/Notebooks/SentinelUtilities/SentinelUtils." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "import SentinelUtils\n# checking Python version\ncheck = SentinelUtils.version_management.ModuleVersionCheck()\npy_check = check.validate_python('3.6.0')\nif py_check.requirement_met == False:\n print('Please select Python 3.6 or Python 3.6 - AzureML at the upper right corner')\nelse:\n print('All OK, please continue')", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "In this notebook we will be using the [Kqlmagic library](https://pypi.org/project/Kqlmagic/) to query data from our Azure Sentinel instance, so we need to check it is installed and the correct version." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "# checking required packages\nmods_check = check.validate_installed_modules(['Kqlmagic>=0.1.105'])\nfor mod_info in mods_check:\n if mod_info.requirement_met == False:\n print('Please install {} {}.'.format(mod_info.name, mod_info.required_version))\n else:\n print(\"All required pacakges installed. Please continue.\")", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "markdown", | ||
| "source": "If the preceding cell asked you to install certain pacakges you can add a cell to do this. Click the + icon at the top of the page to add a new cell below the one you have currently selected. Then to install a pacakge with pip simply type '!pip install <package_name> --upgrade' into the cell and run it. This will install the latest version of the package you specify." | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "Once we have checked that we have all the needed elements installed we can import the modules we are going to be using in this notebook." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "import ipywidgets as widgets\nfrom IPython.display import display\nimport pandas as pd\nimport numpy as np\nfrom msticpy.nbtools.wsconfig import WorkspaceConfig\nfrom msticpy.data.data_providers import QueryProvider\nfrom msticpy.nbtools import nbwidgets\nprint(\"Imports complete.\")", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "Now that we have set up all the elements we need we can look at how to connect to an Azure Sentinel workspace, query it and interact with the output.\n\nThe first stage of this is to connect to our Azure Sentinel workspace. To do this we need to provide an worksapce and tenant ID of the workspace we wish to connect to, this can be provided either via a [config file](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html) or interactively in the cell output." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "tags": [ | ||
| "parameters" | ||
| ], | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "#Update to WorkspaceConfig(workspace=\"WORKSPACE_NAME\") to get alerts from a Workspace other than your default one.\nws_config = WorkspaceConfig()\ntry:\n ws_id = ws_config['workspace_id']\n ten_id = ws_config['tenant_id']\n print(\"Workspace details collected from config file\")\nexcept:\n print('Please go to your Log Analytics workspace, copy the workspace ID and/or tenant Id and paste here to enable connection to the workspace and querying of it.')\n ws_id = nbwidgets.GetEnvironmentKey(env_var='WORKSPACE_ID',\n prompt='Please enter your Log Analytics Workspace Id:', auto_display=True)\n ten_id = nbwidgets.GetEnvironmentKey(env_var='TENANT_ID',\n prompt='Please enter your Log Analytics Tenant Id:', auto_display=True)\n ws_id = ws_id.value\n ten_id = ten_id.value", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "Now that we have collected information on the workspace to connect to we can go ahead an authenticate using Kqlmagic. This uses a device logon process where you are required to authenticate your device using a set of credentials you provide via browser window." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "# You must run this cell to log into Log Analytics to continue\n# Make sure you have 0.1.90 or above, if not, run Kqlmagic installation again\n%reload_ext Kqlmagic\n%kql loganalytics://code;workspace=ws_id;tenant=ten_id;alias=\"Sentinel\" ", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "Once we have connected we can run KQL queries against our workspace by using '%kql our query'. Below we are just getting a list of tables with data in and how much data they have." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "%kql search * | summarize count() by Type", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "Now that we have a list of tables we can build some elements to interact with the data and focus on a specific table. Below we take the list of tables in Azure Sentinel we collected before and we display this in a drop down list, allowing us to select a table to focus on." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "scrolled": true, | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "dbSchema = %kql --schema \"Sentinel@loganalytics\"\ntables = list(dbSchema.keys())\nselected_table = widgets.Dropdown(options=tables, value=tables[1],description='Data Table:')\ndisplay(selected_table)", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "Now that we have selected a table we can collect data from that table and store them in a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html)" | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "%kql {selected_table.value} | where TimeGenerated >= ago(30d) | take 1000\nif len(_kql_raw_result_) > 0:\n df = _kql_raw_result_.to_dataframe()\n print(f\"Data collected from {selected_table.value}\")\nelse:\n df = None\n print(f'No data found for {selected_table.value} in the last 30 days.')", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "To further focus our search in the data we can now get a list of columns in our table and select one of them." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "columns = list(dbSchema[selected_table.value])\ncolumns.sort()\nselected_column = widgets.Dropdown(options=columns,description='Column:')\ndisplay(selected_column)", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "And focussing even further we can use the features of Pandas to get a list of unique values in the column selected." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "if isinstance(df, pd.DataFrame) and not df.empty:\n #Get a unique list of values in our column\n unique_values = df[selected_column.value].replace('', np.nan).dropna().drop_duplicates().sort_values()\n if len(unique_values.index) > 0:\n data_point = widgets.Dropdown(options=unique_values,description='Data value:')\n display(data_point)\nelse:\n print(f\"No data avalaible for {selected_table.value} please try another table.\")", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "Now that we have selected a table, a query, and a specific data value we can pass these variables into a new KQL query in order to just get the dat we want." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "# scope to a table and a column\n%kql {selected_table.value} | where {selected_column.value} contains '{data_point.value}' | take 5", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "cell_type": "markdown", | ||
| "source": "MSTICpy also includes a way to query Azure Sentinel via KQL that can be simpler than using native Kqlmagic. Below is an example of using MSTICpy to create a query, run it using MSTICpy and return the results in a Pandas DataFrame." | ||
| }, | ||
| { | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "cell_type": "code", | ||
| "source": "qry_prov = QueryProvider('LogAnalytics')\nla_connection_string = f'loganalytics://code().tenant(\"{ten_id}\").workspace(\"{ws_id}\")'\nqry_prov.connect(connection_str=f'{la_connection_string}')\n\nquery = \"\"\"\n SecurityAlert\n | where TimeGenerated > ago(30d)\n | take 10\n \"\"\"\n\nqry_prov.exec_query(query)", | ||
| "execution_count": null, | ||
| "outputs": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "celltoolbar": "Tags", | ||
| "hide_input": false, | ||
| "kernelspec": { | ||
| "name": "python36", | ||
| "display_name": "Python 3.6", | ||
| "language": "python" | ||
| }, | ||
| "language_info": { | ||
| "mimetype": "text/x-python", | ||
| "nbconvert_exporter": "python", | ||
| "name": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.6.6", | ||
| "file_extension": ".py", | ||
| "codemirror_mode": { | ||
| "version": 3, | ||
| "name": "ipython" | ||
| } | ||
| }, | ||
| "toc": { | ||
| "base_numbering": 1, | ||
| "nav_menu": {}, | ||
| "number_sections": false, | ||
| "sideBar": true, | ||
| "skip_h1_title": false, | ||
| "title_cell": "Table of Contents", | ||
| "title_sidebar": "Contents", | ||
| "toc_cell": false, | ||
| "toc_position": {}, | ||
| "toc_section_display": true, | ||
| "toc_window_display": false | ||
| }, | ||
| "varInspector": { | ||
| "cols": { | ||
| "lenName": 16, | ||
| "lenType": 16, | ||
| "lenVar": 40 | ||
| }, | ||
| "kernels_config": { | ||
| "python": { | ||
| "delete_cmd_postfix": "", | ||
| "delete_cmd_prefix": "del ", | ||
| "library": "var_list.py", | ||
| "varRefreshCmd": "print(var_dic_list())" | ||
| }, | ||
| "r": { | ||
| "delete_cmd_postfix": ") ", | ||
| "delete_cmd_prefix": "rm(", | ||
| "library": "var_list.r", | ||
| "varRefreshCmd": "cat(var_dic_list()) " | ||
| } | ||
| }, | ||
| "types_to_exclude": [ | ||
| "module", | ||
| "function", | ||
| "builtin_function_or_method", | ||
| "instance", | ||
| "_Feature" | ||
| ], | ||
| "window_display": false | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
Oops, something went wrong.