cloudflare/Azure-Sentinel
Publicmirrored fromhttps://github.com/cloudflare/Azure-Sentinel
Notebooks/Get Started.ipynb
238lines · modecode
7 years ago
| 1 | { |
| 2 | "cells": [ |
| 3 | { |
| 4 | "metadata": { |
| 5 | "collapsed": true |
| 6 | }, |
| 7 | "cell_type": "markdown", |
| 8 | "source": "# Get Started\n\n__Notebook Version:__ 1.0<br>\n__Python Version:__ Python 3.6 (including Python 3.6 - AzureML)<br>\n__Required Packages:__ Kqlmagic 0.1.90<br>\n__Platforms Supported:__<br>\n - Azure Notebooks Free Compute\n - Azure Notebooks DSVM\n__Data Source Required:__<br>\n - Log Analytics - SecurityEvent\n \n### Description\nThe notebook provides you basic knowledge to use Azure Sentinel Notebooks.\n\n<font color=red>When you switch between Azure Notebooks Free Compute and Data Science Virtual Machine (DSVM), you may need to select Python version: please select Python 3.6 for Free Compute, and Python 3.6 - AzureML for DSVM.</font>" |
| 9 | }, |
| 10 | { |
| 11 | "metadata": {}, |
| 12 | "cell_type": "markdown", |
| 13 | "source": "## Prerequisite check" |
| 14 | }, |
| 15 | { |
| 16 | "metadata": { |
| 17 | "trusted": true |
| 18 | }, |
| 19 | "cell_type": "code", |
| 20 | "source": "# only run once, current version 0.1.2\n!pip install --upgrade Sentinel-Utilities", |
| 21 | "execution_count": null, |
| 22 | "outputs": [] |
| 23 | }, |
| 24 | { |
| 25 | "metadata": { |
| 26 | "trusted": true |
| 27 | }, |
| 28 | "cell_type": "code", |
| 29 | "source": "import SentinelUtils\n# checking Python version\ncheck = SentinelUtils.version_management.ModuleVersionCheck()\npy_check = check.validate_python('3.6.0')\nif py_check.requirement_met == False:\n print('Please select Python 3.6 or Python 3.6 - AzureML at the upper right corner')\nelse:\n print('Please continue')", |
| 30 | "execution_count": null, |
| 31 | "outputs": [] |
| 32 | }, |
| 33 | { |
| 34 | "metadata": { |
| 35 | "trusted": true |
| 36 | }, |
| 37 | "cell_type": "code", |
| 38 | "source": "# checking required packages\nmods_check = check.validate_installed_modules(['Kqlmagic>=0.1.90'])\nfor mod_info in mods_check:\n if mod_info.requirement_met == False:\n print('Please install {} {} at the following cell.'.format(mod_info.name, mod_info.required_version))", |
| 39 | "execution_count": null, |
| 40 | "outputs": [] |
| 41 | }, |
| 42 | { |
| 43 | "metadata": { |
| 44 | "trusted": true |
| 45 | }, |
| 46 | "cell_type": "code", |
| 47 | "source": "# Please install required packages based on the check at last cellp\n#!pip install Kqlmagic --upgrade", |
| 48 | "execution_count": null, |
| 49 | "outputs": [] |
| 50 | }, |
| 51 | { |
| 52 | "metadata": {}, |
| 53 | "cell_type": "markdown", |
| 54 | "source": "## Table of Contents\n\n1. Retrieve Log Analytics information \n2. Log into Log Analytics\n3. Get a list of available data tables from Log Analytics\n4. Run a simple Kusto query" |
| 55 | }, |
| 56 | { |
| 57 | "metadata": {}, |
| 58 | "cell_type": "markdown", |
| 59 | "source": "## 1. Retrieve Log Analytics Information" |
| 60 | }, |
| 61 | { |
| 62 | "metadata": { |
| 63 | "trusted": true |
| 64 | }, |
| 65 | "cell_type": "code", |
| 66 | "source": "path = %env PATH\ndsvm = False\nif '/dsvm/' in path:\n dsvm = True\n \nif dsvm == False:\n # Run this if you are using Free Compute\n tenant_id = SentinelUtils.config_reader.ConfigReader.read_config_values(\"config.json\")[0]\n subscription_id = SentinelUtils.config_reader.ConfigReader.read_config_values(\"config.json\")[1]\n resource_group = SentinelUtils.config_reader.ConfigReader.read_config_values(\"config.json\")[2]\n workspace_id = SentinelUtils.config_reader.ConfigReader.read_config_values(\"config.json\")[3]\n workspace_name = SentinelUtils.config_reader.ConfigReader.read_config_values(\"config.json\")[4]\n print('Your Log Analytic Workspace: {}'.format(workspace_name))\nelse:\n # Run this if you are using DSVM. You need to copy the values from config.json, if the file has no value, then you need to go to Log Analytics Portal to get the information.\n tenant_id = input('tenant_id:')\n subscription_id = input('subscription_id:')\n resource_group = input('resource_group:')\n workspace_id = input('workspace_id:')\n workspace_name = input('workspace_name:')", |
| 67 | "execution_count": null, |
| 68 | "outputs": [] |
| 69 | }, |
| 70 | { |
| 71 | "metadata": {}, |
| 72 | "cell_type": "markdown", |
| 73 | "source": "## 2. Log into Log Analytics" |
| 74 | }, |
| 75 | { |
| 76 | "metadata": { |
| 77 | "trusted": true |
| 78 | }, |
| 79 | "cell_type": "code", |
| 80 | "source": "# You must run this cell to log into Log Analytics to continue\n# Make sure you have 0.1.90 or above, if not, run Kqlmagic installation again\n%reload_ext Kqlmagic\n%kql loganalytics://code;workspace=workspace_id;tenant=tenant_id;alias=\"SentinelDB\"", |
| 81 | "execution_count": null, |
| 82 | "outputs": [] |
| 83 | }, |
| 84 | { |
| 85 | "metadata": {}, |
| 86 | "cell_type": "markdown", |
| 87 | "source": "## 3. Get a list of available data tables from Log Analytics" |
| 88 | }, |
| 89 | { |
| 90 | "metadata": {}, |
| 91 | "cell_type": "markdown", |
| 92 | "source": "1. Tables with records" |
| 93 | }, |
| 94 | { |
| 95 | "metadata": { |
| 96 | "trusted": true |
| 97 | }, |
| 98 | "cell_type": "code", |
| 99 | "source": "%kql search * | summarize count() by Type", |
| 100 | "execution_count": null, |
| 101 | "outputs": [] |
| 102 | }, |
| 103 | { |
| 104 | "metadata": {}, |
| 105 | "cell_type": "markdown", |
| 106 | "source": "2. All Tables in drop down" |
| 107 | }, |
| 108 | { |
| 109 | "metadata": { |
| 110 | "scrolled": true, |
| 111 | "trusted": true |
| 112 | }, |
| 113 | "cell_type": "code", |
| 114 | "source": "import ipywidgets as widgets\nfrom IPython.display import display\ndbSchema = %kql --schema \"SentinelDB@loganalytics\"\ntables = list(dbSchema.keys())\nselected_table = widgets.Dropdown(options=tables, value=tables[1],description='Data Table:')\ndisplay(selected_table)", |
| 115 | "execution_count": null, |
| 116 | "outputs": [] |
| 117 | }, |
| 118 | { |
| 119 | "metadata": {}, |
| 120 | "cell_type": "markdown", |
| 121 | "source": "3. Choose columns from the selected table" |
| 122 | }, |
| 123 | { |
| 124 | "metadata": { |
| 125 | "trusted": true |
| 126 | }, |
| 127 | "cell_type": "code", |
| 128 | "source": "columns = list(dbSchema[selected_table.value])\ncolumns.sort()\nselected_columns = widgets.SelectMultiple(options=columns, value=[],description='Data Columns:')\ndisplay(selected_columns)", |
| 129 | "execution_count": null, |
| 130 | "outputs": [] |
| 131 | }, |
| 132 | { |
| 133 | "metadata": {}, |
| 134 | "cell_type": "markdown", |
| 135 | "source": "4. List unique values for selected columns" |
| 136 | }, |
| 137 | { |
| 138 | "metadata": { |
| 139 | "trusted": true |
| 140 | }, |
| 141 | "cell_type": "code", |
| 142 | "source": "%kql {selected_table.value} | where TimeGenerated >= ago(30d) | take 1000\nif len(_kql_raw_result_) > 0:\n df = _kql_raw_result_.to_dataframe()\nelse:\n df = None\n print('no data')", |
| 143 | "execution_count": null, |
| 144 | "outputs": [] |
| 145 | }, |
| 146 | { |
| 147 | "metadata": { |
| 148 | "trusted": true |
| 149 | }, |
| 150 | "cell_type": "code", |
| 151 | "source": "for index, column in enumerate(selected_columns.value):\n unique_values = df[column].drop_duplicates().sort_values()\n values_dropdown = widgets.Dropdown(options=unique_values, description=column)\n display(values_dropdown)", |
| 152 | "execution_count": null, |
| 153 | "outputs": [] |
| 154 | }, |
| 155 | { |
| 156 | "metadata": {}, |
| 157 | "cell_type": "markdown", |
| 158 | "source": "5. Search a term" |
| 159 | }, |
| 160 | { |
| 161 | "metadata": { |
| 162 | "trusted": true |
| 163 | }, |
| 164 | "cell_type": "code", |
| 165 | "source": "# scope to a table and a column\n%kql search in ({selected_table.value}) {selected_columns.value[0]}: '*local service' | take 5", |
| 166 | "execution_count": null, |
| 167 | "outputs": [] |
| 168 | }, |
| 169 | { |
| 170 | "metadata": {}, |
| 171 | "cell_type": "markdown", |
| 172 | "source": "## 4. Run a simple Kusto query" |
| 173 | }, |
| 174 | { |
| 175 | "metadata": { |
| 176 | "trusted": true |
| 177 | }, |
| 178 | "cell_type": "code", |
| 179 | "source": "# Checking required Data Sources\nrequired_data_sources = ['SecurityEvent']\ndbSchema = %kql --schema \"SentinelDB@loganalytics\"\ntables = list(dbSchema.keys())\nfor source in required_data_sources:\n if source not in tables:\n print('You do not have required data source: {}'.format(source))\n else:\n print('Please continue')", |
| 180 | "execution_count": null, |
| 181 | "outputs": [] |
| 182 | }, |
| 183 | { |
| 184 | "metadata": { |
| 185 | "trusted": true |
| 186 | }, |
| 187 | "cell_type": "code", |
| 188 | "source": "%kql SecurityEvent | take 5 | project Account, Computer\ndf = _kql_raw_result_.to_dataframe()\ndf", |
| 189 | "execution_count": null, |
| 190 | "outputs": [] |
| 191 | }, |
| 192 | { |
| 193 | "metadata": { |
| 194 | "trusted": true |
| 195 | }, |
| 196 | "cell_type": "code", |
| 197 | "source": "", |
| 198 | "execution_count": null, |
| 199 | "outputs": [] |
| 200 | } |
| 201 | ], |
| 202 | "metadata": { |
| 203 | "celltoolbar": "Tags", |
| 204 | "hide_input": false, |
| 205 | "kernelspec": { |
| 206 | "name": "python36", |
| 207 | "display_name": "Python 3.6", |
| 208 | "language": "python" |
| 209 | }, |
| 210 | "language_info": { |
| 211 | "pygments_lexer": "ipython3", |
| 212 | "name": "python", |
| 213 | "file_extension": ".py", |
| 214 | "codemirror_mode": { |
| 215 | "name": "ipython", |
| 216 | "version": 3 |
| 217 | }, |
| 218 | "nbconvert_exporter": "python", |
| 219 | "mimetype": "text/x-python", |
| 220 | "version": "3.6.7" |
| 221 | }, |
| 222 | "toc": { |
| 223 | "base_numbering": 1, |
| 224 | "nav_menu": {}, |
| 225 | "number_sections": false, |
| 226 | "sideBar": true, |
| 227 | "skip_h1_title": false, |
| 228 | "title_cell": "Table of Contents", |
| 229 | "title_sidebar": "Contents", |
| 230 | "toc_cell": false, |
| 231 | "toc_position": {}, |
| 232 | "toc_section_display": true, |
| 233 | "toc_window_display": false |
| 234 | } |
| 235 | }, |
| 236 | "nbformat": 4, |
| 237 | "nbformat_minor": 2 |
| 238 | } |