Upgrade ELK stack to v8.x (#75)

* Upgrade ELK stack to v8.x

Upgrades to v7.17 before v8.x

* Fix agent installer, add python container, secure curl requests

* Fix snapshot perms, image tagging, post-install message

* Fix version check

---------

Co-authored-by: naomi <[email protected]>

Author: caffeinatedpixel

ELK_VERSIONS

@@ -0,0 +1,2 @@
+7.17.9
+8.7.0

README.md

@@ -23,7 +23,7 @@ After Sysmon starts sending data to ElasticSearch, Kibana will be ready to go. F
 ## Installation
 
 ### BeaKer Server System Requirements
-* Operating System: The preferred platform is x86 64-bit Ubuntu 16.04 LTS. The system should be patched and up to date using apt-get.
+* Operating System: The preferred platform is x86 64-bit Ubuntu 20.04 LTS. The system should be patched and up to date using apt-get.
   * The automated installer will also support CentOS 7.
 * Processor: Two or more cores. Elasticsearch uses parallel processing and benefits from more CPU cores.
 * Memory: 8-64GB. Monitoring more hosts requires more RAM.
@@ -32,18 +32,23 @@ After Sysmon starts sending data to ElasticSearch, Kibana will be ready to go. F
 ### BeaKer Agent System Requirements
 * Operating System: Windows x86-64 bit OS
 * Powershell Version: 3+
+* Installed WinLogBeats version must be <= the Elasticsearch version installed on the BeaKer server, but at least the minimum supported wire version for the Elasticsearch version
+  * Elasticsearch v8.6.2 supports WinLogBeats 7.17.0 through 8.6.2 
+  * Elasticsearch v7.17.9 supports WinLogBeats 6.8.0 through 7.17.9
 
 ### Automated Install: BeaKer Server
 
 Download the [latest release](https://github.com/activecm/BeaKer/releases/latest) tar file, extract it, and inside the `BeaKer` directory,
 run `./install_beaker.sh` on the Linux machine that will aggregate your Sysmon data and host Kibana.
 
+** Note that existing BeaKer installations must be upgraded to v7.17 before they can be upgraded to v8.x.
 The automated installer will:
   - Install Docker and Docker-Compose
   - Create a configuration directory in `/etc/BeaKer`
   - Install Elasticsearch, Kibana, and load the dashboards
   - Set the Elasticsearch superuser password for the `elastic` account
   - Set the `sysmon-ingest` user password for connecting WinLogBeats
+  - Set up index templates, ILM policy, data streams and ingest pipelines 
 
 The `beaker` script installed to `/usr/local/bin/beaker` is a wrapper around `docker-compose` and can be used to manage BeaKer.
  - To stop BeaKer, run `beaker down`
@@ -55,7 +60,11 @@ After running `./install_beaker.sh` you should be able to access Kibana at `loca
 
 Use the `elastic` account to perform your initial login to Kibana. Additional user accounts can be created using the Kibana interface. The `sysmon-ingest` user account is not allowed to access Kibana.
 
-The Elasticsearch server will begin listening for connections on port 9200 using HTTPS. It expects Sysmon ID 3 Network Events to be published to the ES index `sysmon-%{+YYYY.MM.dd}` using the WinLogBeat schema. See the embedded `winlogbeat.yml` file in `./agent/install-sysmon-beats.ps1` for more info.
+The Elasticsearch server will begin listening for connections on port 9200 using HTTPS. It expects Sysmon ID 3 Network Events to be published to:
+- WinLogBeats less than v7.17.9: ES index `sysmon-%{+YYYY.MM.dd}`
+- WinLogBeats v7.17.9: ES index `winlogbeat-%{[agent.version]}` via data stream
+- WinLogBeats v8.6.2: Ingest Pipeline `winlogbeat-%{[agent.version]}-routing`
+See the embedded `winlogbeat.yml` file in `./agent/install-sysmon-beats.ps1` for more info.
 
 The easiest way to begin sending data to the server is to use the automated BeaKer agent installer.
 

agent/install-sysmon-beats.ps1

@@ -29,13 +29,20 @@ the script will ask for the password at runtime. In order to avoid recording the
 password, consider editing this file. Change the line `[string]$ESPassword="",` to
 `[string]$ESPassword="YOUR_ELASTIC_PASSWORD_HERE",.
 
+.PARAMETER BeatsVersion
+The version of Winlogbeat to install. This will override any logic that handles upgrading to an
+intermediate version of Winlogbeat before upgrading to a higher major version.
+
 .EXAMPLE
 # Asks for Elasticsearch authentication details at runtime
 .\install-sysmon-beats.ps1 my-es-host.com 9200
 
 # Reads Elasticsearch authentication details from the command line aguments
 .\install-sysmon-beats.ps1 my-es-host.com 9200 elastic elastic_password
 
+# Overrides the version of Winlogbeat to install
+.\install-sysmon-beats.ps1 my-es-host.com 9200 elastic elastic_password 8.6.2
+
 .NOTES
 The Elasticsearch credentials are stored locally using Elastic Winlogbeat's secure
 storage facilities. The ESUsername and ESPassword parameters should not be passed
@@ -44,38 +51,67 @@ enter the credentials during the installation process, or edit the parameters' d
 #>
 
 param (
-    [Parameter(Mandatory=$true)][string]$ESHost,
-    [string]$ESPort="9200",
-    [string]$ESUsername="",
-    [string]$ESPassword=""
+    [Parameter(Mandatory = $true)][string]$ESHost,
+    [string]$ESPort = "9200",
+    [string]$ESUsername = "",
+    [string]$ESPassword = "",
+    [string]$BeatsVersion = ""
 )
 
-if (-NOT ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator"))  
-{  
+$ELK_STACK_VERSION = "8.7.0"
+
+if (-NOT ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")) {  
     # Use param values instead of $args because $args doesn't appear to get populated if param values are specified
     # Also set the ExecutionPolicy to Bypass otherwise this will likely fail as script
     # execution is disabled by default.
     $arguments = "-ExecutionPolicy", "Bypass", "-File", $myinvocation.mycommand.definition, $ESHost, $ESPort
-    if($ESUsername) 
-    {
+    if ($ESUsername) {
         # Only add this argument if the user provided it, otherwise it will be blank and will cause an error
         $arguments += $ESUsername
     }
-    if($ESPassword) 
-    {
+    if ($ESPassword) {
         # Only add this argument if the user provided it, otherwise it will be blank and will cause an error
         $arguments += $ESPassword
     }
+    if ($BeatsVersion) {
+        # Only add this argument if the user provided it, otherwise it will be blank and will cause an error
+        $arguments += $BeatsVersion
+    }
 
     Start-Process -FilePath powershell -Verb runAs -ArgumentList $arguments
     Break
 }
 
+
+[bool] $OverrideBeatsVersion = $false
+if ([string]::IsNullOrWhiteSpace("$BeatsVersion")) {
+    $BeatsVersion = "$ELK_STACK_VERSION"
+}
+else {
+    if ($null -eq ("$BeatsVersion" -as [System.Version])) {
+        throw "Beats version $BeatsVersion is not a valid version, please provide a valid version number."
+    }
+    if ([System.Version]$BeatsVersion -lt [System.Version]"7.17.9") {
+        throw "Minimum supported Beats version is 7.17.9, exiting"
+    }
+    $OverrideBeatsVersion = $true
+}
+
+# Check for existing winlogbeat installation via Espy
+if (Test-Path "$Env:programfiles\Winlogbeat-Espy" -PathType Container) {
+    Write-Output "Detected existing winlogbeat installation performed by Espy. Continuing the install may result in a partially working Sysmon/winlogbeat setup."
+    $installAnyway = Read-Host -Prompt "Are you sure you want to continue? [y/n]"
+    if (($installAnyway -eq 'n') -or ($installAnyway -eq 'N')) {
+        Exit
+    }
+}
+
 if (-not (Test-Path "$Env:programfiles\Sysmon" -PathType Container)) {
-  Invoke-WebRequest -OutFile Sysmon.zip https://download.sysinternals.com/files/Sysmon.zip
-  Expand-Archive .\Sysmon.zip
-  rm .\Sysmon.zip
-  mv .\Sysmon\ "$Env:programfiles"
+    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
+    Invoke-WebRequest -OutFile Sysmon.zip https://download.sysinternals.com/files/Sysmon.zip
+    Expand-Archive .\Sysmon.zip
+    rm .\Sysmon.zip
+    mv .\Sysmon\ "$Env:programfiles"
 }
 
 echo @"
@@ -154,32 +190,109 @@ echo @"
 
 & "$Env:programfiles\Sysmon\Sysmon64.exe" -accepteula -i "$Env:programfiles\Sysmon\sysmon-net-only.xml"
 
-if (-not (Test-Path "$Env:programfiles\winlogbeat*" -PathType Container)) {
-  Invoke-WebRequest -OutFile WinLogBeat.zip https://artifacts.elastic.co/downloads/beats/winlogbeat/winlogbeat-7.5.2-windows-x86_64.zip
-  Expand-Archive .\WinLogBeat.zip
-  rm .\WinLogBeat.zip
-  mv .\WinLogBeat\winlogbeat* "$Env:programfiles"
+$InstalledBeatsVersion = ""
+[bool] $DownloadWinlogbeat = $false
+
+# Check for fresh install or pre-7.17 install
+if (-not (Test-Path "$Env:programfiles\Winlogbeat-BeaKer\winlogbeat.exe" -PathType Leaf)) {
+    $DownloadWinlogbeat = $true
+  
+    # Create install directory if it doesn't exist
+    if (-not (Test-Path "$Env:programfiles\Winlogbeat-BeaKer" -PathType Container)) {
+        mkdir "$Env:programfiles\Winlogbeat-BeaKer" > $null
+    }
+  
+    # Check if this is a pre-7.17 upgrade install
+    if ((Test-Path "$Env:programfiles\winlogbeat-7*" -PathType Container)) {
+        ### Make sure that Beats is upgraded to 7.17 before installing v8.x
+        # Install winlogbeat 7.17.9 if the current version is less than 8.x
+        if (!$OverrideBeatsVersion) {
+            $BeatsVersion = "7.17.9"
+        }
+        Copy-Item "$Env:programfiles\winlogbeat-7*\winlogbeat.yml" "$Env:programfiles\Winlogbeat-BeaKer"
+    }
+}
+else {
+    # Check if currently installed version is outdated
+    $InstalledBeatsVersion = (& "$Env:programfiles\Winlogbeat-BeaKer\winlogbeat.exe" version | Select-String -Pattern "(?<=winlogbeat version )(\d+\.\d+\.\d+)").Matches.Value
+    if ($null -eq ("$InstalledBeatsVersion" -as [System.Version])) {
+  
+        if (!$OverrideBeatsVersion) {
+            throw "Unable to retrieve installed winlogbeat version"
+        }
+        else {
+            Write-Output "Unable to retrieve installed winlogbeat version, continuing anyway"
+            $DownloadWinlogbeat = $true
+        }
+    }
+    else {
+        if ([System.Version]"$InstalledBeatsVersion" -lt [System.Version]"$BeatsVersion") {
+            $DownloadWinlogbeat = $true
+        }
+    }
+}
+    
+# Download winlogbeat and move it to install directory
+if ($DownloadWinlogbeat) {
+    Write-Output "######## Downloading winlogbeat version $BeatsVersion ########"
+
+    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
+    Invoke-WebRequest -OutFile WinLogBeat.zip https://artifacts.elastic.co/downloads/beats/winlogbeat/winlogbeat-"$BeatsVersion"-windows-x86_64.zip
+    Expand-Archive .\WinLogBeat.zip
+    rm .\WinLogBeat.zip
+    rm .\WinLogBeat\winlogbeat*\winlogbeat.yml
+
+    # Stop winlogbeat service if it exists 
+    if (Get-Service winlogbeat -ErrorAction SilentlyContinue) {
+        Stop-Service winlogbeat
+        (Get-Service winlogbeat).WaitForStatus('Stopped')
+        Start-Sleep -s 1
+    }
+    Copy-Item -Path .\WinLogBeat\winlogbeat*\* -Destination "$Env:programfiles\Winlogbeat-BeaKer\" -Recurse -Force
+    rm .\Winlogbeat -Recurse
+}
+
+Write-Output "######## Installing winlogbeat version $BeatsVersion ########"
+
+
+# Begin winlogbeat configuration
+Set-Location "$Env:programfiles\Winlogbeat-BeaKer\"
+
+# Backup winlogbeat config if it exists
+if (Test-Path -PathType Leaf .\winlogbeat.yml) {
+    if ($DownloadWinlogbeat) {
+        # Backup config with its version in the name if upgrading to a new Beats version
+        # so that the config isn't overwritten by subsequent upgrades. This is useful in case
+        # breaking changes between configurations need to be referenced in the future for troubleshooting
+        Copy-Item .\winlogbeat.yml .\winlogbeat-$InstalledBeatsVersion-old.yml.bak
+    }
+    else {
+        Copy-Item .\winlogbeat.yml .\winlogbeat.yml.bak
+    }
 }
 
-cd "$Env:programfiles\winlogbeat*\"
 .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore create
-if($ESUsername) {
-  Write-Output "$ESUsername" | .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_USERNAME --stdin
-} else {
-  .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_USERNAME
+if ($ESUsername) {
+    Write-Output "$ESUsername" | .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_USERNAME --stdin
 }
-if($ESPassword) {
-  Write-Output "$ESPassword" | .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_PASSWORD --stdin
-} else {
-  .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_PASSWORD
+else {
+    .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_USERNAME
+}
+if ($ESPassword) {
+    Write-Output "$ESPassword" | .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_PASSWORD --stdin
+}
+else {
+    .\winlogbeat.exe --path.data "C:\ProgramData\winlogbeat" keystore add ES_PASSWORD
 }
 
 # Set ACL's of the $Env:ProgramData\winlogbeat folder to be the same as $Env:ProgramFiles\winlogbeat* (the main install path)
 # This helps ensure that "normal" users aren't able to access the $Env:ProgramData\winlogbeat folder
-Get-ACL -Path "$Env:ProgramFiles\winlogbeat*" | Set-ACL -Path "$Env:ProgramData\winlogbeat"
+Get-ACL -Path "$Env:ProgramFiles\\Winlogbeat-BeaKer*" | Set-ACL -Path "$Env:ProgramData\\winlogbeat"
 
 rm .\winlogbeat.yml
-echo @"
+
+if ([System.Version]$BeatsVersion -lt [System.Version]"8.0.0") {
+    Write-Output @"
 winlogbeat.event_logs:
   - name: Microsoft-Windows-Sysmon/Operational
     event_id: 3
@@ -190,19 +303,40 @@ winlogbeat.event_logs:
           file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
 
 setup.ilm.enabled: false
-setup.template.enabled: true
-setup.template.name: `"sysmon`"
-setup.template.pattern: `"sysmon-*`"
+setup.template.name: `"winlogbeat-%{[agent.version]}`"
+setup.template.pattern: `"winlogbeat-%{[agent.version]}`"
 
 output.elasticsearch:
   hosts:
     - https://${ESHost}:${ESPort}
-  index: `"sysmon-%{+YYYY.MM.dd}`"
+  index: `"winlogbeat-%{[agent.version]}`"
   username: `"`${ES_USERNAME}`"
   password: `"`${ES_PASSWORD}`"
   ssl:
     enabled: true
     verification_mode: none
 "@ > winlogbeat.yml
+}
+else {
+    Write-Output @"
+winlogbeat.event_logs:
+    - name: Microsoft-Windows-Sysmon/Operational
+      event_id: 3
+
+setup.ilm.enabled: false
+
+output.elasticsearch:
+  hosts:
+    - https://${ESHost}:${ESPort}
+  pipeline: winlogbeat-%{[agent.version]}-routing
+  username: `"`${ES_USERNAME}`"
+  password: `"`${ES_PASSWORD}`"
+  ssl:
+    enabled: true
+    verification_mode: none
+"@ > winlogbeat.yml
+}
+
 PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-winlogbeat.ps1
+
 Start-Service winlogbeat

check_kibana/Dockerfile

@@ -0,0 +1,3 @@
+FROM python:3.9
+ADD check_kibana.py .
+CMD ["python3", "./check_kibana.py"] 

check_kibana/check_kibana.py

@@ -0,0 +1,58 @@
+import sys
+import json
+
+# check kibana status for v8.x
+
+
+def check_kibana_migrations_v8(status):
+    if "core" in status:
+        if "savedObjects" in status["core"]:
+            if "summary" in status["core"]["savedObjects"]:
+                if "completed migrations" in status["core"]["savedObjects"]["summary"] \
+                        and "available" in status["core"]["savedObjects"]["summary"]:
+                    sys.exit(0)
+
+    sys.exit(2)
+
+
+def check_kibana_status_v8(status):
+    if "overall" in status:
+        if "level" in status["overall"]:
+            if status["overall"]["level"] == "available":
+                check_kibana_migrations_v8(status)
+    sys.exit(1)
+
+# check kibana status for v7.17
+
+
+def check_kibana_status(status):
+    if "overall" in status:
+        if "state" in status["overall"]:
+            if status["overall"]["state"] == "green":
+                check_migration_status(status)
+    check_kibana_status_v8(status)
+
+
+def check_migration_status(status):
+    if "statuses" in status:
+        for plugin in status["statuses"]:
+            if "core:savedObjects" in plugin["id"]:
+                if "message" in plugin:
+                    if "completed migrations" in plugin["message"] \
+                            and "available" in plugin["message"]:
+                        sys.exit(0)
+    sys.exit(2)
+
+
+# takes piped in curl output from https://localhost:5601/api/status
+# and checks whether or not kibana is online and finished with data migrations (upgrades)
+try:
+    response = json.load(sys.stdin)
+except ValueError:
+    sys.exit(4)
+
+if "status" in response:
+    status = response["status"]
+    check_kibana_status(status)
+
+sys.exit(3)