Malcolm v4.0.0 development merge (#177)

Malcolm v4.0.0 development merge

Malcolm v4.0.0 consists of a major restructuring of the underlying data schema used to represent Zeek logs (and, going forward, logs from other data sources) in the Elasticsearch data store. As the Malcolm project uses [semantic versioning](https://semver.org/) when choosing version numbers, this backwards-compatibility breaking change is the reason for bumping the major version number from 3 to 4 despite no significant new functionality being introduced.

The details of the drivers behind this change can be found at idaholab#64 and idaholab#16. This change, though somewhat painful, will make it easier to integrate more data sources into Malcolm in the future and potentially makes Malcolm's network session data more compatible with other tools that use the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html).

v3.4.0...v4.0.0

**BREAKING CHANGES:**

* as *many* field names have changed, custom saved dashboards and/or bookmarks to Kibana or Arkime visualizations may need to be adjusted accordingly
* old network session data (stored in the `sessions2-*` indices in Elasticsearch) will not be visible (as the indices are now named `arkime-sessions3-*`)

A fresh install of Malcolm is recommended with this release. Upgrading from previous versions of Malcolm to v4.0.0+ is not suggested.

Changes:

* added GitHub [workflow files](./.github/workflows/) which contain instructions for GitHub to build the docker images and [sensor](#Hedgehog) and [Malcolm](#ISO) installer ISOs.
* moved many fields that were named zeek-specific to generic [ECS](https://www.elastic.co/guide/en/ecs/current/index.html)-specified (or at least "ECS-inspired") field names, updating related parsing code and dashboard definitions
* changed Zeek-specific field naming schema (e.g., `zeek_foo.bar` becomes `zeek.foo.bar`)
* added  Corelight's [Microsoft Excel privilege escalation detection (CVE-2021-42292)](https://github.com/corelight/CVE-2021-42292) plugin
* integrated updates to the LDAP parser which improve the detail given from observed LDAP searches
* improved and genericized [the code](./logstash/ruby/mac_lookup.rb) for mapping MAC addresses to vendor OUIs, replacing the use of [logstash-filter-ieee_oui](https://github.com/mmguero-dev/logstash-filter-ieee_oui)
* updated some Dockerfiles to use Debian 11 "bullseye" instead of Debian 10 "buster"

.github/workflows/arkime-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'arkime/**'
       - 'Dockerfiles/arkime.Dockerfile'

.github/workflows/elasticsearch-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'Dockerfiles/elasticsearch.Dockerfile'
       - 'shared/bin/*'

.github/workflows/file-monitor-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'file-monitor/**'
       - 'Dockerfiles/file-monitor.Dockerfile'

.github/workflows/file-upload-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'file-upload/**'
       - 'Dockerfiles/file-upload.Dockerfile'

.github/workflows/filebeat-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'filebeat/**'
       - 'Dockerfiles/filebeat.Dockerfile'

.github/workflows/freq-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'freq-server/**'
       - 'Dockerfiles/freq.Dockerfile'

.github/workflows/htadmin-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'htadmin/**'
       - 'Dockerfiles/htadmin.Dockerfile'

.github/workflows/kibana-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'kibana/**'
       - 'Dockerfiles/kibana.Dockerfile'

.github/workflows/kibana-helper-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'kibana/**'
       - 'Dockerfiles/kibana-helper.Dockerfile'

.github/workflows/logstash-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'logstash/**'
       - 'Dockerfiles/logstash.Dockerfile'

.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml

@@ -0,0 +1,116 @@
+name: malcolm-iso-build-docker-wrap-push-ghcr
+
+on:
+  push:
+    branches:
+      - main
+      - development
+    paths:
+      - 'malcolm-iso/**'
+      - 'shared/bin/*'
+  workflow_dispatch:
+  repository_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    permissions:
+      actions: write
+      packages: write
+      contents: read
+    defaults:
+      run:
+        shell: bash
+    steps:
+      -
+        name: Cancel previous run in progress
+        uses: styfle/[email protected]
+        with:
+          ignore_sha: true
+          all_but_latest: true
+          access_token: ${{ secrets.GITHUB_TOKEN }}
+      -
+        name: Set up QEMU
+        uses: docker/setup-qemu-action@v1
+      -
+        name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v1
+      -
+        name: Build environment setup
+        run: |
+          sudo apt-get -q update
+          sudo env DEBIAN_FRONTEND=noninteractive apt-get install -t focal-backports --no-install-recommends -y -q \
+               apt-transport-https \
+               bc \
+               build-essential \
+               ca-certificates \
+               curl \
+               debhelper-compat \
+               debian-archive-keyring \
+               debootstrap \
+               genisoimage \
+               gettext \
+               git \
+               gnupg2 \
+               imagemagick \
+               jq \
+               pandoc \
+               po4a \
+               rsync \
+               software-properties-common \
+               squashfs-tools \
+               virt-what \
+               xorriso
+            git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20210407' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
+            cd /tmp/live-build
+            dpkg-buildpackage -b -uc -us
+            sudo dpkg -i /tmp/live-build*.deb
+      -
+        name: Checkout
+        uses: actions/checkout@v2
+      -
+        name: Extract branch name
+        shell: bash
+        run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
+        id: extract_branch
+      -
+        name: Extract short hash
+        shell: bash
+        run: echo "##[set-output name=hash;]$(git rev-parse --short $GITHUB_SHA)"
+        id: extract_short_hash
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "##[set-output name=mversion;]$(grep -P "^\s+image:\s*malcolm" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)"
+        id: extract_malcolm_version
+      -
+        name: Build image
+        run: |
+          IMAGES=( $(grep image: docker-compose.yml | awk '{print $2}') )
+          for IMAGE in "${IMAGES[@]}"; do
+            REPO_IMAGE="$(echo "$IMAGE" | sed "s@^\(malcolmnetsec\)@ghcr.io/${{ github.repository_owner }}/\1@" | sed "s/:.*/:${{ steps.extract_branch.outputs.branch }}/")"
+            docker pull "$REPO_IMAGE" && \
+              docker tag "$REPO_IMAGE" "$IMAGE" && \
+              docker rmi "$REPO_IMAGE"
+          done
+          DEST_IMAGES_TGZ=$(pwd)/malcolm_"$(date +%Y.%m.%d_%H:%M:%S)"_${{ steps.extract_short_hash.outputs.hash }}_images.tar.gz
+          docker save "${IMAGES[@]}" | gzip > "$DEST_IMAGES_TGZ"
+          pushd ./malcolm-iso
+          sudo /usr/bin/env bash ./build.sh -d "$DEST_IMAGES_TGZ"
+          rm -rf ./shared/
+          sudo chmod 644 ./malcolm-*.*
+          popd
+      -
+        name: ghcr.io login
+        uses: docker/login-action@v1
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      -
+        name: Build and push ISO image
+        uses: docker/build-push-action@v2
+        with:
+          context: ./malcolm-iso
+          push: true
+          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/malcolm:${{ steps.extract_branch.outputs.branch }}

.github/workflows/name-map-ui-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'name-map-ui/**'
       - 'Dockerfiles/name-map-ui.Dockerfile'

.github/workflows/nginx-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'nginx/**'
       - 'Dockerfiles/nginx.Dockerfile'

.github/workflows/pcap-capture-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'pcap-capture/**'
       - 'Dockerfiles/pcap-capture.Dockerfile'

.github/workflows/pcap-monitor-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'pcap-monitor/**'
       - 'Dockerfiles/pcap-monitor.Dockerfile'

.github/workflows/sensor-iso-build-docker-wrap-push-ghcr.yml

@@ -0,0 +1,105 @@
+name: sensor-iso-build-docker-wrap-push-ghcr
+
+on:
+  push:
+    branches:
+      - main
+      - development
+    paths:
+      - 'sensor-iso/**'
+      - 'shared/bin/*'
+  workflow_dispatch:
+  repository_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    permissions:
+      actions: write
+      packages: write
+      contents: read
+    defaults:
+      run:
+        shell: bash
+    steps:
+      -
+        name: Cancel previous run in progress
+        uses: styfle/[email protected]
+        with:
+          ignore_sha: true
+          all_but_latest: true
+          access_token: ${{ secrets.GITHUB_TOKEN }}
+      -
+        name: Set up QEMU
+        uses: docker/setup-qemu-action@v1
+      -
+        name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v1
+      -
+        name: Build environment setup
+        run: |
+          sudo apt-get -q update
+          sudo env DEBIAN_FRONTEND=noninteractive apt-get install -t focal-backports --no-install-recommends -y -q \
+               apt-transport-https \
+               bc \
+               build-essential \
+               ca-certificates \
+               curl \
+               debhelper-compat \
+               debian-archive-keyring \
+               debootstrap \
+               genisoimage \
+               gettext \
+               git \
+               gnupg2 \
+               imagemagick \
+               jq \
+               pandoc \
+               po4a \
+               rsync \
+               software-properties-common \
+               squashfs-tools \
+               virt-what \
+               xorriso
+            git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20210407' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
+            cd /tmp/live-build
+            dpkg-buildpackage -b -uc -us
+            sudo dpkg -i /tmp/live-build*.deb
+      -
+        name: Checkout
+        uses: actions/checkout@v2
+      -
+        name: Extract branch name
+        shell: bash
+        run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
+        id: extract_branch
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "##[set-output name=mversion;]$(grep -P "^\s+image:\s*malcolm" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)"
+        id: extract_malcolm_version
+      -
+        name: Build image
+        run: |
+          cp -r ./shared ./sensor-iso
+          pushd ./sensor-iso
+          echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
+          echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt
+          sudo /usr/bin/env bash ./build.sh
+          rm -rf ./shared/
+          sudo chmod 644 ./hedgehog-*.*
+          popd
+      -
+        name: ghcr.io login
+        uses: docker/login-action@v1
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      -
+        name: Build and push ISO image
+        uses: docker/build-push-action@v2
+        with:
+          context: ./sensor-iso
+          push: true
+          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/hedgehog:${{ steps.extract_branch.outputs.branch }}

.github/workflows/zeek-build-and-push-ghcr.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - development
     paths:
       - 'zeek/**'
       - 'Dockerfiles/zeek.Dockerfile'

Dockerfiles/kibana-helper.Dockerfile

@@ -52,7 +52,7 @@ ADD kibana/anomaly_detectors /opt/kibana/anomaly_detectors
 ADD kibana/maps /opt/maps
 ADD kibana/scripts /data/
 ADD kibana/supervisord.conf /etc/supervisord.conf
-ADD kibana/zeek_template.json /data/zeek_template.json
+ADD kibana/malcolm_template.json /data/malcolm_template.json
 ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
 ADD shared/bin/elastic_search_status.sh /data/
 ADD shared/bin/elastic_index_size_prune.py /data/
@@ -73,7 +73,7 @@ RUN apk --no-cache add bash python3 py3-pip curl procps psmisc npm shadow jq &&
     chown -R ${PUSER}:${PGROUP} /opt/kibana/dashboards /opt/maps /data/init /opt/kibana/anomaly_detectors && \
     chmod 755 /data/*.sh /data/*.py /data/init && \
     chmod 400 /opt/maps/* && \
-    (echo -e "*/2 * * * * /data/kibana-create-arkime-sessions-index.sh\n0 10 * * * /data/kibana_index_refresh.py --template zeek_template\n*/20 * * * * /data/elastic_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
+    (echo -e "*/2 * * * * /data/kibana-create-arkime-sessions-index.sh\n0 10 * * * /data/kibana_index_refresh.py --template malcolm_template\n*/20 * * * * /data/elastic_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
 
 EXPOSE $KIBANA_OFFLINE_REGION_MAPS_PORT