Malcolm v5.0.3 minor update

- build with latest zeek/spicy-ldap release - build with latest corelight/cve-2021-44228 release - fix idaholab#69 (zeek resists shutdown on sensor during halt/reboot) - bump OpenSearch to v1.2.2 - added convenience script for working with GitHub workflow-built images Signed-off-by: SG <[email protected]>
cisagov · Dec 16, 2021 · 4e8d21c · 4e8d21c
1 parent 3f6f71c
commit 4e8d21c
Show file tree

Hide file tree

Showing 13 changed files with 238 additions and 67 deletions.
diff --git a/Dockerfiles/dashboards.Dockerfile b/Dockerfiles/dashboards.Dockerfile
@@ -14,7 +14,7 @@ ENV PGROUP "dashboarder"
 
 ENV TERM xterm
 
-ARG OPENSEARCH_VERSION="1.2.1"
+ARG OPENSEARCH_VERSION="1.2.2"
 ENV OPENSEARCH_VERSION $OPENSEARCH_VERSION
 
 ARG OPENSEARCH_DASHBOARDS_VERSION="1.2.0"

diff --git a/Dockerfiles/opensearch.Dockerfile b/Dockerfiles/opensearch.Dockerfile
@@ -1,4 +1,4 @@
-FROM opensearchproject/opensearch:1.2.1
+FROM opensearchproject/opensearch:1.2.2
 
 # Copyright (c) 2022 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"

diff --git a/README.md b/README.md
@@ -143,21 +143,21 @@ You can then observe that the images have been retrieved by running `docker imag
 ```
 $ docker images
 REPOSITORY                                                     TAG             IMAGE ID       CREATED      SIZE
-malcolmnetsec/arkime                                           5.0.2           xxxxxxxxxxxx   2 days ago   811MB
-malcolmnetsec/dashboards                                       5.0.2           xxxxxxxxxxxx   2 days ago   970MB
-malcolmnetsec/dashboards-helper                                5.0.2           xxxxxxxxxxxx   2 days ago   154MB
-malcolmnetsec/filebeat-oss                                     5.0.2           xxxxxxxxxxxx   2 days ago   621MB
-malcolmnetsec/file-monitor                                     5.0.2           xxxxxxxxxxxx   2 days ago   586MB
-malcolmnetsec/file-upload                                      5.0.2           xxxxxxxxxxxx   2 days ago   259MB
-malcolmnetsec/freq                                             5.0.2           xxxxxxxxxxxx   2 days ago   132MB
-malcolmnetsec/htadmin                                          5.0.2           xxxxxxxxxxxx   2 days ago   242MB
-malcolmnetsec/logstash-oss                                     5.0.2           xxxxxxxxxxxx   2 days ago   1.27GB
-malcolmnetsec/name-map-ui                                      5.0.2           xxxxxxxxxxxx   2 days ago   142MB
-malcolmnetsec/nginx-proxy                                      5.0.2           xxxxxxxxxxxx   2 days ago   117MB
-malcolmnetsec/opensearch                                       5.0.2           xxxxxxxxxxxx   2 days ago   1.18GB
-malcolmnetsec/pcap-capture                                     5.0.2           xxxxxxxxxxxx   2 days ago   122MB
-malcolmnetsec/pcap-monitor                                     5.0.2           xxxxxxxxxxxx   2 days ago   214MB
-malcolmnetsec/zeek                                             5.0.2           xxxxxxxxxxxx   2 days ago   938MB
+malcolmnetsec/arkime                                           5.0.3           xxxxxxxxxxxx   2 days ago   811MB
+malcolmnetsec/dashboards                                       5.0.3           xxxxxxxxxxxx   2 days ago   970MB
+malcolmnetsec/dashboards-helper                                5.0.3           xxxxxxxxxxxx   2 days ago   154MB
+malcolmnetsec/filebeat-oss                                     5.0.3           xxxxxxxxxxxx   2 days ago   621MB
+malcolmnetsec/file-monitor                                     5.0.3           xxxxxxxxxxxx   2 days ago   586MB
+malcolmnetsec/file-upload                                      5.0.3           xxxxxxxxxxxx   2 days ago   259MB
+malcolmnetsec/freq                                             5.0.3           xxxxxxxxxxxx   2 days ago   132MB
+malcolmnetsec/htadmin                                          5.0.3           xxxxxxxxxxxx   2 days ago   242MB
+malcolmnetsec/logstash-oss                                     5.0.3           xxxxxxxxxxxx   2 days ago   1.27GB
+malcolmnetsec/name-map-ui                                      5.0.3           xxxxxxxxxxxx   2 days ago   142MB
+malcolmnetsec/nginx-proxy                                      5.0.3           xxxxxxxxxxxx   2 days ago   117MB
+malcolmnetsec/opensearch                                       5.0.3           xxxxxxxxxxxx   2 days ago   1.18GB
+malcolmnetsec/pcap-capture                                     5.0.3           xxxxxxxxxxxx   2 days ago   122MB
+malcolmnetsec/pcap-monitor                                     5.0.3           xxxxxxxxxxxx   2 days ago   214MB
+malcolmnetsec/zeek                                             5.0.3           xxxxxxxxxxxx   2 days ago   938MB
 ```
 
 #### Import from pre-packaged tarballs
@@ -370,7 +370,7 @@ Then, go take a walk or something since it will be a while. When you're done, yo
 * `malcolmnetsec/pcap-monitor` (based on `debian:bullseye-slim`)
 * `malcolmnetsec/pcap-zeek` (based on `debian:bullseye-slim`)
 
-Alternately, if you have forked Malcolm on GitHub, [workflow files](./.github/workflows/) are provided which contain instructions for GitHub to build the docker images and [sensor](#Hedgehog) and [Malcolm](#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolmnetsec/image:branch` (e.g., if you've forked Malcolm with the github user `romeogdetlevjr`, the `arkime` container built for the `main` would be named `ghcr.io/romeogdetlevjr/malcolmnetsec/arkime:main`). To run your local instance of Malcolm using these images instead of the official ones, you'll need to edit your `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern.
+Alternately, if you have forked Malcolm on GitHub, [workflow files](./.github/workflows/) are provided which contain instructions for GitHub to build the docker images and [sensor](#Hedgehog) and [Malcolm](#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolmnetsec/image:branch` (e.g., if you've forked Malcolm with the github user `romeogdetlevjr`, the `arkime` container built for the `main` would be named `ghcr.io/romeogdetlevjr/malcolmnetsec/arkime:main`). To run your local instance of Malcolm using these images instead of the official ones, you'll need to edit your `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the bash helper script `./shared/bin/github_image_helper.sh` to pull and re-tag the images.
 
 ## <a name="Packager"></a>Pre-Packaged installation files
 
@@ -1499,7 +1499,7 @@ Building the ISO may take 30 minutes or more depending on your system. As the bu
 
 ```
 …
-Finished, created "/malcolm-build/malcolm-iso/malcolm-5.0.2.iso"
+Finished, created "/malcolm-build/malcolm-iso/malcolm-5.0.3.iso"
 …
 ```
 
@@ -1885,21 +1885,21 @@ Pulling zeek              ... done
 
 user@host:~/Malcolm$ docker images
 REPOSITORY                                                     TAG             IMAGE ID       CREATED      SIZE
-malcolmnetsec/arkime                                           5.0.2           xxxxxxxxxxxx   2 days ago   811MB
-malcolmnetsec/dashboards                                       5.0.2           xxxxxxxxxxxx   2 days ago   970MB
-malcolmnetsec/dashboards-helper                                5.0.2           xxxxxxxxxxxx   2 days ago   154MB
-malcolmnetsec/filebeat-oss                                     5.0.2           xxxxxxxxxxxx   2 days ago   621MB
-malcolmnetsec/file-monitor                                     5.0.2           xxxxxxxxxxxx   2 days ago   586MB
-malcolmnetsec/file-upload                                      5.0.2           xxxxxxxxxxxx   2 days ago   259MB
-malcolmnetsec/freq                                             5.0.2           xxxxxxxxxxxx   2 days ago   132MB
-malcolmnetsec/htadmin                                          5.0.2           xxxxxxxxxxxx   2 days ago   242MB
-malcolmnetsec/logstash-oss                                     5.0.2           xxxxxxxxxxxx   2 days ago   1.27GB
-malcolmnetsec/name-map-ui                                      5.0.2           xxxxxxxxxxxx   2 days ago   142MB
-malcolmnetsec/nginx-proxy                                      5.0.2           xxxxxxxxxxxx   2 days ago   117MB
-malcolmnetsec/opensearch                                       5.0.2           xxxxxxxxxxxx   2 days ago   1.18GB
-malcolmnetsec/pcap-capture                                     5.0.2           xxxxxxxxxxxx   2 days ago   122MB
-malcolmnetsec/pcap-monitor                                     5.0.2           xxxxxxxxxxxx   2 days ago   214MB
-malcolmnetsec/zeek                                             5.0.2           xxxxxxxxxxxx   2 days ago   938MB
+malcolmnetsec/arkime                                           5.0.3           xxxxxxxxxxxx   2 days ago   811MB
+malcolmnetsec/dashboards                                       5.0.3           xxxxxxxxxxxx   2 days ago   970MB
+malcolmnetsec/dashboards-helper                                5.0.3           xxxxxxxxxxxx   2 days ago   154MB
+malcolmnetsec/filebeat-oss                                     5.0.3           xxxxxxxxxxxx   2 days ago   621MB
+malcolmnetsec/file-monitor                                     5.0.3           xxxxxxxxxxxx   2 days ago   586MB
+malcolmnetsec/file-upload                                      5.0.3           xxxxxxxxxxxx   2 days ago   259MB
+malcolmnetsec/freq                                             5.0.3           xxxxxxxxxxxx   2 days ago   132MB
+malcolmnetsec/htadmin                                          5.0.3           xxxxxxxxxxxx   2 days ago   242MB
+malcolmnetsec/logstash-oss                                     5.0.3           xxxxxxxxxxxx   2 days ago   1.27GB
+malcolmnetsec/name-map-ui                                      5.0.3           xxxxxxxxxxxx   2 days ago   142MB
+malcolmnetsec/nginx-proxy                                      5.0.3           xxxxxxxxxxxx   2 days ago   117MB
+malcolmnetsec/opensearch                                       5.0.3           xxxxxxxxxxxx   2 days ago   1.18GB
+malcolmnetsec/pcap-capture                                     5.0.3           xxxxxxxxxxxx   2 days ago   122MB
+malcolmnetsec/pcap-monitor                                     5.0.3           xxxxxxxxxxxx   2 days ago   214MB
+malcolmnetsec/zeek                                             5.0.3           xxxxxxxxxxxx   2 days ago   938MB
 ```
 
 Finally, we can start Malcolm. When Malcolm starts it will stream informational and debug messages to the console. If you wish, you can safely close the console or use `Ctrl+C` to stop these messages; Malcolm will continue running in the background.

diff --git a/docker-compose-standalone.yml b/docker-compose-standalone.yml
@@ -127,7 +127,7 @@ x-pcap-capture-variables: &pcap-capture-variables
 
 services:
   opensearch:
-    image: malcolmnetsec/opensearch:5.0.2
+    image: malcolmnetsec/opensearch:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -164,7 +164,7 @@ services:
       retries: 3
       start_period: 180s
   dashboards-helper:
-    image: malcolmnetsec/dashboards-helper:5.0.2
+    image: malcolmnetsec/dashboards-helper:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -192,7 +192,7 @@ services:
       retries: 3
       start_period: 30s
   dashboards:
-    image: malcolmnetsec/dashboards:5.0.2
+    image: malcolmnetsec/dashboards:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -213,7 +213,7 @@ services:
       retries: 3
       start_period: 210s
   logstash:
-    image: malcolmnetsec/logstash-oss:5.0.2
+    image: malcolmnetsec/logstash-oss:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -248,7 +248,7 @@ services:
       retries: 3
       start_period: 600s
   filebeat:
-    image: malcolmnetsec/filebeat-oss:5.0.2
+    image: malcolmnetsec/filebeat-oss:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -285,7 +285,7 @@ services:
       retries: 3
       start_period: 60s
   arkime:
-    image: malcolmnetsec/arkime:5.0.2
+    image: malcolmnetsec/arkime:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -323,7 +323,7 @@ services:
       retries: 3
       start_period: 210s
   zeek:
-    image: malcolmnetsec/zeek:5.0.2
+    image: malcolmnetsec/zeek:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -349,7 +349,7 @@ services:
       retries: 3
       start_period: 60s
   file-monitor:
-    image: malcolmnetsec/file-monitor:5.0.2
+    image: malcolmnetsec/file-monitor:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -372,7 +372,7 @@ services:
       retries: 3
       start_period: 60s
   pcap-capture:
-    image: malcolmnetsec/pcap-capture:5.0.2
+    image: malcolmnetsec/pcap-capture:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -392,7 +392,7 @@ services:
     volumes:
       - ./pcap/upload:/pcap
   pcap-monitor:
-    image: malcolmnetsec/pcap-monitor:5.0.2
+    image: malcolmnetsec/pcap-monitor:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -415,7 +415,7 @@ services:
       retries: 3
       start_period: 90s
   upload:
-    image: malcolmnetsec/file-upload:5.0.2
+    image: malcolmnetsec/file-upload:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -441,7 +441,7 @@ services:
       retries: 3
       start_period: 60s
   htadmin:
-    image: malcolmnetsec/htadmin:5.0.2
+    image: malcolmnetsec/htadmin:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -463,7 +463,7 @@ services:
       retries: 3
       start_period: 60s
   freq:
-    image: malcolmnetsec/freq:5.0.2
+    image: malcolmnetsec/freq:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -481,7 +481,7 @@ services:
       retries: 3
       start_period: 60s
   name-map-ui:
-    image: malcolmnetsec/name-map-ui:5.0.2
+    image: malcolmnetsec/name-map-ui:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -502,7 +502,7 @@ services:
       retries: 3
       start_period: 60s
   nginx-proxy:
-    image: malcolmnetsec/nginx-proxy:5.0.2
+    image: malcolmnetsec/nginx-proxy:5.0.3
     restart: "no"
     stdin_open: false
     tty: true

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -130,7 +130,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/opensearch.Dockerfile
-    image: malcolmnetsec/opensearch:5.0.2
+    image: malcolmnetsec/opensearch:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -170,7 +170,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/dashboards-helper.Dockerfile
-    image: malcolmnetsec/dashboards-helper:5.0.2
+    image: malcolmnetsec/dashboards-helper:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -201,7 +201,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/dashboards.Dockerfile
-    image: malcolmnetsec/dashboards:5.0.2
+    image: malcolmnetsec/dashboards:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -225,7 +225,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/logstash.Dockerfile
-    image: malcolmnetsec/logstash-oss:5.0.2
+    image: malcolmnetsec/logstash-oss:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -265,7 +265,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/filebeat.Dockerfile
-    image: malcolmnetsec/filebeat-oss:5.0.2
+    image: malcolmnetsec/filebeat-oss:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -306,7 +306,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/arkime.Dockerfile
-    image: malcolmnetsec/arkime:5.0.2
+    image: malcolmnetsec/arkime:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -350,7 +350,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/zeek.Dockerfile
-    image: malcolmnetsec/zeek:5.0.2
+    image: malcolmnetsec/zeek:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -380,7 +380,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/file-monitor.Dockerfile
-    image: malcolmnetsec/file-monitor:5.0.2
+    image: malcolmnetsec/file-monitor:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -406,7 +406,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/pcap-capture.Dockerfile
-    image: malcolmnetsec/pcap-capture:5.0.2
+    image: malcolmnetsec/pcap-capture:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -429,7 +429,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/pcap-monitor.Dockerfile
-    image: malcolmnetsec/pcap-monitor:5.0.2
+    image: malcolmnetsec/pcap-monitor:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -455,7 +455,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/file-upload.Dockerfile
-    image: malcolmnetsec/file-upload:5.0.2
+    image: malcolmnetsec/file-upload:5.0.3
     restart: "no"
     stdin_open: false
     tty: true
@@ -481,7 +481,7 @@ services:
       retries: 3
       start_period: 60s
   htadmin:
-    image: malcolmnetsec/htadmin:5.0.2
+    image: malcolmnetsec/htadmin:5.0.3
     build:
       context: .
       dockerfile: Dockerfiles/htadmin.Dockerfile
@@ -506,7 +506,7 @@ services:
       retries: 3
       start_period: 60s
   freq:
-    image: malcolmnetsec/freq:5.0.2
+    image: malcolmnetsec/freq:5.0.3
     build:
       context: .
       dockerfile: Dockerfiles/freq.Dockerfile
@@ -527,7 +527,7 @@ services:
       retries: 3
       start_period: 60s
   name-map-ui:
-    image: malcolmnetsec/name-map-ui:5.0.2
+    image: malcolmnetsec/name-map-ui:5.0.3
     build:
       context: .
       dockerfile: Dockerfiles/name-map-ui.Dockerfile
@@ -554,7 +554,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfiles/nginx.Dockerfile
-    image: malcolmnetsec/nginx-proxy:5.0.2
+    image: malcolmnetsec/nginx-proxy:5.0.3
     restart: "no"
     stdin_open: false
     tty: true

diff --git a/docs/contributing/README.md b/docs/contributing/README.md
@@ -111,7 +111,7 @@ Another method for modifying your local copies of Malcolm's services' containers
 
 For example, say you wanted to create a Malcolm container which includes a new dashboard for OpenSearch Dashboards and a new enrichment filter `.conf` file for Logstash. After placing these files under `./dashboards/dashboards` and `./logstash/pipelines/enrichment`, respectively, in your Malcolm working copy, run `./build.sh dashboards-helper logstash` to build just those containers. After the build completes, you can run `docker images` and see you have fresh images for `malcolmnetsec/dashboards-helper` and `malcolmnetsec/logstash-oss`. You may need to review the contents of the [Dockerfiles](../../Dockerfiles) to determine the correct service and filesystem location within that service's Docker image depending on what you're trying to accomplish.
 
-Alternately, if you have forked Malcolm on GitHub, [workflow files](../../.github/workflows/) are provided which contain instructions for GitHub to build the docker images and [sensor](#Hedgehog) and [Malcolm](#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolmnetsec/image:branch` (e.g., if you've forked Malcolm with the github user `romeogdetlevjr`, the `arkime` container built for the `main` would be named `ghcr.io/romeogdetlevjr/malcolmnetsec/arkime:main`). To run your local instance of Malcolm using these images instead of the official ones, you'll need to edit your `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern.
+Alternately, if you have forked Malcolm on GitHub, [workflow files](../../.github/workflows/) are provided which contain instructions for GitHub to build the docker images and [sensor](#Hedgehog) and [Malcolm](#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolmnetsec/image:branch` (e.g., if you've forked Malcolm with the github user `romeogdetlevjr`, the `arkime` container built for the `main` would be named `ghcr.io/romeogdetlevjr/malcolmnetsec/arkime:main`). To run your local instance of Malcolm using these images instead of the official ones, you'll need to edit your `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the bash helper script `./shared/bin/github_image_helper.sh` to pull and re-tag the images.
 
 ## <a name="NewImage"></a>Adding a new service (Docker image)
 

diff --git a/sensor-iso/README.md b/sensor-iso/README.md
@@ -416,7 +416,7 @@ Building the ISO may take 90 minutes or more depending on your system. As the bu
 
 ```
 …
-Finished, created "/sensor-build/hedgehog-5.0.2.iso"
+Finished, created "/sensor-build/hedgehog-5.0.3.iso"
 …
 ```
-Original file line number
+Diff line change
@@ Expand Up @@
     ```
     …
-    Finished, created "/sensor-build/hedgehog-5.0.2.iso"
+    Finished, created "/sensor-build/hedgehog-5.0.3.iso"
     …
     ```
@@ Expand Down @@