From b9036d49703ff189cde04f1249078e1a40b93aa5 Mon Sep 17 00:00:00 2001 From: Melanie Pierce <59747276+piercema@users.noreply.github.com> Date: Tue, 11 Feb 2025 18:21:19 -0700 Subject: [PATCH] Staging (#7) * Bump development for v25.01.0, also update copyright year * bump netbox to v4.1.10, osd_transform to v2.18.0, and fluent-bit to v3.2.4 * for cisagov/Malcolm#354, work in progress for Malcolm directly accepting syslog * for cisagov/Malcolm#354, work in progress for Malcolm directly accepting syslog; (dashboard) * cisagov/Malcolm#543, add naviation pane to non-network dashboards * bump jinja to 3.1.5 * Documentation for cisagov/Malcolm#354, syslog * replace old filebeat input for syslog with tcp/udp input and syslog processor, for cisagov/Malcolm#354 * Documentation for cisagov/Malcolm#354, syslog * install.py tweak for cisagov/Malcolm#354 * minor fix for for cisagov/Malcolm#354, set host.name correctly * bump netbox to v4.11.1 and elasticsearch-dsl to v8.17.1 * start of cisagov/Malcolm#356, normalize winlogbeats * WIP of cisagov/Malcolm#356, normalize winlogbeats * WIP of cisagov/Malcolm#356, normalize winlogbeats * WIP of cisagov/Malcolm#356, fix for a dashboard * WIP of cisagov/Malcolm#356, normalize winlogbeats * Work in progress for cisagov/Malcolm#541, making sure conn.log and known_services.log get the ICS protocols assigned to them corrrectly and tagged appropriately * Work in progress for cisagov/Malcolm#541 * standardize ICS protocols in network.protocol field, so they all get tagged with 'ics' properly cisagov/Malcolm#541 * fix cisagov/Malcolm#533, allow keystores to be created on startup even in hedgehog mode * forgot to add file for cisagov/Malcolm#356 * For cisagov/Malcolm#524, handle filenames with spaces in extracted_files_http_server.py * work for cisagov/Malcolm#542, preserve custom field formatting for index pattern on update of index pattern * work for cisagov/Malcolm#542, preserve custom field formatting for index pattern on update of index pattern * bump yq to v4.45.1 * for cisagov/Malcolm#551, URL pivot links from dashboards to arkime * for cisagov/Malcolm#551, URL pivot links from dashboards to arkime * fix pivot from arkime to dashboards and vice-versa when using a traefik or other reverse proxy * for cisagov/Malcolm#551, URL pivot links from dashboards to netbox * for cisagov/Malcolm#551, URL pivot links from dashboards to netbox * for cisagov/Malcolm#551, URL pivot links from netbox to arkime/dashboards * start of cisagov/Malcolm#553, update zeek to v7.1.0 * cisagov/Malcolm#553, handle conn.log for zeek v7.1.0 and documentation update * cisagov/Malcolm#553, handle postgresql.log * cisagov/Malcolm#553, handle postgresql.log * cisagov/Malcolm#553, added PostgreSQL dashboard * for cisagov/Malcolm#551, URL pivot links in dashboards (ignore date/times) * start of omron fins integration, cisagov/Malcolm#554 * wip omron fins integration, , cisagov/Malcolm#554 * arkime to v5.6.0 * bump logstash and filebeat to v8.17.0 * Fix nginx filebeat * WIP omron fins integration, cisagov/Malcolm#554 * WIP omron fins integration, cisagov/Malcolm#554 * WIP omron fins integration, cisagov/Malcolm#554 * WIP omron fins integration, cisagov/Malcolm#554 * WIP omron fins integration, cisagov/Malcolm#554 * dashboards tweaks * fix links for hh redirect download * First pass at adding suricata socket optimization * fix issue with nginx proxy * Setting debug to false * Fixing permissions for socket * html formatting * documentation for workaround for UFW software firewall for Malcolm ISO should automatically open ports for syslog cisagov/Malcolm#560) * Bump for v25.02.0 development * restore _config.yml * fix version * I don't think we need a seperate pod for the socket-based suricata, that's what the offline one does now anyway, right? * restore some comments, black python style * some tweaks for cisagov/Malcolm#457, pulled jjrush's branch into mine for some fixes * some tweaks for cisagov/Malcolm#457 * allow suricata to spawn threads * logging tweaks * more flexible verbosity for suricata * some tweaks for cisagov/Malcolm#457, try to wait until PCAP is finished processing before moving on * First pass at adding suricata socket optimization * Setting debug to false * Fixing permissions for socket * for cisagov/Malcolm#457, a few tweaks of the suricata pcap processing mode after reviewing @jjrush's code * for cisagov/Malcolm#457, monitor suricata.log to know when PCAP is done processing * for cisagov/Malcolm#457, monitor suricata.log to know when PCAP is done processing * for cisagov/Malcolm#457, signal suricata rules to reload after update * for cisagov/Malcolm#457, signal suricata rules to reload after update * for cisagov/Malcolm#457, fix processing of other log types * for cisagov/Malcolm#457, fix processing of other log types * for cisagov/Malcolm#457, signal suricata rules to reload after update * decrease verbosity for log * fix logic for autoarkime/forcearkime * some tweaks for cisagov/Malcolm#457, don't bother keeping track of when suricata is done with a PCAP file. just let filebeat handle it and pick up the resultant eve.json files directly * Standardizing healthcheck scripts, updating docker-compose, updating kubernetes * Adding livenessProbe to htadmin * cisagov/Malcolm#457, handle multiple Suricata PCAP processing threads * cisagov/Malcolm#574, clear screen after auth_setup when using Dialog mode * add the related.user field to the 'nginx Access Logs' table * bump fluent bit to v3.2.5 * fixed import of ECS templates * handle ARKIME_PORT value formatted like a URL in the init of the API container * cisagov/Malcolm#565, warn user about overwriting netbox passwords if they've already been set * fix cisagov/Malcolm#559, ANSI color codes from croc displayed * Exception in build triggers * for cisagov/Malcolm#557, try building dirinit with arm runner * cisagov/Malcolm#557, use arm-hosted runners for github build actions * restore _config.yml * a bit of cleanup for Dockefiles/health check scripts * minor fixes for health checks * Tweaks for health checks * restore _config.yml * Tweaks for health checks * build tweaks for health scripts * bump capa to v9.0.0 * workaround for issue blocking cisagov/Malcolm#475, integration of sigma rules * improvements to workaround for issue blocking cisagov/Malcolm#475, integration of sigma rules * improvements to workaround for issue blocking cisagov/Malcolm#475, integration of sigma rules * for cisagov/Malcolm#475, automatically apply aliases via index templates * for cisagov/Malcolm#475, starting on mappings for security analytics * for cisagov/Malcolm#585, include corelight/zeek-long-connections plugin for long connections (wIP) * for cisagov/Malcolm#585, include corelight/zeek-long-connections plugin for long connections (WIP) * for cisagov/Malcolm#585, include corelight/zeek-long-connections plugin for long connections (WIP) * for cisagov/Malcolm#585, include corelight/zeek-long-connections plugin for long connections (WIP) * demo fix * for cisagov/Malcolm#585, show long connection count on connections dashboard * decouple redis from netbox (cisagov/Malcolm#580) * one more minor change to cisagov/Malcolm#491, moved all container health scripts into one place to make it easier to keep track of them * decouple redis from netbox (cisagov/Malcolm#580) and reorganized some of the other netbox password stuff * updated fluent bit * fix filebeat health --------- Co-authored-by: Seth Grover Co-authored-by: Jason Rush --- .../dashboards/abdd7550-2c7c-40dc-947e-f6d186a158c4.json | 2 +- dashboards/scripts/shared-object-creation.sh | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/dashboards/dashboards/abdd7550-2c7c-40dc-947e-f6d186a158c4.json b/dashboards/dashboards/abdd7550-2c7c-40dc-947e-f6d186a158c4.json index 08f6e1ef8..ebcff86fe 100644 --- a/dashboards/dashboards/abdd7550-2c7c-40dc-947e-f6d186a158c4.json +++ b/dashboards/dashboards/abdd7550-2c7c-40dc-947e-f6d186a158c4.json @@ -194,7 +194,7 @@ "title": "Connections - Log Count Over Time", "uiStateJSON": "{\"vis\":{\"legendOpen\":false}}", "version": 1, - "visState": "{\"title\":\"Connections - Log Count Over Time\",\"type\":\"line\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"params\":{\"field\":\"MALCOLM_NETWORK_INDEX_TIME_FIELD_REPLACER\",\"useNormalizedOpenSearchInterval\":true,\"scaleMetricValues\":false,\"interval\":\"auto\",\"drop_partials\":false,\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\" \"},\"schema\":\"segment\"}],\"params\":{\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{\"text\":\"MALCOLM_NETWORK_INDEX_TIME_FIELD_REPLACER per 12 hours\"}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"\"}}],\"seriesParams\":[{\"show\":true,\"mode\":\"normal\",\"type\":\"histogram\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"linear\",\"lineWidth\":2,\"data\":{\"id\":\"1\",\"label\":\"Count\"},\"valueAxis\":\"ValueAxis-1\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"showCircles\":true,\"interpolate\":\"linear\",\"scale\":\"linear\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"type\":\"histogram\",\"labels\":{},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}" + "visState": "{\"title\":\"Connections - Log Count Over Time\",\"type\":\"line\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"params\":{\"field\":\"MALCOLM_NETWORK_INDEX_TIME_FIELD_REPLACER\",\"timeRange\":{\"from\":\"now-25y\",\"to\":\"now\"},\"useNormalizedOpenSearchInterval\":true,\"scaleMetricValues\":false,\"interval\":\"auto\",\"drop_partials\":false,\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\" \"},\"schema\":\"segment\"}],\"params\":{\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{\"text\":\"MALCOLM_NETWORK_INDEX_TIME_FIELD_REPLACER per 12 hours\"}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"\"}}],\"seriesParams\":[{\"show\":true,\"mode\":\"normal\",\"type\":\"histogram\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"linear\",\"lineWidth\":2,\"data\":{\"id\":\"1\",\"label\":\"Count\"},\"valueAxis\":\"ValueAxis-1\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"showCircles\":true,\"interpolate\":\"linear\",\"scale\":\"linear\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"type\":\"histogram\",\"labels\":{},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}" }, "id": "03eba854-72b5-47d0-a92a-b671a0d7ed19", "migrationVersion": { diff --git a/dashboards/scripts/shared-object-creation.sh b/dashboards/scripts/shared-object-creation.sh index 4b763bd6b..b8e610b4c 100755 --- a/dashboards/scripts/shared-object-creation.sh +++ b/dashboards/scripts/shared-object-creation.sh @@ -589,7 +589,7 @@ if [[ "${CREATE_OS_ARKIME_SESSION_INDEX:-true}" = "true" ]] ; then # OpenSearch security analytics fields mappings echo "Creating $DATASTORE_TYPE security analytics mappings..." - SA_MAPPINGS_IMPORT_DIR="$(mktemp -p "$TMP_WORK_DIR" -d -t sa-mappings-XXXXXX)" + SA_MAPPINGS_IMPORT_DIR="$(mktemp -d -t sa-mappings-XXXXXX)" rsync -a /opt/security_analytics_mappings/ "$SA_MAPPINGS_IMPORT_DIR"/ DoReplacersForDir "$SA_MAPPINGS_IMPORT_DIR" "$DATASTORE_TYPE" sa_mapping for i in "${SA_MAPPINGS_IMPORT_DIR}"/*.json; do @@ -597,13 +597,13 @@ if [[ "${CREATE_OS_ARKIME_SESSION_INDEX:-true}" = "true" ]] ; then RULE_TOPIC="$(jq -r '.rule_topic' 2>/dev/null < "$i")" INDEX_NAME="$(jq -r '.index_name' 2>/dev/null < "$i")" echo "Creating mappings for \"${INDEX_NAME}\" / \"${RULE_TOPIC}\" ..." && \ - CURL_OUT=$(get_tmp_output_filename) - curl "${CURL_CONFIG_PARAMS[@]}" --location --fail-with-body --output "$CURL_OUT" --silent \ + curl "${CURL_CONFIG_PARAMS[@]}" -w "\n" --location --silent --output /dev/null --show-error \ -XPOST "$OPENSEARCH_URL_TO_USE/_plugins/_security_analytics/mappings" \ -H "$XSRF_HEADER:true" -H 'Content-type:application/json' \ - -d "@$i" || ( cat "$CURL_OUT" && echo ) + -d "@$i" set -e done + rm -rf "${SA_MAPPINGS_IMPORT_DIR}" # end OpenSearch security analytics #############################################################################################################################