Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Tons of stuff. * bump moloch to 2.2.0 * reduce log noise * bump version for development to 1.8.2 * set elastalert index settings for a single node * fix issue #97, when using tcpdump the capture files are named .pcap.pcap * check moloch viewer status page periodically for docker container health check * fix docker-compose log verbosity * made kibana_index_refresh.py more robust as suggested by @fabrie in issue #100 use a _find API to get the index ID for a given index name instead of just the saved_objects/index-pattern API Example in test environment: Before adding new fields (dry run): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 465 fields success (dry run only, no write performed) ``` After adding new fields (dry run): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 481 fields success (dry run only, no write performed) ``` After adding new fields (update index mapping): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=False, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 481 fields success ``` * added plugin for detecting cve-2020-0601 * work on issue #102, log access to Malcolm web interface(s) to Elasticsearch for analysis in Kibana * nginx/php adjustments for issue #101, uploading very large pcap files may fail * fix a few of the control bash scripts to use GNU coreutils where applicable (issue #103) A few uses of "grep" and "find" use flags unique to the GNU versions of those tools. As GNU coreutils was already required for a few other utilities I've done the same thing to detect and use ggrep and gfind when needed. As suggested, it might be a good idea to rewrite these to use Python instead to be more portable (although I'll have to take care to make them work with both python 2/3 for various platforms). * bump malcolm version to 1.9.0, moloch version to 2.2.1 * work on implementing control scripts (start,stop,restart,wipe,logs) in python rather than bash for portability (see issue #103). have not as of yet removed the bash versions, as I am still testing these new implementations. for now the auth_setup.sh and malcolm_appliance_packager.sh are still in Bash as well * have ISO use new scripts * bump elastic to 7.5.2 * update iso build scripts to use new python scripts for install * compatibility fixes for scripts under linux * don't source missing files * more reworking of scripts from bash -> python (not complete yet, may be in a broken state) * more work on auth_setup * remove reference to files we're not longer using * Revert "bump elastic to 7.5.2" This reverts commit 440c859. * fix default for external password question * handle missing python package in windows * documentation updates * tweak some codenames * use specified path (rather than absolute path) for compose file * make python3 the default * Revert "make python3 the default" This reverts commit 52e53f4. * fixes for new control scripts * fix logs script hanging * pin filters by default in kibana * put a hack/fix in for vagrant not liking dhcp nat in 6.1 * create a zeek.service_version field to track protocol version in a single place; also, move password up to the zeek root level * added security overview dashboard wip * added freq.Dockerfile to detecting string entropy * if designated by the FREQ_LOOKUP (true) environment variable, look up DNS query hostnames using freq_server.py * update docker ignore ifle * ask about string freq lookup in install.py * added security overview dashboard wip * added security overview dashboard wip * use a ruby block rather than an http filter in order to better handle arrays * fix volume mapping for local.zeek in docker-compose.yml for testing * fix volume mapping for local.zeek in docker-compose.yml for testing * clean up symlinks as well * initial code for generating and parsing smb_cmd.log * initial code for generating and parsing smb_cmd.log * more work on smb command mapping * more work on smb * more work on smb * bump version to 2.0.0 * some field normalization for 2.0.0 - restored kibana swimlane visualization as it has been fixed for 7.5.x - remove some unused fields from records (agent.ephemeral_id, input.type, path portion of log.file.path) - remove "_jsonparsefailure" tag on cleanup - change some places where we were doing calculations to get count values when we already know the count is "1" - normalization of "action" or "command" values to "zeek.action" field - normalization of mime type, file names, fuids, and service version * fixes to SMB action mapping * remove useless prefix before smb action * exclude some domains from freq. analysis * utility script to repackage zeek logs for upload: * fix issue #111, moloch/etc mount in docker-compose.yml causes custom Zeek fields not to be loaded * fix Malcolm issue #110, submitting hunt job crashes viewer unless Zeek logs are filtered out (temporary patch of fix for Moloch issue 1374, arkime/arkime#1374) * Added smb_cmd fields to WISE * fix dashboard referring to zeek_smb.action -> zeek.action * remove tunnel:: prefix from tunnel type * added 'action' panel to overview * added security overview dashboard (wip) to directory * more work on issue #108, create security overview dashboard in kibana * working on issue #109, create ICS security overview dashboard * added ipv4/ipv6 * working on issue #109, create ICS security overview dashboard * working on issue #109, create ICS security overview dashboard * added network layer to connections * fix max font size * bring sensor local.zeek up to match malcolm's * fixed spacing of navigation menu * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * comments * ignore logs that have been renamed and are in transit being archived * updates to dashboards * fix kibana_index_refresh.py for python2 * fix non-ics/iot protocols dashboard * bump version to 7.6.0 for elastic * working with es 7.6, but elastalert had to be temporarily disabled. will work on fixing this next * don't include known_certs in outdated/insecure protocols * Tons of work refining dashboards * tweak connections view * improved maps * remove warnings * improvements to how notices can be used througout the other dashboards * improvements to how notices can be used througout the other dashboards * do frequency analysis on zeek_ssl.server_name * merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113) * experimenting with creating a merged network.mac_oui field that looks like this: ... "network": { "type": "ipv4", "mac_oui": { "00:10:db:ff:10:01": "Juniper Networks", "8c:85:90:65:85:8f": "Apple, Inc." } }, ... however, I may revert this for now because although this works kibana does't really play nicely with the data in visualizations * Revert "experimenting with creating a merged network.mac_oui field that looks like this:" This reverts commit 8bdcefa. * Revert "merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113)" This reverts commit ae60cf2. * make installer work better for vms * build virtualbox guest debs in a clean environment * fixed vagrantfile for malcolm build * only keep vmware/virtualbox guest packages in the right environments * increase build memory requirements * fix typo * updating sensor-iso to match malcolm-iso * fix relative path * removed docker-gen in nginx container, we're not using it any more * update software saved search * Fixed installation of elastalert kibana plugin, but still broken due to this issue: bitsensor/elastalert-kibana-plugin#141 * fix issue #104, Upload without trailing slash redirects to incorrect hostname and scheme this fix includes a few things: - modifying the index.html page to prepend "upload/" before relative HREF/SRC references - removing some useless code in the file-upload default nginx config - added the trailing slash to the proxy-pass directive for the upload section of the main nginx proxy - handle "/server/php" as a separate proxy redirect as that's the XHR where the uploads seem to go - remove the unused Moloch upload page * update copyright * update style of upload screen to match the rest of the app * working on network diff code (wip) * work in progress on network time diff, viewer.js not actually used yet * bump moloch to 2.2.2 * work in progress for network diff * network diff work in progress * comments/work in progress * some test files * update zeek to 3.0.2 * fix reference to zeek::af_packet * install zeek::af_packet with zkg * Revert "install zeek::af_packet with zkg" This reverts commit a20fa9b. * added docker files for running moloch regression tests * added vim to test harnest * fix af_packet zeek build * added promotional poster: * switch test harness branch * temporarily pull from https://github.com/mmguero-dev/moloch fork for issue #2 development rather than aol/moloch releases * update elastic to 7.6.1 for security and bug fixes * update moloch to 2.2.3 * update moloch to 2.2.3 * update zeek to 3.0.3 * update psutil to fix security alert GHSA-qfc5-mcwq-26q8 * zeek updated website, fix broken link * restore stuff for generating web documentation * fix URL for relocated MITRE ATTACK BZAR plugin * fix broken links for build of Zeek, MITRE ATT&CK BZAR plugin * fixed duplicate plugin URL in script * update moloch version in docs to 2.2.3 * remove files no longer needed for testing * remove files no longer needed for testing and update moloch version in documentation * should fix issue #114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files. * should fix issue #114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files. * proof of concept for a segment mapping form * work in progress on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * apply tooltip for table columns * scroll back and forth to selected item * beautify with icons * basic validation client-side * more work on the segment mapping ui (integration with malcolm scripts on logstash startup) * more work on the segment mapping ui (creation of docker image, integration with malcolm's nginx reverse proxy) * Added new icon to malcolm iso for subnet mapping editor * documentation updates * start logstash under supervisord in order to add a process that will watch for changes to the name matching * more work on name-map-ui, allow uploading of the JSON file so it can be pushed to the docker image volume automatically * map location of host/subnet mapping to correct location under name-map-ui container * integrate upload with name-map-ui * add the ability to signal logstash from the net-map-ui container * clear out previous maps between restarts * add ability to save net-map.json from web ui * basic control for restarting logstash via ui controls * put save/restart confirmations in UI * added import button to name map ui * send save-state post value to restart-logstash.php * update documentation * update documentation * remove unused variable * documentation updates * use fonts-symbola instead of fonts-noto-color-emoji * re-enable swimlane visualization * update elasticsearch to 7.6.2; also, fix issue #119 * use default theme in elastalert kibana editor * update kibana plugin version * add user to vboxsf group for using shared folders * the 'run a separate instance of Zeek locally' use case isn't really a big enough use case to have a whole separate docker-compose file for it; especially with the ISO and live capture methods. * ensure all services have a health check * reduce verbosity of health checks in logs
- Loading branch information
