Merge branch 'maintenance' into develop

Intevation · Oct 28, 2020 · 5cec1d8 · 5cec1d8
2 parents f783c2d + 4d10b2b
commit 5cec1d8
Show file tree

Hide file tree

Showing 9 changed files with 69 additions and 26 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -65,6 +65,7 @@ CHANGELOG
 - Added `intelmq.bots.experts.threshold` (PR#1608 by Karl-Johan Karlsson).
 
 #### Outputs
+- `intelmq.bots.outputs.rt`: Added Request Tracker output bot (PR#1589 by Marius Urkis).
 
 ### Documentation
 - Feeds:
@@ -110,14 +111,13 @@ CHANGELOG
 
 ### Known issues
 
-2.2.2 (unreleased)
+
+2.2.3 (unreleased)
 ------------------
 
 ### Configuration
 
 ### Core
-- `intelmq.lib.upgrades`:
-  - Add upgrade function for renamed Shadowserver feed name "Blacklisted-IP"/"Blocklist".
 
 ### Development
 
@@ -126,23 +126,49 @@ CHANGELOG
 ### Bots
 #### Collectors
 
+#### Parsers
+
+#### Experts
+
+#### Outputs
+
+### Documentation
+
+### Packaging
+
+### Tests
+
+### Tools
+
+### Contrib
+
+### Known issues
+
+
+2.2.2 (2020-10-28)
+------------------
+
+### Core
+- `intelmq.lib.upgrades`:
+  - Add upgrade function for renamed Shadowserver feed name "Blacklisted-IP"/"Blocklist".
+
+### Bots
 #### Parsers
 - `intelmq.bots.parsers.shadowserver`:
   - Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
   - Added support for the feeds `Accessible Radmin` and `CAIDA IP Spoofer` (PR#1600 by sinus-x).
 - `intelmq.bots.parsers.anubisnetworks.parser`: Fix parsing error where `dst.ip` was not equal to `comm.http.host`.
 - `intelmq/bots/parsers/danger_rulez/parser`: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus).
 - `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23).
-- `intelmq.bots.parsers.microsoft.parser_ctip`: Add support for `DestinationIpInfo.*` and `Signatures.Sha256` fields, used by the `ctip-c2` feed (PR#1623 by Mikk Margus Möll).
+- `intelmq.bots.parsers.microsoft.parser_ctip`:
+  - Add support for `DestinationIpInfo.*` and `Signatures.Sha256` fields, used by the `ctip-c2` feed (PR#1623 by Mikk Margus Möll).
+  - Use `extra.payload.text` for the feed's field `Payload` if the content cannot be decoded (PR#1610 by Giedrius Ramas).
 
 #### Experts
 - `intelmq.bots.experts.cymru_whois`:
   - Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606).
   - The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606).
 
-#### Outputs
-- `intelmq.bots.outputs.rt`: Added Request Tracker output bot (PR#1589 by Marius Urkis).
-
 ### Documentation
 - README:
   - Add Core Infrastructure Initiative Best Practices Badge.
@@ -172,6 +198,8 @@ CHANGELOG
 - Bash completion scripts: Check both `/opt/intelmq/` as well as LSB-paths (`/etc/intelmq/` and `/var/log/intelmq/`) for loading bot information (#1561, PR#1628 by Birger Schacht).
 
 ### Known issues
+- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
+- Corrupt dump files when interrupted during writing (#870).
 
 
 2.2.1 (2020-07-30)

diff --git a/NEWS.md b/NEWS.md
@@ -38,28 +38,33 @@ The documentation is now available at [intelmq.readthedocs.io](https://intelmq.r
 ### Postgres databases
 
 
-2.2.2 Bugfix release (unreleased)
+2.2.3 Bugfix release (unreleased)
 ---------------------------------
 
-### Bots
-#### Cymru Whois Lookup
-The cache key calculation has been fixed. It previously led to duplicate keys for different IP addresses and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible. Therefore, this bot may take longer processing events than usual after applying this update.
-More details can be found in [issue #1592](https://github.com/certtools/intelmq/issues/1592).
-
 ### Requirements
 
 ### Tools
 
 ### Harmonization
-#### Shadowserver Feed/Parser
-The feed "Blacklisted-IP" has been renamed by ShadowServer to "Blocklist". In IntelMQ, the old name can still be used in IntelMQ until version 3.0.
 
 ### Configuration
 
 ### Libraries
 
 ### Postgres databases
 
+2.2.2 Bugfix release (2020-10-28)
+---------------------------------
+
+### Bots
+#### Cymru Whois Lookup
+The cache key calculation has been fixed. It previously led to duplicate keys for different IP addresses and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible. Therefore, this bot may take longer processing events than usual after applying this update.
+More details can be found in [issue #1592](https://github.com/certtools/intelmq/issues/1592).
+
+### Harmonization
+#### Shadowserver Feed/Parser
+The feed "Blacklisted-IP" has been renamed by ShadowServer to "Blocklist". In IntelMQ, the old name can still be used in IntelMQ until version 3.0.
+
 
 2.2.1 Bugfix release (2020-07-30)
 ----------------------------------

diff --git a/debian/changelog b/debian/changelog
@@ -4,11 +4,17 @@ intelmq (2.3.0~alpha1-1) UNRELEASED; urgency=medium
 
  -- Sebastian Wagner <[email protected]>  Thu, 18 Jun 2020 10:34:50 +0200
 
-intelmq (2.2.2~alpha1-1) UNRELEASED; urgency=medium
+intelmq (2.2.3~alpha1-1) UNRELEASED; urgency=medium
 
-  * Update to 2.2.2 Alpha.
+  * Update to 2.2.3.
 
- -- Sebastian Wagner <[email protected]>  Mon, 03 Aug 2020 12:59:51 +0200
+ -- Sebastian Wagner <[email protected]>  Wed, 28 Oct 2020 21:09:54 +0100
+
+intelmq (2.2.2-1) stable; urgency=medium
+
+  * Update to 2.2.2.
+
+ -- Sebastian Wagner <[email protected]>  Tue, 27 Oct 2020 17:50:36 +0100
 
 intelmq (2.2.1-1) stable; urgency=medium
 

diff --git a/docs/guides/Bots.md b/docs/guides/Bots.md
@@ -1362,6 +1362,9 @@ The parser is capable of parsing both feeds:
 - `ctip-infected-summary`
 The feeds only differ by a few fields, not in the format.
 
+The feeds contain a field called `Payload` which is nearly always a base64 encoded JSON structure.
+If decoding works, the contained fields are saved as `extra.payload.*`, otherwise the field is saved as `extra.payload.text`.
+
 * * *
 
 ### MISP

diff --git a/docs/guides/INSTALL.md b/docs/guides/INSTALL.md
@@ -26,8 +26,8 @@ The following instructions assume the following requirements. Python versions >=
 Supported and recommended operating systems are:
 * CentOS 7
 * Debian 9 and 10
-* OpenSUSE Leap 15.1
-* Ubuntu: 16.04, 18.04, 19.10, 20.04
+* OpenSUSE Leap 15.1, 15.2
+* Ubuntu: 16.04, 18.04, 20.04
 
 Other distributions which are (most probably) supported include CentOS 8, RHEL, Fedora and openSUSE Tumbleweed.
 

diff --git a/intelmq/bots/parsers/microsoft/parser_ctip.py b/intelmq/bots/parsers/microsoft/parser_ctip.py
@@ -139,7 +139,7 @@
     "Payload.response": "extra.payload.response",
     "Payload.handler": "extra.payload.handler",
     "Payload.type": "protocol.application",
-    "Payload": "extra.payload",
+    "Payload": "extra.payload.text",
     "Payload.Time": "extra.payload.time",
     "Payload.SourceIP": "extra.payload.source.ip",
     "Payload.DestIP": "extra.payload.destination.ip",

diff --git a/intelmq/lib/upgrades.py b/intelmq/lib/upgrades.py
@@ -26,7 +26,7 @@
            'v220_azure_collector',
            'v220_feed_changes',
            'v221_feed_changes',
-           'v222_feed_changes_1',
+           'v222_feed_changes',
            ]
 
 
@@ -501,7 +501,7 @@ def v221_feed_changes(defaults, runtime, harmonization, dry_run):
     return messages + ' Remove affected bots yourself.' if messages else changed, defaults, runtime, harmonization
 
 
-def v222_feed_changes_1(defaults, runtime, harmonization, dry_run):
+def v222_feed_changes(defaults, runtime, harmonization, dry_run):
     """
     Migrate Shadowserver feed name
     """
@@ -529,7 +529,8 @@ def v222_feed_changes_1(defaults, runtime, harmonization, dry_run):
     ((2, 1, 3), (v213_deprecations, v213_feed_changes)),
     ((2, 2, 0), (v220_configuration, v220_azure_collector, v220_feed_changes)),
     ((2, 2, 1), (v221_feed_changes, )),
-    ((2, 2, 2), (v222_feed_changes_1, )),
+    ((2, 2, 2), (v222_feed_changes, )),
+    ((2, 2, 3), ()),
 ])
 
 ALWAYS = (harmonization, )
diff --git a/intelmq/tests/bots/parsers/microsoft/test_parser_ctip_azure.py b/intelmq/tests/bots/parsers/microsoft/test_parser_ctip_azure.py
@@ -79,7 +79,7 @@
 "source.geolocation.city": "Vienna",
     "extra.custom_field1": "matsnu5",
     "raw": base64_encode(EXAMPLE_LINES[1]),
-    "extra.payload": 'this is just some text',
+    "extra.payload.text": 'this is just some text',
     'extra.malware': 'Avalanche',
     },
     ]

diff --git a/intelmq/tests/lib/test_upgrades.py b/intelmq/tests/lib/test_upgrades.py
@@ -569,7 +569,7 @@ def test_v221_feed_changes_2(self):
 
     def test_v222_feed_changes(self):
         """ Test v222_feed_changes """
-        result = upgrades.v222_feed_changes_1({}, V222, {}, False)
+        result = upgrades.v222_feed_changes({}, V222, {}, False)
         self.assertTrue(result[0])
         self.assertEqual(V222_OUT, result[2])