fix: ensure AS names are strings

While there, introduce a more comprehensive smoke test that
ensures we will catch this issue in the future.

Fixes ooni/probe#1836

Author: bassosimone

.github/workflows/generate.yml

@@ -23,14 +23,17 @@ jobs:
       run: apt-get update
 
     - name: Install deps
-      run: apt-get install --no-install-recommends -y bgpdump ca-certificates curl libmaxmind-db-writer-perl python3 python3-gdbm
+      run: apt-get install --no-install-recommends -y bgpdump ca-certificates curl libmaxmind-db-writer-perl python3 python3-gdbm python3-maxminddb
 
     - name: Generate database using RIPE BGP dump and CAIDA AS names
       run: ./generator_ripe_caida.sh
 
     - name: End
       run: ls -ltr
 
+    - name: Verify the generated file
+      run: ./smoketest.py
+
     - name: Upload
       uses: actions/upload-artifact@v1
       with:

.gitignore

@@ -1,3 +1,4 @@
+/asn.mmdb
 /caida_asn_to_name.dbm
 /caida_input.gz
 /dbdata

README.md

@@ -22,14 +22,18 @@ them to an intermediate file for later consumption;
 choosing the most common occurrence _and_ merges routing
 data extracted using `bgpdump` to CAIDA ASN data;
 
-- `./writedb.pl` writes the `output.mmdb` file.
+- `./writedb.pl` writes the `asn.mmdb` file.
 
 Each individual script contains a comment explaining with
 greater detail what its input and output look like.
 
 The `VERSION` file changes every time we bless a new release: we
 need something to change in the tree to make a release.
 
+## Testing
+
+Run `./smoketest.py` to smoke test `asn.mmdb`.
+
 ## Dependencies
 
 This generator is designed to run on Debian GNU/Linux. It may work

dedupe_add_caida_orgname.py

@@ -47,15 +47,15 @@ def main():
             if current_net is not None:
                 # Deduplicate ASNs using the most common occurrence
                 oasn = c.most_common(1)[0][0]
-                orgname = asn_to_orgname.get(oasn, "<unknown>")
+                orgname = asn_to_orgname.get(oasn, b"<unknown>").decode()
                 print(f"{current_net} {oasn} {orgname}")
                 c = Counter()
             current_net = net
         c.update([asn])
 
     # Deduplicate ASNs using the most common occurrence
     asn = c.most_common(1)[0][0]
-    orgname = asn_to_orgname.get(asn, "<unknown>")
+    orgname = asn_to_orgname.get(asn, b"<unknown>").decode()
     print(f"{current_net} {asn} {orgname}")
 
     asn_to_orgname.close()