Merge branch 'dev'

CHANGES.rst

@@ -3,6 +3,19 @@ Change log
 
 .. note:: **Upgrade notes**: after upgrading, run the ``arouteserver setup-templates`` command to sync the local templates with those distributed with the new version. More details on the `Upgrading <https://arouteserver.readthedocs.io/en/latest/INSTALLATION.html#upgrading>`__ section of the documentation.
 
+next release
+------------
+
+- New feature: ``configure`` and ``show_config`` *support* commands.
+
+  - ``configure``: it can be used to quickly generate a route server policy definition file (``general.yml``) on the basis of best practices and suggestions.
+
+  - ``show_config``: to display current configuration settings and also options that have been left to their default values.
+
+- New feature: ``ixf-member-export`` command, to build `IX-F Member Export JSON files <https://ml.ix-f.net/>`__ from the list of clients.
+
+- Improvement: cache expiry time values can be set for each external resource type: PeeringDB info, IRR data, ...
+
 v0.14.1
 -------
 

README.rst

@@ -100,6 +100,8 @@ Features
   - `integration <https://arouteserver.readthedocs.io/en/latest/USAGE.html#ixp-manager-integration>`__ with **IXP-Manager**;
   - `fetch lists <https://arouteserver.readthedocs.io/en/latest/USAGE.html#automatic-clients>`__ from **PeeringDB** records and **Euro-IX member list JSON** files.
 
+- **IX-F Member Export** JSON files `creation <https://arouteserver.readthedocs.io/en/latest/USAGE.html#ixf-member-export-command>`__.
+
 - Related tools:
 
   - `Invalid routes reporter <https://arouteserver.readthedocs.io/en/latest/TOOLS.html#invalid-routes-reporter>`__, to log or report invalid routes and their reject reason.

config.d/arouteserver.yml

@@ -58,7 +58,43 @@
 #threads: 4
 
 # Cache expiry time, in seconds.
+#
+# This can be a single integer value or a list of 'keyword: value'
+# pairs. If a single value is provided here, it will be used as
+# the expiry time for every kind of resource used by the program;
+# otherwise, each resource will use its own expiry time (or the
+# 'general' one if its specific expiry time is not given).
+#
+# Resources are identified by the following keywords:
+#
+# pdb_info: info fetched from PeeringDB (AS-SETs that networks
+#       configured in their IRR Record field, IPv4 and IPv6
+#       prefix count).
+#
+# ripe_rpki_roas: RPKI ROAs fetched from the RIPE RPKI Validator.
+#       See the 'general.yml' configuration file comments for more
+#       details.
+#
+# irr_as_sets: ASNs and routes fetched from IRR using clients'
+#       AS-SETs.
+#
+# Please note: if the desired behavior is to completely bypass
+# cache it is advisable to avoid setting expiry time to zero but
+# rather to set it to a duration that is enough to cover the whole
+# configuration processing time. During the configuration building
+# process some resources may be used more than once; a value of
+# zero would require them to be fetched multiple time during the
+# same execution.
+#
+# Single value:
 #cache_expiry: 43200
+#
+# Multiple values (default):
+#cache_expiry:
+#  general: 43200
+#  pdb_info: 86400
+#  ripe_rpki_roas: 43200
+#  irr_as_sets: 43200
 
 # Enable automatic checking for new release.
 # When set to True, the program automatically checks PyPI for

config.d/general.yml

@@ -1,3 +1,15 @@
+# This is the reference route server policy definition file
+# distributed with ARouteServer.
+#
+# -------------------------------------------------------------------
+#                       DO NOT EDIT THIS FILE
+# -------------------------------------------------------------------
+#
+# This file should be only used as a guide where comments about
+# options and settings can be found. Do not edit this file but rather
+# make a copy of it or, even better, use the 'arouteserver configure'
+# command to build your own version of the 'general.yml' file.
+
 cfg:
   # The route server AS number.
   rs_as: 999

config.d/log.ini

@@ -5,7 +5,19 @@ keys=root
 keys=simple,syslog,file
 
 [handlers]
-keys=stderr,smtp,syslog_udp,syslog_local,file
+# PLEASE CONFIGURE THE HANDLERS YOU WANT TO USE TO LOG MESSAGES
+#
+# one or more (comma delimited) handlers between the following:
+#
+#	stderr, smtp, syslog_udp, syslog_local, file, <your_custom_handler>
+#
+# For each handler that is enabled here, the relative "[handler_XXX]" section
+# must be configured below.
+#
+#keys=stderr,file,smtp,syslog_udp,syslog_local
+#keys=stderr,file,smtp
+#keys=stderr,file
+keys=stderr
 
 [logger_root]
 # PLEASE CONFIGURE THE HANDLERS YOU WANT TO USE TO LOG MESSAGES
@@ -17,9 +29,9 @@ keys=stderr,smtp,syslog_udp,syslog_local,file
 # levels: DEBUG, INFO, WARN, ERROR, CRITICAL
 level=INFO
 
-# one or more (comma delimited) of the following handlers:
+# one or more (comma delimited) handlers between the following:
 #
-#	stderr, smtp, syslog_udp, syslog_local, file, <your_custom_handler> 
+#	stderr, smtp, syslog_udp, syslog_local, file, <your_custom_handler>
 #
 # For each handler that is enabled here, the relative "[handler_XXX]" section
 # must be configured below.

docs/EXAMPLES.rst

@@ -83,3 +83,77 @@ Clients from Euro-IX member list JSON file
 Some clients files automatically built from `Euro-IX member list JSON files <https://github.com/euro-ix/json-schemas>`_ are reported here.
 
 https://github.com/pierky/arouteserver/blob/master/examples/clients-from-euroix
+
+``configure`` command output
+----------------------------
+
+The ``configure`` command can be used to quickly generate policy definition files (*general.yml*) which are based on suggested settings and best practices.
+
+A list of BGP communities is also automatically built.
+
+.. code-block:: console
+
+   $ arouteserver configure --output general.yml
+   
+   BGP daemon
+   ==========
+   
+   Depending on the BGP daemon used for the route server some features may not be
+   available.
+   
+   Details here:
+   https://arouteserver.readthedocs.io/en/latest/CONFIG.html#caveats-and-
+   limitations
+   
+   Which BGP daemon will be used? [bird/openbgpd] bird
+   
+   Router server's ASN
+   ===================
+   
+   What's the ASN of the route server? 64496
+   
+   Route server's BGP router-id
+   ============================
+   
+   Please enter the route server BGP router-id: 192.0.2.1
+   
+   List of local networks
+   ======================
+   
+   A list of local IPv4/IPv6 networks must be provided here: routes announced by
+   route server clients for these prefixes will be filtered out.
+   
+   Please enter a comma-separated list of local networks: 192.0.2.0/24,2001:db8::/32
+   
+   
+   Route server policy definition file generated successfully!
+   
+   The content of the general configuration file will now be written to general.yml
+   
+   Some notes:
+   
+    - Accepted prefix lengths are 8-24 for IPv6 and 12-48 for IPv6.
+    - Routes with 'transit-free networks' ASNs in the middle of AS_PATH are
+   rejected.
+    - IRR-based filters are enabled; prefixes that are more specific of those
+   registered are accepted.
+    - PeeringDB is used to fetch AS-SETs for those clients that are not explicitly
+   configured.
+    - RPKI ROAs are used as if they were route objects to further enrich IRR data.
+    - Routes tagged with the GRACEFUL_SHUTDOWN well-known community (65535:0) are
+   processed accordingly to draft-ietf-grow-bgp-gshut.
+
+The textual description (HTML) generated on the basis of the *general.yml* files produced by this command is also reported here.
+
+https://github.com/pierky/arouteserver/blob/master/examples/auto-config
+
+bird-general.yml.html - See the `textual representation of this configuration <_static/examples_auto-config_bird-general.yml.html>`__.
+
+openbgpd62-general.yml.html - See the `textual representation of this configuration <_static/examples_auto-config_openbgpd62-general.yml.html>`__.
+
+IX-F Member Export files
+------------------------
+
+The files reported within this directory were generated using the ``ixf-member-export`` `command <https://arouteserver.readthedocs.io/en/latest/USAGE.html#ixf-member-export-command>`__.
+
+https://github.com/pierky/arouteserver/blob/master/examples/ixf-member-export

docs/FEATURES.rst

@@ -90,6 +90,8 @@ Features
   - `integration <https://arouteserver.readthedocs.io/en/latest/USAGE.html#ixp-manager-integration>`__ with **IXP-Manager**;
   - `fetch lists <https://arouteserver.readthedocs.io/en/latest/USAGE.html#automatic-clients>`__ from **PeeringDB** records and **Euro-IX member list JSON** files.
 
+- **IX-F Member Export** JSON files `creation <https://arouteserver.readthedocs.io/en/latest/USAGE.html#ixf-member-export-command>`__.
+
 - Related tools:
 
   - `Invalid routes reporter <https://arouteserver.readthedocs.io/en/latest/TOOLS.html#invalid-routes-reporter>`__, to log or report invalid routes and their reject reason.

docs/INSTALLATION.rst

@@ -53,6 +53,18 @@ Installation
   The program will ask you to create some directories (under ``~/arouteserver`` by default) and to copy some files there.
   These paths can be changed by editing the ``arouteserver.yml`` program configuration file or by using command line arguments. More information in the :doc:`configuration section <CONFIG>`.
 
+4. Define the route server configuration policies, using the ``configure`` command or manually by editing the ``general.yml`` file:
+
+  .. code:: bash
+
+    # if you installed from GitHub
+    ./scripts/arouteserver configure
+
+    # if you used pip
+    arouteserver configure
+
+  The ``configure`` command asks some questions about the route server environment (ASN, router ID, local subnets) and then it builds a policy definition file based on best practices and suggestions which also includes a rich BGP communities list.
+
 External programs
 -----------------
 
@@ -115,6 +127,9 @@ ARouteServer uses the following external programs:
 Upgrading
 ---------
 
+Often upgrades bring new features and new options, sometimes they also introduce changes that might break backward compatibility with previous versions.
+It is advisable to always check the :doc:`CHANGELOG <CHANGELOG>` to verify what's new: the ``arouteserver show_config`` command can also be used to verify if new configuration options are available and how they are set by default.
+
 To upgrade the program, download the new version...
 
 .. code:: bash

docs/USAGE.rst

@@ -226,6 +226,49 @@ Since the popular `IXP-Manager <https://github.com/inex/IXP-Manager>`_ allows to
         # push it to the route server.
         ...
 
+.. _ixf-member-export-command:
+
+IX-F Member Export JSON file from ``clients.yml``
+-------------------------------------------------
+
+The ``ixf-member-export`` command can be used to generate `IX-F Member Export JSON files <https://ml.ix-f.net/>`__ from the list of clients that are configured on the route server.
+Although the ``clients.yml`` file used by ARouteServer to build the route server configuration contains only those clients that are supposed to connect to the route server itself, it's a quite common practice to preconfigure passive BGP sessions for all the IXP members there. When that's true the clients file contains a comprehensive representation of all the IXP participants.
+
+Please note: the output file generated with this command contains only a subset of the attributes available in the IX-F JSON schema: ASN, IP addresses, max-prefix limits and AS macros. Only information that are hard-coded in the ``clients.yml`` file are exported: AS-SETs or max prefix limits that during the configuration building process are fetched from PeeringDB or other external data sources are not included in the output file.
+
+.. code-block:: console
+
+   $ arouteserver ixf-member-export --clients examples/rich/clients.yml "Test IXP"
+   {
+     "version": "0.6",
+     "timestamp": "2017-11-24T17:23:41Z",
+     "ixp_list": [
+       {
+         "ixp_id": 0,
+         "shortname": "Test IXP",
+         "vlan": [
+           {
+             "id": 0
+           }
+         ]
+       }
+     ],
+     "member_list": [
+       {
+         "asnum": 10745,
+         "connection_list": [
+           {
+             "ixp_id": 0,
+             "vlan_list": [
+               {
+                 "vlan_id": 0,
+                 "ipv4": {
+                   "address": "192.0.2.22"
+                 }
+               },
+   [...]
+
+
 Live tests, development and customization
 -----------------------------------------