Skip to content

Route Server Template Upgrades, Grapher Improvements, More ZF Migration, PeeringDB Integration, and More!
Compare
Choose a tag to compare
@barryo barryo released this 03 Jul 13:39
· 2101 commits to master since this release
v4.8.0
b1abc25

This release adds a bunch of new features and makes further significant progress in the migration from Zend Framework to Laravel and some of these changes may require manual fixes / changes by you. Please read these release notes carefully.

Summary:

$ git --no-pager diff --shortstat --no-merges  v4.7.0 v4.8.0
 669 files changed, 71892 insertions(+), 21802 deletions(-)

Upgrade Instructions

  • perform the manual SQL queries below before doing anything else.
  • please follow the official upgrade documentation without skipping any steps.
  • complete some post-upgrade tasks and manual database changes covered in this release notification.

Pre-Upgrade Database Schema Changes

The following may affect zero database rows for most users. At INEX, some of our database entries date back >10 years so there is a little bit of crud in there from older tools and from times when MySQL was less strict on certain formats.

Please execute the following SQL manually:

UPDATE `contact` SET `created` = NULL WHERE CAST(`created` AS CHAR(20)) = '0000-00-00 00:00:00';
UPDATE `contact` SET `lastupdated` = NULL WHERE CAST(`lastupdated` AS CHAR(20)) = '0000-00-00 00:00:00';
DELETE FROM `user_pref` WHERE `user_pref`.`user_id` NOT IN ( SELECT `id` FROM `user`);
UPDATE `irrdb_asn` SET `last_seen` = NULL WHERE CAST(`last_seen` AS CHAR(20)) = '0000-00-00 00:00:00';
UPDATE `consoleserverconnection` SET `autobaud` = 0 WHERE autobaud IS NULL;

Post-Upgrade Tasks and Database Schema Changes

One of the new features of v4.8.0 is fixing the switch database table which until now could hold switches and console servers. This was awkward in practice and we have split these into distinct database tables. The schema changes you ran during the official upgrade will have put in the necessary database schema. You now need to run the following command to migrate the data:

cd $IXPROOT
php artisan ixp-manager:upgrade:split-conservers

If you get an error like:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-8' for key 'console_server_port_uniq'

it means you have a console server with more than one entry for a given port. You need to root these out and fix them. The following SQL will provide these in the format switchid-consoleserverport:

SELECT CONCAT( `switchid`, '-', `port` ) AS `switchport`, 
            COUNT(CONCAT( `switchid`, '-', `port` )) AS `cnt`  
    FROM `consoleserverconnection` 
    GROUP BY `switchport` HAVING cnt > 1

Please read on through the rest of these release notes as there are a number of other important changes and new configuration options that you may wish to set.

Route Server Template Updates

If you are generating your route server configurations via IXP Manager with IRRDB filtering, then please make sure you add the following option to your local .env file:

# Absolute path to run the bgpq3 utility
# e.g. IXP_IRRDB_BGPQ3_PATH=/usr/local/bin/bgpq3
IXP_IRRDB_BGPQ3_PATH=/path/to/your/bgpq3

DEPRECATION: the former setting for the bgpq3 path in config/ixp_tools.php is no longer used and the entire irrdb section can be removed.

The route servers in their default configuration filter all IPv4 prefixes >/24. This version of IXP Manager now makes that configurable as well as adding the same restriction for IPv6 (at >/48). If you want to change these, add and set the following options to your .env (default values shown):

IXP_IRRDB_MIN_V4_SUBNET_SIZE=24
IXP_IRRDB_MIN_V6_SUBNET_SIZE=48

The IRRDB database update process now also honours these by passing bgpq3 the -m option (2ebcdc7).

We have extended BGP NEXT_HOP hijacking detection and prevention to v6 also (2893103).

We now optionally allow more specifics in route server inbound prefix filtering. This means, for example, if someone has a IRRDB entry for a.b.0.0/16 but advertises this as 2 /17's, these will be accepted if the user's VLAN interface is configured to allow this. This implements a feature as discussed in #281 and requested a number of times by others including @listerr (9f146d6). NB: you need to turn this on per member.

If you wish to do it for all existing members, you can run the following SQL command:

UPDATE vlaninterface SET rsmorespecifics = 1;

Latency Graphs (Smokeping) and Graphing

This release migrates all remaining frontend graphing operations to Grapher and Laravel and also migrates latency graphs (via Smokeping) to Grapher.

Grapher Documentation

We have reworked the graphing documentation: https://docs.ixpmanager.org/grapher/introduction/

UI Improvements

On the UI you will find a new layout which we feel better incorporates latency and peer to peer graphs. We have also updated the peer to peer graphing UI to better handle situations such as multiple interfaces for the same customer on the same peering LAN.

### Permissions

We have added options to make all customer graphs available to all logged in customer users or publicly available. Neither of these are the default - the default remains that a customer's graphs are only available to that customer. See the API & Permissions documentation section.

Latency / Smokeping Graphs

With the Smokeping migration to Grapher, the following .env parameters need to be updated:

### Sample pre-v4.8 configuration:
GRAPHER_BACKENDS="mrtg|sflow"

GRAPHER_SMOKEPING_URL="http://ixp.example.com/smokeping"

### New >=v4.8 configuration:
GRAPHER_BACKENDS="mrtg|sflow|smokeping"

GRAPHER_BACKEND_SMOKEPING_ENABLED=true
GRAPHER_BACKEND_SMOKEPING_URL="http://ixp.example.com/smokeping"

We have also changing the API endpoint for Smokeping configuration generation to Grapher's API system. As such, older URLs such as:

https://ixp.example.com/api/v4/vlan/smokeping/{vlanid}/{protocol}

need to be updated to:

https://ixp.example.com/api/v4/grapher/config?backend=smokeping&vlanid=10&protocol=ipv4

See the Smokeping Grapher documentation for complete details. And also, note particularly that if you are using the supplied Smokeping update script then you will need to update the URL, change the protocols from 4 6 to ipv4 ipv6 and change all instances of ipv${proto} to just $proto in this.

PeeringDB

When adding a new customer to IXP Manager, there is now the option to pre-fill that customer's details from PeeringDB based on their AS number. PeeringDB does not provide access to some information such as NOC contact details unless the query comes from an authenticated user.

If you configure the following settings in your .env file, then IXP Manager will query PeeringDB using an authenticated connection:

IXP_API_PEERING_DB_USERNAME=username
IXP_API_PEERING_DB_PASSWORD=password

(clearly, replacing 'username' and 'password' for valid details for a PeeringDB user in your organisation)

Docker

As the IXP Manager ecosystem grows, it becomes harder to maintain ubiquitous development environments for coding, testing and demonstrating IXP Manager.

Especially for emulating switches, route servers, graphing and tools such as Bird's Eye. Docker is ideally suited to solving these issues as well as providing the perfect environment for IXP Manager workshops.

To that end, we have creating two Docker systems.

  1. The first is ideally suited for demonstrations, 'just playing' with IXP Manager and workshops: https://github.com/inex/ixp-manager-docker
  2. The second is specifically targeted at IXP Manager developers: http://docs.ixpmanager.org/dev/docker/

These are in addition to the existing installation scripts and Vagrant:

The Docker images currently reflect v4.7 but will updated shortly for v4.8.

Customer Notes

Customer note functionality has been ported to Laravel and improved. They now also support Markdown.

We have added documentation for this: https://docs.ixpmanager.org/usage/customer-notes/

One change from previous releases is that notification of customer note changes is now enabled by default for all superadmins. You may want to check system accounts with admin permissions (if you have any) and disable notifications for those as outlined in the documentation.

Exporting Member Details

The older Zend Framework method of exporting member details has been deprecated for sometime and is now removed in this release. If you were using this, please migrate to the new methods:

  1. IX-F Member Export: http://docs.ixpmanager.org/features/ixf-export/
  2. New Member Export: http://docs.ixpmanager.org/features/member-export/

The first link above includes sample JavaScript code for generating member lists using the IX-F Member Export.

Customer Tags

IXP Manager has always supported customer types for: Full, Pro-bono, Internal, Associate.

We appreciate these are sufficient to suit all use cases and from requests from users, we have added tags which allows administrators to tag customers to indicate specific properties that are not built into IXP Manager by default. Some examples might include:

  • commercial vs membership based relationship;
  • special billing relationship (e.g. customer originally connected via 100Mb port which has been discontinued; customer now on 1Gb port but not paying until traffic exceeds 100Mb);

Documentation for tags can be found here: https://docs.ixpmanager.org/usage/customer-tags/

Discovered MAC Addresses

Discovered MAC addresses is a tool which polls your switches via SNMP to build a database of MAC addresses known to peering ports on the switches.

This is done via the update-l2database.pl script.

A long standing issue with this was that it used the database switch.name field to address the switch rather then the more correct switch.hostname field. The script was updated to fix this so you should:

  1. ensure your switch hostnames are correct
  2. recreate the MySQL views as per the upgrade instructions (step 12 at time of writing)
  3. update the script to the latest version

UI Testing with Laravel Dusk

As an end user, how do you know that what you stick in a web-based form gets put into the database correctly? And conversely, how do you know that form represents the database data correctly when editing?

This is an issue we ran into recently around some checkbox logic and a dropdown showing the wrong selected element. These issues are every bit as dangerous to mission critical elements as the output tests we do with PHPUnit.

We now use Laravel Dusk to test mission critical elements on the frontend UI - read all about the details here including an animation of it in action.

Like the PHPUnit tests, the Laravel Dusk tests are performed via Travis CI everytime we push to GitHub.

New Features and Improvements

In addition to those elements mentioned above, the following summarises the new features and improvements in this release.

We have migrated the following controllers to Laravel (including better documentation, code review and improvements):

  • the customer controller. This includes adding, editing and listing customers; adding and editing registration and billing details; customer overview page and all tabs there in; email templates and welcome email; customer logo management; customer notes.
  • the controllers for showing statistic graphs for MRTG and Smokeping have been migrated.
  • all elements around showing peer to peer graphs have been migrated;
  • a new controller for console servers which has been split from switches.
  • a Latency Graph type has been added to Grapher with a concrete implementation for Smokeping.

Other improvements include:

  • You can now permanently delete a customer. Up until now, you just marked a customer account as closed by setting the Date Left on the customer edit form. We would typically only recommend deleting test customer entries as otherwise you will have no customer history in IXP Manager. This is beta and while we believe we have found and handled all associated customer database references, if you find a bug, please open an issue. This closes #338, addresses a number of mailing list requests and internal issue islandbridgenetworks/IXP-Manager-Archive-Yann#108.

  • Implemented the IX-F Member Export Schema v0.7

  • Add a customer using details from PeeringDB. See above.

  • Better proxy support (see v4.7.2 release)

  • Patch Panel enhancements based on real world use:

    • add patch panel actions to the patch panel view page (closes internal issue islandbridgenetworks/IXP-Manager-Archive-Yann#110);
    • LoAs can now be generated and downloaded / emailed at any stage or a patch panel port's lifecycle;
    • add (co-lo) billing ID to patch panel ports;
    • wildcard search for a patch panel port based on co-lo or billing ID from left hand side menu via xc:yyy where yyy is the ID or partial ID;

  • You can disable the frontend links to the peering matrix by setting adding the following to your .env file: IXP_FE_FRONTEND_DISABLED_PEERING_MATRIX=true.

  • [NF] implement garbage collector for bgp session data (closes #182)

Bug Fixes

  • ZF -> Laravel link. Fixes #362
  • ZF -> Laravel links. Fixes #363
  • Ability to change default page title via .env. Fixes #375
  • Do not include AS macro is IX-F export if none configured. Fixes #379
  • Do not show (broken) graphs for non-connected customers. Fixes #376
  • MRTG - escape characters in community strings. Fixes #384
  • Fix AS number popup (currently only works for RIPE registered ASNs) - closes internal issue islandbridgenetworks/IXP-Manager112
  • Issue patch panel ports and apostrophes. Fixes #429
  • and lots more 😄
Assets 2
Loading