Skip to content
This repository has been archived by the owner on Oct 11, 2020. It is now read-only.

Cross devices correlation and closed loop automation (demo with MTU mismatch automatic correction)

Jump to bottom
Khelil Sator edited this page Mar 12, 2019 · 4 revisions

closed loop automation

The then clause in Healtboth rules is used to set actions to be taken.
These actions can be:

  • pre-defined-actions (ex: text message to show, color to show, ...)
  • User-Defined-Actions (UDA).

UDA invoke functions. These functions are implemented in Python.
So you create your own python code, and healthbot will execute it automatically based on the data collected and the rules you defined.

This directory has python functions.
These python functions can be triggered by Healthbot rules.

cross devices correlation

When we configure a rule field, we can use the argument reference to refer to another field from another rule.

Example with MTU

We will use one rule to collect data and update the database, and another rule to process/query the database and take actions.

Collect interfaces MTU

The rule collect-interfaces-mtu.rule uses openconfig to continuously collect interfaces mtu and update the database with this.

The playbook collect-mtu.playbook uses the rule collect-interfaces-mtu.rule.

You need to instanciate the playbook collect-mtu.playbook for a device group.

The playbook collect-mtu.playbook doesnt show any data in Healthbot GUI because the rule collect-interfaces-mtu.rule doesnt use any trigger

Purpose of above rule and playbook is only to collect data and update the database. There is no database processing/queries in the above rule.

Compare interfaces MTU across two differents devices, and enforce same MTU

The rule compare-interfaces-mtu.rule:

  • continuously queries the database updated by the rule collect-interfaces-mtu.rule
  • compares the mtu of 2 connected interfaces (cross devices correlation)
  • and in case of mtu mismacth, it changes the junos configuration (closed loop automation) to enforce the same mtu across the 2 connected interfaces

The rule compare-interfaces-mtu.rule uses User-Defined-Actions (UDA) (python code).
Here's the UDA change-mtu-config.py fired automatically by the rule compare-interfaces-mtu.rule in order to change the junos configurationto enforce the same mtu across the 2 connected interfaces

The rule compare-interfaces-mtu.rule is a network-rule

The rule compare-interfaces-mtu.rule uses these 6 variables to define a connection between two devices (so which interface of which device of which group is connected to which interface of which device of which group):

  • device1-group-variable
  • device1-inter-variable
  • device1-name-variable
  • device2-group-variable
  • device2-inter-variable
  • device2-name-variable

The rule compare-interfaces-mtu.rule uses the field device1-mtu and device2-mtu with the argument reference to refer to data collected by the rule collect-interfaces-mtu.rule

The playbook compare-mtu.playbook uses the rule compare-interfaces-mtu.rule

You need to instanciate the playbook compare-mtu.playbook for a network group

For this topology with 3 spines and 4 leaves,

topology.png

We need 1 instance for each connection.
There are 3 spines, each spine is connected to 4 leaves, so it is 3*4 connections to monitor
So we need 12 instances of the playbook compare-mtu.playbook.
Here's the variables for the 12 instances/connections:

spine vMX1 <-> leaves connections:

device1-group-variable vmx, device1-group-variable vMX1, device1-inter-variable ge-0/0/0 / device2-group-variable vmx, device2-group-variable vMX4, device2-inter-variable ge-0/0/0  
device1-group-variable vmx, device1-group-variable vMX1, device1-inter-variable ge-0/0/1 / device2-group-variable vmx, device2-group-variable vMX5, device2-inter-variable ge-0/0/0  
device1-group-variable vmx, device1-group-variable vMX1, device1-inter-variable ge-0/0/2 / device2-group-variable vmx, device2-group-variable vMX6, device2-inter-variable ge-0/0/0  
device1-group-variable vmx, device1-group-variable vMX1, device1-inter-variable ge-0/0/3 / device2-group-variable vmx, device2-group-variable vMX7, device2-inter-variable ge-0/0/0

spine vMX2 <-> leaves connections:

device1-group-variable vmx, device1-group-variable vMX2, device1-inter-variable ge-0/0/0 / device2-group-variable vmx, device2-group-variable vMX4, device2-inter-variable ge-0/0/1  
device1-group-variable vmx, device1-group-variable vMX2, device1-inter-variable ge-0/0/1 / device2-group-variable vmx, device2-group-variable vMX5, device2-inter-variable ge-0/0/1  
device1-group-variable vmx, device1-group-variable vMX2, device1-inter-variable ge-0/0/2 / device2-group-variable vmx, device2-group-variable vMX6, device2-inter-variable ge-0/0/1  
device1-group-variable vmx, device1-group-variable vMX2, device1-inter-variable ge-0/0/3 / device2-group-variable vmx, device2-group-variable vMX7, device2-inter-variable ge-0/0/1

spine vMX3 <-> leaves connections:

device1-group-variable vmx, device1-group-variable vMX3, device1-inter-variable ge-0/0/0 / device2-group-variable vmx, device2-group-variable vMX4, device2-inter-variable ge-0/0/2  
device1-group-variable vmx, device1-group-variable vMX3, device1-inter-variable ge-0/0/1 / device2-group-variable vmx, device2-group-variable vMX5, device2-inter-variable ge-0/0/2  
device1-group-variable vmx, device1-group-variable vMX3, device1-inter-variable ge-0/0/2 / device2-group-variable vmx, device2-group-variable vMX6, device2-inter-variable ge-0/0/2  
device1-group-variable vmx, device1-group-variable vMX3, device1-inter-variable ge-0/0/3 / device2-group-variable vmx, device2-group-variable vMX7, device2-inter-variable ge-0/0/2

end to end demo

The python script enforce_same_mtu.py executes all the required tasks:

$ python enforce_same_mtu.py
loaded healthbot rule collect-interfaces-mtu.rule
loaded healthbot playbook collect-mtu.playbook
loaded healthbot configuration for the device: vMX1
loaded healthbot configuration for the device: vMX2
loaded healthbot configuration for the device: vMX3
loaded healthbot configuration for the device: vMX4
loaded healthbot configuration for the device: vMX5
loaded healthbot configuration for the device: vMX6
loaded healthbot configuration for the device: vMX7
loaded healthbot configuration for the device group: vmx
healthbot configuration commited!
wait 60 sec
loaded healthbot function change-mtu-config.py
loaded healthbot rule compare-interfaces-mtu.rule
loaded healthbot playbook compare-mtu.playbook
loaded healthbot configuration for the network group: MTU
healthbot configuration commited!
$

Healthbot is collecting the MTU on the interfaces between spines and leaves.
There is currently no MTU mismatch.
MTU_1514.png

In case of MTU mismatch on a link, Healthbot will detect the issue (cross devices correlation) and will fix the MTU mismatch (closed loop automation): it will change the junos configuration to enforce the same MTU across the 2 connected interfaces (in that demo, it will enforce the minimum MTU).

let's connect on vMX1 device, and let's change the MTU on its interface to vMX4. This will create an MTU mismatch issue (between vMX1 and vMX4).

$ ssh 100.123.1.0
Password:
Last login: Mon Mar 11 17:23:00 2019 from 100.123.35.0
--- JUNOS 18.2R1.9 Kernel 64-bit  JNPR-11.0-20180614.6c3f819_buil
jcluser@vMX-addr-0> edit
Entering configuration mode

[edit]
jcluser@vMX-addr-0# set interfaces ge-0/0/0 mtu 1300

[edit]
jcluser@vMX-addr-0# commit and-quit
commit complete
Exiting configuration mode

jcluser@vMX-addr-0>

Healthbot detected the issue MTU-mismatch.png

Healthbot fixed the issue MTU-mismatch-fixed.png

Let's connect on vMX4 to verify

$ ssh 100.123.1.3
Password:
--- JUNOS 18.2R1.9 Kernel 64-bit  JNPR-11.0-20180614.6c3f819_buil
jcluser@vMX-addr-3> show configuration | compare rollback 1
[edit interfaces ge-0/0/0]
+   mtu 1300;

jcluser@vMX-addr-3> show system commit
0   2019-03-12 13:46:41 UTC by jcluser via netconf
Clone this wiki locally