Spanner CI

Table of Contents

• Introduction
• How It Works
• Creating An Account
• Integrations
  • GitHub
  • Gitlab
  • Bitbucket
• Projects
  • Project Settings
• Jobs
• Environment Variables
• Configuration with .spannerci.yml
• Test Scripts
• Build Binary
• Artifacts
• Testboards
• Spanner CLI
• Quick Start Guide

Introduction

Spanner CI is a Continuous Integration Platform for IoT and Embedded Devices. It can be used for automated code quality checks, automated firmware builds and automated functional tests in real hardware that run with every new version of your firmware.

This repository contains everything that is needed to get started with Spanner CI. It's highly recommended to make a fork of this repository and use it as a starting point to understand how Spanner CI can be easily integrated with your own github or gitlab repository. More details about this can be found in the Quick Start Guide section.

In the following sections you'll find detailed information about Spanner CI.

How It Works

As soon as we create an account and make a new Project in Spanner CI Platform, we can add a .spannerci.yml configuration file in the root directory of our source code repository to trigger Spanner CI for every commit or pull request. Each time Spanner CI is triggered, it reads the instructions from .spannerci.yml file and starts one or more virtual environments for firmware building or functional testing of the hardware. For the functional testing, one or more user-defined test scripts contain all the device functional tests. Before running the tests, each device under test is updated with the new firmware. Moreover, a Spanner Testboard can be used to test all inputs and outputs of our device. The Testboard is connected both with the device (wired or wireless) and with the Spanner Platform over the network. Please make sure to check the Quick Start Guide section for more details.

Creating an Account

If you haven't used Spanner CI before, you can create a new account by visiting the Create an Account page. To sign-up, you can either use your GitHub account (recommended) or create a new account directly from Spanner.

Option 1: Sign-Up with GitHub

If you already have a GitHub account click the SIGN UP WITH GITHUB button and authorize Spanner to use GitHub.

After that you'll be asked to fill your organization info:

Next, you have to enable the GitHub integration. For more information about this, please visit the Integrations section.

Option 2: Regular Sign-Up

1. Enter your basic information and click REGISTER to continue.
2. Use the credentials provided above to sign-in.
3. Enter your organization info to access the Spanner UI.
4. Make sure to check the Integrations section, on how to enable an integration with a code repository, either GitHub or GitLab.

Integrations

Spanner CI integrates with multiple third-party services for external authentication, code hosting, notifications etc. For code hosting, either GitLab or GitHub can be used.

This readme contains everything that is needed to get started using Spanner CI integrations. It's highly recommended to use it as a starting point to understand how Spanner CI can be easily integrated with your own source code repositories. More details about this can be found below.

GitHub Integration

Go to the Integrations page, click the GitHub integration and then the Install The Github App button, to install the Spanner GitHub App (needed to get access to user repos).

In the next page, check the All repositories option and click the Install button. You'll be redirected to GitHub, where you'll be able to check the installed Spanner App. Next, go back to the Integrations page, and click the GitHub integration. You'll be able to see the GitHub Authenticated and App Installed messages. Click the Enable Integration button to complete the integration. You're done. From now on, each time that you click on the GitHub Integration, you'll be able to see the available repositories that are synced with Spanner.

Gitlab Integration

• If you already have a Gitlab account, go to the Gitlab Profile Settings

• Click on Access Token on the Left Sidebar Menu
• Give a friendly name to your Access Token and under Scopes select api as shown below:

• Copy the generated token to your clipboard in order to paste it on Spannner CI Platform

• Sign in to Spanner CI Platform and go to the Integrations page

• Click on Gitlab Integration and paste the access token from clipboard to the corresponding field

Update the access token to Spanner CI platform Integrations as many times as you want. Also, do not forget to save your Gitlab access token - you won't be able to access it again.

• To trigger Spanner CI on each push/pull request per repository, copy the GitLab Webhook Secret Token from Spanner CI (as shown below) and then click on Integrations in your Gitlab Project. Fill the URL with http://console.spannerci.com:4000/gitlab/app/hooks and paste your Webhook Secret Token. Choose Trigger events (Push or/and Merge request events) whenever you want to trigger Spanner CI.

Bitbucket Integration

To integrate Spanner CI with your Bitbucket account, you are going to create an app password for your account, as well as setting up a webhook, to automatically trigger Spanner CI on your repository's push and pull-request events.

• To create an app password go to Account Settings > Access Management > App Passwords and press on the Create App Password button. Enter a user-friendly name to remember where this password is used for, and click on the read repository permission checkbox.

  

• Click on the create button, and a new dialog would be shown, containing your new password. Please copy this password and head to Spanner CI integrations page.

  

• On the Spanner CI integrations page, click on the Enable Bitbucket Integration button. Fill the password which you created earlier, and your bitbucket's account username.

  

• Copy the Bitbucket Webhook Secret Token and click the Enable Integration button. This token is needed in order for Spanner CI to associate webhook events to your account.

• Head to the repository you wish to enable webhooks. Click on Settings > Webhooks and click on Create Webhook button. This step is needed in all the repositories you wish to trigger Spanner CI automatically on push or pull-request events.

  The URL of this Webhook example would be: 
  http://console.spannerci.com:4000/bitbucket/app/hooks/?webhook_token=ae8506c80c72310d91f9f9039348c5ac 
  

  

Projects

Spanner supports the creation of one or more Projects. Each Project represents one user source code repository. It is assumed that you've already enabled an integration with GitHub or Gitlab. If not, please check the Integrations section.

To create a new Project:

1. Select Projects from the navigation menu on the left side of the dashboard.
2. Click on New Project.
3. Select the preferred source code repository.
4. Add one or more Testboards.
5. Click Finish to complete the Project creation.

Project Settings

Some useful project settings are:

• Trigger Options :
  1. The Spanner CI has as default the Pull Request option, which triggers a job after each pull request (Merge request in GitLab).
  2. If you want to trigger Spanner CI on each push, you can enable the Pushed branch too.
• Notifications :
  1. Additionally, Spanner CI provides the option of email notifications, for completed jobs. The only step you have to take is to enable the Email checkbox and register an email address in Account Settings page.

Jobs

Each time Spanner CI is triggered from a source code commit or pull request a new Job is created automatically. Every Job gets the instructions on what to do from the .spannerci.yml file and then runs inside a virtual environment. Upon completion, Jobs provide the output result for each running stage and the resulted artifacts (e.g firmware binaries), if any. Moreover, in the Testing stage, the output result for each test case is provided.

New Jobs can also be created manually. Each Job belongs to a specific Project. A list with all the Project Jobs can be found under the Project Info page (accessible by clicking a Project name in the Projects page). From there, it's possible to get various information about each Job, download any artifacts or even watch any Job in runtime or at a later time.

Also note, that because of the Spanner integration with GitHub & Gitlab, it's possible to watch the Job result directly from GitHub or Gitlab, after creating a new Pull Request.

Environment Variables

Spanner supports the definition of environment variables for each Project. They can be defined in the Environment Variables section, under the Project Settings page. They can be referenced in the .spannerci.yml file using the $ prefix, e.g $MY_VAR, and we use them in two differrent ways:

• Throughout the .spannerci.yml file, where they will be replaced by the actual value, as defined in the Project Settings. For example: access_token: $MY_ACCESS_TOKEN.
• Inside the env_vars stage parameter of building or testing stage, where they will be imported directly in the virtual environment, for example:

env_vars:
    - $DB_NAME
    - $ACCESS_TOKEN

And since our test scripts run directly from the virtual environment (which is Linux-based), if needed, we can reference them as any normal linux environment variable, for example:

os.environ['DB_NAME']
os.environ['ACCESS_TOKEN']

Spanner supports a number of pre-defined environment variables, that are either mandatory to use some of the Spanner builders and OTA update methods or just convenient. All Spanner environment variables start with the SPN_ prefix. For example:

• SPN_BUILDER_SDK: is the SDK directory of the selected builder

• SPN_PROJECT_DIR: is the directory in which the source code repository is cloned in the virtual environment

• SPN_OUTPUT_BINARY: is the filename of your generated firmware binary

  ...

Please contact us to get a full list of the Spanner pre-defined environment variables.

Configuration with .spannerci.yml

Spanner CI enables continuous integration by adding a .spannerci.yml file in the root directory of your repository. This, together with some more configuration options that are mentioned later, make every new commit or pull request to automatically trigger Spanner.

Basically, the .spannerci.yml tells Spanner what to do. By default, it runs with two stages:

1. build_binary stage for firmware builds
2. testing stage for functional tests on real hardware

You don't need to use all the above stages and stages with no definitions will be ignored. Each stage contains definitions on what to do. A sample .spannerci.yml file is shown below:

build_binary:
    builder: 'particle'
    binary_name: 'firmware/target/firmware.bin'
    script: cd $SPN_BUILDER_SDK && make PLATFORM=photon APPDIR=$SPN_PROJECT_DIR/firmware/particle

testing:
    script: 'pytest -s --verbose testing/basic-tests/GPIO/read-digital-output/scenario.py'
    device_update:
        ota_method: 'particle'
        devices:
            - $DEVID_1
            - $DEVID_2
        access_token: $PARTICLE_TOKEN
        binary: auto

A stage is defined by a list of parameters that define the stage behavior.

Keyword	Scope	Required	Description
builder	build_binary	Yes	Defines the preferred build environment (1)
binary_name	build_binary	No (5)	Name of the generated binary file based on the SDK platform's documentation
env_vars	build_binary/testing	No	Defines a list with environment variables that will be passed in the virtual environment
script	build_binary/testing	Yes	Defines the script or command to execute in building or testing stage
device_update	testing	No	Enables OTA update of devices before testing
devices	device_update	Yes	Defines a list of devices to apply the OTA update
ota_method	device_update	Yes	Defines the preferred method for OTA updates (2)
binary	device_update	Yes	Defines the binary source for OTA updates (auto (5), URL or repo path)
access_token	device_update	Yes	Defines the access token of the OTA platform
access_key_id	device_update	Yes	Defines the access key id of the OTA platform (AWS only)
secret_access_key	device_update	Yes	Defines the secret key id of the OTA platform (AWS only)
s3_bucket	device_update	Yes	Defines the s3 bucket name of the OTA platform (AWS only)
signer_profile	device_update	Yes	Defines the signer profile name of the OTA platform (AWS only)
aws_region	device_update	Yes	Defines the AWS region of the OTA platform (AWS only)
targets	device_update	Yes	Defines the device targets of the OTA platform (AWS only)
roleArn	device_update	Yes	Defines the ARN role of the OTA platform (AWS only)
pre_flight	global/build_binary/testing	No	Defines commands that should run before a stage (3)
post_flight	global/build_binary/testing	No	Defines commands that should run after a stage (3)
artifacts	build_binary/testing	No	Define a list of paths (4)

(1),(2): Please contact us to get a full list of the currently supported device builders and OTA update methods. To get started, make sure to check the Quick Start Guide section.

(3): pre_flight and post_flight parameters can have global or stage scope. It's possible to overwrite the globally defined pre_flight and post_flight if you set it per stage:

pre_flight:
    - global before building stage

build_binary:
    pre_flight:
        - execute this instead of global pre_flight
    script: my command
    post_flight:
        - execute this after my script

(4): artifacts parameter has stage scope and will create a zip file containing each given path.

build_binary:
    builder: "particle"
    script: cd $SPN_BUILDER_SDK && make PLATFORM=photon APPDIR=$SPN_PROJECT_DIR/app
    artifacts:
        # upload the build's output folder (paths are
        # relative to the repository path)
        - 'app/target/'

testing:
    script: 'pytest --junitxml=output.xml "examples/my_test.py"'
    artifacts:
        # Uploads the generated output file
        - 'output.xml'

You can download the zip file for each script, clicking on button Download Artifacts in Project Info page or in Job Info page of Project.

(5): binary_name, is only required when used with binary:auto option, in the device_update section.

Test Scripts

Test Scripts are user defined scripts that contain a list of functional tests to be performed in one or more devices. Test Scripts are executed within a virtual Linux-based environment. Currently, they can be written in Python and we can trigger them by using the script parameter from the Testing stage. By default the 'pytest' testing framework is supported (already pre-installed in the virtual environment) but there is no restriction if someone wants to use another testing framework. If unsure, we recommend using pytest. In the most primitive form, a test script looks like the one below:

# Sample/Dummy Spanner CI test script. Pytest testing framework is used throughout
# all our examples.
import pytest

# This is a sample test case. Note that each test case starts with 'test_'
def test_dummy_case():
    # We can use the standard Python 'assert'
    assert 1 > 0

One of the most important aspects of Spanner Test Scripts are the Spanner Testboards, that enable the user to write hardware-in-the-loop functional tests. In other words, user is able to test the inputs and outputs of the device. Testboards can be added to the test scripts by importing the SpannerTestboard module. For example:

from SpannerTestboard import SpannerTestboard
testboard = SpannerTestboard("testboard_name")

Example Test Scripts can be found under testing folder in this repository. Choose the one that you want to experiment with, by updating the command in the script parameter of the Testing stage. The Test Scripts are split into three categories:

• 1.basic-tests, which only perform one action and one test, to showcase that individual test function
• 2.simple-tests, which perform a simple real-world scenario, i.e. Turn Light on through Network Command
• 3.complex-tests, which perform a more common and complex real-world scenario, and whose goal is to showcase what an actual Functional test for a real product would test, with more than one assertions and using multiple APIs.

Each Test Script contains documentation for the specific use case. To understand the usage of Test Scripts, make sure to check the Quick Start Guide section.

Build Binary

The Build Binary stage, is the place to build your product's firmware. Some of our pre-configured builders for jump-starting your project, include builders for Particle devices, ESP32 devices and many other hardware platforms. A generic vanilla ubuntu 18.04 builder is also available for maximum flexibility on customizing your own build environment.

The Spanner Build Binary stage, can also handle over the air updates (OTA), for a number of supported builders, via 3rd party cloud services (e.g. Amazon AWS).

An example of the ubuntu:18.04 builder is shown bellow. In this example, we install the gcc arm toolchain, and we export the arm-none-eabi-gcc executable (just for example, in a real scenario the device firmware, would be exported).

# Sample/Dummy Spanner CI build binary.
build_binary:
    builder: 'ubuntu:18.04'
    script:
        - apt-get update && apt-get install -y make libarchive-zip-perl git gcc-multilib vim
        - apt-get install wget
        - wget https://launchpad.net/gcc-arm-embedded/4.9/4.9-2015-q3-update/+download/gcc-arm-none-eabi-4_9-2015q3-20150921-linux.tar.bz2
        - tar xjf gcc-arm*.tar.bz2
        - export PATH=$PATH:$SPN_PROJECT_DIR/gcc-arm-none-eabi-4_9-2015q3/bin
        - echo $PATH && arm-none-eabi-gcc --version
    artifacts:
        - gcc-arm-none-eabi-4_9-2015q3/bin/arm-none-eabi-gcc

Testboards

Spanner Testboards are off the shelf hardware boards loaded with custom firmware from Spanner. They enable the control of inputs and outputs of the Product either wired or wireless. They communicate with Spanner CI Platform over a network interface. Testboards can be added from the Testboards Page in the Spanner CI Platform and they can be then assigned to a Spanner Project. To add a new Testboard, the Testboard's Device ID is needed, which is printed on the board or supplied by Spanner. The Testboard name can be used in the Test Scripts to refer to a specific Testboard.

As soon as we create a new Testboard, we can assign it to a new Project by clicking the assign to Project icon, under the Actions column.

Alternatively, we can do that, during the Project creation.

Spanner CLI

Spanner provides a Command Line Interface (CLI) which can be used instead of the Web Interface. For more information please contact us.

Quick Start Guide

This section is a step-by-step guide for all the new Spanner CI users. If you still have questions, please contact us.

The structure of the current repository is shown below:

• firmware: This folder contains an example firmware application for the Particle Photon device.
• testing: This folder contains a number of example Test Scripts.
• .spannerci.yml: This is the Spanner configuration file
• Readme.md: The current Readme file.

Example 1: Continuous Integration using the Spanner Build Service

• Step 1: Use a GitHub account. Currently Spanner works with either GitHub or GitLab for Git hosting. For the rest of our examples, GitHub is used by default so make sure to use a GitHub account.

• Step 2: Fork the current repository (https://github.com/spannerci/spanner-examples) into your own account. This repository contains all the files that you need to get started, together with examples. From now on it is assumed that you work with the forked repository from your account.

• Step 3: Create a Spanner Account.

• Step 4: Create a Spanner Project. Do not define any Testboards.

• Step 5: Open and review the .spannerci.yml, located in the root of your forked repository. As you can see, only the build_binary stage is enabled. From the build_binary parameters, we understand that the particle builder will be used. The script indicates the make command that will build a binary named firmware/target/firmware.bin declared as a binary_name, ready to be flashed in a Particle Photon device. Leave the default values and close the file.

• Step 6: Now that we setup everything, we will make a change in our firmware in a new branch, create a Pull Request and check how Spanner will be triggered. Open the application.cpp file under the firmware/particle folder, directly from the GitHub page by clicking the Edit this file pencil icon. Just add a new line and then go in the bottom of the page, and check the Create a new branch for this commit and start a pull request.. Click the Commit Changes button.

• Step 7: As soon as you create the Pull Request in the above step, Spanner CI will be triggered and start checking if the Pull Request is valid. In our specific case it will build the firmware again and if everything is good, GiHub will show a All checks have passed notification.

• Step 8: Go to the Spanner CI Platform and check the newly created Job under the Project that you created (click on the project name to see the Project Info and the Job that was created). There you can see more info about the Job and download the binary file for the Particle Photon device.

Repeat Steps 6-8 as many times as you like and enjoy continuous integration in your firmware!

Example 2: Using the Spanner Testing Service

• Step 1: Follow the Steps 1-4 from Example 1.
• Step 2: Open the .spannerci.yml, located in the root of your forked repository. Comment everything in the build_binary stage and uncomment everything from the testing stage. Commit the changes (You can do this directly from the GitHub web page, using the Edit this file pencil icon. As you can see from the script parameter of the testing stage, the sample-test-script.py script will be executed.
• Step 3: Follow the Steps 6-8 from Example 1.

Repeat Step 3 as many times as you like and enjoy continuous integration with automated testing in your firmware!

Example 3: Using Build and Testing Service with Over-The-Air device updates

For this example, a Particle Photon device is required and an active account in the Particle Platform. This example uses the Spanner CI Build Binary service to build a new binary for the Photon device and then uses the Spanner CI Testing Service to update the firmware of the Photon device and run the functional tests included in the sample-test-script.py.

• Step 1: Follow the Steps 1-4 from Example 1.

• Step 2: Go to the Project Settings Page of Spanner CI Platform and add one Environment Variable, with name DEVID_1 and value the Particle Device ID and another with name SPN_PARTICLE_TOKEN and value the Particle Access Token of your account.

• Step 3: Open the .spannerci.yml, located in the root of your forked repository. Add the following lines in the end of the file, to enable the testing stage with OTA device updates with a device binary that was produced in th build_binary stage:

  testing:
      script: 'pytest -s --verbose testing/sample-test-script.py'
      env_vars:
          - $SPN_PARTICLE_TOKEN
      device_update:
          devices:
              - $DEVID_1
          ota_method: 'particle'
          binary: auto
  

• Step 4: Follow the Steps 6-8 from Example 1.

Repeat Step 4 as many times as you like and enjoy continuous integration with automated testing and Over-The-Air updates in your firmware!

Example 4: Using Testboards with the Spanner Testing Service

TODO