[Technical Design][Draft] Unified Functional Test Framework

[MuhammedIrfan]

Introduction

This design document provides a high-level design of our customized framework that will drive automated functional testing of BTP. We are developing this framework to be as generic as possible so that it can adapt to automate any blockchain that we plan to test in the future. The application is developed in Rust which will interact with the test framework in python using Behave BDD(Behavior Driven Development) approach.

Terminology

Term	Definition	Link
Behave	behave is a behavior-driven development framework	Official Website
Test Helper	This library has chain-specific APIs to interact with Sandbox VM	Example
Test Interface	This interface written in rust will be used to interact with the Test Framework
PYO3	PYO3 helps in accessing code written in rust using python	Repo
Maturin	Tool for building and publishing Rust-based Python packages with minimal configuration	Repo

Considerations

Behave is a tool used for Behavior-driven development (BDD) in Python programming language. In an Agile development framework, BDD creates a culture where testers, developers, business analysts, and other stakeholders of the project can contribute to the software development.

Having a unified framework can use the same tests for multiple blockchains. Having worked with NEAR, it has a sandbox VM to run contracts that can be used for functional testing. Our previous implementation used rust for writing test cases. Going forward we would like to use python for the test framework to unify and reuse the tests across multiple integrations.

Design

High-level Architecture

Components

• Test Helper
• Test Interface
• Python Behave BDD Implementation

Test Helper

In order to initiate and deploy the smart contracts, the test-helper play a major role. Each blockchain will have a dedicated test helper to help with initialization and deployment. Test helper is written in rust.

Test helper consists of 2 folders mainly:

• Actions:

  • Will have functions/methods to Initialize and deploy smart contracts.
  • Also will have other methods to retrieve data or perform some action.
    Ex: get_owner , add_owner

• Types:

  • Will have functions/methods to instantiate contracts.
  • Create a new context.

Test Interface

Test Interface is developed in such a way that it can interact with the test helper of any blockchain. Python automation suite will be accessing this interface to call methods in test helper which in turn will communicate with smart contracts in blockchain.
Test Interface will be written in Rust but it will be wrapped with python bindings so that python test cases can access the test interface.

Python Behave BDD Implementation

The basic structure of tests using Behave consists of the feature file, the step definition file, and so on. Feature file should be present under a folder called features and a subfolder called steps should also be created where step definition will be written in python.

Implementation

PyO3 supports the following software versions:

• Python 3.7 and up (CPython and PyPy)
• Rust 1.48 and up

PyO3 can be used to generate a native Python module and to do this we need to install Maturin.

Install Maturin:

• maturin is published as Python binary wheel to PyPI, you can install it using pip:
  pip install maturin
• Or you can from crates.io using cargo.
  cargo install maturin
• Or from Git Repository
  cargo install --git https://github.com/PyO3/maturin.git maturin

Configuration

Major changes or additions are done in configuration files like cargo.toml, lib.rs and pyproject.toml.
When you create a project using ‘maturin init’ cargo.toml, lib.rs and pyproject.toml files are automatically generated with all dependencies required. You can use that project or copy the configuration to a project where testing has to done.

• Pyproject.toml → this file has to be present in the root folder of the project.

[build-system]
requires = ["maturin>=0.12,<0.13"]
build-backend = "maturin"
 
[project]
name = "test_interface"
requires-python = ">=3.7"
classifiers = [
   "Programming Language :: Rust",
   "Programming Language :: Python :: Implementation :: CPython",
   "Programming Language :: Python :: Implementation :: PyPy",
]

• Cargo.toml → Package for test_interface.

[package]
name = "test_interface"
version = "0.1.0"
authors = ["Icon Foundation<[email protected]>"]
edition = "2018"
 
[lib]
name = "test_interface"
crate-type = ["cdylib"]
 
[dependencies]
test-helper = { path = "../test-helper" }
pyo3 = { version = "0.16.5", features = ["extension-module"] }
pyo3-asyncio = { version = "0.16.0", features = ["attributes", "tokio-runtime"] }
tokio = { version = "1.16.1", features = ["macros"] }

crate-type : "cdylib" is necessary to produce a shared library for Python to import from

• lib.rs → Package for test_interface.
  we can make rust function callable from Python using
  • Importing the pyo3 prelude
  • annotate
    • Struct with #[pyclass]
    • the function with #[pyfunction] to turn it into a PyCFunction
    • Impl with #[pymethods]
  • wrap the result in a PyResult
  • add the function to the #[pymodule]

The following code follows illustrates the above steps in the same order:

use pyo3::{*, types::PyModule};
use test_helper::types::Context as TestHelperContext;
 
#[pyclass]
pub struct Context(TestHelperContext);
 
#[pymethods]
impl Context {
   #[new]
   pub fn new() -> Self {
   }
   pub fn deploy(&self, wasm_file: &str) {   
   }
}
 
#[pyfunction]
pub fn deploy_smartcontract(){
}
 
#[pymodule]
fn test_interface(_py: Python, m: &PyModule) -> PyResult<()> {
   m.add_class::<Context>()?;
   Ok(())
}

NOTE : The name of the library in Cargo.toml matches the name of the function that was annotated with #[pymodule]. Only then can we import the module test_interface in step definition file.

Now that we have a rust program with python annotations ready we can use them in test cases by implementing Behave BDD structure.

.
├── features
│   ├── owner.feature
│   └── steps
│       └── manage_owner.py
└── test_interface

Here’s a brief explanation of the files:

• owner.feature: written out tests for the owner feature in Gherkin language.
• manage_owner.py: The code that runs the tests in owner.feature.
• test_interface : The interface between test helper and feature file.

Owner.feature → Here is an example of a feature file in Gherkin language.
Feature: detailed description of the feature.

Scenario: non_bmc_owner_cannot_remove_owners
 Given New Context
 And BMC_CONTRACT_IS_DEPLOYED_AND_INITIALIZED
 And BMC_CONTRACT_IS_OWNED_BY_ALICE
 And CHARLIE_IS_AN_EXISITNG_OWNER_IN_BMC
 And CHUCKS_ACCOUNT_IS_CREATED
 And BMC_CONTRACT_IS_NOT_OWNED_BY_CHUCK
 And CHARLIES_ACCOUNT_ID_IS_PROVIDED_AS_REMOVE_OWNER_PARAM
 When CHUCK_INVOKES_REMOVE_OWNER_IN_BMC 
 Then BMC_SHOULD_THROW_UNAUTHORISED_ERROR_ON_REMOVING_OWNER

Manage_owner.py → Example of step definition file based on the above feature.

import test_interface
from behave import given, when, then
 
@given('New Context')
def step_impl(context):
  pass
 
@given('"{contract:D}"_IS_DEPLOYED_AND_INITIALIZED')
def step_impl(context, contract):
    pass
 
@given('BMC_CONTRACT_IS_OWNED_BY_ALICE')
def step_impl(context):
   raise NotImplementedError('STEP: Given BMC_CONTRACT_IS_OWNED_BY_ALICE')
 
@given('CHARLIE_IS_AN_EXISITNG_OWNER_IN_BMC')
def step_impl(context):
   raise NotImplementedError('STEP: Given CHARLIE_IS_AN_EXISITNG_OWNER_IN_BMC')
 
@given('CHUCKS_ACCOUNT_IS_CREATED')
def step_impl(context):
   raise NotImplementedError('STEP: Given CHUCKS_ACCOUNT_IS_CREATED')
 
@given('BMC_CONTRACT_IS_NOT_OWNED_BY_CHUCK')
def step_impl(context):
   raise NotImplementedError('STEP: Given BMC_CONTRACT_IS_NOT_OWNED_BY_CHUCK')
 
@given('CHARLIES_ACCOUNT_ID_IS_PROVIDED_AS_REMOVE_OWNER_PARAM')
def step_impl(context):
   raise NotImplementedError('STEP: Given CHARLIES_ACCOUNT_ID_IS_PROVIDED_AS_REMOVE_OWNER_PARAM')
 
@when('CHUCK_INVOKES_REMOVE_OWNER_IN_BMC')
def step_impl(context):
   raise NotImplementedError('STEP: When CHUCK_INVOKES_REMOVE_OWNER_IN_BMC')
 
@then('BMC_SHOULD_THROW_UNAUTHORISED_ERROR_ON_REMOVING_OWNER')

def step_impl(context):
   raise NotImplementedError('STEP: Then BMC_SHOULD_THROW_UNAUTHORISED_ERROR_ON_REMOVING_OWNER')

NOTE : we are able to achieve the import test_interface module by using pyo3 configurations made in the test_interface file.

To Run Test Cases

To run test cases here are the steps.

• Maturin build: cd to test_interface and Make sure to run this command from test_interface where cargo.toml is Configured.
• behave: cd back root directory where feature folder is present and run this command to run all feature files
  • To run a particular feature use the feature file name after behave.
  • To print on the console use - - no - capture
  • Check out more behave commands here

Report

To create a JUnit report, run the command given below
behave –junit
A folder called as the reports gets generated within the project, having the name TESTS-.xml.

Conclusion

Here is why the proposed automation suite is best suited for functional testing.

• Single unified framework to automate tests in different blockchain.
• This framework follows the BDD approach where testers, developers, business analysts, and other stakeholders can understand test
  scenario and contribute towards improvising test scenarios/user stories.
• Test scenarios and code to automate test scenarios are present in different folders and files. This will help BA or other stakeholders to understand test scenarios that we are testing as it will be in simple english without any code.
• Excellent online documentation for Behave and Pyo3.
• Redundancy of code can be eliminated. Similar scenario steps can be mapped to single step definition code.
  • Ex : Initialization and deployment is must to execute all functional test cases. Instead of writing the same code for all scenarios we can simply map them to a single step definition file by writing the same scenario.
• For each test scenario it would be testing against a new instance of sandbox vm.
• We can generate reports as JSON, Junit or view report in a web server using allure.