Skip to content

Testing Documentation

Jump to bottom
Adam Boudj edited this page Jun 5, 2024 · 2 revisions

Contents

Branching Tree Testing (BTT)

Branching Tree Testing (BTT) is a method that enhances the readability and maintainability of a test suite by organizing tests in a hierarchical structure that mirrors the logical flow of the application. This approach simplifies navigation and understanding, making it accessible to all team members, including non-technical ones.

Benefits of BTT

  • Improved Readability: A hierarchical structure that clearly shows the relationship between test cases and functionalities.
  • Ease of Maintenance: You can easily track and update changes within the corresponding branch of the test tree.
  • Enhanced Collaboration: Facilitates collaboration by providing a clear and organized view of the test scenarios accessible even to non-technical team members.

Implementing BTT

  1. Define the Tree Structure: Break down the application into logical components and define a tree structure representing these components and their relationships.
  2. Organize Tests Hierarchically: Place each test case within the appropriate branch, reflecting the application's logical flow.
  3. Use Clear and Descriptive Names: Name each branch and test case descriptively to convey their purpose clearly.
  4. Regularly Review and Update: Continuously review and update the test tree to align with the application's evolving structure and requirements.

Further Reading

For practical examples and a detailed exploration of Branching Tree Testing, refer to the following resources:

Foundry Testing

Test Structure

test/foundry
├── fork
│   ├── arbitrum
│   └── base
├── integration
├── shared
│   └── interfaces
├── unit
│   ├── concrete
│   └── fuzz
└── utils
  • Fork: Tests on specific Ethereum forks (e.g., Arbitrum).
  • Integration: Tests for component and module integration.
  • Shared: Common test utilities and interfaces.
  • Unit/Concrete: Unit tests for specific modules and functions.
  • Fuzz: Fuzz tests with random and extreme inputs.
  • Utils: Utility scripts and helper functions.

Note

Foundry supports advanced testing capabilities like fuzz testing and symbolic execution, making it a powerful tool for Solidity testing.

Test Metrics

  • Total Test Cases: 394
  • Types of Tests:
    • Unit Tests: Test individual functions and modules.
    • Integration Tests: Verify interactions between components.
    • Fuzz Tests: Apply random inputs to uncover vulnerabilities.
    • Security Tests: Static analysis for potential security issues.
    • Performance Tests: Measure gas consumption and execution efficiency.

Coverage Report

Overall Coverage:

Metric Coverage Total Hit
Lines 100.0% 308 308
Functions 100.0% 98 98

Directory Coverage:

Directory Line Coverage Line Rate Total Lines Hit Lines Function Coverage Function Rate Total Functions Hit Functions
contracts/ 100.0% 100.0% 120 120 100.0% 100.0% 26 26
contracts/base/ 100.0% 100.0% 86 86 100.0% 100.0% 36 36
contracts/common/ 100.0% 100.0% 7 7 100.0% 100.0% 4 4
contracts/factory/ 100.0% 100.0% 63 63 100.0% 100.0% 18 18
contracts/modules/validators/ 100.0% 100.0% 14 14 100.0% 100.0% 8 8
contracts/utils/ 100.0% 100.0% 18 18 100.0% 100.0% 6 6

Hardhat Testing

Test Structure

test/hardhat
├── common
├── smart-account
└── utils
  • Common: Tests for shared components and utilities.
  • Smart-Account: Tests specific to the Nexus smart account, covering functionalities like batch execution and module management.
  • Utils: Utility scripts, type definitions, and helper functions.

Test Metrics

  • Total Test Cases: 102
  • Types of Tests:
    • Unit Tests: Focus on individual functions and modules.
    • Integration Tests: Verify interactions between multiple components.
    • Fuzz Tests: Apply random inputs to uncover vulnerabilities.
    • Security Tests: Static analysis for potential security issues.
    • Performance Tests: Measure gas consumption and execution efficiency.

Coverage Report

Overall Coverage:

Metric Coverage
Statements 82.91%
Branches 64.08%
Functions 90.32%
Lines 83.96%

Directory Coverage:

Directory % Statements % Branches % Functions % Lines
contracts/ 70.64% 58.47% 73.08% 65.63%
contracts/base/ 90.48% 63.33% 97.3% 91.11%
contracts/common/ 100% 100% 100% 100%
contracts/factory/ 100% 71.05% 100% 100%
contracts/interfaces/ 100% 100% 100% 100%
contracts/interfaces/base/ 100% 100% 100% 100%
contracts/interfaces/common/ 100% 100% 100% 100%
contracts/interfaces/factory/ 100% 100% 100% 100%
contracts/interfaces/modules/ 100% 100% 100% 100%
contracts/modules/validators/ 81.82% 62.5% 87.5% 84.62%
contracts/types/ 100% 100% 100% 100%

Test Automation

We utilize Foundry and Hardhat frameworks for test automation due to their unique advantages:

  • Foundry: Known for its speed and support for fuzz testing, allowing us to run a large number of test cases quickly.
  • Hardhat: Provides excellent integration with the Ethereum ecosystem, enabling us to simulate bundlers and interact with the Nexus SDK. Hardhat also supports a wide range of plugins and tools for comprehensive testing and development workflows.

Gas Usage Reporting Script

Purpose and Importance:

The gas usage reporting script is essential for accurately measuring and reporting the gas consumption of our smart contracts. By automating this process, it ensures transparency and consistency in our performance metrics, which is vital for developers, auditors, and stakeholders.

Scope and Significance:

  • Comprehensive Scenario Coverage: The script includes tests for various scenarios such as ERC20, ERC721, native ETH transfers, and Uniswap V2 swaps. These scenarios are tested under different conditions (e.g., with paymaster, deployed vs. non-deployed Nexus Accounts) to ensure thorough analysis.
  • Accurate Reporting: The script generates detailed gas usage reports that help in evaluating contract performance. These reports aid in making informed decisions and provide a clear view of the contract's efficiency.
  • Addressing Foundry Limitations: Foundry's gas reporting can be influenced by fuzz testing and seed variations, leading to inaccurate gas usage estimates. Our script mitigates these issues by using a specific set of non-fuzz, positive tests prefixed with "test_Gas". This ensures consistent and reliable gas usage data.
  • Benchmark Testing: It includes a suite of benchmark tests covering main usages in both cold and warm storage environments. Testing is done on forks at a fixed block number to ensure consistent gas usage over time.

Important

Regularly run the gas usage reporting script to monitor gas usage

Test Coverage Reporting Script

Purpose and Importance

The test coverage reporting script generates comprehensive coverage reports, highlighting the extent to which the codebase is tested. This is vital for identifying untested areas, improving test suites, and ensuring high code quality and reliability.

Important

Comprehensive test coverage is crucial for identifying gaps in testing and ensuring the reliability of smart contracts.

Scope and Significance

  • Targeted Coverage: The script excludes non-essential directories (e.g., test, mock, node_modules) to focus on relevant parts of the codebase, producing accurate coverage metrics.
  • Visual Reports: It generates HTML reports for better visualization, aiding developers and auditors in assessing test coverage effectively.
  • Enhancing Test Suites: By identifying gaps in coverage, the script helps in refining the test suite, leading to more robust and dependable code.
  • Handling Foundry Limitations: Foundry's coverage tools have specific limitations, such as:
    • Branch Coverage Issues: Some branches are covered by tests but not reported as covered.
    • File Exclusion Difficulties: Foundry does not provide an exclusion mechanism for files. As a workaround, we use lcov commands to filter out unnecessary files and focus on relevant code, thereby ensuring cleaner and more accurate coverage reports.

Tools and Scripts for Testing

Static Analysis Tools

  • Slither: Identifies potential security vulnerabilities through static analysis.
  • OpenZeppelin Code Inspector: Helps mitigate common security issues in smart contracts.

Custom Scripts for Coverage and Gas Reporting

Coverage Script: A shell script that generates lcov.info, installs lcov if necessary, excludes certain directories from the coverage report, and generates an HTML report.

Gas Reporting Script: A JavaScript script that runs forge tests, parses the log file for gas usage, generates a report, and compares it with the previous gas report.

Summary of Testing Efforts

  • Unit Tests: Ensured each function and module works as intended.
  • Integration Tests: Verified seamless interaction between components.
  • Fuzz Tests: Identified vulnerabilities with random and extreme inputs.
  • Security Tests: Preemptively discovered and addressed security issues.
  • Performance Tests: Optimized gas consumption and execution efficiency.

Caution

Failing to maintain rigorous testing practices can lead to undetected vulnerabilities and inefficiencies in smart contracts.

Conclusion

This document serves as a comprehensive guide to our testing strategy, methodologies, and results. Our rigorous testing process ensures the Nexus protocol is robust, secure, and reliable. By following best practices and leveraging advanced tools and techniques, we maintain high standards of quality and security.

Branching Tree Technique (BTT) enhances the readability and maintainability of our test suite, making it accessible even to non-technical team members. Our testing strategy includes unit tests, integration tests, fuzz tests, security tests, and performance tests.

We use Foundry for its speed and advanced features and robust debugging tools. Hardhat is utilized for its excellent integration with the Ethereum ecosystem and support for comprehensive testing workflows.

Custom scripts for coverage and gas reporting ensure accurate and detailed reports. Static analysis tools like Slither and OpenZeppelin Code Inspector help identify and mitigate security vulnerabilities.

Note

High test coverage and automated testing frameworks provide confidence in the robustness and reliability of the Nexus protocol. For more information on best practices, issues, and methodologies, refer to:

Clone this wiki locally