ApiTestEngine

Design Philosophy

Take full reuse of Python's existing powerful libraries: Requests, unittest and Locust. And achieve the goal of API automation test, production environment monitoring, and API performance test, with a concise and elegant manner.

Key Features

• Inherit all powerful features of Requests, just have fun to handle HTTP in human way.
• Define testcases in YAML or JSON format in concise and elegant manner.
• Supports function/variable/extract/validate mechanisms to create full test scenarios.
• With debugtalk.py plugin, module functions can be auto-discovered in recursive upward directories.
• Testcases can be run in diverse ways, with single testset, multiple testsets, or entire project folder.
• Test report is concise and clear, with detailed log records. See PyUnitReport.
• Perfect combination with Jenkins, running continuous integration test and production environment monitoring. Send mail notification with jenkins-mail-py.
• With reuse of Locust, you can run performance test without extra work.
• It is extensible to facilitate the implementation of web platform with Flask framework.

Background Introduction (中文版) | Feature Descriptions (中文版)

Installation/Upgrade

$ pip install git+https://github.com/debugtalk/ApiTestEngine.git#egg=ApiTestEngine --process-dependency-links

To upgrade all specified packages to the newest available version, you should add the -U option.

$ pip install -U git+https://github.com/debugtalk/ApiTestEngine.git#egg=ApiTestEngine --process-dependency-links

If there is a problem with the installation or upgrade, you can check the FAQ.

To ensure the installation or upgrade is successful, you can execute command ate -V to see if you can get the correct version number.

$ ate -V
ApiTestEngine version: 0.6.0

Execute the command ate -h to view command help.

$ ate -h
usage: ate [-h] [-V] [--log-level LOG_LEVEL] [--report-name REPORT_NAME]
           [--failfast]
           [testset_paths [testset_paths ...]]

Api Test Engine.

positional arguments:
  testset_paths         testset file path

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         show version
  --log-level LOG_LEVEL
                        Specify logging level, default is INFO.
  --report-name REPORT_NAME
                        Specify report name, default is generated time.
  --failfast            Stop the test run on the first error or failure.

use jenkins-mail-py plugin

If you want to use ApiTestEngine with Jenkins, you may need to send mail notification, and jenkins-mail-py will be of great help.

To install mail helper, run this command in your terminal:

$ pip install -U git+https://github.com/debugtalk/jenkins-mail-py.git#egg=jenkins-mail-py
$ ate -V
jenkins-mail-py version: 0.2.5
ApiTestEngine version: 0.6.0

With jenkins-mail-py installed, you can see more optional arguments.

$ ate -h
usage: ate [-h] [-V] [--log-level LOG_LEVEL] [--report-name REPORT_NAME]
           [--failfast] [--mailgun-api-id MAILGUN_API_ID]
           [--mailgun-api-key MAILGUN_API_KEY] [--email-sender EMAIL_SENDER]
           [--email-recepients EMAIL_RECEPIENTS] [--mail-subject MAIL_SUBJECT]
           [--mail-content MAIL_CONTENT] [--jenkins-job-name JENKINS_JOB_NAME]
           [--jenkins-job-url JENKINS_JOB_URL]
           [--jenkins-build-number JENKINS_BUILD_NUMBER]
           [testset_paths [testset_paths ...]]

Api Test Engine.

positional arguments:
  testset_paths         testset file path

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         show version
  --log-level LOG_LEVEL
                        Specify logging level, default is INFO.
  --report-name REPORT_NAME
                        Specify report name, default is generated time.
  --failfast            Stop the test run on the first error or failure.
  --mailgun-api-id MAILGUN_API_ID
                        Specify mailgun api id.
  --mailgun-api-key MAILGUN_API_KEY
                        Specify mailgun api key.
  --email-sender EMAIL_SENDER
                        Specify email sender.
  --email-recepients EMAIL_RECEPIENTS
                        Specify email recepients.
  --mail-subject MAIL_SUBJECT
                        Specify email subject.
  --mail-content MAIL_CONTENT
                        Specify email content.
  --jenkins-job-name JENKINS_JOB_NAME
                        Specify jenkins job name.
  --jenkins-job-url JENKINS_JOB_URL
                        Specify jenkins job url.
  --jenkins-build-number JENKINS_BUILD_NUMBER
                        Specify jenkins build number.

Write testcases

It is recommended to write testcases in YAML format.

And here is testset example of typical scenario: get token at the beginning, and each subsequent requests should take the token in the headers.

- config:
    name: "create user testsets."
    variable_binds:
        - user_agent: 'iOS/10.3'
        - device_sn: ${gen_random_string(15)}
        - os_platform: 'ios'
        - app_version: '2.8.6'
    request:
        base_url: http://127.0.0.1:5000
        headers:
            Content-Type: application/json
            device_sn: $device_sn

- test:
    name: get token
    request:
        url: /api/get-token
        method: POST
        headers:
            user_agent: $user_agent
            device_sn: $device_sn
            os_platform: $os_platform
            app_version: $app_version
        json:
            sign: ${get_sign($user_agent, $device_sn, $os_platform, $app_version)}
    extract_binds:
        - token: content.token
    validators:
        - {"check": "status_code", "comparator": "eq", "expected": 200}
        - {"check": "content.token", "comparator": "len_eq", "expected": 16}

- test:
    name: create user which does not exist
    request:
        url: /api/users/1000
        method: POST
        headers:
            token: $token
        json:
            name: "user1"
            password: "123456"
    validators:
        - {"check": "status_code", "comparator": "eq", "expected": 201}
        - {"check": "content.success", "comparator": "eq", "expected": true}

Function invoke is supported in YAML/JSON format testcases, such as gen_random_string and get_sign above. This mechanism relies on the debugtak.py hot plugin, with which we can define functions in debugtak.py file, and then functions can be auto discovered and invoked in runtime.

For detailed regulations of writing testcases, you can read the QuickStart documents.

Run testcases

ApiTestEngine can run testcases in diverse ways.

You can run single testset by specifying testset file path.

$ ate filepath/testcase.yml

You can also run several testsets by specifying multiple testset file paths.

$ ate filepath1/testcase1.yml filepath2/testcase2.yml

If you want to run testsets of a whole project, you can achieve this goal by specifying the project folder path.

$ ate testcases_folder_path

When you do continuous integration test or production environment monitoring with Jenkins, you may need to send test result notification. For instance, you can send email with mailgun service as below.

$ ate filepath/testcase.yml --report-name ${BUILD_NUMBER} \
    --mailgun-api-id samples.mailgun.org \
    --mailgun-api-key key-3ax6xnjp29jd6fds4gc373sgvjxteol0 \
    --email-sender [email protected] \
    --email-recepients ${MAIL_RECEPIENTS} \
    --jenkins-job-name ${JOB_NAME} \
    --jenkins-job-url ${JOB_URL} \
    --jenkins-build-number ${BUILD_NUMBER}

Performance test

With reuse of Locust, you can run performance test without extra work.

$ locusts -V
[2017-08-26 23:45:42,246] bogon/INFO/stdout: Locust 0.8a2
[2017-08-26 23:45:42,246] bogon/INFO/stdout:

For full usage, you can run locusts -h to see help, and you will find that it is the same with locust -h.

The only difference is the -f argument. If you specify -f with a Python locustfile, it will be the same as locust, while if you specify -f with a YAML/JSON testcase file, it will convert to Python locustfile first and then pass to locust.

$ locusts -f examples/first-testcase.yml
[2017-08-18 17:20:43,915] Leos-MacBook-Air.local/INFO/locust.main: Starting web monitor at *:8089
[2017-08-18 17:20:43,918] Leos-MacBook-Air.local/INFO/locust.main: Starting Locust 0.8a2

In this case, you can reuse all features of Locust.

That’s not all about it. With the argument --full-speed, you can even start locust with master and several slaves (default to cpu cores number) at one time, which means you can leverage all cpus of your machine.

$ locusts -f examples/first-testcase.yml --full-speed
[2017-08-26 23:51:47,071] bogon/INFO/locust.main: Starting web monitor at *:8089
[2017-08-26 23:51:47,075] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,078] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,080] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,083] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,084] bogon/INFO/locust.runners: Client 'bogon_656e0af8e968a8533d379dd252422ad3' reported as ready. Currently 1 clients ready to swarm.
[2017-08-26 23:51:47,085] bogon/INFO/locust.runners: Client 'bogon_09f73850252ee4ec739ed77d3c4c6dba' reported as ready. Currently 2 clients ready to swarm.
[2017-08-26 23:51:47,084] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,085] bogon/INFO/locust.runners: Client 'bogon_869f7ed671b1a9952b56610f01e2006f' reported as ready. Currently 3 clients ready to swarm.
[2017-08-26 23:51:47,085] bogon/INFO/locust.runners: Client 'bogon_80a804cda36b80fac17b57fd2d5e7cdb' reported as ready. Currently 4 clients ready to swarm.

Enjoy!

Supported Python Versions

Python 2.7, 3.3, 3.4, 3.5 and 3.6.

ApiTestEngine has been tested on macOS, Linux and Windows platforms.

Development

To develop or debug ApiTestEngine, you can install relevant requirements and use main-ate.py or main-locust.py as entrances.

$ pip install -r requirements_dev.txt
$ python main-ate -h
$ python main-locust -h

To learn more ...

• 《接口自动化测试的最佳工程实践（ApiTestEngine）》
• ApiTestEngine QuickStart
• 《ApiTestEngine 演进之路（0）开发未动，测试先行》
• 《ApiTestEngine 演进之路（1）搭建基础框架》
• 《ApiTestEngine 演进之路（2）探索优雅的测试用例描述方式》
• 《ApiTestEngine 演进之路（3）测试用例中实现 Python 函数的定义》
• 《ApiTestEngine 演进之路（4）测试用例中实现 Python 函数的调用》