Tarantool Functional testing framework

Test Suite

Bunch of tests, that lay down in the subfolder (recursively) with suite.ini file. suite.ini is basic ini-file, that consists of one section default, and a number of fields:

• core
• description - Test Suite description
• script - shebang file to start tarantool with
• disables:
  • disabled - tests that must be skipped
  • release_disabled - tests that must be skipped when Tarantool has been builded with Release
  • valgrind_disabled - tests that must be skipped when Valgrind is enabled
• lua_libs - paths for lua files, that should be copied into the folder, where server is started (delimited with the space, e.g. lua_libs=lua/1.lua lua/2.lua)
• long_run - mark tests as long, enabled only with --long option (delimited with the space, e.g. long_run=t1.test.lua t2.test.lua)
• config - test configuration file name

Field core must be one of:

• luatest - luatest compatible test suite
• tarantool - Test-Suite for Functional Testing
• app - Another functional Test-Suite
• unittest - Unit-Testing Test Suite

Test

Each test consists of files *.test(.lua|.sql|.py)?, *.result, and may have skip condition file *.skipcond. On first run (without .result) .result is generated from output. Each run, in the beggining, .skipcond file is executed. In the local env there's object self, that's Test object. If test must be skipped - you must put self.skip = 1 in this file. Next, .test(.lua|.py)? is executed and file .reject is created, then .reject is compared with .result. If something differs, then 15 last string of this diff file are printed and .reject file is saving in the <vardir>/rejects/<suite> subfolder given in options or set localy as var/rejects/<suite> by default. If not, then .reject file is deleted.

Test configuration

Test configuration file contains config for multiple run. For each test section system runs separated test and compares result with common .result file. For example we need to run one test for different db engines("*" means default configuration):

{
    "my.test.lua": {
        "first": {"a": 1, "b": 2},
        "second": {"a": 1, "b": 3}
    },
    "*": {
        "memtx": {"engine": "memtx"},
        "vinyl": {"engine": "vinyl"}
    }
}

In test case we can get configuration from inspector:

engine = test_run:get_cfg('engine')
-- first run engine is 'memtx'
-- second run engine is 'vinyl'

"engine" value has a special meaning for *.test.sql files: if it is "memtx" or "vinyl", then the corresponding default engine will be set before executing commands from a test file. An engine is set with the following commands:

UPDATE "_session_settings" SET "value" = 'memtx|vinyl' WHERE "name" = 'sql_default_engine'
pragma sql_default_engine='memtx|vinyl'

If the first fails, then the second will be executed. When both fails, fail the test.

Python

Files: <name>.test.py, <name>.result and <name>.skipcond(optionaly).

Environment:

• sql - BoxConnection class. Convert our subclass of SQL into IProto query and then decode it. Print into .result in YAML. Examples:
  • sql("select * from t<space> where k<key>=<string|number>[ limit <number>]")
  • sql("insert into t<space> values ([<string|number> [, <string|number>]*])")
  • sql("delete from t<space> where k<key>=<string|number>")
  • sql("call <proc_name>([string|number]*)")
  • sql("update t<space> set [k<field>=<string|number> [, k<field>=<string|number>]*] where k<key>=<string|number>"")
  • sql("ping")
• admin - AdminConnection - simply send admin query on admin port (LUA), then, receive answer. Examples
  • admin('box.info')

Example:

import os
import time

from lib.admin_connection import AdminConnection
from lib.tarantool_server import TarantoolServer

master = server
admin("box.info.lsn") # equivalent to master.admin("box.info.lsn") and server.admin(...)
sql("select * from t0 where k0=1")
replica = TarantoolServer()
replica.script = 'replication/replica.lua'
replica.vardir = os.path.join(server.vardir, "replica")
replica.deploy()
master.admin("box.insert(0, 1, 'hello')")
print('sleep_1')
time.sleep(0.1)
print('sleep_finished')
print('sleep_2')
admin("require('fiber').sleep(0.1)")
print('sleep_finished')
replica.admin("box.select(0, 0, 1)")
con2 = AdminConnection('localhost', server.admin.port)
con2("box.info.lsn")
replica.stop()
replica.cleanup()
con2.disconnect()

Result:

box.info.lsn
---
- null
...
select * from t0 where k0=1
---
- error:
    errcode: ER_NO_SUCH_SPACE
    errmsg: Space '#0' does not exist
...
box.insert(0, 1, 'hello')
---
- error: '[string "return box.insert(0, 1, ''hello'')"]:1: attempt to call field ''insert''
    (a nil value)'
...
sleep_1
sleep_finished
sleep_2
require('fiber').sleep(0.1)
---
...
sleep_finished
box.select(0, 0, 1)
---
- error: '[string "return box.select(0, 0, 1)"]:1: attempt to call field ''select''
    (a nil value)'
...
box.info.lsn
---
- null
...

Lua

Files: <name>.test.lua, <name>.result and <name>.skipcond(optionaly). Tests interact only with AdminConnection. Supports some preprocessor functions (eg delimiter)

Delimiter example:

env = require('test_run')
test_run = env.new()
box.schema.space.create('temp')
t1 = box.space.temp
t1:create_index('primary', { type = 'hash', parts = {1, 'num'}, unique = true})
t1:insert{0, 1, 'hello'}
test_run:cmd("setopt delimiter ';'")
function test()
    return {1,2,3}
end;
test(
);
test_run:cmd("setopt delimiter ''");
test(
);
test

Delimiter result:

env = require('test_run')
test_run = env.new()
box.schema.space.create('temp')
---
- index: []
  on_replace: 'function: 0x40e4fdf0'
  temporary: false
  id: 512
  engine: memtx
  enabled: false
  name: temp
  field_count: 0
- created
...
t1 = box.space.temp
---
...
t1:create_index('primary', { type = 'hash', parts = {1, 'num'}, unique = true})
---
...
t1:insert{0, 1, 'hello'}
---
- [0, 1, 'hello']
...
test_run:cmd("setopt delimiter ';'")
function test()
    return {1,2,3}
end;
---
...
test(
);
---
- - 1
  - 2
  - 3
...
test_run:cmd("setopt delimiter ''");
test(
---
- error: '[string "test( "]:1: unexpected symbol near ''<eof>'''
...
);
---
- error: '[string "); "]:1: unexpected symbol near '')'''
...
test
---
- 'function: 0x40e533b8'
...

It is possible to use backslash at and of a line to carry it.

function echo(...) \
    return ...     \
end

SQL

*.test.sql files are just SQL statements written line-by-line.

It is possible to mix SQL and Lua commands using \set language lua and \set language sql commands.

Interaction with the test environment

In lua test you can use test_run module to interact with the test environment.

env = require('test_run')
test_run = env.new()
test_run:cmd("<command>")

Base directives:

• setopt delimiter '<delimiter>' - Sets delimiter to <delimiter>\n

Server directives:

• create server <name> with ... - Create server with name <name>, where ... may be:
  • script = '<path>' - script to start
  • rpl_master = <server> - replication master server name
• start server <name> - Run server <name>
• stop server <name> [with signal=<signal>] - Stop server <name>
  • <signal> is a signal name (with or without 'SIG' prefix, uppercased) or a signal number to use instead of default SIGTERM
• cleanup server <name> - Cleanup (basically after server has been stopped)
• restart server <name> - Restart server <name> (you can restart yourself from lua!)

Connection switch:

• switch <name> - Switch connection to server <name> and add test run into global scope

Connection directives(low level):

• create connection <name-con> to <name-serv> - create connection named <name-con> to <name-serv> server
• drop connection <name> - Turn connection <name> off and delete it
• set connection <name> - Set connection <name> to be main, for next commands

Filter directives:

• push filter '<regexp_from>' to '<regexp_to>' - e.g. push filter 'listen: .*' to 'listen: <uri>'

Set variables:

• set variables '<variable_name>' to '<where>' - execute <variable_name> = *<where> where * is value of where. Where must be
  • <server_name>.admin - admin port of this server
  • <server_name>.master - listen port of master of this replica
  • <server_name>.listen - listen port of this server

Dev ops features:

You can power on any tarantool replicas in a loop.

test_run:cmd('setopt delimiter ";"')
function join(inspector, n)
    for i=1,n do
        local rid = tostring(i)
        os.execute('mkdir -p tmp')
        os.execute('cp ../replication/replica.lua ./tmp/replica'..rid..'.lua')
        os.execute('chmod +x ./tmp/replica'..rid..'.lua')
        inspector:cmd("create server replica"..rid.." with rpl_master=default, script='./var/tmp/replica"..rid..".lua'")
        inspector:cmd("start server replica"..rid)
    end
end;
test_run:cmd('setopt delimiter ""');

-- create 30 replicas for current tarantool
join(test_run, 30)

pretest_clean()

Nothing will be done before a Python test and for core = unittest test suites.

For a core = [app|tarantool] test suites this function removes tarantool WAL and snapshot files before each test.

The following files will be removed:

• *.snap
• *.xlog
• *.vylog
• *.inprogress
• [0-9]*/

Tags

Usage:

./test-run.py --tags foo
./test-run.py --tags foo,bar app/ app-tap/

test-run will run only those tests, which have at least one of the provided tags.

Show a list of tags:

./test-run.py --tags
./test-run.py app-tap/ --tags

The tags metainfo should be placed within a first comment of a test file.

Examples:

• .lua file:

  #!/usr/bin/tarantool
  
  -- tags: foo, bar
  -- tags: one, more
  
  <...>

• .sql file:

  -- tags: foo
  -- tags: bar
  <...>

• .py file:

  # tags: foo
  
  <...>

Unsupported features:

• Marking unit tests with tags.
• Multiline comments (use singleline ones for now).

Using luatest

test-run supports tests written in the luatest format. *_test.lua files in a core = luatest test suite are run as part of ./test/test-run.py invocation: no extra actions are needed.

You can also run a particular test using a substring of its full name:

$ ./test/test-run.py foo-luatest/bar_test.lua
$ ./test/test-run.py bar_test.lua
$ ./test/test-run.py bar

If you need to run a particular test case from a luatest compatible test, use luatest command directly. In order to use luatest, which is bundled into test-run, source test-run's environment:

$ . <(./test/test-run.py --env)
$ luatest -v -p my_specific_test_case

Used By

• Tarantool - in-memory database and application server
• memcached - Memcached protocol 'wrapper' for Tarantool
• vshard - sharding based on virtual buckets
• xsync (internal project)