Tests Creation and Exection

In the past arcticdb project has not used custom pytest marks to help processes of filtering tests for execution, traceability etc. With more than 1400 pytests currently and havins support for many kind of storages there is need of ability to select and edit or execute only tests that are needed. To help do step-by=step approach to their introduction will be started with several new marks that would help reduce ammounts of tests being executed, and help their maintenance also.

First important mark is the 'storage' mark. Its is introduced to help select only tests that have fixtures real storages (aws s2 and gcp). This mark is currently placed on all tests that have such fixtures, but in future it must also be placed on new tests that have such fixtures. This mark help us select for execution only tests having it or the opposite all tests that do not have it (pytest -m storage, pytest -m 'not storage')

As the tests belonging to that group/category can be executed agains variety of storage types, not only real storages but also simulated and local storages, several new environment variables have been introduced to limit the options of execution when needed:

    ARCTICDB_LOCAL_STORAGE_TESTS_ENABLED=(0|1) default is 1 if missing
    ARCTICDB_STORAGE_AWS_S3=(0|1) default is 1 if missing
    ARCTICDB_STORAGE_GCP=(0|1) not enabled by default

NOTE: currently all local and simulated s3, gcp etc storages, as well as mongo are in the LOCAL_STORAGE_TESTS_ENABLED category. This is done on purpose for backwards compatibility. All other default values are also considered for backwards compatibility

With those enhancements we have fine control over unatended execution (GitHub) and so the workflow for test execution is now not a checkbox but drop down of 3 possibilities:

• local only
• real aws s3
• real gcpxml

There are 2 other new marks which would help over time reducing the need of havins so many different fixtures. As we have 10 types now of storages there are so many possibilities to combine them along with parameters. What are the marks:

• skip_fixture_params - accepts a list of fixture params that will be skipped - helps defining smaller set of storage types that fixture will generated
• only_fixture_params - accepts a list of fixture params, and only those will be generated for this test

There are 2 more marks:

• authentication - all storages require authentication, and authentication cannot be simulated thus having ability to select only those hight prio tests is important
• bug_ids - allows specifying bug ids list the tests is based on or depends. It has no effect on test, but only creates a link for traceability with external issue database. That helps preserving history and knowledge and over time will be very helphul

NOTE: all marks that we declare should be also listed in toml file as otherwise pytest will list warning. Therefore if new marks are needed, please add them there