TIG-1192 Actor CI Tests can use a real mongo instance (mongodb#64)

* Actor CI Tests can use a real mongo instance * Cleanup, better cmake/Catch2 integration, better reporting on failed tests
roganamy · Jan 3, 2019 · 985fc83 · 985fc83
1 parent 1ed3880
commit 985fc83
Show file tree

Hide file tree

Showing 18 changed files with 638 additions and 48 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -179,3 +179,9 @@ private:
 
 -   Use scoped enums (`enum class`) wherever possible
 -   Use `k` prefixes for enum values e.g. `enum class Color { kRed, kBlue };`
+
+## Catch2
+Genny uses [Catch2](https://github.com/catchorg/Catch2) as its test framework.
+If you'd like to make changes to the vendored copy of Catch2 (e.g. to upgrade
+to a newer copy) Please make sure you read the README in the Catch2 directory
+before making changes.
diff --git a/README.md b/README.md
@@ -61,7 +61,7 @@ and restarting your shell.
 TODO: TIG-1263 This is kind of a hack; using built-in package-location
 mechanisms would avoid having to have OS-specific hacks like this.
 
-### Linux Distributions
+#### Linux Distributions
 
 Have installations of non-vendored dependent packages in your system,
 using the package manger. Generally this is:
@@ -85,7 +85,7 @@ You only need to run cmake once. Other useful targets include:
 - test_gennylib test_driver (builds tests)
 - test (run's tests if they're built)
 
-### Other Operating Systems
+#### Other Operating Systems
 
 If not using OS X, ensure you have a recent C++ compiler and boost
 installation. You will also need packages installed corresponding to the
@@ -105,7 +105,7 @@ apt-get install -y \
 #   https://mongodb.github.io/mongo-cxx-driver/mongocxx-v3/installation/
 ```
 
-### IDEs and Whatnot
+#### IDEs and Whatnot
 
 We follow CMake and C++17 best-practices so anything that doesn't work
 via "normal means" is probably a bug.
@@ -115,16 +115,15 @@ emacs, vim, etc.). Before doing anything cute (see
 [CONTRIBUTING.md](./CONTRIBUTING.md)), please do due-diligence to ensure
 it's not going to make common editing environments go wonky.
 
-Running Genny Self-Tests
-------------------------
+### Running Genny Self-Tests
 
 Genny has self-tests using Catch2. You can run them with the following command:
 
 ```sh
 make -C "build" test_gennylib test_driver test
 ```
 
-### Perf Tests
+#### Perf Tests
 
 The above `make test` line also runs so-called "perf" tests. They can
 take a while to run and may fail occasionally on local developer
@@ -143,8 +142,28 @@ Read more about specifying what tests to run [here][s].
 
 [s]: https://github.com/catchorg/Catch2/blob/master/docs/command-line.md#specifying-which-tests-to-run
 
-Running Genny Workloads
------------------------
+#### Actor Integration Tests
+
+The actor tests use resmoke to set up a real MongoDB cluster and execute
+the test binary. The resmoke yaml config files that define the different
+cluster configurations are defined in `src/resmokeconfig`.
+
+resmoke.py can be run locally as follows:
+```sh
+# Set up virtualenv and install resmoke requirements if needed.
+# From genny's top-level directory.
+python /path/to/resmoke.py --suite src/resmokeconfig/genny_standalone.yml
+```
+
+Each yaml configuration file will only run tests that are associated
+with their specific tags. (Eg. `genny_standalone.yml` will only run
+tests that have been tagged with the "[standalone]" tag.)
+
+When creating a new actor, `create-new-actor.sh` will generate a new test case
+template to ensure the new actor can run against different MongoDB topologies,
+please update the template as needed so it uses the newly created actor.
+
+### Running Genny Workloads
 
 First install mongodb and start a mongod process:
 
@@ -173,16 +192,14 @@ in the above example).
 Post-processing of metrics data is done by Python scripts in the
 `src/python` directory. See [the README there](./src/python/README.md).
 
-Code Style and Limitations
----------------------------
+### Code Style and Limitations
 
 > Don't get cute.
 
 Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for code-style etc.
 Note that we're pretty vanilla.
 
-Sanitizers
-----------
+### Sanitizers
 
 Genny is periodically manually tested to be free of unknown sanitizer
 errors. These are not currently run in a CI job. If you are adding

diff --git a/create-new-actor.sh b/create-new-actor.sh
@@ -103,6 +103,49 @@ create_impl() {
     create_impl_text "$@" > "$(dirname "$0")/src/cast_core/src/actors/${actor_name}.cpp"
 }
 
+create_test() {
+    local actor_name
+    actor_name="$1"
+
+    cat << EOF >> "$(dirname "$0")/src/gennylib/test/${actor_name}_test.cpp"
+#include "test.h"
+
+#include <bsoncxx/json.hpp>
+#include <bsoncxx/builder/stream/document.hpp>
+
+#include <MongoTestFixture.hpp>
+
+namespace {
+using namespace genny::testing;
+namespace bson_stream = bsoncxx::builder::stream;
+
+TEST_CASE_METHOD(MongoTestFixture, "${actor_name} successfully connects to a MongoDB instance.",
+          "[standalone][single_node_replset][three_node_replset][sharded]") {
+
+    dropAllDatabases();
+    auto db = client.database("test");
+
+    SECTION("Insert a document into the database.") {
+        auto builder = bson_stream::document{};
+        bsoncxx::document::value doc_value = builder
+                << "name" << "MongoDB"
+                << "type" << "database"
+                << "count" << 1
+                << "info" << bson_stream::open_document
+                << "x" << 203
+                << "y" << 102
+                << bson_stream::close_document
+                << bson_stream::finalize;
+        bsoncxx::document::view view = doc_value.view();
+        db.collection("test").insert_one(view);
+        // Fail on purpose to encourage contributors to extend automated testing for each new actor.
+        REQUIRE(db.collection("test").count(view) == 0);
+    }
+}
+} // namespace
+EOF
+}
+
 recreate_cast_core_cmake_file() {
     local uuid_tag
     local actor_name
@@ -118,6 +161,21 @@ recreate_cast_core_cmake_file() {
     mv "$$.cmake.txt" "$cmake_file"
 }
 
+recreate_gennylib_cmake_file() {
+    local uuid_tag
+    local actor_name
+    local cmake_file
+    uuid="$1"
+    actor_name="$2"
+    cmake_file="$(dirname "$0")/src/gennylib/CMakeLists.txt"
+
+    < "$cmake_file" \
+    perl -pe "s|((\\s+)# ActorsTestEnd)|\$2test/${actor_name}_test.cpp\\n\$1|" \
+    > "$$.cmake.txt"
+
+    mv "$$.cmake.txt" "$cmake_file"
+}
+
 if [[ "$#" != 1 ]]; then
     usage
     exit 1
@@ -138,3 +196,5 @@ uuid_tag="$("$(dirname "$0")/generate-uuid-tag.sh")"
 create_header                "$uuid_tag" "$actor_name"
 create_impl                  "$uuid_tag" "$actor_name"
 recreate_cast_core_cmake_file "$uuid_tag" "$actor_name"
+create_test                  "$actor_name"
+recreate_gennylib_cmake_file "$uuid_tag" "$actor_name"
diff --git a/evergreen.yml b/evergreen.yml
@@ -79,13 +79,40 @@ tasks:
   - func: f_fetch_source
   - func: f_build_grpc
   - func: f_compile
+  - func: f_checkout_mongodb
+
 - name: t_python_test
   commands:
   - func: f_python_test
+
 - name: t_cmake_test
   commands:
   - func: f_cmake_test
 
+- name: t_integration_test_standalone
+  commands:
+  - func: f_resmoke_test
+    vars:
+      resmoke_suite: genny_standalone.yml
+
+- name: t_integration_test_single_node_replset
+  commands:
+  - func: f_resmoke_test
+    vars:
+      resmoke_suite: genny_single_node_replset.yml
+
+- name: t_integration_test_three_node_replset
+  commands:
+  - func: f_resmoke_test
+    vars:
+      resmoke_suite: genny_three_node_replset.yml
+
+- name: t_integration_test_sharded
+  commands:
+  - func: f_resmoke_test
+    vars:
+      resmoke_suite: genny_sharded.yml
+
 
 ##                ⚡️ Task Groups ⚡️
 
@@ -108,6 +135,10 @@ task_groups:
   - t_compile
   - t_python_test
   - t_cmake_test
+  - t_integration_test_standalone
+  - t_integration_test_single_node_replset
+  - t_integration_test_three_node_replset
+  - t_integration_test_sharded
 
 
 ##                ⚡️ Functions ⚡️
@@ -119,6 +150,26 @@ functions:
     params:
       directory: src
 
+  ##
+  # Download the mongodb binary and then clone and checkout the appropriate mongodb repository branch
+  # that contains the intended gennylib test case (SERVER-38646). Also installs resmoke dependencies.
+  ##
+  f_checkout_mongodb:
+  - command: shell.exec
+    params:
+      working_dir: src/build
+      script: |
+        yes | pacman -S mongodb
+        git clone [email protected]:mongodb/mongo.git mongo
+
+        pushd mongo
+          git checkout 6734c12d17dd4c0e2738a47feb7114221d6ba66d
+        popd
+
+        virtualenv -p python2 venv
+        source venv/bin/activate
+        python -m pip install -r mongo/etc/pip/evgtest-requirements.txt
+
   ##
   # Compile gRPC. This script should be synchronized with DSI's compilation sript for Genny:
   # https://github.com/10gen/dsi/blob/master/configurations/workload_setup/workload_setup.common.yml
@@ -147,7 +198,7 @@ functions:
         mkdir cmake-build
         pushd cmake-build
         cmake ..
-        make
+        make -j$(nproc)
 
   ##
   # Compile the project in src/build.
@@ -170,10 +221,10 @@ functions:
         export CPLUS_INCLUDE_PATH="$cwd/grpc/include:$cwd/grpc/third_party/protobuf/src"
 
         cmake -DCMAKE_CXX_FLAGS="${cmake_cxx_flags}" ..
-        make
+        make -j$(nproc)
 
   ##
-  # Runs tests via `make test`.
+  # Runs tests via ctest.
   # Requires f_compile to have been run first.
   ##
   f_cmake_test:
@@ -186,7 +237,53 @@ functions:
         set -o pipefail
         set -o nounset
 
-        make test
+        ctest --label-exclude "(standalone|sharded|single_node_replset|three_node_replset)"
+
+  ##
+  # Runs tests via resmoke.
+  # Requires f_compile to have been run first.
+  ##
+  f_resmoke_test:
+  - command: shell.exec
+    params:
+      continue_on_err: true
+      working_dir: src
+      script: |
+        set -o errexit
+        set -o pipefail
+        # set -o nounset # the "activate" script has an unbound variable
+
+        source build/venv/bin/activate
+
+        # We rely on catch2 to report test failures, but it doesn't always do so.
+        # See https://github.com/catchorg/Catch2/issues/1210
+        # As a workaround, we generate a dummy report with a failed test that is deleted if resmoke
+        # succeeds.
+
+        cat << EOF >> "build/sentinel.junit.xml"
+        <?xml version="1.0" encoding="UTF-8"?>
+        <testsuites>
+         <testsuite name="resmoke_failure_sentinel" errors="0" failures="1" tests="1" hostname="tbd" time="1.0" timestamp="2019-01-01T00:00:00Z">
+          <testcase classname="resmoke_failure_sentinel" name="Dummy testcase to signal that resmoke failed because a report may not be generated" time="1.0">
+           <failure message="resmoke did not exit cleanly, see task log for detail" type="">
+           </failure>
+          </testcase>
+          <system-out/>
+          <system-err/>
+         </testsuite>
+        </testsuites>
+        EOF
+
+        # The tests themselves do the reporting instead of using resmoke.
+        python build/mongo/buildscripts/resmoke.py \
+               --suite src/resmokeconfig/${resmoke_suite} \
+               --mongod mongod \
+               --mongo mongo \
+               --mongos mongos
+
+        # Remove the sentinel report if resmoke succeeds. This line won't be executed if
+        # resmoke fails because we've set errexit on this shell.
+        rm build/sentinel.junit.xml
 
   ##
   # Runs python nosetests.
@@ -213,6 +310,10 @@ functions:
     params:
       optional: true
       file: src/build/src/*/*.junit.xml
+  - command: attach.xunit_results
+    params:
+      optional: true
+      file: src/build/*.junit.xml
   - command: attach.xunit_results
     params:
       optional: true
@@ -229,4 +330,7 @@ functions:
         set -o pipefail
         set -o nounset
 
-        rm -f src/build/src/*/*.junit.xml src/src/python/nosetests.xml
+        rm -f \
+        src/build/*.junit.xml \
+        src/build/src/*/*.junit.xml \
+        src/src/python/nosetests.xml
diff --git a/generate-uuid-tag.sh b/generate-uuid-tag.sh
@@ -1,4 +1,4 @@
 #/bin/bash
 
 UUID=$(uuidgen | sed s/-/_/g | tr a-z A-Z)
-echo "HEADER_${UUID}"
+echo "HEADER_${UUID}_INCLUDED"
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
@@ -1,3 +1,6 @@
+include(${CMAKE_CURRENT_SOURCE_DIR}/third_party/catch2/ParseAndAddCatchTests.cmake)
+set(AdditionalCatchParameters --reporter junit --durations yes --rng-seed 12345 --success)
+
 add_subdirectory(third_party)
 
 add_subdirectory(gennylib)

diff --git a/src/driver/CMakeLists.txt b/src/driver/CMakeLists.txt
@@ -31,13 +31,4 @@ target_include_directories(test_driver
 )
 target_link_libraries(test_driver driver catch2)
 
-add_test(
-    NAME    test_driver
-    COMMAND test_driver
-        -r junit
-        -d yes
-        --rng-seed 12345
-        --success
-        --out test_driver.junit.xml
-)
-
+ParseAndAddCatchTests(test_driver)