Simplify testing infrastructure by removing DM_TEST_SRC and implementing template-based test generation

.devcontainer/README.md

@@ -55,17 +55,16 @@ This directory contains the devcontainer configuration for the dm package develo
 ### Running Tests with Different Backends
 
 ```bash
-# Test with PostgreSQL (default)
-DM_TEST_SRC=postgres R -e 'testthat::test_local()'
+# Run all tests (includes database-specific tests that will auto-configure)
+R -e 'testthat::test_local()'
 
-# Test with MariaDB
-DM_TEST_SRC=maria R -e 'testthat::test_local()'
+# Run specific database tests
+R -e 'testthat::test_local(filter = "postgres")'
+R -e 'testthat::test_local(filter = "maria")'
+R -e 'testthat::test_local(filter = "mssql")'
 
-# Test with SQL Server (requires ODBC drivers)
-DM_TEST_SRC=mssql R -e 'testthat::test_local()'
-
-# Test with data frames (no database)
-DM_TEST_SRC=df R -e 'testthat::test_local()'
+# Generate database test files from template
+make generate-db-tests
 ```
 
 ## Database Connection in R

.github/CONTRIBUTING.md

@@ -51,9 +51,9 @@ If you’ve found a bug, please file an issue that illustrates the bug with a mi
 will create a remote dm in a different database management system depending on context. 
 This allows tests that use those to be run on different setups through github actions. 
 Additionally `my_db_test_src()` will return the relevant database. 
-In order to test databases locally (typically to debug if CI tests fails and we can't debug from the online log)
-  we can set the environ variable "DM_TEST_SRC" to "postgres", "mariadb", "mssql", "duckdb" or "sqlite".
-  You might have to setup credentials in "helper-config-db.R" to do so.
+Database-specific test files (test-postgres.R, test-maria.R, etc.) are automatically generated 
+from a template and set up their own database connections. To regenerate these files after 
+editing the template, run `make generate-db-tests` or `./generate-db-tests.sh`.
 * Some useful expectations can be found in "helper-expectations.R".
 * In "helper-skip.R" are some helpers to skip tests in some contexts, these might be useful for instance if a feature is not supported on some databases.
 * When using `expect_snapshot()` on a DBMS dependent call (i.e a call that uses `dm_for_filter()` or copies to `my_db_test_src()`), 

.github/versions-matrix.R

@@ -1,18 +1,18 @@
 data.frame(
   os = "ubuntu-22.04",
   r = "release",
-  test_src = c(
-    "test-mssql",
-    "test-postgres",
-    "test-maria",
-    "test-mysql-maria",
-    "test-duckdb",
-    "test-sqlite"
+  setup_db = c(
+    "mssql",
+    "postgres", 
+    "maria",
+    "mysql-maria",
+    "duckdb",
+    "sqlite"
   ),
   covr = "true",
   desc = c(
     "SQL Server with covr",
-    "Postgres with covr",
+    "Postgres with covr", 
     "MariaDB with covr",
     "MySQL with covr",
     "DuckDB with covr",

.github/workflows/custom/after-install/action.yml

@@ -4,45 +4,45 @@ runs:
   using: "composite"
   steps:
     - uses: ankane/setup-postgres@v1
-      if: env.DM_TEST_SRC == 'test-postgres'
+      if: env.DM_SETUP_DB == 'postgres'
 
     - name: Create database
-      if: env.DM_TEST_SRC == 'test-postgres'
+      if: env.DM_SETUP_DB == 'postgres'
       run: |
         createdb ${USER}
       shell: bash
 
     # Must happen after installing system dependencies,
     # https://github.com/ankane/setup-mariadb/issues/2
     - uses: ankane/setup-mariadb@v1
-      if: env.DM_TEST_SRC == 'test-maria'
+      if: env.DM_SETUP_DB == 'maria'
       with:
         mariadb-version: 10.11
 
     - uses: ankane/setup-mysql@v1
-      if: env.DM_TEST_SRC == 'test-mysql-maria'
+      if: env.DM_SETUP_DB == 'mysql-maria'
       with:
         mysql-version: "8.0"
 
     - name: Create database (MariaDB), set it to UTF-8, add time zone info
-      if: env.DM_TEST_SRC == 'test-maria' || env.DM_TEST_SRC == 'test-mysql-maria'
+      if: env.DM_SETUP_DB == 'maria' || env.DM_SETUP_DB == 'mysql-maria'
       run: |
         mysql -e "CREATE DATABASE IF NOT EXISTS test; ALTER DATABASE test CHARACTER SET 'utf8'; FLUSH PRIVILEGES;"
       shell: bash
 
     - uses: ankane/setup-sqlserver@v1
-      if: env.DM_TEST_SRC == 'test-mssql'
+      if: env.DM_SETUP_DB == 'mssql'
       with:
         accept-eula: true
 
     - name: Create database (SQL Server)
-      if: env.DM_TEST_SRC == 'test-mssql'
+      if: env.DM_SETUP_DB == 'mssql'
       run: |
         sqlcmd -U SA -P 'YourStrong!Passw0rd' -Q 'CREATE DATABASE test;'
       shell: bash
 
     - name: Set ODBCSYSINI (SQL Server)
-      if: env.DM_TEST_SRC == 'test-mssql'
+      if: env.DM_SETUP_DB == 'mssql'
       run: |
         ln -s /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.*.so.* /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.so
         echo "ODBCSYSINI=${{ github.workspace }}/.github/odbc" | tee -a $GITHUB_ENV

.github/workflows/custom/before-install/action.yml

@@ -8,9 +8,9 @@ runs:
         echo '_R_CHECK_PKG_SIZES_THRESHOLD_=10' | tee -a $GITHUB_ENV
       shell: bash
 
-    - name: Define DM_TEST_SRC
+    - name: Define DM_SETUP_DB
       run: |
-        echo "DM_TEST_SRC=${{ matrix.config.test-src }}" | tee -a $GITHUB_ENV
+        echo "DM_SETUP_DB=${{ matrix.config.setup-db }}" | tee -a $GITHUB_ENV
       shell: bash
 
     - name: Clean up broken mysql apt

CLAUDE.md

@@ -53,13 +53,21 @@ testthat::test_local(reporter = "summary")
 
 ### Different Backends
 
-**Important**: Set the `DM_TEST_SRC` environment variable to test against various backends. Always test all backends. Supported values include:
+**Important**: Database-specific tests are now self-contained. Each database test file (test-postgres.R, test-maria.R, etc.) sets up its own connection and runs independently. Supported backends include:
 
-- `df`
+- `df` (data frames - default)
 - `postgres`
 - `maria` (MariaDB)
 - `mssql` (SQL Server)
 - `duckdb`
+- `sqlite`
+
+To regenerate database test files from the template:
+```bash
+make generate-db-tests
+# or directly:
+./generate-db-tests.sh
+```
 
 ---
 

Makefile

@@ -1,7 +1,11 @@
 all: qtest
 
+# Generate database-specific test files from template
+generate-db-tests:
+	./generate-db-tests.sh
+
 # Quiet tests
-# Run with make -j $(nproc) -O
+# Run with make -j $(nproc) -O 
 # or with pmake
 qtest: qtest-df qtest-sqlite qtest-postgres qtest-mssql qtest-duckdb qtest-maria
 
@@ -17,20 +21,34 @@ stest: stest-df stest-sqlite stest-postgres stest-mssql stest-duckdb stest-maria
 # Connectivity tests
 connect: connect-sqlite connect-postgres connect-mssql connect-duckdb connect-maria
 
+# Default tests run in data frame mode
+qtest-df:
+	time R -q -e 'options("crayon.enabled" = TRUE); Sys.setenv(TESTTHAT_PARALLEL = FALSE); testthat::test_local(filter = "${DM_TEST_FILTER}")'
+
+test-df:
+	time R -q -e 'Sys.setenv(TESTTHAT_PARALLEL = TRUE); testthat::test_local(filter = "${DM_TEST_FILTER}")'
+
+stest-df:
+	time R -q -e 'options(testthat.progress.max_fails = 1); testthat::test_local(filter = "${DM_TEST_FILTER}", reporter = "silent")'
+
+ltest-df:
+	time R -q -e 'Sys.setenv(TESTTHAT_PARALLEL = TRUE); lazytest::lazytest_local()'
+
+# Database-specific tests run all tests but only database tests that match will work
 qtest-%:
-	DM_TEST_SRC=$@ time R -q -e 'options("crayon.enabled" = TRUE); Sys.setenv(TESTTHAT_PARALLEL = FALSE); testthat::test_local(filter = "${DM_TEST_FILTER}")'
+	time R -q -e 'options("crayon.enabled" = TRUE); Sys.setenv(TESTTHAT_PARALLEL = FALSE); testthat::test_local(filter = "${DM_TEST_FILTER}")'
 
 test-%:
-	DM_TEST_SRC=$@ time R -q -e 'Sys.setenv(TESTTHAT_PARALLEL = TRUE); testthat::test_local(filter = "${DM_TEST_FILTER}")'
+	time R -q -e 'Sys.setenv(TESTTHAT_PARALLEL = TRUE); testthat::test_local(filter = "${DM_TEST_FILTER}")'
 
 stest-%:
-	DM_TEST_SRC=$@ time R -q -e 'options(testthat.progress.max_fails = 1); testthat::test_local(filter = "${DM_TEST_FILTER}", reporter = "silent")'
+	time R -q -e 'options(testthat.progress.max_fails = 1); testthat::test_local(filter = "${DM_TEST_FILTER}", reporter = "silent")'
 
 ltest-%:
-	DM_TEST_SRC=$@ time R -q -e 'Sys.setenv(TESTTHAT_PARALLEL = TRUE); lazytest::lazytest_local()'
+	time R -q -e 'Sys.setenv(TESTTHAT_PARALLEL = TRUE); lazytest::lazytest_local()'
 
 connect-%:
-	DM_TEST_SRC=$@ R -q -e 'suppressMessages(pkgload::load_all()); my_test_con()'
+	R -q -e 'suppressMessages(pkgload::load_all()); tryCatch(get("test_src_$*")(), error = function(e) cat("Error connecting to $*:", e$$message, "\n"))'
 
 db-start:
 	docker-compose up -d --force-recreate

generate-db-tests.R

@@ -0,0 +1,38 @@
+#!/usr/bin/env Rscript
+
+# Script to generate database-specific test files from template
+# Usage: Rscript generate-db-tests.R
+
+# Define the databases we support
+databases <- c("postgres", "maria", "mssql", "sqlite", "duckdb")
+
+# Read the template
+template_path <- file.path("tests", "testthat", "template-db-tests.R")
+if (!file.exists(template_path)) {
+  stop("Template file not found: ", template_path)
+}
+
+template_content <- readLines(template_path, warn = FALSE)
+
+# Generate test files for each database
+for (db in databases) {
+  # Replace placeholder with actual database name
+  content <- gsub("\\{\\{DATABASE\\}\\}", db, template_content)
+
+  # Add header comment indicating this is generated
+  header <- c(
+    paste0("# GENERATED FILE - DO NOT EDIT"),
+    paste0("# This file was generated from template-db-tests.R"),
+    paste0("# Edit the template and run generate-db-tests.R to update"),
+    "",
+    content
+  )
+
+  # Write to output file
+  output_path <- file.path("tests", "testthat", paste0("test-", db, ".R"))
+  writeLines(header, output_path)
+
+  cat("Generated:", output_path, "\n")
+}
+
+cat("Database test generation complete!\n")

generate-db-tests.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+
+# Script to generate database-specific test files from template
+# Usage: ./generate-db-tests.sh
+
+set -e
+
+# Define the databases we support
+databases=("postgres" "maria" "mssql" "sqlite" "duckdb")
+
+# Check if template exists
+template_path="tests/testthat/template-db-tests.R"
+if [ ! -f "$template_path" ]; then
+    echo "Template file not found: $template_path"
+    exit 1
+fi
+
+echo "Generating database test files..."
+
+# Generate test files for each database
+for db in "${databases[@]}"; do
+    output_path="tests/testthat/test-${db}.R"
+
+    # Create header
+    cat > "$output_path" << EOF
+# GENERATED FILE - DO NOT EDIT
+# This file was generated from template-db-tests.R
+# Edit the template and run generate-db-tests.sh to update
+
+EOF
+
+    # Replace placeholder and append content
+    sed "s/{{DATABASE}}/$db/g" "$template_path" >> "$output_path"
+
+    echo "Generated: $output_path"
+done
+
+echo "Database test generation complete!"

tests/testthat/README-db-tests.md

@@ -0,0 +1,52 @@
+# Database Test Generation
+
+This directory contains a template-based system for generating database-specific test files.
+
+## Files
+
+- `template-db-tests.R` - Template file for database-specific tests
+- `generate-db-tests.sh` - Script to generate test files from template  
+- `generate-db-tests.R` - R version of generation script (requires R)
+- `test-{database}.R` - Generated test files (DO NOT EDIT DIRECTLY)
+
+## How it works
+
+1. **Template**: `template-db-tests.R` contains the test logic with `{{DATABASE}}` placeholders
+2. **Generation**: Running the generation script replaces placeholders with actual database names
+3. **Self-contained**: Each generated test file sets up its own database connection and test environment
+4. **Synchronization**: All database test files stay in sync because they're generated from the same template
+
+## Supported Databases
+
+- postgres
+- maria (MariaDB)  
+- mssql (SQL Server)
+- sqlite
+- duckdb
+
+## Usage
+
+### Regenerate all database test files:
+```bash
+make generate-db-tests
+# or
+./generate-db-tests.sh
+```
+
+### Edit tests:
+1. Edit `template-db-tests.R` (not the individual test-*.R files)
+2. Run the generation script to update all database test files
+3. Commit both the template and generated files
+
+### Adding a new database:
+1. Add the database name to the `databases` array in `generate-db-tests.sh`
+2. Create a corresponding `test_src_<database>()` function in `helper-config-db.R`
+3. Run the generation script
+
+## Important Notes
+
+- **Never edit the generated `test-*.R` files directly** - they will be overwritten
+- **Always regenerate after editing the template** - files can get out of sync otherwise
+- **The template uses `{{DATABASE}}` placeholders** - these get replaced during generation
+- **Each test file is self-contained** - no global environment dependencies
+- **Other tests may still use skip_if_src_not()** - these will run in default 'df' mode and skip database-specific functionality. Consider moving database-specific tests to the template if they need variants across all databases.

tests/testthat/helper-src.R

@@ -56,7 +56,9 @@ copy_to_my_test_src <- function(rhs, lhs) {
 }
 
 my_test_src_name <- {
-  src <- Sys.getenv("DM_TEST_SRC")
+  # Default to data frame testing when no specific source is requested
+  # Database-specific test files override this value locally
+  src <- Sys.getenv("DM_TEST_SRC", "df")
   # Allow set but empty DM_TEST_SRC environment variable
   if (src == "") {
     src <- "df"