TestEnough is an attempt at solving the problem of reducing the feedback time around making sure if the changes made to a codebase are fine. It is a library that helps you run a pruned set of tests (a small subset) picked from the entire test suite based on the changes made to production code. Thus, instead of running all the tests, it lets you run only a small, relevant set of tests.

Why?

Typically teams following continuous integration make sure that developers run the build locally and then check in if the build is green. The idea is to keep the main line green as much as possible. However, even with a very fast build of a few minutes, developers need to wait for those few minutes if they are running the build on their box. Even if the build is moved to a personal build server of sorts, so that pre-commit build is not happening on the local developer box, a few minutes still need to be spent before checking in.

On a large team, there could potentially be multiple commits during this few minutes window and you would then have to merge with upstream and re-run the build. It would be quite tedious if one gets into a merge-build loop. This means, ideally, you would want as small a pre-commit build as possible.

It also seems like a huge waste to run all tests when the change made is quite small. Especially, if a team follows the “Small checkin, frequent checkin” philosophy, it would definitely be wasteful to run all the tests all the time.

To address these, is why TestEnough came into existence.

Getting started

• Get hold of the TestEnough package. You can either
  • Download a stable build to get a tar ball or
  • You can source compile. There is an ant build file in the source root. You can run ant pkg in the checkout folder. This runs all the tests and generates the tar ball in target/pkg folder.
• Extract the tar ball
• In order to hook up TestEnough to your tests, you need to add these JVM argument to your test running JVM.

-javaagent:test-enough/test-enough-0.1.jar=configFilePath:test-enough/config.properties=lib:test-enough/lib -Xbootclasspath/a:test-enough/lib/commons-io-2.0.1.jar

In the above arguments, if you notice, you need to provide the path to 4 things:

1. The TestEnough jar
2. The TestEnough configuration
3. The libraries on which TestEnough depends on
4. The commons-io jar.

All these paths are to resources available inside the obtained tar ball. Its preferred to have the extracted directory checked in and the above paths referred to using relative paths

Using TestEnough with Ant

If you are using ANT you can use TestEnough in 2 ways based on how you run your tests

• Fork Mode: If you run your JUnit tests using the fork=true option, you need to pass the above arguments as 2 under tag. 1 for the “javaagent” and the other for the “Xbootclasspath” options. For example:

<jvmarg>-javaagent:test-enough/test-enough-0.1.jar=configFilePath:test-enough/config.properties=lib:test-enough/lib</jvmarg>

<jvmarg>-Xbootclasspath/a:test-enough/lib/commons-io-2.0.1.jar</jvmarg>

• Non-Fork Mode: If you are not forking your tests, then you need to set the environment variable “ANT_OPTS” with both the arguments as the value. For example:

export ANT_OPTS="-javaagent:test-enough/test-enough-0.1.jar=configFilePath:test-enough/config.properties=lib:test-enough/lib -Xbootclasspath/a:test-enough/lib/commons-io-2.0.1.jar"

Here, you should replace the paths to the real path of the resources.

Configuration

TestEnough requires as little configuration as possible in order to get up and running. However, you can customize a bunch of things using the config.properties file.

You need to pass the config file path as an argument to the javaagent. The name of the argument is “configFilePath”. The value is the path. The TestEnough package contains a config.properties file that has all the properties and the corresponding values they can take.

Checkout the config.properties file and set the properties properly.

Properties and meaning

• sampleProductionClass MANDATORY: This property gives the name of any one class in your application under test. This is used to figure out the class loader from which this class gets loaded. This is how we get an access to all the classes that get loaded so that we can BCI them.

• populateIncludePackages: This property specifies a comma separated list of packages that you want to track test information for. If you leave this out all classes (including libraries that your application depends upon) get tracked. It will be a good idea to specify this since instrumenting everything may affect test run times.

• testClassPattern: This property represents the pattern which the names of the test classes follow. Using this, Test Enough figures out what the test class is. Using this information, test enough knows which test class is calling the tracker.

• trackingInformationFilePath: This property represents the file path where the tracking information needs to be persisted. It is this file that will be used in order to figure out which tests need to be run based on production code changes.

• codeToInsert: This is the code that is used for instrumentation. Override this if you know what you want. This will change the default

• behaviour of the tool. This is more for developers to plug things in.

How?

TestEnough is a Java agent library that uses Byte Code Instrumentation to figure out what methods were called. One can configure what is a test and what is production code which would then let TestEnough figure out what all code gets called from a given test. It maintains a map of “Method => [Tests]”.

This way, when running a build, based on what methods of production code have changed, TestEnough constructs a smaller list of tests. You would have a new build task that would use TestEnough and run this as the pre-commit build

Why TestEnough?

My wife thought it would be a good idea to name this after her. Her point was, I spend so much of my personal time hacking and since she is very supportive, why shouldn’t I? We compromised by saying I will mention her intention instead here! (Here you go honey)