To main content
Featured image of Guide: Parallel unit tests with Apache Maven

Guide: Parallel unit tests with Apache Maven

Published by Benjamin Marwell on

Parallel Testing with JUnit Jupiter

This is currently the most popular framework and has (in my opinion) the best configuration options. This was the first part of our joint talk. Use this option if you already use JUnit Jupiter in your project.

How to enable parallel tests in JUnit Jupiter

You have three options here:

Using a properties file

Put a file src/test/resources/junit-platform.properties into each module.

Using surefire’s argLine

Add the configuration parameters as system properties to the surefire argLine variable.

Using surefire’s properties/configurationParameters

Add the same configuration parameters to a special (almost hidden) configuration in the surefire configuration block.

Using annotations

Add @Execution annotations to each class or suite.

If unsure, start with surefire’s properties/configurationParameters like so:

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.2.5</version>
      <configuration>
        <properties>
          <configurationParameters>
              junit.jupiter.execution.parallel.enabled = true
              junit.jupiter.execution.parallel.mode.default = concurrent
              junit.jupiter.execution.parallel.mode.classes.default = concurrent
          </configurationParameters>
        </properties>
      </configuration>
    </plugin>
  </plugins>
</build>

This will run both top-level classes and their methods in parallel. As a bonus, this configuration will be inherited in submodules. Therefore, your whole project will run tests in parallel.

If you want to use argLine, the setup gets a bit more complicated.

Configuring JUnit-Jupiter using surefire’s argLine
<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.2.5</version>
      <configuration>
        <argLine>
          @{argLine}
          -Djunit.jupiter.execution.parallel.enabled=true
          -Djunit.jupiter.execution.parallel.mode.default=concurrent
          -Djunit.jupiter.execution.parallel.mode.classes.default=concurrent
        </argLine>
      </configuration>
    </plugin>
  </plugins>
</build>

Disabling parallel tests selectively

If you want to disable parallel tests selectively, you have a few options here.

Overwrite configurationParameters

This is global for all tests in the current modules. Use this if a whole module’s tests cannot be run in parallel and only few modules need this exception.

Annotation @ExecutionMode(SAME_THREAD)

Will run the tests of the current class in the same thread, including the methods @BeforeAll and @AfterAll. Use this as a first try when a single class will not run using parallel tests.

Annotation @ResourceLock

Use this annotation on multiple classes when a resource would be shared or modified among those classes, hindering parallel execution. A typical example for a shared resource would be a global variable store, static fields accessed by multiple tests at the same time, or caches.

Annotation @Isolated

This annotation will run the test methods sequentially and in isolation of ALL other tests, no matter what. Use this as a last resort.

Parallel testing with Surefire

The method from above (junit.jupiter.execution.parallel) only works for JUnit Jupiter, as JUnit Jupiter brings its own test engine. This means that you cannot use those parameters for JUnit 4, TestNG nor Spock.

This is how you enable parallel tests for JUnit 4, TestNG or Spock using Apache Maven’s surefire-plugin.

Surefire configuration options

Surefire has only a single parameter parallel, but supports all combinations of parallelism for suites, classes and methods. These are the valid values for the parameter parallel:

none

Disables parallel testing completely. This is the default.

suites

Only runs suites in parallel, but not other top-level classes or methods within any class.

suitesAndClasses

Runs suites and top-level classes in parallel.

suitesAndMethods

Runs suites and methods in parallel, but not any methods encountered anywhere.

classesAndMethods

Runs top-level classes in parallel, but not suites.

all

Runs all of it in parallel: Methods in top-level classes, top-level-classes and suites.

How to enable parallel tests in JUnit 4, TestNG or Spock

1. Enable parallel testing via XML configuration in maven-surefire-plugin

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.2.5</version>
      <configuration>
        <parallel>classesAndMethods</parallel>
        <threadCount>4</threadCountSuites>
        <perCoreThreadCount>true</perCoreThreadCount>
      </configuration>
    </plugin>
  </plugins>
</build>

2. Enable parallel testing via command line for maven-surefire

mvn -Dparallel=all -DthreadCount=4 -DperCoreThreadCount=true verify

Advantages and disadvantages of configuration options

Now, obviously you cannot disable parallel testing per module using the command line - it is global. However, you could define a custom property per module to control parallel test execution per module. Since this is a tedious work and does not follow conventions, I personally would abstain from this method.

However, you can just set the configuration in a module which does not (fully) support parallel testing. Or, put the classes which do not support parallel testing into a test suite. You can also do it the other way around: Putting the tests which support parallel testing into a suite.

About forkCount

At this point, I do not advise to use fork count. Forking is relatively expensive.

Demo project: Accelerating Maven Builds

The GitHub repository mthmulders/snail-pace-rocket-speed-demos contains a script called ./demo-2-parallel-tests.sh. Execute it to see the effects of enabling parallel tests!

Here is my outcome.

Apache Shiro without parallel tests

Running mvn verify -pl :shiro-core -am on Apache Shiro without parallel tests takes 1:24 minutes on my machine.

Maven Reactor Summary with sequential Tests on Apache Shiro
Figure 1. Running Apache Shiro up to core takes 1:24 by default

Apache Shiro with parallel tests enabled

Running the same command after putting the files junit-platform.properties into place wil only take 49 seconds!

Maven Reactor Summary with Parallel Tests on Apache Shiro
Figure 2. Running Apache Shiro up to core takes only 49s with parallel tests enabled.

Outlook: JUnit Looming

Christian Stein’s JUnit Looming is a showcase project wich shows how virtual tests can speed up testing. As of my knowledge (Christian may have slipped something 😉), it might get included soon into junit-jupiter.

Making the CI slower

However, please keep in mind that those options can actually hurt performance. In my CI, a few builds got slower, while the local builds on workstations were significantly faster. Reasons for this can include CPU or RAM constraints, slower file systems and more concurrent use of system resources in general. If you encounter this phenomenon, try to tune your parallelism until you find a "sweet spot" which speeds up both your local and your CI builds.

Conclusion: Parallel testing in Apache Maven

While JUnit Jupiter offers the most versatile options to parallelize your tests, a single surefire configuration can handle all the other test frameworks well. IntelliJ will sadly only pick up the property files and the annotations, which is why I use property files in some of my projects. There is a lot of room for improvement when it comes down to comparing the default configuration with a per-project optimal configuration. This is left to the educated reader as each project needs to be tuned individually.