Skip to content

Commit 24bfbca

Browse files
marcphilippmpkorstanjeleonard84
authored
Implement parallel test execution without using ForkJoinPool (#5060)
This commit introduces a new implementation of `HierarchicalTestExecutorService` that runs rests in parallel and has limited work stealing capabilities but is not based on `ForkJoinPool`. It avoids its pitfalls (such as #3945) and #3108 but may require additional threads because its work stealing is limited to direct children. Contrary to the `ForkJoinPool` implementation, the new executor service guarantees that no more than `parallelism` test nodes are executed in parallel. The new implementation is initially going to ship as an opt-in feature that can be enabled via the new `junit.jupiter.execution.parallel.config.executor-service` configuration parameter. Resolves #3108. --------- Co-authored-by: M.P. Korstanje <rien.korstanje@gmail.com> Co-authored-by: Leonard Brünings <lord_damokles@gmx.net>
1 parent d928cf6 commit 24bfbca

File tree

37 files changed

+2556
-189
lines changed

37 files changed

+2556
-189
lines changed

‎documentation/src/docs/asciidoc/release-notes/release-notes-6.1.0-M1.adoc‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,13 +21,20 @@ repository on GitHub.
2121
[[release-notes-6.1.0-M1-junit-platform-deprecations-and-breaking-changes]]
2222
==== Deprecations and Breaking Changes
2323

24-
* ❓
24+
* Deprecate constructors for `ForkJoinPoolHierarchicalTestExecutorService` in favor of the
25+
new `ParallelHierarchicalTestExecutorServiceFactory` that also supports
26+
`WorkerThreadPoolHierarchicalTestExecutorService`.
2527

2628
[[release-notes-6.1.0-M1-junit-platform-new-features-and-improvements]]
2729
==== New Features and Improvements
2830

2931
* Support for creating a `ModuleSelector` from a `java.lang.Module` and using
3032
its classloader for test discovery.
33+
* New `WorkerThreadPoolHierarchicalTestExecutorService` implementation used for parallel
34+
test execution that is backed by a regular thread pool rather than a `ForkJoinPool`.
35+
Engine authors should switch to use `ParallelHierarchicalTestExecutorServiceFactory`
36+
rather than instantiating a concrete `HierarchicalTestExecutorService` implementation
37+
for parallel execution directly.
3138
* `OpenTestReportGeneratingListener` now supports redirecting XML events to a socket via
3239
the new `junit.platform.reporting.open.xml.socket` configuration parameter. When set to a
3340
port number, events are sent to `127.0.0.1:<port>` instead of being written to a file.
@@ -61,6 +68,13 @@ repository on GitHub.
6168
* Enrich `assertInstanceOf` failure using the test subject `Throwable` as cause. It
6269
results in the stack trace of the test subject `Throwable` to get reported along with
6370
the failure.
71+
* Make implementation of `HierarchicalTestExecutorService` used for parallel test
72+
execution configurable via the new
73+
`junit.jupiter.execution.parallel.config.executor-service` configuration parameter to
74+
in order to add support for `WorkerThreadPoolHierarchicalTestExecutorService`. Please
75+
refer to the
76+
<<../user-guide/index.adoc#writing-tests-parallel-execution-config-executor-service, User Guide>>
77+
for details.
6478

6579
[[release-notes-6.1.0-M1-junit-vintage]]
6680
=== JUnit Vintage

‎documentation/src/docs/asciidoc/user-guide/writing-tests.adoc‎

Lines changed: 92 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -3433,6 +3433,33 @@ used instead.
34333433
[[writing-tests-parallel-execution-config]]
34343434
==== Configuration
34353435

3436+
[[writing-tests-parallel-execution-config-executor-service]]
3437+
===== Executor Service
3438+
3439+
If parallel execution is enabled, a thread pool is used behind the scenes to execute tests
3440+
concurrently. You can configure which implementation of `HierarchicalTestExecutorService`
3441+
is used be setting the `junit.jupiter.execution.parallel.config.executor-service`
3442+
configuration parameter to one of the following options:
3443+
3444+
`fork_join_pool` (default)::
3445+
Use an executor service that is backed by a `ForkJoinPool` from the JDK. This will cause
3446+
tests to be executed in a `ForkJoinWorkerThread`. In some cases, usages of
3447+
`ForkJoinPool` in test or production code or calls to blocking JDK APIs may cause the
3448+
number of concurrently executing tests to increase. To avoid this situation, please use
3449+
`worker_thread_pool`.
3450+
3451+
`worker_thread_pool` (experimental)::
3452+
Use an executor service that is backed by a regular thread pool and does not create
3453+
additional threads if test or production code uses `ForkJoinPool` or calls a blocking
3454+
API in the JDK.
3455+
3456+
WARNING: Using `worker_thread_pool` is currently an _experimental_ feature. You're invited
3457+
to give it a try and provide feedback to the JUnit team so they can improve and eventually
3458+
<<api-evolution, promote>> this feature.
3459+
3460+
[[writing-tests-parallel-execution-config-strategies]]
3461+
===== Strategies
3462+
34363463
Properties such as the desired parallelism and the maximum pool size can be configured
34373464
using a `{ParallelExecutionConfigurationStrategy}`. The JUnit Platform provides two
34383465
implementations out of the box: `dynamic` and `fixed`. Alternatively, you may implement a
@@ -3464,13 +3491,12 @@ strategy with a factor of `1`. Consequently, the desired parallelism will be equ
34643491
number of available processors/cores.
34653492

34663493
.Parallelism alone does not imply maximum number of concurrent threads
3467-
NOTE: By default JUnit Jupiter does not guarantee that the number of concurrently
3468-
executing tests will not exceed the configured parallelism. For example, when using one
3469-
of the synchronization mechanisms described in the next section, the `ForkJoinPool` that
3470-
is used behind the scenes may spawn additional threads to ensure execution continues with
3471-
sufficient parallelism.
3472-
If you require such guarantees, it is possible to limit the maximum number of concurrent
3473-
threads by controlling the maximum pool size of the `dynamic`, `fixed` and `custom`
3494+
NOTE: By default, JUnit Jupiter does not guarantee that the number of threads used to
3495+
execute test will not exceed the configured parallelism. For example, when using one
3496+
of the synchronization mechanisms described in the next section, the executor service
3497+
implementation may spawn additional threads to ensure execution continues with sufficient
3498+
parallelism. If you require such guarantees, it is possible to limit the maximum number of
3499+
threads by configuring the maximum pool size of the `dynamic`, `fixed` and `custom`
34743500
strategies.
34753501

34763502
[[writing-tests-parallel-execution-config-properties]]
@@ -3479,86 +3505,66 @@ strategies.
34793505
The following table lists relevant properties for configuring parallel execution. See
34803506
<<running-tests-config-params>> for details on how to set such properties.
34813507

3482-
[cols="d,d,a,d"]
3483-
|===
3484-
|Property |Description |Supported Values |Default Value
3485-
3486-
| ```junit.jupiter.execution.parallel.enabled```
3487-
| Enable parallel test execution
3488-
|
3489-
* `true`
3490-
* `false`
3491-
| ```false```
3492-
3493-
| ```junit.jupiter.execution.parallel.mode.default```
3494-
| Default execution mode of nodes in the test tree
3495-
|
3496-
* `concurrent`
3497-
* `same_thread`
3498-
| ```same_thread```
3499-
3500-
| ```junit.jupiter.execution.parallel.mode.classes.default```
3501-
| Default execution mode of top-level classes
3502-
|
3503-
* `concurrent`
3504-
* `same_thread`
3505-
| ```same_thread```
3506-
3507-
| ```junit.jupiter.execution.parallel.config.strategy```
3508-
| Execution strategy for desired parallelism and maximum pool size
3509-
|
3510-
* `dynamic`
3511-
* `fixed`
3512-
* `custom`
3513-
| ```dynamic```
3514-
3515-
| ```junit.jupiter.execution.parallel.config.dynamic.factor```
3516-
| Factor to be multiplied by the number of available processors/cores to determine the
3517-
desired parallelism for the ```dynamic``` configuration strategy
3518-
| a positive decimal number
3519-
| ```1.0```
3520-
3521-
| ```junit.jupiter.execution.parallel.config.dynamic.max-pool-size-factor```
3522-
| Factor to be multiplied by the number of available processors/cores and the value of
3508+
====== General
3509+
3510+
`junit.jupiter.execution.parallel.enabled=true|false`::
3511+
Enable/disable parallel test execution (defaults to `false`).
3512+
3513+
`junit.jupiter.execution.parallel.mode.default=concurrent|same_thread`::
3514+
Default execution mode of nodes in the test tree (defaults to `same_thread`).
3515+
3516+
`junit.jupiter.execution.parallel.mode.classes.default=concurrent|same_thread`::
3517+
Default execution mode of top-level classes (defaults to `same_thread`).
3518+
3519+
`junit.jupiter.execution.parallel.config.executor-service=fork_join_pool|worker_thread_pool`::
3520+
Type of `HierarchicalTestExecutorService` to use for parallel execution (defaults to
3521+
`fork_join_pool`).
3522+
3523+
`junit.jupiter.execution.parallel.config.strategy=dynamic|fixed|custom`::
3524+
Execution strategy for desired parallelism, maximum pool size, etc. (defaults to `dynamic`).
3525+
3526+
====== Dynamic strategy
3527+
3528+
`junit.jupiter.execution.parallel.config.dynamic.factor=decimal`::
3529+
Factor to be multiplied by the number of available processors/cores to determine the
3530+
desired parallelism for the ```dynamic``` configuration strategy.
3531+
Must be a positive decimal number (defaults to `1.0`).
3532+
3533+
`junit.jupiter.execution.parallel.config.dynamic.max-pool-size-factor=decimal`::
3534+
Factor to be multiplied by the number of available processors/cores and the value of
35233535
`junit.jupiter.execution.parallel.config.dynamic.factor` to determine the desired
3524-
parallelism for the ```dynamic``` configuration strategy
3525-
| a positive decimal number, must be greater than or equal to `1.0`
3526-
| 256 + the value of `junit.jupiter.execution.parallel.config.dynamic.factor` multiplied
3527-
by the number of available processors/cores
3528-
3529-
| ```junit.jupiter.execution.parallel.config.dynamic.saturate```
3530-
| Disable saturation of the underlying fork-join pool for the ```dynamic``` configuration
3531-
strategy
3532-
|
3533-
* `true`
3534-
* `false`
3535-
| ```true```
3536-
3537-
| ```junit.jupiter.execution.parallel.config.fixed.parallelism```
3538-
| Desired parallelism for the ```fixed``` configuration strategy
3539-
| a positive integer
3540-
| no default value
3541-
3542-
| ```junit.jupiter.execution.parallel.config.fixed.max-pool-size```
3543-
| Desired maximum pool size of the underlying fork-join pool for the ```fixed```
3544-
configuration strategy
3545-
| a positive integer, must be greater than or equal to `junit.jupiter.execution.parallel.config.fixed.parallelism`
3546-
| 256 + the value of `junit.jupiter.execution.parallel.config.fixed.parallelism`
3547-
3548-
| ```junit.jupiter.execution.parallel.config.fixed.saturate```
3549-
| Disable saturation of the underlying fork-join pool for the ```fixed``` configuration
3550-
strategy
3551-
|
3552-
* `true`
3553-
* `false`
3554-
| ```true```
3555-
3556-
| ```junit.jupiter.execution.parallel.config.custom.class```
3557-
| Fully qualified class name of the _ParallelExecutionConfigurationStrategy_ to be
3558-
used for the ```custom``` configuration strategy
3559-
| for example, _org.example.CustomStrategy_
3560-
| no default value
3561-
|===
3536+
parallelism for the ```dynamic``` configuration strategy.
3537+
Must be a positive decimal number greater than or equal to `1.0` (defaults to 256 plus
3538+
the value of `junit.jupiter.execution.parallel.config.dynamic.factor` multiplied by the
3539+
number of available processors/cores)
3540+
3541+
`junit.jupiter.execution.parallel.config.dynamic.saturate=true|false`::
3542+
Enable/disable saturation of the underlying `ForkJoinPool` for the ```dynamic```
3543+
configuration strategy (defaults to `true`). Only used if
3544+
`junit.jupiter.execution.parallel.config.executor-service` is set to `fork_join_pool`.
3545+
3546+
====== Fixed strategy
3547+
3548+
`junit.jupiter.execution.parallel.config.fixed.parallelism=integer`::
3549+
Desired parallelism for the ```fixed``` configuration strategy (no default value). Must
3550+
be a positive integer.
3551+
3552+
`junit.jupiter.execution.parallel.config.fixed.max-pool-size=integer`::
3553+
Desired maximum pool size of the underlying fork-join pool for the ```fixed```
3554+
configuration strategy. Must be a positive integer greater than or equal to
3555+
`junit.jupiter.execution.parallel.config.fixed.parallelism` (defaults to 256 plus the
3556+
value of `junit.jupiter.execution.parallel.config.fixed.parallelism`).
3557+
3558+
`junit.jupiter.execution.parallel.config.fixed.saturate=true|false`::
3559+
Enable/disable saturation of the underlying `ForkJoinPool` for the ```fixed```
3560+
configuration strategy (defaults to `true`). Only used if
3561+
`junit.jupiter.execution.parallel.config.executor-service` is set to `fork_join_pool`.
3562+
3563+
====== Custom strategy
3564+
3565+
`junit.jupiter.execution.parallel.config.custom.class=classname`::
3566+
Fully qualified class name of the `ParallelExecutionConfigurationStrategy` to be used
3567+
for the ```custom``` configuration strategy (no default value).
35623568

35633569
[[writing-tests-parallel-execution-synchronization]]
35643570
==== Synchronization

‎documentation/src/test/resources/junit-platform.properties‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
junit.jupiter.execution.parallel.enabled=true
22
junit.jupiter.execution.parallel.mode.default=concurrent
3+
junit.jupiter.execution.parallel.config.executor-service=worker_thread_pool
34
junit.jupiter.execution.parallel.config.strategy=fixed
45
junit.jupiter.execution.parallel.config.fixed.parallelism=6
56

‎documentation/src/test/resources/log4j2-test.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
xsi:schemaLocation="https://logging.apache.org/xml/ns https://logging.apache.org/xml/ns/log4j-config-2.xsd">
66
<Appenders>
77
<Console name="Console" target="SYSTEM_OUT">
8-
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
8+
<PatternLayout pattern="%d{HH:mm:ss.SSSSSS} [%-18t] %-5level %logger{1.} - %msg%n"/>
99
</Console>
1010
</Appenders>
1111
<Loggers>

‎junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/Constants.java‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@
3939
import org.junit.jupiter.engine.config.JupiterConfiguration;
4040
import org.junit.platform.commons.util.ClassNamePatternFilterUtils;
4141
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfigurationStrategy;
42+
import org.junit.platform.engine.support.hierarchical.ParallelHierarchicalTestExecutorServiceFactory;
43+
import org.junit.platform.engine.support.hierarchical.ParallelHierarchicalTestExecutorServiceFactory.ParallelExecutorServiceType;
4244

4345
/**
4446
* Collection of constants related to the {@link JupiterTestEngine}.
@@ -237,7 +239,21 @@ public final class Constants {
237239
@API(status = STABLE, since = "5.10")
238240
public static final String DEFAULT_CLASSES_EXECUTION_MODE_PROPERTY_NAME = Execution.DEFAULT_CLASSES_EXECUTION_MODE_PROPERTY_NAME;
239241

240-
static final String PARALLEL_CONFIG_PREFIX = "junit.jupiter.execution.parallel.config.";
242+
/**
243+
* Property name used to determine the desired
244+
* {@link ParallelExecutorServiceType ParallelExecutorServiceType}:
245+
* {@value}
246+
*
247+
* <p>Value must be
248+
* {@link ParallelExecutorServiceType#FORK_JOIN_POOL FORK_JOIN_POOL} or
249+
* {@link ParallelExecutorServiceType#WORKER_THREAD_POOL WORKER_THREAD_POOL},
250+
* ignoring case.
251+
*
252+
* @since 6.1
253+
* @see ParallelHierarchicalTestExecutorServiceFactory
254+
*/
255+
@API(status = EXPERIMENTAL, since = "6.1")
256+
public static final String PARALLEL_CONFIG_EXECUTOR_SERVICE_PROPERTY_NAME = JupiterConfiguration.PARALLEL_CONFIG_EXECUTOR_SERVICE_PROPERTY_NAME;
241257

242258
/**
243259
* Property name used to select the
@@ -249,7 +265,7 @@ public final class Constants {
249265
* @since 5.3
250266
*/
251267
@API(status = STABLE, since = "5.10")
252-
public static final String PARALLEL_CONFIG_STRATEGY_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
268+
public static final String PARALLEL_CONFIG_STRATEGY_PROPERTY_NAME = JupiterConfiguration.PARALLEL_CONFIG_PREFIX
253269
+ CONFIG_STRATEGY_PROPERTY_NAME;
254270

255271
/**
@@ -261,7 +277,7 @@ public final class Constants {
261277
* @since 5.3
262278
*/
263279
@API(status = STABLE, since = "5.10")
264-
public static final String PARALLEL_CONFIG_FIXED_PARALLELISM_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
280+
public static final String PARALLEL_CONFIG_FIXED_PARALLELISM_PROPERTY_NAME = JupiterConfiguration.PARALLEL_CONFIG_PREFIX
265281
+ CONFIG_FIXED_PARALLELISM_PROPERTY_NAME;
266282

267283
/**
@@ -275,7 +291,7 @@ public final class Constants {
275291
* @since 5.10
276292
*/
277293
@API(status = MAINTAINED, since = "5.13.3")
278-
public static final String PARALLEL_CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
294+
public static final String PARALLEL_CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME = JupiterConfiguration.PARALLEL_CONFIG_PREFIX
279295
+ CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME;
280296

281297
/**
@@ -291,7 +307,7 @@ public final class Constants {
291307
* @since 5.10
292308
*/
293309
@API(status = MAINTAINED, since = "5.13.3")
294-
public static final String PARALLEL_CONFIG_FIXED_SATURATE_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
310+
public static final String PARALLEL_CONFIG_FIXED_SATURATE_PROPERTY_NAME = JupiterConfiguration.PARALLEL_CONFIG_PREFIX
295311
+ CONFIG_FIXED_SATURATE_PROPERTY_NAME;
296312

297313
/**
@@ -304,7 +320,7 @@ public final class Constants {
304320
* @since 5.3
305321
*/
306322
@API(status = STABLE, since = "5.10")
307-
public static final String PARALLEL_CONFIG_DYNAMIC_FACTOR_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
323+
public static final String PARALLEL_CONFIG_DYNAMIC_FACTOR_PROPERTY_NAME = JupiterConfiguration.PARALLEL_CONFIG_PREFIX
308324
+ CONFIG_DYNAMIC_FACTOR_PROPERTY_NAME;
309325

310326
/**
@@ -315,7 +331,7 @@ public final class Constants {
315331
* @since 5.3
316332
*/
317333
@API(status = STABLE, since = "5.10")
318-
public static final String PARALLEL_CONFIG_CUSTOM_CLASS_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
334+
public static final String PARALLEL_CONFIG_CUSTOM_CLASS_PROPERTY_NAME = JupiterConfiguration.PARALLEL_CONFIG_PREFIX
319335
+ CONFIG_CUSTOM_CLASS_PROPERTY_NAME;
320336

321337
/**

‎junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/JupiterTestEngine.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@
2929
import org.junit.platform.engine.UniqueId;
3030
import org.junit.platform.engine.support.config.PrefixedConfigurationParameters;
3131
import org.junit.platform.engine.support.discovery.DiscoveryIssueReporter;
32-
import org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService;
3332
import org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine;
3433
import org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutorService;
34+
import org.junit.platform.engine.support.hierarchical.ParallelHierarchicalTestExecutorServiceFactory;
3535
import org.junit.platform.engine.support.hierarchical.ThrowableCollector;
3636

3737
/**
@@ -79,8 +79,8 @@ public TestDescriptor discover(EngineDiscoveryRequest discoveryRequest, UniqueId
7979
protected HierarchicalTestExecutorService createExecutorService(ExecutionRequest request) {
8080
JupiterConfiguration configuration = getJupiterConfiguration(request);
8181
if (configuration.isParallelExecutionEnabled()) {
82-
return new ForkJoinPoolHierarchicalTestExecutorService(new PrefixedConfigurationParameters(
83-
request.getConfigurationParameters(), Constants.PARALLEL_CONFIG_PREFIX));
82+
return ParallelHierarchicalTestExecutorServiceFactory.create(new PrefixedConfigurationParameters(
83+
request.getConfigurationParameters(), JupiterConfiguration.PARALLEL_CONFIG_PREFIX));
8484
}
8585
return super.createExecutorService(request);
8686
}

‎junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/DefaultJupiterConfiguration.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@
1616
import static org.junit.jupiter.api.io.TempDir.DEFAULT_CLEANUP_MODE_PROPERTY_NAME;
1717
import static org.junit.jupiter.api.io.TempDir.DEFAULT_FACTORY_PROPERTY_NAME;
1818
import static org.junit.jupiter.engine.config.FilteringConfigurationParameterConverter.exclude;
19+
import static org.junit.platform.engine.support.hierarchical.ParallelHierarchicalTestExecutorServiceFactory.ParallelExecutorServiceType.FORK_JOIN_POOL;
20+
import static org.junit.platform.engine.support.hierarchical.ParallelHierarchicalTestExecutorServiceFactory.ParallelExecutorServiceType.WORKER_THREAD_POOL;
1921

2022
import java.util.List;
2123
import java.util.Optional;
@@ -100,6 +102,17 @@ private void validateConfigurationParameters(DiscoveryIssueReporter issueReporte
100102
Please remove it from your configuration.""".formatted(key));
101103
issueReporter.reportIssue(warning);
102104
}));
105+
if (isParallelExecutionEnabled()
106+
&& configurationParameters.get(PARALLEL_CONFIG_EXECUTOR_SERVICE_PROPERTY_NAME).isEmpty()) {
107+
var info = DiscoveryIssue.create(Severity.INFO,
108+
"Parallel test execution is enabled but the default ForkJoinPool-based executor service will be used. "
109+
+ "Please give the new implementation based on a regular thread pool a try by setting the '"
110+
+ PARALLEL_CONFIG_EXECUTOR_SERVICE_PROPERTY_NAME + "' configuration parameter to '"
111+
+ WORKER_THREAD_POOL + "' and report any issues to the JUnit team. "
112+
+ "Alternatively, set the configuration parameter to '" + FORK_JOIN_POOL
113+
+ "' to hide this message and keep using the original implementation.");
114+
issueReporter.reportIssue(info);
115+
}
103116
}
104117

105118
@Override

0 commit comments

Comments
 (0)