Document discovery issue reporting

Issue: #242

Author: marcphilipp

documentation/src/docs/asciidoc/link-attributes.adoc

@@ -25,6 +25,8 @@ endif::[]
 :ClasspathResourceSelector:                  {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/ClasspathResourceSelector.html[ClasspathResourceSelector]
 :ClasspathRootSelector:                      {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/ClasspathRootSelector.html[ClasspathRootSelector]
 :ClassSelector:                              {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/ClassSelector.html[ClassSelector]
+:DiscoveryIssue:                             {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/DiscoveryIssue.html[DiscoveryIssue]
+:DiscoveryIssueReporter:                     {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/support/discovery/DiscoveryIssueReporter.html[DiscoveryIssueReporter]
 :DirectorySelector:                          {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/DirectorySelector.html[DirectorySelector]
 :DiscoverySelectors:                         {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/DiscoverySelectors.html[DiscoverySelectors]
 :DiscoverySelectors_selectClasspathResource: {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/DiscoverySelectors.html#selectClasspathResource(java.lang.String)[selectClasspathResource]
@@ -40,6 +42,7 @@ endif::[]
 :DiscoverySelectors_selectPackage:           {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/DiscoverySelectors.html#selectPackage(java.lang.String)[selectPackage]
 :DiscoverySelectors_selectUniqueId:          {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/DiscoverySelectors.html#selectUniqueId(java.lang.String)[selectUniqueId]
 :DiscoverySelectors_selectUri:               {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/DiscoverySelectors.html#selectUri(java.lang.String)[selectUri]
+:EngineDiscoveryListener:                    {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/EngineDiscoveryListener.html[EngineDiscoveryListener]
 :EngineDiscoveryRequest:                     {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/EngineDiscoveryRequest.html[EngineDiscoveryRequest]
 :FileSelector:                               {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/discovery/FileSelector.html[FileSelector]
 :HierarchicalTestEngine:                     {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/support/hierarchical/HierarchicalTestEngine.html[HierarchicalTestEngine]
@@ -57,6 +60,7 @@ endif::[]
 :TestEngine:                                 {javadoc-root}/org.junit.platform.engine/org/junit/platform/engine/TestEngine.html[TestEngine]
 // Platform Launcher API
 :junit-platform-launcher:                    {javadoc-root}/org.junit.platform.launcher/org/junit/platform/launcher/package-summary.html[junit-platform-launcher]
+:DiscoveryIssueException:                    {javadoc-root}/org.junit.platform.launcher/org/junit/platform/launcher/core/DiscoveryIssueException.html[DiscoveryIssueException]
 :Launcher:                                   {javadoc-root}/org.junit.platform.launcher/org/junit/platform/launcher/Launcher.html[Launcher]
 :LauncherConfig:                             {javadoc-root}/org.junit.platform.launcher/org/junit/platform/launcher/core/LauncherConfig.html[LauncherConfig]
 :LauncherDiscoveryListener:                  {javadoc-root}/org.junit.platform.launcher/org/junit/platform/launcher/LauncherDiscoveryListener.html[LauncherDiscoveryListener]

documentation/src/docs/asciidoc/release-notes/release-notes-5.13.0-M3.adoc

@@ -35,6 +35,23 @@ repository on GitHub.
   and
   <<../user-guide/index.adoc#launcher-api-managing-state-across-test-engines, request-scoped>>
   resources.
+* Introduce a mechanism for `TestEngine` implementations to report issues encountered
+  during test discovery. If an engine reports a `DiscoveryIssue` with a `Severity` equal
+  to or higher than a configurable critical severity, its tests will not be executed.
+  Instead, the engine will be reported as failed during execution with a failure message
+  listing all critical issues. Non-critical issues will be logged but will not prevent the
+  engine from executing its tests. The critical severity can be configured via a new
+  configuration parameter and, currently, defaults to `ERROR`. Please refer to the
+  <<../user-guide/index.adoc#running-tests-discovery-issues, User Guide>> for details.
++
+If you're a test engine maintainer, please see the
+<<../user-guide/index.adoc#test-engines-discovery-issues, User Guide>> for details on how
+to start reporting discovery issues.
+* Start reporting discovery issues for problematic `@Suite` classes:
+  - Invalid `@Suite` class declarations (for example, when `private`)
+  - Invalid `@BeforeSuite`/`@AfterSuite` method declarations (for example, when not
+    `static`)
+  - Cyclic dependencies between `@Suite` classes
 
 
 [[release-notes-5.13.0-M3-junit-jupiter]]
@@ -53,10 +70,21 @@ repository on GitHub.
 [[release-notes-5.13.0-M3-junit-jupiter-new-features-and-improvements]]
 ==== New Features and Improvements
 
-* Return types of `@TestFactory` methods are now validated during discovery rather than
-  during execution. Invalid `@TestFactory` methods are no longer executed but reported as
-  discovery issues for consistency with how `@Test` and `@TestTemplate` methods are
-  handled.
+* Start reporting discovery issues for potentially problematic test classes:
+    - Invalid `@Test` and `@TestTemplate` method declarations (for example, when return
+      type is not `void`)
+    - Invalid `@TestFactory` methods (for example, when return type is invalid)
+    - Multiple method-level annotations (for example, `@Test` and `@TestTemplate`)
+    - Invalid test class and `@Nested` class declarations (for example, `static` `@Nested`
+      classes)
+    - Potentially missing `@Nested` annotations (for example, non-abstract inner classes
+      that contain test methods)
+    - Invalid lifecycle method declarations (for example, when `private`)
+    - Invalid `@Tag` syntax
+    - Blank `@DisplayName` declarations
+    - Blank `@SentenceFragment` declarations
+    - `@BeforeParameterizedClassInvocation` and `@AfterParameterizedClassInvocation`
+      methods declared in non-parameterized test classes
 
 
 [[release-notes-5.13.0-M3-junit-vintage]]

documentation/src/docs/asciidoc/user-guide/advanced-topics/engines.adoc

@@ -120,3 +120,22 @@ compatibility with build tools and IDEs:
   siblings or other nodes that are required for the execution of the selected tests.
 * `TestEngines` _should_ support <<running-tests-tags, tagging>> tests and containers so
   that tag filters can be applied when discovering tests.
+
+[[test-engines-discovery-issues]]
+==== Reporting Discovery Issues
+
+Test engines should report <<running-tests-discovery-issues, discovery issues>> if they
+encounter any problems or potential misconfigurations during test discovery. This is
+especially important if the issue could lead to tests not being executed at all or only
+partially.
+
+In order to report a `{DiscoveryIssue}`, a test engine should call the
+`issueEncountered()` method on the `{EngineDiscoveryListener}` available via the
+`{EngineDiscoveryRequest}` passed to its `discover()` method. Rather than passing the
+listener around, the `{DiscoveryIssueReporter}` interface should be used. It also provides
+a way to create a `Condition` that reports a discovery issue if its check fails and may
+be used as a `Predicate` or `Consumer`. Please refer to the implementations of the
+<<test-engines-junit, test engines provided by JUnit>> for examples.
+
+Moreover, <<testkit-engine-discovery, Engine Test Kit>> provides a way to write tests for
+reported discovery issues.

documentation/src/docs/asciidoc/user-guide/running-tests.adoc

@@ -1288,3 +1288,40 @@ never be excluded.
 
 In addition, all elements prior to and including the first call from the JUnit Platform
 Launcher will be removed.
+
+[[running-tests-discovery-issues]]
+=== Discovery Issues
+
+Test engines may encounter issues during test discovery. For example, the declaration of a
+test class or method may be invalid. To avoid such issues from going unnoticed, the JUnit
+Platform provides a <<test-engines-discovery-issues, mechanism for test engines>> to
+report them with different severity levels:
+
+INFO::
+Indicates that the engine encountered something that could be potentially problematic, but
+could also happen due to a valid setup or configuration.
+
+WARNING::
+Indicates that the engine encountered something that is problematic and might lead to
+unexpected behavior or will be removed or changed in a future release.
+
+ERROR::
+Indicates that the engine encountered something that is definitely problematic and will
+lead to unexpected behavior.
+
+If an engine reports an issue with a severity equal to or higher than a configurable
+_critical_ severity, its tests will not be executed. Instead, the engine will be reported
+as failed during execution with a `{DiscoveryIssueException}` listing all critical issues.
+Non-critical issues will be logged but will not prevent the engine from executing its
+tests. The `junit.platform.discovery.issue.severity.critical`
+<<running-tests-config-params, configuration parameter>> can be used to set the critical
+severity level. Currently, the default value is `ERROR` but it may be changed in a future
+release.
+
+TIP: To surface all discovery issues in your project, it is recommended to set the
+`junit.platform.discovery.issue.severity.critical` configuration parameter to `INFO`.
+
+In addition, registered `{LauncherDiscoveryListener}` implementations can receive
+discovery issues via the `issueEncountered()` method. This allows IDEs and build tools to
+report issues to the user in a more user-friendly way. For example, IDEs may choose to
+display all issues in a list or table.