Polling Confirmations: Renamed from Polling Expectations

- Include commenting on why not pure counts (i.e. why counts + interval)
- Specify the configuration traits
- Specify callsite configuration of counts + interval

Author: younata

proposals/testing/NNNN-polling-expectations.md renamed to proposals/testing/NNNN-polling-confirmations.md

@@ -1,6 +1,6 @@
-# Polling Expectations
+# Polling Confirmations
 
-* Proposal: [ST-NNNN](NNNN-polling-expectations.md)
+* Proposal: [ST-NNNN](NNNN-polling-confirmations.md)
 * Authors: [Rachel Brindle](https://github.com/younata)
 * Review Manager: TBD
 * Status: **Awaiting review**
@@ -42,7 +42,9 @@ actor Aquarium {
 
 This proposal introduces new members of the `confirmation` family of functions:
 `confirmPassesEventually` and `confirmAlwaysPasses`. These functions take in
-a closure to be continuously evaluated until the specific condition passes.
+a closure to be repeatedly evaluated until the specific condition passes,
+waiting at least some amount of time - specified by `pollingInterval` and
+defaulting to 1 millisecond - before evaluating the closure again.
 
 `confirmPassesEventually` will evaluate the closure until the first time it
 returns true or a non-nil value. `confirmAlwaysPasses` will evaluate the
@@ -75,6 +77,21 @@ testing library:
 /// - Parameters:
 ///   - comment: An optional comment to apply to any issues generated by this
 ///     function.
+///   - maxPollingIterations: The maximum amount of times to attempt polling.
+///     If nil, this uses whatever value is specified under the last
+///     ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or
+///     suite.
+///     If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then
+///     polling will be attempted 1000 times before recording an issue.
+///     `maxPollingIterations` must be greater than 0.
+///   - pollingInterval: The minimum amount of time to wait between polling
+///     attempts.
+///     If nil, this uses whatever value is specified under the last
+///     ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or
+///     suite.
+///     If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then
+///     polling will wait at least 1 millisecond between polling attempts.
+///     `pollingInterval` must be greater than 0.
 ///   - isolation: The actor to which `body` is isolated, if any.
 ///   - sourceLocation: The source location to whych any recorded issues should
 ///     be attributed.
@@ -87,6 +104,8 @@ testing library:
 @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *)
 public func confirmPassesEventually(
   _ comment: Comment? = nil,
+  maxPollingIterations: Int? = nil,
+  pollingInterval: Duration? = nil,
   isolation: isolated (any Actor)? = #isolation,
   sourceLocation: SourceLocation = #_sourceLocation,
   _ body: @escaping () async throws -> Bool
@@ -97,6 +116,20 @@ public func confirmPassesEventually(
 /// - Parameters:
 ///   - comment: An optional comment to apply to any issues generated by this
 ///     function.
+///   - maxPollingIterations: The maximum amount of times to attempt polling.
+///     If nil, this uses whatever value is specified under the last
+///     ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or
+///     suite.
+///     If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then
+///     polling will be attempted 1000 times before recording an issue.
+///     `maxPollingIterations` must be greater than 0.
+///   - pollingInterval: The minimum amount of time to wait between polling
+///     attempts.
+///     If nil, this uses whatever value is specified under the last
+///     ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite.
+///     If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then
+///     polling will wait at least 1 millisecond between polling attempts.
+///     `pollingInterval` must be greater than 0.
 ///   - isolation: The actor to which `body` is isolated, if any.
 ///   - sourceLocation: The source location to whych any recorded issues should
 ///     be attributed.
@@ -114,6 +147,8 @@ public func confirmPassesEventually(
 @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *)
 public func confirmPassesEventually<R>(
   _ comment: Comment? = nil,
+  maxPollingIterations: Int? = nil,
+  pollingInterval: Duration? = nil,
   isolation: isolated (any Actor)? = #isolation,
   sourceLocation: SourceLocation = #_sourceLocation,
   _ body: @escaping () async throws -> R?
@@ -124,6 +159,19 @@ public func confirmPassesEventually<R>(
 /// - Parameters:
 ///   - comment: An optional comment to apply to any issues generated by this
 ///     function.
+///   - maxPollingIterations: The maximum amount of times to attempt polling.
+///     If nil, this uses whatever value is specified under the last
+///     ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite.
+///     If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then
+///     polling will be attempted 1000 times before recording an issue.
+///     `maxPollingIterations` must be greater than 0.
+///   - pollingInterval: The minimum amount of time to wait between polling
+///     attempts.
+///     If nil, this uses whatever value is specified under the last
+///     ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite.
+///     If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then
+///     polling will wait at least 1 millisecond between polling attempts.
+///     `pollingInterval` must be greater than 0.
 ///   - isolation: The actor to which `body` is isolated, if any.
 ///   - sourceLocation: The source location to whych any recorded issues should
 ///     be attributed.
@@ -135,36 +183,12 @@ public func confirmPassesEventually<R>(
 @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *)
 public func confirmAlwaysPasses(
   _ comment: Comment? = nil,
+  maxPollingIterations: Int? = nil,
+  pollingInterval: Duration? = nil,
   isolation: isolated (any Actor)? = #isolation,
   sourceLocation: SourceLocation = #_sourceLocation,
   _ body: @escaping () async throws -> Bool
 ) async
-
-/// Confirm that some expression always returns a non-optional value
-///
-/// - Parameters:
-///   - comment: An optional comment to apply to any issues generated by this
-///     function.
-///   - isolation: The actor to which `body` is isolated, if any.
-///   - sourceLocation: The source location to whych any recorded issues should
-///     be attributed.
-///   - body: The function to invoke.
-///
-/// - Returns: The value from the last time `body` was invoked.
-///
-/// - Throws: A `PollingFailedError` will be thrown if `body` ever returns a
-///   non-optional value
-///
-/// Use polling confirmations to check that an event while a test is running in
-/// complex scenarios where other forms of confirmation are insufficient. For
-/// example, confirming that some state does not change.
-@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *)
-public func confirmAlwaysPasses<R>(
-  _ comment: Comment? = nil,
-  isolation: isolated (any Actor)? = #isolation,
-  sourceLocation: SourceLocation = #_sourceLocation,
-  _ body: @escaping () async throws -> R?
-) async throws -> R where R: Sendable
 ```
 
 ### New Error Type
@@ -184,9 +208,99 @@ A new Issue.Kind, `confirmationPollingFailed` will be added to represent the
 case here confirmation polling fails. This issue kind will be recorded when
 polling fails.
 
+### New Traits
+
+Two new traits will be added to change the default values for the
+`maxPollingIterations` and `pollingInterval` arguments. Test authors often
+want to poll for the `passesEventually` behavior more than they poll for the
+`alwaysPasses` behavior, which is why there are separate traits for configuring
+defaults for these functions.
+
+```swift
+/// A trait to provide a default polling configuration to all usages of
+/// ``confirmPassesEventually`` within a test or suite.
+///
+/// To add this trait to a test, use the
+/// ``Trait/confirmPassesEventuallyDefaults`` function.
+@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *)
+public struct ConfirmPassesEventuallyConfigurationTrait: TestTrait, SuiteTrait {
+  public var maxPollingIterations: Int?
+  public var pollingInterval: Duration?
+
+  public var isRecursive: Bool { true }
+
+  public init(maxPollingIterations: Int?, pollingInterval: Duration?)
+}
+
+/// A trait to provide a default polling configuration to all usages of
+/// ``confirmPassesAlways`` within a test or suite.
+///
+/// To add this trait to a test, use the ``Trait/confirmAlwaysPassesDefaults``
+/// function.
+@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *)
+public struct ConfirmAlwaysPassesConfigurationTrait: TestTrait, SuiteTrait {
+  public var maxPollingIterations: Int?
+  public var pollingInterval: Duration?
+
+  public var isRecursive: Bool { true }
+
+  public init(maxPollingIterations: Int?, pollingInterval: Duration?)
+}
+
+@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *)
+extension Trait where Self == ConfirmPassesEventuallyConfigurationTrait {
+  /// Specifies defaults for ``confirmPassesEventually`` in the test or suite.
+  ///
+  /// - Parameters:
+  ///   - maxPollingIterations: The maximum amount of times to attempt polling.
+  ///     If nil, polling will be attempted up to 1000 times.
+  ///     `maxPollingIterations` must be greater than 0.
+  ///   - pollingInterval: The minimum amount of time to wait between polling
+  ///     attempts.
+  ///     If nil, polling will wait at least 1 millisecond between polling
+  ///     attempts.
+  ///     `pollingInterval` must be greater than 0.
+  public static func confirmPassesEventuallyDefaults(
+    maxPollingIterations: Int? = nil,
+    pollingInterval: Duration? = nil
+  ) -> Self
+}
+
+@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *)
+extension Trait where Self == ConfirmPassesAlwaysConfigurationTrait {
+  /// Specifies defaults for ``confirmAlwaysPasses`` in the test or suite.
+  ///
+  /// - Parameters:
+  ///   - maxPollingIterations: The maximum amount of times to attempt polling.
+  ///     If nil, polling will be attempted up to 1000 times.
+  ///     `maxPollingIterations` must be greater than 0.
+  ///   - pollingInterval: The minimum amount of time to wait between polling
+  ///     attempts.
+  ///     If nil, polling will wait at least 1 millisecond between polling
+  ///     attempts.
+  ///     `pollingInterval` must be greater than 0.
+  public static func confirmAlwaysPassesDefaults(
+    maxPollingIterations: Int? = nil,
+    pollingInterval: Duration? = nil
+  ) -> Self
+}
+```
+
+Specifying `maxPollingIterations` or `pollingInterval` directly on either
+`confirmPassesEventually` or `confirmAlwaysPasses` will override any value
+provided by the trait.
+
+### Default Polling Configuration
+
+For both `confirmPassesEventually` and `confirmsAlwaysPasses`, the Testing
+library will default `maxPollingIterations` to 1000, and `pollingInterval` to
+1 millisecond. This allows for tests on lightly-loaded systems such as developer
+workstations to run in a little over 1 second wall-clock time, while still
+being able to gracefully handle running on large loads.
+
 ### Platform Availability
 
-Polling expectations will not be available on platforms that do not support
+Polling confirmations will not be available on platforms that do not support
 Swift Concurrency.
 
 ### Usage
@@ -211,7 +325,8 @@ will end, and no failure will be reported.
 
 Polling will be stopped in the following cases:
 
-- After the expression has been evaluated 1 million times.
+- After the expression has been evaluated up to the count specified in
+  `maxPollingInterations`.
 - If the task that started the polling is cancelled.
 - For `confirmPassesEventually`: The first time the closure returns true or a
   non-nil value
@@ -253,6 +368,31 @@ tests are executed serially, the concurrent test runner the testing library uses
 means that timeouts are inherently unreliable. Importantly, timeouts become more
 unreliable the more tests in the test suite.
 
+### Use only polling iterations
+
+Another option considered was only using polling iterations. Naively, this
+would write the main polling loop as:
+
+```swift
+func poll(iterations: Int, expression: () -> Bool) async -> Bool {
+    for _ in 0..<iterations {
+        if expression() { return true }
+        Task.yield()
+    }
+    return false
+}
+```
+
+However, while this works and is resistant to many of the issues timeouts face
+in concurrent testing environments, it is extremely difficult for test authors
+to predict a good-enough polling iterations value. It is much easier for a
+human to predict that something should take a second or two than it is for a
+human to predict how many polling intervals it should take before the state
+changes - polling even a million times on a lightly-loaded system can finish in
+well under a millisecond. Because of this, we decided to add on the polling
+interval argument: a minimum duration to wait between polling, to make it much
+easier for test authors to predict a good-enough guess for when to stop polling.
+
 ### Use macros instead of functions
 
 Instead of adding new bare functions, polling could be written as additional