Skip to content

Commit 59e71ff

Browse files
younatayounata
committed
New pitch for testing: Polling Expectations
1 parent 624d429 commit 59e71ff

File tree

1 file changed

+278
-0
lines changed

1 file changed

+278
-0
lines changed

‎proposals/testing/NNNN-polling-expectations.md‎

Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
# Polling Expectations
2+
3+
* Proposal: [ST-NNNN](NNNN-polling-expectations.md)
4+
* Authors: [Rachel Brindle](https://github.com/younata)
5+
* Review Manager: TBD
6+
* Status: **Awaiting implementation** or **Awaiting review**
7+
* Implementation: (Working on it)
8+
* Review: (Working on it)
9+
10+
## Introduction
11+
12+
Test authors frequently need to wait for some background activity to complete
13+
or reach an expected state before continuing. This proposal introduces a new API
14+
to enable polling for an expected state.
15+
16+
## Motivation
17+
18+
Test authors can currently utilize the existing [`confirmation`](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/confirmation(_:expectedcount:isolation:sourcelocation:_:)-5mqz2)
19+
APIs or awaiting on an `async` callable in order to block test execution until
20+
a callback is called, or an async callable returns. However, this requires the
21+
code being tested to support callbacks or return a status as an async callable.
22+
23+
This proposal adds another avenue for waiting for code to update to a specified
24+
value, by proactively polling the test closure until it passes or a timeout is
25+
reached.
26+
27+
More concretely, we can imagine a type that updates its status over an
28+
indefinite timeframe:
29+
30+
```swift
31+
actor Aquarium {
32+
var dolphins: [Dolphin]
33+
34+
func raiseDolphins() async {
35+
// over a very long timeframe
36+
dolphins.append(Dolphin())
37+
}
38+
}
39+
```
40+
41+
## Proposed solution
42+
43+
This proposal introduces new overloads of the `#expect()` and `#require()`
44+
macros that take, as arguments, a closure and a timeout value. When called,
45+
these macros will continuously evaluate the closure until either the specific
46+
condition passes, or the timeout has passed. The timeout period will default
47+
to 1 second.
48+
49+
There are 2 Polling Behaviors that we will add: Passes Once and Passes Always.
50+
Passes Once will continuously evaluate the expression until the expression
51+
returns true. If the timeout passes without the expression ever returning true,
52+
then a failure will be reported. Passes Always will continuously execute the
53+
expression until the first time expression returns false or the timeout passes.
54+
If the expression ever returns false, then a failure will be reported.
55+
56+
Tests will now be able to poll code updating in the background using
57+
either of the new overloads:
58+
59+
```swift
60+
let subject = Aquarium()
61+
Task {
62+
await subject.raiseDolphins()
63+
}
64+
await #expect(until: .passesOnce) {
65+
subject.dolphins.count() == 1
66+
}
67+
```
68+
69+
## Detailed design
70+
71+
### New expectations
72+
73+
We will introduce the following new overloads of `#expect()` and `#require()` to
74+
the testing library:
75+
76+
```swift
77+
/// Continuously check an expression until it matches the given PollingBehavior
78+
///
79+
/// - Parameters:
80+
/// - until: The desired PollingBehavior to check for.
81+
/// - timeout: How long to run poll the expression until stopping.
82+
/// - comment: A comment describing the expectation.
83+
/// - sourceLocation: The source location to which the recorded expectations
84+
/// and issues should be attributed.
85+
/// - expression: The expression to be evaluated.
86+
///
87+
/// Use this overload of `#expect()` when you wish to poll whether a value
88+
/// changes as the result of activity in another task/queue/thread.
89+
@freestanding(expression) public macro expect(
90+
until pollingBehavior: PollingBehavior,
91+
timeout: Duration = .seconds(1),
92+
_ comment: @autoclosure () -> Comment? = nil,
93+
sourceLocation: SourceLocation = #_sourceLocation,
94+
expression: @Sendable () async throws -> Bool
95+
) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro")
96+
97+
/// Continuously check an expression until it matches the given PollingBehavior
98+
///
99+
/// - Parameters:
100+
/// - until: The desired PollingBehavior to check for.
101+
/// - timeout: How long to run poll the expression until stopping.
102+
/// - comment: A comment describing the expectation.
103+
/// - sourceLocation: The source location to which the recorded expectations
104+
/// and issues should be attributed.
105+
/// - expression: The expression to be evaluated.
106+
///
107+
/// Use this overload of `#expect()` when you wish to poll whether a value
108+
/// changes as the result of activity in another task/queue/thread, and you
109+
/// expect the expression to throw an error as part of succeeding
110+
@freestanding(expression) public macro expect<E>(
111+
until pollingBehavior: PollingBehavior,
112+
throws error: E,
113+
timeout: Duration = .seconds(1),
114+
_ comment: @autoclosure () -> Comment? = nil,
115+
sourceLocation: SourceLocation = #_sourceLocation,
116+
expression: @Sendable () async throws(E) -> Bool
117+
) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro")
118+
where E: Error & Equatable
119+
120+
@freestanding(expression) public macro expect<E>(
121+
until pollingBehavior: PollingBehavior,
122+
timeout: Duration = .seconds(1),
123+
_ comment: @autoclosure () -> Comment? = nil,
124+
sourceLocation: SourceLocation = #_sourceLocation,
125+
performing expression: @Sendable () async throws(E) -> Bool,
126+
throws errorMatcher: (E) async throws -> Bool
127+
) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro")
128+
where E: Error
129+
130+
/// Continuously check an expression until it matches the given PollingBehavior
131+
///
132+
/// - Parameters:
133+
/// - until: The desired PollingBehavior to check for.
134+
/// - timeout: How long to run poll the expression until stopping.
135+
/// - comment: A comment describing the expectation.
136+
/// - sourceLocation: The source location to which the recorded expectations
137+
/// and issues should be attributed.
138+
/// - expression: The expression to be evaluated.
139+
///
140+
/// Use this overload of `#require()` when you wish to poll whether a value
141+
/// changes as the result of activity in another task/queue/thread.
142+
@freestanding(expression) public macro require(
143+
until pollingBehavior: PollingBehavior,
144+
timeout: Duration = .seconds(1),
145+
_ comment: @autoclosure () -> Comment? = nil,
146+
sourceLocation: SourceLocation = #_sourceLocation,
147+
expression: @Sendable () async throws -> Bool
148+
) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro")
149+
150+
@freestanding(expression) public macro require<R>(
151+
until pollingBehavior: PollingBehavior,
152+
timeout: Duration = .seconds(1),
153+
_ comment: @autoclosure () -> Comment? = nil,
154+
sourceLocation: SourceLocation = #_sourceLocation,
155+
expression: @Sendable () async throws -> R?
156+
) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro")
157+
where R: Sendable
158+
159+
@freestanding(expression) public macro require<E>(
160+
until pollingBehavior: PollingBehavior,
161+
throws error: E,
162+
timeout: Duration = .seconds(1),
163+
_ comment: @autoclosure () -> Comment? = nil,
164+
sourceLocation: SourceLocation = #_sourceLocation,
165+
expression: @Sendable () async throws(E) -> Bool
166+
) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro")
167+
where E: Error & Equatable
168+
169+
@freestanding(expression) public macro require<E>(
170+
until pollingBehavior: PollingBehavior,
171+
timeout: Duration = .seconds(1),
172+
_ comment: @autoclosure () -> Comment? = nil,
173+
sourceLocation: SourceLocation = #_sourceLocation,
174+
expression: @Sendable () async throws(E) -> Bool,
175+
throwing errorMatcher: (E) async throws -> Bool,
176+
) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro")
177+
where E: Error
178+
```
179+
180+
### Polling Behavior
181+
182+
A new type, `PollingBehavior`, to represent the behavior of a polling
183+
expectation:
184+
185+
```swift
186+
public enum PollingBehavior {
187+
/// Continuously evaluate the expression until the first time it returns
188+
/// true.
189+
/// If it does not pass once by the time the timeout is reached, then a
190+
/// failure will be reported.
191+
case passesOnce
192+
193+
/// Continuously evaluate the expression until the first time it returns
194+
/// false.
195+
/// If the expression returns false, then a failure will be reported.
196+
/// If the expression only returns true before the timeout is reached, then
197+
/// no failure will be reported.
198+
/// If the expression does not finish evaluating before the timeout is
199+
/// reached, then a failure will be reported.
200+
case passesAlways
201+
}
202+
```
203+
204+
### Usage
205+
206+
These macros can be used with an async test function:
207+
208+
```swift
209+
@Test func `The aquarium's dolphin nursery works`() async {
210+
let subject = Aquarium()
211+
Task {
212+
await subject.raiseDolphins()
213+
}
214+
await #expect(until: .passesOnce) {
215+
subject.dolphins.count() == 1
216+
}
217+
}
218+
```
219+
220+
With the definition of `Aquarium` above, the closure will only need to be
221+
evaluated a few times before it starts returning true. At which point the macro
222+
will end, and no failure will be reported.
223+
224+
If the expression never returns a value within the timeout period, then a
225+
failure will be reported, noting that the expression was unable to be evaluated
226+
within the timeout period:
227+
228+
```swift
229+
await #expect(until: .passesOnce, timeout: .seconds(1)) {
230+
// Failure: The expression timed out before evaluation could finish.
231+
try await Task.sleep(for: .seconds(10))
232+
}
233+
```
234+
235+
In the case of `#require` where the expression returns an optional value, under
236+
`PollingBehavior.passesOnce`, the expectation is considered to have passed the
237+
first time the expression returns a non-nil value, and that value will be
238+
returned by the expectation. Under `PollingBehavior.passesAlways`, the
239+
expectation is considered to have passed if the expression always returns a
240+
non-nil value. If it passes, the value returned by the last time the
241+
expression is evaluated will be returned by the expectation.
242+
243+
When no error is expected, then the first time the expression throws any error
244+
will cause the polling expectation to stop & report the error as a failure.
245+
246+
When an error is expected, then the expression is not considered to pass
247+
unless it throws an error that equals the expected error or returns true when
248+
evaluated by the `errorMatcher`. After which the polling continues under the
249+
specified PollingBehavior.
250+
251+
## Source compatibility
252+
253+
This is a new interface that is unlikely to collide with any existing
254+
client-provided interfaces. The typical Swift disambiguation tools can be used
255+
if needed.
256+
257+
## Integration with supporting tools
258+
259+
We will expose the polling mechanism under ForToolsIntegrationOnly spi so that
260+
tools may integrate with them.
261+
262+
## Future directions
263+
264+
The timeout default could be configured as a Suite or Test trait. Additionally,
265+
it could be configured in some future global configuration tool.
266+
267+
## Alternatives considered
268+
269+
Instead of creating the `PollingBehavior` type, we could have introduced more
270+
macros to cover that situation: `#expect(until:)` and `#expect(always:)`.
271+
However, this would have resulted in confusion for the compiler and test authors
272+
when trailing closure syntax is used.
273+
274+
## Acknowledgments
275+
276+
This proposal is heavily inspired by Nimble's [Polling Expectations](https://quick.github.io/Nimble/documentation/nimble/pollingexpectations/).
277+
In particular, thanks to [Jeff Hui](https://github.com/jeffh) for writing the
278+
original implementation of Nimble's Polling Expectations.

0 commit comments

Comments
 (0)