11# Running tests
22
3- You can run the entire test collection using ` x ` . But note that running the
4- * entire* test collection is almost never what you want to do during local
5- development because it takes a really long time. For local development, see the
6- subsection after on how to run a subset of tests.
3+ You can run the entire test collection using ` x ` .
4+ But note that running the * entire* test collection is almost never what you want to do during local
5+ development because it takes a really long time.
6+ For local development, see the subsection after on how to run a subset of tests.
77
88<div class =" warning " >
99
10- Running plain ` ./x test ` will build the stage 1 compiler and then run the whole
11- test suite. This not only includes ` tests/ ` , but also ` library/ ` , ` compiler/ ` ,
10+ Running plain ` ./x test ` will build the stage 1 compiler and then run the whole test suite.
11+ This not only includes ` tests/ ` , but also ` library/ ` , ` compiler/ ` ,
1212` src/tools/ ` package tests and more.
1313
1414You usually only want to run a subset of the test suites (or even a smaller set
15- of tests than that) which you expect will exercise your changes. PR CI exercises
16- a subset of test collections, and merge queue CI will exercise all of the test
15+ of tests than that) which you expect will exercise your changes.
16+ PR CI exercises a subset of test collections, and merge queue CI will exercise all of the test
1717collection.
1818
1919</div >
@@ -22,8 +22,8 @@ collection.
2222./x test
2323```
2424
25- The test results are cached and previously successful tests are ` ignored ` during
26- testing. The stdout/stderr contents as well as a timestamp file for every test
25+ The test results are cached and previously successful tests are ` ignored ` during testing.
26+ The stdout/stderr contents as well as a timestamp file for every test
2727can be found under ` build/<target-tuple>/test/ ` for the given
2828` <target-tuple> ` . To force-rerun a test (e.g. in case the test runner fails to
2929notice a change) you can use the ` --force-rerun ` CLI option.
@@ -40,18 +40,17 @@ notice a change) you can use the `--force-rerun` CLI option.
4040
4141## Running a subset of the test suites
4242
43- When working on a specific PR, you will usually want to run a smaller set of
44- tests. For example, a good "smoke test" that can be used after modifying rustc
43+ When working on a specific PR, you will usually want to run a smaller set of tests.
44+ For example, a good "smoke test" that can be used after modifying rustc
4545to see if things are generally working correctly would be to exercise the ` ui `
4646test suite ([ ` tests/ui ` ] ):
4747
4848``` text
4949./x test tests/ui
5050```
5151
52- Of course, the choice of test suites is
53- somewhat arbitrary, and may not suit the task you are doing. For example, if you
54- are hacking on debuginfo, you may be better off with the debuginfo test suite:
52+ Of course, the choice of test suites is somewhat arbitrary, and may not suit the task you are doing.
53+ For example, if you are hacking on debuginfo, you may be better off with the debuginfo test suite:
5554
5655``` text
5756./x test tests/debuginfo
@@ -77,8 +76,8 @@ Likewise, you can test a single file by passing its path:
7776./x test tests/ui/const-generics/const-test.rs
7877```
7978
80- ` x ` doesn't support running a single tool test by passing its path yet. You'll
81- have to use the ` --test-args ` argument as described
79+ ` x ` doesn't support running a single tool test by passing its path yet.
80+ You'll have to use the ` --test-args ` argument as described
8281[ below] ( #running-an-individual-test ) .
8382
8483``` text
@@ -97,8 +96,8 @@ have to use the `--test-args` argument as described
9796./x test --stage 0 library/std
9897```
9998
100- Note that this only runs tests on ` std ` ; if you want to test ` core ` or other
101- crates, you have to specify those explicitly.
99+ Note that this only runs tests on ` std ` ;
100+ if you want to test ` core ` or other crates, you have to specify those explicitly.
102101
103102### Run the tidy script and tests on the standard library
104103
@@ -117,8 +116,8 @@ you avoid having to run tests for components you did not change at all.
117116
118117<div class =" warning " >
119118
120- Note that bors only runs the tests with the full stage 2 build; therefore, while
121- the tests ** usually** work fine with stage 1, there are some limitations.
119+ Note that bors only runs the tests with the full stage 2 build;
120+ therefore, while the tests ** usually** work fine with stage 1, there are some limitations.
122121
123122</div >
124123
@@ -128,8 +127,8 @@ the tests **usually** work fine with stage 1, there are some limitations.
128127./x test --stage 2
129128```
130129
131- <div class =" warning " >
132- You almost never need to do this; CI will run these tests for you.
130+ <div class =" warning " > You almost never need to do this;
131+ CI will run these tests for you.
133132</div >
134133
135134## Run unit tests on the compiler/library
@@ -140,7 +139,8 @@ You may want to run unit tests on a specific file with following:
140139./x test compiler/rustc_data_structures/src/thin_vec/tests.rs
141140```
142141
143- But unfortunately, it's impossible. You should invoke the following instead:
142+ But unfortunately, it's impossible.
143+ You should invoke the following instead:
144144
145145``` text
146146./x test compiler/rustc_data_structures/ --test-args thin_vec
@@ -149,24 +149,23 @@ But unfortunately, it's impossible. You should invoke the following instead:
149149## Running an individual test
150150
151151Another common thing that people want to do is to run an ** individual test** ,
152- often the test they are trying to fix. As mentioned earlier, you may pass the
153- full file path to achieve this, or alternatively one may invoke ` x ` with the
154- ` --test-args ` option:
152+ often the test they are trying to fix.
153+ As mentioned earlier, you may pass the
154+ full file path to achieve this, or alternatively one may invoke ` x ` with the ` --test-args ` option:
155155
156156``` text
157157./x test tests/ui --test-args issue-1234
158158```
159159
160160Under the hood, the test runner invokes the standard Rust test runner (the same
161161one you get with ` #[test] ` ), so this command would wind up filtering for tests
162- that include "issue-1234" in the name. Thus, ` --test-args ` is a good way to run
163- a collection of related tests.
162+ that include "issue-1234" in the name.
163+ Thus, ` --test-args ` is a good way to run a collection of related tests.
164164
165165## Passing arguments to ` rustc ` when running tests
166166
167167It can sometimes be useful to run some tests with specific compiler arguments,
168- without using ` RUSTFLAGS ` (during development of unstable features, with ` -Z `
169- flags, for example).
168+ without using ` RUSTFLAGS ` (during development of unstable features, with ` -Z ` flags, for example).
170169
171170This can be done with ` ./x test ` 's ` --compiletest-rustc-args ` option, to pass
172171additional arguments to the compiler when building the tests.
@@ -176,8 +175,7 @@ additional arguments to the compiler when building the tests.
176175If you have changed the compiler's output intentionally, or you are making a new
177176test, you can pass ` --bless ` to the test subcommand.
178177
179- As an example,
180- if some tests in ` tests/ui ` are failing, you can run this command:
178+ As an example, if some tests in ` tests/ui ` are failing, you can run this command:
181179
182180``` text
183181./x test tests/ui --bless
@@ -192,8 +190,9 @@ just like when running the tests without the `--bless` flag.
192190There are a few options for running tests:
193191
194192* ` bootstrap.toml ` has the ` rust.verbose-tests ` option. If ` false ` , each test will
195- print a single dot (the default). If ` true ` , the name of every test will be
196- printed. This is equivalent to the ` --quiet ` option in the [ Rust test
193+ print a single dot (the default).
194+ If ` true ` , the name of every test will be printed.
195+ This is equivalent to the ` --quiet ` option in the [ Rust test
197196 harness] ( https://doc.rust-lang.org/rustc/tests/ ) .
198197* The environment variable ` RUST_TEST_THREADS ` can be set to the number of
199198 concurrent threads to use for testing.
@@ -202,24 +201,24 @@ There are a few options for running tests:
202201
203202Pass UI tests now have three modes, ` check-pass ` , ` build-pass ` and ` run-pass ` .
204203When ` --pass $mode ` is passed, these tests will be forced to run under the given
205- ` $mode ` unless the directive ` //@ ignore-pass ` exists in the test file. For
206- example, you can run all the tests in ` tests/ui ` as ` check-pass ` :
204+ ` $mode ` unless the directive ` //@ ignore-pass ` exists in the test file.
205+ For example, you can run all the tests in ` tests/ui ` as ` check-pass ` :
207206
208207``` text
209208./x test tests/ui --pass check
210209```
211210
212- By passing ` --pass $mode ` , you can reduce the testing time. For each mode,
213- please see [ Controlling pass/fail
211+ By passing ` --pass $mode ` , you can reduce the testing time.
212+ For each mode, please see [ Controlling pass/fail
214213expectations] ( ui.md#controlling-passfail-expectations ) .
215214
216215## Running tests with different "compare modes"
217216
218- UI tests may have different output depending on certain "modes" that the
219- compiler is in. For example, when using the Polonius mode, a test ` foo.rs ` will
217+ UI tests may have different output depending on certain "modes" that the compiler is in.
218+ For example, when using the Polonius mode, a test ` foo.rs ` will
220219first look for expected output in ` foo.polonius.stderr ` , falling back to the
221- usual ` foo.stderr ` if not found. The following will run the UI test suite in
222- Polonius mode:
220+ usual ` foo.stderr ` if not found.
221+ The following will run the UI test suite in Polonius mode:
223222
224223``` text
225224./x test tests/ui --compare-mode=polonius
@@ -229,26 +228,28 @@ See [Compare modes](compiletest.md#compare-modes) for more details.
229228
230229## Running tests manually
231230
232- Sometimes it's easier and faster to just run the test by hand. Most tests are
233- just ` .rs ` files, so after [ creating a rustup
231+ Sometimes it's easier and faster to just run the test by hand.
232+ Most tests are just ` .rs ` files, so after [ creating a rustup
234233toolchain] ( ../building/how-to-build-and-run.md#creating-a-rustup-toolchain ) , you
235234can do something like:
236235
237236``` text
238237rustc +stage1 tests/ui/issue-1234.rs
239238```
240239
241- This is much faster, but doesn't always work. For example, some tests include
240+ This is much faster, but doesn't always work.
241+ For example, some tests include
242242directives that specify specific compiler flags, or which rely on other crates,
243243and they may not run the same without those options.
244244
245245## Running tests on a remote machine
246246
247247Tests may be run on a remote machine (e.g. to test builds for a different
248- architecture). This is done using ` remote-test-client ` on the build machine to
248+ architecture).
249+ This is done using ` remote-test-client ` on the build machine to
249250send test programs to ` remote-test-server ` running on the remote machine.
250- ` remote-test-server ` executes the test programs and sends the results back to
251- the build machine. ` remote-test-server ` provides * unauthenticated remote code
251+ ` remote-test-server ` executes the test programs and sends the results back to the build machine.
252+ ` remote-test-server ` provides * unauthenticated remote code
252253execution* so be careful where it is used.
253254
254255To do this, first build ` remote-test-server ` for the remote machine, e.g. for
@@ -258,13 +259,12 @@ RISC-V
258259./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
259260```
260261
261- The binary will be created at
262- ` ./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server ` . Copy this
263- over to the remote machine.
262+ The binary will be created at ` ./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server ` .
263+ Copy this over to the remote machine.
264264
265265On the remote machine, run the ` remote-test-server ` with the `--bind
266- 0.0.0.0:12345` flag (and optionally ` -v` for verbose output). Output should look
267- like this:
266+ 0.0.0.0:12345` flag (and optionally ` -v` for verbose output).
267+ Output should look like this:
268268
269269``` text
270270$ ./remote-test-server -v --bind 0.0.0.0:12345
@@ -273,12 +273,13 @@ listening on 0.0.0.0:12345!
273273```
274274
275275Note that binding the server to 0.0.0.0 will allow all hosts able to reach your
276- machine to execute arbitrary code on your machine. We strongly recommend either
276+ machine to execute arbitrary code on your machine.
277+ We strongly recommend either
277278setting up a firewall to block external access to port 12345, or to use a more
278279restrictive IP address when binding.
279280
280- You can test if the ` remote-test-server ` is working by connecting to it and
281- sending ` ping\n ` . It should reply ` pong ` :
281+ You can test if the ` remote-test-server ` is working by connecting to it and sending ` ping\n ` .
282+ It should reply ` pong ` :
282283
283284``` text
284285$ nc $REMOTE_IP 12345
287288```
288289
289290To run tests using the remote runner, set the ` TEST_DEVICE_ADDR ` environment
290- variable then use ` x ` as usual. For example, to run ` ui ` tests for a RISC-V
291- machine with the IP address ` 1.2.3.4 ` use
291+ variable then use ` x ` as usual.
292+ For example, to run ` ui ` tests for a RISC-V machine with the IP address ` 1.2.3.4 ` use
292293
293294``` text
294295export TEST_DEVICE_ADDR="1.2.3.4:12345"
@@ -315,34 +316,34 @@ run "/tmp/work/test1018/a"
315316[...]
316317```
317318
318- Tests are built on the machine running ` x ` not on the remote machine. Tests
319- which fail to build unexpectedly (or ` ui ` tests producing incorrect build
319+ Tests are built on the machine running ` x ` not on the remote machine.
320+ Tests which fail to build unexpectedly (or ` ui ` tests producing incorrect build
320321output) may fail without ever running on the remote machine.
321322
322323## Testing on emulators
323324
324- Some platforms are tested via an emulator for architectures that aren't readily
325- available. For architectures where the standard library is well supported and
325+ Some platforms are tested via an emulator for architectures that aren't readily available.
326+ For architectures where the standard library is well supported and
326327the host operating system supports TCP/IP networking, see the above instructions
327328for testing on a remote machine (in this case the remote machine is emulated).
328329
329- There is also a set of tools for orchestrating running the tests within the
330- emulator. Platforms such as ` arm-android ` and ` arm-unknown-linux-gnueabihf ` are
331- set up to automatically run the tests under emulation on GitHub Actions. The
332- following will take a look at how a target's tests are run under emulation.
330+ There is also a set of tools for orchestrating running the tests within the emulator.
331+ Platforms such as ` arm-android ` and ` arm-unknown-linux-gnueabihf ` are
332+ set up to automatically run the tests under emulation on GitHub Actions.
333+ The following will take a look at how a target's tests are run under emulation.
333334
334- The Docker image for [ armhf-gnu] includes [ QEMU] to emulate the ARM CPU
335- architecture. Included in the Rust tree are the tools [ remote-test-client] and
335+ The Docker image for [ armhf-gnu] includes [ QEMU] to emulate the ARM CPU architecture.
336+ Included in the Rust tree are the tools [ remote-test-client] and
336337[ remote-test-server] which are programs for sending test programs and libraries
337- to the emulator, and running the tests within the emulator, and reading the
338- results. The Docker image is set up to launch ` remote-test-server ` and the
338+ to the emulator, and running the tests within the emulator, and reading the results.
339+ The Docker image is set up to launch ` remote-test-server ` and the
339340build tools use ` remote-test-client ` to communicate with the server to
340341coordinate running tests (see [ src/bootstrap/src/core/build_steps/test.rs] ).
341342
342- To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as
343- a "remote" machine. A curious detail here is that the network is shared between
344- the simulator instance and the host macOS, so we can use the local loopback
345- address ` 127.0.0.1 ` . Something like the following should work:
343+ To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as a "remote" machine.
344+ A curious detail here is that the network is shared between
345+ the simulator instance and the host macOS, so we can use the local loopback address ` 127.0.0.1 ` .
346+ Something like the following should work:
346347
347348``` sh
348349# Build the test server for the iOS simulator:
0 commit comments