@@ -50,47 +50,6 @@ a specific test case, which is useful when developing functions.
5050$ pytest array_api_tests/test_creation_functions.py::test_zeros
5151```
5252
53- ## Releases
54-
55- The test suite has tagged releases on
56- [ GitHub] ( https://github.com/data-apis/array-api-tests/releases ) . If you run
57- the test suite in your CI system, we recommend pinning against a release tag.
58-
59- We use [ calender versioning] ( https://calver.org/ ) for the releases. You should
60- expect that any version may be "breaking" compared to the previous one, in the
61- sense that there may have been additional tests added which cause a previously
62- passing library to fail.
63-
64- For now, the test suite is
65- not installable as a Python package. You can use it by cloning the repo and
66- running ` pytest ` as described above. If it would help you to be able to
67- install it as a package, [ please let us
68- know] ( https://github.com/data-apis/array-api-tests/issues/85 ) .
69-
70- * Test suite maintainer note:* to make a release of the test suite, make an
71- annotated tag with the version:
72-
73- ```
74- git tag -a 2022.1
75- ```
76-
77- (for the message, just write something like "array-api-tests version 2022.1").
78- Be sure to use the calver version number for the tag name. Versioneer will
79- automatically set the version number of the ` array_api_tests ` package based on
80- the git tag.
81-
82- Then push the tag to GitHub
83-
84- ```
85- git push --tags origin 2022.1
86- ```
87-
88- Finally go to the [ tags page on
89- GitHub] ( https://github.com/data-apis/array-api-tests/tags ) and convert the tag
90- into a release. If you want, you can add release notes to the release page on
91- GitHub.
92-
93-
9453## What the test suite covers
9554
9655We are interested in array libraries conforming to the
@@ -147,7 +106,7 @@ of the functions and some miscellaneous things.
147106 functions interact with them correctly.
148107
149108Be aware that some aspects of the spec are impractical or impossible to actually
150- test, so they are not covered in the suite <!-- TODO: note what these are -->
109+ test, so they are not covered in the suite. <!-- TODO: note what these are -->
151110
152111## Interpreting errors
153112
@@ -172,7 +131,28 @@ behaviour different from the spec, or test something that is not documented,
172131this is a bug—please [ report such
173132issues] ( https://github.com/data-apis/array-api-tests/issues/ ) to us.
174133
175- ## Configuration
134+
135+ ## Running on CI
136+
137+ See our existing [ GitHub Actions workflow for
138+ Numpy] ( https://github.com/data-apis/array-api-tests/blob/master/.github/workflows/numpy.yml )
139+ for an example of using the test suite on CI.
140+
141+ ### Releases
142+
143+ We recommend pinning against a [ release tag] ( https://github.com/data-apis/array-api-tests/releases )
144+ when running on CI.
145+
146+ We use [ calender versioning] ( https://calver.org/ ) for the releases. You should
147+ expect that any version may be "breaking" compared to the previous one, in that
148+ new tests (or improvements to existing tests) may cause a previously passing
149+ library to fail.
150+
151+ ### Configuration
152+
153+ Use the ` --ci ` flag to run only the primary and special cases tests. You can
154+ ignore the other test cases as they are redundant for the purposes of checking
155+ compliance.
176156
177157By default, tests for the optional Array API extensions such as
178158[ ` linalg ` ] ( https://data-apis.org/array-api/latest/extensions/linear_algebra_functions.html )
@@ -183,10 +163,9 @@ The tests make heavy use
183163[ Hypothesis] ( https://hypothesis.readthedocs.io/en/latest/ ) . You can configure
184164how many examples are generated using the ` --max-examples ` flag, which defaults
185165to 100. Lower values can be useful for quick checks, and larger values should
186- result in more rigorous runs. For example, ` --max-examples 10000 ` may find bugs
187- where default runs don't, but will take a much longer time .
166+ result in more rigorous runs. For example, ` --max-examples 10_000 ` may find bugs
167+ where default runs don't but will take much longer to run .
188168
189- <!-- TODO: howto on CI -->
190169
191170## Contributing
192171
@@ -231,6 +210,31 @@ where `path/to/array-api` is the path to a local clone of the [`array-api`
231210repo] ( https://github.com/data-apis/array-api/ ) . Edit ` generate_stubs.py ` to make
232211changes to the generated files.
233212
213+
214+ ### Release
215+
216+ To make a release, first make an annotated tag with the version, e.g.:
217+
218+ ```
219+ git tag -a 2022.01.01
220+ ```
221+
222+ Be sure to use the calver version number for the tag name. Don't worry too much
223+ on the tag message, e.g. just write "2022.01.01".
224+
225+ Versioneer will automatically set the version number of the ` array_api_tests `
226+ package based on the git tag. Push the tag to GitHub:
227+
228+ ```
229+ git push --tags upstream 2022.1
230+ ```
231+
232+ Then go to the [ tags page on
233+ GitHub] ( https://github.com/data-apis/array-api-tests/tags ) and convert the tag
234+ into a release. If you want, you can add release notes, which GitHub can
235+ generate for you.
236+
237+
234238## Future plans
235239
236240Keeping full coverage of the spec is an on-going priority as the Array API
0 commit comments