Merge branch 'master' into bugfix/perflog-compat-headers

Author: vkarak

docs/config_reference.rst

@@ -115,7 +115,7 @@ System Configuration
    :required: Yes
 
    A list of hostname regular expression patterns in Python `syntax <https://docs.python.org/3.8/library/re.html>`__, which will be used by the framework in order to automatically select a system configuration.
-   For the auto-selection process, see `here <configure.html#picking-a-system-configuration>`__.
+   For the auto-selection process, check the configuration of the :attr:`~config.autodetect_methods` option.
 
 .. py:attribute:: systems.max_local_jobs
 
@@ -244,6 +244,8 @@ System Configuration
       This option is broken in 4.0.
 
 
+.. _system-partition-configuration:
+
 System Partition Configuration
 ==============================
 
@@ -470,7 +472,7 @@ System Partition Configuration
       .. versionadded:: 4.0.0
 
         ReFrame also allows you to register your own custom launchers simply by defining them in the configuration.
-        You can follow a small tutorial `here <tutorial_advanced.html#adding-a-custom-launcher-to-a-partition>`__.
+        You can follow a small tutorial :ref:`here <custom-launchers>`.
 
 
 .. py:attribute:: systems.partitions.access
@@ -550,7 +552,7 @@ System Partition Configuration
    :default: ``8``
 
    The maximum number of concurrent regression tests that may be active (i.e., not completed) on this partition.
-   This option is relevant only when ReFrame executes with the `asynchronous execution policy <pipeline.html#execution-policies>`__.
+   This option is relevant only when ReFrame executes with the :ref:`asynchronous execution policy <execution-policies>`.
 
 
 .. py:attribute:: systems.partitions.prepare_cmds
@@ -568,7 +570,7 @@ System Partition Configuration
    :required: No
    :default: ``[]``
 
-   A list of job scheduler `resource specification <config_reference.html#custom-job-scheduler-resources>`__ objects.
+   A list of job scheduler `resource specification <#custom-job-scheduler-resources>`__ objects.
 
 
 .. py:attribute:: systems.partitions.processor
@@ -976,7 +978,7 @@ They are associated with `system partitions <#system-partition-configuration>`__
    :default: ``{}``
 
    Scheduler resources associated with this environments.
-   
+
    This is the equivalent of a test's :attr:`~reframe.core.pipeline.RegressionTest.extra_resources`.
 
    .. versionadded:: 4.6
@@ -1231,6 +1233,8 @@ All logging handlers share the following set of common attributes:
    In addition to the format directives supported by the standard library's `time.strftime() <https://docs.python.org/3.8/library/time.html#time.strftime>`__ function, ReFrame allows you to use the ``%:z`` directive -- a GNU ``date`` extension --  that will print the time zone difference in a RFC3339 compliant way, i.e., ``+/-HH:MM`` instead of ``+/-HHMM``.
 
 
+.. _file-handler:
+
 The ``file`` log handler
 ------------------------
 

docs/deferrables.rst

@@ -3,7 +3,7 @@ Understanding the Mechanism of Deferrable Functions
 ===================================================
 
 This section describes the mechanism behind deferrable functions, which in ReFrame, they are used for sanity and performance checking.
-Generally, writing a new sanity function in a :class:`~reframe.core.pipeline.RegressionTest` is as straightforward as decorating a simple member function with the built-in :func:`~reframe.core.pipeline.RegressionMixin.sanity_function` decorator.
+Generally, writing a new sanity function in a :class:`~reframe.core.pipeline.RegressionTest` is as straightforward as decorating a simple member function with the built-in :func:`~reframe.core.builtins.sanity_function` decorator.
 Behind the scenes, this decorator will convert the Python function into a deferrable function and schedule its evaluation for the sanity stage of the test.
 However, when dealing with more complex scenarios such as a deferrable function taking as an argument the results from other deferrable functions, it is crucial to understand how a deferrable function differs from a regular Python function, and when is it actually evaluated.
 
@@ -181,7 +181,7 @@ There are some exceptions to this rule, though.
 The logical :keyword:`and`, :keyword:`or` and :keyword:`not` operators as well as the :keyword:`in` operator cannot be deferred automatically.
 These operators try to take the truthy value of their arguments by calling :func:`bool <python:bool>` on them.
 As we shall see later, applying the :func:`bool <python:bool>` function on a deferred expression causes its immediate evaluation and returns the result.
-If you want to defer the execution of such operators, you should use the corresponding :func:`and_ <reframe.utility.sanity.and_>`, :func:`or_ <reframe.utility.sanity.or_>`, :func:`not_ <reframe.utility.sanity.not_>` and :func:`contains <reframe.utility.sanity.contains>` functions in :mod:`reframe.utility.sanity`, which basically wrap the expression in a deferrable function.
+If you want to defer the execution of such operators, you should use the corresponding :func:`~reframe.utility.sanity.and_`, :func:`~reframe.utility.sanity.or_`, :func:`~reframe.utility.sanity.not_` and :func:`~reframe.utility.sanity.contains` functions in :mod:`reframe.utility.sanity`, which basically wrap the expression in a deferrable function.
 
 In summary deferrable functions have the following characteristics:
 

docs/dependencies.rst

@@ -2,7 +2,7 @@
 How Test Dependencies Work In ReFrame
 =====================================
 
-Dependencies in ReFrame are defined at the test level using the :func:`depends_on` function, but are projected to the `test cases <pipeline.html>`__ space.
+Dependencies in ReFrame are defined at the test level using the :func:`depends_on` function, but are projected to the :doc:`test cases <pipeline>` space.
 We will see the rules of that projection in a while.
 The dependency graph construction and the subsequent dependency analysis happen also at the level of the test cases.
 

docs/howto.rst

@@ -25,10 +25,10 @@ Finally, if none of the above build systems fits, users are allowed to use their
 Using Make, CMake or Autotools
 ------------------------------
 
-We have seen already in the `tutorial <tutorial.html#compiling-the-test-code>`__ how to build a test with a single source file.
+We have seen already in the :ref:`tutorial <compiling-the-test-code>` how to build a test with a single source file.
 ReFrame can also build test code using common build systems, such as `Make <https://www.gnu.org/software/make/>`__, `CMake <https://cmake.org/>`__ or `Autotools <https://www.gnu.org/software/automake/>`__.
 The build system to be used is selected by the :attr:`build_system` test attribute.
-This is a "magic" attribute where you assign it a string and ReFrame will create the appropriate `build system object <regression_test_api.html#build-systems>`__.
+This is a "magic" attribute where you assign it a string and ReFrame will create the appropriate :ref:`build system object <build-systems>`.
 Each build system can define its own properties, but some build systems have a common set of properties that are interpreted accordingly.
 Let's see a version of the STREAM benchmark that uses ``make``:
 
@@ -84,7 +84,7 @@ In this case, only the flags requested by the test will be emitted.
 
 The Autotools and CMake build systems are quite similar.
 For passing ``configure`` options, the :attr:`~reframe.core.buildsystems.ConfigureBasedBuildSystem.config_opts` build system attribute should be set, whereas for ``make`` options the :attr:`~reframe.core.buildsystems.ConfigureBasedBuildSystem.make_opts` should be used.
-The `OSU benchmarks <tutorial.html#multi-node-tests>`__ in the main tutorial use the Autotools build system.
+The :ref:`OSU benchmarks <multi-node-tests>` in the main tutorial use the Autotools build system.
 
 Finally, in all three build systems, the :attr:`~reframe.core.buildsystems.Make.max_concurrency` can be set to control the number of parallel make jobs.
 
@@ -118,7 +118,7 @@ For running this test, we need the following Docker image:
    docker run -h myhost --mount type=bind,source=$(pwd)/examples/,target=/home/user/reframe-examples --workdir=/home/user/reframe-examples/tutorial -it reframe-eb-spack:latest  /bin/bash -l
 
 
-EasyBuild requires a `modules system <#working-with-environment-modules>`__ to run, so we need a configuration file that sets the modules system of the current system:
+EasyBuild requires a :ref:`modules system <working-with-environment-modules>` to run, so we need a configuration file that sets the modules system of the current system:
 
 .. literalinclude:: ../examples/tutorial/config/baseline_modules.py
    :caption:
@@ -404,10 +404,10 @@ If you used an environment module to load ReFrame, e.g., ``reframe``, you can us
 Working with low-level dependencies
 ===================================
 
-We have seen that `test fixtures <tutorial.html#test-fixtures>`__ fixtures introduce dependencies between tests along with a scope.
+We have seen that :ref:`test fixtures <test-fixtures>` fixtures introduce dependencies between tests along with a scope.
 It is possible to define test dependencies without a scope using the low-level test dependency API.
 In fact, test fixtures translate to that low-level API.
-In this how-to, we will rewrite the `OSU benchmarks example <tutorial.html#multi-node-tests>`__ of the main tutorial to use the low-level dependency API.
+In this how-to, we will rewrite the :ref:`OSU benchmarks example <multi-node-tests>` of the main tutorial to use the low-level dependency API.
 
 Here is the full code:
 
@@ -560,7 +560,7 @@ The following is an example of ``.gitlab-ci.yml`` file that does exactly that:
 
 It defines two stages.
 The first one, called ``generate``, will call ReFrame to generate the pipeline specification for the desired tests.
-All the usual `test selection options <manpage.html#test-filtering>`__ can be used to select specific tests.
+All the usual :ref:`test selection options <test-filtering>` can be used to select specific tests.
 ReFrame will process them as usual, but instead of running the selected tests, it will generate the correct steps for running each test individually as a Gitlab job in a child pipeline.
 The generated ReFrame command that will run each individual test reuses the :option:`-C`, :option:`-R`, :option:`-v` and :option:`--mode` options passed to the initial invocation of ReFrame that was used to generate the pipeline.
 Users can define CI-specific execution modes in their configuration in order to pass arbitrary options to the ReFrame invocation in the child pipeline.
@@ -569,7 +569,7 @@ Finally, we pass the generated CI pipeline file to the second phase as an artifa
 If ``image`` keyword is defined in ``.gitlab-ci.yml``, the emitted pipeline will use the same image as the one defined in the parent pipeline.
 Besides, each job in the generated pipeline will output a separate junit report which can be used to create GitLab badges.
 
-The following figure shows one part of the automatically generated pipeline for the test graph depicted `above <#fig-deps-complex>`__.
+The following figure shows one part of an automatically generated pipeline.
 
 .. figure:: _static/img/gitlab-ci.png
    :align: center
@@ -1036,7 +1036,7 @@ Rerunning with increased verbosity as the message suggests will give a full trac
 Debugging sanity and performance patterns
 -----------------------------------------
 
-When creating a new test that requires a complex output parsing for the sanity checking or for extracting the figures of merit, tuning the functions decorated by :attr:`@sanity_function<reframe.core.pipeline.RegressionMixin.sanity_function>` or :attr:`@performance_function<reframe.core.pipeline.RegressionMixin.performance_function>` may involve some trial and error to debug the complex regular expressions required.
+When creating a new test that requires a complex output parsing for the sanity checking or for extracting the figures of merit, tuning the functions decorated by :attr:`@sanity_function<reframe.core.builtins.sanity_function>` or :attr:`@performance_function<reframe.core.builtins.performance_function>` may involve some trial and error to debug the complex regular expressions required.
 For lightweight tests which execute in few seconds, this trial and error may not be an issue at all.
 However, when dealing with tests which take longer to run, this method can quickly become tedious and inefficient.
 
@@ -1097,7 +1097,7 @@ After loading the configuration, ReFrame will print out its relevant environment
 Before attempting to load a file, it will validate it and check if it looks like a ReFrame test.
 If it does, it will load that file by importing it.
 This is where any ReFrame tests are instantiated and initialized (see ``Loaded 3 test(s)``), as well as the actual test cases (combination of tests, system partitions and environments) are generated.
-Then the test cases are filtered based on the various `filtering command line options <manpage.html#test-filtering>`__ as well as the programming environments that are defined for the currently selected system.
+Then the test cases are filtered based on the various :ref:`filtering command line options <test-filtering>` as well as the programming environments that are defined for the currently selected system.
 Finally, the test case dependency graph is built and everything is ready for running (or listing).
 
 Try passing a specific system or partition with the :option:`--system` option or modify the test (e.g., removing the decorator that registers it) and see how the logs change.
@@ -1124,7 +1124,7 @@ The following is the actual implementation of the ``mpirun`` launcher in ReFrame
 
 Each launcher must derive from the abstract base class :class:`~reframe.core.launchers.JobLauncher` ands needs to implement the :func:`~reframe.core.launchers.JobLauncher.command` method and, optionally, change the default :func:`~reframe.core.launchers.JobLauncher.run_command` method.
 
-The :func:`~reframe.core.launchers.JobLauncher.command` returns a list of command tokens that will be combined with any user-supplied `options <regression_test_api.html#reframe.core.launchers.JobLauncher.options>`__ by the :func:`~reframe.core.launchers.JobLauncher.run_command` method to generate the actual launcher command line.
+The :func:`~reframe.core.launchers.JobLauncher.command` returns a list of command tokens that will be combined with any user-supplied job launcher :attr:`~reframe.core.launchers.JobLauncher.options` by the :func:`~reframe.core.launchers.JobLauncher.run_command` method to generate the actual launcher command line.
 Notice you can use the ``job`` argument to get job-specific information that will allow you to construct the correct launcher invocation.
 
 If you use a Python-based configuration file, you can define your custom launcher directly inside your config as follows:

docs/manpage.rst

@@ -12,7 +12,7 @@ Synopsis
 Description
 -----------
 
-ReFrame provides both a `programming interface <regression_test_api.html>`__ for writing regression tests and a command-line interface for managing and running the tests, which is detailed here.
+ReFrame provides both a :doc:`programming interface <regression_test_api>` for writing regression tests and a command-line interface for managing and running the tests, which is detailed here.
 The ``reframe`` command is part of ReFrame's frontend.
 This frontend is responsible for loading and running regression tests written in ReFrame.
 ReFrame executes tests by sending them down to a well defined pipeline.
@@ -385,7 +385,7 @@ Options controlling ReFrame output
 
    Directory prefix for logging performance data.
 
-   This option is relevant only to the ``filelog`` `logging handler <config_reference.html#the-filelog-log-handler>`__.
+   This option is relevant only to the ``filelog`` :ref:`logging handler <filelog-handler>`.
 
    This option can also be set using the :envvar:`RFM_PERFLOG_DIR` environment variable or the :attr:`~config.logging.handlers_perflog..filelog..basedir` logging handler configuration parameter.
 
@@ -441,7 +441,7 @@ Options controlling ReFrame output
 
    Save ReFrame log files in the output directory before exiting.
 
-   Only log files generated by ``file`` `log handlers <config_reference.html#the-file-log-handler>`__ will be copied.
+   Only log files generated by ``file`` :ref:`log handlers <file-handler>` will be copied.
 
    This option can also be set using the :envvar:`RFM_SAVE_LOG_FILES` environment variable or the :attr:`~config.general.save_log_files` general configuration parameter.
 
@@ -1175,7 +1175,7 @@ Notice that this example leads to a name conflict with the old naming scheme, si
 
 Each test is also associated with a hash code that is derived from the test name, its parameters and their values.
 As in the example listing above, the hash code of each test is printed with the :option:`-l` option and individual tests can be selected by their hash using the :option:`-n` option, e.g., ``-n /1c51609b``.
-The stage and output directories, as well as the performance log file of the ``filelog`` `performance log handler <config_reference.html#the-filelog-log-handler>`__ will use the hash code for the test-specific directories and files.
+The stage and output directories, as well as the performance log file of the ``filelog`` :ref:`performance log handler <filelog-handler>` will use the hash code for the test-specific directories and files.
 This might lead to conflicts for tests as the one above when executing them with the asynchronous execution policy, but ensures consistency of performance record files when parameter values are added to or deleted from a test parameter.
 More specifically, the test's hash will not change if a new parameter value is added or deleted or even if the parameter values are shuffled.
 Test variants on the other side are more volatile and can change with such changes.
@@ -1843,13 +1843,10 @@ If no configuration file can be found in any of the predefined locations, ReFram
 This configuration file is located in |reframe/core/settings.py|_.
 Users may *not* modify this file.
 
-For a complete reference of the configuration, please refer to |reframe.settings(8)|_ man page.
+For a complete reference of the configuration, please refer to :doc:`reframe.settings(8) <config_reference>` man page.
 
 .. |reframe/core/settings.py| replace:: ``reframe/core/settings.py``
 .. _reframe/core/settings.py: https://github.com/reframe-hpc/reframe/blob/master/reframe/core/settings.py
-.. |reframe.settings(8)| replace:: ``reframe.settings(8)``
-.. _reframe.settings(8): config_reference.html
-
 
 Reporting Bugs
 --------------