Keep the tempdir in case test setup/teardown fails

When an issue occurs inside a fixture (e.g. servers fail to start/stop),
the test result won't be detected as failed, but rather an error will be
thrown.

To ensure the tempdir is kept even if the test itself passes but the
system_test() fixture throws an error, a different mechanism is needed.
At the start of the critical test setup section, note that the fixture
hasn't finished yet. When this is detected in the system_test_dir()
fixture, it is recognized as error in test setup/teardown and the temp
directory is kept.

This may seem cumbersome, because it is. It's basically a workaround for
the way pytest handles fixtures and test errors in general.

Author: nicki-krizek

bin/tests/system/conftest.py

@@ -66,6 +66,7 @@ def control_port():
 
     # ----------------------- Globals definition -----------------------------
 
+    FIXTURE_OK = pytest.StashKey[bool]()  # pylint: disable=no-member
     LOG_FORMAT = "%(asctime)s %(levelname)7s:%(name)s  %(message)s"
     XDIST_WORKER = os.environ.get("PYTEST_XDIST_WORKER", "")
     FILE_DIR = os.path.abspath(Path(__file__).parent)
@@ -313,6 +314,10 @@ def get_test_result():
                 logger.debug("--noclean requested, keeping temporary directory")
             elif result == "failed":
                 logger.debug("test failure detected, keeping temporary directory")
+            elif not request.node.stash[FIXTURE_OK]:
+                logger.debug(
+                    "test setup/teardown issue detected, keeping temporary directory"
+                )
             else:
                 logger.debug("deleting temporary directory")
                 shutil.rmtree(testdir)
@@ -453,10 +458,17 @@ def get_core_dumps():
         port = int(env["PORT"])
         logger.info("using port range: <%d, %d>", port, port + PORTS_PER_TEST - 1)
 
+        request.node.stash[FIXTURE_OK] = True
+
         # Perform checks which may skip this test.
         check_net_interfaces()
         check_prerequisites()
 
+        # Store the fact that this fixture hasn't successfully finished yet.
+        # This is checked before temporary directory teardown to decide whether
+        # it's okay to remove the directory.
+        request.node.stash[FIXTURE_OK] = False
+
         setup_test()
         try:
             start_servers()
@@ -466,3 +478,4 @@ def get_core_dumps():
             logger.debug("test(s) finished")
             stop_servers()
             get_core_dumps()
+            request.node.stash[FIXTURE_OK] = True