[libc++][hardening] In production hardening modes, trap rather than abort (#78561)

In the hardening modes that can be used in production (`fast` and
`extensive`), make a failed assertion invoke a trap instruction rather
than calling verbose abort. In the debug mode, still keep calling
verbose abort to provide a better user experience and to allow us to
keep our existing testing infrastructure for verifying assertion
messages. Since the debug mode by definition enables all assertions, we
can be sure that we still check all the assertion messages in the
library when running the test suite in the debug mode.

The main motivation to use trapping in production is to achieve better
code generation and reduce the binary size penalty. This way, the
assertion handler can compile to a single instruction, whereas the
existing mechanism with verbose abort results in generating a function
call that in general cannot be optimized away (made worse by the fact
that it's a variadic function, imposing an additional penalty). See the
[RFC](https://discourse.llvm.org/t/rfc-hardening-in-libc/73925) for more
details. Note that this mechanism can now be completely [overridden at
CMake configuration
time](#77883).

This patch also significantly refactors `check_assertion.h` and expands
its test coverage. The main changes:
- when overriding `verbose_abort`, don't do matching inside the function
-- just print the error message to `stderr`. This removes the need to
set a global matcher and allows to do matching in the parent process
after the child finishes;
- remove unused logic for matching source locations and for using
wildcards;
- make matchers simple functors;
- introduce `DeathTestResult` that keeps data about the test run,
primarily to make it easier to test.

In addition to the refactoring, `check_assertion.h` can now recognize
when a process exits due to a trap.

Author: var-const

libcxx/docs/BuildingLibcxx.rst

@@ -485,28 +485,25 @@ LLVM-specific options
 .. _assertion-handler:
 
 Overriding the default assertion handler
-==========================================
-
-When the library wants to terminate due to an unforeseen condition (such as
-a hardening assertion failure), the program is aborted through a special verbose
-termination function. The library provides a default function that prints an
-error message and calls ``std::abort()``. Note that this function is provided by
-the static or shared library, so it is only available when deploying to
-a platform where the compiled library is sufficiently recent. On older
-platforms, the program will terminate in an unspecified unsuccessful manner, but
-the quality of diagnostics won't be great.
-
-However, vendors can also override that mechanism at CMake configuration time.
-When a hardening assertion fails, the library invokes the
-``_LIBCPP_ASSERTION_HANDLER`` macro. A vendor may provide a header that contains
-a custom definition of this macro and specify the path to the header via the
-``LIBCXX_ASSERTION_HANDLER_FILE`` CMake variable. If provided, this header will
-be included by the library and replace the default implementation. The header
-must not include any standard library headers (directly or transitively) because
-doing so will almost always create a circular dependency. The
-``_LIBCPP_ASSERTION_HANDLER(message)`` macro takes a single parameter that
-contains an error message explaining the hardening failure and some details
-about the source location that triggered it.
+========================================
+
+When the library wants to terminate due to a hardening assertion failure, the
+program is aborted by invoking a trap instruction (or in debug mode, by
+a special verbose termination function that prints an error message and calls
+``std::abort()``). This is done to minimize the code size impact of enabling
+hardening in the library. However, vendors can also override that mechanism at
+CMake configuration time.
+
+Under the hood, a hardening assertion will invoke the
+``_LIBCPP_ASSERTION_HANDLER`` macro upon failure. A vendor may provide a header
+that contains a custom definition of this macro and specify the path to the
+header via the ``LIBCXX_ASSERTION_HANDLER_FILE`` CMake variable. If provided,
+this header will be included by the library and replace the default
+implementation. The header must not include any standard library headers
+(directly or transitively) because doing so will almost always create a circular
+dependency. The ``_LIBCPP_ASSERTION_HANDLER(message)`` macro takes a single
+parameter that contains an error message explaining the hardening failure and
+some details about the source location that triggered it.
 
 When a hardening assertion fails, it means that the program is about to invoke
 library undefined behavior. For this reason, the custom assertion handler is

libcxx/docs/ReleaseNotes/18.rst

@@ -107,13 +107,18 @@ Deprecations and Removals
   macro is provided to restore the previous behavior, and it will be supported in the LLVM 18 release only.
   In LLVM 19 and beyond, ``_LIBCPP_ENABLE_NARROWING_CONVERSIONS_IN_VARIANT`` will not be honored anymore.
 
-- The only supported way to customize the assertion handler that gets invoked when a hardening assertion fails
-  is now by setting the ``LIBCXX_ASSERTION_HANDLER_FILE`` CMake variable and providing a custom header. See
-  the documentation on overriding the default assertion handler for details.
+- Overriding `__libcpp_verbose_abort` no longer has any effect on library assertions. The only supported way
+  to customize the assertion handler that gets invoked when a hardening assertion fails is now by setting the
+  ``LIBCXX_ASSERTION_HANDLER_FILE`` CMake variable and providing a custom header. See the documentation on
+  overriding the default assertion handler for details. The ability to override `__libcpp_verbose_abort` will
+  be removed in an upcoming release in favor of the new overriding mechanism.
+
+- In safe mode (which is now equivalent to the ``extensive`` hardening mode), a failed assertion will now
+  generate a trap rather than a call to verbose abort.
 
 - The ``_LIBCPP_AVAILABILITY_CUSTOM_VERBOSE_ABORT_PROVIDED`` macro is not honored anymore in LLVM 18.
-  Please see the updated documentation about the hardening modes in libc++ and in particular the
-  ``_LIBCPP_VERBOSE_ABORT`` macro for details.
+  Please see the updated documentation about the hardening modes in libc++ and in particular on
+  overriding the default assertion handler.
 
 - The headers ``<experimental/deque>``, ``<experimental/forward_list>``, ``<experimental/list>``,
   ``<experimental/map>``, ``<experimental/memory_resource>``, ``<experimental/regex>``, ``<experimental/set>``,
@@ -136,13 +141,15 @@ Deprecations and Removals
 Upcoming Deprecations and Removals
 ----------------------------------
 
+- The ability to override ``__libcpp_verbose_abort`` will be removed in an upcoming release.
+
 LLVM 19
 ~~~~~~~
 
 - The ``LIBCXX_ENABLE_ASSERTIONS`` CMake variable that was used to enable the safe mode will be deprecated and setting
   it will trigger an error; use the ``LIBCXX_HARDENING_MODE`` variable with the value ``extensive`` instead. Similarly,
-  the ``_LIBCPP_ENABLE_ASSERTIONS`` macro will be deprecated (setting it to ``1`` still enables the extensive mode the
-  LLVM 19 release while also issuing a deprecation warning). See :ref:`the hardening documentation
+  the ``_LIBCPP_ENABLE_ASSERTIONS`` macro will be deprecated (setting it to ``1`` still enables the extensive mode in
+  the LLVM 19 release while also issuing a deprecation warning). See :ref:`the hardening documentation
   <using-hardening-modes>` for more details.
 
 - The base template for ``std::char_traits`` has been marked as deprecated and will be removed in LLVM 19. If you

libcxx/test/libcxx/algorithms/alg.sorting/assert.min.max.pass.cpp

@@ -11,7 +11,7 @@
 // REQUIRES: has-unix-headers
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 // UNSUPPORTED: libcpp-hardening-mode=none
-// XFAIL: availability-verbose_abort-missing
+// XFAIL: libcpp-hardening-mode=debug && availability-verbose_abort-missing
 
 #include <algorithm>
 #include <array>

libcxx/test/libcxx/algorithms/alg.sorting/assert.sort.invalid_comparator.pass.cpp

@@ -11,7 +11,7 @@
 // REQUIRES: has-unix-headers
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 // UNSUPPORTED: !libcpp-hardening-mode=debug
-// XFAIL: availability-verbose_abort-missing
+// XFAIL: libcpp-hardening-mode=debug && availability-verbose_abort-missing
 // ADDITIONAL_COMPILE_FLAGS: -D_LIBCPP_DEBUG_STRICT_WEAK_ORDERING_CHECK
 // When the debug mode is enabled, this test fails because we actually catch on the fly that the comparator is not
 // a strict-weak ordering before we catch that we'd dereference out-of-bounds inside std::sort, which leads to different

libcxx/test/libcxx/assertions/customize_verbose_abort.compile-time.pass.cpp

@@ -22,6 +22,6 @@ void my_abort(char const*, ...) {
 }
 
 int main(int, char**) {
-  _LIBCPP_ASSERT(false, "message");
+  _LIBCPP_VERBOSE_ABORT("%s", "message");
   return EXIT_FAILURE;
 }

libcxx/test/libcxx/assertions/customize_verbose_abort.link-time.pass.cpp

@@ -19,6 +19,6 @@ void std::__libcpp_verbose_abort(char const*, ...) {
 }
 
 int main(int, char**) {
-  _LIBCPP_ASSERT(false, "message");
+  std::__libcpp_verbose_abort("%s", "message");
   return EXIT_FAILURE;
 }

libcxx/test/libcxx/assertions/default_verbose_abort.pass.cpp

@@ -7,7 +7,9 @@
 //===----------------------------------------------------------------------===//
 
 // Test that the default verbose termination function aborts the program.
+// XFAIL: availability-verbose_abort-missing
 
+#include <__verbose_abort>
 #include <csignal>
 #include <cstdlib>
 
@@ -19,6 +21,6 @@ void signal_handler(int signal) {
 
 int main(int, char**) {
   if (std::signal(SIGABRT, signal_handler) != SIG_ERR)
-    _LIBCPP_ASSERT(false, "foo");
+    std::__libcpp_verbose_abort("%s", "foo");
   return EXIT_FAILURE;
 }

libcxx/test/libcxx/assertions/modes/enabling_assertions_enables_extensive_mode.pass.cpp

@@ -12,8 +12,9 @@
 
 // `check_assertion.h` is only available starting from C++11 and requires Unix headers and regex support.
 // UNSUPPORTED: c++03, !has-unix-headers, no-localization
-// The ability to set a custom abort message is required to compare the assertion message.
-// XFAIL: availability-verbose_abort-missing
+// The ability to set a custom abort message is required to compare the assertion message (which only happens in the
+// debug mode).
+// XFAIL: libcpp-hardening-mode=debug && availability-verbose_abort-missing
 // Note that GCC doesn't support `-Wno-macro-redefined`.
 // ADDITIONAL_COMPILE_FLAGS: -U_LIBCPP_HARDENING_MODE -D_LIBCPP_ENABLE_ASSERTIONS=1
 

libcxx/test/libcxx/assertions/modes/extensive.pass.cpp

@@ -14,7 +14,6 @@
 // UNSUPPORTED: c++03
 // `check_assertion.h` requires Unix headers.
 // REQUIRES: has-unix-headers
-// XFAIL: availability-verbose_abort-missing
 
 #include <cassert>
 #include "check_assertion.h"

libcxx/test/libcxx/assertions/modes/fast.pass.cpp

@@ -14,7 +14,6 @@
 // UNSUPPORTED: c++03
 // `check_assertion.h` requires Unix headers.
 // REQUIRES: has-unix-headers
-// XFAIL: availability-verbose_abort-missing
 
 #include <cassert>
 #include "check_assertion.h"