Enable ThreadSanitizer across the entire multi-threaded JIT pipeline

[jserv]

ThreadSanitizer (TSAN) can now detect race conditions across the entire multi-threaded JIT pipeline with full 4GB address space emulation. This enables testing of the tier-2 LLVM compilation thread while maintaining production memory layout.

Memory Layout (TSAN-compatible):

• Main memory: MAP_FIXED at 0x7d0000000000 (4GB)
• JIT buffer: MAP_FIXED at 0x7d1000000000
• Both allocations within TSAN app range (0x7cf-0x7ff trillion)
• Prevents conflicts with TSAN shadow memory (0x02a-0x7ce trillion)

ASLR Mitigation:

• Added setarch -R wrapper for TSAN test execution
• Disables ASLR to prevent random allocations in shadow memory
• Only affects test runs, not production builds

SDL Conflict Resolution:

• SDL (uninstrumented system library) creates threads TSAN cannot track
• Disabled SDL when TSAN enabled to focus on built-in race detection
• Production builds still fully support SDL

---

Summary by cubic

Enables ThreadSanitizer across the multi-threaded JIT pipeline (including T2C) with FULL4G emulation on x86-64 and ARM64, and adds JIT debug checks to catch regalloc and cache coherency issues early.

• New Features

  • TSAN mode: -fsanitize=thread; fixed FULL4G/JIT mappings (x86_64: 0x7d0000000000/0x7d1000000000, arm64: 0x150000000000/0x151000000000 + MAP_JIT); tests run via setarch -R; SDL and LTO disabled only under TSAN; quieter TSAN defaults.
  • JIT debug mode (ENABLE_JIT_DEBUG=1) with regmap conflict checks and cache coherency verification; wired into CI and .ci/jit-debug-test.sh.

• Bug Fixes

  • T2C races: atomic load/store for hot2/compiled/n_invoke; enqueue uses cache key (PC|SATP), condition variable for wakeups, SATP verification; proper signaling on enqueue and shutdown.
  • JIT stability: fixed icache coherency (write-protect ordering and invalidation) and regalloc conflicts in load ops; only execute translated blocks on success.
  • User mode: auto-enable ELF_LOADER when SYSTEM=0 (except arch tests) to fix failing runs.
  • Build/CI: remove unconditional DTB deps to avoid dtc errors; ensure DTB is built for SYSTEM builds (including emscripten and macOS/arm64) and filter DTB files from linker; add apt-get update retries on arm64 runners; fix emcc builds by running plain distclean to clear stale .config.

^{Written for commit 874ba07. Summary will update automatically on new commits.}

[jserv]

Benchmarks

Benchmark suite	Current: 874ba07	Previous: 7711e50	Ratio
Dhrystone	1315 Average DMIPS over 10 runs	1290 Average DMIPS over 10 runs	0.98
Coremark	902.988 Average iterations/sec over 10 runs	915.002 Average iterations/sec over 10 runs	1.01

This comment was automatically generated by workflow using github-action-benchmark.

[cubic-dev-ai]

3 issues found across 13 files

Prompt for AI agents (all 3 issues)

Understand the root cause of the following 3 issues and fix them.


<file name=".github/workflows/main.yml">

<violation number="1" location=".github/workflows/main.yml:308">
Piping `apt update` through `tee` hides the command’s non-zero exit status, so cases like signature failures or apt lock conflicts skip retries and we continue with a broken package index. Please preserve the real exit code (e.g., enable `set -o pipefail` or capture `${PIPESTATUS[0]}` after the pipeline) before deciding to break.</violation>

<violation number="2" location=".github/workflows/main.yml:310">
The failure filter ignores `InRelease`, so an `apt update` error like `Failed to fetch … InRelease` is treated as success and we skip the remaining retries. Please include `InRelease` in the pattern so we keep retrying on those critical errors.</violation>
</file>

<file name="src/main.c">

<violation number="1" location="src/main.c:31">
The new TSAN options block is guarded by __SANITIZE_THREAD__ only, so clang TSAN builds skip __tsan_default_options entirely and never apply the intended runtime configuration.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

2 issues found across 13 files

Prompt for AI agents (all 2 issues)

Understand the root cause of the following 2 issues and fix them.


<file name="Makefile">

<violation number="1" location="Makefile:109">
`setarch` is Linux-only; with TSAN enabled on macOS the test recipe expands to `setarch $(uname -m) -R …` and fails because the binary is missing. Please gate the wrapper by host OS or provide a macOS-safe alternative so TSAN checks remain usable cross-platform.</violation>
</file>

<file name=".github/workflows/main.yml">

<violation number="1" location=".github/workflows/main.yml:310">
`apt update` reports release failures as `Failed to fetch …/InRelease`, but this regex only looks for Packages/Sources/Release. As a result the retry loop treats an `InRelease` failure as success and never retries, undermining the fix and leaving the job without refreshed indices. Please include `InRelease` in the match so real release failures still trigger retries.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

setarch is Linux-only; with TSAN enabled on macOS the test recipe expands to setarch $(uname -m) -R … and fails because the binary is missing. Please gate the wrapper by host OS or provide a macOS-safe alternative so TSAN checks remain usable cross-platform.

Prompt for AI agents

Address the following comment on Makefile at line 109:

<comment>`setarch` is Linux-only; with TSAN enabled on macOS the test recipe expands to `setarch $(uname -m) -R …` and fails because the binary is missing. Please gate the wrapper by host OS or provide a macOS-safe alternative so TSAN checks remain usable cross-platform.</comment>

<file context>
@@ -80,6 +73,44 @@ endif
+override ENABLE_LTO := 0       # LTO interferes with TSAN instrumentation
+CFLAGS += -DTSAN_ENABLED       # Signal code to use TSAN-compatible allocations
+# Disable ASLR for TSAN tests to prevent allocations in TSAN shadow memory
+BIN_WRAPPER = setarch $(shell uname -m) -R
+else
+BIN_WRAPPER =
</file context>

[jserv]

@cubic-dev-ai The review finding about MAP_JIT is incorrect.

Verification: The code at src/jit.c:2473-2477 (x86_64 TSAN path) already includes MAP_JIT:

state->buf = mmap(jit_addr, size, PROT_READ | PROT_WRITE | PROT_EXEC,
                  MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED
#if defined(__APPLE__)
                      | MAP_JIT  // ← Present on line 2476
#endif
                  ,
                  -1, 0);

All three mmap paths include MAP_JIT on macOS:

1. x86_64 TSAN path (line 2476): ✅ Has MAP_JIT
2. ARM64 TSAN path (line 2492): ✅ Has MAP_JIT
3. Standard path (line 2506): ✅ Has MAP_JIT

The MAP_JIT flag was added in commit f1b685e ("Add Arm64 TSAN support and fix JIT cache coherency") and is present in the current PR.

Please re-scan or verify against the actual code at commit 7711e50.

[cubic-dev-ai]

2 issues found across 13 files

Prompt for AI agents (all 2 issues)

Understand the root cause of the following 2 issues and fix them.


<file name="src/main.c">

<violation number="1" location="src/main.c:307">
Raising the default log level to LOG_INFO suppresses the rv_log_trace() stream that powers the -t instruction trace (rv_log_set_level(attr-&gt;log_level) now filters those messages), so trace mode stops emitting output.</violation>
</file>

<file name="Makefile">

<violation number="1" location="Makefile:109">
TSAN runs now always prepend `setarch`, but macOS does not ship this binary; enabling TSAN on mac hosts will make `make check` fail before the emulator starts. Please gate the wrapper to Linux or gracefully skip it when `setarch` is unavailable.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

TSAN runs now always prepend setarch, but macOS does not ship this binary; enabling TSAN on mac hosts will make make check fail before the emulator starts. Please gate the wrapper to Linux or gracefully skip it when setarch is unavailable.

Prompt for AI agents

Address the following comment on Makefile at line 109:

<comment>TSAN runs now always prepend `setarch`, but macOS does not ship this binary; enabling TSAN on mac hosts will make `make check` fail before the emulator starts. Please gate the wrapper to Linux or gracefully skip it when `setarch` is unavailable.</comment>

<file context>
@@ -80,6 +73,44 @@ endif
+override ENABLE_LTO := 0       # LTO interferes with TSAN instrumentation
+CFLAGS += -DTSAN_ENABLED       # Signal code to use TSAN-compatible allocations
+# Disable ASLR for TSAN tests to prevent allocations in TSAN shadow memory
+BIN_WRAPPER = setarch $(shell uname -m) -R
+else
+BIN_WRAPPER =
</file context>

Suggested change

	BIN_WRAPPER = setarch $(shell uname -m) -R
	BIN_WRAPPER = $(if $(filter Linux,$(UNAME_S)),setarch $(shell uname -m) -R,)