Simplify `jemalloc` setup #146627

madsmtm · 2025-09-16T10:12:20Z

In the past, #[used] had to appear in the top-level crate to have a consistent effect on the linker. This has been fixed a while ago for ELF with the introduction of the symbols.o file in #95604, and more recently for Mach-O in #133832, which means that libraries can now implement the required workarounds themselves. This allows moving these #[used] declarations out of our main.rs.

Specifically, I have moved them into tikv-jemalloc-sys where they belong in tikv/jemallocator#109 and done the same for mimalloc in purpleprotocol/mimalloc_rust#146 (in case we want to experiment with switching to that one day).

Test with:

./x build library src/tools/rustdoc src/tools/clippy --set rust.jemalloc=true

# macOS
lldb -- ./build/host/stage1/bin/rustc -vV
(lldb) b _rjem_je_zone_register
(lldb) run
# Should breakpoint, this means that the allocator was properly linked

# Linux
lldb -- ./build/host/stage1/bin/rustc -vV
(lldb) b malloc
(lldb) run
# Should breakpoint, inspect that the `malloc` symbol comes from the `rustc` binary and not from `libc`

try-job: aarch64-gnu
try-job: dist-aarch64-linux
try-job: dist-x86_64-musl
try-job: dist-x86_64-apple
try-job: dist-aarch64-apple

madsmtm · 2025-10-25T21:52:13Z

tikv-jemalloc-sys v0.6.1 has been released with the mentioned PR, so this is now ready. I'm less certain about the Linux parts here, so please double-check that.

Also, in the second commit, I fixed librustdoc and clippy building with --features jemalloc / ./x build --set rust.jemalloc=true, unclear to me how this wasn't caught by CI? Are these built differently there?

@rustbot ready
r? compiler
@bors try

Simplify `jemalloc` setup try-job: `aarch64-gnu` try-job: `dist-aarch64-linux` try-job: `dist-x86_64-musl` try-job: `dist-x86_64-apple` try-job: `dist-aarch64-apple`

rustbot · 2025-10-25T21:52:34Z

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

The Miri subtree was changed

cc @rust-lang/miri

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

rust-bors · 2025-10-26T00:20:24Z

☀️ Try build successful (CI)
Build commit: 8d39231 (8d39231d22563844d4088a7af200b6885c749c85, parent: 79966ae420f38c5861d177356a3446023c090d6d)

compiler/rustc/src/main.rs

bors · 2025-11-03T03:58:19Z

☔ The latest upstream changes (presumably #148412) made this pull request unmergeable. Please resolve the merge conflicts.

jdonszelmann · 2025-11-10T10:49:47Z

src/librustdoc/lib.rs

 use rustc_middle::ty::TyCtxt;
 use rustc_session::config::{ErrorOutputType, RustcOptGroup, make_crate_type_option};
 use rustc_session::{EarlyDiagCtxt, getopts};
+/// See docs in https://github.com/rust-lang/rust/blob/HEAD/compiler/rustc/src/main.rs


could you also link to this PR here? It's much easier for someone to accidentally change something in rustc and this reference to go stale, while if you link to this PR the reasoning always stays clear.

r=me after that

@rustbot author

Or perhaps pin the reference to a commit?

I went with:

/// See docs in https://github.com/rust-lang/rust/blob/HEAD/compiler/rustc/src/main.rs /// and https://github.com/rust-lang/rust/pull/146627 for why we need this `use` statement.

(There wasn't really a commit I could pin it to while changing the contents in the same commit).

rustbot · 2025-11-10T10:49:53Z

Error: shortcut handler unexpectedly failed in this comment: failed to remove Label { name: "S-waiting-on-review" }

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #triagebot on Zulip.

…zelmann Simplify `jemalloc` setup In the past, `#[used]` had to appear in the top-level crate to have a consistent effect on the linker. This has been fixed a while ago for ELF with the introduction of the `symbols.o` file in rust-lang#95604, and more recently for Mach-O in rust-lang#133832, which means that libraries can now implement the required workarounds themselves. This allows moving these `#[used]` declarations out of our `main.rs`. Specifically, I have moved them into `tikv-jemalloc-sys` where they belong in tikv/jemallocator#109 and done the same for `mimalloc` in purpleprotocol/mimalloc_rust#146 (in case we want to experiment with switching to that one day). Test with: ```sh ./x build library src/tools/rustdoc src/tools/clippy --set rust.jemalloc=true # macOS lldb -- ./build/host/stage1/bin/rustc -vV (lldb) b _rjem_je_zone_register (lldb) run # Should breakpoint, this means that the allocator was properly linked # Linux lldb -- ./build/host/stage1/bin/rustc -vV (lldb) b malloc (lldb) run # Should breakpoint, inspect that the `malloc` symbol comes from the `rustc` binary and not from `libc` ``` try-job: `aarch64-gnu` try-job: `dist-aarch64-linux` try-job: `dist-x86_64-musl` try-job: `dist-x86_64-apple` try-job: `dist-aarch64-apple`

Rollup of 12 pull requests Successful merges: - #146627 (Simplify `jemalloc` setup) - #147753 (Suggest add bounding value for RangeTo) - #147974 (Improve diagnostics for buffer reuse with borrowed references) - #148080 ([rustdoc] Fix invalid jump to def macro link generation) - #148424 (bootstrap: Add snapshot tests for path-to-step handling) - #148500 (Update git index before running diff-index) - #148536 (cmse: add test for `async` and `const` functions) - #148770 (implement `feature(c_variadic_naked_functions)`) - #148819 (Remove specialized warning for removed target) - #148830 (miri subtree update) - #148833 (Update rustbook dependencies) - #148841 (Remove more `#[must_use]` from portable-simd) r? `@ghost` `@rustbot` modify labels: rollup

…zelmann Simplify `jemalloc` setup In the past, `#[used]` had to appear in the top-level crate to have a consistent effect on the linker. This has been fixed a while ago for ELF with the introduction of the `symbols.o` file in rust-lang#95604, and more recently for Mach-O in rust-lang#133832, which means that libraries can now implement the required workarounds themselves. This allows moving these `#[used]` declarations out of our `main.rs`. Specifically, I have moved them into `tikv-jemalloc-sys` where they belong in tikv/jemallocator#109 and done the same for `mimalloc` in purpleprotocol/mimalloc_rust#146 (in case we want to experiment with switching to that one day). Test with: ```sh ./x build library src/tools/rustdoc src/tools/clippy --set rust.jemalloc=true # macOS lldb -- ./build/host/stage1/bin/rustc -vV (lldb) b _rjem_je_zone_register (lldb) run # Should breakpoint, this means that the allocator was properly linked # Linux lldb -- ./build/host/stage1/bin/rustc -vV (lldb) b malloc (lldb) run # Should breakpoint, inspect that the `malloc` symbol comes from the `rustc` binary and not from `libc` ``` try-job: `aarch64-gnu` try-job: `dist-aarch64-linux` try-job: `dist-x86_64-musl` try-job: `dist-x86_64-apple` try-job: `dist-aarch64-apple`

Rollup of 16 pull requests Successful merges: - #146627 (Simplify `jemalloc` setup) - #147753 (Suggest add bounding value for RangeTo) - #147832 (rustdoc: Don't pass `RenderOptions` to `DocContext`) - #147974 (Improve diagnostics for buffer reuse with borrowed references) - #148080 ([rustdoc] Fix invalid jump to def macro link generation) - #148465 (Adjust spans into the `for` loops context before creating the new desugaring spans.) - #148500 (Update git index before running diff-index) - #148531 (rustc_target: introduce Abi, Env, Os) - #148536 (cmse: add test for `async` and `const` functions) - #148770 (implement `feature(c_variadic_naked_functions)`) - #148780 (fix filecheck typos in tests) - #148819 (Remove specialized warning for removed target) - #148830 (miri subtree update) - #148833 (Update rustbook dependencies) - #148834 (fix(rustdoc): Color doctest errors) - #148841 (Remove more `#[must_use]` from portable-simd) r? `@ghost` `@rustbot` modify labels: rollup

bors · 2025-11-12T04:45:10Z

⌛ Testing commit 65f0b7a with merge 0b329f8...

Rollup merge of #146627 - madsmtm:jemalloc-simplify, r=jdonszelmann Simplify `jemalloc` setup In the past, `#[used]` had to appear in the top-level crate to have a consistent effect on the linker. This has been fixed a while ago for ELF with the introduction of the `symbols.o` file in #95604, and more recently for Mach-O in #133832, which means that libraries can now implement the required workarounds themselves. This allows moving these `#[used]` declarations out of our `main.rs`. Specifically, I have moved them into `tikv-jemalloc-sys` where they belong in tikv/jemallocator#109 and done the same for `mimalloc` in purpleprotocol/mimalloc_rust#146 (in case we want to experiment with switching to that one day). Test with: ```sh ./x build library src/tools/rustdoc src/tools/clippy --set rust.jemalloc=true # macOS lldb -- ./build/host/stage1/bin/rustc -vV (lldb) b _rjem_je_zone_register (lldb) run # Should breakpoint, this means that the allocator was properly linked # Linux lldb -- ./build/host/stage1/bin/rustc -vV (lldb) b malloc (lldb) run # Should breakpoint, inspect that the `malloc` symbol comes from the `rustc` binary and not from `libc` ``` try-job: `aarch64-gnu` try-job: `dist-aarch64-linux` try-job: `dist-x86_64-musl` try-job: `dist-x86_64-apple` try-job: `dist-aarch64-apple`

Rollup of 16 pull requests Successful merges: - rust-lang/rust#146627 (Simplify `jemalloc` setup) - rust-lang/rust#147753 (Suggest add bounding value for RangeTo) - rust-lang/rust#147832 (rustdoc: Don't pass `RenderOptions` to `DocContext`) - rust-lang/rust#147974 (Improve diagnostics for buffer reuse with borrowed references) - rust-lang/rust#148080 ([rustdoc] Fix invalid jump to def macro link generation) - rust-lang/rust#148465 (Adjust spans into the `for` loops context before creating the new desugaring spans.) - rust-lang/rust#148500 (Update git index before running diff-index) - rust-lang/rust#148531 (rustc_target: introduce Abi, Env, Os) - rust-lang/rust#148536 (cmse: add test for `async` and `const` functions) - rust-lang/rust#148770 (implement `feature(c_variadic_naked_functions)`) - rust-lang/rust#148780 (fix filecheck typos in tests) - rust-lang/rust#148819 (Remove specialized warning for removed target) - rust-lang/rust#148830 (miri subtree update) - rust-lang/rust#148833 (Update rustbook dependencies) - rust-lang/rust#148834 (fix(rustdoc): Color doctest errors) - rust-lang/rust#148841 (Remove more `#[must_use]` from portable-simd) r? `@ghost` `@rustbot` modify labels: rollup

Zalathar · 2025-11-12T05:19:21Z

Bors, this has already been merged.

@bors r- retry

Zalathar · 2025-11-13T06:51:53Z

Perf results from rollup: Rollup of 16 pull requests #148851 (comment)
Comparison URL

Regressions in doc benchmarks across the board; this seems unintended?

RalfJung · 2025-11-13T07:35:20Z

Maybe rustdoc accidentally is not using jemalloc any more?

Kobzol · 2025-11-13T09:25:30Z

Some allocation changes have definitely happened there:

  496,710,988  ???:
  -408,481,523    realloc
   261,159,975    do_rallocx
   253,281,882    _rjem_je_arena_ralloc
  -193,667,679    tcache_bin_flush_small
   159,404,361    _rjem_je_arena_ralloc_no_move
    96,036,695    _rjem_je_tcache_bin_flush_small
    81,229,215    tcache_bin_flush_edatas_lookup
   -76,118,340    malloc_default
    67,153,310    rtree_read
    64,297,968    rtree_metadata_read
   -54,428,351    pa_alloc
    51,577,458    _rjem_je_arena_cache_bin_fill_small
    29,630,447    _rjem_je_malloc_default
    18,729,715    _rjem_je_emap_update_edata_state
    17,881,154    _rjem_je_eset_fit
    13,595,487    emap_try_acquire_edata_neighbor_impl
    11,825,510    _rjem_je_te_event_trigger
   -10,311,127    free_default
   -10,132,168    ecache_evict
     8,316,595    _rjem_je_eset_remove
     8,300,824    _rjem_je_tcache_bin_flush_stashed
     8,176,825    _rjem_je_eset_insert
     8,124,547    emap_rtree_leaf_elms_lookup
     7,930,250    base_alloc_impl
    -7,411,000    tcache_bin_flush_stashed
     7,385,380    _rjem_je_hook_invoke_dalloc
     7,385,320    _rjem_je_hook_invoke_alloc
     7,226,602    extent_recycle
    -7,008,483    eset_remove

I would suggest perf. testing any PR that touches things like memory allocation, for the next time 😅

I will post a revert, so that we can figure out what happened here.

Kobzol · 2025-11-13T09:30:39Z

#148896

… r=jdonszelmann" This reverts commit 5dc3c19, reversing changes made to 11339a0.

Revert "Rollup merge of #146627 - madsmtm:jemalloc-simplify, r=jdonszelmann"

madsmtm · 2025-11-13T20:08:38Z

Some allocation changes have definitely happened there:

[snip]

Could you tell me how you arrived at that table? I'm unfamiliar with Linux perf tooling.

I would suggest perf. testing any PR that touches things like memory allocation, for the next time 😅

Yeah, sorry about that, I should've done that!

Kobzol · 2025-11-13T20:13:46Z

Could you tell me how you arrived at that table? I'm unfamiliar with Linux perf tooling.

Yeah, sorry. If you open a benchmark row from the comparison table, it shows "Command for profiling locally", which you can copy paste in a checkout of https://github.com/rust-lang/rustc-perf to do a Cachegrind diff before/after the PR.

Revert "Rollup merge of #146627 - madsmtm:jemalloc-simplify, r=jdonszelmann" This reverts commit 5dc3c19, reversing changes made to 11339a0. Reverts #146627 due to a [perf regression](#148851 (comment)). r? `@Zalathar`

Revert "Rollup merge of #146627 - madsmtm:jemalloc-simplify, r=jdonszelmann" This reverts commit 5dc3c19417d0fbfd979c5a1ecd8bebe2d8d9110e, reversing changes made to 11339a0ef5ed586bb7ea4f85a9b7287880caac3a. Reverts rust-lang/rust#146627 due to a [perf regression](rust-lang/rust#148851 (comment)). r? `@Zalathar`

madsmtm added A-linkage Area: linking into static, shared libraries and binaries T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 16, 2025

This comment has been minimized.

Sign in to view

madsmtm added O-linux Operating system: Linux O-macos Operating system: macOS labels Sep 17, 2025

madsmtm force-pushed the jemalloc-simplify branch from 15a8fbc to 551e285 Compare October 25, 2025 21:24

This comment has been minimized.

Sign in to view

rustbot assigned jdonszelmann Oct 25, 2025

madsmtm marked this pull request as ready for review October 25, 2025 21:52

madsmtm added T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. A-allocators Area: Custom and system allocators and removed T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Oct 25, 2025

RalfJung reviewed Oct 26, 2025

View reviewed changes

compiler/rustc/src/main.rs Show resolved Hide resolved

madsmtm force-pushed the jemalloc-simplify branch from de2de0b to 2c0d266 Compare October 26, 2025 11:06

madsmtm force-pushed the jemalloc-simplify branch from 2c0d266 to aa3fd28 Compare November 3, 2025 15:15

This comment has been minimized.

Sign in to view

jdonszelmann requested changes Nov 10, 2025

View reviewed changes

Zalathar mentioned this pull request Nov 11, 2025

Rollup of 13 pull requests #148844

Closed

Zalathar mentioned this pull request Nov 11, 2025

Rollup of 12 pull requests #148845

Closed

Zalathar mentioned this pull request Nov 12, 2025

Rollup of 16 pull requests #148851

Merged

bors merged commit 5dc3c19 into rust-lang:main Nov 12, 2025
11 checks passed

rustbot added this to the 1.93.0 milestone Nov 12, 2025

madsmtm deleted the jemalloc-simplify branch November 12, 2025 20:47

Kobzol mentioned this pull request Nov 13, 2025

Revert "Rollup merge of #146627 - madsmtm:jemalloc-simplify, r=jdonszelmann" #148896

Merged

Kobzol added a commit to Kobzol/rust that referenced this pull request Nov 13, 2025

Revert "Rollup merge of rust-lang#146627 - madsmtm:jemalloc-simplify,…

ccd8795

… r=jdonszelmann" This reverts commit 5dc3c19, reversing changes made to 11339a0.

rust-bors bot added a commit that referenced this pull request Nov 13, 2025

Auto merge of #148896 - Kobzol:revert-146627, r=<try>

961b503

Revert "Rollup merge of #146627 - madsmtm:jemalloc-simplify, r=jdonszelmann"

madsmtm mentioned this pull request Nov 13, 2025

Fix performance regression with jemalloc #148925

Draft

Simplify jemalloc setup #146627

Simplify jemalloc setup #146627

Conversation

madsmtm commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

madsmtm commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

rustbot commented Oct 25, 2025

Uh oh!

rust-bors bot commented Oct 26, 2025

Uh oh!

Uh oh!

bors commented Nov 3, 2025

Uh oh!

This comment has been minimized.

jdonszelmann Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

mati865 Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

madsmtm Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustbot commented Nov 10, 2025

Uh oh!

Uh oh!

bors commented Nov 12, 2025

Uh oh!

Zalathar commented Nov 12, 2025

Uh oh!

Zalathar commented Nov 13, 2025

Uh oh!

RalfJung commented Nov 13, 2025

Uh oh!

Kobzol commented Nov 13, 2025

Uh oh!

Kobzol commented Nov 13, 2025

Uh oh!

madsmtm commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Simplify `jemalloc` setup #146627

Simplify `jemalloc` setup #146627

madsmtm commented Sep 16, 2025 •

edited

Loading

madsmtm commented Oct 25, 2025 •

edited

Loading

madsmtm Nov 11, 2025 •

edited

Loading

madsmtm commented Nov 13, 2025 •

edited

Loading