Failed testing proc-macros built with `-Zbuild-std`

[madsmtm]

Problem

When testing proc-macros with -Zbuild-std, the binary under test fails to load the standard library (at least on macOS):

     Running unittests src/lib.rs (target/debug/deps/foo_proc_macro-0dc0d3e74fa08def)
dyld[99947]: Library not loaded: @rpath/libstd-7cbc1cc35ff1ddd0.dylib
  Referenced from: <E6DCD012-647E-354E-B422-B62E4D01C5BF> target/debug/deps/foo_proc_macro-0dc0d3e74fa08def
  Reason: tried: '$(pwd)/target/debug/deps/libstd-7cbc1cc35ff1ddd0.dylib' (no such file), '$(pwd)/target/debug/libstd-7cbc1cc35ff1ddd0.dylib' (no such file), '$HOME/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/libstd-7cbc1cc35ff1ddd0.dylib' (no such file), '$HOME/lib/libstd-7cbc1cc35ff1ddd0.dylib' (no such file), '/usr/local/lib/libstd-7cbc1cc35ff1ddd0.dylib' (no such file), '/usr/lib/libstd-7cbc1cc35ff1ddd0.dylib' (no such file, not in dyld cache)
error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `$(pwd)/target/debug/deps/foo_proc_macro-0dc0d3e74fa08def` (signal: 6, SIGABRT: process abort signal)

Steps

# Create basic proc-macro
cargo new foo-proc-macro --lib
cd foo-proc-macro/
echo "\n[lib]\nproc-macro = true" >> Cargo.toml
echo > src/lib.rs

# Test it with -Zbuild-std
cargo +nightly test --lib -Zbuild-std --target=aarch64-apple-darwin

Notes

This started happening since rust-lang/rust#131188, because now $HOME/.rustup/toolchains/$TOOLCHAIN/lib (which cargo is adding to the DYLD_FALLBACK_LIBRARY_PATH/LD_LIBRARY_PATH) no longer includes libstd-*.dylib, that is only present in $HOME/.rustup/toolchains/$TOOLCHAIN-$HOST/lib/rustlib/$HOST/lib (which Cargo normally adds, but doesn't add when using -Zbuild-std).

This was found in sonos/dinghy#238.

Possible Solution(s)

Not sure, perhaps the code that sets DYLD_FALLBACK_LIBRARY_PATH/LD_LIBRARY_PATH

cargo/src/cargo/core/compiler/compilation.rs

Lines 293 to 339 in 80d82ca

	let mut search_path = Vec::new();
	if tool_kind.is_rustc_tool() {
	if matches!(tool_kind, ToolKind::Rustdoc) {
	// HACK: `rustdoc --test` not only compiles but executes doctests.
	// Ideally only execution phase should have search paths appended,
	// so the executions can find native libs just like other tests.
	// However, there is no way to separate these two phase, so this
	// hack is added for both phases.
	// TODO: handle doctest-xcompile
	search_path.extend(super::filter_dynamic_search_path(
	self.native_dirs.iter(),
	&self.root_output[&CompileKind::Host],
	));
	}
	search_path.push(self.deps_output[&CompileKind::Host].clone());
	} else {
	search_path.extend(super::filter_dynamic_search_path(
	self.native_dirs.iter(),
	&self.root_output[&kind],
	));
	search_path.push(self.deps_output[&kind].clone());
	search_path.push(self.root_output[&kind].clone());
	// For build-std, we don't want to accidentally pull in any shared
	// libs from the sysroot that ships with rustc. This may not be
	// required (at least I cannot craft a situation where it
	// matters), but is here to be safe.
	if self.gctx.cli_unstable().build_std.is_none() {
	search_path.push(self.sysroot_target_libdir[&kind].clone());
	}
	}
	let dylib_path = paths::dylib_path();
	let dylib_path_is_empty = dylib_path.is_empty();
	search_path.extend(dylib_path.into_iter());
	if cfg!(target_os = "macos") && dylib_path_is_empty {
	// These are the defaults when DYLD_FALLBACK_LIBRARY_PATH isn't
	// set or set to an empty string. Since Cargo is explicitly setting
	// the value, make sure the defaults still work.
	if let Some(home) = self.gctx.get_env_os("HOME") {
	search_path.push(PathBuf::from(home).join("lib"));
	}
	search_path.push(PathBuf::from("/usr/local/lib"));
	search_path.push(PathBuf::from("/usr/lib"));
	}
	let search_path = paths::join_paths(&search_path, paths::dylib_path_envvar())?;
	cmd.env(paths::dylib_path_envvar(), &search_path);

needs to be adjusted? Or perhaps different handling when testing proc macros is needed altogether?

Version

cargo 1.84.0-nightly (cf53cc5 2024-10-18)
release: 1.84.0-nightly
commit-hash: cf53cc5
commit-date: 2024-10-18
host: aarch64-apple-darwin
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.7.1 (sys:0.4.74+curl-8.9.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w 11 Sep 2023
os: Mac OS 14.7.0 [64-bit]

[fredszaq]

I can confirm that this also happens on linux, looks like it started with the 2024-10-06 nightly

$ cargo +nightly-2024-10-05 test -Zbuild-std --target=x86_64-unknown-linux-gnu 
   Compiling my-proc-macro v0.1.0 (/tmp/my-proc-macro)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.25s
     Running unittests src/lib.rs (target/debug/deps/my_proc_macro-7e0727f90075e76f)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests my_proc_macro

running 0 tests

$ cargo +nightly-2024-10-06 test -Zbuild-std --target=x86_64-unknown-linux-gnu
   Compiling my-proc-macro v0.1.0 (/tmp/my-proc-macro)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.40s
     Running unittests src/lib.rs (target/debug/deps/my_proc_macro-7e0727f90075e76f)
/tmp/my-proc-macro/target/debug/deps/my_proc_macro-7e0727f90075e76f: error while loading shared libraries: libstd-f157c25fb2dbfbe0.so: cannot open shared object file: No such file or directory
error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `/tmp/my-proc-macro/target/debug/deps/my_proc_macro-7e0727f90075e76f` (exit status: 127)
note: test exited abnormally; to see the full output pass --nocapture to the harness.