Rollup merge of #140044 - tshepang:rdg-push, r=jieyouxu

rustc-dev-guide subtree update

r? ``@ghost``

Author: ChrisDenton

src/doc/rustc-dev-guide/README.md

@@ -43,13 +43,13 @@ rustdocs][rustdocs].
 To build a local static HTML site, install [`mdbook`](https://github.com/rust-lang/mdBook) with:
 
 ```
-> cargo install mdbook mdbook-linkcheck2 mdbook-toc mdbook-mermaid
+cargo install mdbook mdbook-linkcheck2 mdbook-toc mdbook-mermaid
 ```
 
 and execute the following command in the root of the repository:
 
 ```
-> mdbook build --open
+mdbook build --open
 ```
 
 The build files are found in the `book/html` directory.
@@ -61,8 +61,8 @@ checking is **not** run by default locally, though it is in CI. To enable it
 locally, set the environment variable `ENABLE_LINKCHECK=1` like in the
 following example.
 
-```console
-$ ENABLE_LINKCHECK=1 mdbook serve
+```
+ENABLE_LINKCHECK=1 mdbook serve
 ```
 
 ### Table of Contents
@@ -86,14 +86,14 @@ Older versions of `josh-proxy` may not round trip commits losslessly so it is im
 1) Checkout a new branch that will be used to create a PR into `rust-lang/rustc-dev-guide`
 2) Run the pull command
     ```
-    $ cargo run --manifest-path josh-sync/Cargo.toml rustc-pull
+    cargo run --manifest-path josh-sync/Cargo.toml rustc-pull
     ```
 3) Push the branch to your fork and create a PR into `rustc-dev-guide`
 
 ### Push changes from this repository into `rust-lang/rust`
 1) Run the push command to create a branch named `<branch-name>` in a `rustc` fork under the `<gh-username>` account
     ```
-    $ cargo run --manifest-path josh-sync/Cargo.toml rustc-push <branch-name> <gh-username>
+    cargo run --manifest-path josh-sync/Cargo.toml rustc-push <branch-name> <gh-username>
     ```
 2) Create a PR from `<branch-name>` into `rust-lang/rust`
 
@@ -106,5 +106,5 @@ You may observe "Nothing to pull" even if you *know* rustc-pull has something to
 To minimize the likelihood of this happening, you may wish to keep a separate *minimal* git config that *only* has `[user]` entries from global git config, then repoint system git to use the minimal git config instead. E.g.
 
 ```
-$ GIT_CONFIG_GLOBAL=/path/to/minimal/gitconfig GIT_CONFIG_SYSTEM='' cargo +stable run --manifest-path josh-sync/Cargo.toml -- rustc-pull
+GIT_CONFIG_GLOBAL=/path/to/minimal/gitconfig GIT_CONFIG_SYSTEM='' cargo +stable run --manifest-path josh-sync/Cargo.toml -- rustc-pull
 ```

src/doc/rustc-dev-guide/rust-version

@@ -1 +1 @@
-25a615bf829b9f6d6f22da537e3851043f92e5f2
+a7c39b68616668a45f0afd62849a1da7c8ad2516

src/doc/rustc-dev-guide/src/SUMMARY.md

@@ -10,9 +10,9 @@
 - [How to build and run the compiler](./building/how-to-build-and-run.md)
     - [Quickstart](./building/quickstart.md)
     - [Prerequisites](./building/prerequisites.md)
-    - [Suggested Workflows](./building/suggested.md)
+    - [Suggested workflows](./building/suggested.md)
     - [Distribution artifacts](./building/build-install-distribution-artifacts.md)
-    - [Building Documentation](./building/compiler-documenting.md)
+    - [Building documentation](./building/compiler-documenting.md)
     - [Rustdoc overview](./rustdoc.md)
     - [Adding a new target](./building/new-target.md)
     - [Optimized build](./building/optimized-build.md)
@@ -42,11 +42,11 @@
     - [with the linux perf tool](./profiling/with_perf.md)
     - [with Windows Performance Analyzer](./profiling/wpa_profiling.md)
     - [with the Rust benchmark suite](./profiling/with_rustc_perf.md)
-- [crates.io Dependencies](./crates-io.md)
+- [crates.io dependencies](./crates-io.md)
 
 # Contributing to Rust
 
-- [Contribution Procedures](./contributing.md)
+- [Contribution procedures](./contributing.md)
 - [About the compiler team](./compiler-team.md)
 - [Using Git](./git.md)
 - [Mastering @rustbot](./rustbot.md)
@@ -56,7 +56,7 @@
 - [Stabilizing Features](./stabilization_guide.md)
 - [Feature Gates](./feature-gates.md)
 - [Coding conventions](./conventions.md)
-- [Procedures for Breaking Changes](./bug-fix-procedure.md)
+- [Procedures for breaking changes](./bug-fix-procedure.md)
 - [Using external repositories](./external-repos.md)
 - [Fuzzing](./fuzzing.md)
 - [Notification groups](notification-groups/about.md)
@@ -81,36 +81,43 @@
 - [How Bootstrap does it](./building/bootstrapping/how-bootstrap-does-it.md)
 - [Writing tools in Bootstrap](./building/bootstrapping/writing-tools-in-bootstrap.md)
 - [Debugging bootstrap](./building/bootstrapping/debugging-bootstrap.md)
+- [cfg(bootstrap) in dependencies](./building/bootstrapping/bootstrap-in-dependencies.md)
 
 # High-level Compiler Architecture
 
 - [Prologue](./part-2-intro.md)
 - [Overview of the compiler](./overview.md)
 - [The compiler source code](./compiler-src.md)
 - [Queries: demand-driven compilation](./query.md)
-    - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md)
+    - [The Query Evaluation Model in detail](./queries/query-evaluation-model-in-detail.md)
     - [Incremental compilation](./queries/incremental-compilation.md)
-    - [Incremental compilation In Detail](./queries/incremental-compilation-in-detail.md)
-    - [Debugging and Testing](./incrcomp-debugging.md)
+    - [Incremental compilation in detail](./queries/incremental-compilation-in-detail.md)
+    - [Debugging and testing](./incrcomp-debugging.md)
     - [Salsa](./queries/salsa.md)
-- [Memory Management in Rustc](./memory.md)
-- [Serialization in Rustc](./serialization.md)
-- [Parallel Compilation](./parallel-rustc.md)
+- [Memory management in rustc](./memory.md)
+- [Serialization in rustc](./serialization.md)
+- [Parallel compilation](./parallel-rustc.md)
 - [Rustdoc internals](./rustdoc-internals.md)
     - [Search](./rustdoc-internals/search.md)
 	- [The `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md)
+- [Autodiff internals](./autodiff/internals.md)
+    - [Installation](./autodiff/installation.md)
+    - [How to debug](./autodiff/debugging.md)
+    - [Autodiff flags](./autodiff/flags.md)
+    - [Current limitations](./autodiff/limitations.md)
+
 # Source Code Representation
 
 - [Prologue](./part-3-intro.md)
 - [Syntax and the AST](./syntax-intro.md)
-    - [Lexing and Parsing](./the-parser.md)
+    - [Lexing and parsing](./the-parser.md)
     - [Macro expansion](./macro-expansion.md)
     - [Name resolution](./name-resolution.md)
     - [Attributes](./attributes.md)
-    - [`#[test]` Implementation](./test-implementation.md)
-    - [Panic Implementation](./panic-implementation.md)
-    - [AST Validation](./ast-validation.md)
-    - [Feature Gate Checking](./feature-gate-ck.md)
+    - [`#[test]` implementation](./test-implementation.md)
+    - [Panic implementation](./panic-implementation.md)
+    - [AST validation](./ast-validation.md)
+    - [Feature gate checking](./feature-gate-ck.md)
     - [Lang Items](./lang-items.md)
 - [The HIR (High-level IR)](./hir.md)
     - [Lowering AST to HIR](./ast-lowering.md)
@@ -129,7 +136,7 @@
     - [Example: Type checking](./rustc-driver/interacting-with-the-ast.md)
     - [Example: Getting diagnostics](./rustc-driver/getting-diagnostics.md)
     - [Remarks on perma-unstable features](./rustc-driver/remarks-on-perma-unstable-features.md)
-- [Errors and Lints](diagnostics.md)
+- [Errors and lints](diagnostics.md)
     - [Diagnostic and subdiagnostic structs](./diagnostics/diagnostic-structs.md)
     - [Translation](./diagnostics/translation.md)
     - [`LintStore`](./diagnostics/lintstore.md)
@@ -175,14 +182,14 @@
 - [Type checking](./type-checking.md)
     - [Method Lookup](./method-lookup.md)
     - [Variance](./variance.md)
-    - [Coherence Checking](./coherence.md)
-    - [Opaque Types](./opaque-types-type-alias-impl-trait.md)
+    - [Coherence checking](./coherence.md)
+    - [Opaque types](./opaque-types-type-alias-impl-trait.md)
         - [Inference details](./opaque-types-impl-trait-inference.md)
         - [Return Position Impl Trait In Trait](./return-position-impl-trait-in-trait.md)
         - [Region inference restrictions][opaque-infer]
 - [Const condition checking](./effects.md)
 - [Pattern and Exhaustiveness Checking](./pat-exhaustive-checking.md)
-- [Unsafety Checking](./unsafety-checking.md)
+- [Unsafety checking](./unsafety-checking.md)
 - [MIR dataflow](./mir/dataflow.md)
 - [Drop elaboration](./mir/drop-elaboration.md)
 - [The borrow checker](./borrow_check.md)

src/doc/rustc-dev-guide/src/about-this-guide.md

@@ -3,33 +3,41 @@
 This guide is meant to help document how rustc – the Rust compiler – works,
 as well as to help new contributors get involved in rustc development.
 
-There are seven parts to this guide:
+There are several parts to this guide:
 
-1. [Building `rustc`][p1]:
+1. [Building and debugging `rustc`][p1]:
    Contains information that should be useful no matter how you are contributing,
    about building, debugging, profiling, etc.
-2. [Contributing to `rustc`][p2]:
+1. [Contributing to Rust][p2]:
    Contains information that should be useful no matter how you are contributing,
    about procedures for contribution, using git and Github, stabilizing features, etc.
-3. [High-Level Compiler Architecture][p3]:
+1. [Bootstrapping][p3]:
+   Describes how the Rust compiler builds itself using previous versions, including
+   an introduction to the bootstrap process and debugging methods.
+1. [High-level Compiler Architecture][p4]:
    Discusses the high-level architecture of the compiler and stages of the compile process.
-4. [Source Code Representation][p4]:
+1. [Source Code Representation][p5]:
    Describes the process of taking raw source code from the user
    and transforming it into various forms that the compiler can work with easily.
-5. [Analysis][p5]:
-   discusses the analyses that the compiler uses to check various properties of the code
+1. [Supporting Infrastructure][p6]:
+   Covers command-line argument conventions, compiler entry points like rustc_driver and
+   rustc_interface, and the design and implementation of errors and lints.
+1. [Analysis][p7]:
+   Discusses the analyses that the compiler uses to check various properties of the code
    and inform later stages of the compile process (e.g., type checking).
-6. [From MIR to Binaries][p6]: How linked executable machine code is generated.
-7. [Appendices][p7] at the end with useful reference information.
+1. [MIR to Binaries][p8]: How linked executable machine code is generated.
+1. [Appendices][p9] at the end with useful reference information.
    There are a few of these with different information, including a glossary.
 
 [p1]: ./building/how-to-build-and-run.html
 [p2]: ./contributing.md
-[p3]: ./part-2-intro.md
-[p4]: ./part-3-intro.md
-[p5]: ./part-4-intro.md
-[p6]: ./part-5-intro.md
-[p7]: ./appendix/background.md
+[p3]: ./building/bootstrapping/intro.md
+[p4]: ./part-2-intro.md
+[p5]: ./part-3-intro.md
+[p6]: ./cli.md
+[p7]: ./part-4-intro.md
+[p8]: ./part-5-intro.md
+[p9]: ./appendix/background.md
 
 ### Constant change
 

src/doc/rustc-dev-guide/src/ast-validation.md

@@ -1,4 +1,4 @@
-# AST Validation
+# AST validation
 
 _AST validation_ is a separate AST pass that visits each
 item in the tree and performs simple checks. This pass

src/doc/rustc-dev-guide/src/autodiff/debugging.md

@@ -0,0 +1,113 @@
+# Reporting backend crashes
+
+If after a compilation failure you are greeted by a large amount of llvm-ir code, then our enzyme backend likely failed to compile your code. These cases are harder to debug, so your help is highly appreciated. Please also keep in mind that release builds are usually much more likely to work at the moment.
+
+The final goal here is to reproduce your bug in the enzyme [compiler explorer](https://enzyme.mit.edu/explorer/), in order to create a bug report in the [Enzyme](https://github.com/enzymead/enzyme/issues) repository.
+
+We have an `autodiff` flag which you can pass to `rustflags` to help with this. it will print the whole llvm-ir module, along with some `__enzyme_fwddiff` or `__enzyme_autodiff` calls. A potential workflow on linux could look like:  
+
+## Controlling llvm-ir generation
+
+Before generating the llvm-ir, keep in mind two techniques that can help ensure the relevant rust code is visible for debugging:
+
+- **`std::hint::black_box`**: wrap rust variables or expressions in `std::hint::black_box()` to prevent rust and llvm from optimizing them away. This is useful when you need to inspect or manually manipulate specific values in the llvm-ir.
+- **`extern "rust"` or `extern "c"`**: if you want to see how a specific function declaration is lowered to llvm-ir, you can declare it as `extern "rust"` or `extern "c"`. You can also look for existing `__enzyme_autodiff` or similar declarations within the generated module for examples.
+
+## 1) Generate an llvm-ir reproducer
+
+```sh
+rustflags="-z autodiff=enable,printmodbefore" cargo +enzyme build --release &> out.ll 
+```
+
+This also captures a few warnings and info messages above and below your module. open out.ll and remove every line above `; moduleid = <somehash>`. Now look at the end of the file and remove everything that's not part of llvm-ir, i.e. remove errors and warnings. The last line of your llvm-ir should now start with `!<somenumber> = `, i.e. `!40831 = !{i32 0, i32 1037508, i32 1037538, i32 1037559}` or `!43760 = !dilocation(line: 297, column: 5, scope: !43746)`.
+
+The actual numbers will depend on your code.  
+
+## 2) Check your llvm-ir reproducer
+
+To confirm that your previous step worked, we will use llvm's `opt` tool. find your path to the opt binary, with a path similar to `<some_dir>/rust/build/<x86/arm/...-target-tripple>/build/bin/opt`. also find `llvmenzyme-19.<so/dll/dylib>` path, similar to `/rust/build/target-tripple/enzyme/build/enzyme/llvmenzyme-19`. Please keep in mind that llvm frequently updates it's llvm backend, so the version number might be higher (20, 21, ...). Once you have both, run the following command:
+
+```sh
+<path/to/opt> out.ll -load-pass-plugin=/path/to/llvmenzyme-19.so -passes="enzyme" -s
+```
+
+If the previous step succeeded, you are going to see the same error that you saw when compiling your rust code with cargo. 
+
+If you fail to get the same error, please open an issue in the rust repository. If you succeed, congrats! the file is still huge, so let's automatically minimize it.
+
+## 3) Minimize your llvm-ir reproducer
+
+First find your `llvm-extract` binary, it's in the same folder as your opt binary. then run:
+
+```sh
+<path/to/llvm-extract> -s --func=<name> --recursive --rfunc="enzyme_autodiff*" --rfunc="enzyme_fwddiff*" --rfunc=<fnc_called_by_enzyme> out.ll -o mwe.ll 
+```
+
+This command creates `mwe.ll`, a minimal working example.
+
+Please adjust the name passed with the last `--func` flag. You can either apply the `#[no_mangle]` attribute to the function you differentiate, then you can replace it with the rust name. otherwise you will need to look up the mangled function name. To do that, open `out.ll` and search for `__enzyme_fwddiff` or `__enzyme_autodiff`. the first string in that function call is the name of your function. example:
+
+```llvm-ir 
+define double @enzyme_opt_helper_0(ptr %0, i64 %1, double %2) {
+  %4 = call double (...) @__enzyme_fwddiff(ptr @_zn2ad3_f217h3b3b1800bd39fde3e, metadata !"enzyme_const", ptr %0, metadata !"enzyme_const", i64 %1, metadata !"enzyme_dup", double %2, double %2)
+  ret double %4
+}
+```
+
+Here, `_zn2ad3_f217h3b3b1800bd39fde3e` is the correct name. make sure to not copy the leading `@`. redo step 2) by running the `opt` command again, but this time passing `mwe.ll` as the input file instead of `out.ll`. Check if this minimized example still reproduces the crash.
+
+## 4) (Optional) Minimize your llvm-ir reproducer further.
+
+After the previous step you should have an `mwe.ll` file with ~5k loc. let's try to get it down to 50. find your `llvm-reduce` binary next to `opt` and `llvm-extract`. Copy the first line of your error message, an example could be:
+
+```sh
+opt: /home/manuel/prog/rust/src/llvm-project/llvm/lib/ir/instructions.cpp:686: void llvm::callinst::init(llvm::functiontype*, llvm::value*, llvm::arrayref<llvm::value*>, llvm::arrayref<llvm::operandbundledeft<llvm::value*> >, const llvm::twine&): assertion `(args.size() == fty->getnumparams() || (fty->isvararg() && args.size() > fty->getnumparams())) && "calling a function with bad signature!"' failed.
+```
+
+If you just get a `segfault` there is no sensible error message and not much to do automatically, so continue to 5).  
+otherwise, create a `script.sh` file containing
+
+```sh
+#!/bin/bash
+<path/to/your/opt> $1 -load-pass-plugin=/path/to/llvmenzyme-19.so -passes="enzyme" \
+    |& grep "/some/path.cpp:686: void llvm::callinst::init"
+```
+
+Experiment a bit with which error message you pass to grep. it should be long enough to make sure that the error is unique. However, for longer errors including `(` or `)` you will need to escape them correctly which can become annoying. Run
+
+```sh 
+<path/to/llvm-reduce> --test=script.sh mwe.ll 
+```
+
+If you see `input isn't interesting! verify interesting-ness test`, you got the error message in script.sh wrong, you need to make sure that grep matches your actual error. If all works out, you will see a lot of iterations, ending with a new `reduced.ll` file. Verify with `opt` that you still get the same error.
+
+### Advanced debugging: manual llvm-ir investigation
+
+Once you have a minimized reproducer (`mwe.ll` or `reduced.ll`), you can delve deeper:
+
+- **manual editing:** try manually rewriting the llvm-ir. for certain issues, like those involving indirect calls, you might investigate enzyme-specific intrinsics like `__enzyme_virtualreverse`. Understanding how to use these might require consulting enzyme's documentation or source code.
+- **enzyme test cases:** look for relevant test cases within the [enzyme repository](https://github.com/enzymead/enzyme/tree/main/enzyme/test) that might demonstrate the correct usage of features or intrinsics related to your problem.
+
+## 5) Report your bug.
+
+Afterwards, you should be able to copy and paste your `mwe.ll` (or `reduced.ll`) example into our [compiler explorer](https://enzyme.mit.edu/explorer/).
+
+- Select `llvm ir` as language and `opt 20` as compiler.
+- Replace the field to the right of your compiler with `-passes="enzyme"`, if it is not already set.
+- Hopefully, you will see once again your now familiar error.
+- Please use the share button to copy links to them.
+- Please create an issue on [https://github.com/enzymead/enzyme/issues](https://github.com/enzymead/enzyme/issues) and share `mwe.ll` and (if you have it) `reduced.ll`, as well as links to the compiler explorer. Please feel free to also add your rust code or a link to it.
+
+#### Documenting findings
+
+some enzyme errors, like `"attempting to call an indirect active function whose runtime value is inactive"`, have historically caused confusion. If you investigate such an issue, even if you don't find a complete solution, please consider documenting your findings. If the insights are general to enzyme and not specific to its rust usage, contributing them to the main [enzyme documentation](https://github.com/enzymead/www) is often the best first step. You can also mention your findings in the relevant enzyme github issue or propose updates to these docs if appropriate. This helps prevent others from starting from scratch.
+
+With a clear reproducer and documentation, hopefully an enzyme developer will be able to fix your bug. Once that happens, the enzyme submodule inside the rust compiler will be updated, which should allow you to differentiate your rust code. Thanks for helping us to improve rust-ad.
+
+# Minimize rust code
+
+Beyond having a minimal llvm-ir reproducer, it is also helpful to have a minimal rust reproducer without dependencies. This allows us to add it as a test case to ci once we fix it, which avoids regressions for the future.
+
+There are a few solutions to help you with minimizing the rust reproducer. This is probably the most simple automated approach: [cargo-minimize](https://github.com/nilstrieb/cargo-minimize).
+
+Otherwise we have various alternatives, including [`treereduce`](https://github.com/langston-barrett/treereduce), [`halfempty`](https://github.com/googleprojectzero/halfempty), or [`picireny`](https://github.com/renatahodovan/picireny), potentially also [`creduce`](https://github.com/csmith-project/creduce).