Auto merge of #2634 - RalfJung:miri-script, r=oli-obk

./miri updates: toolchain, rustc-push, rustc-pull

This merges the `./rustup-toolchain` script into `./miri` as `./miri toolchain`, and adds two new commands for josh-based syncing.

r? `@rust-lang/miri`

Author: bors

src/tools/miri/.github/workflows/ci.yml

@@ -67,9 +67,9 @@ jobs:
         shell: bash
         run: |
           if [[ ${{ github.event_name }} == 'schedule' ]]; then
-            ./rustup-toolchain HEAD --host ${{ matrix.host_target }}
+            ./miri toolchain HEAD --host ${{ matrix.host_target }}
           else
-            ./rustup-toolchain "" --host ${{ matrix.host_target }}
+            ./miri toolchain "" --host ${{ matrix.host_target }}
           fi
 
       - name: Show Rust version
@@ -118,7 +118,7 @@ jobs:
       - name: Install "master" toolchain
         shell: bash
         run: |
-          ./rustup-toolchain "" -c clippy
+          ./miri toolchain
 
       - name: Show Rust version
         run: |

src/tools/miri/.gitpod.yml

@@ -4,6 +4,6 @@ tasks:
   - before: echo "..."
     init: |
       cargo install rustup-toolchain-install-master
-      ./rustup-toolchain
+      ./miri toolchain
       ./miri build
-    command: echo "Run tests with ./miri test"
+    command: echo "Run tests with ./miri test"

src/tools/miri/CONTRIBUTING.md

@@ -23,13 +23,13 @@ tested against. Other versions will likely not work. After installing
 [`rustup-toolchain-install-master`], you can run the following command to
 install that exact version of rustc as a toolchain:
 ```
-./rustup-toolchain
+./miri toolchain
 ```
 This will set up a rustup toolchain called `miri` and set it as an override for
 the current directory.
 
 You can also create a `.auto-everything` file (contents don't matter, can be empty), which
-will cause any `./miri` command to automatically call `rustup-toolchain`, `clippy` and `rustfmt`
+will cause any `./miri` command to automatically call `./miri toolchain`, `clippy` and `rustfmt`
 for you. If you don't want all of these to happen, you can add individual `.auto-toolchain`,
 `.auto-clippy` and `.auto-fmt` files respectively.
 
@@ -132,7 +132,7 @@ development version of Miri using
 and then you can use it as if it was installed by `rustup`.  Make sure you use
 the same toolchain when calling `cargo miri` that you used when installing Miri!
 Usually this means you have to write `cargo +miri miri ...` to select the `miri`
-toolchain that was installed by `./rustup-toolchain`.
+toolchain that was installed by `./miri toolchain`.
 
 There's a test for the cargo wrapper in the `test-cargo-miri` directory; run
 `./run-test.py` in there to execute it. Like `./miri test`, this respects the
@@ -217,7 +217,7 @@ for changes in rustc. In both cases, `rustc-version` needs updating.
 
 To update the `rustc-version` file and install the latest rustc, you can run:
 ```
-./rustup-toolchain HEAD
+./miri toolchain HEAD
 ```
 
 Now edit Miri until `./miri test` passes, and submit a PR. Generally, it is
@@ -293,16 +293,14 @@ cargo run --release -p josh-proxy -- --local=$(pwd)/local --remote=https://githu
 
 ### Importing changes from the rustc repo
 
+Josh needs to be running, as described above.
 We assume we start on an up-to-date master branch in the Miri repo.
 
 ```sh
-# Fetch rustc side of the history. Takes ca 5 min the first time.
-# Do NOT change that commit ID, it needs to be exactly this!
-git fetch http://localhost:8000/rust-lang/rust.git:at_commit=75dd959a3a40eb5b4574f8d2e23aa6efbeb33573[:prefix=src/tools/miri]:/src/tools/miri.git master
-# Include that history into ours.
-git merge FETCH_HEAD -m "merge rustc history"
+# Fetch and merge rustc side of the history. Takes ca 5 min the first time.
+./miri rustc-pull
 # Update toolchain reference and apply formatting.
-./rustup-toolchain HEAD && ./miri fmt
+./miri toolchain HEAD && ./miri fmt
 git commit -am "rustup"
 ```
 
@@ -313,16 +311,15 @@ needed.
 
 ### Exporting changes to the rustc repo
 
-We will use the josh proxy to push to your fork of rustc. You need to make sure
-that the master branch of your fork is up-to-date. Also make sure that there
-exists no branch called `miri` in your fork. Then run the following in the Miri
-repo, assuming we are on an up-to-date master branch:
+Josh needs to be running, as described above. We will use the josh proxy to push
+to your fork of rustc. Run the following in the Miri repo, assuming we are on an
+up-to-date master branch:
 
 ```sh
 # Push the Miri changes to your rustc fork (substitute your github handle for YOUR_NAME).
-# Do NOT change that commit ID, it needs to be exactly this!
-git push http://localhost:8000/YOUR_NAME/rust.git:at_commit=75dd959a3a40eb5b4574f8d2e23aa6efbeb33573[:prefix=src/tools/miri]:/src/tools/miri.git -o base=master HEAD:miri
+./miri rustc-push YOUR_NAME miri
 ```
 
-This will create a new branch in your fork, and the output should include a link
-to create a rustc PR that will integrate those changes into the main repository.
+This will create a new branch called 'miri' in your fork, and the output should
+include a link to create a rustc PR that will integrate those changes into the
+main repository.

src/tools/miri/README.md

@@ -419,9 +419,9 @@ Some native rustc `-Z` flags are also very relevant for Miri:
 
 Moreover, Miri recognizes some environment variables:
 
-* `MIRI_AUTO_OPS` indicates whether the automatic execution of rustfmt, clippy and rustup-toolchain
-  should be skipped. If it is set to any value, they are skipped. This is used for avoiding
-  infinite recursion in `./miri` and to allow automated IDE actions to avoid the auto ops.
+* `MIRI_AUTO_OPS` indicates whether the automatic execution of rustfmt, clippy and toolchain setup
+  should be skipped. If it is set to any value, they are skipped. This is used for avoiding infinite
+  recursion in `./miri` and to allow automated IDE actions to avoid the auto ops.
 * `MIRI_LOG`, `MIRI_BACKTRACE` control logging and backtrace printing during
   Miri executions, also [see "Testing the Miri driver" in `CONTRIBUTING.md`][testing-miri].
 * `MIRIFLAGS` (recognized by `cargo miri` and the test suite) defines extra

src/tools/miri/miri

@@ -42,6 +42,22 @@ many different seeds.
 Runs the benchmarks from bench-cargo-miri in hyperfine. hyperfine needs to be installed.
 <benches> can explicitly list the benchmarks to run; by default, all of them are run.
 
+./miri rustc-pull:
+Pull and merge Miri changes from the rustc repo.
+
+./miri rustc-push <github user> <branch>:
+Push Miri changes back to the rustc repo. This will update the 'master' branch
+in the Rust fork of the given user to upstream. It will also pull a copy of the
+rustc history into the Miri repo, unless you set the RUSTC_GIT env var to an
+existing clone of the rustc repo.
+
+./miri toolchain <commit> <flags>:
+Update and activate the rustup toolchain 'miri'. If no commit is given, updates
+to the commit given in the `rust-version` file. If the commit is `HEAD`, updates
+to the latest upstream rustc commit.
+`rustup-toolchain-install-master` must be installed for this to work. Any extra
+flags are passed to `rustup-toolchain-install-master`.
+
   ENVIRONMENT VARIABLES
 
 MIRI_SYSROOT:
@@ -52,37 +68,96 @@ Pass extra flags to all cargo invocations. (Ignored by `./miri cargo`.)
 EOF
 )
 
-## We need to know where we are.
+## We need to know which command to run and some global constants.
+COMMAND="$1"
+if [ -z "$COMMAND" ]; then
+    echo "$USAGE"
+    exit 1
+fi
+shift
 # macOS does not have a useful readlink/realpath so we have to use Python instead...
 MIRIDIR=$(python3 -c 'import os, sys; print(os.path.dirname(os.path.realpath(sys.argv[1])))' "$0")
+# Used for rustc syncs.
+JOSH_FILTER=":at_commit=75dd959a3a40eb5b4574f8d2e23aa6efbeb33573[:prefix=src/tools/miri]:/src/tools/miri"
 
-## Run the auto-things.
-if [ -z "$MIRI_AUTO_OPS" ]; then
-    export MIRI_AUTO_OPS=42
-
-    # Run this first, so that the toolchain doesn't change after
-    # other code has run.
-    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-toolchain" ] ; then
-        (cd "$MIRIDIR" && ./rustup-toolchain)
+## Early commands, that don't do auto-things and don't want the environment-altering things happening below.
+case "$COMMAND" in
+toolchain)
+    cd "$MIRIDIR"
+    # Make sure rustup-toolchain-install-master is installed.
+    if ! which rustup-toolchain-install-master >/dev/null; then
+        echo "Please install rustup-toolchain-install-master by running 'cargo install rustup-toolchain-install-master'"
+        exit 1
     fi
-
-    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-fmt" ] ; then
-        $0 fmt
+    # Determine new commit.
+    if [[ "$1" == "" ]]; then
+        NEW_COMMIT=$(cat rust-version)
+    elif [[ "$1" == "HEAD" ]]; then
+        NEW_COMMIT=$(git ls-remote https://github.com/rust-lang/rust/ HEAD | cut -f 1)
+    else
+        NEW_COMMIT="$1"
     fi
-
-    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-clippy" ] ; then
-        $0 clippy -- -D warnings
+    echo "$NEW_COMMIT" > rust-version
+    shift || true # don't fail if shifting fails because no commit was given
+    # Check if we already are at that commit.
+    CUR_COMMIT=$(rustc +miri --version -v 2>/dev/null | grep "^commit-hash: " | cut -d " " -f 2)
+    if [[ "$CUR_COMMIT" == "$NEW_COMMIT" ]]; then
+        echo "miri toolchain is already at commit $CUR_COMMIT."
+        rustup override set miri
+        exit 0
     fi
-fi
-
-## Determine command and toolchain.
-COMMAND="$1"
-[ $# -gt 0 ] && shift
-# Doing this *after* auto-toolchain logic above, since that might change the toolchain.
-TOOLCHAIN=$(cd "$MIRIDIR"; rustup show active-toolchain | head -n 1 | cut -d ' ' -f 1)
-
-## Handle some commands early, since they should *not* alter the environment.
-case "$COMMAND" in
+    # Install and setup new toolchain.
+    rustup toolchain uninstall miri
+    rustup-toolchain-install-master -n miri -c cargo -c rust-src -c rustc-dev -c llvm-tools -c rustfmt -c clippy "$@" -- "$NEW_COMMIT"
+    rustup override set miri
+    # Cleanup.
+    cargo clean
+    # Call 'cargo metadata' on the sources in case that changes the lockfile
+    # (which fails under some setups when it is done from inside vscode).
+    cargo metadata --format-version 1 --manifest-path "$(rustc --print sysroot)/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.toml" >/dev/null
+    # Done!
+    exit 0
+    ;;
+rustc-pull)
+    cd "$MIRIDIR"
+    git fetch http://localhost:8000/rust-lang/rust.git$JOSH_FILTER.git master
+    git merge FETCH_HEAD
+    exit 0
+    ;;
+rustc-push)
+    USER="$1"
+    BRANCH="$2"
+    if [ -z "$USER" ] || [ -z "$BRANCH" ]; then
+        echo "Usage: $0 rustc-push <github user> <branch>"
+        exit 1
+    fi
+    if [ -n "$RUSTC_GIT" ]; then
+        # Use an existing fork for the branch updates.
+        cd "$RUSTC_GIT"
+    else
+        # Do this in the local Miri repo.
+        echo "This will pull a copy of the rust-lang/rust history into this Miri checkout, growing it by about 1GB."
+        read -r -p "To avoid that, abort now and set the RUSTC_GIT environment variable to an existing rustc checkout. Proceed? [y/N] "
+        if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+            exit 1
+        fi
+        cd "$MIRIDIR"
+    fi
+    # Prepare the branches. For reliable pushing we need to push to a non-existent branch
+    # and set `-o base` to a branch that holds current rustc master.
+    echo "Preparing $USER/rust..."
+    if git fetch https://github.com/$USER/rust $BRANCH &>/dev/null; then
+        echo "The '$BRANCH' seems to already exist in $USER/rust. Please delete it and try again."
+        exit 1
+    fi
+    git fetch https://github.com/rust-lang/rust master
+    git push https://github.com/$USER/rust FETCH_HEAD:master
+    # Do the actual push.
+    cd "$MIRIDIR"
+    echo "Pushing Miri changes..."
+    git push http://localhost:8000/$USER/rust.git$JOSH_FILTER.git HEAD:$BRANCH -o base=master
+    exit 0
+    ;;
 many-seeds)
     for SEED in $({ echo obase=16; seq 0 255; } | bc); do
         echo "Trying seed: $SEED"
@@ -106,9 +181,28 @@ bench)
     ;;
 esac
 
+## Run the auto-things.
+if [ -z "$MIRI_AUTO_OPS" ]; then
+    export MIRI_AUTO_OPS=42
+
+    # Run this first, so that the toolchain doesn't change after
+    # other code has run.
+    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-toolchain" ] ; then
+        $0 toolchain
+    fi
+
+    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-fmt" ] ; then
+        $0 fmt
+    fi
+
+    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-clippy" ] ; then
+        $0 clippy -- -D warnings
+    fi
+fi
+
 ## Prepare the environment
 # Determine some toolchain properties
-# export the target so its available in miri
+TOOLCHAIN=$(cd "$MIRIDIR"; rustup show active-toolchain | head -n 1 | cut -d ' ' -f 1)
 TARGET=$(rustc +$TOOLCHAIN --version --verbose | grep "^host:" | cut -d ' ' -f 2)
 SYSROOT=$(rustc +$TOOLCHAIN --print sysroot)
 LIBDIR=$SYSROOT/lib/rustlib/$TARGET/lib
@@ -227,10 +321,7 @@ cargo)
     $CARGO "$@"
     ;;
 *)
-    if [ -n "$COMMAND" ]; then
-      echo "Unknown command: $COMMAND"
-      echo
-    fi
-    echo "$USAGE"
+    echo "Unknown command: $COMMAND"
     exit 1
+    ;;
 esac

src/tools/miri/rust-version

@@ -1 +1 @@
-85d089b41e2a0c0f07ab34f6c5a7c451389f25e6
+607878d069267e1402ad792c9331b426e4c6d0f9