Auto merge of #2667 - RalfJung:josh, r=RalfJung

adjust josh pushing, remove './miri toolchain HEAD/commit'

Looks like pushing works much better if we use as base commit the last commit we pulled from rustc and merged into Miri. Which I guess is fair. Conveniently, the `rust-version` file should usually be pretty much exactly that commit. Also adjust `rustc-pull` to update that file.

Pulling is now the way to update to a newer rustc, so I also removed `./miri toolchain HEAD`.

Author: bors

src/tools/miri/.github/workflows/ci.yml

@@ -67,10 +67,10 @@ jobs:
         shell: bash
         run: |
           if [[ ${{ github.event_name }} == 'schedule' ]]; then
-            ./miri toolchain HEAD --host ${{ matrix.host_target }}
-          else
-            ./miri toolchain "" --host ${{ matrix.host_target }}
+            echo "Building against latest rustc git version"
+            git ls-remote https://github.com/rust-lang/rust/ HEAD | cut -f 1 > rust-version
           fi
+          ./miri toolchain --host ${{ matrix.host_target }}
 
       - name: Show Rust version
         run: |

src/tools/miri/CONTRIBUTING.md

@@ -209,23 +209,6 @@ We described above the simplest way to get a working build environment for Miri,
 which is to use the version of rustc indicated by `rustc-version`. But
 sometimes, that is not enough.
 
-### Updating `rustc-version`
-
-The `rustc-version` file is regularly updated to keep Miri close to the latest
-version of rustc. Usually, new contributors do not have to worry about this. But
-sometimes a newer rustc is needed for a patch, and sometimes Miri needs fixing
-for changes in rustc. In both cases, `rustc-version` needs updating.
-
-To update the `rustc-version` file and install the latest rustc, you can run:
-```
-./miri toolchain HEAD
-```
-
-Now edit Miri until `./miri test` passes, and submit a PR. Generally, it is
-preferred to separate updating `rustc-version` and doing what it takes to get
-Miri working again, from implementing new features that rely on the updated
-rustc. This avoids blocking all Miri development on landing a big PR.
-
 ### Building Miri with a locally built rustc
 
 [building Miri with a locally built rustc]: #building-miri-with-a-locally-built-rustc
@@ -299,9 +282,10 @@ We assume we start on an up-to-date master branch in the Miri repo.
 
 ```sh
 # Fetch and merge rustc side of the history. Takes ca 5 min the first time.
+# This will also update the 'rustc-version' file.
 ./miri rustc-pull
-# Update toolchain reference and apply formatting.
-./miri toolchain HEAD && ./miri fmt
+# Update local toolchain and apply formatting.
+./miri toolchain && ./miri fmt
 git commit -am "rustup"
 ```
 

src/tools/miri/miri

@@ -42,21 +42,21 @@ many different seeds.
 Runs the benchmarks from bench-cargo-miri in hyperfine. hyperfine needs to be installed.
 <benches> can explicitly list the benchmarks to run; by default, all of them are run.
 
+./miri toolchain <flags>:
+Update and activate the rustup toolchain 'miri' to the commit given in the
+`rust-version` file.
+`rustup-toolchain-install-master` must be installed for this to work. Any extra
+flags are passed to `rustup-toolchain-install-master`.
+
 ./miri rustc-pull:
-Pull and merge Miri changes from the rustc repo.
+Pull and merge Miri changes from the rustc repo. The fetched commit is stored in
+the `rust-version` file, so the next `./miri toolchain` will install the rustc
+we just pulled.
 
 ./miri rustc-push <github user> <branch>:
-Push Miri changes back to the rustc repo. This will update the 'master' branch
-in the Rust fork of the given user to upstream. It will also pull a copy of the
-rustc history into the Miri repo, unless you set the RUSTC_GIT env var to an
-existing clone of the rustc repo.
-
-./miri toolchain <commit> <flags>:
-Update and activate the rustup toolchain 'miri'. If no commit is given, updates
-to the commit given in the `rust-version` file. If the commit is `HEAD`, updates
-to the latest upstream rustc commit.
-`rustup-toolchain-install-master` must be installed for this to work. Any extra
-flags are passed to `rustup-toolchain-install-master`.
+Push Miri changes back to the rustc repo. This will pull a copy of the rustc
+history into the Miri repo, unless you set the RUSTC_GIT env var to an existing
+clone of the rustc repo.
 
   ENVIRONMENT VARIABLES
 
@@ -86,21 +86,12 @@ TOOLCHAIN=$(cd "$MIRIDIR"; rustup show active-toolchain | head -n 1 | cut -d ' '
 case "$COMMAND" in
 toolchain)
     cd "$MIRIDIR"
+    NEW_COMMIT=$(cat rust-version)
     # Make sure rustup-toolchain-install-master is installed.
     if ! which rustup-toolchain-install-master >/dev/null; then
         echo "Please install rustup-toolchain-install-master by running 'cargo install rustup-toolchain-install-master'"
         exit 1
     fi
-    # Determine new commit.
-    if [[ "$1" == "" ]]; then
-        NEW_COMMIT=$(cat rust-version)
-    elif [[ "$1" == "HEAD" ]]; then
-        NEW_COMMIT=$(git ls-remote https://github.com/rust-lang/rust/ HEAD | cut -f 1)
-    else
-        NEW_COMMIT="$1"
-    fi
-    echo "$NEW_COMMIT" > rust-version
-    shift || true # don't fail if shifting fails because no commit was given
     # Check if we already are at that commit.
     CUR_COMMIT=$(rustc +miri --version -v 2>/dev/null | grep "^commit-hash: " | cut -d " " -f 2)
     if [[ "$CUR_COMMIT" == "$NEW_COMMIT" ]]; then
@@ -122,8 +113,18 @@ toolchain)
     ;;
 rustc-pull)
     cd "$MIRIDIR"
+    FETCH_COMMIT=$(git ls-remote https://github.com/rust-lang/rust/ HEAD | cut -f 1)
+    # We can't pull from a commit with josh
+    # (https://github.com/josh-project/josh/issues/1034), so we just hope that
+    # nothing gets merged into rustc *during* this pull.
     git fetch http://localhost:8000/rust-lang/rust.git$JOSH_FILTER.git master
+    # Just verify that `master` didn't move.
+    if [[ $FETCH_COMMIT != $(git ls-remote https://github.com/rust-lang/rust/ HEAD | cut -f 1) ]]; then
+        echo "Looks like something got merged into Rust *while we were pulling*. Aborting. Please try again."
+    fi
+    echo "$FETCH_COMMIT" > rust-version # do this *before* merging as merging will fail in case of conflicts
     git merge FETCH_HEAD --no-ff -m "Merge from rustc"
+    git commit rust-version --amend -m "Merge from rustc"
     exit 0
     ;;
 rustc-push)
@@ -145,19 +146,21 @@ rustc-push)
         fi
         cd "$MIRIDIR"
     fi
-    # Prepare the branches. For reliable pushing we need to push to a non-existent branch
-    # and set `-o base` to a branch that holds current rustc master.
-    echo "Preparing $USER/rust..."
-    if git fetch https://github.com/$USER/rust $BRANCH &>/dev/null; then
-        echo "The branch '$BRANCH' seems to already exist in $USER/rust. Please delete it and try again."
+    # Prepare the branch. Pushing works much better if we use as base exactly
+    # the commit that we pulled from last time, so we use the `rust-version`
+    # file as a good approximation of that.
+    BASE=$(cat "$MIRIDIR/rust-version")
+    echo "Preparing $USER/rust (base: $BASE)..."
+    if git fetch "https://github.com/$USER/rust" "$BRANCH" &>/dev/null; then
+        echo "The branch '$BRANCH' seems to already exist in 'https://github.com/$USER/rust'. Please delete it and try again."
         exit 1
     fi
-    git fetch https://github.com/rust-lang/rust master
-    git push https://github.com/$USER/rust FETCH_HEAD:master
+    git fetch https://github.com/rust-lang/rust $BASE
+    git push https://github.com/$USER/rust $BASE:refs/heads/$BRANCH -f
     # Do the actual push.
     cd "$MIRIDIR"
     echo "Pushing Miri changes..."
-    git push http://localhost:8000/$USER/rust.git$JOSH_FILTER.git HEAD:$BRANCH -o base=master
+    git push http://localhost:8000/$USER/rust.git$JOSH_FILTER.git HEAD:$BRANCH
     exit 0
     ;;
 many-seeds)