Auto merge of #31123 - alexcrichton:who-doesnt-want-two-build-systems, r=brson

This series of commits adds the initial implementation of a new build system for
the compiler and standard library based on Cargo. The high-level architecture
now looks like:

1. The `./configure` script is run with `--enable-rustbuild` and other standard
   configuration options.
2. A `Makefile` is generate which proxies commands to the new build system.
3. The new build system has a Python script entry point which manages
   downloading both a Rust and Cargo nightly. This initial script also manages
   building the build system itself (which is written in Rust).
4. The build system, written in rust and called `bootstrap`, architects how to
   call `cargo` and manages building all native libraries and such.

One might reasonably ask "why rewrite the build system?", which is a good
question! The Rust project has used Makefiles for as long as I can remember at
least, and while ugly and difficult to use are undeniably robust as they contain
years worth of tweaking and tuning for working on as many platforms in as many
situation as possible. The rationale behind this PR, however is:

* The makefiles are impenetrable to all but a few people on this
  planet. This means that contributions to the build system are almost
  nonexistent, and furthermore if a build system change is needed it's
  incredibly difficult to figure out how to do so. This hindrance prevents us
  from doing some "perhaps fancier" things we may wish to do in make.

* Our build system, while portable, is unfortunately not infinitely portable
  everywhere.  For example the recently-introduced MSVC target is quite unlikely
  to have `make` installed by default (e.g. it requires building inside of an
  MSYS2 shell currently). Conversely, the portability of make comes at a cost of
  crazy and weird hacks to work around all sorts of versions of software
  everywhere, especially when it comes to the configure script and makefiles.
  By rewriting this logic in one of the most robust platforms there is, Rust,
  we get to assuage all of these worries for free!

* There's a standard tool to build Rust crates, Cargo, but the standard library
  and compiler don't use it. This means that they cannot benefit easily from the
  crates.io ecosystem, nor can the ecosystem benefit from a standard way to
  build this repository itself. Moving to Cargo should help assuage both of
  these needs. This has the added benefit of making the compiler more
  approachable for newbies as working on the compiler will just happen to be
  working on a large Cargo project, all the same standard tools and tricks will
  apply.

* There's a huge amount of portability information in the main distribution, for
  example around cross compiling, compiling on new OSes, etc. Pushing this logic
  into standard crates (like `gcc`) enables the community to immediately benefit
  from new build logic.

Despite these benefits, it's going to be a long road to actually replace our
current build system. This PR is just the beginning and doesn't implement the
full suite of functionality as the current one, but there are many more to
follow! The current implementation strategy hopes to look like:

1. Land a second build system in-tree that can be itereated on an and
   contributed to. This will not be used just yet in terms of gating new commits
   to the repo.
2. Over time, bring the second build system to feature parity with the old build
   system, start setting up CI for both build systems.
3. At some point in the future, switch the default to the new build system, but
   keep the old one around.
4. At some further point in the future, delete the entire old build system.

---

Alright, so with all that out of the way, here's some more info on this PR
itself. The inital build system here is contained in the `src/bootstrap`
directory and just adds the necessary minimum bits to bootstrap the compiler
itself. There is currently no support for building documentation, running tests,
or installing, but the implemented support is:

* Compiling LLVM with `cmake` instead of `./configure` + `make`. The LLVM
  project is removing their autotools build system, so we'd have to make this
  transition eventually anyway.

* Compiling compiler-rt with `cmake` as well (for the same rationale as above).

* Adding `Cargo.toml` to map out the dependency graph to all crates, and also
  adding `build.rs` files where appropriate. For example `alloc_jemalloc` has a
  script to build jemalloc, `flate` has a script to build `miniz.c`, `std` will
  build `libbacktrace`, etc.

* Orchestrating all the calls to `cargo` to build the standard distribution,
  following the normal bootstrapping process. This also tracks dependencies
  between steps to ensure cross-compilation targets happen as well.

* Configuration is intended to eventually be done through a `config.toml` file,
  so support is implemented for this. The most likely vector of configuration
  for now, however, is likely through `config.mk` (what `./configure` emits), so
  the build system currently parses this information.

There's still quite a few steps left to do, and I'll open up some follow-up
issues (as well as a tracking issue) for this migration, but hopefully this is a
great start to get going! This PR is currently tested on all the
Windows/Linux/OSX triples for x86\_64 and x86, but more portability is always
welcome!

---

Future functionality left to implement

* [ ] Re-verify that multi-host builds work
* [ ] Verify android build works
* [ ] Verify iOS build work (mostly compiler-rt)
* [ ] Verify sha256 and ideally gpg of downloaded nightly compiler and nightly rustc
* [ ] Implement testing -- this is a huge bullet point with lots of sub-bullets
* [ ] Build and generate documentation (plus the various tools we have in-tree)
* [ ] Move various src/etc scripts into Rust -- not sure how this interacts with `make` build system
* [ ] Implement `make install` - like testing this is also quite massive
* [x] Deduplicate version information with makefiles

Author: bors

.gitignore

@@ -93,3 +93,4 @@ tmp.*.rs
 version.md
 version.ml
 version.texi
+/target

configure

@@ -606,6 +606,7 @@ opt stage0-landing-pads 1 "enable landing pads during bootstrap with stage0"
 opt dist-host-only 0 "only install bins for the host architecture"
 opt inject-std-version 1 "inject the current compiler version of libstd into programs"
 opt llvm-version-check 1 "check if the LLVM version is supported, build anyway"
+opt rustbuild 0 "use the rust and cargo based build system"
 
 # Optimization and debugging options. These may be overridden by the release channel, etc.
 opt_nosave optimize 1 "build optimized rust code"
@@ -625,7 +626,7 @@ valopt llvm-root "" "set LLVM root"
 valopt python "" "set path to python"
 valopt jemalloc-root "" "set directory where libjemalloc_pic.a is located"
 valopt build "${DEFAULT_BUILD}" "GNUs ./configure syntax LLVM build triple"
-valopt android-cross-path "/opt/ndk_standalone" "Android NDK standalone path (deprecated)"
+valopt android-cross-path "" "Android NDK standalone path (deprecated)"
 valopt i686-linux-android-ndk "" "i686-linux-android NDK standalone path"
 valopt arm-linux-androideabi-ndk "" "arm-linux-androideabi NDK standalone path"
 valopt aarch64-linux-android-ndk "" "aarch64-linux-android NDK standalone path"
@@ -1422,7 +1423,7 @@ done
 step_msg "configuring submodules"
 
 # Have to be in the top of src directory for this
-if [ -z $CFG_DISABLE_MANAGE_SUBMODULES ]
+if [ -z $CFG_DISABLE_MANAGE_SUBMODULES ] && [ -z $CFG_ENABLE_RUSTBUILD ]
 then
     cd ${CFG_SRC_DIR}
 
@@ -1481,7 +1482,11 @@ do
         ;;
     esac
 
-    if [ -z $CFG_LLVM_ROOT ]
+    if [ -n "$CFG_ENABLE_RUSTBUILD" ]
+    then
+        msg "not configuring LLVM, rustbuild in use"
+        do_reconfigure=0
+    elif [ -z $CFG_LLVM_ROOT ]
     then
         LLVM_BUILD_DIR=${CFG_BUILD_DIR}$t/llvm
         if [ -n "$CFG_DISABLE_OPTIMIZE_LLVM" ]
@@ -1812,8 +1817,15 @@ do
     putvar $CFG_LLVM_INST_DIR
 done
 
+if [ -n "$CFG_ENABLE_RUSTBUILD" ]
+then
+    INPUT_MAKEFILE=src/bootstrap/mk/Makefile.in
+else
+    INPUT_MAKEFILE=Makefile.in
+fi
+
 msg
-copy_if_changed ${CFG_SRC_DIR}Makefile.in ./Makefile
+copy_if_changed ${CFG_SRC_DIR}${INPUT_MAKEFILE} ./Makefile
 move_if_changed config.tmp config.mk
 rm -f config.tmp
 touch config.stamp

mk/crates.mk

@@ -144,8 +144,7 @@ TARGET_CRATES += alloc_jemalloc
 DEPS_std += alloc_jemalloc
 DEPS_alloc_jemalloc := core libc native:jemalloc
 ONLY_RLIB_alloc_jemalloc := 1
-else
-RUSTFLAGS_rustc_back := --cfg disable_jemalloc
+RUSTFLAGS_rustc_back := --cfg 'feature="jemalloc"'
 endif
 
 ################################################################################

mk/llvm.mk

@@ -102,7 +102,7 @@ $(foreach host,$(CFG_HOST), \
 define LLVM_LINKAGE_DEPS
 $$(TLIB$(1)_T_$(2)_H_$(3))/stamp.rustc_llvm: $$(LLVM_LINKAGE_PATH_$(2))
 RUSTFLAGS$(1)_rustc_llvm_T_$(2) += $$(shell echo $$(LLVM_ALL_COMPONENTS_$(2)) | tr '-' '_' |\
-	sed -e 's/^ //;s/\([^ ]*\)/\-\-cfg have_component_\1/g')
+	sed -e 's/^ //;s/\([^ ]*\)/\-\-cfg "llvm_component=\\"\1\\""/g')
 endef
 
 $(foreach source,$(CFG_HOST), \

src/bootstrap/Cargo.lock

@@ -0,0 +1,29 @@
+[package]
+authors = ["The Rust Project Developers"]
+name = "bootstrap"
+version = "0.0.0"
+
+[lib]
+name = "bootstrap"
+path = "lib.rs"
+
+[[bin]]
+name = "bootstrap"
+path = "main.rs"
+
+[[bin]]
+name = "rustc"
+path = "rustc.rs"
+
+[dependencies]
+build_helper = { path = "../build_helper" }
+cmake = "0.1.10"
+filetime = "0.1"
+num_cpus = "0.2"
+toml = "0.1"
+getopts = "0.2"
+rustc-serialize = "0.3"
+winapi = "0.2"
+kernel32-sys = "0.2"
+gcc = "0.3.17"
+libc = "0.2"

src/bootstrap/Cargo.toml

@@ -0,0 +1,110 @@
+# Bootstrapping Rust
+
+This is an in-progress README which is targeted at helping to explain how Rust
+is bootstrapped and in general some of the technical details of the build
+system.
+
+> **Note**: This build system is currently under active development and is not
+> intended to be the primarily used one just yet. The makefiles are currently
+> the ones that are still "guaranteed to work" as much as possible at least.
+
+## Using the new build system
+
+When configuring Rust via `./configure`, pass the following to enable building
+via this build system:
+
+```
+./configure --enable-rustbuild
+```
+
+## ...
+
+## Directory Layout
+
+This build system houses all output under the `target` directory, which looks
+like this:
+
+```
+# Root folder of all output. Everything is scoped underneath here
+build/
+
+  # Location where the stage0 compiler downloads are all cached. This directory
+  # only contains the tarballs themselves as they're extracted elsewhere.
+  cache/
+    2015-12-19/
+    2016-01-15/
+    2016-01-21/
+    ...
+
+  # Output directory for building this build system itself. The stage0
+  # cargo/rustc are used to build the build system into this location.
+  bootstrap/
+    debug/
+    release/
+
+  # Each remaining directory is scoped by the "host" triple of compilation at
+  # hand.
+  x86_64-unknown-linux-gnu/
+
+    # The build artifacts for the `compiler-rt` library for the target this
+    # folder is under. The exact layout here will likely depend on the platform,
+    # and this is also built with CMake so the build system is also likely
+    # different.
+    compiler-rt/build/
+
+    # Output folder for LLVM if it is compiled for this target
+    llvm/
+
+      # build folder (e.g. the platform-specific build system). Like with
+      # compiler-rt this is compiled with CMake
+      build/
+
+      # Installation of LLVM. Note that we run the equivalent of 'make install'
+      # for LLVM to setup these folders.
+      bin/
+      lib/
+      include/
+      share/
+      ...
+
+    # Location where the stage0 Cargo and Rust compiler are unpacked. This
+    # directory is purely an extracted and overlaid tarball of these two (done
+    # by the bootstrapy python script). In theory the build system does not
+    # modify anything under this directory afterwards.
+    stage0/
+
+    # These to build directories are the cargo output directories for builds of
+    # the standard library and compiler, respectively. Internally these may also
+    # have other target directories, which represent artifacts being compiled
+    # from the host to the specified target.
+    #
+    # Essentially, each of these directories is filled in by one `cargo`
+    # invocation. The build system instruments calling Cargo in the right order
+    # with the right variables to ensure these are filled in correctly.
+    stageN-std/
+    stageN-rustc/
+
+    # This is a special case of the above directories, **not** filled in via
+    # Cargo but rather the build system itself. The stage0 compiler already has
+    # a set of target libraries for its own host triple (in its own sysroot)
+    # inside of stage0/. When we run the stage0 compiler to bootstrap more
+    # things, however, we don't want to use any of these libraries (as those are
+    # the ones that we're building). So essentially, when the stage1 compiler is
+    # being compiled (e.g. after libstd has been built), *this* is used as the
+    # sysroot for the stage0 compiler being run.
+    #
+    # Basically this directory is just a temporary artifact use to configure the
+    # stage0 compiler to ensure that the libstd we just built is used to
+    # compile the stage1 compiler.
+    stage0-rustc/lib/
+
+    # These output directories are intended to be standalone working
+    # implementations of the compiler (corresponding to each stage). The build
+    # system will link (using hard links) output from stageN-{std,rustc} into
+    # each of these directories.
+    #
+    # In theory there is no extra build output in these directories.
+    stage1/
+    stage2/
+    stage3/
+```