rustbuild: Document many more parts of the build

This commit expands the bootstrap build system's `README.md` as well as ensuring
that all API documentation is present and up-to-date. Additionally a new
`config.toml.example` file is checked in with commented out versions of all
possible configuration values.

Author: alexcrichton

src/bootstrap/README.md

@@ -1,4 +1,4 @@
-# Bootstrapping Rust
+# rustbuild - Bootstrapping Rust
 
 This is an in-progress README which is targeted at helping to explain how Rust
 is bootstrapped and in general some of the technical details of the build
@@ -8,20 +8,64 @@ system.
 > intended to be the primarily used one just yet. The makefiles are currently
 > the ones that are still "guaranteed to work" as much as possible at least.
 
-## Using the new build system
+## Using rustbuild
 
 When configuring Rust via `./configure`, pass the following to enable building
 via this build system:
 
 ```
 ./configure --enable-rustbuild
+make
 ```
 
-## ...
+Afterwards the `Makefile` which is generated will have a few commands like
+`make check`, `make tidy`, etc. For finer-grained control, the
+`bootstrap.py` entry point can be used:
+
+```
+python src/bootstrap/bootstrap.py
+```
+
+This accepts a number of options like `--stage` and `--step` which can configure
+what's actually being done.
+
+## Configuring rustbuild
+
+There are currently two primary methods for configuring the rustbuild build
+system. First, the `./configure` options serialized in `config.mk` will be
+parsed and read. That is, if any `./configure` options are passed, they'll be
+handled naturally.
+
+Next, rustbuild offers a TOML-based configuration system with a `config.toml`
+file in the same location as `config.mk`. An example of this configuration can
+be found at `src/bootstrap/config.toml.example`, and the configuration file
+can also be passed as `--config path/to/config.toml` if the build system is
+being invoked manually (via the python script).
+
+## Build stages
+
+The rustbuild build system goes through a few phases to actually build the
+compiler. What actually happens when you invoke rustbuild is:
+
+1. The entry point script, `src/bootstrap/bootstrap.py` is run. This script is
+   responsible for downloading the stage0 compiler/Cargo binaries, and it then
+   compiles the build system itself (this folder). Finally, it then invokes the
+   actual `boostrap` binary build system.
+2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
+   sanity checks (compilers exist for example), and then start building the
+   stage0 artifacts.
+3. The stage0 `cargo` downloaded earlier is used to build the standard library
+   and the compiler, and then these binaries are then copied to the `stage1`
+   directory. That compiler is then used to generate the stage1 artifacts which
+   are then copied to the stage2 directory, and then finally the stage2
+   artifacts are generated using that compiler.
+
+The goal of each stage is to (a) leverage Cargo as much as possible and failing
+that (b) leverage Rust as much as possible!
 
 ## Directory Layout
 
-This build system houses all output under the `target` directory, which looks
+This build system houses all output under the `build` directory, which looks
 like this:
 
 ```
@@ -42,6 +86,12 @@ build/
     debug/
     release/
 
+  # Output of the dist-related steps like dist-std, dist-rustc, and dist-docs
+  dist/
+
+  # Temporary directory used for various input/output as part of various stages
+  tmp/
+
   # Each remaining directory is scoped by the "host" triple of compilation at
   # hand.
   x86_64-unknown-linux-gnu/
@@ -50,7 +100,8 @@ build/
     # folder is under. The exact layout here will likely depend on the platform,
     # and this is also built with CMake so the build system is also likely
     # different.
-    compiler-rt/build/
+    compiler-rt/
+      build/
 
     # Output folder for LLVM if it is compiled for this target
     llvm/
@@ -67,6 +118,17 @@ build/
       share/
       ...
 
+    # Output folder for all documentation of this target. This is what's filled
+    # in whenever the `doc` step is run.
+    doc/
+
+    # Output for all compiletest-based test suites
+    test/
+      run-pass/
+      compile-fail/
+      debuginfo/
+      ...
+
     # Location where the stage0 Cargo and Rust compiler are unpacked. This
     # directory is purely an extracted and overlaid tarball of these two (done
     # by the bootstrapy python script). In theory the build system does not
@@ -82,7 +144,9 @@ build/
     # invocation. The build system instruments calling Cargo in the right order
     # with the right variables to ensure these are filled in correctly.
     stageN-std/
+    stageN-test/
     stageN-rustc/
+    stageN-tools/
 
     # This is a special case of the above directories, **not** filled in via
     # Cargo but rather the build system itself. The stage0 compiler already has
@@ -96,7 +160,7 @@ build/
     # Basically this directory is just a temporary artifact use to configure the
     # stage0 compiler to ensure that the libstd we just built is used to
     # compile the stage1 compiler.
-    stage0-rustc/lib/
+    stage0-sysroot/lib/
 
     # These output directories are intended to be standalone working
     # implementations of the compiler (corresponding to each stage). The build
@@ -108,3 +172,69 @@ build/
     stage2/
     stage3/
 ```
+
+## Cargo projects
+
+The current build is unfortunately not quite as simple as `cargo build` in a
+directory, but rather the compiler is split into three different Cargo projects:
+
+* `src/rustc/std_shim` - a project which builds and compiles libstd
+* `src/rustc/test_shim` - a project which builds and compiles libtest
+* `src/rustc` - the actual compiler itself
+
+Each "project" has a corresponding Cargo.lock file with all dependencies, and
+this means that building the compiler involves running Cargo three times. The
+structure here serves two goals:
+
+1. Facilitating dependencies coming from crates.io. These dependencies don't
+   depend on `std`, so libstd is a separate project compiled ahead of time
+   before the actual compiler builds.
+2. Splitting "host artifacts" from "target artifacts". That is, when building
+   code for an arbitrary target you don't need the entire compiler, but you'll
+   end up needing libraries like libtest that depend on std but also want to use
+   crates.io dependencies. Hence, libtest is split out as its own project that
+   is sequenced after `std` but before `rustc`. This project is built for all
+   targets.
+
+There is some loss in build parallelism here because libtest can be compiled in
+parallel with a number of rustc artifacts, but in theory the loss isn't too bad!
+
+## Build tools
+
+We've actually got quite a few tools that we use in the compiler's build system
+and for testing. To organize these, each tool is a project in `src/tools` with a
+corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
+independent `Cargo.lock` files) and do not currently explicitly depend on the
+compiler or standard library. Compiling each tool is sequenced after the
+appropriate libstd/libtest/librustc compile above.
+
+## Extending rustbuild
+
+So you'd like to add a feature to the rustbuild build system or just fix a bug.
+Great! One of the major motivational factors for moving away from `make` is that
+Rust is in theory much easier to read, modify, and write. If you find anything
+excessively confusing, please open an issue on this and we'll try to get it
+documented or simplified pronto.
+
+First up, you'll probably want to read over the documentation above as that'll
+give you a high level overview of what rustbuild is doing. You also probably
+want to play around a bit yourself by just getting it up and running before you
+dive too much into the actual build system itself.
+
+After that, each module in rustbuild should have enough documentation to keep
+you up and running. Some general areas that you may be interested in modifying
+are:
+
+* Adding a new build tool? Take a look at `build/step.rs` for examples of other
+  tools, as well as `build/mod.rs`.
+* Adding a new compiler crate? Look no further! Adding crates can be done by
+  adding a new directory with `Cargo.toml` followed by configuring all
+  `Cargo.toml` files accordingly.
+* Adding a new dependency from crates.io? We're still working on that, so hold
+  off on that for now.
+* Adding a new configuration option? Take a look at `build/config.rs` or perhaps
+  `build/flags.rs` and then modify the build elsewhere to read that option.
+* Adding a sanity check? Take a look at `build/sanity.rs`.
+
+If you have any questions feel free to reach out on `#rust-internals` on IRC or
+open an issue in the bug tracker!

src/bootstrap/build/cc.rs

@@ -8,6 +8,29 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
+//! C-compiler probing and detection.
+//!
+//! This module will fill out the `cc` and `cxx` maps of `Build` by looking for
+//! C and C++ compilers for each target configured. A compiler is found through
+//! a number of vectors (in order of precedence)
+//!
+//! 1. Configuration via `target.$target.cc` in `config.toml`.
+//! 2. Configuration via `target.$target.android-ndk` in `config.toml`, if
+//!    applicable
+//! 3. Special logic to probe on OpenBSD
+//! 4. The `CC_$target` environment variable.
+//! 5. The `CC` environment variable.
+//! 6. "cc"
+//!
+//! Some of this logic is implemented here, but much of it is farmed out to the
+//! `gcc` crate itself, so we end up having the same fallbacks as there.
+//! Similar logic is then used to find a C++ compiler, just some s/cc/c++/ is
+//! used.
+//!
+//! It is intended that after this module has run no C/C++ compiler will
+//! ever be probed for. Instead the compilers found here will be used for
+//! everything.
+
 use std::process::Command;
 
 use build_helper::{cc2ar, output};

src/bootstrap/build/channel.rs

@@ -8,7 +8,13 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
-use std::env;
+//! Build configuration for Rust's release channels.
+//!
+//! Implements the stable/beta/nightly channel distinctions by setting various
+//! flags like the `unstable_features`, calculating variables like `release` and
+//! `package_vers`, and otherwise indicating to the compiler what it should
+//! print out as part of its version information.
+
 use std::fs::{self, File};
 use std::io::prelude::*;
 use std::process::Command;
@@ -19,6 +25,9 @@ use md5;
 use build::Build;
 
 pub fn collect(build: &mut Build) {
+    // Currently the canonical source for the release number (e.g. 1.10.0) and
+    // the prerelease version (e.g. `.1`) is in `mk/main.mk`. We "parse" that
+    // here to learn about those numbers.
     let mut main_mk = String::new();
     t!(t!(File::open(build.src.join("mk/main.mk"))).read_to_string(&mut main_mk));
     let mut release_num = "";
@@ -32,7 +41,8 @@ pub fn collect(build: &mut Build) {
         }
     }
 
-    // FIXME: this is duplicating makefile logic
+    // Depending on the channel, passed in `./configure --release-channel`,
+    // determine various properties of the build.
     match &build.config.channel[..] {
         "stable" => {
             build.release = release_num.to_string();
@@ -58,6 +68,8 @@ pub fn collect(build: &mut Build) {
     }
     build.version = build.release.clone();
 
+    // If we have a git directory, add in some various SHA information of what
+    // commit this compiler was compiled from.
     if fs::metadata(build.src.join(".git")).is_ok() {
         let ver_date = output(Command::new("git").current_dir(&build.src)
                                       .arg("log").arg("-1")
@@ -80,11 +92,14 @@ pub fn collect(build: &mut Build) {
         build.short_ver_hash = Some(short_ver_hash);
     }
 
+    // Calculate this compiler's bootstrap key, which is currently defined as
+    // the first 8 characters of the md5 of the release string.
     let key = md5::compute(build.release.as_bytes());
     build.bootstrap_key = format!("{:02x}{:02x}{:02x}{:02x}",
                                   key[0], key[1], key[2], key[3]);
-    env::set_var("RUSTC_BOOTSTRAP_KEY", &build.bootstrap_key);
 
+    // Slurp up the stage0 bootstrap key as we're bootstrapping from an
+    // otherwise stable compiler.
     let mut s = String::new();
     t!(t!(File::open(build.src.join("src/stage0.txt"))).read_to_string(&mut s));
     if let Some(line) = s.lines().find(|l| l.starts_with("rustc_key")) {