Auto merge of #40018 - japaric:ld, r=alexcrichton

-Z linker-flavor

(Please read the commit message first)

This PR is an alternative to #36120 (internal lld linker). The
main goal of this PR is to make it *possible* to use LLD as a linker to allow
out of tree experimentation. Now that LLD is going to be shipped with LLVM 4.0,
it should become easier to get a hold of LLD (hopefully, it will be packaged by
Linux distros soon).

Since LLD is a multiarch linker, it has the potential to make cross compilation
easier (less tools need to be installed). Supposedly, LLD is also faster than
the gold linker so LLD may improve build times where link times are significant
(e.g. 100% incremental compilation reuse).

The place where LLD shines is at linking Rust programs that don't depend on
system libraries. For example, here's how you would link a bare metal ARM
Cortex-M program:

```
$ xargo rustc --target thumbv7m-none-eabi -- -Z linker-flavor=ld -C linker=ld.lld -Z print-link-args
"ld.lld" \
  "-L" \
  "$XARGO_HOME/lib/rustlib/thumbv7m-none-eabi/lib" \
  "$PWD/target/thumbv7m-none-eabi/debug/deps/app-de1f86df314ad68c.0.o" \
  "-o" \
  "$PWD/target/thumbv7m-none-eabi/debug/deps/app-de1f86df314ad68c" \
  "--gc-sections" \
  "-L" \
  "$PWD/target/thumbv7m-none-eabi/debug/deps" \
  "-L" \
  "$PWD/target/debug/deps" \
  "-L" \
  "$XARGO_HOME/lib/rustlib/thumbv7m-none-eabi/lib" \
  "-Bstatic" \
  "-Bdynamic" \
  "$XARGO_HOME/lib/rustlib/thumbv7m-none-eabi/lib/libcore-11670d2bd4951fa7.rlib"

$ file target/thumbv7m-none-eabi/debug/app
app: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped, with debug_info
```

This doesn't require installing the `arm-none-eabi-gcc` toolchain.

Even cooler (but I'm biased) is that you can link Rust programs that use
[`steed`] (`steed` is a `std` re-implementation free of C dependencies for Linux
systems) instead of `std` for a bunch of different architectures without having
to install a single cross toolchain.

[`steed`]: https://github.com/japaric/steed

```
$ xargo rustc --target aarch64-unknown-linux-steed --example hello --release -- -Z print-link-args
"ld.lld" \
  "-L" \
  "$XARGO_HOME/lib/rustlib/aarch64-unknown-linux-steed/lib" \
  "$PWD/target/aarch64-unknown-linux-steed/release/examples/hello-80c130ad884c0f8f.0.o" \
  "-o" \
  "$PWD/target/aarch64-unknown-linux-steed/release/examples/hello-80c130ad884c0f8f" \
  "--gc-sections" \
  "-L" \
  "$PWD/target/aarch64-unknown-linux-steed/release/deps" \
  "-L" \
  "$PWD/target/release/deps" \
  "-L" \
  "$XARGO_HOME/lib/rustlib/aarch64-unknown-linux-steed/lib" \
  "-Bstatic" \
  "-Bdynamic" \
  "/tmp/rustc.lAybk9Ltx93Q/libcompiler_builtins-589aede02de78434.rlib"

$ file target/aarch64-unknown-linux-steed/release/examples/hello
hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped, with debug_info
```

All these targets (architectures) worked with LLD:

- [aarch64-unknown-linux-steed](https://github.com/japaric/steed/blob/lld/docker/aarch64-unknown-linux-steed.json)
- [arm-unknown-linux-steedeabi](https://github.com/japaric/steed/blob/lld/docker/arm-unknown-linux-steedeabi.json)
- [arm-unknown-linux-steedeabihf](https://github.com/japaric/steed/blob/lld/docker/arm-unknown-linux-steedeabihf.json)
- [armv7-unknown-linux-steedeabihf](https://github.com/japaric/steed/blob/lld/docker/armv7-unknown-linux-steedeabihf.json)
- [i686-unknown-linux-steed](https://github.com/japaric/steed/blob/lld/docker/i686-unknown-linux-steed.json)
- [mips-unknown-linux-steed](https://github.com/japaric/steed/blob/lld/docker/mips-unknown-linux-steed.json)
- [mipsel-unknown-linux-steed](https://github.com/japaric/steed/blob/lld/docker/mipsel-unknown-linux-steed.json)
- [powerpc-unknown-linux-steed](https://github.com/japaric/steed/blob/lld/docker/powerpc-unknown-linux-steed.json)
- [powerpc64-unknown-linux-steed](https://github.com/japaric/steed/blob/lld/docker/powerpc64-unknown-linux-steed.json)
- [x86_64-unknown-linux-steed](https://github.com/japaric/steed/blob/lld/docker/x86_64-unknown-linux-steed.json)

---

The case where lld is unergonomic is linking binaries that depend on system
libraries. Like "Hello, world" for `x86_64-unknown-linux-gnu`. Because you have
to pass as linker arguments: the path to the startup objects, the path to the
dynamic linker and the library search paths. And all those are system specific
so they can't be encoded in the target itself.

```
$ cargo \
  rustc \
  --release \
  -- \
  -C \
  linker=ld.lld \
  -Z \
  linker-flavor=ld \
  -C \
  link-args='-dynamic-linker /lib64/ld-linux-x86-64.so.2 -L/usr/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/6.3.1 /usr/lib/Scrt1.o /usr/lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.1/crtbeginS.o /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.1/crtendS.o /usr/lib/crtn.o'
```

---

Another case where `-Z linker-flavor` may come in handy is directly calling
Solaris' linker which is also a multiarch linker (or so I have heard). cc
@binarycrusader

cc @alexcrichton
Heads up: [breaking-change] due to changes in the target specification format.

Author: bors

src/doc/unstable-book/src/SUMMARY.md

@@ -108,6 +108,7 @@
 - [link_llvm_intrinsics](link-llvm-intrinsics.md)
 - [linkage](linkage.md)
 - [linked_list_extras](linked-list-extras.md)
+- [linker_flavor](linker-flavor.md)
 - [log_syntax](log-syntax.md)
 - [lookup_host](lookup-host.md)
 - [loop_break_value](loop-break-value.md)

src/doc/unstable-book/src/linker-flavor.md

@@ -0,0 +1,61 @@
+# `linker-flavor`
+
+The tracking issue for this feature is: None
+
+------------------------
+
+Every `rustc` target defaults to some linker. For example, Linux targets default
+to gcc. In some cases, you may want to override the default; you can do that
+with the unstable CLI argument: `-Z linker-flavor`.
+
+Here how you would use this flag to link a Rust binary for the
+`thumbv7m-none-eabi` using LLD instead of GCC.
+
+``` text
+$ xargo rustc --target thumbv7m-none-eabi -- \
+    -C linker=ld.lld \
+    -Z linker-flavor=ld \
+    -Z print-link-args | tr ' ' '\n'
+"ld.lld"
+"-L"
+"$SYSROOT/lib/rustlib/thumbv7m-none-eabi/lib"
+"$PWD/target/thumbv7m-none-eabi/debug/deps/app-512e9dbf385f233c.0.o"
+"-o"
+"$PWD/target/thumbv7m-none-eabi/debug/deps/app-512e9dbf385f233c"
+"--gc-sections"
+"-L"
+"$PWD/target/thumbv7m-none-eabi/debug/deps"
+"-L"
+"$PWD/target/debug/deps"
+"-L"
+"$SYSROOT/lib/rustlib/thumbv7m-none-eabi/lib"
+"-Bstatic"
+"$SYSROOT/lib/rustlib/thumbv7m-none-eabi/lib/libcore-e1ccb7dfb1cb9ebb.rlib"
+"-Bdynamic"
+```
+
+Whereas the default is:
+
+``` text
+$ xargo rustc --target thumbv7m-none-eabi -- \
+    -C link-arg=-nostartfiles \
+    -Z print-link-args | tr ' ' '\n'
+"arm-none-eabi-gcc"
+"-L"
+"$SYSROOT/lib/rustlib/thumbv7m-none-eabi/lib"
+"$PWD/target/thumbv7m-none-eabi/debug/deps/app-961e39416baa38d9.0.o"
+"-o"
+"$PWD/target/thumbv7m-none-eabi/debug/deps/app-961e39416baa38d9"
+"-Wl,--gc-sections"
+"-nodefaultlibs"
+"-L"
+"$PWD/target/thumbv7m-none-eabi/debug/deps"
+"-L"
+"$PWD/target/debug/deps"
+"-L"
+"$SYSROOT/lib/rustlib/thumbv7m-none-eabi/lib"
+"-Wl,-Bstatic"
+"$SYSROOT/lib/rustlib/thumbv7m-none-eabi/lib/libcore-e1ccb7dfb1cb9ebb.rlib"
+"-nostartfiles"
+"-Wl,-Bdynamic"
+```

src/librustc/session/config.rs

@@ -19,7 +19,7 @@ pub use self::DebugInfoLevel::*;
 use session::{early_error, early_warn, Session};
 use session::search_paths::SearchPaths;
 
-use rustc_back::PanicStrategy;
+use rustc_back::{LinkerFlavor, PanicStrategy};
 use rustc_back::target::Target;
 use lint;
 use middle::cstore;
@@ -641,12 +641,14 @@ macro_rules! options {
             Some("either `panic` or `abort`");
         pub const parse_sanitizer: Option<&'static str> =
             Some("one of: `address`, `leak`, `memory` or `thread`");
+        pub const parse_linker_flavor: Option<&'static str> =
+            Some(::rustc_back::LinkerFlavor::one_of());
     }
 
     #[allow(dead_code)]
     mod $mod_set {
         use super::{$struct_name, Passes, SomePasses, AllPasses, Sanitizer};
-        use rustc_back::PanicStrategy;
+        use rustc_back::{LinkerFlavor, PanicStrategy};
 
         $(
             pub fn $opt(cg: &mut $struct_name, v: Option<&str>) -> bool {
@@ -777,6 +779,14 @@ macro_rules! options {
             }
             true
         }
+
+        fn parse_linker_flavor(slote: &mut Option<LinkerFlavor>, v: Option<&str>) -> bool {
+            match v.and_then(LinkerFlavor::from_str) {
+                Some(lf) => *slote = Some(lf),
+                _ => return false,
+            }
+            true
+        }
     }
 ) }
 
@@ -979,6 +989,8 @@ options! {DebuggingOptions, DebuggingSetter, basic_debugging_options,
           "pass `-install_name @rpath/...` to the macOS linker"),
     sanitizer: Option<Sanitizer> = (None, parse_sanitizer, [TRACKED],
                                    "Use a sanitizer"),
+    linker_flavor: Option<LinkerFlavor> = (None, parse_linker_flavor, [UNTRACKED],
+                                           "Linker flavor"),
 }
 
 pub fn default_lib_output() -> CrateType {

src/librustc/session/mod.rs

@@ -36,7 +36,7 @@ use syntax::{ast, codemap};
 use syntax::feature_gate::AttributeType;
 use syntax_pos::{Span, MultiSpan};
 
-use rustc_back::PanicStrategy;
+use rustc_back::{LinkerFlavor, PanicStrategy};
 use rustc_back::target::Target;
 use rustc_data_structures::flock;
 use llvm;
@@ -363,6 +363,9 @@ impl Session {
     pub fn panic_strategy(&self) -> PanicStrategy {
         self.opts.cg.panic.unwrap_or(self.target.target.options.panic_strategy)
     }
+    pub fn linker_flavor(&self) -> LinkerFlavor {
+        self.opts.debugging_opts.linker_flavor.unwrap_or(self.target.target.linker_flavor)
+    }
     pub fn no_landing_pads(&self) -> bool {
         self.opts.debugging_opts.no_landing_pads || self.panic_strategy() == PanicStrategy::Abort
     }

src/librustc_back/lib.rs

@@ -52,6 +52,48 @@ pub mod dynamic_lib;
 
 use serialize::json::{Json, ToJson};
 
+macro_rules! linker_flavor {
+    ($(($variant:ident, $string:expr),)+) => {
+        #[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd, Hash,
+                 RustcEncodable, RustcDecodable)]
+        pub enum LinkerFlavor {
+            $($variant,)+
+        }
+
+        impl LinkerFlavor {
+            pub const fn one_of() -> &'static str {
+                concat!("one of: ", $($string, " ",)+)
+            }
+
+            pub fn from_str(s: &str) -> Option<Self> {
+                Some(match s {
+                    $($string => LinkerFlavor::$variant,)+
+                    _ => return None,
+                })
+            }
+
+            pub fn desc(&self) -> &str {
+                match *self {
+                    $(LinkerFlavor::$variant => $string,)+
+                }
+            }
+        }
+
+        impl ToJson for LinkerFlavor {
+            fn to_json(&self) -> Json {
+                self.desc().to_json()
+            }
+        }
+    }
+}
+
+linker_flavor! {
+    (Em, "em"),
+    (Gcc, "gcc"),
+    (Ld, "ld"),
+    (Msvc, "msvc"),
+}
+
 #[derive(Clone, Copy, Debug, PartialEq, Hash, RustcEncodable, RustcDecodable)]
 pub enum PanicStrategy {
     Unwind,

src/librustc_back/target/aarch64_apple_ios.rs

@@ -8,6 +8,7 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
+use LinkerFlavor;
 use target::{Target, TargetOptions, TargetResult};
 use super::apple_ios_base::{opts, Arch};
 
@@ -22,6 +23,7 @@ pub fn target() -> TargetResult {
         target_os: "ios".to_string(),
         target_env: "".to_string(),
         target_vendor: "apple".to_string(),
+        linker_flavor: LinkerFlavor::Gcc,
         options: TargetOptions {
             features: "+neon,+fp-armv8,+cyclone".to_string(),
             eliminate_frame_pointer: false,

src/librustc_back/target/aarch64_linux_android.rs

@@ -8,6 +8,7 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
+use LinkerFlavor;
 use target::{Target, TargetOptions, TargetResult};
 
 // See https://developer.android.com/ndk/guides/abis.html#arm64-v8a
@@ -28,6 +29,7 @@ pub fn target() -> TargetResult {
         target_os: "android".to_string(),
         target_env: "".to_string(),
         target_vendor: "unknown".to_string(),
+        linker_flavor: LinkerFlavor::Gcc,
         options: TargetOptions {
             abi_blacklist: super::arm_base::abi_blacklist(),
             .. base

src/librustc_back/target/aarch64_unknown_freebsd.rs

@@ -8,6 +8,7 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
+use LinkerFlavor;
 use target::{Target, TargetOptions, TargetResult};
 
 pub fn target() -> TargetResult {
@@ -26,6 +27,7 @@ pub fn target() -> TargetResult {
         target_os: "freebsd".to_string(),
         target_env: "".to_string(),
         target_vendor: "unknown".to_string(),
+        linker_flavor: LinkerFlavor::Gcc,
         options: TargetOptions {
             abi_blacklist: super::arm_base::abi_blacklist(),
             .. base

src/librustc_back/target/aarch64_unknown_fuchsia.rs

@@ -8,6 +8,7 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
+use LinkerFlavor;
 use target::{Target, TargetOptions, TargetResult};
 
 pub fn target() -> TargetResult {
@@ -23,6 +24,7 @@ pub fn target() -> TargetResult {
         target_os: "fuchsia".to_string(),
         target_env: "".to_string(),
         target_vendor: "unknown".to_string(),
+        linker_flavor: LinkerFlavor::Gcc,
         options: TargetOptions {
             abi_blacklist: super::arm_base::abi_blacklist(),
             .. base

src/librustc_back/target/aarch64_unknown_linux_gnu.rs

@@ -8,6 +8,7 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
+use LinkerFlavor;
 use target::{Target, TargetOptions, TargetResult};
 
 pub fn target() -> TargetResult {
@@ -26,6 +27,7 @@ pub fn target() -> TargetResult {
         arch: "aarch64".to_string(),
         target_os: "linux".to_string(),
         target_vendor: "unknown".to_string(),
+        linker_flavor: LinkerFlavor::Gcc,
         options: TargetOptions {
             abi_blacklist: super::arm_base::abi_blacklist(),
             .. base