Auto merge of rust-lang#52401 - semarie:tidy-extdeps, r=alexcrichton

tidy: add a new test for external dependencies

ensure all packages in Cargo.lock will be vendored, and fail if the
source packages isn't whitelisted.

the purpose is to avoid such kind of issues:
- rust-lang#52029 Rustfmt isn't vendored correctly
- rust-lang#42719 building beta with vendor=true fail due to network dependencies

as Rust comes with several external dependencies (clippy, miri, rustfmt, rls), it is important to have a way to catch some errors in the update of this submodules.

The new check in tidy quickly reads `Cargo.lock` to search for the `source` of all packages. This attribute is present when the package comes from external source (like `crates.io-index` or some `git` repository). Some sources are whitelisted (like `crates.io-index`) as the crates are vendored.

`Cargo.lock` extract with several cases (git, crates.io, and local).
```
[[package]]
name = "rustfmt-nightly"
version = "0.8.2"
source = "git+https://github.com/rust-lang-nursery/rustfmt?rev=5e5992517d3591e2708d4ca6b155dfcbdf3344b9#5e5992517d3591e2708d4ca6b155dfcbdf3344b9"
dependencies = [
...
]

[[package]]
name = "same-file"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
...
]

[[package]]
name = "rustdoc-themes"
version = "0.1.0"
```

r? @alexcrichton

Author: bors

src/tools/tidy/src/extdeps.rs

@@ -0,0 +1,50 @@
+// Copyright 2018 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+// ! Check for external package sources. Allow only vendorable packages.
+
+use std::fs::File;
+use std::io::Read;
+use std::path::Path;
+
+/// List of whitelisted sources for packages
+static WHITELISTED_SOURCES: &'static [&'static str] = &[
+    "\"registry+https://github.com/rust-lang/crates.io-index\"",
+];
+
+/// check for external package sources
+pub fn check(path: &Path, bad: &mut bool) {
+    // Cargo.lock of rust: src/Cargo.lock
+    let path = path.join("Cargo.lock");
+
+    // open and read the whole file
+    let mut cargo_lock = String::new();
+    t!(t!(File::open(path)).read_to_string(&mut cargo_lock));
+
+    // process each line
+    let mut lines = cargo_lock.lines();
+    while let Some(line) = lines.next() {
+
+        // consider only source entries
+        if ! line.starts_with("source = ") {
+            continue;
+        }
+
+        // extract source value
+        let parts: Vec<&str> = line.splitn(2, "=").collect();
+        let source = parts[1].trim();
+
+        // ensure source is whitelisted
+        if !WHITELISTED_SOURCES.contains(&&*source) {
+            println!("invalid source: {}", source);
+            *bad = true;
+        }
+    }
+}

src/tools/tidy/src/lib.rs

@@ -49,6 +49,7 @@ pub mod features;
 pub mod cargo;
 pub mod pal;
 pub mod deps;
+pub mod extdeps;
 pub mod ui_tests;
 pub mod unstable_book;
 pub mod libcoretest;

src/tools/tidy/src/main.rs

@@ -46,6 +46,7 @@ fn main() {
         deps::check(&path, &mut bad);
     }
     deps::check_whitelist(&path, &cargo, &mut bad);
+    extdeps::check(&path, &mut bad);
     ui_tests::check(&path, &mut bad);
 
     if bad {