Auto merge of #653 - Mark-Simulacrum:watch-disk-often, r=Mark-Simulacrum

Check disk space more often

Currently seeing a high rate of failure in git clone due to disk space limits
being exhausted, which is presumably due to this check not running often enough,
though we don't have great evidence for that. With current instance disk space
(100 GB) we would need to always grow slower than 15GB in 5 minutes (roughly
3GB/minute), which seems like it should be reasonable, but maybe isn't.

The new bound is every 30 seconds, and we'd need to grow 20 GB in that time, which seems very unlikely.

It's likely also worth exploring a retry strategy for error'd crates, but
that'll take more design work to make sure we do eventually complete.

Author: bors

src/db/migrations.rs

@@ -5,6 +5,7 @@ use std::collections::HashSet;
 
 enum MigrationKind {
     SQL(&'static str),
+    #[allow(clippy::type_complexity)]
     Code(Box<dyn Fn(&Transaction) -> ::rusqlite::Result<()>>),
 }
 

src/runner/mod.rs

@@ -16,8 +16,8 @@ use std::collections::HashMap;
 use std::sync::Mutex;
 use std::time::Duration;
 
-const DISK_SPACE_WATCHER_INTERVAL: Duration = Duration::from_secs(300);
-const DISK_SPACE_WATCHER_THRESHOLD: f32 = 0.85;
+const DISK_SPACE_WATCHER_INTERVAL: Duration = Duration::from_secs(30);
+const DISK_SPACE_WATCHER_THRESHOLD: f32 = 0.80;
 
 #[derive(Debug, Fail)]
 #[fail(display = "overridden task result to {}", _0)]