Check disk space more often

Currently seeing a high rate of failure in git clone due to disk space limits
being exhausted, which is presumably due to this check not running often enough,
though we don't have great evidence for that. With current instance disk space
(100 GB) we would need to always grow slower than 15GB in 5 minutes (roughly
3GB/minute), which seems like it should be reasonable, but maybe isn't.

It's likely also worth exploring a retry strategy for error'd crates, but
that'll take more design work to make sure we do eventually complete.

Author: Mark-Simulacrum

src/runner/mod.rs

@@ -16,8 +16,8 @@ use std::collections::HashMap;
 use std::sync::Mutex;
 use std::time::Duration;
 
-const DISK_SPACE_WATCHER_INTERVAL: Duration = Duration::from_secs(300);
-const DISK_SPACE_WATCHER_THRESHOLD: f32 = 0.85;
+const DISK_SPACE_WATCHER_INTERVAL: Duration = Duration::from_secs(30);
+const DISK_SPACE_WATCHER_THRESHOLD: f32 = 0.80;
 
 #[derive(Debug, Fail)]
 #[fail(display = "overridden task result to {}", _0)]