Workaround: wait longer for IPC socket file to appear (#1197)

* trinity/utils/ipc: increase wait_for_ipc() timeout to 10 seconds.

It seems that on machines with slow I/O, creation of the socket
file may take a long while.

* trinity: log duration of wait for IPC file.

* trinity/main: minor, move wait_for_ipc() one line down.

It is the networking process that needs the socket to be present,
so move the wait-for-socket line closer to starting the former.

* trinity/utils/ipc: WARN on timeout reached, demote to DEBUG on success.

* trinity/utils/ipc: minor, change condition from `>` to `>=`.

Dummy commit to force CircleCI re-run.

* Revert previous four commits - taking alrenative approach.

Will reuse previous loop structure in wait_for_ipc(), and raise an
exception on timeout instead.

* trinity: raise exception in wait_for_ipc() on reaching timeout.

The boot procedure still won't try to shut down cleanly, but will
at least print an error message _and_ a warning about proceeding
anyway.

* Change to exit trinity if database IPC connection cannot be made.

Author: veox

trinity/main.py

@@ -194,7 +194,14 @@ def trinity_boot(args: Namespace,
     # start the processes
     database_server_process.start()
     logger.info("Started DB server process (pid=%d)", database_server_process.pid)
-    wait_for_ipc(chain_config.database_ipc_path)
+
+    # networking process needs the IPC socket file provided by the database process
+    try:
+        wait_for_ipc(chain_config.database_ipc_path)
+    except TimeoutError as e:
+        logger.error("Timeout waiting for database to start.  Exiting...")
+        kill_process_gracefully(database_server_process, logger)
+        sys.exit(1)
 
     networking_process.start()
     logger.info("Started networking process (pid=%d)", networking_process.pid)

trinity/utils/ipc.py

@@ -8,12 +8,19 @@
 from typing import Callable
 
 
-def wait_for_ipc(ipc_path: pathlib.Path, timeout: int=1) -> None:
+def wait_for_ipc(ipc_path: pathlib.Path, timeout: int=10) -> None:
+    """
+    Waits up to ``timeout`` seconds for the IPC socket file to appear at path
+    ``ipc_path``, or raises a :exc:`TimeoutError` otherwise.
+    """
     start_at = time.time()
     while time.time() - start_at < timeout:
         if ipc_path.exists():
-            break
-        time.sleep(0.05)
+            return
+        else:
+            time.sleep(0.05)
+    # haven't `return`ed by now - raise unconditionally
+    raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
 
 
 DEFAULT_SIGINT_TIMEOUT = 10