subservers: fail LiT startup when integrated sub-server boot fails #1183
Conversation
Summary of ChangesHello @ffranr, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the reliability of the LiT startup sequence by elevating the status of integrated sub-servers to critical. Previously, LiT could continue operating even if some of its integrated sub-servers failed to launch. Now, any failure in an integrated sub-server's boot process will cause LiT itself to cease its startup, ensuring that the system only reaches an operational state when all essential integrated components are functioning correctly. This change provides a more robust and predictable initialization experience. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request modifies the startup logic to treat integrated sub-server startup failures as fatal. The changes correctly propagate errors from StartIntegratedServers up to the main start function, causing LiT to fail on startup as intended. The documentation has also been updated to reflect this new behavior. The implementation is sound, but I have one suggestion to improve the consistency of error handling.
* Treat integrated sub-servers as fatal to startup and return an error if any fail to start. * Propagate integrated sub-server startup errors to LiT so it stops launching and records the failure status. * Update docs to reflect integrated sub-servers are now critical to startup.
ffranr
force-pushed
the
wip/fail-startup-on-subserver-err
branch
from
57ee325 to
86e8ec5
Compare
There was a problem hiding this comment.
Thanks for this @ffranr 🙏!
In addition to the feedback I've commented below, this new behaviour definitely needs itest coverage.
I think we actually started working on this at the same time, and I have local branch with draft code implementing this + itest coverage. If you want to, i can clean that up and push it so that you can cherry-pick that to make it more simple for you. Let me know if that'd be helpful :).
| // started in integrated mode. An error is returned if any integrated sub-server | ||
| // fails to start. |
There was a problem hiding this comment.
We only want errors in the tapd sub-server to cause the startup process to error, and not any sub-server. The motivation for why this should specifically target tapd right now, is that we've realized that tapd is critically coupled to lnd in the scenario that the user has taproot-asset channels, in the sense that lnd can't function if tapd registers as the aux-unit, but then shuts down.
Therefore, I suggest that you add a map of "critical" sub-server names as a param to the StartIntegratedServers function. If a sub-server then errors during the startup, you'll check if that sub-server name exists in the passed map, and if so return an error.
We actually originally designed this function specifically to not error the whole startup process if any of the integrated sub-servers fails to start. The motivation behind that is that we do not want bugs in one sub-server, to disable users from using the other sub-servers until a fix for the erroring sub-server has been released. I.e. a bug that causes pool to error on startup, shouldn't disable people from performing loops until it's been fixed. Of course users can alawys disable pool in that scenario, but for UX purposes we do not want require users to understand that it's pool that's actually failing the startup process, and then additionally understand how they disable pool, as this adds friction.
| ) | ||
| if err != nil { | ||
| s.statusServer.SetErrored(ss.Name(), err.Error()) | ||
| errs = append(errs, fmt.Sprintf("%s: %v", ss.Name(), err)) |
There was a problem hiding this comment.
I think we want to return an error imminently here, and not proceed to start all sub-servers before we return the error. The motivation for that is that we want lnd to shutdown asap if a "critical" sub-server errors during startup, and that could be delayed if we proceed to start the rest of the sub-servers before returning the error, and hence proceed to shutdown lnd.
| if err != nil { | ||
| return fmt.Errorf("could not start integrated sub-servers: %w", err) | ||
| } |
There was a problem hiding this comment.
Just returning an error here will currently actually not shutdown lnd. Instead this will cause the Run function to wait until the shutdownInterceptor.ShutdownChannel() is triggered, and keep lnd up and running until that happens. This will lead to the same errors that we've seen previously, as lnd will still proceed to try to call into tapd.
Instead we want to make sure lnd is shutdown asap here. One way of achieving that would be to call shutdownInterceptor.RequestShutdown() here. Alternatively, you could call g.basicClient.StopDaemon() here.
I suggest the second alternative, as the single Interceptor pattern is a bit of a code smell that we eventually actually want to get rid of :).
Closes #1181