Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "starting" info logging in the various components that have non-trivial startup cost #12544

Open
rvagg opened this issue Oct 2, 2024 · 2 comments

Comments

@rvagg
Copy link
Member

rvagg commented Oct 2, 2024

On larger nodes, particularly archival nodes, the Lotus startup time is very expensive. Logging out of badger suggests that it gets done with loading its indexes relatively fast but then there's a significant wait until the node is ready. e.g. we know that FEVM archival nodes take at least 30 minutes to properly start up. This is less pronounced but still noticeable on splitstore nodes.

For the purpose of diagnosing and for helping users understand what's going on, it would be helpful to info-log the start process of the various modules that do more than just instantiate. Various modules have startup procedures that they need to undertake before they are "ready", so we should log the start and the ready times of these.

This might be a matter of walking through node/builder_chain.go and looking at each of the modules and considering whether more logging should be added to them or not.

@github-project-automation github-project-automation bot moved this to 📌 Triage in FilOz Oct 2, 2024
@rjan90
Copy link
Contributor

rjan90 commented Oct 22, 2024

How much overlap is it between this issue ticket, and the one written up here: #12568 ?

@rvagg
Copy link
Member Author

rvagg commented Nov 20, 2024

Not really any overlap, this one asks for logging so we can see what's going on. I suspect once we start this we find out that most of it is not useful information so they should be debug-level but there are some modules that are quite heavy for which info-level would be helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 📌 Triage
Development

No branches or pull requests

2 participants