Add server health monitoring to detect and auto-restart when interval server loses connection

[codegen-sh]

Overview

This PR adds a comprehensive server health monitoring system to detect and automatically recover from the "Nothing here yet" issue that occurs when the interval server loses connection to hosts.

Key Changes

1. New Server Health Monitoring System:

   • Added serverHealth.ts with health checks that monitor the server's state
   • Implements auto-restart capability when the server is detected to be in an unhealthy state
   • Uses a cooldown period to prevent restart loops

2. Enhanced Connection Monitoring:

   • Improved logging for connection creation and closure
   • Added detailed tracking of ping failures
   • Better diagnostics for connection issues

3. Automatic Recovery:

   • Server will automatically restart itself when it detects a state that would cause the "Nothing here yet" issue
   • Properly cleans up database state before restarting to ensure clean reconnection

How It Works

The system performs periodic health checks that:

1. Verify database connectivity
2. Compare connected hosts with database records
3. Check for inconsistencies between WebSocket connections and database state
4. Automatically restart the server when multiple consecutive checks fail

This should eliminate the need for manual restarts when the server gets into the state where it shows "Nothing here yet" in the web UI.

Testing

This can be deployed to Railway and will automatically monitor itself. The logs will show detailed information about the server's health state and any actions taken to recover from issues.

Fixes SW-49

---

💻 View my work • 👤 Initiated by @zeevl • About Codegen
⛔ Remove Codegen from PR • 🚫 Ban action checks

[linear]

SW-49 interval server keeps losing connection?

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Join our Discord community for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.