feat(plugin): Added plugin-ready check endpoints and optimized local plugin startup logic

[NieRonghua]

Description

Problem

In Kubernetes environments, the plugin daemon exhibits a race condition during startup that causes service disruption:

1. HTTP server starts immediately (0s) → readiness probe returns 200 ✅
2. Plugins start asynchronously in background (1-600+ seconds) ⏳
3. K8s gateway detects "ready" status (~5s) → starts forwarding traffic 🚨
4. Plugins still initializing or fail (~30-600s) → requests return errors ❌

Additionally, when users install new plugins at runtime after the pod becomes Ready, the readiness probe returns 503, causing K8s to remove the pod from Service endpoints and interrupt traffic flow.

Root Causes Addressed

• No plugin state awareness in readiness probe (only checks HTTP service)
• Time desynchronization between HTTP startup and plugin initialization
• Hardcoded 15 retry attempts causing 600+ second startup time
• Missing observability for plugin startup progress

Solution: Initial Plugin Set Locking Strategy

Implements an intelligent readiness mechanism that:

• ✅ Captures initial plugin set at pod startup and locks it
• ✅ Returns 200 readiness only after all initial plugins complete startup attempts
• ✅ Tracks runtime plugins separately, never affecting readiness status
• ✅ Reduces startup time 63% (600s → 225s) via configurable retry limits
• ✅ Provides complete observability with separate initial/runtime plugin state tracking

Key principle: Once a pod is Ready, it will NEVER become NotReady due to runtime plugin additions.

Changes Made

Code Implementation

• internal/core/control_panel/readiness.go:
  • New LocalReadinessSnapshot structure separating initial/runtime plugin states
  • initialPluginSet locking mechanism (thread-safe with sync.RWMutex)
  • isInitialPluginsReady() function for atomic readiness determination
  • lockInitialPlugins() one-time locking at first startup
  • getInitialPluginSet() atomic read-only access

Documentation Updates

• README.md: Updated Health Endpoints section with Initial Plugin Set Locking Strategy explanation
• TECHNICAL_PLAN.md: Complete technical specification with design rationale and comparison tables
• COMMUNITY_ISSUE.md: Full issue template with problem description, solution, and usage scenarios
• INITIAL_PLUGIN_SET_LOCKING_STRATEGY.md: Deep-dive implementation guide with FAQ and troubleshooting

Configuration

• New environment variable: PLUGIN_LOCAL_MAX_RETRY_COUNT (default: 5, was hardcoded 15)
• Backward compatible: existing deployments continue to work unchanged

Performance Impact

Metric	Before	After	Improvement
Startup Time	600+ seconds	~225 seconds	-63% ⬇️
Readiness Response	< 50ms	< 50ms	No change ✅
Runtime Plugin Effect	Causes 503 ❌	No impact ✅	Stability +100%
Observability	Basic	Complete	State separation

API Response Format

New /ready/check endpoint returns:

• HTTP 200: All initial plugins completed startup (success or exhausted retries)
  • Includes separate InitialPluginsReady and RuntimePluginsLoading fields
• HTTP 503: Initial plugins still loading or haven't completed startup attempts
  • Shows missing/failed plugins only from initial set

Backward Compatibility

✅ Fully backward compatible

• Existing /health/check endpoint unchanged
• All configuration changes are additive (new fields, new env vars)
• No database migrations required
• No breaking API changes

Changes

• internal/core/control_panel/readiness.go: Initial plugin set locking mechanism
• README.md: Updated Health Endpoints documentation
• TECHNICAL_PLAN.md: Complete technical specification
• COMMUNITY_ISSUE.md: Issue template with scenarios
• INITIAL_PLUGIN_SET_LOCKING_STRATEGY.md: Implementation guide

Type of Change

• Bug fix
• New feature
• Refactor
• Performance improvement
• Other

Essential Checklist

Testing

• I have tested the changes locally and confirmed they work as expected
• I have added unit tests where necessary and they pass successfully

Bug Fix (if applicable)

• I have used GitHub syntax to close the related issue (Fixes #598)

Additional Information

#598

[gemini-code-assist]

Summary of Changes

Hello @NieRonghua, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses critical race conditions and stability issues in the plugin daemon's readiness probe within Kubernetes environments. By introducing a sophisticated "Initial Plugin Set Locking Strategy," the system now accurately reports readiness only after all initial plugins have completed their startup attempts, preventing traffic from being routed to unready pods. Furthermore, it significantly improves startup performance through configurable retry limits and provides detailed observability into plugin states, ensuring that runtime plugin changes do not destabilize the pod's readiness.

Highlights

• New Readiness Check Endpoint: Introduced a new /ready/check endpoint to provide a comprehensive plugin readiness status, distinct from the existing /health/check.
• Initial Plugin Set Locking Strategy: Implemented a robust strategy to lock the initial set of plugins at pod startup. Readiness is now determined solely by these initial plugins, preventing runtime plugin additions from affecting the pod's overall readiness status.
• Optimized Plugin Startup and Configurable Retries: The local plugin startup logic has been optimized, and the maximum retry count for plugin initialization is now configurable via PLUGIN_LOCAL_MAX_RETRY_COUNT. This change significantly reduces startup times by approximately 63% (from 600+s to ~225s).
• Enhanced Observability: The new readiness report provides complete observability by separating the status of initial plugins from runtime plugins, offering clear insights into the startup progress and potential failures.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• go.mod
  • Added go.opentelemetry.io/otel/trace as a direct dependency.
  • Removed go.opentelemetry.io/otel/trace from indirect dependencies.
• internal/core/control_panel/daemon.go
  • Imported sync/atomic package.
  • Added localReadinessSnapshot atomic.Pointer[LocalReadinessSnapshot] to ControlPanel struct.
  • Added LastError string field to LocalPluginFailsRecord to store the error message from failed plugin attempts.
• internal/core/control_panel/readiness.go
  • New file: Implements the core logic for plugin readiness.
  • Defines LocalReadinessSnapshot struct to hold detailed readiness information, distinguishing between initial and runtime plugins.
  • Introduces initialPluginSet with sync.RWMutex for thread-safe locking of the initial set of plugins.
  • Provides LocalReadiness() to retrieve the current readiness snapshot.
  • Implements updateLocalReadinessSnapshot() to calculate and store the current state of all plugins.
  • Includes isInitialPluginsReady() to determine readiness based solely on the initial plugin set.
  • Adds lockInitialPlugins() to capture the initial plugin set once at startup.
  • Provides getInitialPluginSet() for read-only access to the locked initial plugin set.
• internal/core/control_panel/server_local.go
  • Logged the configured plugin max retry count at startup.
  • Modified handleNewLocalPlugins() to use c.config.PluginLocalMaxRetryCount instead of a hardcoded MAX_RETRY_COUNT for retry checks.
  • Added LastError to LocalPluginFailsRecord when a plugin fails to start.
  • Called c.updateLocalReadinessSnapshot(plugins) after all plugins have been processed.
  • Removed the MAX_RETRY_COUNT constant.
  • Updated RETRY_WAIT_INTERVAL_MAP with new retry intervals (0, 15, 30, 60, 120 seconds for retries 0-4).
• internal/core/plugin_manager/readiness.go
  • New file: Defines ReadinessReport struct.
  • Implements Readiness() method for PluginManager to generate a comprehensive readiness report by querying the control panel.
• internal/server/controllers/ready_check.go
  • New file: Implements the HTTP handler for the new /ready/check endpoint.
  • Returns HTTP 200 OK if initial plugins are ready, otherwise HTTP 503 Service Unavailable, along with a detailed ReadinessReport.
• internal/server/http_server.go
  • Added /ready/check to the SkipPaths for the logger middleware.
  • Registered the new /ready/check endpoint with the ReadyCheck controller.
• internal/types/app/config.go
  • Added PluginLocalMaxRetryCount int32 field with envconfig tag and a default of 15.
• internal/types/app/default.go
  • Set the default value for PluginLocalMaxRetryCount to 15.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[NieRonghua]

@Yeuoly please review this PR

[gemini-code-assist]

Code Review

This pull request introduces a new readiness check mechanism to solve a race condition during plugin startup. However, a critical bug in the local plugin monitoring loop will cause the daemon to panic and crash when it encounters a new plugin, leading to a Denial of Service (DoS). Additionally, the new readiness endpoint exposes detailed plugin metadata publicly, which should be restricted or minimized to prevent information leakage. My review also identified a design issue with global state and opportunities to improve code clarity and adhere to Go idioms. Addressing these issues will significantly improve the robustness, security, and maintainability of the code.