BUG: Plugin Daemon Signals Readiness Before Plugins Complete Startup in Kubernetes Deployments

[NieRonghua]

Self Checks

To make sure we get to you in time, please check the following :)

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• "Please do not modify this template :) and fill in all the required fields."

Versions

1. dify-plugin-daemon Version: 0.5.1
2. dify-api Version: 1.11.1

Describe the bug

In Kubernetes environments, the plugin daemon has a race condition during startup:

1. HTTP service starts immediately (0s) → readiness probe gets 200 ✅
2. Plugins start asynchronously in background (1-600+ seconds) ⏳
3. K8s Gateway detects ready status (~5s) → starts forwarding traffic 🚨
4. Plugins still initializing or startup fails (~30-600s) → requests return errors ❌

Root Causes

#	Issue	Impact
1	No plugin state awareness	readiness probe only checks HTTP service, not plugin readiness
2	Time desynchronization	HTTP fast (1-5s), plugins slow (30-600s), async startup
3	Poor observability	Cannot see plugin startup progress or failure reasons
4	Hardcoded retry limit	15 retries fixed in code, startup takes 600+ seconds

Current Behavior

0s      : HTTP server running ✅
         readiness probe → 200 OK (but plugins not ready!)

~5s     : K8s Gateway traffic begins flowing
         Requests fail if plugins still loading

~30-600s: Plugins finally ready (or failed)
         Too late! Traffic already flowing

Business Constraints

• Private cloud deployment - no manual intervention possible
• Partial availability acceptable - service should start even if some plugins fail
• Fast startup required - K8s gateway needs sufficient time to detect true ready state

---

To Reproduce

Steps to reproduce the behavior in Kubernetes:

1. Deploy dify-plugin-daemon to Kubernetes with current readiness probe pointing to /health/check
2. Observe initial readiness probe response time (returns 200 immediately)
3. Wait for plugins to fully load (10-600+ seconds depending on plugin count/type)
4. Monitor traffic during this window:
   • Traffic arrives before plugins fully loaded
   • Requests fail with "plugin not ready" or timeout errors
5. Check logs to see retry attempts are hardcoded at 15 times

---

Expected behavior

• ✅ readiness probe returns 503 while plugins are still starting
• ✅ readiness probe returns 200 only after all plugin startup attempts complete (success or failure after max retries)
• ✅ Configurable retry limit instead of hardcoded 15
• ✅ Visible plugin startup state and error information
• ✅ Startup time reduced from 600s to ~225s (63% improvement)

---

Screenshots
The Kubernetes pod health check has passed, and the pod has started accepting API requests forwarded by the gateway.

However, the plugin in local mode is still starting up, causing the plugin to be called.

Additional context
Dify is deployed via Kubernetes.

[dosubot]

You've correctly identified a real issue: the current readiness probe at /health/check always returns 200 OK as soon as the HTTP server starts, without checking if plugins have finished initializing. This means Kubernetes routes traffic before plugins are ready, causing errors like "plugin not ready" or "plugin runtime not found" during startup windows. The readiness endpoint is implemented in internal/server/controllers/health_check.go and does not consult plugin state or lifecycle at all—it just reports basic service status and metrics, regardless of plugin readiness [source].

Internally, plugin startup state is tracked using the PluginRuntime and PluginRuntimeState structs, which record status (active, launching, pending, etc.), restarts, timestamps, and logs [source]. However, this state is not surfaced in the readiness probe or health endpoint.

There are currently no configuration options or environment variables to control readiness probe behavior or plugin startup retry limits. The retry limit is hardcoded, and the only relevant plugin startup config is PLUGIN_LOCAL_LAUNCHING_CONCURRENT, which controls parallel launches but not readiness or retries [source].

To address this, you would need to modify the /health/check handler to check the state of all plugin runtimes and return 503 until all plugins have either started successfully or failed after their maximum retry attempts. You can access plugin states via the plugin manager and aggregate their statuses. Once all plugins are in a terminal state (active or stopped), return 200; otherwise, return 503 and include plugin startup progress or error info in the response for better observability.

There are no merged PRs or existing config flags for this, so this would require a code change. For distributed/multi-pod setups, be aware that plugin state is not synchronized across pods, which can cause additional "plugin not found" errors unless you implement a distributed lock or run a single replica [source].

If you need a starting point, focus on updating internal/server/controllers/health_check.go to consult plugin states and expose them in the readiness response. This will make the readiness probe accurately reflect true service readiness and help avoid traffic routing before plugins are ready.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}