256 integrate health check and monitoring for the server e5 #305
+949
−92
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
Ngha-Boris
commented
Feb 3, 2025
•
Ngha-Boris
commented
Ngha-Boris
requested review from
Hermann-Core,
IngridPuppet,
Blindspot22,
Christiantyemele and
ndefokou
Blindspot22
requested changes
Feb 4, 2025
…6-integrate-health-check-and-monitoring-for-the-server-e5
Hermann-Core
requested changes
Feb 7, 2025
Hermann-Core
approved these changes
Feb 12, 2025
…6-integrate-health-check-and-monitoring-for-the-server-e5
docs/monitoring-and-health-checks.md
Outdated
Comment on lines
22
to
59
| ``` | ||
| # | ||
| ### Metrics Enpoint | ||
| - **Route:** ```/metrics``` | ||
| - **Description:** Exposes Prometheus-compatible metrics, which are scraped by Prometheus for monitoring. | ||
| - **Response:** Raw Prometheus metrics data (e.g., CPU, memory usage). | ||
| # | ||
| ### Prometheus Metrics | ||
|
|
||
| Prometheus collects the following metrics: | ||
| - **CPU Usage:** Tracks CPU time spent in different modes (idle, system, user). Alert if usageexceeds 80% for 2 minutes. | ||
| - **Memory Usage:** Monitors available memory. Alert if usage exceeds 85% for 2 minutes. | ||
| - **Disk Usage:** Monitors disk space. Alert if disk space is below 20%. | ||
| - **API Response Time:** Measures API response times. Alert if the 95th percentile response time exceeds 1 second. | ||
| - **HTTP Errors:** Tracks failed HTTP requests. Alert if failure rate exceeds 5% over 5 minutes. | ||
| # | ||
| ### Alerting | ||
|
|
||
| Prometheus triggers alerts based on defined conditions: | ||
|
|
||
| **Important Alerts** | ||
| - **InstanceDown:** Triggered if the server is unreachable for 30 seconds. | ||
| - **HighCPUUsage:** Triggered if CPU usage is over 80% for 2 minutes. | ||
| - **HighMemoryUsage:** Triggered if memory usage exceeds 85% for 2 minutes. | ||
| - **SlowAPIResponse:** Triggered if API response times are too slow (95th percentile > 1s). | ||
| - **DiskSpaceLow:** Triggered if disk space is below 20%. | ||
| Alerts are routed to Alertmanager, which handles notification via "any" application(e.g slack discord etc) (configured in ```alertmanager.yml```). | ||
| # | ||
| ### Docker Setup | ||
| The monitoring stack is managed using Docker Compose. It includes: | ||
| - **Prometheus:** Collects metrics. | ||
| - **Grafana:** Visualizes metrics. | ||
| - **Alertmanager:** Sends notifications based on alerts. | ||
| - **Node Exporter:** Exposes system metrics (CPU, memory, disk usage). | ||
| ### Build and run | ||
| ```bash | ||
| docker compose up --build | ||
| ``` |
There was a problem hiding this comment.
Can you double check the markdown here? When it is all red, it generally means the syntax is not correct. Maybe the hash signs with remaining lines empty are not necessary...
Comment on lines
+1
to
+2
| // src/main.rs | ||
| use axum::routing::get; |
There was a problem hiding this comment.
What's the purpose of adding this comment?
There was a problem hiding this comment.
That was an error i just removed it this morning. I will soon push.
Comment on lines
+8
to
+9
| mod health; | ||
| mod metrics; |
There was a problem hiding this comment.
I'd prefer you simply import these modules than creating two hierarchies:
use didcomm_mediator::{app, health, metrics};
docs/monitoring-and-health-checks.md
Outdated
Comment on lines
17
to
21
| - **Response:** | ||
| ```json | ||
| { | ||
| "Status": "OK", | ||
| } |
There was a problem hiding this comment.
I could find that GitHub's problem is the level of indentation at this point, and possibly the extra comma too.
Suggested change
| - **Response:** | |
| ```json | |
| { | |
| "Status": "OK", | |
| } | |
| - **Response:** | |
| ```json | |
| { | |
| "Status": "OK" | |
| } |
There was a problem hiding this comment.
I just fixed the issue but i am still working on updating the document.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.