Docker Swarm Monitoring Stack

An all-in-one monitoring stack for Docker Swarm environments.

Stack Components

• Docker Swarm
• Elastic Stack (Elasticsearch, Logstash, Kibana, Filebeat, Fleet Server, and Elastic Agent)
• Kafka & Zookeeper
• Prometheus & Grafana
• Node Exporter & cAdvisor
• Nginx
• ElastAlert

Features

• "Elastic"-based log monitoring pipeline using the Elastic Stack and Kafka

  • Fully supports global Elasticsearch cluster deployment without hardcoding
  • Full Elasticsearch security integration
  • Integration with Elastic Agent and Fleet
  • Example Logstash configuration included
  • Nginx proxy for Elasticsearch
  • Real-time alerting via ElastAlert

• Docker Swarm node and service monitoring using Prometheus and Grafana

  • Collects Swarm metrics using Node Exporter and cAdvisor
  • Includes two pre-built dashboards

• Shell script for easy monitoring stack control

Setup

Requirements: A Linux environment with Docker, Docker Compose, and an active Docker Swarm cluster. If you're unsure how to set this up, refer to BASIC_GUIDE.md.

1. Increase vm.max_map_count for Elasticsearch

vi /etc/sysctl.conf

# Add or update the following line
vm.max_map_count=262144

# Apply the changes
sysctl -p

2. Configure daemon.json to expose Docker metrics for monitoring containers

vi /etc/docker/daemon.json

# Example configuration
{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

3. Run Filebeat on the server to be monitored

See filebeat/README.md for details.

4. Start the Kafka server

Refer to kafka/README.md.

5. Configure Logstash to receive input from Kafka

Modify the bootstrap_servers setting in logstash.yml to point to your Kafka server's IP.

# logstash/config/logstash.yml

input {
  kafka {
    bootstrap_servers => "your_kafka_server_ip:9094"
    topics => ["test_topic"]
    codec => json
    decorate_events => true
  }
}

6. Set up Elasticsearch security

To use all features, basic security must be enabled in Elasticsearch. Follow the minimal security and basic security guides. Copy the generated certificate to ./elasticsearch/config.

If you do not need security, you can disable it in elasticsearch/config/elasticsearch.yml.

7. Set up Fleet Server security

After enabling basic security, you can create certificates for Fleet Server. Follow the Fleet server setup guide and certificate generation guide. Copy the certificates to ./fleet-server.

Fleet Server is optional. If not needed, you can remove the related configuration.

Running the Monitoring Stack

Use the provided shell script to start, stop, or manage the monitoring stack:

sh monitor_service.sh start
sh monitor_service.sh stop
sh monitor_service.sh restart
sh monitor_service.sh status
sh monitor_service.sh logs {service_name}
sh monitor_service.sh update {service_name}

Access Dashboards

Once all services are running, you can access the following dashboards:

Service	URL	Description
Grafana	http://<your_server_ip>:3000	Default: admin/admin
Kibana	http://<your_server_ip>:5601