Elasticsearch Cluster Monitor

A simple dockerized Django-based Elasticsearch monitor. Have all your clusters at one place and immediately know if any one of them is down.

Information you get:

1. Is your Elasticsearch Cluster down?
2. Is your Ingest pipeline/logstash working properly?
3. What is the health of the cluster? (Green/Yellow/Red)

Instructions:

DO NOT MISS THE FOLLOWING STEPS

1. Clone the repo

   $ git clone https://github.com/govind-menon110/django-elasticsearch-monitor-dashboard.git

2. Configure the env variables

   $ cd django-elasticsearch-monitor-dashboard

   $ vi .env

   SECRET_KEY=5(15ds+i2+%ik6z&!yer+ga9m=e%jcqiz_5wszg)r-z!2--b2d #Change this
   DB_NAME=postgres #Note this
   DB_USER=postgres 
   DB_PASS=postgres
   DB_SERVICE=postgres
   DB_PORT=5432
   SU_NAME=admin #Change this
   SU_PASS=Admin123@ #Change this
   SU_EMAIL=admin@test.com

   $ vi .env.db

   POSTGRES_USER=postgres
   POSTGRES_PASSWORD=postgres
   POSTGRES_DB=postgres #Same as DB_Name

3. Configure the docker-compose file

   $ vi docker-compose.yml

   version: '3.7'
   
   services:
   web:
       restart: always
       build:
           context: ./django_es
           dockerfile: Dockerfile
       expose:
       - "8000"
       links:
       - postgres:postgres
       volumes:
       - web-static:/static
       env_file: .env
       command: sh -c "/usr/local/bin/gunicorn django_es.wsgi:application -w 3 -b :8000 && /bin/python manage.py collectstatic --no-input"
   
   nginx:
       restart: always
       build: ./nginx
       ports:
       - "80:80"
       volumes:
       - web-static:/static
       links:
       - web:web
   
   postgres: # Must be same as DB_Name
       restart: always
       image: postgres:latest
       ports:
       - "5432:5432"
       env_file:
       - ./.env.db
       volumes:
       - pgdata:/var/lib/postgresql/data/
   
   volumes:
   web-static:
   pgdata:

4. Update the nginx config if you need to

5. Fill in details of your elasticsearch cluster (I assume you are still in the root directory of the project)

   $ vi ./django_es/config.json

   If for example your elasticsearch cluster has the following properties:

   1. Name: My_ES
   2. ES_URL: http://1.1.1.1:9200
   3. Kibana_URL: http://1.1.1.1:5601
   4. Query Index: /logstash-*/_search?pretty or 'x' # Note that the url slug must begin with '/'
   5. Kibana_Filter: URL of Dashboard or 'x'
   6. doc_limit: INTEGER (Baseline of logs that one generally observes for half hour)
   7. ES_DESC: Description of ES

   The config.json would look like the following:

   [
   {
       "id":"1",
       "es_name":"My ES",
       "es_url":"http://1.1.1.1:9200",
       "kibana_url":"http://1.1.1.1:5601",
       "query_index":"/logstash-*/_search?pretty",
       "kibana_filter":"http://1.1.1.1:5601",
       "doc_limit":"1000",
       "es_desc":"This is my ES for testing the ES Monitor application"
   },
   {
       "id":"2",
       "es_name":"YOUR_ES_NAME",
       "es_url":"https://ES_URL",
       "kibana_url":"KIBANA_URL",
       "query_index":"/ES_INDEX-*/_search?pretty",
       "kibana_filter":"DASHBOARD/CUSTOM_SEARCH URL",
       "doc_limit":"0",
       "es_desc":"DESCRIPTION"
   }
   ]

6. Although not necessary, do customize the dashboad according to your needs through the template files base.html and home.html located here

Running the Application:

1. Install Docker and Docker-compose according to your system

   $ sudo apt install docker docker-compose -y

   OR

   $ sudo yum install docker docker-compose -y

2. Start docker daemon

   $ systemctl start docker

3. Run docker-compose

   $ sudo `which docker-compose` up -d

   1. On some systems there may be a this version of docker-compose unsupported error. Please refer to this document for the same
   2. Some systems may face a permission denied error in docker which is why sudo is added. Reference this discussion to solve the same

4. Migrate the database

   $ sudo `which docker-compose` run -d web python manage.py migrate

5. Wait for 120s (> 2mins) or so

Your dashboard would be ready!

Change config after deployment:

If you wish to add/remove Elasticsearch clusters or change anything in the config.json file or in the template files, then, after updating the same in the folder on host system, run the following to reflect the changes in docker as well:

$ sudo `which docker-compose` up -d --build --force-recreate

References:

1. Dockerizing Django Blog Post 1
2. Dockerizing Django Blog Post 2
3. Dockerizing Django Blog Post 3 and its github repo
4. Dockerizing Django Blog Post 4
5. Rebuilding Docker container after config change
6. Performing Migrations in Django Docker container
7. Creating a default superuser in Django Docker containercreate-django-super-user-in-a-docker-container-without-inputting-password
8. Clean Restart of Docker Containers
9. Executing multiple commands in docker-compose