Elastic Community: Local Testing with Logstash AMA with Rich Horace

Here are the resources from the Elastic Community Event on September 21, 2021
Description: How do you ensure your logstash configurations work as expected before ingesting into an elastisearch cluster? Test before you ingest!

Just Enough Docker

Configuration Examples Walkthrough

Just Enough Docker

The Elastic Stack version is controllered in the .env file. If the image is not already local, docker will pull down the official Elastic docker image. Docker Containers can be launched in detached mode by adding -d after up.

There are three docker-compose files:
docker-compose.yml:
Consider this configuration purely for reviewing the latest Elastic Stack features. This will launch Kibana and Elasticsearch containers with an ephermeral environment based on the version specified in .env and any changes will not be save after the environment is stopped.

# Launch containers in detached mode with docker-compose.yml as default
docker-compose up -d

docker-compose-persistent.yml:
This configuration will launch Kibana and Elasticsearch containers and create a docker volume that will persist data and changes after the containers are stopped with down or stop commands.

# Launch containers in detached mode with -f to specify an alternate docker-compose file
docker-compose -f docker-compose-persistent.yml up -d

# Will stop containers
docker-compose -f docker-compose-persistent.yml stop

# Will stop and remove containers and -v is used to specify deletion the docker volume
docker-compose -f docker-compose-persistent.yml down -v

docker-compose-ingest.yml:
This configuration will launch Kibana, Elasticsearch and Logstash containers create a docker volume that will persist data and changes after the containers are stopped with down or stop commands.

# Launch containers in detached mode with -f to specify an alternate docker-compose file
docker-compose -f docker-compose-ingest.yml up -d

# Will stop the kibana container, but keep elasticsearch and logstash running
docker-compose -f docker-compose-ingest.yml stop kibana

# Will stop and remove containers and -v is used to specify deletion the docker volume
docker-compose -f docker-compose-ingest.yml down -v

# Will only launch logstash container based on the configuration for the logstash service in the docker-compose file.
docker-compose -f docker-compose-ingest.yml up -d logstash

Other useful docker commands

# List running containers
docker ps

# Docker exec you can access a running container. This example will access logstash container from the docker-compose file
docker exec -it logstash_ama_logstash_1 bash

# Docker run will allow you to specify a docker image and remove the container once you exit the bash session.
docker run --rm -it docker.elastic.co/logstash/logstash:7.14.1 bash

Additional docker resources:

• Docker Office Documents:
• Elastic Logstash Docker:
• Elastic Elasticearch Docker:
• Elastic Kibana Docker:

Configuration Examples Walkthrough

This walkthrough will be driven by multiple logstash pipelines with examples of different configurations that will be enabled by uncommenting and commenting in the elastic-stack/config/logstash/pipelines.yml

To see the results have the logstash container running in the foreground.

docker-compose -f docker-compose-ingest.yml up logstash

Each pipeline example will build on previous example, but can also be run by itself.

• logstash-main.conf: bare config mininum to run logstash without errors.
  
• 1-generate-example: Introduces the Generator Input Plugin which allows to pass message directly into input with count of events to produce. This is great for isolating testing to specific message for debugging.
  
• 2-dissect-example: Introduces File Input Plugin with sincedb and starting position settings. Then use Dissect Filter Plugin and Date Filter Plugin to enrich the data. A site to test Dissect Configuration Lastly, use File Output Plugin with codec setting to producing a json output file.
  
• 3-more-complicated-example: Introduces the Grok Filter Plugin and conditionals.
  
• 4-multiple-config-example: Takes logstash.conf from Example 3 and creates seperating files for Input/Filter/Output
  -- 100-input.conf
  -- 500-filter.conf
  -- 900-output.conf
  
• 5-output-es: Pulls it all together with multiple file inputs, conditional filtering and outputing to multiple indices using Elasticsearch Output Plugin
  

Run all three container for 5-output-es

docker-compose -f docker-compose-ingest.yml up 

Access Kibana localhost:5601

Thanks a wrap!