Merge pull request #43 from xneelo/sb_filebeat_ilm

Support filebeat ilm configuration

CHANGELOG.md

@@ -1,41 +1,48 @@
-##2020-11-20 - Release 2.0.1
-###Summary
+## [2.1.0](https://github.com/xneelo/hetzner-filebeats/tree/2.1.0) (2020-11-25)
+
+### Add support configuring filebeat ilm
+Filebeat 7.0 sets ilm on by default so we need to be able to configure this, the new config options are:
+* [setup.ilm.enabled](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-option)
+* [setup.ilm.rollover_alias](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-rollover_alias-option)
+* [setup.ilm.pattern](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-pattern-option)
+* [setup.ilm.policy_name](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-policy_name-option)
+* [setup.ilm.policy_file](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-policy_file-option)
+* [setup.ilm.check_exits](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-check_exists-option)
+* [setup.ilm.overwrite](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-overwrite-option)
+
+## [2.0.1](https://github.com/xneelo/hetzner-filebeats/tree/2.0.1) (2020-11-20)
 
 Minor tweak to enable/disable module
-* Only attempt to enable if it is disabled, only disable if it is enabled
+- Only attempt to enable if it is disabled, only disable if it is enabled
 
-##2020-11-03 - Release 2.0.0
-###Summary
+## [2.0.0](https://github.com/xneelo/hetzner-filebeats/tree/2.0.0) (2020-11-03)
 
-Major version change 6.x => 7.x
-* https://www.elastic.co/guide/en/beats/libbeat/7.9/breaking-changes-7.0.html#breaking-changes-7.0
+### [Major version change 6.x => 7.x](https://www.elastic.co/guide/en/beats/libbeat/7.9/breaking-changes-7.0.html#breaking-changes-7.0)
 
-##2019-12-09 - Release 1.0.4
-###Summary
+## [1.0.4](https://github.com/xneelo/hetzner-filebeats/tree/1.0.4) (2019-12-09)
 
 Ensure Apt update runs before package installation
-* Credit to Thodoris Sotiropoulos - theosotr
+- Credit to Thodoris Sotiropoulos - theosotr
 
-##2019-06-28 - Release 1.0.3
-###Summary
+## 2019-06-28 - Release 1.0.3
+### Summary
 
 NFR: Fix README format
 
-##2019-06-27 - Release 1.0.2
-###Summary
+## 2019-06-27 - Release 1.0.2
+### Summary
 
 Add support for the following config options to logstash output
 * ttl
 * bulk_max_size
 
-##2019-04-25 - Release 1.0.0
-###Summary
+## 2019-04-25 - Release 1.0.0
+### Summary
 
-Update input type setting in flebeats configuration to conform to 6.X syntax:
-https://www.elastic.co/guide/en/beats/libbeat/6.2/breaking-changes-6.0.html#breaking-changes-types
+Update input type setting in filebeats configuration to conform to [6.X syntax](https://www.elastic.co/guide/en/beats/libbeat/6.2/breaking-changes-6.0.html#breaking-changes-types)
 
-##2018-08-16 - Release 0.3.0
-###Summary
+## 2018-08-16 - Release 0.3.0
+### Summary
 
 Update both logstash output and elasticsearch output for filebeat 5.X syntax
 * Rename params to reflect which filebeat output they affect
@@ -50,88 +57,88 @@ Update both logstash output and elasticsearch output for filebeat 5.X syntax
 *     json.add_error_key
 *     Credit to https://github.com/hundredacres
 
-##2017-09-17 - Release 0.2.5
-###Summary
+## 2017-09-17 - Release 0.2.5
+### Summary
 
 Remove unintended blank line in filebeats.yml
 
-##2017-06-05 - Release 0.2.4
-###Summary
+## 2017-06-05 - Release 0.2.4
+### Summary
 
 Add exclude_line config option
 
-##2017-06-05 - Release 0.2.3
-###Summary
+## 2017-06-05 - Release 0.2.3
+### Summary
 
 Add include_line config option
 
-##2017-02-03 - Release 0.2.2
-###Summary
+## 2017-02-03 - Release 0.2.2
+### Summary
 
 Add bootstrapping of filebeat service
 
-##2017-02-03 - Release 0.2.1
-###Summary
+## 2017-02-03 - Release 0.2.1
+### Summary
 
 Support puppetlabs apt module 2.3
 * Bump dependency on puppetlabs apt
 
-##2017-02-02 - Release 0.1.6
-###Summary
+## 2017-02-02 - Release 0.1.6
+### Summary
 
 Add support for prospector fields
 * Added new prospector option, fields
 * Thanks for the contribution @belskiiartem
 
-##2015-04-13 - Release 0.1.5
-###Summary
+## 2015-04-13 - Release 0.1.5
+### Summary
 
 Adding logstash output
 * Added new parameters for logstash output
 * Added param to specify index to ship for both elasticsearch and logstash
 * Added service_state param to allow for overriding
 * Added loadbalancing param for logstash output
 
-##2015-04-08 - Release 0.1.4
-###Summary
+## 2015-04-08 - Release 0.1.4
+### Summary
 
 Fix documentation issues
 * Minor indetation in documentation
 
-##2015-04-08 - Release 0.1.3
-###Summary
+## 2015-04-08 - Release 0.1.3
+### Summary
 
 Fix documentation issues
 * Fix code examples in README, still trying to get a hang of this
 
-##2015-04-08 - Release 0.1.2
-###Summary
+## 2015-04-08 - Release 0.1.2
+### Summary
 
 Fix documentation issues
 * Fix code examples in README
 
-##2015-04-07 - Release 0.1.1
-###Summary
+## 2015-04-07 - Release 0.1.1
+### Summary
 
 Minor update to include multiple prospectors
 * This update allows for an array of hashes to configure multiple prospectors
 * This includes setting of input_type and document type in each hash
 
-##2015-04-06 - Release 0.1.0
-###Summary
+## 2015-04-06 - Release 0.1.0
+### Summary
 
 First major release with tested (on our environment) log exporting
 * Added logging options for filebeats
 
-##2015-04-01 - Release 0.0.13
-###Summary
+## 2015-04-01 - Release 0.0.13
+### Summary
 
 * Minor bug fixes
 * Added TLS/SSL support
 * Added protocol option
 
-##2015-03-24 - Release 0.0.9
-###Summary
+## 2015-03-24 - Release 0.0.9
+### Summary
 
 First working release of basic puppet filebeats module
 * This version only included very basic funcionality for installing and configuring file beats on Debian Wheezy

README.md

@@ -87,6 +87,10 @@ A array containing the hostname/s of your elasticsearch host/s used for send the
 to Elasticsearch by using the Elasticsearch HTTP API.
 If left empty then all other elasticsearch options are ignored
 
+#`elasticsearch_proxy_host`
+
+A string containing the hostname of your proxy host used for load balancing your cluster.
+
 #`elasticsearch_username`
 
 The username filebeats should use to authenticate should your cluster make use of shield
@@ -103,6 +107,10 @@ A string containing the protocol used by filebeats, defaults to http.
 
 A string that specifies the index to use for the elasticsearch output, defaults to '[filebeat-]YYYY.MM.DD' as per the package.
 
+#`elasticsearch_ilm`
+
+A boolean that specifies whether to enable Elastic's ILM option, defaults to false
+
 #`elasticsearch_ssl_certificate_authorities`
 
 An array of Strings that specifies paths to Certificate authority files.
@@ -135,6 +143,37 @@ A string that specifies the path to the template file
 
 An array of Strings that specifies which logs the filebeats application must export.
 
+#`ilm_check_exits`
+
+A boolean when set to false, disables the check for an existing lifecycle policy. The default is true. You need to disable 
+this check if the Filebeat user connecting to a secured cluster doesn’t have the read_ilm privilege
+
+#`ilm_enabled`
+
+A string that Enables or disables index lifecycle management on any new indices created by Filebeat. Valid values are
+true, false, and auto (because auto is also an option, this can be a puppet Boolean)
+
+#`ilm_overwrite`
+
+A boolean when set to true, the lifecycle policy is overwritten at startup
+
+#`ilm_pattern`
+
+A string that specifies the rollover index pattern. Date math is supported in this setting
+
+#`ilm_policy_file`
+
+A string that specifies the path to a JSON file that contains a lifecycle policy configuration. Use this setting to load your
+own lifecycle policy
+
+#`ilm_policy_name`
+
+A string that specifies the name to use for the lifecycle policy
+
+#`ilm_rollover_alias`
+
+A string that specifies the index lifecycle write alias name
+
 #`log_settings`
 
 A puppet Hash containing log level ('debug', 'warning', 'error' or 'critical'), to_syslog(true/false), path('/var/log/filebeat'), keepfiles(7), rotateeverybytes(10485760), name(filebeats.log)
@@ -237,7 +276,7 @@ Does not support all options available to filebeats configuration.
 If you're running a Filebeat version lower than 7.x, e.g 6.8.10. You need to install the `pre-v7.x` release in order for the module
 to be compatible - https://github.com/xneelo/hetzner-filebeats/releases
 
-If you're running Filebeat version 7 and up. You can install the `production` branch of this module or refer to the releases section again.
+If you're running Filebeat version 7 and up, release 2.0.0 and newer is supported.
 
 ## Development
 

manifests/config.pp

@@ -1,9 +1,5 @@
 #Basic filebeat configuration
-# Params:
-#   export_log_paths: Array of Strings - path to log files to export.
-#   shield_username: String - Username for shield authentication.
-#   shield_password: String - Password for shield authentication.
-#More details on shield: https://www.elastic.co/guide/en/shield/current/getting-started.html
+#More details: https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html
 
 class filebeats::config (
   Array   $elasticsearch_hosts,
@@ -33,6 +29,13 @@
   Hash    $modules,
   String  $modules_conf_dir,
   Array   $inputs,
+  Boolean $ilm_check_exits,
+  String  $ilm_enabled,
+  Boolean $ilm_overwrite,
+  String  $ilm_pattern,
+  String  $ilm_policy_file,
+  String  $ilm_policy_name,
+  String  $ilm_rollover_alias,
 ){
   $config_path = $filebeats::params::config_path
 
@@ -51,14 +54,18 @@
   if empty($inputs) {
     validate_array($export_log_paths)
 
-    $inputs_array =  [{'paths'         => $export_log_paths,
-                            'input_type'    => 'log',
-                            'doc_type' => 'log'
-                          }]
+    $inputs_array =  [{ 'paths'      => $export_log_paths,
+                        'input_type' => 'log',
+                        'doc_type'   => 'log'
+                      }]
   } else {
     $inputs_array = $inputs
   }
 
+  if ! ($ilm_enabled in [ 'auto', 'true', 'false' ]) {
+    fail("Parameter \$ilm_enabled with content '${ilm_enabled}': must be one of [ 'auto', 'true', 'false' ]")
+  }
+
   file {"${config_path}/filebeat.yml":
     ensure  => present,
     owner   => root,

manifests/init.pp

@@ -11,7 +11,8 @@
 # * `export_log_paths`
 # An array of Strings that specifies which logs the filebeats application must export.
 # * `inputs`
-# An array of Hashes that specifies which groups of inputs (formally known as prospectors) log entries the filebeats application must export.
+# An array of Hashes that specifies which groups of inputs (formally known as prospectors) log entries the filebeats application 
+#  must export.
 # * `elasticsearch_username`
 # The username filebeats should use to authenticate should your cluster make use of elasticsearch
 # * `elasticsearch_password`
@@ -21,11 +22,11 @@
 # If left empty it will default to exporting logs to your local host on port 9200.
 # * `elasticsearch_protocol`
 # A string containing the protocl used by filebeats to send logs.
-# * `ssl_certificate_authorities`
+# * `elasticsearch_ssl_certificate_authorities`
 # An array of Strings that specifies paths to Certificate authority files.
-# * `ssl_certificate`
+# * `elasticsearch_ssl_certificate`
 # A String that specifies a path to your hosts certificate to use when connecting to elasticsearch.
-# * `ssl_certificate_key`
+# * `elasticsearch_ssl_certificate_key`
 # A String that specifies a path to your hosts certificate key to use when connecting to elasticsearch.
 # * `log_settings`
 # A puppet Hash containing log level ('debug', 'warning', 'error' or 'critical'),
@@ -43,6 +44,12 @@
 # *`logstash_bulk_max_size`
 # A number representing the maximum number of events to bulk in a single Logstash request, e.g 2048
 #   Setting this to zero or negative disables the splitting of batches.
+# * `logstash_ssl_certificate_authorities`
+# An array of Strings that specifies paths to Certificate authority files when connecting to logstash.
+# * `logstash_ssl_certificate`
+# A String that specifies a path to your hosts certificate to use when connecting to logstash.
+# * `logstash_ssl_certificate_key`
+# A String that specifies a path to your hosts certificate key to use when connecting to logstash.
 # *`logstash_ttl`
 # A string that specifies the Time To Live for a connection to Logstash, you must use a elastic duration e.g. '5m', '1h', '45s'
 #  see https://www.elastic.co/guide/en/beats/libbeat/master/config-file-format-type.html#_duration
@@ -52,6 +59,25 @@
 # A string that specifies the index to use for the elasticsearch output, defaults to '[filebeat-]YYYY.MM.DD' as per the package.
 # *`elasticsearch_ilm`
 # A boolean that specifies whether to enable Elastic's ILM option, defaults to false
+# *`ilm_check_exits`
+# A boolean when set to false, disables the check for an existing lifecycle policy. The default is true. You need to disable 
+#  this check if the Filebeat user connecting to a secured cluster doesn’t have the read_ilm privilege
+# *`ilm_enabled`
+# A string that Enables or disables index lifecycle management on any new indices created by Filebeat. Valid values are
+#  true, false, and auto (because auto is also an option, this can be a puppet Boolean)
+# *`ilm_overwrite`
+# A boolean when set to true, the lifecycle policy is overwritten at startup
+# *`ilm_pattern`
+# A string that specifies the rollover index pattern. Date math is supported in this setting
+# Because the variable can use %, you must cater for it like this "%%{}{now/d}-000001"
+# *`ilm_policy_file`
+# A string that specifies the path to a JSON file that contains a lifecycle policy configuration. Use this setting to load your
+#  own lifecycle policy
+# *`ilm_policy_name`
+# A string that specifies the name to use for the lifecycle policy
+# *`ilm_rollover_alias`
+# A string that specifies the index lifecycle write alias name
+# Because the variable can use %, you must cater for it like this "filebeat-%%{}{[agent.version]}"
 #
 # Example
 # --------
@@ -102,6 +128,13 @@
   $logstash_worker                           = $filebeats::params::logstash_worker,
   $modules                                   = $filebeats::params::modules,
   $modules_conf_dir                          = $filebeats::params::modules_conf_dir,
+  $ilm_check_exits                           = $filebeats::params::ilm_check_exits,
+  $ilm_enabled                               = $filebeats::params::ilm_enabled,
+  $ilm_overwrite                             = $filebeats::params::ilm_overwrite,
+  $ilm_pattern                               = $filebeats::params::ilm_pattern,
+  $ilm_policy_file                           = $filebeats::params::ilm_policy_file,
+  $ilm_policy_name                           = $filebeats::params::ilm_policy_name,
+  $ilm_rollover_alias                        = $filebeats::params::ilm_rollover_alias,
   $inputs                                    = $filebeats::params::inputs,
   $service_bootstrapped                      = $filebeats::params::service_bootstrapped,
   $service_state                             = $filebeats::params::service_state,
@@ -143,6 +176,13 @@
     modules                                   => $modules,
     modules_conf_dir                          => $modules_conf_dir,
     inputs                                    => $inputs,
+    ilm_check_exits                           => $ilm_check_exits,
+    ilm_enabled                               => $ilm_enabled,
+    ilm_overwrite                             => $ilm_overwrite,
+    ilm_pattern                               => $ilm_pattern,
+    ilm_policy_file                           => $ilm_policy_file,
+    ilm_policy_name                           => $ilm_policy_name,
+    ilm_rollover_alias                        => $ilm_rollover_alias,
   }
 
   Class['::filebeats::params']-> Class['::filebeats::config']