Skip to content

Commit

Permalink
rollback: rollback to 2.32.0
Browse files Browse the repository at this point in the history
Signed-off-by: Matthew Fala <[email protected]>
  • Loading branch information
matthewfala committed Mar 29, 2024
1 parent e575760 commit 9a80d21
Show file tree
Hide file tree
Showing 36 changed files with 207 additions and 2,784 deletions.
2 changes: 1 addition & 1 deletion AWS_FOR_FLUENT_BIT_PREVIOUS_STABLE_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.32.0.20240122
2.31.12.20231011
2 changes: 1 addition & 1 deletion AWS_FOR_FLUENT_BIT_STABLE_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.32.0.20240304
2.32.0.20240122
2 changes: 1 addition & 1 deletion AWS_FOR_FLUENT_BIT_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.32.1
2.32.0.20240304
15 changes: 2 additions & 13 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,5 @@
# Changelog

### 2.32.1
This release includes:
* Fluent Bit [1.9.10](https://github.com/fluent/fluent-bit/tree/v1.9.10)
* Amazon CloudWatch Logs for Fluent Bit 1.9.4
* Amazon Kinesis Streams for Fluent Bit 1.10.2
* Amazon Kinesis Firehose for Fluent Bit 1.7.2

Compared to `2.32.0` this release adds:
* Enhancement - Add `aws_fluent_bit_init_ignore_firelens_config` option to ECS init tag. As shown in the [example](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/init-ignore-firelens) and [documentation](https://github.com/aws/aws-for-fluent-bit/tree/mainline/use_cases/init-process-for-fluent-bit#how-to-ignore-the-generated-firelens-configuration), this option tells init to ignore the built-in generated FireLens configuration which allows users to fully override it with their own configuration.
* Enhancement - Upgrade to latest stable Go 1.20.1. Each future AWS for Fluent Bit release will be built with the latest stable Go release. Go is used to build the init process for ECS use cases and the [Go plugins](https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#aws-go-plugins-vs-aws-core-c-plugins).

### 2.32.0.20240304 Linux re-build

*This release has the same Fluent Bit contents as 2.32.0, and is simply a linux-only re-build for recent patches in dependencies installed in the image. There are no windows images for this release.*
Expand Down Expand Up @@ -39,8 +28,8 @@ This release includes:
* Amazon Kinesis Firehose for Fluent Bit 1.7.2

Compared to `2.31.12` this release adds:
* Enhancement - Customers can send metrics to Amazon Managed Prometheus via added sigv4 authentication on `prometheus_remote_write`. Refer to [amazon-ecs-firelens-examples](https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/amazon-managed-service-for-prometheus/README.md) for information on how to export metrics to AMP on ECS [aws-for-fluent-bit:256](https://github.com/aws/aws-for-fluent-bit/issues/256)
* Enhancement - Support multiline parsers with the init tag. Multiline parsers can be specified same as a standard parser file [aws-for-fluent-bit:537](https://github.com/aws/aws-for-fluent-bit/issues/537)
* Feature - Customers can send metrics to Amazon Managed Prometheus via added sigv4 authentication on `prometheus_remote_write`. Refer to [amazon-ecs-firelens-examples](https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/amazon-managed-service-for-prometheus/README.md) for information on how to export metrics to AMP on ECS [aws-for-fluent-bit:256](https://github.com/aws/aws-for-fluent-bit/issues/256)
* Feature - Support multiline parsers with the init tag. Multiline parsers can be specified same as a standard parser file [aws-for-fluent-bit:537](https://github.com/aws/aws-for-fluent-bit/issues/537)
* Enhancement - Customers can route logs to CloudWatch Logs at higher throughputs by increasing number of output workers as `cloudwatch_logs` output plugin removed sequence tokens from API requests [aws-for-fluent-bit:526](https://github.com/aws/aws-for-fluent-bit/issues/526)
* Fix - Fix multiline input behavior when multiple streams are parsed (stderr, stdout) together. Multiline logs are no longer terminated when streams are switched between [fluent-bit:7469](https://github.com/fluent/fluent-bit/pull/7469)
* Fix - Fix networking edgecase causing data loss and OOM related issues via net timeout event injection resolution [fluent-bit:7728](https://github.com/fluent/fluent-bit/pull/7728/files)
Expand Down
4 changes: 1 addition & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,6 @@ The process for pushing out new builds with CVE patches in the base image or ins

For Windows, every month after the [B release date/"patch tuesday"](https://learn.microsoft.com/en-us/windows/deployment/update/release-cycle#monthly-security-update-release), we re-build and update all Windows images currently found in the [windows.versions](windows.versions) file in this repo with the newest base images from Microsoft. The Fluent Bit and go plugin binaries are copied into the newly released base windows image. Thus, the windows image tags are not immutable images; only the Fluent Bit and Go plugin binaries are immutable over time.

At any point in time, [windows.versions](windows.versions) file will contain at least 5 versions, including latest and latest stable. AWS for Fluent Bit Windows are guaranteed to be patched for 4 months after their release date. Therefore, the [windows.versions](windows.versions) file always contains all versions released in the last 4 months, and may contain more if the latest stable release is older than 4 months.

For Linux, each image tag is immutable. When there is a report of high or critical CVEs reported in the base amazon linux image or installed linux packages, we will work to push out a new image [per our patching policy](#compliance-and-patching). However, we will not increment the semantic version number to simply re-build to pull in new linux dependencies. Instead, we will add a 4th version number signifying the date the image was built.

For example, a series of releases in time might look like:
Expand Down Expand Up @@ -161,7 +159,7 @@ Prior to being designated as the latest stable, a version must pass the followin

* It has been out for at least 2 weeks or is a CVE patch with no Fluent Bit changes. Stable designation is based on the Fluent Bit code in the image. A version released for CVE patches can be made stable if the underlying if the underlying Fluent Bit code is already designated as stable.
* No bugs have been reported in Fluent Bit which we expect will have high impact for AWS customers. This means bugs in the components that are most frequently used by AWS customers, such as the AWS outputs or the tail input.
* The version has passed our long running stability tests for at least 2 weeks or is a CVE patch with no Fluent Bit changes that has passed our long running stability tests for at least 1 day. The version would have already passed our simple integration and load tests when it was first released as the latest image.
* The version has passed our long running stability tests for at least 2 weeks. The version would have already passed our simple integration and load tests when it was first released as the latest image.

#### CVE scans and latest stable

Expand Down
103 changes: 30 additions & 73 deletions init/fluent_bit_init_process.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package main
import (
"encoding/json"
"io"
"io/ioutil"
"net/http"
"os"
"path/filepath"
Expand All @@ -18,20 +19,12 @@ import (
"github.com/sirupsen/logrus"
)

// env vars for user configuration
// (?i) makes the match case insensitive
const (
initS3ConfigFilePattern = "(?i)aws_fluent_bit_init_s3"
initLocalConfigFilePattern = "(?i)aws_fluent_bit_init_file"
initIgnoreFireLensConfig = "(?i)aws_fluent_bit_init_ignore_firelens"
)

// static paths
const (
s3FileDirectoryPath = "/init/fluent-bit-init-s3-files/"
initConfigFilePath = "/init/fluent-bit-init.conf"
firelensGeneratedConfigFilePath = "/fluent-bit/etc/fluent-bit.conf"
invokeFilePath = "/init/invoke_fluent_bit.sh"
s3FileDirectoryPath = "/init/fluent-bit-init-s3-files/"
mainConfigFile = "/init/fluent-bit-init.conf"
originalMainConfigFile = "/fluent-bit/etc/fluent-bit.conf"
invokeFile = "/init/invoke_fluent_bit.sh"
)

var (
Expand Down Expand Up @@ -84,7 +77,7 @@ func getECSTaskMetadata(httpClient HTTPClient) ECSTaskMetadata {
logrus.Fatalf("[FluentBit Init Process] Failed to get ECS Metadata via HTTP Get: %s\n", err)
}

response, err := io.ReadAll(res.Body)
response, err := ioutil.ReadAll(res.Body)
if err != nil {
logrus.Fatalf("[FluentBit Init Process] Failed to read ECS Metadata from HTTP response: %s\n", err)
}
Expand All @@ -107,7 +100,7 @@ func getECSTaskMetadata(httpClient HTTPClient) ECSTaskMetadata {
metadata.ECS_TASK_DEFINITION = metadata.ECS_FAMILY + ":" + metadata.ECS_REVISION

// per ECS task metadata docs, Cluster can be an ARN or the name
if strings.Contains(metadata.ECS_CLUSTER, "/") {
if (strings.Contains(metadata.ECS_CLUSTER, "/")) {
clusterARN, err := arn.Parse(metadata.ECS_CLUSTER)
if err != nil {
logrus.Fatalf("[FluentBit Init Process] Failed to parse ECS Cluster ARN: %s %s\n", metadata.ECS_CLUSTER, err)
Expand Down Expand Up @@ -164,9 +157,6 @@ func getAllConfigFiles() {
// get all env vars in the container
envs := os.Environ()

s3Regex := regexp.MustCompile(initS3ConfigFilePattern)
fileRegex := regexp.MustCompile(initLocalConfigFilePattern)

// find all env vars match specified prefix
for _, env := range envs {
var envKey string
Expand All @@ -179,59 +169,27 @@ func getAllConfigFiles() {
envKey = string(env_kv[0])
envValue = string(env_kv[1])

matchedS3 := s3Regex.MatchString(envKey)
matchedFile := fileRegex.MatchString(envKey)
s3_regex, _ := regexp.Compile("aws_fluent_bit_init_[sS]3")
file_regex, _ := regexp.Compile("aws_fluent_bit_init_[fF]ile")

matched_s3 := s3_regex.MatchString(envKey)
matched_file := file_regex.MatchString(envKey)

// if this env var's value is an arn, download the config file first, then process it
if matchedS3 {
if matched_s3 {
s3FilePath := getS3ConfigFile(envValue)
s3FileName := strings.SplitN(s3FilePath, "/", -1)
processConfigFile(s3FileDirectoryPath + s3FileName[len(s3FileName)-1])
}
// if this env var's value is a local config fil, process is directly
if matchedFile {
// if this env var's value is a path of our built-in config file, process is derectly
if matched_file {
processConfigFile(envValue)
}
}
}

func processFireLensConfigFile() {
includeFireLensConfig := true
envs := os.Environ()

ignoreRegex := regexp.MustCompile(initIgnoreFireLensConfig)

// docs say to use aws_fluent_bit_init_ignore_firelens
// this supports case insensitive prefix matching, in case someone
// tries to capitalize FireLens, or uses aws_fluent_bit_init_ignore_firelens_config
for _, env := range envs {
var envKey string
var envValue string
env_kv := strings.SplitN(env, "=", 2)
if len(env_kv) != 2 {
logrus.Fatalf("[FluentBit Init Process] Unrecognizable environment variables: %s\n", env)
}

envKey = string(env_kv[0])
envValue = string(env_kv[1])

matchedIgnore := ignoreRegex.MatchString(envKey)

if matchedIgnore {
if strings.EqualFold(envValue, "true") || strings.EqualFold(envValue, "on") {
includeFireLensConfig = false
}
}
}

if includeFireLensConfig {
// add @INCLUDE in main config file to include original main config file
writeInclude(firelensGeneratedConfigFilePath, initConfigFilePath)
}
}

func processConfigFile(path string) {
contentBytes, err := os.ReadFile(path)
contentBytes, err := ioutil.ReadFile(path)
if err != nil {
logrus.Errorln(err)
logrus.Fatalf("[FluentBit Init Process] Cannot open file: %s\n", path)
Expand All @@ -244,7 +202,7 @@ func processConfigFile(path string) {
updateCommand(path)
} else {
// this is not a parser config file. @INCLUDE
writeInclude(path, initConfigFilePath)
writeInclude(path, mainConfigFile)
}
}

Expand Down Expand Up @@ -349,15 +307,15 @@ func downloadS3ConfigFile(s3Downloader S3Downloader, s3FilePath, bucketName, s3F
}

// use @INCLUDE to add config files to the main config file
func writeInclude(configFilePath string, initConfigFilePath string) {
initConfigFile := openFile(initConfigFilePath)
defer initConfigFile.Close()
func writeInclude(configFilePath, mainConfigFilePath string) {
mainConfigFile := openFile(mainConfigFilePath)
defer mainConfigFile.Close()

writeContent := "@INCLUDE " + configFilePath + "\n"
_, err := initConfigFile.WriteString(writeContent)
_, err := mainConfigFile.WriteString(writeContent)
if err != nil {
logrus.Errorln(err)
logrus.Fatalf("[FluentBit Init Process] Cannot write %s in main config file: %s\n", writeContent[:len(writeContent)-2], initConfigFilePath)
logrus.Fatalf("[FluentBit Init Process] Cannot write %s in main config file: %s\n", writeContent[:len(writeContent)-2], mainConfigFilePath)
}
}

Expand Down Expand Up @@ -414,24 +372,23 @@ func main() {
// create the invoke_fluent_bit.sh
// which will declare ECS Task Metadata as environment variables
// and finally invoke Fluent Bit
createFile(invokeFilePath, true)
createFile(invokeFile, true)

// get ECS Task Metadata and set the region for S3 client
httpClient := &http.Client{}
metadata := getECSTaskMetadata(httpClient)

// set ECS Task Metada as env vars in the invoke_fluent_bit.sh
setECSTaskMetadata(metadata, invokeFilePath)
setECSTaskMetadata(metadata, invokeFile)

// create main config file which will be used invoke Fluent Bit
createFile(initConfigFilePath, true)
createFile(mainConfigFile, true)

// create Fluent Bit command to use "-c" to specify new main config file
createCommand(&baseCommand, initConfigFilePath)
// add @INCLUDE in main config file to include original main config file
writeInclude(originalMainConfigFile, mainConfigFile)

// include the FireLens generated config
// unless the user has set aws_fluent_bit_init_ignore_firelens
processFireLensConfigFile()
// create Fluent Bit command to use "-c" to specify new main config file
createCommand(&baseCommand, mainConfigFile)

// get our built in config files or files from s3
// process built-in config files directly
Expand All @@ -442,5 +399,5 @@ func main() {
// this function will be called at the end
// any error appear above will cause exit this process,
// will not write Fluent Bit command in the finvoke_fluent_bit.sh so Fluent Bit will not be invoked
modifyInvokeFile(invokeFilePath)
modifyInvokeFile(invokeFile)
}
2 changes: 1 addition & 1 deletion linux.version
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"linux": {
"version": "2.32.1",
"version": "2.32.0.20240304",
"latest": "true",
"build": "1",
"fluent-bit": "1.9.10",
Expand Down
6 changes: 3 additions & 3 deletions load_tests/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 9a80d21

Please sign in to comment.