diff --git a/SUMMARY.md b/SUMMARY.md index cafaf1057..64eef78bb 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -192,6 +192,7 @@ * [Graylog Extended Log Format (GELF)](pipeline/outputs/gelf.md) * [Google Chronicle](pipeline/outputs/chronicle.md) * [Google Cloud BigQuery](pipeline/outputs/bigquery.md) + * [Google Cloud Storage](pipeline/outputs/gcs.md) * [HTTP](pipeline/outputs/http.md) * [InfluxDB](pipeline/outputs/influxdb.md) * [Kafka Producer](pipeline/outputs/kafka.md) diff --git a/pipeline/outputs/gcs.md b/pipeline/outputs/gcs.md new file mode 100644 index 000000000..deb53b577 --- /dev/null +++ b/pipeline/outputs/gcs.md @@ -0,0 +1,104 @@ +--- +description: Send logs to Google Cloud Storage +--- + +# Google Cloud Storage + +{% hint style="info" %} +**Supported event types:** `logs` +{% endhint %} + +The _Google Cloud Storage_ (`gcs`) output plugin lets you upload the records received through the input plugin to a [Google Cloud Storage](https://cloud.google.com/storage) bucket. Records are buffered locally and uploaded as objects when the configured upload timeout elapses. + +This plugin is available in Fluent Bit version 5.1 and greater. It's included in official builds. If you compile Fluent Bit from source, the `FLB_OUT_GCS` build option is enabled by default. + +## Google Cloud configuration + +Before using the plugin, you must: + +1. Create a [Google Cloud service account](https://docs.cloud.google.com/iam/docs/service-accounts-create) for Fluent Bit and [create a JSON key](https://docs.cloud.google.com/iam/docs/keys-create-delete) for it. +1. [Create the bucket](https://docs.cloud.google.com/storage/docs/creating-buckets) that receives your data. Fluent Bit doesn't create buckets. +1. Grant the service account permission to write objects to that bucket. The plugin requests the `https://www.googleapis.com/auth/devstorage.read_write` `OAuth 2.0` scope. +1. Make the credentials available to Fluent Bit, either by setting the `google_service_credentials` option to the absolute path of the JSON key file, or by setting the `GOOGLE_SERVICE_CREDENTIALS` environment variable to that path. + +The plugin uploads objects to `storage.googleapis.com` over TLS using the Cloud Storage JSON API. + +## Configuration parameters + +| Key | Description | Default | +| :--- | :--- | :--- | +| `bucket` | Name of the Cloud Storage bucket that receives the objects. | _none_ | +| `canned_acl` | [Predefined ACL](https://docs.cloud.google.com/storage/docs/access-control/lists#predefined-acl) applied to uploaded objects. Accepted values: `authenticated-read`, `bucket-owner-full-control`, `bucket-owner-read`, `private`, `project-private`, and `public-read`. The camel case forms used by the Cloud Storage API, such as `publicRead`, are also accepted. | _none_ | +| `compression` | Compression applied to uploaded objects. Accepted values: `none`, `gzip`. | `none` | +| `content_type` | Value of the `Content-Type` metadata set on uploaded objects. | `application/json` | +| `gcs_key_format` | Format string for object names in the bucket. See [Object key format](#object-key-format). | `fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` | +| `gcs_key_format_tag_delimiters` | Characters used to split the tag into the parts referenced by `$TAG[n]` in `gcs_key_format`. | `.` | +| `google_service_credentials` | Absolute path to a Google Cloud service account credentials JSON file. | Value of the environment variable `$GOOGLE_SERVICE_CREDENTIALS`. | +| `preserve_data_ordering` | When an upload request fails, the last received chunk might swap with a later chunk, resulting in data shuffling. This option prevents shuffling by using queue logic for uploads. | `false` | +| `send_content_md5` | Send the `Content-MD5` header with uploads so that Cloud Storage verifies object integrity. | `false` | +| `static_file_path` | Disables the behavior where a random suffix appends to the object name when `$UUID` isn't provided in `gcs_key_format`. | `false` | +| `store_chunk_limit` | Maximum number of buffered chunks kept in `store_dir`. Set to `0` for unlimited. | `0` | +| `store_dir` | Directory used to locally buffer data before uploading it. | `/tmp/fluent-bit/gcs` | +| `store_dir_limit_size` | Limits the amount of data buffered in `store_dir` to limit disk usage. When the limit is reached, data is discarded. Set to `0` for unlimited. | `0` | +| `upload_timeout` | When this amount of time elapses, Fluent Bit uploads the buffered data and starts a new object. Set to `60m` to upload a new object every hour. | `10m` | +| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` | + +## Object key format + +The `gcs_key_format` option supports the same formatters as the [Amazon S3](s3.md#s3-key-format-and-tag-delimiters) output plugin: + +- `$TAG`: The full tag. +- `$TAG[n]`: The nth part of the tag, with the index starting at zero. Tag parts are separated using the characters set in `gcs_key_format_tag_delimiters`. +- `$UUID`: A random string. +- `$INDEX`: An integer that increments with each upload. The value is stored in `store_dir` so it survives a restart. +- [strftime](https://man7.org/linux/man-pages/man3/strftime.3.html) formatters such as `%Y`, `%m`, and `%d`. The time used is the timestamp of the first record in the object. + +When `gcs_key_format` doesn't contain `$UUID` and `static_file_path` is `false`, Fluent Bit appends a random suffix to each object name so that concurrent uploads don't overwrite each other. Set `static_file_path` to `true` to keep the object name exactly as the format string produces it. + +## Buffering + +This plugin buffers records as files in `store_dir` and uploads them when `upload_timeout` elapses, so it requires a writeable filesystem. Because the plugin has its own buffering system, the `storage.total_limit_size` parameter isn't meaningful. Use `store_dir_limit_size` and `store_chunk_limit` to limit disk usage instead. + +## Get started + +The following configuration buffers CPU metrics and uploads a gzip-compressed object to the `my-logs` bucket every five minutes: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: cpu + tag: cpu + + outputs: + - name: gcs + match: '*' + bucket: my-logs + google_service_credentials: /path/to/credentials.json + gcs_key_format: /fluent-bit/$TAG/%Y/%m/%d/%H_%M_%S/$UUID.gz + compression: gzip + upload_timeout: 5m +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name cpu + Tag cpu + +[OUTPUT] + Name gcs + Match * + Bucket my-logs + Google_Service_Credentials /path/to/credentials.json + Gcs_Key_Format /fluent-bit/$TAG/%Y/%m/%d/%H_%M_%S/$UUID.gz + Compression gzip + Upload_Timeout 5m +``` + +{% endtab %} +{% endtabs %} diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml index bcbc230eb..df3393364 100644 --- a/vale-styles/FluentBit/Headings.yml +++ b/vale-styles/FluentBit/Headings.yml @@ -76,6 +76,7 @@ exceptions: - Google Cloud - Google Cloud BigQuery - Google Cloud Platform + - Google Cloud Storage - GPG - GPU - Grafana