AWS Lambda support
This issue is intended to give some context and track a couple of PRs that are adding features independently of the goal: support for AWS lambda. This comes after stumbling about this PR aws-samples/aws-lambda-extensions#42 attempting to add Fluentbit as an AWS Lambda extension. The goal of this possible implementation would be to allow Fluentbit to be embedded within lambda and processing logs received from the lambda telemetry API.
Lambda extensions run alongside the lambda workload and receive logs as they come before being frozen once the extension hands back the environment until the next lambda gets invoked unthawing the environment. This event-driven workflow requires Fluentbit to be event-driven as well for its flush mechanism to ensure logs are force flushed to the outputs in a deterministic manner (doc).
PRs to make the feature possible:
- Flush now: modifies the HTTP server to add a
/api/v2/flush route which when called, will flush the in-transit chunks, returning when acknowledged by the engine. PR
- In-flight tasks metrics: modifies the HTTP server's
/api/v2/metrics route adding an optional ?recalculate=true GET parameter to force recalculation of metrics which are currently delayed. The purpose is to inspect fluentbit_storage_chunks in real-time to ensure all chunks are flushed before handing back the environment to AWS lambda for freezing.
The lambda extension (an intermediary piece of software) starting Fluentbit would then normally flush logs during the lifetime of the lambda workload, and leverage the aforementioned features upon termination of the lambda to force flushing every chunk. Upon lambda terminating, the extension would call the /api/v2/flush route, and then the /api/v2/metrics?recalculate=true endpoint either until all chunks are flushed, or the extension times out waiting for chunks to be flushed. The extension then hands back the environment to AWS for freezing until the next lambda invocation.
Would the above approach (force flush + poll metrics) be the preferred way to solve this? I'm open to alternative, simpler solutions from experts more familiar with the codebase. In the meantime, I'll open a PR for the flush feature, and work on the second one if the above is deemed reasonable.
AWS Lambda support
This issue is intended to give some context and track a couple of PRs that are adding features independently of the goal: support for AWS lambda. This comes after stumbling about this PR aws-samples/aws-lambda-extensions#42 attempting to add Fluentbit as an AWS Lambda extension. The goal of this possible implementation would be to allow Fluentbit to be embedded within lambda and processing logs received from the lambda telemetry API.
Lambda extensions run alongside the lambda workload and receive logs as they come before being frozen once the extension hands back the environment until the next lambda gets invoked unthawing the environment. This event-driven workflow requires Fluentbit to be event-driven as well for its flush mechanism to ensure logs are force flushed to the outputs in a deterministic manner (doc).
PRs to make the feature possible:
/api/v2/flushroute which when called, will flush the in-transit chunks, returning when acknowledged by the engine. PR/api/v2/metricsroute adding an optional?recalculate=trueGET parameter to force recalculation of metrics which are currently delayed. The purpose is to inspectfluentbit_storage_chunksin real-time to ensure all chunks are flushed before handing back the environment to AWS lambda for freezing.The lambda extension (an intermediary piece of software) starting Fluentbit would then normally flush logs during the lifetime of the lambda workload, and leverage the aforementioned features upon termination of the lambda to force flushing every chunk. Upon lambda terminating, the extension would call the
/api/v2/flushroute, and then the/api/v2/metrics?recalculate=trueendpoint either until all chunks are flushed, or the extension times out waiting for chunks to be flushed. The extension then hands back the environment to AWS for freezing until the next lambda invocation.Would the above approach (force flush + poll metrics) be the preferred way to solve this? I'm open to alternative, simpler solutions from experts more familiar with the codebase. In the meantime, I'll open a PR for the flush feature, and work on the second one if the above is deemed reasonable.