Skip to content

out_s3, out_chronicle: grow log_key buffer to avoid dropping oversized values #12137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vitaly-sinev wants to merge 2 commits into
base: master
Choose a base branch
from
+67 −9
Open

out_s3, out_chronicle: grow log_key buffer to avoid dropping oversized values #12137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eac4057
out_s3: grow log_key buffer on JSON serialization overflow
vitaly-sinev Jul 22, 2026
032f864
out_chronicle: grow log_key buffer on JSON serialization overflow
vitaly-sinev Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions plugins/out_chronicle/chronicle.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -526,9 +526,12 @@ static flb_sds_t flb_pack_msgpack_extract_log_key(void *out_context, uint64_t by
int ret;
struct flb_chronicle *ctx = out_context;
char *val_buf;
char *tmp_buf;
char *key_str = NULL;
size_t key_str_size = 0;
size_t msgpack_size = bytes + bytes / 4;
size_t new_size;
size_t avail;
size_t val_offset = 0;
flb_sds_t out_buf;
msgpack_object map;
Expand Down Expand Up @@ -592,11 +595,31 @@ static flb_sds_t flb_pack_msgpack_extract_log_key(void *out_context, uint64_t by
val_offset++;
}
else {
ret = flb_msgpack_to_json(val_buf + val_offset,
msgpack_size - val_offset, &val,
config->json_escape_unicode);
if (ret < 0) {
break;
/*
* Serialize the value as JSON, growing the scratch buffer
* and retrying if it does not fit, so an oversized value
* is neither truncated nor dropped.
*/
ret = -1;
while (1) {
avail = msgpack_size - val_offset;
if (avail > 1) {
ret = flb_msgpack_to_json(val_buf + val_offset,
avail, &val,
config->json_escape_unicode);
if (ret > 0) {
break;
}
}
new_size = msgpack_size * 2;
tmp_buf = flb_realloc(val_buf, new_size);
if (tmp_buf == NULL) {
flb_errno();
flb_free(val_buf);
return NULL;
}
val_buf = tmp_buf;
msgpack_size = new_size;
}
val_offset += ret;
val_buf[val_offset] = '\0';
Expand Down
43 changes: 39 additions & 4 deletions plugins/out_s3/s3.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3589,9 +3589,12 @@ static flb_sds_t flb_pack_msgpack_extract_log_key(void *out_context, const char
int alloc_error = 0;
struct flb_s3 *ctx = out_context;
char *val_buf;
char *tmp_buf;
char *key_str = NULL;
size_t key_str_size = 0;
size_t msgpack_size = bytes + bytes / 4;
size_t new_size;
size_t avail;
size_t val_offset = 0;
flb_sds_t out_buf;
msgpack_object map;
Expand Down Expand Up @@ -3683,10 +3686,33 @@ static flb_sds_t flb_pack_msgpack_extract_log_key(void *out_context, const char
val_offset++;
}
else {
ret = flb_msgpack_to_json(val_buf + val_offset,
msgpack_size - val_offset, &val,
config->json_escape_unicode);
if (ret < 0) {
/*
* Serialize the value as JSON, growing the scratch
* buffer and retrying if it does not fit, so an
* oversized value is neither truncated nor dropped.
*/
ret = -1;
while (1) {
avail = msgpack_size - val_offset;
if (avail > 1) {
ret = flb_msgpack_to_json(val_buf + val_offset,
avail, &val,
config->json_escape_unicode);
if (ret > 0) {
break;
}
}
new_size = msgpack_size * 2;
tmp_buf = flb_realloc(val_buf, new_size);
if (tmp_buf == NULL) {
flb_errno();
alloc_error = 1;
break;
Comment on lines +3709 to +3710

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Return NULL when S3 log_key growth fails

For S3 chunks using log_key, if a later non-string value needs this buffer growth and flb_realloc() fails after earlier records have already advanced val_offset, setting alloc_error only exits the decoder loop; the function then falls through and creates an SDS from the partial val_buf whenever val_offset != 0. The flush path treats that non-NULL chunk as data to upload, so the current and remaining records can be dropped while the output reports success; free the buffer and return NULL on this allocation failure instead of falling through with partial data.

Useful? React with 👍 / 👎.

}
val_buf = tmp_buf;
msgpack_size = new_size;
}
if (alloc_error == 1) {
Comment thread
coderabbitai[bot] marked this conversation as resolved.
break;
}
val_offset += ret;
Expand All @@ -3713,6 +3739,15 @@ static flb_sds_t flb_pack_msgpack_extract_log_key(void *out_context, const char

flb_log_event_decoder_destroy(&log_decoder);

/*
* If the scratch buffer could not be grown mid-chunk, do not return a
* partial payload: fail so the chunk can be retried as a whole.
*/
if (alloc_error == 1) {
flb_free(val_buf);
return NULL;
}

/* If nothing was read, destroy buffer */
if (val_offset == 0) {
flb_free(val_buf);
Expand Down