bug(AMI): Log Rotation Issue Causes Message Fragmentation

[mikheevsv]

What happened: We are experiencing an issue with log rotation where lengthy log messages are being split between two different log files. This results in fragmented and incomplete log entries.

What you expected to happen: Log messages should be complete and not split between log files

How to reproduce it (as minimally and precisely as possible):
For testing, I just run the following python-script on EKS:

import time
message = "0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 "
i = 1
while i < 10000000:
  i += 1
  print(message)
time.sleep(3600)

And changed containerLogMaxFiles to 10.

That's what I see in the logs:

$ for i in `ls *.gz`; do echo file: $i; zcat $i|head -n 1; zcat $i|tail -n 1; done
...
file: 0.log.20241223-161939.gz
2024-12-23T16:19:24.969362937Z stdout F 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 
2024-12-23T16:19:39.66287371Z stdout P 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 012345678

file: 0.log.20241223-161950.gz
2024-12-23T16:19:39.696095101Z stdout F 9 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 
2024-12-23T16:19:50.943256914Z stdout F 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 0123456789 

Environment:

• AWS Region: eu-west-1
• Instance Type(s): t3., m5a., m5n., m6a.
• Cluster Kubernetes version: 1.29
• Node Kubernetes version: v1.29.6-eks-1552ad0
• AMI Version: amazon-eks-node-1.29-v20240807