stdout logging from Kubernetes pods

0

Hi,

I ran into an issue with our logging from Kubernetes pods. We store logging in JSON format, but saw that with big JSON logs it was breaking. So I investigated the isssue. Found out that when you are sending big logs from the pods with for example cat log/test.log > /proc/1/fd/1

It will ends in multiple parts into the node log folder (/var/log/pods/pod) 2023-03-24T11:53:33.107458625Z stdout P YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY 2023-03-24T11:53:33.107458625Z stdout P YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY 2023-03-24T11:53:33.107458625Z stdout P YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

This breaks the JSON. How can we avoid that sending big log files to /proc/1/fd/1, will be splitted into multiple parts?

Thank you!

asked a year ago640 views
1 Answer
0

I believe this is by design, files larger than 16K. Fluent has options like (https://docs.fluentbit.io/manual/v/1.3/input/tail#docker_mode) docker mode if you are using that which tries to recombine.

profile pictureAWS
Roly
answered a year ago
  • Yes, I'm using Fluent-Bit, and also has the docker mode. But I find that it stuggles to combine those again.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions