Long lines are unexpectedly split into multiple OTEL Log records

[cwegener]

Component(s)

pkg/stanza/fileconsumer, receiver/filelog

What happened?

Description

When using the filelog receiver, any lines that are longer than the default buffer size (16KB) will be split into multiple log records instead of one single log record.

Steps to Reproduce

1. Create simple default filelog receiver config
2. Ingest a file that has a line length that is greater than 16KB)

Expected Result

Lines longer than 16KB gets emitted as one single OTEL Log record

Actual Result

Lines longer than 16KB get emitted as multiple OTEL Log records, each record getting split at the 16KB boundary.

Collector version

v0.108.0

Environment information

Environment

OS: Archlinux
Compiler(if manually compiled): go 1.23.0

OpenTelemetry Collector configuration

receivers:
  filelog:
    start_at: beginning
    include:
      - "/tmp/metrics_101089747.json"

exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs/raw:
      receivers:
        - filelog
      exporters:
        - debug

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/stanza/fileconsumer: @djaglowski
• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

Although the buffer size is 16KB, the maximum log size is 1024KB.
The incomplete token issue might be due to the flush timeout expiring. Could you try setting force_flush_period to 0 and see if that resolves the problem?

[cwegener]

The bug still behaves the same when using different values for the flush time out.

Although, I have not actually tested a value of 0 ... I wonder what 0 means?

[cwegener]

Ok.

Did another quick test and the issue is:

• the nature of the input file is causing the problem
• the input files that I am using (they are generated by the awss3exporter contrib component) are missing a \n at the end of the line, which is then triggering the behaviour that I originally observed above.

Adding a \n at the end of the line, fixes the issue as expected.

I'll start investigating why the \n is missing in the files generated by https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awss3exporter

[cwegener]

Closing this issue, as filelog receiver is behaving as expected (logs are force flushed if no log line separator token is seen within the force flush period.

[djaglowski]

I'm reopening this issue as I've stumbled onto the same behavior.

To clarify, the issue occurs when a log is:

1. Longer than the default buffer size (16KB)
2. Shorter than the max log size
3. NOT terminated

What happens is that the flush function kicks in and ejects the token.

What I do not understand yet is why the buffer size does not increase up to max log size before the ejection occurs. If it did, then the ejected token would be the entire log (even though it is not explicitly terminated).