New component: Queue Processor

[akats7]

The purpose and use-cases of the new component

The purpose of a queue processor would be to decouple the queuing mechanism from the exporterhelper util. This stems from a discussion in #33007, where the ask was for the failover connector to support queueing directly in the component. As the failover connector relies on backwards propagated errors to trigger failover but with queueing enabled on the exporters will only failover once the queue is full, leading to delayed failover and without a properly configured queue dropped data. For that reason certain users have resorted to disabling the queue.

Instead of enabling a queue mechanism directly in the failover connector, it was discussed if it might be beneficial to develop a queue processor that could be re-usable for similar uses cases in the future.

The queue processor would be a component that sits directly in front of the exporters(last in the processor chain) in the pipeline and would be responsible for providing an async export queue to the nextConsumer exporters to consume from without enabling the queue on the exporter itself.

I'd be happy to contribute this component if we are able to get it sponsored.

cc. @djaglowski @sinkingpoint

Example configuration for the component

receivers:
  otlp:

processors:
  batch:
  queue:

exporters:
  otlp:


service:
  pipelines:
    traces:
       receivers: [otlp]
       processors: [batch, queue]
       exporters: [otlp]

Telemetry data types supported

traces->traces
metrics->metrics
logs->logs

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

akats7

Sponsor (optional)

No response

Additional context

No response

[atoulme]

Please see the guidelines for new components here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md

Who's the sponsor for your component. A sponsor is an approver or maintainer who will be the official reviewer of the code and a code owner for the component. You will need to find a sponsor for the component in order for it to be accepted.

To secure a sponsor, please communicate on the CNCF slack and/or attend a SIG meeting to make the case for your component.

Please also consider hosting your component in a separate repository if it suits you.

[jmacd]

I read through #33007 and agreed with the arguments against adding exporterhelper functions to the failover connector.

This RFC on error propagation for the batch processor,
open-telemetry/opentelemetry-collector#11947
speaks a lot about making the batch processor identical to the exporterhelper support. I have also wondered whether it makes sense to create a processor component that is just the exporterhelper logic, outputting to the next consumer as pdata. I.e., all the features of exporterhelper, where the export function calls Consume() on the following component (exporter/processor/connector).

Since this doesn't have a name, I was thinking of it as a "pipeline processor". If there was a pipeline processor exposing all the features of the exporterhelper (queue, batch, retry, timeout), then today's batch processor could be replaced by the pipeline processor. @akats7 the queue processor you propose would be just one configuration, and overall we would have to maintain less code. wdyt?

[akats7]

Hey @jmacd, I would be on board with this, I do like the idea of shared configuration and allocations. However, it does seem to be a further expansion of the queue retry processor that was deprecated a while back. Outside of the application to connectors what else do you see as probable use cases, presuming that the majority of exporters would still rely on exporter helper.

[jmacd]

@akats7 I hope you like #37787, which I have an interest in seeing finished. If you agree, feel free to close this and add support to that one!

[akats7]

Sounds good, thanks @jmacd!