Periodic Metric exporter unable to collect data points on Passenger-based servers

[divyam-rai]

Description of the bug

Due to Passenger's smart spawning mechanism which creates worker processes by forking an initial pre-loader process, we notice that the thread for the periodic metric exporter is only attached to the pre-loader process and as such does not share memory with the other child processes which handle the actual incoming requests.

This causes the periodic metric reader to never have any data points since the metrics are stored in the memory of the child processes and are not shared with the thread which is attached to the pre-loader process.

The spawning mechanism, as well as its caveats, are described here: https://www.phusionpassenger.com/library/indepth/ruby/spawn_methods/#the-smart-spawning-method

To resolve this, the periodic metric reader needs to be initialised for each of the forked worker processes instantiated by Passenger, which was done as followed based on events that are made available within the internals of Passenger. This allowed us to send metrics as required.

if defined?(PhusionPassenger)
    PhusionPassenger.on_event(:starting_worker_process) do |forked|
      if forked
        metric_reader = OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(
          exporter: OpenTelemetry::Exporter::OTLP::Metrics::MetricsExporter.new,
        )

        OpenTelemetry.meter_provider.add_metric_reader(metric_reader)
      end
    end
end

[github-actions]

👋 This issue has been marked as stale because it has been open with no activity. You can: comment on the issue or remove the stale label to hold stale off for a while, add the keep label to hold stale off permanently, or do nothing. If you do nothing this issue will be closed eventually by the stale bot.

[chrisholmes]

I've run into the same issue while I've been testing this out on one of our applications that uses puma and forking puma workers. It should apply to all systems that use a forking webserver as forking kills threads.

@kaylareopelle, I believe I have a generic enough patch for this. Would you like me to open a PR or is somebody else looking at this?

[kaylareopelle]

Hi @chrisholmes, thanks for reaching out! A PR with your patch would be great. I'm not aware of anyone looking into this issue yet, and we would appreciate the support.

[chrisholmes]

@kaylareopelle, here's the PR: #1823