NullPointerException because "datadog.trace.api.civisibility.InstrumentationBridge.TEST_EVENTS_HANDLER_FACTORY" is null

[staktrace]

Hi, I'm seeing a NullPointerException from the datadog java agent library when running junit tests on gradle modules that produce gradle plugins. In CI we're seeing this on version 1.35.2 of com.datadoghq:dd-java-agent but I also tried version 1.38.1 (latest on maven central) and got the same error.

expand for full stack trace

Caused by: java.lang.NullPointerException: Cannot invoke "datadog.trace.api.civisibility.events.TestEventsHandler$Factory.create(String, datadog.trace.bootstrap.ContextStore, datadog.trace.bootstrap.ContextStore)" because "datadog.trace.api.civisibility.InstrumentationBridge.TEST_EVENTS_HANDLER_FACTORY" is null	
at datadog.trace.api.civisibility.InstrumentationBridge.createTestEventsHandler(InstrumentationBridge.java:28)	
at datadog.trace.instrumentation.junit5.TestEventsHandlerHolder.start(TestEventsHandlerHolder.java:40)	
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:50)	
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)	
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)	
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)	
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)	
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)	
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)	
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)	
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)	
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:124)	
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:99)	
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:94)	
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:63)	
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)	
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)	
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	
at java.lang.reflect.Method.invoke(Method.java:568)	
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)	
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)	
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)	
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:92)	
at jdk.proxy3.$Proxy48.stop(Unknown Source)	
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:200)	
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:132)	
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)	
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)	
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)	
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:121)	
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)	
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)	
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

In our buildkite infra we set up the following datadog system properties:

  DD_PROFILER_ARGS="-javaagent:${dd_java_agent_path}" # Path for the agent jar
  DD_PROFILER_ARGS="${DD_PROFILER_ARGS} -Ddd.profiling.ddprof.enabled=false" # Disable the wallclock profiling engine, as it breaks all the things
  DD_PROFILER_ARGS="${DD_PROFILER_ARGS} -Ddd.profiling.enabled=true" # Enable the profiler
  DD_PROFILER_ARGS="${DD_PROFILER_ARGS} -Ddd.agent.host=dd-agent" # Agent runs in a different docker container, available on dd-agent
  DD_PROFILER_ARGS="${DD_PROFILER_ARGS} -Ddd.jmxfetch.statsd.host=dd-agent" # Agent runs in a different docker container, available on dd-agent
  DD_PROFILER_ARGS="${DD_PROFILER_ARGS} -Ddd.service=${BUILDKITE_PIPELINE_SLUG}-pipeline" # Profiler tags setup: service
  DD_PROFILER_ARGS="${DD_PROFILER_ARGS} -Ddd.env=cashkite" # Profiler tags setup: env
  DD_PROFILER_ARGS="${DD_PROFILER_ARGS} -Ddd.pipeline=${BUILDKITE_PIPELINE_SLUG:-}" # Profiler tags setup: pipeline

and then in our gradle convention plugin we additionally set the following:

    tasks.withType(Test::class.java).configureEach { t ->
        t.systemProperties(
          mapOf(
            "dd.civisibility.enabled" to true,
            "dd.profiling.enabled" to false,
            "dd.trace.enabled" to true,
            "dd.jmxfetch.enabled" to false,
            "dd.civisibility.code.coverage.enabled" to false,
            "dd.civisibility.git.upload.enabled" to false,
            "dd.integration.opentracing.enabled" to true,
            "dd.instrumentation.telemetry.enabled" to false,
          )
     }

For most gradle modules this works fine. However modules that contain gradle plugins (i.e. that are applying the java-gradle-plugin plugin) seem to be hitting this exception on junit startup. I can reliably reproduce the problem so can collect more info and/or try to make a minimal reproducer that I can share. Let me know if those would be helpful. Thanks!

[nikita-tkachenko-datadog]

Hi @staktrace, thank you for the detailed report!
The fix for the issue will be included in the next tracer release, which is scheduled for the following week.

As a side note, I have noticed that you seem to be injecting the tracer only in your test JVM processes, rather than in your Gradle build parent process.
There are some advantages to instrumenting the Gradle Daemon (like reduced overhead, having less configuration and seeing all the modules from the same build as one session in Datadog), besides it should resolve your problem without the need to wait for the new tracer release.

If you want to try running your tests while instrumenting the entire Gradle build, rather than individual test tasks, you can do so by modifying your build command to include org.gradle.jvmargs system property which injects the agent like this (more details can be found in the docs):

./gradlew cleanTest test -Dorg.gradle.jvmargs=\
-javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=\
dd.civisibility.enabled=true,\
dd.env=ci,\
dd.service=my-java-app

[staktrace]

Thank you for the quick fix!

As a side note, I have noticed that you seem to be injecting the tracer only in your test JVM processes, rather than in your Gradle build parent process.

Yes, this is a result of how our CI infrastructure is set up and cost-optimization. The gradle invocation happens in shared CI infra code, so if we turn on the instrumentation there it would happen across a large set of CI builds that we don't care about. Instead we control it in a more fine-grained way via our gradle plugins so that it is restricted to things we care about and don't incur extra costs for the other stuff. That being said, thanks for also providing the workaround, it is good to know in case we need it.

Looking forward to the new version with the fix! Cheers :)