Hystrix 1.5.5 and Jackson Error on Metrics-Event-Stream

[caarlos0]

Last version of hystrix-metrics-event-stream (1.5.5) running on a JBoss AS 7 app server, when accessing the metric stream, give me this error:

Error log

14:30:04,638 java.lang.NullPointerException GtOHbn9FSiIkdARjIGvK6L7R app-d821bd09.prod.foo.local WARN  [org.jboss.modules] (RxComputationScheduler-1) Failed to define class com.fasterxml.jackson.dataformat.cbor.CBORGenerator in Module "deployment.foo-app.ear.foo-app-presentation.war:main" from Service Module Loader: java.lang.VerifyError: class com.fasterxml.jackson.dataformat.cbor.CBORGenerator overrides final method copyCurrentEvent.(Lcom/fasterxml/jackson/core/JsonParser;)V
        at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_72]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800) [rt.jar:1.7.0_72]
        at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:361)
        at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:482)
        at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:277)
        at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:92)
        at org.jboss.modules.Module.loadModuleClass(Module.java:568)
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:205)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389)
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134)
        at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$1.call(HystrixMetricsStreamServlet.java:66) [hystrix-metrics-event-stream-1.5.5.jar:1.5.5]
        at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$1.call(HystrixMetricsStreamServlet.java:63) [hystrix-metrics-event-stream-1.5.5.jar:1.5.5]
        at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:259) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OperatorOnBackpressureDrop$2.onNext(OperatorOnBackpressureDrop.java:83) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeRefCount$2.onNext(OnSubscribeRefCount.java:120) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OperatorPublish$PublishSubscriber.dispatch(OperatorPublish.java:588) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OperatorPublish$PublishSubscriber.onNext(OperatorPublish.java:286) [rxjava-1.1.8.jar:1.1.8]
        at rx.observers.Subscribers$5.onNext(Subscribers.java:235) [rxjava-1.1.8.jar:1.1.8]
        at rx.observers.Subscribers$5.onNext(Subscribers.java:235) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeTimerPeriodically$1.call(OnSubscribeTimerPeriodically.java:52) [rxjava-1.1.8.jar:1.1.8]
        at rx.Scheduler$Worker$1.call(Scheduler.java:136) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189) [rxjava-1.1.8.jar:1.1.8]
        at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.8.jar:1.1.8]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_72]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_72]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [rt.jar:1.7.0_72]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [rt.jar:1.7.0_72]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_72]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_72]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_72]

And the error I got is MaxConcurrentConnections reached: 5.

Any tips on how to fix that?

Thanks

[mattnelson]

There is a configuration property for the max concurrent connections, the default is 5.

https://github.com/Netflix/Hystrix/blob/v1.5.5/hystrix-contrib/hystrix-metrics-event-stream/src/main/java/com/netflix/hystrix/contrib/metrics/eventstream/HystrixMetricsStreamServlet.java#L55-L56

[caarlos0]

@mattflix yeah, as I said, the problem isn't that, the exception is masking it...

The problem seems to be that the jackson version inside jboss as is older that the version used by metrics-stream... but the error shown is the number of connections, which is unrelated to the root cause...

[caarlos0]

FWIW: version 1.5.3 works

[mattrjacobs]

Hystrix 1.5.4 broke out the serialization of metrics into its own module (hystrix-serialization). That module depends on 2.7.5 Jackson libraries.

In your application, do you have the hystrix-serialization jar? What versions of Jackson do you have there?

[caarlos0]

@mattrjacobs yes, it was working until I updated it.

It's a dependency hell... just so you have a slight idea:

I'll try to bump all, for now, the report that remains is the misleading error message 👍

[mattrjacobs]

I know that feeling :) I suspect that the mismatched Jackson is the problem. Please let me know if that resolves things.

[caarlos0]

@mattrjacobs yeah, I'll try to do that...

[mukteshkrmishra]

I got the same problem in 1.5.5 and rectified it by excluding Jackson.

[pablote]

I was able to get this working when using an older jackson dep by adding hystrix like this:

<dependency>
            <groupId>com.netflix.hystrix</groupId>
            <artifactId>hystrix-metrics-event-stream</artifactId>
            <version>1.5.5</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.dataformat</groupId>
                    <artifactId>jackson-dataformat-cbor</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

[mattrjacobs]

This looks resolved. Please re-open if there's more to discuss

[mechko]

Hi, I'm using 1.5.8 of hystrix-metrics-event-stream. First I had the same verifyError like the original poster, I added the exclusion like @pablote suggested then, but now I get the following ClassNotFound:

Exception in thread "RxComputationScheduler-1" java.lang.IllegalStateException: Fatal Exception thrown on Scheduler.Worker thread. at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/dataformat/cbor/CBORParser at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$1.call(HystrixMetricsStreamServlet.java:66) at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$1.call(HystrixMetricsStreamServlet.java:63) at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:259) at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144) at rx.internal.operators.OperatorOnBackpressureDrop$2.onNext(OperatorOnBackpressureDrop.java:98) at rx.internal.operators.OnSubscribeRefCount$2.onNext(OnSubscribeRefCount.java:120) at rx.internal.operators.OperatorPublish$PublishSubscriber.dispatch(OperatorPublish.java:587) at rx.internal.operators.OperatorPublish$PublishSubscriber.onNext(OperatorPublish.java:285) at rx.observers.Subscribers$5.onNext(Subscribers.java:235) at rx.observers.Subscribers$5.onNext(Subscribers.java:235) at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) at rx.internal.operators.OnSubscribeTimerPeriodically$1.call(OnSubscribeTimerPeriodically.java:52) at rx.Scheduler$Worker$1.call(Scheduler.java:137) at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189) at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ... 7 more Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.dataformat.cbor.CBORParser at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1333) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167) ... 22 more

Now I'm not sure how to work around this, since there is obviously a Jackson on classpath already, but this is obviously not compatible to the one hystrix is using. Any hints?

[mattrjacobs]

Do you have the hystrix-serialization jar on your classpath?

[mechko]

Yes. Parts of the application I'm trying to use with Hystrix is depending on Jackson 2.2.1, but at this early time there was obviously no CBORParser, thats why excluding jackson-dataformat-cbor means class CBORParser is not there at all.

I guess I can only solve this dilemma by using a very early version of Hystrix (say 1.4.26 or the like). What do you think, would that make sense?

Edit: Fixed suggested Hystrix Version.

[mattrjacobs]

So I think the problematic path for your use-case is that SerialHystrixDashboardData has static methods for String/JSON/CBOR. I will experiment if splitting those methods into separate classes allows some of the Jackson JARs to be excluded.

Otherwise, pinning Hystrix back would work as well

[mattrjacobs]

I think a better design of this set of functionality is to be modular for serialization. If you need JSON serialization, then there should be a set of methods with a dependency on jackson-json. If you need CBOR serialization, then there should be a set of methods with a dependency on jackson-cbor.

In this PR, I've deprecated (and hollowed out) the byte[] implementations of serializations/deserializations. They can come back in a different module when I reintroduce the ReactiveSocket impls of streams (#1325).

Thoughts?

[mechko]

Thank you so much Matt, looks good so far. Is there a timeline already when 1.5.9 will be available from Maven Central?

[mattrjacobs]

It usually takes a couple of hours from the time I press the button, but it's out of my hands. Can you re-try?

[mechko]

Hi Matt, I just tested the new release 1.5.9 from Maven Central and voila: It works like a charm now! No class loader errors at runtime. My effective dependency tree looks like this now (the omitted and newer Jackson version does obviously not affect the functionality of the JSON serialization that is needed by Hystrix).

Thank you so much for your quick reaction and that nice release!

[mattrjacobs]

Great- putting all the serialization options in the same jar was a mistake by me. I'll figure out a better way to do it once we have pieces that need the CBOR serialization.