Try to speed up kafka admin CLI invocations #25

solsson · 2019-06-13T05:29:47Z

... using native image ideally. If that fails a small improvement might be achieved by bypassing the shell wrapper and reducing the volume of classpath libs.

./bin/kafka-topics.sh to kafka-topics is the first step, but it currently fails on GraalVM native-image with jopt-simple.

based on the observation that, bar custom env, the analysis done at every start is static. Utilities that run as for example kube jobs could be speedier if we mod the entrypoint.

Might still be useful becasue it hides the hideous classpath string and is less picky than regular java invocations in CLI args ordering

…-zookeeper

but why does it require the lz4 lib at all for mere topic admin? The stack trace indicates an issue in CompressionType which could be what is worked around in https://github.com/quarkusio/quarkus/tree/0.16/extensions/kafka-client/runtime/src/main/java/io/quarkus/kafka/client/runtime/graal Can we instead block analysis before it ends up in CompressionType stuff, for the topic admin use case?

…issues

though https://ops.gitlab.net/ongresinc/pgio seems to succeed

... and with these resource limits it takes ~20s on my local dev cluster.

solsson · 2019-06-15T04:54:38Z

solsson/kafka-topics@sha256:3506069ce16ee683352d7d9811f0588f4433bdd584985186a095352b76ecafeb

Caused by: java.lang.InstantiationException: Type `sun.security.provider.ConfigFile` can not be instantiated reflectively as it does not have a no-parameter constructor or the no-parameter constructor has not been added explicitly to the native image.

"or the no-parameter constructor has not been added explicitly to the native image"

…le in state: CONNECTING

com.fasterxml.jackson.databind.ext.Java7Support <clinit> WARNING: Unable to load JDK7 types (annotations, java.nio.file.Path): no Java7 support added

…718)

but I don't want to change download paths now and here we're not using it as a runtime image.

back to a reduced classpath: Discovered unresolved type during parsing: io.netty.util.internal.logging.Log4J2Logger

solsson · 2019-12-23T05:58:21Z

Did some experiments with Kafka 2.4.0 (as Yolean/kubernetes-kafka#297 says it would be great to not have to specify replication factor and partitions in topic-create jobs) but I think we should wait until Quarkus is at Graal 19.3.0 and Kafka 2.4.0. Currently builds fail with:

Parameter 0 of org.apache.kafka.common.utils.Utils.closeQuietly(AutoCloseable, String) has declared type java.lang.AutoCloseable which is incompatible with types in state
...
Detailed message:
Call path from entry point to org.apache.kafka.common.utils.Utils.closeQuietly(AutoCloseable, String): 
	at org.apache.kafka.common.utils.Utils.closeQuietly(Utils.java:832)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1220)

solsson · 2020-07-10T06:04:33Z

#29 builds on the discoveries from this PR and produces up to date images solsson/kafka:native-cli.

solsson added 14 commits June 11, 2019 05:18

Adds an experimental way around the bash scripts,

0ef16c0

based on the observation that, bar custom env, the analysis done at every start is static. Utilities that run as for example kube jobs could be speedier if we mod the entrypoint.

The CLI seems to run well enough without all the flags

d9e4172

Adds unsuccessful native image build

79b2686

Might still be useful becasue it hides the hideous classpath string and is less picky than regular java invocations in CLI args ordering

Minimizes the set of libs, based on trial and error, no support for -…

fc50dfa

…-zookeeper

Supports --zookeeper

c000241

Fixes logging, but we might want NOP

8d3dcb8

Native compile finds more dependencies despite a working command

f03f3d9

Creates a new topic every time

92044e5

Builds the fallback image, so we get consistent entrypoints

af69e6c

native-image passes now, but the resulting executable has reflection …

3d6954e

…issues

Making jopt-simple work with native-image might be a science project,

e862b8c

though https://ops.gitlab.net/ongresinc/pgio seems to succeed

Progress: segfault at actual kafka access, i.e. CLI args work

5396514

Real progress: Without --static and with graal's linux the test passes

1c5e9f8

... and with these resource limits it takes ~20s on my local dev cluster.

solsson added 15 commits June 17, 2019 06:46

Latest graal + an attempt to fix ZK client

c71123d

Use kubectl create for test jobs. TTL is coming.

a443627

Another guess, based on the part of the error message that says:

0f7c1ea

"or the no-parameter constructor has not been added explicitly to the native image"

ClassNotFoundException: org.apache.zookeeper.ClientCnxnSocketNIO

7af2959

NoSuchMethodException: org.apache.zookeeper.ClientCnxnSocketNIO.<init>

ddf9ffa

Did some digging into source

764e1ca

ZooKeeperClientTimeoutException: Timed out waiting for connection whi…

85757ad

…le in state: CONNECTING

The binary grows quite a bit for every class here

60377e1

This build can connect to a local zookeeper 3.5.5

0b7f7d4

Last build logged a Jackson error at runtime:

6cf1b02

com.fasterxml.jackson.databind.ext.Java7Support <clinit> WARNING: Unable to load JDK7 types (annotations, java.nio.file.Path): no Java7 support added

Scripts often need to combine different CLIs

75d09f3

The server saved <10% compilation time; prefer separate layers

cdbc7eb

We frequently need kafka-consumer-groups as well

3515569

Tests pass; only kafka-configs get the jackson Java7Support error

90e1263

solsson added 19 commits June 19, 2019 17:26

Makes this image a drop-in replacement w.r.t. scripting

e26f811

Make Kafka client compression libs available, in case we get JNI working

7603fe5

Fix autobuild

6237867

Try to support kafka-consumer-groups, i.e. arg type long

50590c0

Easier than maven publish :)

0071f4a

Runs but requires a String deserializer

74023c7

at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:…

a2bbbae

…718)

Serializers are insantiated using the no-args constructor

3f8d6c0

You don't know until real testing which reflections are missing

a2cacaf

Yay, Kafka 2.4.0 has a lot more jars in classpath

489a25b

Latest graalvm

4abc4bf

Use Quarkus 1.1.0 jars

b9ed4ea

This is the full classpath from the aforementioned docker run+exec

087c6b9

Try compiling with the full classpath, no manual work

197d3fd

The official curl image run nonroot,

6f9f187

but I don't want to change download paths now and here we're not using it as a runtime image.

Quarkus' kafka JMX replace didn't match 2.4.0

acd2870

native-image bailed on an error that hints we might need to revert

2f54c99

back to a reduced classpath: Discovered unresolved type during parsing: io.netty.util.internal.logging.Log4J2Logger

Reuses the old reduced classpath, but bumps versions

572e34d

Fixes a class not found

ba7af9b

solsson added 3 commits March 25, 2020 06:32

Quarkus 1.3.0 has kafka client 2.4.0 so we'll include that

3dac2e0

Run on nonroot-ish, we still want PATH, user 65532

ebe20f1

Don't just quit silently

688327a

solsson mentioned this pull request Apr 24, 2020

Native-image i.e. GraalVM builds of Zookeeper and CLIs #29

Merged

solsson closed this Jul 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try to speed up kafka admin CLI invocations #25

Try to speed up kafka admin CLI invocations #25

solsson commented Jun 13, 2019

solsson commented Jun 15, 2019

solsson commented Dec 23, 2019 •

edited

Loading

solsson commented Jul 10, 2020

Try to speed up kafka admin CLI invocations #25

Try to speed up kafka admin CLI invocations #25

Conversation

solsson commented Jun 13, 2019

solsson commented Jun 15, 2019

solsson commented Dec 23, 2019 • edited Loading

solsson commented Jul 10, 2020

solsson commented Dec 23, 2019 •

edited

Loading