Job logging to GCS is broken on 0.50.44

[kzvezdarov]

Helm Chart Version

0.50.55

What step the error happened?

On deploy

Revelant information

We recently upgraded from 0.50.42 to 0.50.44, which caused job logging to stop functioning.

Our deployment is running on GKE Autopilot, with job logging and state storage configured to write to GCS (S3 and minio are fully disabled). This worked fine up until upgrading to 0.50.44, after which the worker and server fail to initialize Log4j due to a failure to validate the (empty) value set for S3_LOG_BUCKET_REGION.

No configuration other than the version bump has been changed and as mentioned. The logging configuration looks like this:

# S3/Minio Log Configuration
S3_LOG_BUCKET=
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=
S3_PATH_STYLE_ACCESS=
STATE_STORAGE_MINIO_BUCKET_NAME=
STATE_STORAGE_MINIO_ENDPOINT=

# GCS Log Configration
GCS_LOG_BUCKET=<gcs logs bucket>
STATE_STORAGE_GCS_BUCKET_NAME=<gcs state storage bucket>

# Cloud Storage Configuration
WORKER_LOGS_STORAGE_TYPE=gcs
WORKER_STATE_STORAGE_TYPE=gcs

Adding token (but valid) values for S3_LOG_BUCKET_REGION fails due to what appears to be a configuration conflict with having both GCS and S3 configured (partially).

Relevant log output

java.lang.RuntimeException: Cannot build appender due to errors
	at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:147)
	at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:25)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:124)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1162)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1083)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:323)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:295)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:253)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.appender.rewrite.RewriteAppender.append(RewriteAppender.java:89)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
	at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2933)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2886)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2868)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2652)
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2400)
	at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:673)
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193)
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.discover(AirbyteIntegrationLauncher.java:175)
	at io.airbyte.workers.general.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:77)
	at io.airbyte.workers.general.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:50)
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142)
	at io.airbyte.workers.temporal.discover.catalog.DiscoverCatalogActivityImpl.lambda$run$1(DiscoverCatalogActivityImpl.java:168)
	at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57)
	at io.airbyte.workers.temporal.discover.catalog.DiscoverCatalogActivityImpl.run(DiscoverCatalogActivityImpl.java:153)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
	at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
	at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216)
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Cannot create enum from ${env:S3_LOG_BUCKET_REGION} value!
	at com.amazonaws.regions.Regions.fromName(Regions.java:105)
	at com.van.logging.aws.S3Configuration.resolveRegion(S3Configuration.java:204)
	at com.van.logging.aws.S3Configuration.setRegion(S3Configuration.java:125)
	at com.van.logging.log4j2.Log4j2AppenderBuilder.getS3ConfigIfEnabled(Log4j2AppenderBuilder.java:172)
	at com.van.logging.log4j2.Log4j2AppenderBuilder.createCachePublisher(Log4j2AppenderBuilder.java:271)
	at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:140)
	... 50 more

[kzvezdarov]

Just tested 0.50.43, logging appears to work fine. Unfortunately I am unable to downgrade because 0.50.42 and 0.50.43 have an unrelated issue which prevents connector version upgrades (which 0.50.44 fixes).

[marcosmarxm]

Duplicated of #34539. @kzvezdarov a fix was merged waiting a new release to be made.