Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Helm] Sync Job S3 Log Error: Only one of Region or EndpointConfiguration may be set #34348

Closed
SEAC-Tam opened this issue Jan 18, 2024 · 6 comments
Closed

[Helm] Sync Job S3 Log Error: Only one of Region or EndpointConfiguration may be set #34348

SEAC-Tam opened this issue Jan 18, 2024 · 6 comments
Labels
area/platform issues related to the platform autoteam community helm team/compose team/platform-move type/bug Something isn't working

Comments

@SEAC-Tam
Copy link

SEAC-Tam commented Jan 18, 2024 •
edited
Loading

Helm Chart Version

0.51.5

What step the error happened?

During the Sync

Revelant information

Error on worker & replication-orchestrator pod after upgrading the chart version from 0.50.21 to 0.51.5
The Sync job did not generate any logs due to the failed initialization of Log4j2Appender.
image

my current values.yaml

minio:
  enabled: false
global:
  logs:
    minio:
      enabled: false
    storage:
      type: "S3"
    s3:
      enabled: true
      bucket: "MY_BUCKET"
      bucketRegion: "MY_REGION"
    accessKey:
      existingSecret: "my-aws-secret"
      existingSecretKey: "AWS_ACCESS_KEY_ID"
    secretKey:
      existingSecret: "my-aws-secret"
      existingSecretKey: "AWS_SECRET_ACCESS_KEY"
  state:
    storage:
      type: "S3"
worker:
  extraEnv:
    - name: STATE_STORAGE_S3_BUCKET_NAME
      value: "MY_BUCKET"
    - name: STATE_STORAGE_S3_REGION
      value: "MY_REGION"

Relevant log output

Unsetting empty environment variable 'S3_MINIO_ENDPOINT'
Unsetting empty environment variable 'ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS'
Unsetting empty environment variable 'GOOGLE_APPLICATION_CREDENTIALS'
Unsetting empty environment variable 'OTEL_COLLECTOR_ENDPOINT'
Unsetting empty environment variable 'STATE_STORAGE_MINIO_ENDPOINT'
Unsetting empty environment variable 'JOB_MAIN_CONTAINER_CPU_REQUEST'
Unsetting empty environment variable 'JOB_MAIN_CONTAINER_MEMORY_REQUEST'
Unsetting empty environment variable 'ACTIVITY_MAX_ATTEMPT'
Unsetting empty environment variable 'JOB_MAIN_CONTAINER_CPU_LIMIT'
Unsetting empty environment variable 'VERSION'
Unsetting empty environment variable 'WORKFLOW_FAILURE_RESTART_DELAY_SECONDS'
Unsetting empty environment variable 'METRIC_CLIENT'
Unsetting empty environment variable 'JOB_MAIN_CONTAINER_MEMORY_LIMIT'
Unsetting empty environment variable 'S3_PATH_STYLE_ACCESS'
Unsetting empty environment variable 'GCS_LOG_BUCKET'
Unsetting empty environment variable 'ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS'
INFO StatusConsoleListener Loading mask data from '/seed/specs_secrets_mask.yaml

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
        : airbyte-workers :

  Micronaut (v3.10.1)

i.m.c.e.DefaultEnvironment(<init>):167 - Established active environments: [k8s, cloud, ec2, control-plane]
i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):111 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
i.a.f.ConfigFileClient(<init>):105 - path /flags does not exist, will return default flag values
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable STATE_STORAGE_S3_ACCESS_KEY: ''
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable STATE_STORAGE_S3_SECRET_ACCESS_KEY: ''
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable METRIC_CLIENT: ''
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable METRIC_CLIENT: ''
i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`. 
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable STATE_STORAGE_S3_ACCESS_KEY: ''
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable STATE_STORAGE_S3_SECRET_ACCESS_KEY: ''
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
i.a.c.EnvConfigs(getEnvOrDefault):693 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):148 - Waiting for temporal server...
i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):159 - Waiting for namespace default to be initialized in temporal...
i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}
i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):176 - Temporal namespace default initialized!
i.a.w.ApplicationInitializer(initializeCommonDependencies):167 - Initializing common worker dependencies.
i.a.w.ApplicationInitializer(configureTemporal):322 - Configuring Temporal....
ERROR StatusConsoleListener Could not create plugin of type class com.van.logging.log4j2.Log4j2Appender for element Log4j2Appender: java.lang.RuntimeException: Cannot build appender due to errors
 java.lang.RuntimeException: Cannot build appender due to errors
	at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:147)
	at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:25)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:124)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1162)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1083)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:323)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:295)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:253)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.appender.rewrite.RewriteAppender.append(RewriteAppender.java:89)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
	at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2933)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2886)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2868)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2652)
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2400)
	at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
	at io.airbyte.workers.ApplicationInitializer.configureTemporal(ApplicationInitializer.java:322)
	at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:177)
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:143)
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:57)
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262)
	at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60)
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229)
	at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:600)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:594)
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298)
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104)
	at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:79)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:323)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:309)
	at io.airbyte.workers.Application.main(Application.java:15)
Caused by: java.lang.IllegalStateException: Only one of Region or EndpointConfiguration may be set.
	at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:450)
	at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
	at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
	at com.van.logging.aws.AwsClientHelpers.buildClient(AwsClientHelpers.java:88)
	at com.van.logging.aws.S3PublishHelper.<init>(S3PublishHelper.java:52)
	at com.van.logging.log4j2.Log4j2AppenderBuilder.lambda$createCachePublisher$0(Log4j2AppenderBuilder.java:276)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at com.van.logging.log4j2.Log4j2AppenderBuilder.createCachePublisher(Log4j2AppenderBuilder.java:271)
	at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:140)
	... 46 more
ERROR StatusConsoleListener Unable to invoke factory method in class com.van.logging.log4j2.Log4j2Appender for element Log4j2Appender: java.lang.IllegalStateException: No factory method found for class com.van.logging.log4j2.Log4j2Appender
 java.lang.IllegalStateException: No factory method found for class com.van.logging.log4j2.Log4j2Appender
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:268)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:140)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1162)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1083)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:323)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:295)
	at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:253)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.appender.rewrite.RewriteAppender.append(RewriteAppender.java:89)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
	at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2933)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2886)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2868)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2652)
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2400)
	at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
	at io.airbyte.workers.ApplicationInitializer.configureTemporal(ApplicationInitializer.java:322)
	at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:177)
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:143)
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:57)
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262)
	at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60)
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229)
	at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:600)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:594)
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298)
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104)
	at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:79)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:323)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:309)
	at io.airbyte.workers.Application.main(Application.java:15)
ERROR StatusConsoleListener Unable to create Appender of type Log4j2Appender
@SEAC-Tam SEAC-Tam added area/platform issues related to the platform needs-triage type/bug Something isn't working labels Jan 18, 2024
@AronsonDan
Copy link
Contributor

AronsonDan commented Jan 29, 2024 •
edited
Loading

Hey, Just ran into that issue.
Any ETA on the fix?
Should I rollback to 0.50.21 for now?

Im currently using 0.51.5

@SEAC-Tam SEAC-Tam changed the title [replication-orchestrator] Error Writing Sync Job Log to S3 [Helm] Sync Job S3 Log Error: Only one of Region or EndpointConfiguration may be set Jan 29, 2024
@SEAC-Tam
Copy link
Author

@AronsonDan
I have tested all of my sync jobs on version 0.51.5, and it seems to work fine. However, it still does not produce any logs. If you require logging, rollback to the 0.50.21 might be an option.

@SEAC-Tam
Copy link
Author

SEAC-Tam commented Feb 5, 2024

@AronsonDan
The issue has been resolved by commit #11025 with this configuration.

global:
  log4jConfig: "log4j2-no-minio.xml"

@SEAC-Tam SEAC-Tam closed this as completed Feb 5, 2024
@AronsonDan
Copy link
Contributor

Can I kindly ask which version of Airbyte helm should I install in order to resolve that?

@SEAC-Tam
Copy link
Author

SEAC-Tam commented Feb 5, 2024

@AronsonDan
I have tested with chart version 0.53.52.

@raphaelauv
Copy link

raphaelauv commented Feb 5, 2024 •
edited
Loading

I make it work airbyte/worker:0.50.47 and helm chart 0.53.52

-> #31988 (comment)

@robertomczak robertomczak mentioned this issue Feb 21, 2024
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/platform issues related to the platform autoteam community helm team/compose team/platform-move type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@marcosmarxm @raphaelauv @AronsonDan @octavia-squidington-iii @SEAC-Tam