DEADLINE_EXCEEDED StatusRuntimeException #4975

carloslme · 2021-07-25T18:55:25Z

Enviroment

Airbyte version: 0.27.0-alpha
OS Version / Instance: Ubuntu, 20.04 LTS
Deployment: Docker
Source Connector and version: MySQL 0.4.0
Destination Connector and version: S3 0.1.9
Severity: High
Step where error happened: Sync job

Current Behavior

GA connector fails with DEADLINE_EXCEEDED exception

Expected Behavior

GA connector sync succeeds

Logs

logs-46617-2.txt

Steps to Reproduce

Run GA 0.27.0-alpha

sherifnada · 2021-07-26T14:20:37Z

@jrhizor FYI as this seems to be an issue in the scheduler

jrhizor · 2021-07-26T17:04:52Z

I think we've only seen this issue when the machine is either at max disk capacity or really low on memory.

@carloslme what is your VM disk usage / memory usage like when you encounter this error?

carloslme · 2021-07-27T18:18:31Z

@jrhizor , when we encounter the error the VM disk usage was 24%, and the memory usage 90%.
We resized the instance to give it more resources, once done, and although the instance was ready, Airbyte was not able to continue working as expected, all the connectors failed with the three attempts. The sending of the information continues to fail so far.

jrhizor · 2021-07-28T00:43:55Z

Would you mind providing logs for the database and the temporal pod?

davinchia · 2021-08-23T14:10:29Z

@carloslme have you seen this since?

carloslme · 2021-09-06T20:05:52Z

Hi guys! Unfortunately, we could recover neither logs for the database nor the temporary pod because we have some internal issues.
The good news is that we are running the last version of Airbyte and everything looks great.
Thanks for your time and consideration.

davinchia · 2021-09-07T00:31:30Z

Great! Let us know if this happens again!

sherifnada · 2021-11-29T07:33:52Z

We're seeing this issue happen in a connector here: #6417

@davinchia @jrhizor is it possible to look into where in the platform this issue might be happening again? The linked ticket contains repro instructions

sherifnada · 2021-11-29T07:34:20Z

this is also happening here: #4536

davinchia · 2021-11-29T13:42:44Z

centralising error messages:

from #6417:

2021-11-17 21:53:31 WARN () ActivityExecutionContextImpl(doHeartBeat):153 - Heartbeat failed
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.999163700s. [closed=[], open=[[remote_addr=airbyte-temporal/172.18.0.8:7233]]]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.recordActivityTaskHeartbeat(WorkflowServiceGrpc.java:2710) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.sendHeartbeatRequest(ActivityExecutionContextImpl.java:203) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.doHeartBeat(ActivityExecutionContextImpl.java:147) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.heartbeat(ActivityExecutionContextImpl.java:108) ~[temporal-sdk-1.0.4.jar:?]
	at io.airbyte.workers.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:46) ~[io.airbyte-airbyte-workers-0.32.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$4(TemporalAttemptExecution.java:216) ~[io.airbyte-airbyte-workers-0.32.0-alpha.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

from #4536

2021-07-04 10:35:24 ERROR (/tmp/workspace/1032/0) DefaultAirbyteStreamFactory(internalLog):108 - time=2021-07-04 10:35:19 name=googleapiclient.discovery level=INFO message=URL being requested: POST https://analyticsreporting.googleapis.com/v4/reports:batchGet?alt=json

2021-07-04 10:35:24 ERROR (/tmp/workspace/1032/0) DefaultAirbyteStreamFactory(internalLog):108 - time=2021-07-04 10:35:21 name=googleapiclient.discovery level=INFO message=URL being requested: POST https://analyticsreporting.googleapis.com/v4/reports:batchGet?alt=json

2021-07-04 10:35:24 ERROR (/tmp/workspace/1032/0) DefaultAirbyteStreamFactory(internalLog):108 - time=2021-07-04 10:35:22 name=googleapiclient.discovery level=INFO message=URL being requested: POST https://analyticsreporting.googleapis.com/v4/reports:batchGet?alt=json

2021-07-04 10:44:39 WARN (/tmp/workspace/1032/0) ActivityExecutionContextImpl(doHeartBeat):153 - Heartbeat failed
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 8.397817165s. [remote_addr=airbyte-temporal/172.19.0.3:7233]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.recordActivityTaskHeartbeat(WorkflowServiceGrpc.java:2710) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.sendHeartbeatRequest(ActivityExecutionContextImpl.java:203) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.doHeartBeat(ActivityExecutionContextImpl.java:147) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.heartbeat(ActivityExecutionContextImpl.java:108) [temporal-sdk-1.0.4.jar:?]
	at io.airbyte.workers.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:68) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$4(TemporalAttemptExecution.java:194) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
2021-07-04 10:45:22 WARN (/tmp/workspace/1032/0) GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 120.999974677s. [remote_addr=airbyte-temporal/172.19.0.3:7233]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) [temporal-sdk-1.0.4.jar:?]
	at com.sun.proxy.$Proxy41.run(Unknown Source) [?:?]
	at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:123) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:143) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:122) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
2021-07-04 10:49:51 WARN (/tmp/workspace/1032/0) GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 120.992742305s. [remote_addr=airbyte-temporal/172.19.0.3:7233]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) [temporal-sdk-1.0.4.jar:?]
	at com.sun.proxy.$Proxy41.run(Unknown Source) [?:?]
	at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:123) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:143) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:122) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
2021-07-04 10:51:55 WARN (/tmp/workspace/1032/0) GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 120.992584644s. [remote_addr=airbyte-temporal/172.19.0.3:7233]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) [temporal-sdk-1.0.4.jar:?]
	at com.sun.proxy.$Proxy41.run(Unknown Source) [?:?]
	at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:123) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:143) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:122) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
2021-07-04 10:56:00 WARN (/tmp/workspace/1032/0) GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 120.988033142s. [remote_addr=airbyte-temporal/172.19.0.3:7233]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) [temporal-sdk-1.0.4.jar:?]
	at com.sun.proxy.$Proxy41.run(Unknown Source) [?:?]
	at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:123) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:143) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:122) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
2021-07-04 10:59:23 WARN (/tmp/workspace/1032/0) GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 120.972979011s. [remote_addr=airbyte-temporal/172.19.0.3:7233]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) [temporal-sdk-1.0.4.jar:?]
	at com.sun.proxy.$Proxy41.run(Unknown Source) [?:?]
	at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:123) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:143) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:122) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
2021-07-04 11:07:15 WARN (/tmp/workspace/1032/0) GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 120.990681832s. [remote_addr=airbyte-temporal/172.19.0.3:7233]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.34.1.jar:1.34.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) [temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) [temporal-sdk-1.0.4.jar:?]
	at com.sun.proxy.$Proxy41.run(Unknown Source) [?:?]
	at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:123) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:143) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:122) [io.airbyte-airbyte-workers-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.25.0-alpha.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

davinchia · 2021-11-29T13:44:32Z

Something is causing Temporal to not respond to the status check RPC call. Trying to reproduce on Cloud with the Stripe -> BQ connector as noted in #6417

sherifnada · 2021-12-01T17:37:59Z

same issue happening here : logs-2912-0 .txt

PVMer · 2021-12-03T06:30:45Z

@sherifnada any updates on this issue would be much appreciated.

awoehrl · 2022-02-01T11:50:00Z

Hi! I'm seeing the same error while using the Facebook Pages connector.

alafanechere · 2022-02-04T16:41:30Z

This error often occurs when the Airbyte instance is lacking resources. For users stumbling across this issue please try to upsize your instance. We have recommendations about sizing your Airbyte instance here. Please at least try with 16GB RAM, 4CPU, and 30GB disk.

swyxio · 2022-05-11T19:43:23Z

getting this error out of the box with default docker-compose locally. do we have recommendations on how much resources are needed for the default airbyte docker-compose experience? (literally when running the Open Source QuickStart just on a standard m1 mac mini right now and its dying)

davinchia · 2022-05-12T10:08:13Z

What are your resource settings now? I'm running Airbyte Docker on 0.38.2-alpha with 1 CPU and 2.5GB RAM and it's working fine. It might be because you are running non-ARM images and thus incurring a big performance penalty. Can you try again with 0.38.2-alpha?

swyxio · 2022-05-13T01:13:09Z

ok i was on 0.38.1 and moving to 0.38.2 (where ARM images were introduced) fixed it. m1 mac mini, i guess 1 CPU and 5GB RAM

evantahler · 2022-08-01T22:49:01Z

Is this issue still reproducible?

davinchia · 2022-08-01T23:07:48Z

@evantahler this is no longer an ARM issue, and I think this generally happens either if 1) there aren't enough resources 2) something on the job execution path has a bug. I would close this for now as this error message is a red herring with too many root causes to be useful as a single issue.

davinchia · 2022-08-01T23:08:09Z

I'm closing this for now.

carloslme added the type/bug Something isn't working label Jul 25, 2021

sherifnada added the area/platform issues related to the platform label Jul 26, 2021

davinchia closed this as completed Sep 7, 2021

sherifnada reopened this Nov 29, 2021

sherifnada mentioned this issue Dec 1, 2021

Iterable Sync timeout/cancelation due to Memory Error/Heartbeat Failure #7558

Closed

alafanechere mentioned this issue Feb 4, 2022

airbyte-server & worker - grpc DEADLINE_EXCEEDED #10053

Closed

bleonard added autoteam team/compose team/platform-move labels Apr 26, 2022

alafanechere mentioned this issue Jul 12, 2022

Kubernetes: Airbyte check connection issue on GKE #14536

Closed

evantahler added team/tse Technical Support Engineers and removed team/tse Technical Support Engineers labels Aug 1, 2022

davinchia closed this as completed Aug 1, 2022

nullmaxwell mentioned this issue Oct 26, 2022

airbyte-server StatusRuntimeException: DEADLINE_EXCEEDED on Initial Deployment #18516

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEADLINE_EXCEEDED StatusRuntimeException #4975

DEADLINE_EXCEEDED StatusRuntimeException #4975

carloslme commented Jul 25, 2021

sherifnada commented Jul 26, 2021

jrhizor commented Jul 26, 2021

carloslme commented Jul 27, 2021

jrhizor commented Jul 28, 2021

davinchia commented Aug 23, 2021

carloslme commented Sep 6, 2021

davinchia commented Sep 7, 2021

sherifnada commented Nov 29, 2021

sherifnada commented Nov 29, 2021

davinchia commented Nov 29, 2021

davinchia commented Nov 29, 2021

sherifnada commented Dec 1, 2021

PVMer commented Dec 3, 2021

awoehrl commented Feb 1, 2022

alafanechere commented Feb 4, 2022

swyxio commented May 11, 2022 •

edited

Loading

davinchia commented May 12, 2022 •

edited

Loading

swyxio commented May 13, 2022 •

edited

Loading

evantahler commented Aug 1, 2022

davinchia commented Aug 1, 2022

davinchia commented Aug 1, 2022

DEADLINE_EXCEEDED StatusRuntimeException #4975

DEADLINE_EXCEEDED StatusRuntimeException #4975

Comments

carloslme commented Jul 25, 2021

Enviroment

Current Behavior

Expected Behavior

Logs

Steps to Reproduce

sherifnada commented Jul 26, 2021

jrhizor commented Jul 26, 2021

carloslme commented Jul 27, 2021

jrhizor commented Jul 28, 2021

davinchia commented Aug 23, 2021

carloslme commented Sep 6, 2021

davinchia commented Sep 7, 2021

sherifnada commented Nov 29, 2021

sherifnada commented Nov 29, 2021

davinchia commented Nov 29, 2021

davinchia commented Nov 29, 2021

sherifnada commented Dec 1, 2021

PVMer commented Dec 3, 2021

awoehrl commented Feb 1, 2022

alafanechere commented Feb 4, 2022

swyxio commented May 11, 2022 • edited Loading

davinchia commented May 12, 2022 • edited Loading

swyxio commented May 13, 2022 • edited Loading

evantahler commented Aug 1, 2022

davinchia commented Aug 1, 2022

davinchia commented Aug 1, 2022

swyxio commented May 11, 2022 •

edited

Loading

davinchia commented May 12, 2022 •

edited

Loading

swyxio commented May 13, 2022 •

edited

Loading