airbyte-server & worker - grpc DEADLINE_EXCEEDED

[magefister]

Environment

• Airbyte version: 0.35.12-alpha
• Deployment: airbyte-server & airbyte-worker
• OS Version / Instance: centOS 7.2 , Docker version 20.10.12, build e91ed57
• Severity: High
• Step where error happened: Deploy

Current Behavior

airbyte-server & airbyte-worker GRPC Deadline Exceeded

Expected Behavior

no error / heartbeat success

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG

Full Log: airbyte.log

airbyte-bootloader  | 2022-02-04 03:08:35 INFO i.a.c.p.DatabaseConfigPersistence(updateConfigsFromSeed):1486 - Connector definitions have been updated (152 new connectors, and 0 updates)
airbyte-bootloader  | 2022-02-04 03:08:35 INFO i.a.b.BootloaderApp(lambda$new$0):88 - Loaded seed data..
airbyte-bootloader  | 2022-02-04 03:08:35 INFO i.a.b.BootloaderApp(load):130 - Finished running post load Execution..
airbyte-bootloader  | 2022-02-04 03:08:35 INFO i.a.b.BootloaderApp(load):133 - Finished bootstrapping Airbyte environment..
airbyte-bootloader exited with code 0
airbyte-scheduler   | 2022-02-04 03:08:36 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...
airbyte-temporal    | {"level":"info","ts":"2022-02-04T03:08:36.681Z","msg":"temporal-sys-tq-scanner-workflow workflow successfully started","service":"worker","logging-call-at":"scanner.go:194"}
airbyte-worker      | 2022-02-04 03:08:37 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure
airbyte-worker      | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.966487413s. 
airbyte-worker      |   at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.42.1.jar:1.42.1]
airbyte-worker      |   at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.42.1.jar:1.42.1]
airbyte-worker      |   at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.42.1.jar:1.42.1]
airbyte-worker      |   at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.42.1.jar:1.42.1]
airbyte-worker      |   at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:282) ~[temporal-serviceclient-1.6.0.jar:?]
airbyte-worker      |   at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) [temporal-serviceclient-1.6.0.jar:?]
airbyte-worker      |   at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) [temporal-serviceclient-1.6.0.jar:?]
airbyte-worker      |   at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:275) [temporal-serviceclient-1.6.0.jar:?]
airbyte-worker      |   at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:182) [temporal-serviceclient-1.6.0.jar:?]
airbyte-worker      |   at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51) [temporal-serviceclient-1.6.0.jar:?]
airbyte-worker      |   at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41) [temporal-serviceclient-1.6.0.jar:?]
airbyte-worker      |   at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:50) [io.airbyte-airbyte-workers-0.35.12-alpha.jar:?]
airbyte-worker      |   at io.airbyte.workers.WorkerApp.main(WorkerApp.java:324) [io.airbyte-airbyte-workers-0.35.12-alpha.jar:?]
airbyte-scheduler   | 2022-02-04 03:08:38 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...
^CGracefully stopping... (press Ctrl+C again to force)

Steps to Reproduce

1. docker-compose up

Are you willing to submit a PR?

Remove this with your answer.

[alafanechere]

Hi @magefister this is often related to the sizing of the Airbyte instance. Could you please try to provide more resources (memory+cpu) do your instance and check if the error persists? Please check this doc, it has some hints about sizing.
What is your current instance size?

[alafanechere]

Closing because it's a duplicate of #4975

[seyal84]

we keep closing this issue but there is no resolution posted anywhere