Unexpected error when using Cloud Run jobs command in BashOperator

[janos0207]

Apache Airflow version

2.5.1

What happened

When I ran the Cloud Run jobs execute command in BashOperator:

gcloud beta run jobs execute extract-dev-****** --project "products-*****" --region "us-east1" --wait 

And the task failed, and I got an error whose message said that the task process group received SIGTERM:

• bashop_cloud_run_jobs_error.log

But It went green when I dropped the "wait" option.

What you think should happen instead

The error message says, "WARNING - Recorded pid 6032 does not match the current pid 6033"
and so Airflow sent Signals.SIGTERM to group 6033, whose PIDs of all processes in the group: [6036, 6033].

I think this SIGTERM signal makes the BashOperator crash.

How to reproduce

No response

Operating System

Ubuntu 22.04.1 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-common-sql==1.3.3
apache-airflow-providers-ftp==3.3.0
apache-airflow-providers-google==8.8.0
apache-airflow-providers-http==4.1.1
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-sqlite==3.3.1

Deployment

Virtualenv installation

Deployment details

No response

Anything else

There are some issues similar to this problem:

• Task processes killed with WARNING - Recorded pid does not match the current pid #17507
• Recorded pid<> does not match the current pid <> #20992

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Taragolis]

As I understand that it only happen when you add --wait flag, and did failed if not.
I'm not familiar with gcloud but if it create new process and close previous than this behaviour of local_task_job.py is valid, it detects that process which initially create task run not exists anymore and kill all process group.

[janos0207]

Thank you for the comment, and sorry for my late response.
So if the gcloud CLI ended the original process and created a new one in this case, is this the correct behavior of BashOperator?
Should I report the crash problem to the gcloud CLI dev. group rather than Airflow's group?

[Taragolis]

It is just my assumption about gcloud I could be wrong.
The main question are you have any problem with other tasks?