Skip to content

Commit

Permalink
AWS System Test: EMR Container (#42353)
Browse files Browse the repository at this point in the history
Ensure that the EMR cluster stays running for long enough for all the
tasks in the DAG to complete their operations against the cluster.
  • Loading branch information
o-nikolas authored Sep 19, 2024
1 parent 4da35b5 commit 3464633
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion tests/system/providers/amazon/aws/example_emr.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,10 @@
"InstanceCount": 1,
},
],
"KeepJobFlowAliveWhenNoSteps": False,
# If the EMR steps complete too quickly the cluster will be torn down before the other system test
# tasks have a chance to run (such as the modify cluster step, the addition of more EMR steps, etc).
# Set KeepJobFlowAliveWhenNoSteps to False to avoid the cluster from being torn down prematurely.
"KeepJobFlowAliveWhenNoSteps": True,
"TerminationProtected": False,
},
"Steps": SPARK_STEPS,
Expand Down

0 comments on commit 3464633

Please sign in to comment.