Skip to content

Commit

Permalink
AWS System Test: EMR Container (apache#42353)
Browse files Browse the repository at this point in the history
Ensure that the EMR cluster stays running for long enough for all the
tasks in the DAG to complete their operations against the cluster.
  • Loading branch information
o-nikolas authored and joaopamaral committed Oct 21, 2024
1 parent ef9740c commit 430bc4b
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion tests/system/providers/amazon/aws/example_emr.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,10 @@
"InstanceCount": 1,
},
],
"KeepJobFlowAliveWhenNoSteps": False,
# If the EMR steps complete too quickly the cluster will be torn down before the other system test
# tasks have a chance to run (such as the modify cluster step, the addition of more EMR steps, etc).
# Set KeepJobFlowAliveWhenNoSteps to False to avoid the cluster from being torn down prematurely.
"KeepJobFlowAliveWhenNoSteps": True,
"TerminationProtected": False,
},
"Steps": SPARK_STEPS,
Expand Down

0 comments on commit 430bc4b

Please sign in to comment.