-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(glue-alpha): refactor spark jobs code-related props handling #33567
base: main
Are you sure you want to change the base?
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #33567 +/- ##
=======================================
Coverage 82.20% 82.20%
=======================================
Files 119 119
Lines 6862 6862
Branches 1158 1158
=======================================
Hits 5641 5641
Misses 1118 1118
Partials 103 103
Flags with carried forward coverage won't be shown. Click here to find out more.
|
After the second commit, I had to export Another thought I'm having is that there's a lot of duplicative logic in the different spark job classes, and I wonder if it can be simplified by either (1) Extracting common protected methods into |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Issue # (if applicable)
Related to #33356
Reason for this change
address the feedback section of extraJars, extraFiles, extraPythonFiles, extraJarsFirst + refactor to reduce duplicative code across the different spark job classes
Description of changes
extraPythonFiles
,extraFiles
,extraJars
, andextraJarsFirst
available in pyspark jobsextraFiles
,extraJars
, andextraJarsFirst
available in scala spark jobsSparkJob
class andSparkJobProps
to reduce duplicate logic across different spark job classessetupSparkCodeArguments
protected
method along with an interfaceSparkExtraCodeProps
to handle setting upCode
-related argumentssparkUI
propspark-ui-utils.ts
intospark-job.ts
and retire itDescribe any new or updated permissions being added
N/A
Description of how you validated changes
updated unit tests
Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license