Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[275] Removes default storage account dependency when running in Terra #285

Merged
merged 13 commits into from
Jul 14, 2023

Conversation

giventocode
Copy link
Contributor

@giventocode giventocode commented Jul 12, 2023

Closes: #275

This PR includes:

  • New methods in IStorageAccessProvider to handle storage URL creation and I/O operations for TES internal and TES tasks internal files.
    • These methods remove the dependency of the local path mapping to URLs in the scheduler and config utils to facilitate the upcoming refactoring for the TES runner.
    • For Terra, the destination is: <LZ storage account>/<workspace-container>/<batchscheduling-prefix>/tes-internal or <LZ storage account>/<workspace-container>/<batchscheduling-prefix>/tes-internal/<task-id>
      • These values can be overwritten by setting the internal_path_prefix
  • Config files for allowed and supported VMs are now in /tes-internal/configuration/ and internal_path_prefix is used if provided.

@giventocode giventocode changed the title [275] Removes defaults storage account dependency when running in Terra [275] Removes default storage account dependency when running in Terra Jul 12, 2023
Copy link
Collaborator

@BMurri BMurri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integration test failed with SYSTEMERROR, indicating that something failed with the task in TES itself.

2023-07-12T20:30:53.0444476Z Running a test task...
2023-07-12T20:33:19.1549238Z TES Task State: SYSTEMERROREnum
2023-07-12T20:33:19.1555074Z Running a test task... Completed in 146s
2023-07-12T20:33:19.1555179Z
2023-07-12T20:33:19.1555319Z Test task failed.

@giventocode giventocode requested a review from BMurri July 13, 2023 23:55
Copy link
Collaborator

@BMurri BMurri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I'd like to have @MattMcL4475's opinion, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TES internal files not configured for Terra for uploading TES runner and storing the supported-vm-sizes
3 participants