New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize BatchScheduler #142

Closed

BMurri wants to merge 87 commits into main from bmurri/parallelize-scheduler

Collaborator

BMurri commented Mar 21, 2023

Fixes #138

BMurri added 2 commits

March 20, 2023 23:38


          Parallelize BatchScheduler

6c6345a


          Missed filter

MattMcL4475 reviewed

View reviewed changes

src/TesApi.Web/Scheduler.cs Outdated Show resolved Hide resolved

MattMcL4475 reviewed

View reviewed changes

src/TesApi.Web/AzureProxy.cs Show resolved Hide resolved

MattMcL4475 reviewed

View reviewed changes

src/TesApi.Web/AzureProxy.cs Outdated Show resolved Hide resolved

MattMcL4475 reviewed

View reviewed changes

src/TesApi.Web/BatchScheduler.cs Outdated Show resolved Hide resolved

MattMcL4475 reviewed

View reviewed changes

src/TesApi.Web/BatchScheduler.cs Outdated Show resolved Hide resolved

BMurri added 3 commits

March 21, 2023 16:45


          Log to debug. These two entries should be removed before merging

1310f1c


          Completed making BatchScheduler multi-threaded and consolidated retry

a0d29fa

management, along with other fixes


          Merge branch 'main' into bmurri/parallelize-scheduler

bf28454

BMurri marked this pull request as ready for review

March 23, 2023 19:50

BMurri added 19 commits

March 23, 2023 13:11


          Merge branch 'main' into bmurri/parallelize-scheduler

f2ee8ef


          Added CancelationToken to all places touched, directly or indirectly,

abf22b2

from Scheduler processing TesTasks


          Merge branch 'main' into bmurri/parallelize-scheduler

5ee850f


          Merge branch 'main' into bmurri/parallelize-scheduler

2a5c7dc


          Merge branch 'main' into bmurri/parallelize-scheduler

27542d9


          Merge branch 'main' into bmurri/parallelize-scheduler

8698ca6


          Merge branch 'main' into bmurri/parallelize-scheduler

e5ab188


          Prevent CloudException from failing task during pool creation


          Merge branch 'main' into bmurri/parallelize-scheduler

c07ab9a


          Merge branch 'main' into bmurri/parallelize-scheduler

b3a091a


          Fix incorrect resource cleanup and make code layout more consistent

4726da0


          Address issues discovered when investigating failed unit tests

67e4b93


          Fix remaining tests and include exception type in reports

81c5671


          Merge branch 'main' into bmurri/parallelize-scheduler

f16b05b


          Batch is suddenly unable to remove unusable nodes when using ComputeN…

4cf7cda

…odeDeallocationOption.TaskCompletion


          Don't retain any on-compute-node task artifacts

989acf2


          Make calling batch api more robust, reduce batch calls for pool

04d7036

maintenance, and make state machine more robust


          Lowercase the hashes in pool/job names

9758cbd


          minor edits

995a4f7

BMurri added 8 commits

April 24, 2023 16:11


          Merge branch 'main' into bmurri/parallelize-scheduler

e4938b3


          Formatting/whitespace and usings

02fd651


          Finish making continuation token task retrieval work

90691f5


          Streamline schedualer task handling

28ae82e


          Merge branch 'main' into bmurri/parallelize-scheduler


          Various minor issues

a8bd2a8


          Handle ArgumentExceptions thrown in BatchScheduler.WhenEach()

b31f0f6


          Update TesRepositoryLazyCache.cs

015c1d5

fix error in cache handling

MattMcL4475 added this to the 4.4.0 milestone

BMurri added 10 commits

April 28, 2023 15:43


          Change cache provider

e2db1c6


          minor, yet necessary, fix

7d7b482


          Discovered that the caching code wasn't caching

fcbffd9


          Change terminal item expiration

5eea94f


          fix test

03c832f


          Merge branch 'bmurri/parallelize-scheduler' of https://github.com/mic…

fbcbbe4

…rosoft/ga4gh-tes into bmurri/parallelize-scheduler


          Merge branch 'main' into bmurri/parallelize-scheduler

665d494


          Missed edit in merge

4c9a9c9


          Cache optimization

ef537ca


          Clean database

c7970d2

MattMcL4475 marked this pull request as draft

May 17, 2023 17:42

MattMcL4475 assigned BMurri

BMurri modified the milestones: 4.4.0, next

BMurri added 4 commits

June 29, 2023 16:28


          Merge branch 'main' into bmurri/parallelize-scheduler

7697b5f


          Merge branch 'main' into bmurri/parallelize-scheduler

433d84a


          Merge branch 'main' into bmurri/parallelize-scheduler

8be736e


          fix whitespace

649d44c

Collaborator Author

BMurri commented Jan 29, 2024

With the node runner, we now have events

BMurri closed this

BMurri deleted the bmurri/parallelize-scheduler branch

January 29, 2024 16:37

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet