Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix possible double score submission when auto-retrying via perfect mod #26333

Merged
merged 1 commit into from
Jan 3, 2024

Conversation

bdach
Copy link
Collaborator

@bdach bdach commented Jan 2, 2024

Closes #26035.

submitOnFailOrQuit(), as the name suggests, can be called both when the player has failed, or when the player screen is being exited from. Notably, when perfect mod with auto-retry is active, the two happen almost simultaneously.

This double call exposes a data race in submitScore() concerning the handling of scoreSubmissionSource. The race could be experimentally confirmed by applying the following patch:

diff --git a/osu.Game/Screens/Play/SubmittingPlayer.cs b/osu.Game/Screens/Play/SubmittingPlayer.cs index 83adf1f960..76dd29bbdb 100644
--- a/osu.Game/Screens/Play/SubmittingPlayer.cs
+++ b/osu.Game/Screens/Play/SubmittingPlayer.cs
@@ -228,6 +228,7 @@ private Task submitScore(Score score)
                 return Task.CompletedTask;
             }

+            Logger.Log($"{nameof(scoreSubmissionSource)} is {(scoreSubmissionSource == null ? "null" : "not null")}");
             if (scoreSubmissionSource != null)
                 return scoreSubmissionSource.Task;

@@ -237,6 +238,7 @@ private Task submitScore(Score score)

             Logger.Log($"Beginning score submission (token:{token.Value})...");

+            Logger.Log($"creating new {nameof(scoreSubmissionSource)}");
             scoreSubmissionSource = new TaskCompletionSource<bool>();
             var request = CreateSubmissionRequest(score, token.Value);

which would result in the following log output:

[runtime] 2024-01-02 09:54:13 [verbose]: scoreSubmissionSource is null
[runtime] 2024-01-02 09:54:13 [verbose]: scoreSubmissionSource is null
[runtime] 2024-01-02 09:54:13 [verbose]: Beginning score submission (token:36780)...
[runtime] 2024-01-02 09:54:13 [verbose]: creating new scoreSubmissionSource
[runtime] 2024-01-02 09:54:13 [verbose]: Beginning score submission (token:36780)...
[runtime] 2024-01-02 09:54:13 [verbose]: creating new scoreSubmissionSource
[network] 2024-01-02 09:54:13 [verbose]: Performing request osu.Game.Online.Solo.SubmitSoloScoreRequest
[network] 2024-01-02 09:54:14 [verbose]: Request to https://dev.ppy.sh/api/v2/beatmaps/869310/solo/scores/36780 successfully completed!
[network] 2024-01-02 09:54:14 [verbose]: SubmitSoloScoreRequest finished with response size of 639 bytes
[network] 2024-01-02 09:54:14 [verbose]: Performing request osu.Game.Online.Solo.SubmitSoloScoreRequest
[runtime] 2024-01-02 09:54:14 [verbose]: Score submission completed! (token:36780 id:20247)
[network] 2024-01-02 09:54:14 [verbose]: Request to https://dev.ppy.sh/api/v2/beatmaps/869310/solo/scores/36780 successfully completed!
[network] 2024-01-02 09:54:14 [verbose]: SubmitSoloScoreRequest finished with response size of 639 bytes
[runtime] 2024-01-02 09:54:14 [error]: An unhandled error has occurred.
[runtime] 2024-01-02 09:54:14 [error]: System.InvalidOperationException: An attempt was made to transition a task to a final state when it had already completed.
[runtime] 2024-01-02 09:54:14 [error]: at osu.Game.Screens.Play.SubmittingPlayer.<>c__DisplayClass30_0.<submitScore>b__0(MultiplayerScore s) in /home/dachb/Documents/opensource/osu/osu.Game/Screens/Play/SubmittingPlayer.cs:line 250

The intention of the submission logic was to only ever create one scoreSubmissionSource, and then reuse this one if a redundant submission request was made. However, because of the temporal proximity of fail and quit in this particular case, combined with the fact that the calls to submitScore() are taking place on TPL threads, means that there is a read-write data race on scoreSubmissionSource, wherein the source can be actually created twice.

This leads to two concurrent score submission requests, which, upon completion, attempt to transition only the second scoreSubmissionSource to a final state (this is because the API success/failure request callbacks capture this, i.e. the entire SubmittingPlayer instance, rather than the scoreSubmissionSource reference specifically).

To fix, ensure correct synchronisation on the read-write critical section, which should prevent the scoreSubmissionSource from being created multiple times.

Closes ppy#26035.

`submitOnFailOrQuit()`, as the name suggests, can be called both when
the player has failed, or when the player screen is being exited from.
Notably, when perfect mod with auto-retry is active, the two happen
almost simultaneously.

This double call exposes a data race in `submitScore()` concerning the
handling of `scoreSubmissionSource`. The race could be experimentally
confirmed by applying the following patch:

diff --git a/osu.Game/Screens/Play/SubmittingPlayer.cs b/osu.Game/Screens/Play/SubmittingPlayer.cs
index 83adf1f960..76dd29bbdb 100644
--- a/osu.Game/Screens/Play/SubmittingPlayer.cs
+++ b/osu.Game/Screens/Play/SubmittingPlayer.cs
@@ -228,6 +228,7 @@ private Task submitScore(Score score)
                 return Task.CompletedTask;
             }

+            Logger.Log($"{nameof(scoreSubmissionSource)} is {(scoreSubmissionSource == null ? "null" : "not null")}");
             if (scoreSubmissionSource != null)
                 return scoreSubmissionSource.Task;

@@ -237,6 +238,7 @@ private Task submitScore(Score score)

             Logger.Log($"Beginning score submission (token:{token.Value})...");

+            Logger.Log($"creating new {nameof(scoreSubmissionSource)}");
             scoreSubmissionSource = new TaskCompletionSource<bool>();
             var request = CreateSubmissionRequest(score, token.Value);

which would result in the following log output:

	[runtime] 2024-01-02 09:54:13 [verbose]: scoreSubmissionSource is null
	[runtime] 2024-01-02 09:54:13 [verbose]: scoreSubmissionSource is null
	[runtime] 2024-01-02 09:54:13 [verbose]: Beginning score submission (token:36780)...
	[runtime] 2024-01-02 09:54:13 [verbose]: creating new scoreSubmissionSource
	[runtime] 2024-01-02 09:54:13 [verbose]: Beginning score submission (token:36780)...
	[runtime] 2024-01-02 09:54:13 [verbose]: creating new scoreSubmissionSource
	[network] 2024-01-02 09:54:13 [verbose]: Performing request osu.Game.Online.Solo.SubmitSoloScoreRequest
	[network] 2024-01-02 09:54:14 [verbose]: Request to https://dev.ppy.sh/api/v2/beatmaps/869310/solo/scores/36780 successfully completed!
	[network] 2024-01-02 09:54:14 [verbose]: SubmitSoloScoreRequest finished with response size of 639 bytes
	[network] 2024-01-02 09:54:14 [verbose]: Performing request osu.Game.Online.Solo.SubmitSoloScoreRequest
	[runtime] 2024-01-02 09:54:14 [verbose]: Score submission completed! (token:36780 id:20247)
	[network] 2024-01-02 09:54:14 [verbose]: Request to https://dev.ppy.sh/api/v2/beatmaps/869310/solo/scores/36780 successfully completed!
	[network] 2024-01-02 09:54:14 [verbose]: SubmitSoloScoreRequest finished with response size of 639 bytes
	[runtime] 2024-01-02 09:54:14 [error]: An unhandled error has occurred.
	[runtime] 2024-01-02 09:54:14 [error]: System.InvalidOperationException: An attempt was made to transition a task to a final state when it had already completed.
	[runtime] 2024-01-02 09:54:14 [error]: at osu.Game.Screens.Play.SubmittingPlayer.<>c__DisplayClass30_0.<submitScore>b__0(MultiplayerScore s) in /home/dachb/Documents/opensource/osu/osu.Game/Screens/Play/SubmittingPlayer.cs:line 250

The intention of the submission logic was to only ever create one
`scoreSubmissionSource`, and then reuse this one if a redundant
submission request was made. However, because of the temporal proximity
of fail and quit in this particular case, combined with the fact that
the calls to `submitScore()` are taking place on TPL threads, means that
there is a read-write data race on `scoreSubmissionSource`, wherein the
source can be actually created twice.

This leads to two concurrent score submission requests, which, upon
completion, attempt to transition only _the second_
`scoreSubmissionSource` to a final state (this is because the API
success/failure request callbacks capture `this`, i.e. the entire
`SubmittingPlayer` instance, rather than the `scoreSubmissionSource`
reference specifically).

To fix, ensure correct synchronisation on the read-write critical
section, which should prevent the `scoreSubmissionSource` from being
created multiple times.
@bdach
Copy link
Collaborator Author

bdach commented Jan 2, 2024

blocking because i don't like some test failures here

@bdach bdach added the blocked Don't merge this. label Jan 2, 2024
@peppy peppy self-requested a review January 3, 2024 04:00
@peppy peppy merged commit 4e7b1e1 into ppy:master Jan 3, 2024
@bdach bdach deleted the retry-perfect-unobserved-error branch January 3, 2024 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unhandled error randomly occurs when failing a map with perfect mod enabled
2 participants