Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breaking change in error handling in Serverless API with handler using generator #397

Open
dimitribarbot opened this issue Jan 21, 2025 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@dimitribarbot
Copy link

Describe the bug
After upgrading to the 1.7.7 version of the Runpod python package, I'm facing a breaking change in the way errors are handled when using streaming in a serverless endpoint and my handler is using an async generator: errors are no longer sent in the streaming output, I have to call the status endpoint to get the error details.

This seems to be related to changes in the PR #384. While I understand the purpose of this PR, which is to return a FAILED status instead of a COMPLETED one, which is great, it seems to introduce two side effects in my project:
1- Errors are no longer sent to my streaming output (because the stream_result function is not called anymore),
2- I was sending information along the error field that I can no longer retrieve (because only the error field is kept in the stream_output), in particular the refresh_worker field. However, when I look at the code, it seems that the refresh_worker field is not handled in generators (but this is another issue).

To Reproduce
Steps to reproduce the behavior:

  1. Start from example described here: https://docs.runpod.io/serverless/workers/handlers/handler-async#writing-asynchronous-handlers
  2. Replace the yield output line with:
yield {
  "error": "An error has occurred.",
  "context": "Some context.",
  "refresh_worker": True
}
  1. Create a new job by using the run endpoint as described here.
  2. Use the stream endpoint as described here to get the streaming output and check that it is empty but with a FAILED status in runpod version 1.7.7 whereas previous versions contains the yielded output but with a COMPLETED status.
  3. Use the status endpoint as described here to get the previously yielded error (but without the context field).

Expected behavior
I'm expecting to get the error in the streaming result to avoid a new call to the status API endpoint, as well as being able to add context to the error output (although I can send it using the error field).

Otherwise, maybe add some documentation explaining how to retrieve errors when using streaming and handlers with a generator? Or maybe there is another way of getting my error information?

Screenshots
N/A

Desktop (please complete the following information):
N/A

Smartphone (please complete the following information):
N/A

@dimitribarbot dimitribarbot added the bug Something isn't working label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants