Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] pydantic validation errors for ChatCompletion #3637

Open
2 of 5 tasks
xxll88 opened this issue Feb 17, 2025 · 11 comments · May be fixed by #3793
Open
2 of 5 tasks

[Bug] pydantic validation errors for ChatCompletion #3637

xxll88 opened this issue Feb 17, 2025 · 11 comments · May be fixed by #3793
Assignees

Comments

@xxll88
Copy link

xxll88 commented Feb 17, 2025

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

when use autogen with qwen2.5

messages=[SystemMessage(content='You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.', type='SystemMessage'), UserMessage(content='shanghai weather', source='user', type='UserMessage')],
client.chat.completions.create(
messages=messages,
stream=False,
tools=converted_tools,
**create_args,
)
when using the SGLang inference framework, it can normally infer and call the function named "get_weather." However, if the conversation continues and carries the history of previous chats, there will be an issue.

messages=[SystemMessage(content='You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.', type='SystemMessage'), UserMessage(content='shanghai weather', source='user', type='UserMessage'), AssistantMessage(content=[FunctionCall(id='0', arguments='{"city": "shanghai"}', name='get_weather')], source='weather_agent', type='AssistantMessage'), FunctionExecutionResultMessage(content=[FunctionExecutionResult(content='The weather in shanghai is 73 degrees and Sunny.', call_id='0', is_error=False)], type='FunctionExecutionResultMessage'), UserMessage(content='beijing weather', source='user', type='UserMessage')],

pydantic_core._pydantic_core.ValidationError: 5 validation errors for ChatCompletionRequest
messages.2.ChatCompletionMessageGenericParam.content.str
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/string_type
messages.2.ChatCompletionMessageGenericParam.content.list[ChatCompletionMessageContentTextPart]
Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/list_type
messages.2.ChatCompletionMessageUserParam.role
Input should be 'user' [type=literal_error, input_value='assistant', input_type=str]
For further information visit https://errors.pydantic.dev/2.10/v/literal_error
messages.2.ChatCompletionMessageUserParam.content.str
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/string_type
messages.2.ChatCompletionMessageUserParam.content.list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart]]
Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/list_type

Reproduction

python3 -m sglang.launch_server --model-path /mnt/vdb1/model/qwen/Qwen2___5-7B-Instruct-GPTQ-Int4/ --served-model-name qwen2-72b --host 0.0.0.0 --port 8001 --trust-remote-code --mem-fraction-static 0.2 --chunked-prefill-size 8192 --schedule-conservativeness 0.3 --tool-call-parser qwen25

Environment

ubuntu
autogen 0.4.6

@shuaills shuaills self-assigned this Feb 17, 2025
@xxll88 xxll88 changed the title [Bug] validation errors for ChatCompletionRequest [Bug] pydantic validation errors for ChatCompletionRequest Feb 17, 2025
@xxll88 xxll88 changed the title [Bug] pydantic validation errors for ChatCompletionRequest [Bug] pydantic validation errors for ChatCompletion Feb 17, 2025
@shuaills
Copy link
Collaborator

@DarkSharpness

@Edenzzzz
Copy link
Contributor

Edenzzzz commented Feb 22, 2025

I'm also seeing this in my PR #2798, when the input is None?

Image

@Superskyyy
Copy link

Superskyyy commented Feb 22, 2025

@shuaills Seeing the exact same issue, any tool call will fail on qwen25 model (I'm using Qwen2.5-72B-Instruct, apparently deepseek model have the same problem). Can someone help look into this? This is blocking and I had to fallback to vLLM beacuse of this :( I appreciate any insights to this problem.

I can confirm my LangGraph script is correct because the same script when using GPT4o API works, but fails when using SGLang.

The sglang server errors when a toolcall result (ToolMessage) is added to the context: (flow works like following snippet output from LangGraph). It is the same flow as the issue author @xxll88 , SGLang can call tools properly, but it cannot receive the tool results.

Original Query -> Model Choose Tool -> Tool Response -> Model (Fails here)

[-1:checkpoint] State at the end of step -1:
{'messages': []}
[0:tasks] Starting 1 task for step 0:
- __start__ -> {'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9')]}
[0:writes] Finished step 0 with writes to 1 channel:
- messages -> [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9')]
[0:checkpoint] State at the end of step 0:
{'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9')]}
[1:tasks] Starting 1 task for step 1:
- agent -> {'is_last_step': False,
 'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9')],
 'remaining_steps': 24}
[1:writes] Finished step 1 with writes to 1 channel:
- messages -> [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}})]
[1:checkpoint] State at the end of step 1:
{'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9'),
              AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}})]}
[2:tasks] Starting 1 task for step 2:
- tools -> {'is_last_step': False,
 'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9'),
              AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}})],
 'remaining_steps': 23}
[2:writes] Finished step 2 with writes to 1 channel:
- messages -> [ToolMessage(content='[{"snippet": "From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.", "title": "How to Boil and Eat Lobster - Simply Recipes", "link": "https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/"}, {"snippet": "Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.", "title": "How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time", "link": "https://oliviaoven.com/how-to-cook-lobster/"}, {"snippet": "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it\'s done. 4.", "title": "Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results", "link": "https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/"}, {"snippet": "Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.", "title": "How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home", "link": "https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide"}]', name='duckduckgo_results_json', tool_call_id='0', artifact=[{'snippet': 'From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.', 'title': 'How to Boil and Eat Lobster - Simply Recipes', 'link': 'https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/'}, {'snippet': 'Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.', 'title': 'How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time', 'link': 'https://oliviaoven.com/how-to-cook-lobster/'}, {'snippet': "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it's done. 4.", 'title': 'Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results', 'link': 'https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/'}, {'snippet': 'Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.', 'title': 'How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home', 'link': 'https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide'}])]
[2:checkpoint] State at the end of step 2:
{'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9'),
              AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}}),
              ToolMessage(content='[{"snippet": "From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.", "title": "How to Boil and Eat Lobster - Simply Recipes", "link": "https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/"}, {"snippet": "Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.", "title": "How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time", "link": "https://oliviaoven.com/how-to-cook-lobster/"}, {"snippet": "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it\'s done. 4.", "title": "Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results", "link": "https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/"}, {"snippet": "Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.", "title": "How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home", "link": "https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide"}]', name='duckduckgo_results_json', id='82c9c5ed-a9ba-4c19-9ce1-60344c5ceeef', tool_call_id='0', artifact=[{'snippet': 'From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.', 'title': 'How to Boil and Eat Lobster - Simply Recipes', 'link': 'https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/'}, {'snippet': 'Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.', 'title': 'How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time', 'link': 'https://oliviaoven.com/how-to-cook-lobster/'}, {'snippet': "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it's done. 4.", 'title': 'Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results', 'link': 'https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/'}, {'snippet': 'Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.', 'title': 'How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home', 'link': 'https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide'}])]}
[3:tasks] Starting 1 task for step 3:
- agent -> {'is_last_step': False,
 'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9'),
              AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}}),
              ToolMessage(content='[{"snippet": "From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.", "title": "How to Boil and Eat Lobster - Simply Recipes", "link": "https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/"}, {"snippet": "Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.", "title": "How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time", "link": "https://oliviaoven.com/how-to-cook-lobster/"}, {"snippet": "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it\'s done. 4.", "title": "Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results", "link": "https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/"}, {"snippet": "Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.", "title": "How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home", "link": "https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide"}]', name='duckduckgo_results_json', id='82c9c5ed-a9ba-4c19-9ce1-60344c5ceeef', tool_call_id='0', artifact=[{'snippet': 'From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.', 'title': 'How to Boil and Eat Lobster - Simply Recipes', 'link': 'https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/'}, {'snippet': 'Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.', 'title': 'How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time', 'link': 'https://oliviaoven.com/how-to-cook-lobster/'}, {'snippet': "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it's done. 4.", 'title': 'Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results', 'link': 'https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/'}, {'snippet': 'Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.', 'title': 'How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home', 'link': 'https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide'}])],
 'remaining_steps': 22}

Actual OpenAI SDK sent request:

{'messages': [{'content': 'CALL A Search TOOL NOW, on how to cook a lobster', 'role': 'user'}, {'content': None, 'role': 'assistant', 'tool_calls': [{'type': 'function', 'id': '0', 'function': {'name': 'duckduckgo_results_json', 'arguments': '{"query": "how to cook a lobster"}'}}]}, {'content': '[{"snippet": "From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.", "title": "How to Boil and Eat Lobster - Simply Recipes", "link": "https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/"}, {"snippet": "Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.", "title": "How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time", "link": "https://oliviaoven.com/how-to-cook-lobster/"}, {"snippet": "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it\'s done. 4.", "title": "Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results", "link": "https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/"}, {"snippet": "Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.", "title": "How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home", "link": "https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide"}]', 'role': 'tool', 'tool_call_id': '0'}], 'model': 'default', 'audio': NOT_GIVEN, 'frequency_penalty': NOT_GIVEN, 'function_call': NOT_GIVEN, 'functions': NOT_GIVEN, 'logit_bias': NOT_GIVEN, 'logprobs': NOT_GIVEN, 'max_completion_tokens': 256, 'max_tokens': NOT_GIVEN, 'metadata': NOT_GIVEN, 'modalities': NOT_GIVEN, 'n': NOT_GIVEN, 'parallel_tool_calls': False, 'prediction': NOT_GIVEN, 'presence_penalty': NOT_GIVEN, 'reasoning_effort': NOT_GIVEN, 'response_format': NOT_GIVEN, 'seed': NOT_GIVEN, 'service_tier': NOT_GIVEN, 'stop': NOT_GIVEN, 'store': NOT_GIVEN, 'stream': False, 'stream_options': NOT_GIVEN, 'temperature': 0.0, 'tool_choice': NOT_GIVEN, 'tools': [{'type': 'function', 'function': {'name': 'duckduckgo_results_json', 'description': 'A wrapper around Duck Duck Go Search. Useful for when you need to answer questions about current events. Input should be a search query.', 'parameters': {'properties': {'query': {'description': 'search query to look up', 'type': 'string'}}, 'required': ['query'], 'type': 'object'}}}], 'top_logprobs': NOT_GIVEN, 'top_p': NOT_GIVEN, 'user': NOT_GIVEN}

Only messages part in json:

{
  "messages": [
    {
      "content": "CALL A Search TOOL NOW, on how to cook a lobster",
      "role": "user"
    },
    {
      "content": null,
      "role": "assistant",
      "tool_calls": [
        {
          "type": "function",
          "id": "0",
          "function": {
            "name": "duckduckgo_results_json",
            "arguments": "{\"query\": \"how to cook a lobster\"}"
          }
        }
      ]
    },
    {
      "content": "[{\"snippet\": \"From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.\", \"title\": \"How to Boil and Eat Lobster - Simply Recipes\", \"link\": \"https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/\"}, {\"snippet\": \"Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.\", \"title\": \"How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time\", \"link\": \"https://oliviaoven.com/how-to-cook-lobster/\"}, {\"snippet\": \"For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it's done. 4.\", \"title\": \"Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results\", \"link\": \"https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/\"}, {\"snippet\": \"Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.\", \"title\": \"How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home\", \"link\": \"https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide\"}]",
      "role": "tool",
      "tool_call_id": "0"
    }
  ]
}

Image

Error message from SGLang side:

[2025-02-22 05:03:38] INFO:     127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
[2025-02-22 05:03:38] ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/usr/local/lib/python3.10/dist-packages/starlette/_utils.py", line 76, in collapse_excgroups
  |     yield
  |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 178, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
  |     raise BaseExceptionGroup(
  | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |   File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    |     return await self.app(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    |     raise exc
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 177, in __call__
    |     with recv_stream, send_stream, collapse_excgroups():
    |   File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    |     self.gen.throw(typ, value, traceback)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/_utils.py", line 82, in collapse_excgroups
    |     raise exc
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 179, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |   File "/sgl-workspace/sglang/python/sglang/srt/utils.py", line 611, in authentication
    |     return await call_next(request)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 154, in call_next
    |     raise app_exc
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 141, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    |     await self.app(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
    |     await route.handle(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
    |     await self.app(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
    |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
    |     response = await f(request)
    |   File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
    |     raw_response = await run_endpoint_function(
    |   File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    |     return await dependant.call(**values)
    |   File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/http_server.py", line 406, in openai_v1_chat_completions
    |     return await v1_chat_completions(_global_state.tokenizer_manager, raw_request)
    |   File "/sgl-workspace/sglang/python/sglang/srt/openai_api/adapter.py", line 1200, in v1_chat_completions
    |     all_requests = [ChatCompletionRequest(**request_json)]
    |   File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 214, in __init__
    |     validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
    | pydantic_core._pydantic_core.ValidationError: 5 validation errors for ChatCompletionRequest
    | messages.1.ChatCompletionMessageGenericParam.content.str
    |   Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    |     For further information visit https://errors.pydantic.dev/2.10/v/string_type
    | messages.1.ChatCompletionMessageGenericParam.content.list[ChatCompletionMessageContentTextPart]
    |   Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
    |     For further information visit https://errors.pydantic.dev/2.10/v/list_type
    | messages.1.ChatCompletionMessageUserParam.role
    |   Input should be 'user' [type=literal_error, input_value='assistant', input_type=str]
    |     For further information visit https://errors.pydantic.dev/2.10/v/literal_error
    | messages.1.ChatCompletionMessageUserParam.content.str
    |   Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    |     For further information visit https://errors.pydantic.dev/2.10/v/string_type
    | messages.1.ChatCompletionMessageUserParam.content.list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart]]
    |   Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
    |     For further information visit https://errors.pydantic.dev/2.10/v/list_type
    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 177, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 179, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/sgl-workspace/sglang/python/sglang/srt/utils.py", line 611, in authentication
    return await call_next(request)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 154, in call_next
    raise app_exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 141, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/http_server.py", line 406, in openai_v1_chat_completions
    return await v1_chat_completions(_global_state.tokenizer_manager, raw_request)
  File "/sgl-workspace/sglang/python/sglang/srt/openai_api/adapter.py", line 1200, in v1_chat_completions
    all_requests = [ChatCompletionRequest(**request_json)]
  File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 5 validation errors for ChatCompletionRequest
messages.1.ChatCompletionMessageGenericParam.content.str
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/string_type
messages.1.ChatCompletionMessageGenericParam.content.list[ChatCompletionMessageContentTextPart]
  Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/list_type
messages.1.ChatCompletionMessageUserParam.role
  Input should be 'user' [type=literal_error, input_value='assistant', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
messages.1.ChatCompletionMessageUserParam.content.str
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/string_type
messages.1.ChatCompletionMessageUserParam.content.list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart]]
  Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/list_type

@Superskyyy
Copy link

Superskyyy commented Feb 22, 2025

I'm also seeing this in my PR #2798, when the input is None?

Image

Are you also calling tools? @Edenzzzz

@shuaills
Copy link
Collaborator

Thanks, we are looking into this.

@DarkSharpness
Copy link
Contributor

@Superskyyy Currently, the content field of a message can't be null. If there's no message, simply set it as an empty string (like this).

{
      "content": "",
      "role": "assistant",
      "tool_calls": [
        {
          "type": "function",
          "id": "0",
          "function": {
            "name": "duckduckgo_results_json",
            "arguments": "{\"query\": \"how to cook a lobster\"}"
          }
        }
      ]
}

@DarkSharpness
Copy link
Contributor

Hello @xxll88. Could you please provide the messages in json? It seems that the content field in one of your messages happens to be None (= null in json), which leads to the fatal error:

messages.2.ChatCompletionMessageGenericParam.content.str
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]

@Edenzzzz
Copy link
Contributor

@Superskyyy I'm not using tool calling, just running a tester https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server.py

@Superskyyy
Copy link

I can confirm that changing to an empty string works as a workaround.

@Edenzzzz
Copy link
Contributor

So we should PR to make it convert None to an empty string?

@Superskyyy
Copy link

So we should PR to make it convert None to an empty string?

I think so, it can avoid unexpected complications.

@Edenzzzz Edenzzzz linked a pull request Feb 23, 2025 that will close this issue
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants