[Bug]: vllm server dead issue

[pimang62]

Your current environment

I used version of v0.7.0

• Here is my envirionment, and I want to use kv_transfer_config with LMCache
• I run out the issue with only tensor parallel 4, in kubenetes + docker environment

INFO 03-27 05:06:19 llm_engine.py:232] Initializing an LLM engine (v0.7.0) with config: model='LGAI-EXAONE/EXAONE-3.5-32B-Instruct', speculative_config=None, tokenizer='LGAI-EXAONE/EXAONE-3.5-32B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=exaone1, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":128}, use_cached_outputs=True, 
WARNING 03-27 05:06:19 multiproc_worker_utils.py:298] Reducing Torch parallelism from 48 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
...
(VllmWorkerProcess pid=352) INFO 03-27 05:06:59 lmcache_connector.py:39] Initializing LMCacheConfig under kv_transfer_config kv_connector='LMCacheConnector' kv_buffer_device='cuda' kv_buffer_size=1000000000.0 kv_role='kv_both' kv_rank=None kv_parallel_size=1 kv_ip='127.0.0.1' kv_port=14579

🐛 Describe the bug

• This is server's dead message.
• I just requested 1 prompt.

INFO:     10.244.0.236:56376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 03-27 05:16:15 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250327-051615.pkl...
(VllmWorkerProcess pid=352) INFO 03-27 05:16:15 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250327-051615.pkl...
(VllmWorkerProcess pid=351) INFO 03-27 05:16:15 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250327-051615.pkl...
INFO 03-27 05:16:15 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20250327-051615.pkl.
(VllmWorkerProcess pid=352) INFO 03-27 05:16:15 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20250327-051615.pkl.
(VllmWorkerProcess pid=351) INFO 03-27 05:16:15 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20250327-051615.pkl.
ERROR 03-27 05:16:15 serving_chat.py:596] Error in chat completion stream generator.
ERROR 03-27 05:16:15 serving_chat.py:596] Traceback (most recent call last):
ERROR 03-27 05:16:15 serving_chat.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 320, in chat_completion_stream_generator
ERROR 03-27 05:16:15 serving_chat.py:596]     async for res in result_generator:
ERROR 03-27 05:16:15 serving_chat.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/client.py", line 656, in _process_request
ERROR 03-27 05:16:15 serving_chat.py:596]     raise request_output
ERROR 03-27 05:16:15 serving_chat.py:596] vllm.engine.multiprocessing.MQEngineDeadError: Engine loop is not running. Inspect the stacktrace to find the original error: DistBackendError('Error in model execution (input dumped to /tmp/err_execute_model_input_20250327-051615.pkl): NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:317, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5\nncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. \nLast error:\nError while creating shared memory segment /dev/shm/nccl-8Dg8Fo (size 7340384)').

What's the matter? Could you fix it?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[lengrongfu]

@pimang62 can you use latest version try it?