Misc. bug: Concurrency Limitation: Only 6 Inferences Run Simultaneously When Setting --parallel > 6
#12013
Assignees
Labels
Comments
|
I can reproduce this. Seems like the 7th and 8th requests are not processed by the |
|
I'm unable to reproduce this issue. Did you test without the webui? (i.e. with python or curl) Here is my command: And the test code in javascript: import OpenAI from 'openai';
const client = new OpenAI({
apiKey: 'test',
baseURL: 'http://localhost:8080/v1',
});
async function cmpl(onToken) {
const stream = await client.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: 'write a short poem about cats' }],
stream: true,
});
for await (const chunk of stream) {
//process.stdout.write(chunk.choices[0]?.delta?.content || '');
onToken(chunk.choices[0]?.delta?.content || '');
}
//process.stdout.write('\n');
}
async function main() {
const np = 8;
const counters = Array(np).fill(0);
const printCounters = () => {
console.log(counters);
}
const wrapper = async (i) => {
await cmpl((token) => {
counters[i] += token.length;
printCounters();
});
};
await Promise.all(Array(np).fill(0).map((_, i) => wrapper(i)));
}
main(); |
|
The JS script works, but using the browser does not work: 
|
|
Seems like this may be related to a limitation in browser:
Though I'm not yet tested, will see a bit later |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Name and Version
❯ llama-cli --version
version: 4568 (a4417dd)
built with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.2.0
Operating systems
Mac Studio M2 Ultra (192GB)
Which llama.cpp modules do you know to be affected?
llama-server
Command line
Problem description & steps to reproduce
Description: I encountered an issue when trying to run multiple concurrent inferences with
llama-server. Even though I set the--parallelflag to 8, the server processes only 6 inferences concurrently. The remaining inferences stay in a waiting queue until one of the active processes is completed.Steps to Reproduce:
Launch the Server:
Execute the following command:
llama-server \ -m DeepSeek-R1-Distill-Llama-8B-Q8_0.gguf \ --temp 0.6 \ --ctx-size 81920 \ --parallel 8 \ --flash-attnPrepare the Requests:
Open 8 browser windows and navigate to
http://127.0.0.1:8080/.Send Prompts:
In each browser window, paste the desired prompt into the input textbox. After all windows have the prompt ready, click the “Send” button simultaneously in all 8 windows.
Observe the Behavior:
Only 6 requests begin processing immediately. The remaining 2 are queued and will only start once one of the initial 6 inferences finishes.
Expected Behavior: All 8 inferences should run concurrently as specified by the
--parallelflag.Actual Behavior: Regardless of setting
--parallelto 8 (or even using a different--ctx-sizevalue such as 18), only 6 concurrent inferences are allowed. The rest remain queued until one of the active processes is completed. RAM usage is about 50 GB, excluding the memory issue.Any insights or suggestions would be appreciated. Thank you!
First Bad Commit
No response
Relevant log output
The text was updated successfully, but these errors were encountered: