[Kernel] Initial commit containing new Triton kernels for multi lora serving.

[FurtherAI]

SGMV Triton Kernels

New Triton kernels for multi lora computation. These (should) handle any shape and data type, apply to loras in a paged format and compute at the actual lora rank and also speed up for grouped lora requests (especially prefill).

The PR contains the kernels, tests for the kernels and benchmarks. A follow up PR will work on adding the ability to use these in vLLM.

ping @Yard1

[FurtherAI]

@Yard1 Any idea why the import fails? Works locally, but the kernel is in a new folder so maybe that path has to be added somewhere?

[Yard1]

@FurtherAI add __init__.py to the new folder

[FurtherAI]

@Yard1 Is there a way to rerun the tests without an empty commit?

[Yard1]

@FurtherAI Making an empty commit with git commit --allow-empty -m "Trigger CI" is the way to go

[tensimixt]

@FurtherAI Does this allow for larger vocabulary Sizes? For example NeMO-12B has a vocab size of 131072
If i run this version of vllm from the pull request with LoRA enabled = True will it still say

When using LoRA, vocab size must be "32000 >= vocab_size <= 128512"

[FurtherAI]

@tensimixt Yeah I think it does. It shouldn't have any issues with different sizes. I'll test it at some point

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

[github-actions]

This pull request has been automatically closed due to inactivity. Please feel free to reopen if you intend to continue working on it. Thank you!