Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FlashMLA support #1159

Open
wants to merge 31 commits into
base: master
Choose a base branch
from
Open

FlashMLA support #1159

wants to merge 31 commits into from

Conversation

EricLBuehler
Copy link
Owner

@EricLBuehler EricLBuehler commented Feb 25, 2025

Support FlashMLA for improved throughput for MLA models (DeepSeek V2, V3/R1) on CUDA.

EricLBuehler/candle#74

https://github.com/deepseek-ai/FlashMLA

Copy link

Code Metrics Report
  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           41           22           10            9
 JSON                   12          105          104            0            1
 Python                 69         2926         2534           77          315
 Shell                   1           58           22           18           18
 Plain Text              3         3723            0         2413         1310
 TOML                   19          527          489            2           36
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               48         3894            0         2953          941
 |- BASH                 6          103          100            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                15          512          433            0           79
 |- TOML                 2           75           63            0           12
 (Total)                           4717          717         2953         1047
-------------------------------------------------------------------------------
 Rust                  318       103707        92870         2075         8762
 |- Markdown           156         1763           25         1604          134
 (Total)                         105470        92895         3679         8896
===============================================================================
 Total                 479       115037        96088         7550        11399
===============================================================================
  

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant