Adding support for switch-transformer / NLLB-MoE

[yl3469]

Hi vLLM Team,

Thanks again for the great work!

Recently my collaborator and I have been working on MoE and like to try to integrate vLLM with SwitchTransformer / FairSeq. I was wondering if you mind sharing if there is any effort already going on and what would be the major challenges?

https://huggingface.co/docs/transformers/main/model_doc/nllb-moe
https://github.dev/huggingface/transformers/blob/main/src/transformers/models/switch_transformers/modeling_switch_transformers.py

Thanks,
Lisa

[hmellor]

@yl3469 do you still intend to add support for this model archutecture?

[thangld201]

Hi @hmellor, if I want to add support for NLLB in vllm, what should I do ?

[hmellor]

You can follow the steps at https://docs.vllm.ai/en/latest/models/adding_model.html

[thangld201]

You can follow the steps at https://docs.vllm.ai/en/latest/models/adding_model.html

This is for CausalLM, where as NLLB is encoder-decoder. I suppose that it would be very different from what's written here ?

[hmellor]

Ah I see, in that case you'll want to head over to #187 to discuss Encoder-Decoder model support.