Feasible to implement non-autoregressive LMs? (M2M)

[Axreub]

Hey. I'm starting to try to implement M2M in vLLM and noticed that all the currently supported models use Causal language models (decoder only), while M2M is non-autoregressive and has an encoder-decoder architecture. Is it still possible to implement a vLLM version of M2M or should I just give up 🥲 Appreciate any help with this : )

[irasin]

Hi, @Axreub, I just wonder what is the non-autoregressive LMs(M2M), can you share the paper or code about it? Thanks a lot.

[Axreub]

I mean specifically M2M-100, the translation model by Meta AI.arXiv paper 418M params model card on HF 1.2B params model card on HF. Sorry for not being clear enough previously. I'm mostly concerned about the fact that non-autoregressive models include a decoder with separate attention etc so I'm wondering if it's still feasible.

[hmellor]

Closing as duplicate of #187