Add Qwen2MoE

[bozheng-hit]

Adding Qwen2MoE

This PR adds the support of codes for the coming Qwen2MoE models. For information about Qwen, please visit https://github.com/QwenLM/Qwen. @ArthurZucker

[ArthurZucker]

Thanks a few small comments for a small review for now!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Thanks! Overall alright, I think one thing we need to be careful about is to test in the terst_modeling_qwen2_moe when there are sparse and non sparse. The loss needs to ignore None router logits!

[JustinLin610]

Hey, I guess our update of the testing solves the issues mentioned. Take another look?

[ArthurZucker]

LGTM Almost not comment, I'll do another full review tomorrow but it is pretty much mergable ! 🔥

Mostly using the latest api would be nice for cache_positions in the Model, but it's really optional!

[bozheng-hit]

@ArthurZucker Hi, I have rebased our branch and solved all the conflicts. I think the codes are ready to be merged now.

[ArthurZucker]

Thanks for your efforts! merging 🥳

[ydshieh]

Hi @bozheng-hit

Thank you for adding this model 🚀

I see Qwen/Qwen1.5-MoE-A2.7B is used in the tests, but I could not find it on the Hub. See

https://huggingface.co/models?search=Qwen1.5-MoE

Could you check this, open a PR to make some necessary updates, and make sure the integration tests pass by running

RUN_SLOW=1 TF_FORCE_GPU_ALLOW_GROWTH=yes python3 -m pytest -v tests/models/qwen2_moe

Thank you in advance

[lucasjinreal]

Didn't saw Qwen/Qwen1.5-MoE-A2.7B anywhere, time estimation on new model release?