Adding Support for Qwen3.5

[bozheng-hit]

This PR adds the support of codes for the upcoming Qwen3.5 series models. For information about Qwen, please visit:
👉https://qwen.ai

Special thanks to @JJJYmmm for helping complete the code in this PR. We also appreciate the valuable feedback and thorough review provided by @vasqu and @ArthurZucker ! 🙏

[1-bytes]

尽快审批～ 期待

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, qwen3_5, qwen3_5_moe

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[MaoJianwei]

wait for a long time

[BestJuly]

Hi @bozheng-hit , thanks for the PR. I have a simple question, that we noticed that the dtypes for A_log and out_norm in the HF released ckpt are different from Qwen3-Next. Should the storage of these two parameters also be in FP32? Or computation is necessary in FP32 but storage could remain the same as Qwen3-Next, in BF16.

Current implementation in MCore is using BF16 for storage and computation for these parts are in FP32. But if the storage dtypes are also required in FP32, there require make some changes (draft PR). Thank you.

[adithyabsk]

I have a similar question as @BestJuly and a follow up. @bozheng-hit wanted to better understand the choice of dtype here for A_log and out_norm and it also seems we force qkv, and a to float as well.

transformers/src/transformers/models/qwen3_5/modeling_qwen3_5.py

Lines 262 to 264 in fa6c830

	query, key, value, beta, g = [
	x.transpose(1, 2).contiguous().to(torch.float32) for x in (query, key, value, beta, g)
	]

transformers/src/transformers/models/qwen3_5/modeling_qwen3_5.py

Line 518 in fa6c830

g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)

Empirically, I see the choice of numerics for these ops matters a lot. It seems that there is a perf / capability trade off here. Has there been any investigations into different dtypes for these key ops and the subsequent impact for SFT / RL?