Qwen 3 support

[maziyarpanahi]

Model description

Hi,

It seems Qwen3 models are not supported:

2025-04-29T08:58:35.746805Z  WARN text_generation_router::server: router/src/server.rs:1648: Tokenizer_config None - Some("/data/hub/models--Qwen--Qwen3-32B/snapshots/30b8421510892303dc5ddd6cd0ac90ca2053478d/tokenizer_config.json")
2025-04-29T08:58:39.544459Z  WARN text_generation_router::server: router/src/server.rs:1698: Could not parse config Error("unknown variant `qwen3`, expected one of `qwen2_5_vl`, `qwen2_vl`, `llava_next`, `clip_vision_model`, `mistral`, `mamba`, `idefics`, `mllama`, `idefics2`, `idefics3`, `ssm`, `gpt_bigcode`, `granite`, `santacoder`, `bloom`, `mpt`, `gpt2`, `gptj`, `gpt_neox`, `phi`, `phi-msft`, `phi3`, `phimoe`, `llama`, `llama4`, `baichuan`, `paligemma`, `gemma`, `gemma2`, `gemma3`, `gemma3_text`, `cohere`, `drbx`, `falcon`, `mixtral`, `starcoder2`, `qwen2`, `opt`, `t5`, `deepseek_v2`, `deepseek_v3`", line: 16, column: 23)
2025-04-29T08:58:39.544487Z  INFO text_generation_router::server: router/src/server.rs:1716: Using config None
2025-04-29T08:58:39.544545Z  WARN text_generation_router::server: router/src/server.rs:1879: Invalid hostname, defaulting to 0.0.0.0
2025-04-29T08:58:39.625639Z  INFO text_generation_router::server: router/src/server.rs:2266: Connected
2025-04-29T09:36:32.015498Z ERROR chat_completions:apply_chat_template: text_generation_router::infer: router/src/infer/mod.rs:238: Template error: invalid operation: cannot convert number to u64 (in <string>:18)

I am getting invalid operation: cannot convert number to u64 (in <string>:18) error which might be connected to the warning before, or there is something in the chat template that raise an exception inside apply_chat_template.

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f

[mht-sharma]

Hi could you please test with latest, this should now be fixed with #3200

[Ichigo3766]

Also tried awq format and that does not work either with the latest commits. Seems like its defaulting to AutoModel where its not supported.

[bcxbb]

Mine is defaulting to automodel as well

[maziyarpanahi]

Hi could you please test with latest, this should now be fixed with #3200

Hi @mht-sharma
This PR is merged, but it hasn't been released as a new image. The latest I have is 3.2.3 from a last month.

[v3ss0n]

@maziyarpanahi clone and run from that commit.

[maziyarpanahi]

@maziyarpanahi clone and run from that commit.

I have confident that commit has fixed the issue, we just need it to be officially released so we can use it in production. (custom builds are not allowed in production, they become messy)

[Ichigo3766]

I pulled the latest commit when testing and built the image manually and it was not working with that commit. I think the model may be working without quant with this commit but quants are not working yet.

[bcxbb]

I pulled the latest commit when testing and built the image manually and it was not working with that commit. I think the model may be working without quant with this commit but quants are not working yet.

I'm having the same issue

[cutiechi]

I pulled the latest commit when testing and built the image manually and it was not working with that commit. I think the model may be working without quant with this commit but quants are not working yet.

+1