Examples of using llms with PT2E workflow?

[cjm715]

Are there examples of using llms with PT2E workflow? I'm interested in static quantization using qwen3 .

[Xia-Weiwen]

Hi @cjm715 PT2E static quantization is for conventional CNN models other than LLMs. It produces poor accuracy when applying to LLMs. Please consider other quantization methods like AWQ, GPTQ and SmoothQuant.

[namgyu-youn]

@cjm715 for LLM task (like Qwen, DeepSeek family), you can try AWQ, GPTQ and SmoothQuant as @Xia-Weiwen mentioned. Please try

# AWQ (W4A16-INT)
python quantize_and_upload.py --model_id Qwen/Qwen3-8B --quant AWQ-INT4 --push_to_hub --calibration_limit 128

# SmoothQuant (W8A8-INT)
python quantize_and_upload.py --model_id Qwen/Qwen3-8B --quant SMOOTHQUANT-W8A8 --push_to_hub --populate_model_card_template

to publish to hugging face. TorchAO benchmarks memory usage, accuracy, throughput in vLLM. You can also check https://huggingface.co/pytorch/Phi-4-mini-instruct-AWQ-INT4 for preview.