fix(quantization): Skip weight initialization for quantized models

[mauricioharley]

Description

This Pull Request fixes a RuntimeError that occurs when loading llmcompressor W8A8 quantized models (e.g., RedHatAI/Qwen2.5-VL-7B-Instruct-quantized.w8a8) due to an attempt to initialize int8 weights using torch.nn.init.normal_(), which only supports floating-point dtypes.

The issue was identified in modeling_utils.py within the _initialize_missing_keys method. When is_quantized is True, the else branch was still calling self.initialize_weights(), leading to the RuntimeError.

Proposed Change

Added a conditional check if not is_quantized: before the call to self.initialize_weights() in the else branch of the _initialize_missing_keys method. This ensures that weight initialization is skipped for quantized models, as their weights are either already defined or will be loaded from a pretrained state dictionary, making the initialization redundant and problematic.

Related Issue

Closes #39366

[ydshieh]

cc @MekkCyber @SunMarc

[SunMarc]

quantized models shouldn't be handled differently. Maybe we need to set _is_hf_initialized somewhere in the modules that are impacted.

[mauricioharley]

Thanks for the catch, @SunMarc! I restored the regular initialize_weights() path so quantized models no longer skip it, and added a hook in the quantizers to mark the tensors we create as _is_hf_initialized. That way the initialization logic still runs, but the quantized parameters stay untouched.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: fbgemm_fp8, finegrained_fp8, higgs, hqq, mxfp4

[ArthurZucker]

hey! Sorry for the radio silence, if you still want to work on this we merged #41580 which changes this a lot!