Skip to content
This repository was archived by the owner on Oct 25, 2024. It is now read-only.

Commit 44a24ec

Browse files
Improve SQ model restored from json (#1600)
Signed-off-by: Wang, Chang <chang1.wang@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 14734de commit 44a24ec

File tree

1 file changed

+6
-0
lines changed
  • intel_extension_for_transformers/transformers/utils

1 file changed

+6
-0
lines changed

intel_extension_for_transformers/transformers/utils/utility.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -462,6 +462,8 @@ def recover_model_from_json(fp32_model_name_or_path, json_file_path, trust_remot
462462
(object): quantized model
463463
"""
464464
from transformers import AutoModelForCausalLM
465+
466+
# ipex recovered int8 model from configure.json requests float32 model input and on cpu device.
465467
user_model = AutoModelForCausalLM.from_pretrained(fp32_model_name_or_path,
466468
trust_remote_code=trust_remote_code).float()
467469
if user_model.config.model_type in IPEX_OPT_LLM_SUPPORTED:
@@ -496,5 +498,9 @@ def recover_model_from_json(fp32_model_name_or_path, json_file_path, trust_remot
496498
from intel_extension_for_transformers.transformers.llm.evaluation.models import (
497499
TSModelCausalLMForITREX,
498500
)
501+
origin_model_type = config.model_type
502+
if origin_model_type in ["chatglm", "qwen", "baichuan"]:
503+
config.model_type = "qwen2"
499504
user_model = TSModelCausalLMForITREX(user_model, config=config)
505+
user_model.config.model_type = origin_model_type
500506
return user_model

0 commit comments

Comments
 (0)