Skip to content

Commit eaeca3c

Browse files
yyzxwhmellor
andauthored
[Bugfix] Parse SpeculativeConfig Error (vllm-project#25142)
Signed-off-by: zxw <1020938856@qq.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
1 parent 12c1287 commit eaeca3c

File tree

1 file changed

+12
-6
lines changed

1 file changed

+12
-6
lines changed

vllm/engine/arg_utils.py

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1465,12 +1465,18 @@ def _is_v1_supported_oracle(self, model_config: ModelConfig) -> bool:
14651465
return False
14661466

14671467
# V1 supports N-gram, Medusa, and Eagle speculative decoding.
1468-
if (self.speculative_config is not None
1469-
and self.speculative_config.get("method") == "draft_model"):
1470-
raise NotImplementedError(
1471-
"Speculative decoding with draft model is not supported yet. "
1472-
"Please consider using other speculative decoding methods "
1473-
"such as ngram, medusa, eagle, or deepseek_mtp.")
1468+
if self.speculative_config is not None:
1469+
# speculative_config could still be a dict at this point
1470+
if isinstance(self.speculative_config, dict):
1471+
method = self.speculative_config.get("method", None)
1472+
else:
1473+
method = self.speculative_config.method
1474+
1475+
if method == "draft_model":
1476+
raise NotImplementedError(
1477+
"Draft model speculative decoding is not supported yet. "
1478+
"Please consider using other speculative decoding methods "
1479+
"such as ngram, medusa, eagle, or deepseek_mtp.")
14741480

14751481
V1_BACKENDS = [
14761482
"FLASH_ATTN_VLLM_V1",

0 commit comments

Comments
 (0)