Skip to content

Commit 99b09be

Browse files
authored
[bugfix][#280] MLA layer size calculated wrong (#281)
* Fix the layer_size calculated wrong for mla * Fix the style
1 parent 477dc28 commit 99b09be

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

ucm/integration/vllm/uc_connector.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -187,9 +187,9 @@ def DataOffset(self, kv_layer, rank, layer_id, is_v):
187187
kv_layer[1][0].numel() if not self.is_mla else 0
188188
) * elem_size
189189
# When tp > 1 layer_size = (k_min_data_block_size + v_min_data_block_size) * tp_size
190-
layer_size = (
191-
k_min_data_block_size + v_min_data_block_size
192-
) * self.total_tp_size
190+
layer_size = (k_min_data_block_size + v_min_data_block_size) * (
191+
self.total_tp_size if not self.is_mla else 1
192+
)
193193
if is_v:
194194
# Offset of v = Offset of k + k_min_data_block_size
195195
return int(

0 commit comments

Comments
 (0)