[None][ci] run speculative unit tests serially (#9080)

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

Author: QiJune

tests/integration/defs/agg_unit_mem_df.csv

@@ -113,5 +113,4 @@ unittest/_torch/thop/parallel,NVIDIA B200,16,
 "unittest/_torch/auto_deploy/unit/singlegpu -k ""not test_trtllm_bench_backend_comparison""",NVIDIA B200,4,
 unittest/_torch/attention,NVIDIA H100,4,
 unittest/_torch/misc,NVIDIA H100,4,
-unittest/_torch/speculative,NVIDIA H100,2,
 unittest/_torch/thop/parallel,NVIDIA H100,16,