[None][feat] Nano-v3 stack PRs v2 #9062
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Add conv act fusion Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> fix unit tests Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> fix tests Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Address reviewer's comments Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> fix typo Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Fixes and UT Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Use trtllm moe for relu2 mlp case Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Fix the runGemmProfile Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Replace the FP8 fused MoE backend Before: torch.ops.auto_deploy.triton_quant_fp8_moe After: torch.ops.auto_deploy.trtllm_quant_fp8moe_fused Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Code refactoring Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> syntax error fixes Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> remove dead code Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> fix moe operator function name Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Add skips if not hopper+ Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> remove unused code Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com>
1 task
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
force-pushed
the
user/williamj/nano-v3-fp8-stack
branch
from
49a0c40 to
7664abb
Compare
Member
|
fyi, just merged #8812 --> so you can drop the PR from the list next time you rebase |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Features:
This PR stacked the following PRs on top of TRTLLM main.
[None][fixes] Add tool call parsing fixes and Qwen3 coder parser #8817 Qwen3 code tool parser
[#8763][feature] AutoDeploy: configurable dtype for caching #8812 mamba cache dtype config (Can be bf16 or fp32)
[None][feat] AutoDeploy: Perf improvement for mamba layers #8991 fuse silu to causal conv1d
fix prefill nv-auto-deploy/TensorRT-LLM#156 fix prefill
[#8732][feat] Update TRTLLM Cutlass MoE kernels with ReLU2 #9011 TRTLLM cutlass MoE kernel
Added nano_v3_bench.yaml and nano_v3_accuracy.yaml for bench and accuracy checking.
CMD to launch trtllm-serve for bench: