Performance of llama.cpp with Vulkan #10879
Replies: 221 comments 348 replies
-
|
AMD FirePro W8100
|
Beta Was this translation helpful? Give feedback.
-
|
With the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
With FA:
|
Beta Was this translation helpful? Give feedback.
-
|
AMD RX 470
|
Beta Was this translation helpful? Give feedback.
-
|
With the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
With FA:
|
Beta Was this translation helpful? Give feedback.
-
|
i got the mining edition 8 gb ran slightly better
build: e288693 (6242) |
Beta Was this translation helpful? Give feedback.
-
|
ubuntu 24.04, vulkan and cuda installed from official APT packages.
build: 4da69d1 (4351) vs CUDA on the same build/setup
build: 4da69d1 (4351) |
Beta Was this translation helpful? Give feedback.
-
|
Macbook Air M2 on Asahi Linux ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
For the record I think this is slow on the HoneyKrisp side rather than llama.cpp |
Beta Was this translation helpful? Give feedback.
-
|
Can you share how you got vulkan to build on Asahi? I can't seem to get cmake to notice it. |
Beta Was this translation helpful? Give feedback.
-
|
Spoke too soon, got it working! |
Beta Was this translation helpful? Give feedback.
-
|
Gentoo Linux on ROG Ally (2023) Ryzen Z1 Extreme ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
ggml_vulkan: Found 4 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
Cool setup! Could you also post the result of 1, 2 and 3 7900 XTX GPUs? You can use only the first GPU with |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
For this multi GPU case getting Vulkan to support #6017 pipeline parallelism might help improve the prompt processing speed. |
Beta Was this translation helpful? Give feedback.
-
|
@netrunnereve I updated the commit id in all my result. |
Beta Was this translation helpful? Give feedback.
-
|
build: 0d52a69 (4439) NVIDIA GeForce RTX 3090 (NVIDIA)
AMD Radeon RX 6800 XT (RADV NAVI21) (radv)
AMD Radeon (TM) Pro VII (RADV VEGA20) (radv)
Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver)
|
Beta Was this translation helpful? Give feedback.
-
|
@netrunnereve Some of the tg results here are a little low, I think they might be debug builds. The cmake step (at least on Linux) might require |
Beta Was this translation helpful? Give feedback.
-
|
I've added My release numbers for the RX 470 are basically identical to the ones I posted earlier without the flag.
|
Beta Was this translation helpful? Give feedback.
-
|
Maybe not in your case, but some other results are suspiciously low in tg (for example the RTX 3080) |
Beta Was this translation helpful? Give feedback.
-
|
Build: 8d59d91 (4450)
Lack of proper Xe coopmat support in the ANV driver is a setback honestly.
edit: retested both with the default batch size. |
Beta Was this translation helpful? Give feedback.
-
I put my A770 into a Windows PC and gave Intel GPA and vtune a shot: GPA just crashes most of the time, I couldn't get it to trace anything useful. vtune works, but does not support Vulkan. It just shows some high-level metrics in that case, not really useful sadly. |
Beta Was this translation helpful? Give feedback.
-
I did build it with cmake with build type Release. |
Beta Was this translation helpful? Give feedback.
-
|
In that case it's something else, cause it should be performing similarly to my A770. I suspect the mesa version, there was something in newer mesa versions that slowed down tg on Intel. |
Beta Was this translation helpful? Give feedback.
-
|
A750 has 448 CUs, A770 has 512 CUs I think. Personally, I am not worried about tg. I am worried about pp here. The gemm batch quickly saturates my GPU. |
Beta Was this translation helpful? Give feedback.
-
|
@0cc4m https://gitlab.freedesktop.org/mesa/mesa/-/issues/12585 |
Beta Was this translation helpful? Give feedback.
-
|
Here's something exotic: An AMD FirePro S10000 dual GPU from 2012 with 2x 3GB GDDR5. build: 914a82d (4452)
|
Beta Was this translation helpful? Give feedback.
-
|
Very interesting, and looks like it's pretty close to the W8100 in tg despite being a dual GPU card. Your backend scales pretty well with layer splitting which is why I find it worthwhile to run my RX470 and W8100 together (I end up getting results that are close to the average of both cards).
|
Beta Was this translation helpful? Give feedback.
-
|
Latest arch with For the sake of consistency I run every bit in a script and also build every target from scratch (for some reason kill -STOP -1
timeout 240s $COMMAND
kill -CONT -1
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Xe Graphics (TGL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | matrix cores: none
build: ff3fcab (4459)
This bit seems to underutilise both GPU and CPU in real conditions based on
|
Beta Was this translation helpful? Give feedback.
-
|
Unless you reduce the number of GPU layers, threads and openblas/non-openblas is not gonna make any difference. Try it with ngl 0, then only prompt processing is accelerated using Vulkan, the rest runs on CPU. This is often a good setting for integrated GPUs. |
Beta Was this translation helpful? Give feedback.
-
|
That's something I didn't think about, with
build: ba8a1f9 (4460)
|
Beta Was this translation helpful? Give feedback.
-
|
It seems latest patches has improved the results a bit: ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
A few months later and I get:
build: f3a4b16 (5568) I run it on Linux (Arch with lama.cpp-vulkan-git package compiled by GCC 15). From my tests, only Vulkan backend ( I'm curious why I cannot go over 6 t/s. Is this an issue with the newer llama.cpp version or with my OS configuration? |
Beta Was this translation helpful? Give feedback.
-
|
Intel ARC A770 on Windows:
build: ba8a1f9 (4460) |
Beta Was this translation helpful? Give feedback.
-
Single GPU VulkanRadeon Instinct MI25 ggml_vulkan: 0 = AMD Radeon Instinct MI25 (RADV VEGA10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Radeon PRO VII ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Multi GPU Vulkanggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Single GPU RocmDevice 0: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no
build: 2739a71 (4461) Device 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
build: 2739a71 (4461) Multi GPU RocmDevice 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
build: 2739a71 (4461) Layer split
build: 2739a71 (4461) Row split
build: 2739a71 (4461) Single GPU speed is decent, but multi GPU trails Rocm by a wide margin, especially with large models due to the lack of row split. |
Beta Was this translation helpful? Give feedback.
-
|
What is the power profile for this MI25? Mine is 110W but its running slower than yours on git from today. |
Beta Was this translation helpful? Give feedback.
-
|
Mine defaults to 220w. |
Beta Was this translation helpful? Give feedback.
-
|
AMD Radeon RX 5700 XT on Arch using mesa-git and setting a higher GPU power limit compared to the stock card.
I also think it could be interesting adding the flash attention results to the scoreboard (even if the support for it still isn't as mature as CUDA's).
|
Beta Was this translation helpful? Give feedback.
-
|
There is no Vulkan flash attention support (except with coopmat2 on very new nvidia drivers). What you're measuring here is a CPU fallback. |
Beta Was this translation helpful? Give feedback.
-
|
I see, I was sure about the CPU fallback but didn't know there was no flash attention support at all. |
Beta Was this translation helpful? Give feedback.
-
|
I tried but there's nothing after 1 hrs , ok, might be 40 mins... Anyway I run the llama_cli for a sample eval...
Meanwhile OpenBLAS |
Beta Was this translation helpful? Give feedback.
-
|
Even at below 1t/s |
Beta Was this translation helpful? Give feedback.
-
|
Truth is ... (0.79 tokens per second), 3788.63 ms / 3 tokens So it's not even...it just slower... |
Beta Was this translation helpful? Give feedback.
-
|
Nvidia MX150 (+200 core, +1000 mem), i7-8550U with 1 channel DDR4-2400:
build: 21d31e0 (7122) CUDA:
build: 4949ac0 (1) |
Beta Was this translation helpful? Give feedback.
-
|
Tiger Lake GT2 (96EU), i7-1185G7, Ubuntu 24.04, Mesa 25.0.7. This is very slow so I didn't bother trying SYCL.
build: f1ffbba (7124) |
Beta Was this translation helpful? Give feedback.
-
|
Apple M1 Mac Mini with 7-core GPU, Asahi Linux (Fedora 43) and mesa 25.2.6. Slight improvement since my last benchmark in August. #10879 (comment)
build: 2370665 (7123) |
Beta Was this translation helpful? Give feedback.
-
|
NVIDIA GeForce RTX 3060
build: 92c0b38 (7118) |
Beta Was this translation helpful? Give feedback.
-
|
NVIDIA Tesla M40 24GB Driver Version: 580.105.08 ggml_vulkan: Found 1 Vulkan devices:
build: b8372ee (7146) Strangely, PP seems to be only 1/3 of what I get with CUDA, and also slower than GTX 980. |
Beta Was this translation helpful? Give feedback.
-
|
Apple M2 Ultra on Arch Linux ARM (with Asahi drivers):
build: dbb852b (7142) |
Beta Was this translation helpful? Give feedback.
-
|
What happens if you run this in ollama on mac os? Do you get better speed? |
Beta Was this translation helpful? Give feedback.
-
|
I don't know. I do not own this device. |
Beta Was this translation helpful? Give feedback.
-
|
AMD Radeon RX 6650 XT on Arch Linux:
build: dbb852b (7142) |
Beta Was this translation helpful? Give feedback.
-
|
I've noticed a regression in build b6764 (#16203) for pp512 (AMD only) and in build b6975 (#17046) for tg128 (AMD and NVIDIA). If you can replicate these (see script below), can you add them as a comment, please, and I'll update the table.
Using pre-built binaries, e.g. https://github.com/ggml-org/llama.cpp/releases/download/b6763/llama-b6763-bin-ubuntu-vulkan-x64.zip
#!/usr/bin/env bash
# Usage: ./bench_releases.sh b6763 b6764 b6974 b6975 ...
set -euo pipefail
MODEL="models/llama-2-7b.Q4_0.gguf"
DVICE="Vulkan0"
for ver in "$@"; do
TARGET="llama-${ver}-bin-ubuntu-vulkan-x64"
[ -d "$TARGET" ] && continue
ZIP="${TARGET}.zip"
[ -f "$ZIP" ] && continue
URL="https://github.com/ggml-org/llama.cpp/releases/download/${ver}/${ZIP}"
wget -q "$URL"
rm -rf build
unzip -q "$ZIP"
mv build "$TARGET"
done
for ver in "$@"; do
BENCH="./llama-${ver}-bin-ubuntu-vulkan-x64/bin/llama-bench"
[ -x "$BENCH" ] || continue
"$BENCH" -dev "$DEVICE" -m "$MODEL"
done |
Beta Was this translation helpful? Give feedback.
-
|
NVIDIA A30 ggml_vulkan: Found 1 Vulkan devices:
build: 583cb83 (7157) For some reason it fails to allocate memory for 32768 context: |
Beta Was this translation helpful? Give feedback.
-
|
Nvidia GeForce GTX 1070 8gb ggml_vulkan: Found 3 Vulkan devices:
build: eec1e33 (7166) |
Beta Was this translation helpful? Give feedback.
-
|
AMD Radeon RX Vega 56
build: 92c0b38 (7118) |
Beta Was this translation helpful? Give feedback.
-
|
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
|
Beta Was this translation helpful? Give feedback.
-
Strix Halo is really great for large MoE models, I'm having a lot of Qwen right now. Prompt processing speed is very sensitive to device power state, but token generation isn't, probably because the bottleneck is the memory bandwidth. |
Beta Was this translation helpful? Give feedback.
-
Understood, but even if slow, you can run them. Currently I can run them only on CPU, which is exceedingly slow. Think 0.1 t/s and magnitudes slower. So even if the speed is 5-10 t/s, that would be more than acceptable. Heck, even 2-4 t/s would be fine. |
Beta Was this translation helpful? Give feedback.
-
|
@mrglutton At least on Linux you can do this on any AMD with integrated GPU, thanks to the ttm module. This is what you should be able to get from your current Ryzen's iGPU (Vulkan, recent Mesa, using ttm to extend GTT). Probably more if you've increased the TDP:
What I find this useful for is (a) running medium sized MoE models like Qwen 30B.A3B at decent speeds; and (b) running some basic models at 5W standby on my 24x7 home server and have HAProxy route to my faster system when it's online. I could run 100B+ Q4 models or large context windows on the iGPU of my 5600H or 5700U (25Watt) with 64GiB of RAM. The problem is that shaders, memory bandwidth, caches and latency do not scale. If you double the model's size your token speeds drop by at least 2x. Systems with faster APUs, higher TDPs and faster RAM (like the Strix Halo) will have higher throughput but drop at the same rate. The usefulness of the latest Ryzen AI offerings will probably end at 30B Q4 dense, 120B Q4 MoE (depending on active weights) or finetuning smaller models. |
Beta Was this translation helpful? Give feedback.
-
|
@thielj Shared memory is amazing. I did testing of mobile ryzen 5 and the results were excellent. Compiled llama.ccp for vulkan and it worked out of the box. I have 32 GB ram on my laptop and I was able to run GPT-OSS 20B. It is kind of slowish, but it is amazing that it can run it at all. I did some additional testing, and if I split offloading layers between cpu and GPU I can even utilise both processors to get some more speed (RAM speed is the same). The only two other options for large models is spending thousands on GPUs and running them on split, or running on CPU, which is glacial. |
Beta Was this translation helpful? Give feedback.
-
|
@mrglutton configure ttm and try running everything on the GPU. Splitting slows you down and eats into the GPUs power budget. With 32GiB of RAM, you should be able to run Qwen3 30B.A3B with the full 256KB context window. This uses approx 16GiB "VRAM" and 12GiB GTT. If you can't reserve large VRAM values in the BIOS, try the minimum instead (512MB). And configure 30GiB of GTT. Grep dmesg for You can get MI50 32GB cards for 150€+. They're not really fast but useable. See benchmark results in this discussion (Radeon Instinct MI50 387.37 ± 0.33 71.46 ± 0.10 ). If less than 16GiB is your sweet spot, BC-250 self contained mining blades are PS/5 hardware and deliver similar performance as the MI50 (375 and 63 t/s for the 8B llama model - faster than the strix halo). I bought mine for 60 Euro each. By the way, GTT memory is virtual and swappable. I haven't experimented with this yet, but I would expect that at the minimum, I should be able to overcommit multiple models I don't use at the same time. Swapping weights, buffers and cache back in should be faster than creating them from scratch each time. I haven't investigated how MoE weights are allocated, but they might be able to benefit from only keeping used experts in memory. |
Beta Was this translation helpful? Give feedback.
-
|
Tesla P100 16GB SXM2 For some reason I'm getting terrible PP performance, less than 1/10th of what it's supposed to be.
build: eec1e33 (7166) Edit: much better performance with
build: eec1e33 (7166) |
Beta Was this translation helpful? Give feedback.
-
|
That might be a driver thing. As far as I know, Tesla P40 supports int dot (DP4A), but not fp16, while Tesla P100 supports fp16, but not int dot. Yours reports int dot to be enabled though. Can you try running with the environment variable |
Beta Was this translation helpful? Give feedback.
-
|
Oh, didn't notice this.
build: eec1e33 (7166) |
Beta Was this translation helpful? Give feedback.
-
|
Then it's a driver issue and they shouldn't be showing support for integer dot.. |
Beta Was this translation helpful? Give feedback.
-
|
GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1 makes RADV TG speed close to AMDVLK (stopped working) ggml_vulkan: 0 = AMD Radeon RX 9070 XT (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: KHR_coopmat
build: d82b7a7 (7193) |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
This is similar to the Apple Silicon benchmark thread, but for Vulkan! We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.
Instructions
Either run the commands below or download one of our Vulkan releases. If you have multiple GPUs please run the test on a single GPU using
-sm none -mg YOUR_GPU_NUMBERunless the model is too big to fit in VRAM.Share your llama-bench results along with the git hash and Vulkan info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.
If multiple entries are posted for the same setup I'll prioritize newer commits with substantial Vulkan updates, otherwise I'll pick the one with the highest overall score at my discretion. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that the memory speed and number of channels will greatly affect your inference speed!
Vulkan Scoreboard (Click on the headings to expand the section)
Llama 2 7B, Q4_0, no FA
Llama 2 7B, Q4_0, FA enabled
Beta Was this translation helpful? Give feedback.
All reactions