CPU SIMD and pipeline optimizations across vec/mmq/ops/kv-cache/repack

[NoahOksuz]

Summary

I was really bored in some lectures last week so i scoured through the repo for some optimisable/improvable parts so this PR accelerates multiple hot paths in ggml-cpu via multi‑ISA SIMD, better threading/cache locality and tighter inner loops. Touches vector activations, quantization, normalization kernels, KV cache, and repack paths.

• Vector ops: SIMD hardswish/hardsigmoid; improved trailing‑element handling
• Matmul/quant: parallelized A‑quant for cache locality and core utilization
• Norms: SIMD reductions in RMSNorm (fwd/bwd), GroupNorm, L2 norm
• KV cache: reordered conditions, hoisted invariants, simplified mask generation
• Repack: SIMD absolute‑max for generic quant flows (Q8_0 4x4/4x8, Q8_K 4x8)

Architectures: AVX512/AVX2/SSE2 (x86), NEON/SVE (ARM), RVV (RISC‑V), with scalar fallbacks.

Changes by area

• vec (ggml/src/ggml-cpu/vec.cpp, vec.h)
  • Added SIMD implementations for hardswish/hardsigmoid across ISAs
  • Reduced overhead for tails (clean scalar tails or single‑width fallbacks)
• mmq (mmq.cpp)
  • Parallelized quantization of matrix A; chunked to preserve locality, reduce contention
• ops (ggml/src/ggml-cpu/ops.cpp)
  • RMSNorm forward: SIMD sum‑of‑squares
  • RMSNorm backward: SIMD for sum‑of‑squares + dot
  • L2 norm: SIMD reduction
  • GroupNorm: SIMD sum and sum‑of‑squares
• KV cache (llama-kv-cache.cpp)
  • Condition reordering for better branch prediction
  • Hoisted frequently accessed values outside inner loops
  • Simplified mask generation logic
• repack (ggml/src/ggml-cpu/repack.cpp)
  • SIMD absolute‑max in generic quant functions for Q8 paths

Performance (CPU backend)

A/B vs prior commit (53d7d21) shows:

• ADD_ID: up to ~3.8x (shape‑dependent), commonly 1.3–2.0x
• MUL_MAT / MUL_MAT_ID (quantized paths): many cases 1.2–3.0x; f16/f32 often +5–30%
• FLASH_ATTN_EXT: frequent 1.2–1.7x gains; a few small‑shape regressions
• PAD_REFLECT_1D: ~2–6x
• CPY / SOFT_MAX / CONV2D: mixed; many +5–30%, some regressions (−10–40%) on specific shapes

[CISC]

I suggest using Add suggestion to batch and commit them all at once (from Files changed).

[CISC]

It looks like this is crashing:
https://github.com/ggml-org/llama.cpp/actions/runs/19199643005/job/54885609603?pr=17113#step:3:9570
https://github.com/ggml-org/llama.cpp/actions/runs/19199643005/job/54885609411?pr=17113#step:9:270

[NoahOksuz]

passed my local tests. ill look into it

[taronaeo]

I may be wrong but shouldn't this be in ggml/src/ggml-cpu/arch/x86/repack.cpp instead?

llama.cpp/ggml/src/ggml-cpu/arch/x86/repack.cpp

Line 178 in aa3b7a9

void ggml_quantize_mat_q8_0_4x8(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k) {

[Djip007]

it is best to reduce only at the end.
But modern compiler can same job with some small help:
try this on https://godbolt.org/ with for exemple
GCC / "-march=znver4 -O3 -fopenmp"

#define VECT_SIZE 16

float rmse(float* v, int N) {
    float res_v[VECT_SIZE] = {0};
    int i = 0;
    for (; i<N/VECT_SIZE; ++i) {
#       pragma omp simd
        for (int k=0; k<VECT_SIZE; ++k) {
            res_v[k] += v[i*VECT_SIZE+k]*v[i*VECT_SIZE+k];
        }
    }
    // redution;
    float res = 0;
    for (int k=0; k<VECT_SIZE; ++k) {
        res += res_v[k];
    }
    i *= VECT_SIZE;
    for (; i<N; ++i) {
        res += v[i]*v[i];
    }

    return res;
}

form me in most case intrinsic is only needed when you can't have the same on C like:

__m512 _mm512_dpbf16_ps (__m512 src, __m512bh a, __m512bh b)

or event simple:

#define VECT_SIZE 16

float rmse(float* v, int N) {
    float res = 0;
#   pragma omp simd simdlen(VECT_SIZE) reduction(+:res)
    for (int i = 0; i<N; ++i) {
        res += v[i]*v[i];
    }
    return res;
}