Catalog / llama.cpp

0.4.0

yesterdayaddedfixedchangedOriginal notes

Overview

llama.cpp 0.4.0 adds initial Qwen3.8-Flash-Next and Nemotron-3-Puzzle support, on-demand tensor reading, per-slot server context limits, video input options, and a ggml update to 0.23.0 with major sparse flash attention and RDMA work.

API changes

  • Added llama_lazy_mode and lazy_mode (#27794).
  • Added max_buf_size to quantize params (#27795).
  • Bumped session/state versions for KV-cell token tracking (#27762).
  • Added mtmd_input_part and mtmd_tokenize_from_parts() (#28250).
  • Made mtmd_tokenize const-correct (#28310).
  • Made mtmd_bitmap_init_lazy const-correct (#28307).
  • Added mtmd helper video/init options (#24318).
  • Server multimodal helpers accept mtmd_helper_init_opt (#24318).

New models

  • Added initial Qwen3.8-Flash-Next (qwen4exp) architecture support; optimization improvements are still pending (#27742).
  • Added NVIDIA Nemotron-3-Puzzle-75B-A9B support (#25444).
  • Added DSpark support for Nemotron 3.5 (#27804).
  • Added support for nanbeige4.2-3B (#27730).

Core changes

  • Added lazy tensor reading (#27794).
  • Renamed option to --lazy-mode (#27969).
  • Added --n-cpu-ffn (#26622).
  • Added per-layer expert routing/FFN (#28323).
  • Added KV-cell token tracking (#27762).
  • Added n-gram history lookup (#28040).
  • Optimized KV cache restore (#27991).
  • Stopped early sequence scans (#28011).
  • Added quantizer RAM cap (#27795).
  • Added quantizer row-slab streaming (#27830).
  • Prevented RAM peaking during load (#27483).
  • Autoscaled YaRN training context (#28030).
  • Disabled non-fused GDN/LID (#27877).
  • Simplified MiniMax-01 graph (#27790).
  • Added DFlash2 support (#27816).
  • Fused DFlash encoder into KV injection (#27310).
  • Added sparse flash attention for DeepSeek-V4/GLM and Qwen4exp (#27970).
  • Added Apple RDMA as an RPC transport (#26421).
  • Added RPC event/async backend APIs (#18626).
  • Fixed Qwen4exp graph issues (#27941).
  • Added Qwen4exp recurrent rollback (#28123).
  • Summed Qwen4exp indexer heads (#28023).
  • Reduced Qwen4exp graph splits (#27880).
  • Fixed DeepSeek-V4 input vision (#28154).
  • Fixed Gemma-4 vision handling (#28335).
  • Fixed Gemma-4 assistant (#28183).
  • Added synthetic spec acceptance options (#27711).
  • Made build info stream configurable (#28322).
  • Fixed JSON-schema empty object grammar (#28279).

Multi-modality changes

  • Added mtmd_tokenize_from_parts() (#28250).
  • Added DeepSeek-V4-Flash-Vision-Exp (#28133).
  • Added DeepSeek-V4 input vision handling (#28154).
  • Fixed Gemma-4 vision handling (#28335).
  • Fixed Idefics3 preprocessing (#28273).
  • Fixed Qwen3-TTS (#28231).
  • Added video parameters/--video-* (#24318).
  • Made image preprocessors const-correct (#28310).
  • Made audio preprocessors const-correct (#28307).

Server changes

  • Added per-slot context limit (#24124).
  • Accept data: URLs for media (#27735).
  • Enabled preserve_reasoning by default (#28174).
  • Reject prefilled assistant tool calls (#27626).
  • Added synthetic spec acceptance options (#27711).
  • Added pytest-xdist server tests (#28298).

UI changes

  • Improved Chat Form Actions UI/UX (#27746).
  • Replaced MCP overrides with tool policy (#27745).
  • Moved Settings/MCP routes to dialogs (#27744).
  • Updated dialog styling (#27743).
  • Copy grouped agentic response text (#27832).
  • Updated ESLint config (#27700).
  • Updated package scripts (#28270).

ggml changes

  • Bumped ggml from v0.22.0 to v0.23.0 (release).

Version 0.23.0 adds new operators and backend scheduling APIs for sparse attention, asynchronous execution, and allocation-dependency tracking. The most notable additions are sparse flash attention (ggml_flash_attn_ext_set_n_kv_max), RPC event/async APIs, and Apple RDMA transport support. It also includes broad correctness and performance work across CPU, GPU, and accelerator backends.

Assets

Nightly build: b10809

More info

Changelog since v0.3.0

5266f24da llama.cpp : bump version to 0.4.0 (#28386) 64a155d24 sync : ggml (#28379) 163a40796 model, mtmd: fix gemma4 vision handling (#28335) 8f83678fd metal : add remaining fa-vec tunings for M3 Max (#28373) 86b351fd6 ggml : replace compile definitions with version.h.in (#28364) d509cb1e8 Don't use npx inside a package.json script (#28270) 4cbe8b070 ggml : don't crash when backend search path can't be read (#28271) 24f5bf8a4 ggml : remove GGML_CUDA_PEER_MAX_BATCH_SIZE (#28177) a529af96e docs : update maintainer PRs link and regenerate AUTHORS (#28365) 38521ec33 vendor: update BoringSSL to 0.20260903.0 (#28354) 0ef4d560e ci : disable failing openvino tests (#28347) c390d0abb common : make build info output stream configurable (#28322) 832fd6f17 ggml-cpu(s390x) : fix q5_1 uninitialized v_acc (#28332) 9a4843cf2 src : add n_expert_used_max function (#28323) 6703d7894 sycl: fuse rms_norm+mul+add and add+add residual chains (#27610) f9f09f02c SYCL: Refactor GGML_SYCL_ENABLE_MKL_FA to global var (#26863) d230ddd76 llama: fix whole source code rebuilt on each new commit (#28278) c5a5535e6 common/json-schema : fix GBNF grammar generation for empty object schemas (#28279) 95ef7fc16 opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations (speculative decoding/MTP) (#26477) 8c1a25166 tune MMVQ to MMQ crossover for SM87 (#28285) d30500b83 snapdragon: ci updates to use new run script (#28293) e107984bc ops: add Hexagon to ops.md and update main README.md (#28263) 42f0225fe server : use pytest-xdist for server tests (#28298) de8656bd9 mtmd: propagate const to preproc class (#28310) 7bb0fc18f metal : add sparse FA (#28098) 0df017d6d metal : fix glu dispatch with ne00 = 1 (#28306) f45576aa8 mtmd : add const in various places (#28307) 0ba6499c3 CUDA: Allow concurrent streams per split for multi-GPU (#28198) c7bda030e vulkan: fix FA dequant path engagement (#28190) 0df974d77 sycl : enhance the api to support peer-to-peer copy (#27550) d646c9d15 convert : skip bias_vl tensor in DeepSeek-V4 DSpark conversion (#28294) 5ec4eab69 misc : prevent RAM peaking at model loading stage (#27483) 4aa6ffba2 sycl: reduce redundant work in Q4_K multi-column MMVQ (#27062) c61b98b87 model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444) 67a17c17c mtmd: fix idefics3 preproc (#28273) 159b74142 finetune: fix no KV cache (#27199) 9cffdcc80 server : accept data: URLs for input_video and input_audio (#27735) f027c4f1b ggml-hexagon: add F16 support for unary ops (#28228) 733905474 mtmd: add mtmd_tokenize_from_parts() (#28250) 9cc33944f metal : add fa-vec tunings for M3 (#28236) 8c0b9cd04 metal : fix memory query under low-memory conditions (#27701) 03dbcc53e ci : check for missing autoreleasepools (#27884) cff184438 Update ROCm to 10.0.0 release (#27803) 9400c8946 model: correctly support input vision for deepseek4 (#28154) d5fec32a8 ci : enable hf-jobs on server-cuda (#28258) 3d3d7c818 ggml-cuda : remove unused vars (#28235) e750b887a common, server : enable preserve_reasoning kwarg by default, log its effective state (#28174) 7798007a2 mtmd: support DeepSeek-V4-Flash-Vision-Exp (#28133) 8e93a9773 CUDA + ggml: add sparse-fa for DSV4/GLM (#27970) 0f3a71be1 mtmd: Fix Qwen3-tts-0.6b (#28231) b81c99b47 ggml: avoid KleidiAI buffer type init on dispatch (#27891) 960dffab0 hexagon: MUL_MAT and MUL_MAT_ID fusion and fixes (#28202) ba8818cbf vulkan: handle larger batch sizes (>4) efficiently for IQ3_S mat-vec (#27449) 56dd8150c vulkan : only request VK_KHR_shader_bfloat16 extension if supported (#28155) 2637dfe37 ggml-cpu : conditionally add SpacemiT IME kernel sources (#27961) 43d87ff2d opencl: fix out‐of‐bound reads in the Adreno image kernels (#27632) 69320fef1 hexagon: add missing FARF logs for cpy/get_rows/set_rows/gdn ops (#28217) b96806d96 metal : add metallib build support for xcframework (#28163) 3466812d1 cuda: fuse MoE weighted expert reduction (#25952) b356fa262 kv-cells: look up the n-gram history in the sequence position index (#28040) dfc29b64e context : autoscale n_ctx_train when yarn scaling specified (#28030) f28493c78 models : appropriately flag noscan ssm_a tensors (#28121) 73159c303 model : fix gemma4-assistant (#28183) d11b3cc7e model : load relevant arrays with n_layer_all (#28173) c845263f8 Revert "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (#28184) 1f3d31873 sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280 (#28016) 8887a48f0 metal : add fa-vec tuning for M2 Pro (#28122) be789c344 metal : add fa-vec tunings for A18 Pro (MacBook Neo) (#28152) 9d817213a model : load hparams.n_layer_nextn before n_layer() calls (#28159) fe2120bc9 metal : fix more leaks due to missing autoreleasepools (#27883) d08c7872d metal : add fa-vec tuning for M2 Max (#28015) 5eec3ad01 sycl : support limit max alloc memory within 2GB for host-pinned memory (#27559) 36b101543 qwen4exp: fix seq_cp, block position keying, mtmd input, cuda abort, add tests (#27941) d086dbb34 tests : fix log verbosity for test-llama-archs (#28147) 1b89a43e3 quantize: row-slab stream to avoid thread starvation (#27830) d5d993a09 metal: enable Metal 4.0 tensor API on M5+/A19+ (#27461) 234a6ebaa ci: Bump ggml-org/ccache-action to v1.2.24 (#28083) 518b76236 kleidiai : Update KleidiAI Documentation (#26078) 0eadefebd qwen4exp: support recurrent state rollback (#28123) 09412af38 qwen4exp: sum the indexer heads by slices (#28023) 458681e1d metal : add fa-vec tunings for M1 Ultra (#28088) e4b9af007 CUDA: XOR swizzle flash attn K,V smem fp16 tiles (#25635) ab0b3bd3c metal : add concat support for quantized types (#28116) 85c55223c AVX2: Speed up large batch size prompt processing of IQ models (#27402) 2a74817f9 metal : add top-k radix implementation (#28073) 2d8d612e4 kv-cache : optimize restoring non-contiguous cells (#27991) 010be9683 opencl: tune the quant paths for Intel Xe-LP GPUs to improve its TG and PP performance (#26438) 774ee0e20 ui: copy the displayed text of grouped agentic responses (#27832) 8e53fcefd webgpu : avoid crash when offset is not multiple of 4 in WebGPU ggml_backend_tensor_get() implementation (#28045) f8dbcd618 ROCm: add radix TOP_K for long rows (#27466) 5d4a3be26 metal : add fa-vec tunings for M1 (#28078) 41ef91f7c CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token (#27621) a32af33de sycl : Enhance to get the free memory of Intel GPU (#27968) 580e88d8b ci : add check for unzip (#28082) 662a0b012 spec : fuse the DFlash encoder into the KV cache injection (#27310) 2cdae802e vulkan: tune mat-vec rows for batched inference on Strix Halo (#27909) 557614e02 ggml : add MUL_MAT to the list of ops that may need additional memory (for WebGPU) (#28071) daef7b687 vulkan: top_k radix select for k >= 1024 for Qwen 3.8 Flash Next (#28032) 9723942ad hexagon: fix CPY fence bug (#28033) bd55e6aae metal : add remaining Q4_1/Q5_0/Q5_1 fa-vec tunings for M2 (#28017) a7cc83bba rpc: avoid serializing buffers from other servers (#26500) 6d1479c14 ggml : fix ggml_backend_buft_get_alloc_size() guard (#28038) 62acc89c2 kv-cells: stop the sequence scan once all sequences are seen (#28011) 0190529ec ggml: add SWIGLU_CLAMP (#27930) 257813839 llama: improve TENSOR_READ_LAZY handling (#27837) f1793c1c4 CUDA: use the fast mm_ids_helper path for any n_expert_used (#27978) 0b5be7e4a hip: tune rdna 3 mmq config (#26284) e42214804 hip : optimize Q2_0 dot-product path for gfx1201 (#26753) cc231cb0d dflash: pass missing NVFP4 scales to attention operations (#28000) bebc9350e common: rename --tensor-read-lazy to --lazy-mode, add -lzm shorthand (#27969) 73f56d105 ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC (#27960) 742347b2e rpc: fix apple rdma error spew on teardown (#27908) 093adb242 metal: add fa-vec tunings for M3 Ultra (#27999) b8b743c3c metal : Add fa-vec tuning for M3 Pro (#27963) dc7aecf70 vendor : update cpp-httplib to 0.54.0 (#27919) 2bf041515 rpc : fix pre-rdma macOS versions (#27815) 9e54e687c hexagon: support for device discovery and create sessions on demand (#27785) 370cb12e8 sycl: split long rows in TOP_K instead of one work-group per row (#27847) d882575cc metal : fix null-pipeline crash for F16 src1 mul_mat/mul_mat_id (#25648) bdf395515 memory : copy Hadamard matrix to k_rot tensor only if it has buffer assigned to prevent crashes during context shift of unquantized K cache (#27967) 57291f264 ggml: allow passing alloc dependencies in graph_optimize (#27301) c589f0ed1 metal : add fa-vec tunings for M2 (#27940) c841aeeb8 opencl: use a better matmul path on two Adreno GPU generations (#27640) 3173a5647 metal : assert shared memory padding (#27951) 17252c769 metal : add remaining fa-vec tunings for M4 Pro (#27915) cc83d7b48 sycl: make --fit respect --fit-target better (#27629) c9ca51c1f vulkan: combine duplicated fastdiv functions, rename the one optimizing small divs (#27526) 5ea1b124e metal : add fa-vec tunings for M1 Max (#27932) 77f132cb1 vulkan: Change mul_mat_id to pad K rather than N (#27925) d7bd3bfca snapdragon: python SDK setup (Windows) (#27903) 50f068fff bench: add --tensor-read-lazy (#27881) 6fe749801 model: qwen4exp: reduce number of graph splits (#27880) b387ddfd8 vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize (#27812) a43c3986b ggml : fix conv_transpose_2d for multiple batches (#26132) 90c26fcd4 Vulkan: add hoisting support for row IDs and expert count in shaders (#26686) 866322481 context : disable non-fused GDN and LID ops (#27877) f5e85d43a metal : add fa-vec tunings for M4 (#27875) 511f9c137 OpenVINO: Update OV to 2026.3.1, whisper.cpp support, Qwen3.5 on NPU, and new ops (#27843) b19cbe925 convert: prevent ndarray conversion in LazyChunkedTensor (#27869) d077b4c21 sycl: use TILE for quantized KV decode on BMG (#26689) be876204a sycl: bind the f16 KV cache in place for the oneDNN SDPA path (#27468) 8963a9bdc metal : add fa-vec tunings for M3 Max, M5 and M5 Pro (#27863) 6d6b697cd metal : add fa-vec tunings for M4 Pro (#27824) 4e97ac86e tests : run test-save-load-state across all architectures (#27755) ca3d5a3e1 model: add DSpark support for Nemotron3.5 (#27804) e70802a01 ggml-hexagon: add HTP unary ops for ABS and LOG (#27786) 83d855c5a hex-unary: fix RMS_NORM_MUL weight-offset bugs for grouped/broadcast norms (#27798) 18443257a server: add ctx-per-slot (--kv-unified-per-slot) (#24124) 32176338a ci : build only the ggml-hip backend for windows-rocm release (#27753) 6c84c7d5d model: add Qwen3.8-Flash-Next (qwen4exp) (#27742) 6fdd0ac89 ci : bundle HIP runtime DLLs with Windows ROCm release (#26973) b10f9ca58 spec : add DFlash2 support (local convolution + candidate selector) (#27342) (#27816) 58546250c opencl: add bin kernels kernel_gemm_moe_q4_0_q8_1_dp4a_bin, kernel_gemm_moe_mxfp4_q8_1_dp4a_bin (#27768) 732707dff quantize: cap working memory size to avoid loading big tensors onto RAM (#27795) cb300598d Feature: Added LIGHTNING_INDEXER support for Deepseek V4 ops on Vulkan Backend (#27453) 1a946ec74 pr2wt : use ssh/https remote in worktree depending on base (#27800) fac889fb3 llama: model_loader: add TENSOR_READ_LAZY (#27794) cae63579b ui: Improve Chat Form Actions UI/UX (models selector, add panel) (#27746) bcb6084a4 convert : fix Nemotron-H LoRA GGUF conversion (#27356) fe235f434 ui: Replace per-conversation MCP overrides with per-conversation tool policy (#27745) 2bb9bddaf spec: Add benchmark-only synthetic speculative acceptance options (#27711) deae5ee13 model : simplify MiniMax-01 graph (#27790) f29551215 args: add --video-* CLI arguments (#24318) 915dc6d38 metal : fix memory leaks due to missing autoreleasepools (#27758) c5fc7e348 llama : add --n-cpu-ffn option (#26622) d7a207411 models : support nanbeige4.2-3B (#27730) 192067b72 hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (#26501) 925e11799 llama: add token ID tracking to KV cell (#27762) 539f24529 ui: Move Settings and MCP Servers routes to dialog-based views (#27744) 0379a19f0 ui: Update Dialog component styling (#27743) 5e6a37cb1 vulkan: warptiles currently assume warp sizes <= 64, clamp to work around larger warps (#27726) bf9421646 Implemented vulkan cross_entropy_loss and cross_entropy_loss_back (#27216) d0132a680 rpc : implement event and async backend APIs (#18626) 4d19b2876 ci: Clean up UI builds from releases (#27706) fc35562ba cuda: unblock mmq for MoE on sm_60 (#26264) da9b5d68c ci : make cache bucket public (#27728) dac869b0a conversion : fix Nemotron 3.5 Lightning layers (#27729) 11cd98842 ggml-metal: add chunked SSD MMA for Mamba-2 prefill optimization (#26647) 5d5cb4c3a ggml-meta: propagate buffer usage and call init on the new tensors (#27586) d222767c7 kleidiai: Rework KleidiAI Build System/Integration (#26077) eab8ee41f ci : update OS used for ROCM to Ubuntu 24.04 (#27681) b114b4739 rpc: support apple RDMA as an RPC transport (#26421) 0a5ac49bc devops: use GGML_NATIVE=OFF for OpenVINO (#27338) 1729ed537 server : reject prefilled assistant messages with tool calls (#27626) 0cc5b1495 chat : scope qwen3-coder workarounds (#27679) 790b5713c ci : store ccache on HF buckets (test with cuda-ubuntu for now) (#27699) f1357e499 ui: ESLint config updates (#27700) 3737e4137 metal : null-check buffer alloc to fix OOM crash (#25371)