kade/llama.cpp
1
0
Fork
You've already forked llama.cpp
 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions 6

ATSInfer benchmark campaign P0-P3 results and recommendations #2

New issue
Open
opened 2026-08-05 16:46:43 +02:00 by kade · 0 comments
kade commented 2026-08-05 16:46:43 +02:00
Owner
Copy link

Campaign Summary: Ornith-1.0-35B on RTX 4090

Profiles Tested

Profile Key Change ATSInfer Preload Decode (2-seat agg)
P0 (baseline) Production config FAILED (budget 15668) 57.8-75.0 t/s (4 reps)
P1 (budget 21500) Explicit budget override PASSED 75.8 t/s
P2 (q5_1/q5_0 KV) KV quantization FAILED (budget 15668) 71.2 t/s
P3 (ctx-131k) Halved context FAILED (budget 15668) 64.3 t/s

Key Findings

  1. P1 is the production winner — ATSInfer preload passes, decode is competitive (75.8 t/s), VRAM headroom thin but sufficient (1200 MiB free).
  2. The budget calc bug — --cache-ram 11500 moves ~11 GB KV to system RAM, but the preload budget estimate ignores this. Fix: credit cache-ram in the budget calculation.
  3. Variance is real — 57.8-75.0 t/s spread on P0 alone (29% range). N>=3 reps per profile is required.
  4. LCP crash (kade/llama.cpp#1) — deterministic segfault on identical-prompt retry. Blocker for production.

Related Issues

Full Journal

See ~/wiki/atsinfer/journal-2026-08-05-profile-benchmarking.md for complete campaign log.

## Campaign Summary: Ornith-1.0-35B on RTX 4090 ### Profiles Tested | Profile | Key Change | ATSInfer Preload | Decode (2-seat agg) | |---------|-----------|------------------|---------------------| | P0 (baseline) | Production config | FAILED (budget 15668) | 57.8-75.0 t/s (4 reps) | | P1 (budget 21500) | Explicit budget override | **PASSED** | 75.8 t/s | | P2 (q5_1/q5_0 KV) | KV quantization | FAILED (budget 15668) | 71.2 t/s | | P3 (ctx-131k) | Halved context | FAILED (budget 15668) | 64.3 t/s | ### Key Findings 1. **P1 is the production winner** — ATSInfer preload passes, decode is competitive (75.8 t/s), VRAM headroom thin but sufficient (1200 MiB free). 2. **The budget calc bug** — `--cache-ram 11500` moves ~11 GB KV to system RAM, but the preload budget estimate ignores this. Fix: credit cache-ram in the budget calculation. 3. **Variance is real** — 57.8-75.0 t/s spread on P0 alone (29% range). N>=3 reps per profile is required. 4. **LCP crash** (kade/llama.cpp#1) — deterministic segfault on identical-prompt retry. Blocker for production. ### Related Issues - kade/llama.cpp#1: LCP slot reuse segfault on identical-prompt retry ### Full Journal See `~/wiki/atsinfer/journal-2026-08-05-profile-benchmarking.md` for complete campaign log.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
apply-pr-26323
fix/out-ids-host-buffer-crash
pr/26640-slot-save-restore
NVTX-profiler-markers
merge/laguna-pr25165
fix-cuda-convert-kernels
backup/pre-upstream-merge-20260730
feature/prismml-bonsai-port
laguna
pr-25952
feature/weaver-dflash-tfm
feature/atsinfer-hybrid-scheduling
merge/upstream-buun-integration
pr-25165
merge/upstream-buun-integration-pre-upstream-merge
merge/upstream-buun-integration-pre-rebase-snap
feature/recurrent-state-shrink-expand
backport/cache-optimize
merge/upstream-buun-integration-backup
main
stable
fix/qwen-hybrid-checkpoint-reuse
diffusiongemma
merge/upstream-612
pr/24423-diffusiongemma
squash-clean
pr-16857
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kade/llama.cpp#2
No description provided.