36 lines
2.2 KiB
Markdown
36 lines
2.2 KiB
Markdown
# Medium microbatch quick A/B, 2026-09-20
|
|
|
|
User-authorized first-priority test: microbatch512 versus256, all other Medium
|
|
model settings unchanged. Three sequential requests per arm: frozen German and
|
|
code decode prompts (768-token caps), then frozen24576-target prefill/recall
|
|
(24674 actual prompt tokens,512 output-token cap). No reasoning in these speed
|
|
probes; same sampling/seeds as archived requests. Small matched512 control is
|
|
necessary because the frozen baseline has different slot/vision/context settings;
|
|
this is not a repeat of the broad Qwen benchmark or an overwrite of that baseline.
|
|
|
|
Supervisor snapshots actual production image/Args into production.json. Runner
|
|
copies them, changes only microbatch and isolated bind address/port, and sets
|
|
log verbosity3. Vision remains loaded on3060; context160000 shared by two slots,
|
|
Pure model,85:15 split,MTP3, q4_0 target KV, f16 MTP KV, batch2048, six CPU threads,
|
|
cache-RAM32768, TTS resident. Requests disable prompt cache for cold measurement.
|
|
No concurrency/vision task/full-context validation in this small speed probe.
|
|
|
|
Bounded600-second child; production requests drain before stopping the existing
|
|
router/controller/Medium. Isolated read-only model mount,26GiB memory cap with no
|
|
additional swap allowance,512MiB loaded VRAM reserve,85C/3GiB host RAM guard,
|
|
no core dumps. Supervisor restores the same production containers and waits for
|
|
healthy status. Kernel/network/driver untouched. Each result directory is unique
|
|
and cannot be overwritten. New scripts are not run automatically from checkout.
|
|
|
|
Check startup logs for successful allocation versus explicit fallback without
|
|
pipeline parallelism. Absence of a fallback line alone is not proof of actual
|
|
pipeline utilization; source/verbose logs may be needed to establish activation.
|
|
Judge walltime, prompt/decode rates, VRAM and the three-needle recall together.
|
|
|
|
## Outcome
|
|
|
|
512 control completed;256 loaded without the earlier fallback message but had
|
|
only461MiB free on5080, below the512MiB guard. No256 inference was executed.
|
|
Args differ only in --ubatch-size. Production512 restored and checked. See
|
|
../../docs/MEDIUM_MICROBATCH_20260920.md. Do not claim a256 speedup or quality result.
|