Document Athena containers and expand Hermes operator skill

This commit is contained in:
Mikei386 committed 2026-09-09 16:39:34 +02:00
1 parent 87a2ae5704
commit b4f4bf37fd
13 files changed
+232 -29

No files matched your search

@@ -0,0 +1,41 @@
# Model evaluation
## Before downloading
1. Read all of `docs/TESTED_MODELS.md`; it is the no-repeat register.
2. Record the exact repository, revision, filename, base model, fine-tune,
quantization, license, format and estimated disk/VRAM/RAM requirements.
3. Confirm the model adds a genuinely new candidate rather than a renamed
artifact already tested.
4. Do not stop the active production model merely to download or prepare the
candidate. Put temporary scripts under `/tmp`; durable code belongs in Git.
## Quality rule
Standard text profiles must not fall below Q4. A smaller Q3 quantization may be
kept only when an A/B test documents that its quality loss is negligible for
Mike's intended workload. Smaller files are not presumed faster: GPU split,
memory bandwidth, kernels, cache formats and cross-GPU traffic must be measured.
## Fair A/B test
Hold these equal wherever the models permit it:
- prompt set and conversation history;
- context size and filled-context test point;
- KV-cache quantization, slots, batch/uBatch, MTP and sampling;
- GPU visibility and tensor split;
- warm-up state and output-token limit.
Measure prompt processing, short decode, long-context decode, peak VRAM and
wall time. Test meaning preservation, uncertainty, negation, ordered safety
constraints, German language consistency, tool-call schema and long-context
recall. Do not replace a model on synthetic benchmark scores alone.
## Completion
Write the exact artifact, settings, raw result path, interpretation and decision
to `docs/TESTED_MODELS.md` in the same commit. If promoted, update the manifest,
Compose/env examples, profile matrix and user documentation. If rejected,
remove candidate-only weights, images and containers after preserving the
result. Restore and functionally test the previous profile, then publish.