Document Athena containers and expand Hermes operator skill
This commit is contained in:
1 parent
87a2ae5704
commit
b4f4bf37fd
13 files changed
+232
-29
No files matched your search
@@ -0,0 +1,41 @@
|
||||
# Model evaluation
|
||||
|
||||
## Before downloading
|
||||
|
||||
1. Read all of `docs/TESTED_MODELS.md`; it is the no-repeat register.
|
||||
2. Record the exact repository, revision, filename, base model, fine-tune,
|
||||
quantization, license, format and estimated disk/VRAM/RAM requirements.
|
||||
3. Confirm the model adds a genuinely new candidate rather than a renamed
|
||||
artifact already tested.
|
||||
4. Do not stop the active production model merely to download or prepare the
|
||||
candidate. Put temporary scripts under `/tmp`; durable code belongs in Git.
|
||||
|
||||
## Quality rule
|
||||
|
||||
Standard text profiles must not fall below Q4. A smaller Q3 quantization may be
|
||||
kept only when an A/B test documents that its quality loss is negligible for
|
||||
Mike's intended workload. Smaller files are not presumed faster: GPU split,
|
||||
memory bandwidth, kernels, cache formats and cross-GPU traffic must be measured.
|
||||
|
||||
## Fair A/B test
|
||||
|
||||
Hold these equal wherever the models permit it:
|
||||
|
||||
- prompt set and conversation history;
|
||||
- context size and filled-context test point;
|
||||
- KV-cache quantization, slots, batch/uBatch, MTP and sampling;
|
||||
- GPU visibility and tensor split;
|
||||
- warm-up state and output-token limit.
|
||||
|
||||
Measure prompt processing, short decode, long-context decode, peak VRAM and
|
||||
wall time. Test meaning preservation, uncertainty, negation, ordered safety
|
||||
constraints, German language consistency, tool-call schema and long-context
|
||||
recall. Do not replace a model on synthetic benchmark scores alone.
|
||||
|
||||
## Completion
|
||||
|
||||
Write the exact artifact, settings, raw result path, interpretation and decision
|
||||
to `docs/TESTED_MODELS.md` in the same commit. If promoted, update the manifest,
|
||||
Compose/env examples, profile matrix and user documentation. If rejected,
|
||||
remove candidate-only weights, images and containers after preserving the
|
||||
result. Restore and functionally test the previous profile, then publish.
|
||||
Reference in new issue
Block a user