Contact Open models tested for reliable agent work. Compare the shared screening results, inspect hardware and serving recipes, and see where each model still fails.
Loading model results…
Loading benchmark filters…
Speed and answer correctness are shown together. These are measured workloads, not maximum-capacity claims.
Context rows are approximate input-size targets. Columns are client concurrency limits, separate from server slots and observed overlap. Open a run for actual token counts, timings and limitations.
Warm-engine unique-prefix pilot; about 4K input, client C1
Four NVIDIA DGX Spark systems, GB10, 128 GB unified memory per system · Warm engine, unique prefixes
| Input target | C1Client limit | C4Client limit | C8Client limit |
|---|---|---|---|
| 4K | 6/8 correct8/8 transport completed6/8 verified primary21.18 tok/sIncludes reasoning + final tokens
| Not measured | Not measured |
| 32K | Not measured | Not measured | Not measured |
| 128K | Not measured | Not measured | Not measured |
First generated output may be hidden reasoning. First final-content arrival is not proof of a useful or correct answer. Throughput spans the full measured batch. Speculative streaming intervals describe chunks, not individual tokens.
The same model can behave differently when memory, shared tasks and recollection are available.
For the finalists, compare base Hermes with Hermes plus Protagine on 48 state-dependent scenarios. A further 12-scenario control withholds relevant memories or substitutes unrelated ones.
Provide the same experience and feedback, close the session, then test untouched task variants. Repeat a subset with a different processor. Measure useful transfer and regressions separately.
These are planned system-contribution experiments. Endpoint or extraction tests alone do not demonstrate persistent-agent behavior. Learning uses Protagine's ordinary pathways, with invented people and isolated state.
About Protagine180 planned scenarios. A scenario may span several turns, tools or sessions. This is an agent-work benchmark, not an AGI score.
Model endpoint, native Hermes, and Hermes plus Protagine. Consumer tests remain labeled separately. A valid JSON response is not evidence that the stored fact is correct.
A 24-case screen comes from 120 development scenarios. Freeze settings before opening 60 held-out scenarios. Three repetitions measure consistency, not three times as many independent tasks.
Check tool effects, memory changes and code with executable oracles first. Free-form answers use blind semantic rubrics. Publish sample counts, disagreements and task-level uncertainty when measured.
Report queue delay, first useful content, completed tasks and latency under mixed load. Separate cold prefill from warm caches, single-request speed from aggregate throughput, and configured context from demonstrated usable context.
Pin fixtures, supporting models, runtime and budgets. Record exact checkpoints and quantizations. Modified weights are a separate deployment. Keep failed configurations and distinguish model errors from invalid tests.
Publish named result fields and synthetic examples. Owner conversations, memories, credentials, internal network details and private reasoning are excluded. Downloadable artifacts identify what was actually tested.
Evidence, uncertainty, false premises and honest completion claims.
Correct actions, arguments, structured output, streaming and recovery.
Useful facts, attribution and temporal scope, without admitting fiction or clutter.
Automatic recall, updated facts, forgetting and a different model in a new session.
Evidence use, contradictions, synthesis and compression, beyond finding a hidden phrase.
Dependencies, changed constraints, delegation and verified final task state.
Concurrent channels, shared commitments, steering and durable task ownership.
Synthetic contacts, legitimate access, impersonation and private-data boundaries.
Complete concise answers, justified disagreement and stable state across processors.
Feedback applied to unseen variants, delayed recall, transfer and regressions.
Small and multi-file repairs, instructions, executable tests and accurate reports.
Grounded interpretation, ambiguity, interruption and conversation continuity.
Timeouts, bounded retries, fallback attribution and isolated service recovery.
Foreground conversation and steering while substantial background work continues.
Recipes, case-set hashes, evaluator versions and limitations accompany each published run. Snapshots update after test batches; this page does not expose a live inference endpoint. Historical measurements elsewhere on the site are not scores in this suite.