Model benchmarks

TPS includes queueing and prefill. Background work may be uncounted.

Wins and losses compare Protagine with Hermes. Ties include both passing or both failing.

Shared endpoint; other traffic may affect timing. Deployment-defined budgets. Equal total work is unverified.

Dataset

paired-agent-reviewed-2 · development

Dataset SHA-256

9d513d5288be1ef6043a930c66ff1513e810c7fcefb7a58576668e5c804e7127

Source SHA-256

888e6982b9239dd76b35a2f26feb851172ed070c9f932eab1bef431953e09a74

Manifest SHA-256

5a5f6c5630edfe67aada8ca6d506550b282e897f2404a7491124c4cac142c1fa

Comparison key

94d3846a77535d69661893323da9297a4150c0669205d67b47eb9ff393b42b8e

Hermes version

0.21.3

Runtime image

sha256:ea1c805d9b8cff55695ac250fa0d4ef54cfca6754c6e8da31ea4314467f4e6d0

Hermes payload

080ba118c2500e65e6b5c4f2cfe7fa72cd3ce37e6508e5489b4964efc1159bfa

Protagine payload

f0e1e455b29fc591c3f08529f58a5d9b9c1c7ecd6df48a0b77752182bdd42b46

• Public development data, not a held-out benchmark or a comprehensive production qualification.
• Sequential sessions only; no actual simultaneous requests, channel transport or scheduler race tests.
• Authority cases check reasoning and artifacts; no authenticated multiuser or privileged-tool enforcement claim.
• Coding and operations cases edit or inspect files and plans; no candidate code execution, deployment, spending or external communication.
• Memory facts enter through ordinary turns. Forgetting checks later output, not physical erasure from every store.
• Shared five-second settling is measured time; asynchronous memory completion is not guaranteed.
• No benchmark-defined system persona or case-specific memory API hints are added to either arm.
• Endpoint usage is operator-declared; competing traffic may slow the live agent and distort these measurements.
• Public development results do not establish model tiers or a causal improvement estimate.
• Request throughput includes the whole HTTP request, not isolated decode; completion tokens may include reasoning.
• Auxiliary model work is partially observed. Missing observations are not zero.
• Report-only attribution correction, applied symmetrically to both arms. A returned native container with empty output counts as noncompletion only with independent serialized candidate requests and returned-model evidence, no fallback, confirmed cleanup, and consistent recorded native incomplete turns. Saved outcomes, primary attribution and checks are unchanged; artifact grading is not rerun.

Model

GLM-5.3 Flash Abliterated

Checkpoint revision

c8f58e6aa9117c73607d692978b22f091d80450c

Weight identity

Not independently verified

Quantization

EXL3 4 bpw weights; NVFP4 MLA KV cache

Hardware

1 × Two NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation GPUs, 96 GB per GPU

Configured context

327,680

Configured request slots

4

Serving runtime

vLLM 0.26.1rc0+infernal.invocation.cu133.r19.vllm174c789.b12x12c4263

Configured limits, not measured capacity.

• Checkpoint manifest is stale for 43 modified shards; full tensor identity is not established by the repository revision alone.
• Production incumbent was concurrently available to ordinary agent traffic, so screening latency is not an isolated throughput benchmark.
• Public runtime image is pinned and reachable; independent launch reproduction has not been performed.
• The linked upstream launch profile differs from this benchmark. Use the recorded arguments below to reproduce the tested configuration.

TPS includes queueing and prefill. Background work may be uncounted.

Wins and losses compare Protagine with Hermes. Ties include both passing or both failing.

Endpoint traffic unknown. Deployment-defined budgets. Equal total work is unverified.

Dataset

paired-agent-reviewed-2 · development

Dataset SHA-256

9d513d5288be1ef6043a930c66ff1513e810c7fcefb7a58576668e5c804e7127

Source SHA-256

888e6982b9239dd76b35a2f26feb851172ed070c9f932eab1bef431953e09a74

Manifest SHA-256

6fd8e977fb3ef62c2ab4f734df3200064e11b7a7dac2d7a5a74d18f5dde418a0

Comparison key

f4abf2032d990c047d40b8b7a3d9fc0005ba5bb2e0cd9bc5b116fd223206487a

Hermes version

0.21.3

Runtime image

sha256:ea1c805d9b8cff55695ac250fa0d4ef54cfca6754c6e8da31ea4314467f4e6d0

Hermes payload

080ba118c2500e65e6b5c4f2cfe7fa72cd3ce37e6508e5489b4964efc1159bfa

Protagine payload

f0e1e455b29fc591c3f08529f58a5d9b9c1c7ecd6df48a0b77752182bdd42b46

• Public development data, not a held-out benchmark or a comprehensive production qualification.
• Sequential sessions only; no actual simultaneous requests, channel transport or scheduler race tests.
• Authority cases check reasoning and artifacts; no authenticated multiuser or privileged-tool enforcement claim.
• Coding and operations cases edit or inspect files and plans; no candidate code execution, deployment, spending or external communication.
• Memory facts enter through ordinary turns. Forgetting checks later output, not physical erasure from every store.
• Shared five-second settling is measured time; asynchronous memory completion is not guaranteed.
• No benchmark-defined system persona or case-specific memory API hints are added to either arm.
• Endpoint usage is operator-declared; competing traffic may slow the live agent and distort these measurements.
• Public development results do not establish model tiers or a causal improvement estimate.
• Request throughput includes the whole HTTP request, not isolated decode; completion tokens may include reasoning.
• Auxiliary model work is partially observed. Missing observations are not zero.
• Report-only attribution correction, applied symmetrically to both arms. A returned native container with empty output counts as noncompletion only with independent serialized candidate requests and returned-model evidence, no fallback, confirmed cleanup, and consistent recorded native incomplete turns. Saved outcomes, primary attribution and checks are unchanged; artifact grading is not rerun.

Model

GLM-5.3 Uncensored

Checkpoint revision

409ddf484d16cb7b13d390e8b30942890164d5af

Weight identity

Verified; see evidence

Quantization

NVFP4 weights; FP8 MLA KV cache

Hardware

16 × NVIDIA DGX Spark GB10, 128 GB per node

Configured context

1,048,576

Configured request slots

16

Serving runtime

vLLM with upstream structured-output repair v19 + upstream PR44993

Configured limits, not measured capacity.

• A clean Dockerfile and exact upstream method repair are supplied. The script reproduces the verified source-file hash; a clean image build and independent inference reproduction have not been performed.
• The campaign image was assembled as a deterministic delta. A conventional Docker build has different metadata and image ID; retain original campaign identities.
• Argument snapshot includes the checkpoint architecture index pattern; retain the exact pattern.

12 distinct workflows. Repetitions use the same tasks with fresh state; they are not independent task designs.

Controls supply the current facts directly. They are comparable examples, not an exact causal estimate of memory benefit.

TPS includes queueing and prefill. Background work may be uncounted.

Wins and losses compare Protagine with Hermes. Ties include both passing or both failing.

Shared endpoint; other traffic may affect timing. Deployment-defined budgets. Equal total work is unverified.

Dataset

paired-agent-workflows-1 · frozen_public_evaluation

Dataset SHA-256

1f6bff097336bb13ead67c9caa93cf0d2cad8e3e5098160b7680c05984d84ab6

Source SHA-256

93f39753e75ea945da5199fb7a006497d76db28fef7d678682877c956f997c1d

Manifest SHA-256

75eae932f025b666768c2eea6319d821f7077ab11665173a941a4d4ac82ce786

Comparison key

62635aa43d8bdd432d221583b8ecde9881a08d509049c661e9596956c3d68d2c

Hermes version

0.21.3

Runtime image

sha256:ea1c805d9b8cff55695ac250fa0d4ef54cfca6754c6e8da31ea4314467f4e6d0

Hermes payload

080ba118c2500e65e6b5c4f2cfe7fa72cd3ce37e6508e5489b4964efc1159bfa

Protagine payload

f0e1e455b29fc591c3f08529f58a5d9b9c1c7ecd6df48a0b77752182bdd42b46

• Public frozen synthetic workflow, not a secret holdout or unseen-task claim.
• Sessions are sequential. Process restart is real; no host failure, simultaneous-session race or live channel transport is exercised.
• Only declared local artifacts and checkpoints are graded; no external action or scheduled outreach is executed.
• Memory-relevant tasks permit legitimate workspace notes and native session retrieval in both arms; no mechanism is required or rewarded.
• Exactly twelve small workflows do not establish production qualification, real-time autonomy, multimodal competence, statistical superiority or long-term learning.
• Synthetic read fault targets native read_file; alternative legitimate access can bypass it and must be reported rather than credited as fault recovery.
• Foreground output and private stores are outside these artifact checks; no all-channel confidentiality or physical-forgetting claim.
• Source checks detect changed bytes at declared checkpoints and final capture, not a modification restored between snapshots.
• Checkpoints and read faults are part of this versioned protocol. Do not silently edit frozen inputs or graders after inspecting model results.
• Endpoint usage is operator-declared; competing traffic may slow the live agent and distort these measurements.
• Public development results do not establish model tiers or a causal improvement estimate.
• Request throughput includes the whole HTTP request, not isolated decode; completion tokens may include reasoning.
• Auxiliary model work is partially observed. Missing observations are not zero.
• Report-only attribution correction, applied symmetrically to both arms. A returned native container with empty output counts as noncompletion only with independent serialized candidate requests and returned-model evidence, no fallback, confirmed cleanup, and consistent recorded native incomplete turns. Saved outcomes, primary attribution and checks are unchanged; artifact grading is not rerun.

Model

GLM-5.3 Flash Abliterated

Checkpoint revision

c8f58e6aa9117c73607d692978b22f091d80450c

Weight identity

Not independently verified

Quantization

EXL3 4 bpw weights; NVFP4 MLA KV cache

Hardware

1 × Two NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation GPUs, 96 GB per GPU

Configured context

327,680

Configured request slots

4

Serving runtime

vLLM 0.26.1rc0+infernal.invocation.cu133.r19.vllm174c789.b12x12c4263

Configured limits, not measured capacity.

• Checkpoint manifest is stale for 43 modified shards; full tensor identity is not established by the repository revision alone.
• Production incumbent was concurrently available to ordinary agent traffic, so screening latency is not an isolated throughput benchmark.
• Public runtime image is pinned and reachable; independent launch reproduction has not been performed.
• The linked upstream launch profile differs from this benchmark. Use the recorded arguments below to reproduce the tested configuration.

12 distinct workflows. Repetitions use the same tasks with fresh state; they are not independent task designs.

Controls supply the current facts directly. They are comparable examples, not an exact causal estimate of memory benefit.

TPS includes queueing and prefill. Background work may be uncounted.

Wins and losses compare Protagine with Hermes. Ties include both passing or both failing.

Endpoint traffic unknown. Deployment-defined budgets. Equal total work is unverified.

Dataset

paired-agent-workflows-1 · frozen_public_evaluation

Dataset SHA-256

1f6bff097336bb13ead67c9caa93cf0d2cad8e3e5098160b7680c05984d84ab6

Source SHA-256

93f39753e75ea945da5199fb7a006497d76db28fef7d678682877c956f997c1d

Manifest SHA-256

c4c0b566af844fdab48c6d1ad762316f742c7b1cbea2488353b1153da27586c1

Comparison key

fa0f6843e3435beb15923c01c10f48e12b8a714e31fa1b8e31dc5e6fcde59d1e

Hermes version

0.21.3

Runtime image

sha256:ea1c805d9b8cff55695ac250fa0d4ef54cfca6754c6e8da31ea4314467f4e6d0

Hermes payload

080ba118c2500e65e6b5c4f2cfe7fa72cd3ce37e6508e5489b4964efc1159bfa

Protagine payload

f0e1e455b29fc591c3f08529f58a5d9b9c1c7ecd6df48a0b77752182bdd42b46

• Public frozen synthetic workflow, not a secret holdout or unseen-task claim.
• Sessions are sequential. Process restart is real; no host failure, simultaneous-session race or live channel transport is exercised.
• Only declared local artifacts and checkpoints are graded; no external action or scheduled outreach is executed.
• Memory-relevant tasks permit legitimate workspace notes and native session retrieval in both arms; no mechanism is required or rewarded.
• Exactly twelve small workflows do not establish production qualification, real-time autonomy, multimodal competence, statistical superiority or long-term learning.
• Synthetic read fault targets native read_file; alternative legitimate access can bypass it and must be reported rather than credited as fault recovery.
• Foreground output and private stores are outside these artifact checks; no all-channel confidentiality or physical-forgetting claim.
• Source checks detect changed bytes at declared checkpoints and final capture, not a modification restored between snapshots.
• Checkpoints and read faults are part of this versioned protocol. Do not silently edit frozen inputs or graders after inspecting model results.
• Endpoint usage is operator-declared; competing traffic may slow the live agent and distort these measurements.
• Public development results do not establish model tiers or a causal improvement estimate.
• Request throughput includes the whole HTTP request, not isolated decode; completion tokens may include reasoning.
• Auxiliary model work is partially observed. Missing observations are not zero.
• Report-only attribution correction, applied symmetrically to both arms. A returned native container with empty output counts as noncompletion only with independent serialized candidate requests and returned-model evidence, no fallback, confirmed cleanup, and consistent recorded native incomplete turns. Saved outcomes, primary attribution and checks are unchanged; artifact grading is not rerun.

Model

GLM-5.3 Uncensored

Checkpoint revision

409ddf484d16cb7b13d390e8b30942890164d5af

Weight identity

Verified; see evidence

Quantization

NVFP4 weights; FP8 MLA KV cache

Hardware

16 × NVIDIA DGX Spark GB10, 128 GB per node

Configured context

1,048,576

Configured request slots

16

Serving runtime

vLLM with upstream structured-output repair v19 + upstream PR44993

Configured limits, not measured capacity.

• A clean Dockerfile and exact upstream method repair are supplied. The script reproduces the verified source-file hash; a clean image build and independent inference reproduction have not been performed.
• The campaign image was assembled as a deterministic delta. A conventional Docker build has different metadata and image ID; retain original campaign identities.
• Argument snapshot includes the checkpoint architecture index pattern; retain the exact pattern.