Finance Agent Reliability Benchmark for Cohere Command A+

A pre-registered design for testing financial correctness, artifact integrity, provenance, reliability, and review burden—before any result is known.

PROTOCOL · NO RESULTS REPORTED

Protocol status: design draft / pre-registration. No benchmark runs, scores, latency measurements, cost measurements, or comparative results are reported in this article.

Why publish the protocol before the result?

A benchmark is most useful when its scoring rules are fixed before anyone sees the outcome. Publishing the protocol first makes it harder to select only favorable tasks, change a validator after a failure, or confuse a product demonstration with a controlled comparison.

This design asks a narrow question: how should a finance agent built with Cohere Command A+ be evaluated when the output is a working spreadsheet or presentation rather than a paragraph of text?

The intended output is a reproducible evaluation package, not a claim that one model or harness is already better.

Claims and non-claims

This protocol defines:

• the target task set;
• the baseline and orchestrated system configurations;
• the artifacts that each run must retain;
• deterministic and review-based checks;
• the failure taxonomy; and
• the reporting template.

This protocol does not claim:

• that the task set has been completed;
• that Command A+ has already been run on these tasks;
• that Krappa improves any score;
• that any model outperforms another product;
• that latency or cost has been measured; or
• that the test set represents every finance workflow.

All result fields remain explicitly marked not run until a reproducible release exists.

Research question

Can an agent complete bounded finance-artifact tasks while preserving financial correctness, document integrity, provenance, and reviewability?

The benchmark is designed to distinguish model capability from harness quality. It therefore compares the same versioned model under two configurations:

1. Direct baseline: one model session with the minimum documented tools and prompt required to attempt the task.
2. Orchestrated system: the same model endpoint inside a staged workflow with planning, specialized tools, deterministic validation, trace capture, and bounded retries.

The model identifier, API version, prompts, tool schemas, sampling parameters, retry policy, and validator commit must be frozen in the release manifest.

Planned task set

The target is 30 sanitized tasks: large enough to expose recurring failures while still allowing every artifact to be manually reviewed. Thirty is a protocol target, not a completed dataset count.

Each task must include:

• a sanitized input artifact;
• a precise objective;
• permitted tools;
• prohibited shortcuts;
• deterministic checks where possible;
• a human-review rubric;
• a maximum execution budget; and
• a disclosure of any synthetic data.

Task inclusion rules

A task may enter the test set only when:

1. it can be shared without customer, employer, or personal information;
2. a reviewer can determine whether the requested change was completed;
3. at least one meaningful failure mode is observable;
4. the input does not favor a proprietary internal convention; and
5. the expected artifact is not embedded in the prompt.

Tasks used while developing prompts or validators belong to a development set. They cannot later be counted as held-out test tasks.

Evaluation dimensions

Task completion

Did the resulting artifact satisfy every explicit requirement? Partial completion must be recorded separately from a full pass.

Financial correctness

Do accounting identities, roll-forwards, valuation logic, signs, units, and period conventions remain correct?

Artifact integrity

Are formulas, references, named ranges, sheet structures, slide layouts, and editable objects preserved where required?

Provenance

Can each material output be connected to a supplied source, calculation, or tool action?

Reliability

Does the workflow complete consistently across repeated runs under the frozen configuration? The number of repetitions must be declared before execution.

Review burden

How many material issues require human correction, and at what severity? A full pass should not hide costly manual cleanup.

Efficiency

Wall-clock latency, model tokens, tool calls, retries, and estimated API cost should be captured, but they must not be reported until the measurement code and pricing assumptions are released.

Deterministic validators

Deterministic checks should carry the decision whenever correctness is computable. Planned validator families include:

• balance-sheet and cash-flow identities;
• schedule-to-summary reconciliation;
• formula-presence and formula-pattern checks;
• reference and range integrity;
• required sheet, section, or slide existence;
• unit, sign, and period consistency;
• editable-object and template-preservation checks; and
• unexpected-change detection outside the permitted region.

A validator must return a machine-readable result with an identifier, version, severity, observed value, expected condition, and affected artifact location.

Model-based grading may help assess clarity or presentation quality, but it must not override a failed deterministic financial check.

Failure taxonomy

Every failed or partial run should receive one primary category and any applicable secondary categories:

The public report should include examples of important failures, not only aggregate pass rates.

Required trace

Every run should retain a sanitized trace with:

run manifest
  task id
  system configuration
  model and API version
  prompt and tool-schema hashes
  validator commit

execution trace
  plan revisions
  tool calls and structured outputs
  file mutations
  retry and stop decisions

evaluation record
  deterministic checks
  human-review rubric
  failure categories
  final disposition

Secrets, personal data, and proprietary document content must be removed before publication. A redacted trace should never be represented as complete if the omitted context affects interpretation.

Run procedure

1. Freeze the task set and publish its hashes.
2. Freeze both system configurations.
3. Run a small infrastructure check that cannot enter the scored set.
4. Execute the held-out tasks under identical budgets.
5. Store artifacts and traces before reviewing aggregate results.
6. Run versioned deterministic validators.
7. Have a reviewer assess the anonymized outputs using the frozen rubric.
8. Assign failure categories without changing the original run record.
9. Publish all exclusions, crashes, and missing measurements.
10. Release the report only with enough artifacts for another person to reproduce the calculation.

Empty reporting template

The table below is intentionally empty of results.

Any future result must include denominators and uncertainty. A percentage without the underlying task and run counts is insufficient.

Reproducibility package

A credible release should include:

• sanitized task inputs;
• task definitions and split manifest;
• prompts and tool schemas;
• system configuration and dependency versions;
• deterministic validator code;
• run-level traces;
• resulting artifacts;
• reviewer rubric and adjudication notes;
• analysis code; and
• a concise model card describing limitations.

If part of the system must remain private, the report should identify that boundary and narrow its claims accordingly.

Known limitations

Thirty tasks cannot establish universal finance capability. Spreadsheet and presentation quality also contain legitimate judgment calls. API behavior, tool implementations, and pricing change over time. Results from one model version or one harness should not be generalized to another.

The benchmark is therefore intended as a transparent engineering case study: a way to expose failure modes, improve reliability, and make future claims inspectable.

Release checklist

• Sanitize and license the task inputs
• Implement and version the validators
• Freeze the direct and orchestrated configurations
• Publish the task manifest before scored runs
• Execute the held-out suite
• Review and categorize every failure
• Release traces, artifacts, and analysis code
• Replace every “Not run” field only with reproducible measurements

Until those items are complete, this page remains a protocol—not a benchmark result.

Published by Rayane Oubelkas