EvalArc/ evidence lab

OPEN SOURCE   /   RESEARCH PREVIEW 0.6

Look past
the score.

A tool returned an error. The write already happened.
A retry added the same note twice.

EvalArc tests whether an agent's grader catches the difference between plausible progress and a completed task.

Inspect the acceptance gate ↘ Every repeated attempt ↓ Compare revision regressions ↓
RECORDED CONTROL / 01NOT RESOLVED
93.75%

Partial score. Duplicated write.

add_notecommittedresponsetransient errorretry with new keyduplicate note

Scripted negative control · public seed 17
No model call or live customer data.

New in v0.6: generate Python or JavaScript candidates and audit independent controls against the same task contracts. Try both runtimes ↗. The recorded showcases below retain their original versions and fingerprints.

2task packs
15declared faults caught
JSONdownloadable evidence
CPUreproduce without a model API

NEW IN 0.5 / DECLARE YOUR ACCEPTANCE RULES

Same score. Different gate.

One frozen policy.
Two explicit acceptance rules.

Both support jobs score 93.75% and resolve 0/2 attempts. A deliberately permissive gate accepts the result; requiring every notes check to pass rejects it. The task outcome stays the same.

Loading the recorded suite…

Recorded Docker suite: three jobs, five attempts, 31 case executions. JUnit distinguishes a rejected gate from an environment error; a hosted CI importer was not exercised. This is scripted development evidence.

Prefer a filterable table or Python? Open the Hugging Face Casebook ↗ and compare gate_accepted with fully_resolved. The original records accompany every row.

NEW IN 0.4 / REPEAT ONE FROZEN CANDIDATE

One run is not the whole record.

Same cases. Fresh state.
Every attempt stays visible.

Three Docker attempts per scripted control. The reference passes every time; the duplicate-write policy keeps its 93.75% score and fails acceptance every time. Switch controls and open any attempt's full evidence.

Loading the recorded attempts…

These v0.4 recordings show fixed public cases, with no observed check variation in either control. They are not stochastic model trials or an estimate of reliability on unseen tasks. The v0.3 comparison below retains its original records and grading fingerprint.

NEW IN 0.3 / COMPARE TWO REVISIONS

Better score. New failure.

Two closure checks improve.
A previously passing note check fails.

These recorded policies use the same task, seeds, grader and runtime. Select a changed case to compare the observed outcomes.

Loading the recorded comparison…

FOLLOW THE EVIDENCE

One score never tells the whole story.

Switch the control. Inspect the checks.
Follow each recorded state change.

Loading recorded audit evidence…

FROM THE BROWSER TO YOUR TERMINAL

Make the grader
earn your trust.

Run the known-good reference and the declared faulty controls. Keep the outcomes, seeds, runtime limits and fingerprints together.

Installation & execution guide ↗
TRUSTED BUNDLED CONTROLS / LINUX / PYTHON 3.11+
git clone https://github.com/noteflowai/evalarc.git
cd evalarc
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .

evalarc audit --task support-routing \
  --backend local --trust-local \
  --output runs/support

Local mode runs with your user privileges. Use the documented Docker backend for candidate isolation.

What this evidence establishes

The bundled references pass and the 15 declared faulty implementations fail their intended checks. These are public development tasks and scripted policies. They do not establish frontier-model performance, coverage of arbitrary reward hacks, a human time horizon, or RL training gains. The browser replays saved reports; it does not execute submissions.