Prime Intellect Just Split the AI Agent Eval Into Three Swappable Parts. The Harness Is Now Its Own Layer.

TL;DR

Prime Intellect just shipped Verifiers v1, landing in version 0.2.0 of its open-source library. It is a ground-up rewrite that stops treating an agent evaluation as one tangled blob of code and splits every environment into three swappable layers: a taskset (the work), a harness (the agent that does the work), and a runtime (where it runs). Any task can now run under any agent, inside any sandbox, and the same environment feeds straight into reinforcement-learning training through prime-rl 0.7.0.

Why an eval used to be one blob

If you have ever tried to benchmark a coding agent, you know the pain. The benchmark data, the agent loop that solves each task, and the sandbox it all runs in were fused into a single file. Want to swap the agent? Rewrite the eval. Want to run the same tasks on a different provider? Rewrite it again.

Every serious lab ended up with a drawer full of one-off harnesses that only their author understood, and half of them were duct tape and a prayer. The v1 rewrite pulls those three concerns apart. Verifiers is the environment stack behind Prime Intellect's INTELLECT models, so this is the tooling a frontier open-weight lab actually trains on, not a toy.

The three layers

Taskset, the "what." A taskset holds the data, the tools a task exposes, and the scoring logic, all independent of how the task gets solved. Under the hood it separates a serializable per-row TaskData from a lifecycle-managing Task class, so a dataset row and the machinery around it are no longer the same object.

Harness, the "how." The harness is the program that actually produces a rollout: a plain ReAct loop, a CLI-based agent like Codex or Terminus 2, or something you write yourself. This is the part everyone hand-rolled before, now promoted to a first-class, reusable component.

Runtime, the "where." The runtime decides where execution happens: a local subprocess, Docker, or a remote sandbox such as Prime Sandboxes or Modal. Same task, same agent, different blast radius.

The tidy way to picture it: the taskset is the recipe, the harness is the cook, and the runtime is the kitchen. Because they are separate, you can put any cook in any kitchen making any recipe, instead of hiring a new chef every time you change the menu. The payoff is combinatorial: a handful of tasksets times a handful of harnesses times a handful of runtimes covers a matrix you used to build one bespoke script at a time.

Bring your own harness

The rewrite also opens the door to formats built outside Prime Intellect. Harbor is the first fully supported third-party format, shipping with a taskset base, reusable built-in harnesses, container-aware runtime config, and a catalog of production benchmarks. NeMo Gym and OpenEnv arrive with alpha support. The point is that a v1 environment is a contract, not a walled garden.

Everything is typed with strict Pydantic configs across tasksets, tasks, harnesses, and runtimes. You scaffold a new environment package with uv run init, expose taskset- or task-scoped tools over MCP, attach user simulators to fake a human in the loop, and score traces with rewards, metrics, group rewards, reusable helpers, or configurable LLM judges. Evaluation artifacts now stream to a traces.jsonl format that stays memory-efficient across long or concurrent runs.

Why builders should care

Two reasons. First, evals stop rotting. When the benchmark, the agent, and the sandbox are separate typed components, you can rerun last month's eval against this month's model without reverse-engineering your own code. Reproducibility becomes the default instead of a heroic effort.

Second, the eval and the training loop are now the same object. A v1 environment plugs directly into prime-rl 0.7.0, whose rollout path consumes the same taskset and harness blocks end to end and lets you bring your own harness into training. The thing you graded your agent with is the thing you train it on, which is exactly how you avoid the classic trap of optimizing against a metric your eval never actually measured.

This lands in the same week the AI-engineering crowd has decided that harness design, not raw model quality, is where the next round of gains hides. Verifiers v1 is a bet that the harness deserves to be a real, swappable layer rather than a script you apologize for.

The fine print

Keep expectations calibrated. This is version 0.2.0 previewing the v1 namespace, not a 1.0: the v1 and legacy APIs are meant to stay unmixed, and two of the three third-party formats are still alpha. It is a library for people building RL environments and evals, not a one-click product, and the hard part, designing a harness that faithfully represents the task, is still on you. Verifiers just makes that harness something you can name, reuse, and hand to someone else.

Key Takeaways

• Verifiers v1 (in version 0.2.0) rewrites Prime Intellect's open-source eval stack around three swappable layers: taskset, harness, runtime.
• Taskset is the work (data, tools, scoring), harness is the agent that solves it, runtime is where it runs (local, Docker, or a remote sandbox).
• Any task runs under any compatible harness inside any runtime, turning a pile of bespoke scripts into a mix-and-match matrix.
• Harbor is the first fully supported outside format; NeMo Gym and OpenEnv are in alpha.
• The same v1 environment feeds reinforcement-learning training via prime-rl 0.7.0, so you train on exactly what you evaluate.
• It is a 0.2.0 preview and a library, not a finished product: designing a faithful harness is still the hard, human part.

Sources: Prime Intellect, "verifiers v1", Verifiers GitHub releases, prime-rl GitHub releases, Prime Intellect.