Sourcebot

What is agenteval?

agenteval is a framework-agnostic Python toolkit that scores LLM agents with a statistical pass rate instead of a binary unit-test result. You assert on tool use and behavior across repeated runs, then track that pass rate across releases.

Quickstart

Install agenteval-py, write a test with tracer.wrap and tracer.assert_that, and get a 10-run pass rate from the CLI.

Agent Guide

Signatures, tables, and copy-paste commands for wiring @agenteval.test, Tracer, AssertionSet, adapters, and the CLI from a single page.

How agenteval Works

See the four moving parts of agenteval, Tracer, Runner, AssertionSet, and Reporter, and how they turn repeated agent runs into a pass rate.

The Inbound/Outbound Email Model

Explains the normalized email object every Primitive SDK returns and the receive → inspect → send/reply/forward workflow, including wait mode and the four terminal delivery statuses.

Webhook Contract and Event Types

The webhook contract is the canonical email.received JSON Schema plus a shared signature scheme and event-type catalog that every Primitive SDK validates and generates code from identically.

x402 Payments Model

The non-custodial payment flow shared identically by every Primitive SDK and the CLI: register a payout address, issue a challenge, sign it locally, and settle on chain, all governed by an org-level spend policy.

The @agenteval.test decorator

Decorate a test function so the Runner executes it N times and scores a pass rate against your threshold. Configure n, threshold, and tags for async or sync functions.

Framework adapters

Decide whether to wrap tools with tracer.wrap or use an OpenAI, Anthropic, or LangChain adapter. This page covers the shared wrap_tools and extract_token_usage contract and when to skip adapters.

CLI Reference

Lookup for every agenteval run flag, exit code, and the JSON report written by --output. Use it when you wire the CLI into a script or a CI step.

Python API Reference

Signatures, parameters, and defaults for every name the agenteval package exports, from @agenteval.test and run_suite to Tracer, AssertionSet, and RichReporter.

Installation and extras

Choose the pip command for agenteval-py and the extra that matches your stack so you get the CLI, the import, and only the adapter or dev dependencies you need.

Errors and Exceptions

Diagnose DiscoveryError, collected assertion failures, and TracerError by matching the message you see to the code that raised it. Each section states the cause and the concrete fix.

Common issues and FAQ

Fix tests that never show up, pass rates that swing under concurrency, and wide confidence intervals that make a green threshold unreliable.

Reporting Results

Print color-coded pass rates with RichReporter after every suite, then export the same SuiteResult as JSON for dashboards.

Using agenteval in CI

Fail the GitHub Actions build when an agent drops below its pass-rate threshold, and keep eval-report.json as an artifact you can inspect after the job ends.

Crawl an entire website

Scrape a web page

Run an AI agent to gather structured data