{"title":"Sourcebot","description":"Sourcebot Docs","canonical_base_url":"https://help.newtools.space","llms_full_txt":"https://help.newtools.space/llms-full.txt","categories":[{"name":"Getting Started","articles":[{"title":"What is agenteval?","url":"https://help.newtools.space/what-is-agenteval.md","description":"agenteval is a framework-agnostic Python toolkit that scores LLM agents with a statistical pass rate instead of a binary unit-test result. You assert on tool use and behavior across repeated runs, then track that pass rate across releases."},{"title":"Quickstart","url":"https://help.newtools.space/quickstart-e8b4c85e.md","description":"Install agenteval-py, write a test with tracer.wrap and tracer.assert_that, and get a 10-run pass rate from the CLI."},{"title":"Agent Guide","url":"https://help.newtools.space/agent-guide-3895cb0a.md","description":"Signatures, tables, and copy-paste commands for wiring @agenteval.test, Tracer, AssertionSet, adapters, and the CLI from a single page."}]},{"name":"Core Concepts","articles":[{"title":"How agenteval Works","url":"https://help.newtools.space/how-it-works.md","description":"See the four moving parts of agenteval, Tracer, Runner, AssertionSet, and Reporter, and how they turn repeated agent runs into a pass rate."},{"title":"The Inbound/Outbound Email Model","url":"https://help.newtools.space/email-model.md","description":"Explains the normalized email object every Primitive SDK returns and the receive → inspect → send/reply/forward workflow, including wait mode and the four terminal delivery statuses."},{"title":"Webhook Contract and Event Types","url":"https://help.newtools.space/webhook-contract.md","description":"The webhook contract is the canonical email.received JSON Schema plus a shared signature scheme and event-type catalog that every Primitive SDK validates and generates code from identically."},{"title":"x402 Payments Model","url":"https://help.newtools.space/x402-payments-model.md","description":"The non-custodial payment flow shared identically by every Primitive SDK and the CLI: register a payout address, issue a challenge, sign it locally, and settle on chain, all governed by an org-level spend policy."},{"title":"The @agenteval.test decorator","url":"https://help.newtools.space/test-decorator.md","description":"Decorate a test function so the Runner executes it N times and scores a pass rate against your threshold. Configure n, threshold, and tags for async or sync functions."}]},{"name":"Integrations","articles":[{"title":"Framework adapters","url":"https://help.newtools.space/adapters-overview.md","description":"Decide whether to wrap tools with tracer.wrap or use an OpenAI, Anthropic, or LangChain adapter. This page covers the shared wrap_tools and extract_token_usage contract and when to skip adapters."},{"title":"Instrument a LangChain agent","url":"https://help.newtools.space/langchain-adapter.md","description":"Record LangChain tool start, end, and error events automatically by passing AgentEvalCallbackHandler into invoke or ainvoke. Skip per-tool tracer.wrap() when the agent already runs through LangChain."}]},{"name":"Reference","articles":[{"title":"CLI Reference","url":"https://help.newtools.space/cli-reference-902a307e.md","description":"Lookup for every agenteval run flag, exit code, and the JSON report written by --output. Use it when you wire the CLI into a script or a CI step."},{"title":"Python API Reference","url":"https://help.newtools.space/python-api-reference.md","description":"Signatures, parameters, and defaults for every name the agenteval package exports, from @agenteval.test and run_suite to Tracer, AssertionSet, and RichReporter."},{"title":"Installation and extras","url":"https://help.newtools.space/installation-and-extras.md","description":"Choose the pip command for agenteval-py and the extra that matches your stack so you get the CLI, the import, and only the adapter or dev dependencies you need."}]},{"name":"Troubleshooting","articles":[{"title":"Errors and Exceptions","url":"https://help.newtools.space/errors-and-exceptions.md","description":"Diagnose DiscoveryError, collected assertion failures, and TracerError by matching the message you see to the code that raised it. Each section states the cause and the concrete fix."},{"title":"Common issues and FAQ","url":"https://help.newtools.space/faq.md","description":"Fix tests that never show up, pass rates that swing under concurrency, and wide confidence intervals that make a green threshold unreliable."}]},{"name":"Guides","articles":[{"title":"Reporting Results","url":"https://help.newtools.space/reporting-results.md","description":"Print color-coded pass rates with RichReporter after every suite, then export the same SuiteResult as JSON for dashboards."},{"title":"Using agenteval in CI","url":"https://help.newtools.space/ci-integration.md","description":"Fail the GitHub Actions build when an agent drops below its pass-rate threshold, and keep eval-report.json as an artifact you can inspect after the job ends."}]},{"name":"other","articles":[{"title":"Crawl an entire website","url":"https://help.newtools.space/crawl-an-entire-website.md","description":"Use the Crawl endpoint to fetch data for every page on a given website."},{"title":"Scrape a web page","url":"https://help.newtools.space/scrape-a-web-page.md","description":"Use the Scrape endpoint playground to fetch LLM-ready content from a single URL."},{"title":"Run an AI agent to gather structured data","url":"https://help.newtools.space/run-an-ai-agent-to-gather-structured-data.md","description":"Use the Agent feature to run a natural-language data-gathering task across multiple URLs or a prompt."}]}]}