Pricing

    Gate AI releases from the command line.

    Run and gate evals from your terminal, CI, or agentic harness in minutes.

    Get started freeSee the pricing
    No credit card · bring your own key · read the docs
    Autoeval CLI terminal: autoeval quickstart runs a sample evaluation, autoeval --help lists all commands
    Without And With Plumloom

    From agent run to release decision, without stitching tools together.

    Without Plumloom

    Your harness + observability + eval + gating glue.

    Agent runthe harness already produced this
    SDK / export
    ↓
    Instrument and export the trace
    ↓
    Store in an observability platforminstall SDK · configure storage · dashboards · retention · operate it↻ Duplicate work: captures the run your harness already produced
    ↓
    Configure evals and scorers
    ↓
    Run evaluations
    ↓
    Interpret results and reliability
    ↓
    Build and maintain a release gateyou build and operate this
    ↓
    Release decision

    With Plumloom

    Your harness + Autoeval.

    Agent runthe same harness, the same run
    ↓
    Autoevalopen source
    evaluate · measure reliability · gate
    ↓
    Release decision
    How It Works

    From The Run You Already Produce To The Answer

    Upstream · your stack

    Your agentic harness

    e.g. DeepSeek Harness, coding agents

    The agent does the work and the run is already the artifact: scenarios, conversations, and agent traces come straight out of the harness. No logging pipeline to wire up first.

    scenario · conversation · agent trace
    the run

    Local · open source

    Autoeval

    The CLI. Author, run, and gate a run from your terminal, in CI, or driven by a coding agent over MCP. Bring your own OpenAI, Anthropic, or Together.ai key.

    CLI · CI · MCP · BYOK
    eval run

    Server · the reliability engine

    EvalCore judges the run, and tells you if it holds

    Calibrated judges grade to your experts' bar, several judges score each answer independently, and scenarios get statistical multi-run evaluations with adaptive stopping and confidence intervals.

    calibrated judges · multi-judge · statistical multi-run (scenarios) · confidence intervals
    What Makes It Reliable

    What Makes The Verdict Trustworthy

    The engine behind Plumloom Autoeval is EvalCore. It turns a raw model score into a result you can act on.

    Calibrated judges

    Judges grade against anchors set by your own experts, so the standard stays consistent from run to run and does not drift over time. A 4.2 means the same thing every time.

    Reliability For Your Loop

    Your Harness Runs The Loop. Plumloom Makes It Trustworthy.

    Agent acts

    The agentic harness already gives you the operational loop: the agent acts, the result is judged, and the harness decides what to keep and what to improve next. It is fast, and it is autonomous.

    Result judged

    Every decision in that loop rests on the judgment underneath it. Judge with a single run and the same case can pass one moment and fail the next, so the loop moves quickly on a signal that is part noise.

    Harness decides

    The harness keeps, promotes, or discards based on a verdict that may not hold. One moment a change looks like an improvement; the next, it is a regression you cannot see yet.

    Improves

    Plumloom makes the judgment step reliable: calibrated to your experts and scored by multiple judges. For scenarios, it adds statistical multi-run evaluations with adaptive stopping and confidence intervals. The loop keeps its speed, and its decisions now rest on evidence that holds.

    The Operational Loop

    Plumloom makes the judgment step reliable without slowing the loop.

    Agent actsThe harness runs the work
    Result judgedPlumloom sits here
    Harness decidesKeep, promote, or discard
    ImprovesThe loop runs again
    • Single run, verdict can flip
    • Calibrated, multi-judge, statistical multi-run (scenarios), holds
    Why Plumloom

    Most Tools Hand You The Runs. Plumloom Hands You The Answer.

    Not an eval framework

    Frameworks run your evaluation and give the runs back, leaving the judgment to you. Plumloom returns the judgment: calibrated, scored by multiple judges, and for scenarios using statistical multi-run evaluations with adaptive stopping and confidence intervals.

    Not observability

    Observability shows what happened after a response ships, and needs your app instrumented first. Plumloom tells you whether to ship at all, with a provider key and nothing to install.

    FAQs
    Do I need to instrument my app or send you my logs?
    No. You bring the run: a scenario, a conversation transcript, or an agent trace your harness already produces. There is no SDK to install and no logging pipeline to wire up first.
    Which models does it use?
    Bring your own key: OpenAI, Anthropic, or Together.ai. The models you select run on your key, and there is no provider markup.
    Is Autoeval open source?
    Yes. Autoeval is the open-source CLI. Run it locally, in CI, or drive it from a coding agent over MCP.
    How is this different from observability tools?
    Observability shows what happened after a response ships, and needs your app instrumented first. Plumloom Autoeval tells you whether to ship at all, before release, with a provider key and nothing to install.
    What does reliable mean here?
    It means you can trust the score. Judges are calibrated to your experts' bar and several judges score each result so you see the agreement behind the number. For scenarios, EvalCore uses statistical multi-run evaluations with adaptive stopping and confidence intervals: multiple trials that stop early once the score is stable, then report a confidence interval.
    What can it evaluate?
    Three kinds of run: scenarios (text), conversations (a transcript against an expected outcome), and agent traces (OpenTelemetry or OpenInference), with a trace's outcome and path scored separately.
    Does it work in CI?
    Yes. A single evaluation or a suite returns a pass or fail result with a non-zero exit code on failure. Wire it as a required check and a known regression cannot merge.

    Find Out Whether Your Improvement Is Real

    Run one evaluation from your terminal and see whether the number holds before you ship it.

    Get started freeSee the pricing
    No credit card · bring your own key · read the docs