From endpoint to evidence.

01Your first evaluation

Create an account and connect your company endpoint. Activate the monthly platform plan, save a card, and fund your prepaid wallet in Usage & billing before real provider requests. Then generate an allevals API key. Set your deployment’s base URL and keep the key in an environment variable. The example samples 25 items; use "limit": 1, 5, or 10 for an even smaller run.

export ALLEVALS_URL="https://www.allevals.com"
export ALLEVALS_API_KEY="your_api_key"

curl "$ALLEVALS_URL/v1/evaluations" \
  -H "Authorization: Bearer $ALLEVALS_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: release-v2-mmlu-001" \
  -d '{
    "endpoint": "my-model",
    "benchmark": "mmlu-test",
    "limit": 25,
    "expected_collection_mode": "prepaid"
  }'

The request queues work and returns an evaluation resource. Poll its URL for progress. Save the evaluation ID for later inspection.

02Add your connection

In Connections, add a name, a unique workspace slug, and the complete HTTPS URL that receives inference requests. You can configure a bearer token, a custom authentication header, and additional HTTP headers.

Connection creation and editing are session-authenticated dashboard actions, not bearer-key provisioning endpoints. Once saved, reference the connection’s slug from the REST API or SDK when creating an evaluation. You never need to repeat endpoint credentials in an evaluation request.

List saved connections with GET /v1/connections or SDK client.connections.list() using a key with connections:read. The response includes all active connections in your workspace without credentials or custom header values. Use a returned slug as endpoint in an evaluation.

{
  "endpoint": "production-v2",
  "benchmark": "mmlu-test",
  "limit": 25,
  "seed": 42,
  "expected_collection_mode": "prepaid"
}

Omit seed for a fresh random sample. Reuse a seed with the same benchmark version and limit to select the same rows and order. The resolved seed is returned in configuration.seed. Idempotent retries keep the original run; separate random samples can overlap.

Endpoint requirements

Save your connection in the dashboard, then submit an evaluation through the API with a small sample size such as limit: 1. This sends a real request and uses evaluation allowance and applicable prepaid credits. Saving or editing a connection never starts an evaluation.

• Public HTTPS URLs only. Local, private-network, and metadata-service destinations are not allowed.
• Redirects are not followed. Use the final inference URL.
• Saved API keys and custom header values are encrypted and write-only.
• Unsafe transport headers, including Host and Cookie, are not accepted.
• Choose a custom company endpoint, OpenRouter with your own key, or Cloudflare AI Gateway with your own credentials. OpenRouter uses its fixed trusted URL; your company endpoint or gateway uses its configured HTTPS URL and optional headers. The endpoint must support the benchmark’s required protocol; a chat router does not support every native benchmark. Billing mode is immutable after creation.

Completions request and response contract

Use the complete HTTPS POST URL, not only a base URL. The adapter sends one non-streaming user message and expects JSON with text at choices[0].message.content. Tool-call-only responses and streaming are unsupported.

{
  "model": "your-model",
  "messages": [
    {
      "role": "user",
      "content": "Which number is even?\nA. Three\nB. Four\nAnswer with the option letter only."
    }
  ],
  "max_tokens": 64,
  "temperature": 0,
  "stream": false
}

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "B"
      }
    }
  ],
  "usage": {
    "prompt_tokens": 28,
    "completion_tokens": 1
  }
}

Usage fields are optional. Missing usage is not a zero-cost guarantee. Responses are processed transiently and are not retained in run results. Both adapters require an effective model: save a default on the connection, or pass model with each evaluation. Saving without a default is allowed, but running without either model returns model_required.

Completion probabilities for original MMLU

Choose Completion logprobs and a raw completion endpoint. The author evaluator sends a five-example prompt with max_tokens: 1, temperature: 0,echo: true and logprobs: 100. These settings are fixed by the benchmark.

The last entry of choices[0].logprobs.top_logprobs must contain finite log probabilities for each of " A", " B", " C" and " D". Generated letters, chat probabilities and decision scores do not satisfy this contract. The author container selects and grades the answer; missing probabilities fail the sample.

This format uses custom endpoint billing. Saving the connection declares its capabilities; a connection smoke check confirms text delivery only. Actual MMLU responses are checked separately.

03Evaluate typed decision models

The decision adapter uses a System One-compatible JSON shape: a state object, named questions with explicit types, and an answers object in the response. This is a wire-format contract, not a claim that every decision provider uses the same API.

{
  "model": "your-model",
  "state": {
    "question": "Which number is even?"
  },
  "questions": {
    "answer": {
      "type": "choice",
      "instructions": "Select the correct answer to the question.",
      "criteria": {
        "A": "Three",
        "B": "Four"
      }
    }
  }
}

{
  "answers": {
    "answer": {
      "type": "choice",
      "choice": "B",
      "probabilities": {
        "A": 0.1,
        "B": 0.9
      }
    }
  },
  "usage": {
    "input_tokens": 28,
    "output_tokens": 1
  }
}

• answers.answer.type must be choice, and choice must be one of the request’s criteria keys.
• probabilities must include exactly every criteria key, with finite numeric values from 0 to 1. Their sum must be within 0.02 of 1.
• The selected choice must have the highest probability (ties accepted; numerical tolerance 1e-9). An optional finite confidence value is accepted during transient normalization, but the response and confidence trace are not stored in run results.
• This adapter currently supports text-only multiple-choice questions. Native score, noul, image, and free-form decision tasks are not supported.

Only benchmarks with an explicit decision adapter are runnable against decision endpoints. Converting a free-form answer task into a typed decision changes its protocol, so those results must remain distinct from chat-completion scores.

04Create and manage evaluations

Native prediction endpoints

Save a custom native_predictions endpoint for benchmarks with an original final-prediction interface. The model adapter owns generation and final-answer extraction. Keep its implementation and configuration stable and identify its version in the model name; the connection revision freezes the saved URL, credentials and model, but cannot pin remote adapter code.

drop-public-dev-author-predictions sends model, protocol, request_sha256 and request. The request contains the original query id, source index, and inputs with each original passage_id, row, passage and question. Echo the hash and return exactly {request_sha256, prediction, usage: {input_tokens, output_tokens}}. Prediction is the native string or string list, including an empty answer; chat envelopes and extra reasoning fields are rejected. The hash binds the original protocol and request, and does not include the model field.

DROP has 9,535 query IDs and 9,536 question occurrences. One duplicate query includes both inputs and requires one shared prediction. The original evaluator weights both occurrences, preserves exact match and F1, and determines the reported F1 score. Omit temperature, output-token and judge settings. Each graded query incurs one platform fee; provider usage is billed directly. Use a one-item DROP run to check the adapter: generic connection tests do not support this format. This edition does not reproduce a fixed-model paper result.

mmmu-validation-original-json-author uses the same outer request and response envelope, with one prediction string. Its native request contains original id, question, options, question_typeand seven image_1 through image_7 slots. Each image is null or {bytes_base64, path}; the PNG bytes are unchanged. The hash covers this native request object. The largest frozen request is about 12.2 MB. Return a final multiple-choice label exactly, or an open-answer string; the platform performs no additional answer extraction. Predictions are limited to 262,144 UTF-8 bytes.

This MMMU edition preserves the original validation answer JSON, including two labels that differ from later Hugging Face corrections. Original subject, domain and overall reports use instance counts and accuracies rounded to three decimals. Omit generation and judge settings. One graded item incurs one platform fee; model adapter provenance and generation settings remain the caller’s responsibility.

SOB original structured-output profiles

sob-full-openai-author, sob-text-openai-author,sob-image-openai-author and sob-audio-openai-authorpreserve 5,324, 5,000, 209 and 115 original occurrences respectively. Use a custom chat endpoint with the author's supported GPT-4o, GPT-4.1 or GPT-5 model family. Original requests require JSON-object response format and max_completion_tokens: 2048; the author owns model-specific temperature and reasoning fields. Omit generation overrides and judge_endpoint.

Image and audio records already contain the author's normalized text contexts; these profiles do not perform OCR or speech recognition. Original schema validation, eleven row metrics, weighted/unweighted bootstrap reports and the coverage-adjusted overall_adj leaderboard score run in the native evaluator. Row scores are leaf_value_em, not the run's primary score. Modality coverage uses fixed original population counts, even for sampled runs. Numeric reports and checkpoint indices are exportable; source contexts, schemas, schema paths, references and responses are not retained.

{
  "benchmark": "sob-text-openai-author",
  "endpoint": "sob-candidate",
  "limit": 25,
  "expected_collection_mode": "prepaid"
}

ERQA original visual report

The erqa profile uses the original image ordering, PNG bytes, prompt and response normalization. Its custom chat endpoint receives temperature 0 and max_tokens: 300. Native requests can approach 14 MiB; image reduction is not part of this profile.

Completed runs retain the native overall correct count and accuracy, single-image and multi-image correct/total counts, and original by_type correct/total counts for represented question types. The original scalar accuracy remains unchanged. Reports and exports contain these counts only, without questions, images or responses.

MuSR original CoT+ GPT-4 profile

musr-cot-plus-gpt4-author requires a custom chat endpoint fixed to gpt-4 and limit: "all". All 756 original questions run in the author's execution order. The original request fixes temperature 1, top-p 1, maximum 2,400 tokens, one completion and null stop. Omit generation overrides and judge_endpoint.

The native parser includes a gold-biased answer check and random fallback; fallback draws affect later domain shuffles. The platform preserves this behavior and stores only numeric decisions. A completed run has score: null and three original domain reports: murder mysteries (250), object placements (256), and team allocation (250), with correct counts and percentages rounded to one decimal place. No overall mean or partial-sample score is defined. Source questions and responses stay transient.

T2-RAGBench original oracle prediction profiles

t2-ragbench-convfinqa-oracle-native-author, t2-ragbench-finqa-oracle-native-author and t2-ragbench-tat-dqa-oracle-native-author preserve 3,458, 8,281 and 11,349 original records respectively. Use a custom native_predictions endpoint with model-owned prompts, generation and final prediction extraction. Omit generation overrides and judge_endpoint.

The hash-bound native request contains exactly original id, context_id, split, question and complete context. Return one prediction string and usage. Original NumberMatch ignores numeric sign and powers-of-ten scale, uses case-sensitive yes/no, and reports 0–100. Platform score is that native report divided by 100. All 11 empty questions and 110 TAT-DQA native-zero references remain. These named oracle metric profiles do not claim retrieval or full author generation-driver reproduction; raw inputs, references and predictions remain transient.

WinoGrande development prediction profile

The reviewed winogrande-dev-native-author profile uses the original 1,267 development items and native accuracy. Its custom native_predictions adapter receives the same outer envelope described above. The native request contains exactly qID, sentence, option1 and option2, without gold. The hash covers that native request object. Return the hash, token usage and the literal prediction string "1" or "2". Numbers, choice letters, additional reasoning fields and labels with whitespace are rejected.

The model adapter owns generation and final-label extraction; omit generation and judge settings from the evaluation request. Original qIDs and source order are retained. The author’s unchanged metric reports accuracy and metrics.accat its original floating-point precision. This is one model’s development accuracy, without the five XS/S/M/L/XL training conditions, learning-curve AUC or hidden-test evaluation. Source text, gold and predictions do not appear in result exports.

Global-MMLU native likelihood endpoints

Save a custom loglikelihood endpoint. It must use the selected model’s actual tokenizer and compute the summed conditional log probability of each continuation given its context. The greedy flag reports whether every continuation token is the model’s greedy choice. Do not substitute a generated answer, normalized choice probability, a different tokenizer, or top-token probabilities.

{
  "model": "your-model",
  "requests": [
    {
      "context": "Exact author five-shot context ending in Answer:",
      "continuation": " A"
    },
    {
      "context": "Exact author five-shot context ending in Answer:",
      "continuation": " B"
    },
    {
      "context": "Exact author five-shot context ending in Answer:",
      "continuation": " C"
    },
    {
      "context": "Exact author five-shot context ending in Answer:",
      "continuation": " D"
    }
  ]
}

{
  "results": [
    {
      "loglikelihood": -1.2,
      "is_greedy": true
    },
    {
      "loglikelihood": -2.3,
      "is_greedy": false
    },
    {
      "loglikelihood": -3.4,
      "is_greedy": false
    },
    {
      "loglikelihood": -4.5,
      "is_greedy": false
    }
  ],
  "usage": {
    "input_tokens": 1000,
    "output_tokens": 4
  }
}

Return exactly four ordered finite, nonpositive likelihoods and integer token usage. The endpoint owns correct token boundaries and context handling; insufficient model context must fail rather than silently truncate the author prompt. Omit temperature and output-token settings when creating the evaluation. The pinned author framework owns five-shot sampling, prompts, scoring and aggregation. A provider request is never automatically replayed.

The additional original 15-language Lite release is global-mmlu-lite-author. Full 42, English and Telugu use global-mmlu-full-author, global-mmlu-en-author and global-mmlu-te-author. Multilingual runs complete with score: null and native per-language accuracy/count in author_metrics; the author defines no cross-language average. English and Telugu report their native scalar accuracy.

Full 42 indexes all 589,764 test rows. Current Pro limits remain 1,000 numeric items per run, 20,000 for a complete benchmark and 100,000 reserved items per month. A complete Full 42 request is quota-restricted; numeric samples within those limits work. Synthetic native-runtime probes do not validate a complete benchmark run. The legacy 23-language Lite release remains blocked by its missing Welsh development split. Generic connection checks do not support likelihood endpoints; verify with an explicitly requested native benchmark sample.

HRM8K with HRET MathVerify

mmmlu-openai-chatgpt-author contains the 196,588 original translated rows across 14 locales, excluding English. It uses OpenAI's original ChatGPT sampler system message, temperature 0.5, and 2,048 output tokens. Omit generation overrides and use a custom chat-completions endpoint. The author parses and grades each response and reports locale/category means and population standard deviations. Multilingual runs have no overall scalar. Existing sample and plan limits apply; this catalog entry does not expand the 20,000-item full-run limit.

multiloko-dev-author-chat uses all 7,750 original locally sourced dev questions across 31 languages, unchanged zero-shot chat prompts and the historical 4225 scorer. Temperature0 is fixed. Supply max_output_tokens explicitly (1–4096 through this API); the author does not specify a default output limit. One-user-message HTTP transport is disclosed because upstream releases text prompts rather than a complete generation driver. The main score is native equal-language average EM; all seven metrics, 28 group aggregates and per-language metrics are retained with their native units. No MTE or LE, translated subset or private test data is added. A custom chat endpoint is required; questions and responses remain transient.

hrm8k-hret-math-verify-author runs the original HRET zero-shot pipeline with its supported mixed expression and LaTeX extraction settings. Use a custom chat endpoint and explicitly provide temperature and max_output_tokens; the author toolkit does not freeze those choices.

This named edition uses native row-weighted accuracy, not the older five-subset macro average or a claimed paper reproduction. Native parse and verification failure rates and the ancillary Korean language indicator appear in author_metrics. The unseeded language indicator can vary without changing accuracy. Original reference-verification quirks are preserved; no prompts or responses are retained.

MathArena original June and August API editions

arxivmath-june-2026-author evaluates 48 current June questions using four independent native attempts and the original parser. Each attempt may receive the author’s last-chance prompt, for up to eight candidate requests per question. This edition is separate from arxivmath-june-2026-historical49-author, which retains all 49 historical questions, including question 18, and uses the same original four-attempt June parser with the author-supported historical source configuration.

arxivmath-august-2026-author-api evaluates 57 August questions using the author-supported API profile without tools. It requires a separate custom chat judge endpoint with model gemini-3.8-flash. Four candidate responses pass through the original judge, which fixes max_tokens: 65536 and high reasoning effort. Confirmed judge-format failures may request up to three judge attempts per candidate.

Both editions require a custom chat candidate endpoint and no generation overrides. The original candidate profile supplies its 16,000-token bound and model-specific token-limit field. Native accuracy averages all four attempts; it does not choose the best answer. A question may require up to eight June requests or sixteen August requests. Provider inference is billed directly per request, with one platform fee after the question is fully graded. Unknown provider outcomes and the whole-question runtime limit stop execution without automatic replay or a partial score. Saved results contain only source IDs, four correctness flags and numeric aggregate metrics.

Pencil Puzzle original Direct Ask

pencil-puzzle-direct-ask-local-author evaluates all300 original golden puzzles through the released local-provider profile. Use a custom streaming chat endpoint and omit generation overrides. The author builds the prompt, parses the streamed response, executes moves in pzprjs, and computes the native pass count.

The native executor permits board-reset and edit commands; this named edition preserves that behavior. Fully received responses can request original retries; uncertain provider failures stop without replay. Provider usage is billed per request and the platform charges once per graded row. Only source locators and scalar completion flags persist.

MMLU-Pro universal API edition

mmlu-pro-apix-author preserves the published streaming API driver over all 12,032 test questions. Use a custom chat endpoint accepting streaming deltas and final usage, temperature 0.1, top_p 0.95 and max_tokens 32768. Omit temperature and max_output_tokens when creating the evaluation. Native prompts include five original examples from the question’s category.

The original token budget may request up to three attempts; wrong answers are not retried. Unknown provider failures end the row without another request. Provider usage is billed directly per attempt, with one platform charge per successfully graded row. Native category counts and sample-weighted accuracy appear in author_metrics. Prediction ordinals remain private; text and streams are not retained. This named edition is distinct from fixed-model paper results.

KMMLU original direct chat

mmlu-redux-ok-author is the separately named author-endorsed community harness edition: 5,330 ok-only rows, zero-shot, temperature 0, max_tokens 256, generation seed 1234 and stop </s>. It requires a custom chat endpoint preserving the native message fields. The request seed selects rows only. Native first-uppercase A–D extraction and weighted subject/group metrics are retained. Historical 5,668-row correction/CoT packages and HELM paper results are distinct. Provider usage is billed directly; one platform fee per graded row.

kmmlu-direct-author and kmmlu-hard-direct-author require a custom chat endpoint that accepts the author’s temperature 0, max_tokens 256 and four original stop sequences. Omit generation settings or supply these exact values. The native backend truncates output at the original stops and runs the original ExactMatch metric. Prompts and responses are not retained.

The primary score is the original framework equal-subject macro option prescribed by the author README. Native row-weighted micro and per-subject counts and accuracies are available in metadata exports. Samples cover only represented subjects. Full direct KMMLU has 35,030 rows and exceeds the current 20,000-item full-run plan limit; numeric samples remain limited to 1,000. Hard has 4,104 rows. Provider charges are direct and one platform fee applies to each successfully graded row.

Original SimpleQA author chat sampler

SimpleVQA original GPT-4o profile

simplevqa-gpt4o-parallel-author preserves all 2,025 rows of the pinned original Parquet edition. Choose two separate custom chat-completions endpoints with model gpt-4o, and pass the judge endpoint slug as judge_endpoint. Omit temperature and output-token overrides: the original parallel driver sends its own request fields and grades the raw candidate text with its fixed judge.

Original image bytes, including AVIF source bytes, retain the author's data:image/jpg;base64, URL prefix. The endpoint must support that exact native image input. The primary score is the original rounded metrics.is_correct; attempted accuracy, F1, and all original class fractions are also preserved. Prompts, image bytes, responses and judge text remain transient; reports contain scalar metadata only.

{
  "benchmark": "simplevqa-gpt4o-parallel-author",
  "endpoint": "simplevqa-candidate",
  "judge_endpoint": "simplevqa-judge",
  "limit": 1
}

simpleqa-original-author-chat requires separate custom chat candidate and judge endpoints. Select a judge whose model is gpt-4.1-2025-04-14 using judge_endpoint; omit generation overrides. Both original samplers use temperature 0.5, max_tokens 2048 and the helpful-assistant system message. One candidate and one judge call incur provider usage; one platform fee follows successful grading. Original attempted-answer accuracy, F1 and standard deviations are included in the result metadata. This is distinct from Google SimpleQA Verified.

SimpleQA Verified native Google GenAI edition

mrcr-v2-8needle-64k-128k-genai-author and mrcr-v2-8needle-128k-256k-genai-author run the original MRCR GenAI profile on exact 103- and 141-query intervals. Use a custom generate_content endpoint whose URL model matches the saved model, omit generation overrides and judge_endpoint, and provision enough context for the selected interval. The native request uses one complete text part at temperature 1.0; the original similarity mean and four-decimal display are returned. One provider call and one platform fee per successfully graded query.

simpleqa-verified-kaggle-genai-author runs the author-endorsed Kaggle notebook over Google DeepMind’s 1,000 source rows. Save a custom generate_content candidate endpoint with an HTTPS URL whose path is exactly /v1beta/models/<model>:generateContent. Its model must match the URL; query strings and fragments are not accepted. For Google’s Developer API, use https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent with model gemini-2.5-flash, and save the API key using the custom authentication header x-goog-api-key.

Select a separate custom chat judge endpoint with model gpt-4.1-2025-04-14 using judge_endpoint. The candidate model must differ from the judge model because the original notebook dispatches the candidate first. Omit generation overrides: the original candidate and judge functions own their requests. Both providers bill inference directly; one platform fee applies per successfully graded row. Generic connection checks are unsupported for this format; use a small evaluation to verify the complete contract.

{
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "What is the capital of France?"
        }
      ]
    }
  ]
}

{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "Paris."
          }
        ]
      }
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 8,
    "candidatesTokenCount": 2
  }
}

The complete response stays transient and is interpreted by the original Google GenAI SDK, including thought-part filtering and empty candidates. Return the native response object. The notebook’s original grade parser, including its uppercase A/B/C matching quirks, is preserved. The primary score is native F1; metadata includes accuracy given attempted, correct/incorrect/not-attempted counts, and source IDs, source indices and class flags per row. Prompts, answers and model or judge responses are never exported.

{
  "benchmark": "simpleqa-verified-kaggle-genai-author",
  "endpoint": "my-genai-candidate",
  "judge_endpoint": "simpleqa-verified-judge",
  "limit": 1,
  "expected_collection_mode": "prepaid"
}

LiveBench legacy noncoding author profile

livebench-legacy-noncoding-author-local requires separate custom chat candidate and judge endpoints. Select a judge with model gpt-5-mini using judge_endpoint and omit generation overrides. The native default local profile sends one candidate request at temperature 0 with a 4,096-token budget. The original math grader may request one judge call; objective rows need only the candidate. Providers bill those calls directly, with one platform fee after successful grading.

This named edition contains 682 legacy noncoding questions surviving the June 25, 2026 cutoff. It excludes coding tasks and does not apply registered-model leaderboard presets. Original task means, category means and rounding determine the aggregate; native task and category percentages are available in metadata exports. Samples use the original ignore-missing-judgments option and are not complete benchmark scores. Prompts, responses and judge transcripts are not retained.

KINA original n=1 author CLI

medxpertqa-text-ao-author selects the original Text answer-only option, separate from the default chain-of-thought profile. Use a custom chat endpoint with an original supported ordinary model name such as gpt-4o-2024-11-20 and omit generation overrides. The author sends temperature 0, max_tokens 2048, logprobs true and seed 0; native reports preserve rounded percentages by question type. The source is for evaluation only: no training on original or rewritten data and no online examples.

kina-author-n1 requires a custom chat endpoint and no generation overrides. The original CLI sends n: 1 and a 16,384-token budget using max_tokens or max_completion_tokens according to its exact model-name branch. Names containing gpt-5 receive its original medium reasoning effort. Temperature is omitted.

Native lighteval0.9.2 extraction and Pass@1 score each response. Original content, null, reasoning and reasoning_content field presence are preserved for the author fallback; no reasoning stripping occurs. The complete content fields must fit 256 KiB. Native aggregation retains full-precision accuracy, the original rounded percentage and the empty-extraction count. These metadata are exported without source questions or responses. This is the no-search n=1 edition, distinct from legacy EvalScope scoring.

IFBench released author driver

ifbench-test-loose uses the pinned original driver: temperature 0.6, max_tokens 4096 and generation seed 42. Omit generation overrides, or supply exactly those temperature and output-token values. The evaluation seed controls item selection separately. This edition differs from the paper temperature-0 setting.

The custom chat endpoint must accept the exact author request. The original evaluator runs strict before loose checks without removing reasoning text. The primary score is loose prompt accuracy; author_metrics contains both prompt and instruction accuracy plus instruction-tier reports. Result metadata retains only strict/loose boolean verdicts. Prompts and responses are never retained.

EnterpriseRAG and Responses endpoints

The author BM25 baseline retrieves from the full pinned corpus and runs the original correction and grading pipeline. Save a custom responses candidate endpoint and a separate custom Responses judge endpoint with model gpt-5.4. Both must accept streaming Responses requests with medium reasoning and automatic reasoning summaries. Use the complete provider URL, such as https://api.openai.com/v1/responses.

{
  "benchmark": "enterpriserag-bm25-author",
  "endpoint": "my-responses-candidate",
  "judge_endpoint": "rag-judge",
  "limit": 1,
  "seed": 42,
  "expected_collection_mode": "prepaid"
}

Omit temperature and output-token overrides: the original requests own those settings. The seed freezes per-row author randomization. Provider usage is billed directly and the platform charges one fee after a successfully graded row. Judge calls vary with citation checks, document consensus and gold-answer correction. Stream interruptions stop the row without automatically repeating paid work. Connection checks use a nonstreaming Responses request, so endpoints must also support that form.

Results expose native correctness, completeness, document recall and invalid-extra-document metrics in author_metrics. The primary score is the author's combined correctness/completeness percentage divided by 100. Prompts, generated answers and revised gold answers are not included in result exports. A sample is not a full benchmark score.

AMO author grading

AMO publishes grading code, with no fixed candidate generation settings. Create two custom chat endpoints in Connections: your candidate and a separate judge with model o4-mini. The judge must accept the author’s exact request: temperature: 1, max_tokens: 8192,reasoning_effort: "low", and nonstreaming chat completions. Select the judge explicitly and provide candidate temperature and output limit:

{
  "benchmark": "amo-bench",
  "endpoint": "my-candidate",
  "judge_endpoint": "amo-judge",
  "limit": 1,
  "temperature": 0,
  "max_output_tokens": 4096,
  "expected_collection_mode": "prepaid"
}

The author parser grades 39 tasks. Each of the other 11 tasks needs five judge calls: a full run sends 50 candidate calls and 55 judge calls. Each provider bills usage directly, including failed or interrupted calls; the platform charges once per successfully graded row. Saved judge settings are frozen at admission; optionalexpected_judge_endpoint_revision checks the revision you reviewed. Responses stay in memory, with a 256 KiB text limit per call. An interrupted row fails without automatic paid retries. This edition does not reproduce a historical candidate generation configuration.

Sample first, then evaluate the full benchmark

A numeric limit selects a reproducible sample, from just 1 item. Start with 5, 10, or the default 25; you never need to run the full benchmark. Pro workspaces can send "limit": "all" to include every item in a ready benchmark within their full-run allowance. A full run can send thousands of billable requests to your endpoint; it is never selected automatically.

Accepted runs reserve their requested item count against the workspace’s UTC calendar-month budget. Cancellation does not refund this reservation. reserved_items tracks admitted work; attempted_items counts claimed execution slots, not confirmed HTTP delivery or charges. Preflight failures or crashes can occur before a request is sent. Check current allowances in Usage & billing.

Usage is prepaid. A manual top-up charges your card through Stripe; only a verified successful payment adds credits. The server reserves credits before paid work, deducts settled platform fees, and blocks new paid requests when available credit is exhausted. Pending provider costs from historical managed runs keep their reservation until reconciled and are never treated as zero. Credit reservations are separate from the item-count allowances above. Your monthly platform subscription does not fund the wallet.

Optional auto-reload uses your chosen threshold, amount, and UTC daily cap and requires explicit consent to off-session saved-card charges. It can be disabled. Replacing the saved card disables auto-reload until you consent again. Unresolved attempts reserve the daily cap until reconciled. Failed payments or authentication-required payments add no advance credits. Only one potentially chargeable payment may remain unresolved at a time: a pending manual top-up blocks auto-reload, and a pending automatic payment blocks new manual top-ups. Resume an existing manual payment with its original request identity; do not create another charge while its outcome is unknown. Card setup is separate and remains available, but does not resolve an uncertain payment. Authentication-required or review states need reconciliation before new funding proceeds. A checkout return URL alone never proves payment.

Safe retries

API clients may include expected_endpoint_revision and expected_benchmark_version from the configuration they reviewed. If either changed, the server rejects the request instead of silently running a different configuration. Real requests require expected_collection_mode: "prepaid" to confirm credit collection. API clients may also send expected_billing_mode and expected_row_unit_price_usd; changed pricing returns 412 billing_terms_changed instead of silently accepting new terms. A null row rate asserts no per-row fee; it does not cover provider inference. Revision and rate safeguards are optional; prepaid collection confirmation is required for real paid work.

Send an Idempotency-Key header when creating an evaluation. Reuse it for a retry of the same request—not for a different release or benchmark configuration.

Cancel a run

curl -X POST "$ALLEVALS_URL/v1/evaluations/EVALUATION_ID/cancel" \
  -H "Authorization: Bearer $ALLEVALS_API_KEY"

Cancellation stops further scheduled work. An inference request already in progress may finish and may still be charged by your provider.

05Inspect progress and results

curl "$ALLEVALS_URL/v1/evaluations/EVALUATION_ID" \
  -H "Authorization: Bearer $ALLEVALS_API_KEY"

curl "$ALLEVALS_URL/v1/evaluations/EVALUATION_ID/results?limit=50" \
  -H "Authorization: Bearer $ALLEVALS_API_KEY"

Real benchmark packages first enter preparing, then queued, running, and a terminal state: completed, failed, or canceled. Successful items and failed requests are counted separately.

prepared_items measures package preparation, not model execution. Even at the total item count, preparing remains active until final integrity validation finishes. The sample hash is null until preparation completes. Keep polling, and allow cancellation in every active state.

Item results contain task and evaluation IDs, source item IDs, zero-based position, status, score, latency, token counts, timestamps, upstream HTTP status, and a fixed error code. Execution prompts, expected answers, model outputs, reasoning, and decision responses are not retained or returned. A wrong answer is not a provider error; a partial run is not a full benchmark score.

Filter the full result set with outcome=correct, incorrect, failed, pending, or canceled. Filters apply before cursor pagination. Omit the filter or use outcome=all for every item.

Scoring processes the response transiently; the saved result is metadata only. Evaluation duration_ms measures elapsed run time from started_at, including preparation and queue gaps; model_time_ms separately sums model-request latency.

Retrieve the frozen attribution at /v1/evaluations/:id/provenance. Dashboard JSON exports use allevals.evaluation-metadata.v1 and include the complete frozen notice alongside strictly selected run and task metadata. License and conversion documents are preserved; benchmark questions and execution traces are not exported. Missing or invalid attribution stops the export. Source license terms still apply to the attribution documents.

Dashboard exports are available after a run ends. Failed or canceled runs export partial evidence with no aggregate score. Browser exports stop at 20,000 items or 64 MiB and never silently truncate. For larger evidence, iterate the cursor-paginated results API or SDK and retain the complete provenance notice.

The dashboard’s export_summary distinguishes selected items, prepared rows, actual exported rows, unprepared items, and execution attempts. If preparation did not finalize, sample_indices_hash remains null and the export explicitly says final selection integrity was not established. The source provenance’s item count describes its benchmark package, not the number of exported task records.

Usage cost evidence

Read GET /v1/evaluations/:id/cost (SDK: evaluations.cost(id)) with evaluations:read. Amounts are exact USD decimal strings. Customer-owned endpoint totals include only the platform row fee; separately paid provider cost is null. For historical managed runs, amounts remain null while provider cost is pending. Distinguish reserved credit, unsettled historical provider cost, and settled wallet deductions. A known cost is not itself proof that a wallet settlement has completed. Legacy billing evidence remains historical and is not recharged to the wallet.

Correct and incorrect graded rows both count. Provider errors and unexecuted canceled rows do not incur a row fee; completed-before-cancel rows remain billable. Your provider may charge for a call even if its output cannot be graded. Real one-sample smoke checks follow these same rules; only local synthetic endpoints are exempt.

06API access you control

Create keys in API keys. The complete secret is shown once. Give each integration its own key, choose only the scopes it needs, and set an expiry when appropriate.

Every key is tied to a workspace. Workspace IDs or another user’s resource IDs do not grant access. Endpoint credentials are separate from allevals API keys.

07Predictable lists and errors

List responses return data, has_more, and next_cursor. Pass the returned cursor unchanged in the next request.

{
  "data": [],
  "has_more": false,
  "next_cursor": null
}

Errors include a stable code, a readable message, and a request ID. Validation, authentication, authorization, unavailable integrations, and provider failures are not silently converted into success.

An endpoint response with HTTP 401, 402, 403, 404, or 405 stops remaining work instead of repeating a request the endpoint has refused. A provider’s 402 payment or credit refusal is separate from an allevals plan_limit admission error; changing your allevals plan does not refill your provider account. Fix the endpoint’s access, credit, or route before starting another evaluation.

{
  "error": {
    "code": "validation_error",
    "message": "Check the request fields.",
    "request_id": "request_identifier"
  }
}

08Scores need context

A useful score identifies the benchmark and grading protocol. allevals keeps the benchmark version, evaluation configuration, connection revision, and task-level score metadata—not the execution question, answer, or model response.

• Generated multiple-choice answers are not likelihood-based MMLU scoring.
• A decision-model protocol is distinct from a chat-completion protocol.
• A sampled evaluation is not the full benchmark.
• Synthetic test fixtures verify plumbing—not model performance.
• Adapted protocols are not official leaderboard reproductions.

09Available benchmarks

The catalog and GET /v1/benchmarks list supported benchmarks with verified installed packages. Browsing is public and does not require billing setup. Execution still checks your endpoint, executor availability, and account before accepting paid work.

Each entry includes source revision, manifest identity, licenses, metrics, required capabilities, and reviewed generation and few-shot settings. GET /v1/benchmarks/:id returns the same metadata for one benchmark. Unknown facts are shown as null; catalog entries never expose sample questions, answers, or customer responses.

Add ?endpoint_type=chat_completions for chat completion models or ?endpoint_type=decisions for SystemOne decision models. Omit the filter to list all installed benchmarks. Benchmarks supporting both appear in either list.