Execute LLM Judge

Records the submission and queues it. Returns immediately; the judgement itself runs asynchronously and is retrieved with GET /v2/judge/llm/{evaluation_id}. Unlike POST /v2/judge/agent (Agent Judge), which returns a written assessment, this endpoint produces a structured verdict — dimension scores plus reasoning — on a fixed rubric (Factuality / Completeness / Relevance / Grounding, each 1-5) with a server-computed overall_score. Every run uses the same rubric so runs are comparable across callers and time.

POST /v2/judge/llm

Submit an LLM Judge evaluation

cURL

curl --request POST \
  --url https://api.eu.linqalpha.com/v2/judge/llm \
  --header 'Content-Type: application/json' \
  --header 'X-API-KEY: <api-key>' \
  --data '
{
  "answer": "<string>",
  "query": "<string>",
  "time_window": {
    "end_time": "2026-08-19T05:32:11Z",
    "start_time": "2025-01-01T00:00:00Z"
  },
  "prompt": "<string>",
  "references": [],
  "search_type": "external"
}
'

202

{
  "error": null,
  "payload": {
    "evaluation_id": "2fde560a-e7eb-45e0-8cd3-04e57c50d1d3",
    "status": "pending"
  }
}

What it does

Submits an answer plus your judge prompt and the sources you want it graded against, and returns an evaluation_id immediately. The judge verifies the sources within the selected source scope, then a deterministic LLM grades the answer on a fixed rubric — Factuality / Completeness / Relevance / Grounding, each 1–5 — and returns a structured verdict (dimension scores plus a short reasoning, with a server-computed overall_score mean).

Unlike Execute Agent Judge, which returns a written assessment against primary sources, this endpoint returns numeric scores you can compare across runs. The rubric is the same for every run so scores are comparable across callers and across time.

That still takes minutes, not seconds, so this endpoint does not return the verdict. 202 means the submission is recorded and will be evaluated — never that it has been.

POST /v2/judge/llm                  ->  202  { evaluation_id, status: "pending" }
                                         |
                                         |  the judge runs on our side
                                         v
GET  /v2/judge/llm/{evaluation_id}  ->  the verdict, once it settles

status is pending for a new submission, but read it rather than assuming it. An Idempotency-Key retry returns the evaluation that key already names — which may have finished in the meantime — so it can come back completed or failed. The value set is the same one Get LLM Judge Evaluation returns.

Retrieve the result with Get LLM Judge Evaluation.

Example

import requests

resp = requests.post(
    "https://api.eu.linqalpha.com/v2/judge/llm",
    headers={
        "X-API-KEY": "<your-api-key>",
        "Content-Type": "application/json",
        # Optional but recommended: makes a retry safe.
        "Idempotency-Key": "run-2026-08-25-0001",
    },
    json={
        # OPTIONAL. Your framing for the judge — delivered verbatim as the system message
        # when supplied. The scoring dimensions themselves are fixed by the endpoint; use
        # `prompt` to steer emphasis (e.g. "penalise unhedged numeric claims") rather than
        # to change the rubric structure. Omit the field entirely to fall back to Linq's
        # default judge prompt.
        "prompt": (
            "You are grading whether the ANSWER is supported by the SOURCES and by the "
            "verification verdicts. Score each dimension in the fixed rubric strictly, "
            "and justify the scores in a short reasoning."
        ),
        "query": "How did Apple perform in Q3 FY2024?",
        "answer": (
            "Apple reported record Services revenue in Q3 FY2024 while iPhone revenue "
            "slipped year-over-year."
        ),
        # Inclusive search range. `start_time` is optional; `end_time` is required.
        "time_window": {
            "start_time": "2024-07-01T00:00:00Z",
            "end_time": "2026-08-19T05:32:11Z",
        },
        # Optional. `external` is the default; use `rms` for RMS-only answers and
        # `all` when the answer combines RMS and external sources.
        "search_type": "external",
        # Structured references match the agent-judge payload shape. `content` is what
        # the final judge source-grounds against.
        "references": [
            {
                "title": "Apple Q3 FY2024 Press Release",
                "content": (
                    "Apple today announced financial results for its fiscal 2024 third "
                    "quarter. Services revenue reached an all-time high."
                ),
                # Optional locator for the source the excerpt came from.
                "url": "https://www.apple.com/newsroom/2024/08/apple-reports-third-quarter-results/",
                "metadata": {
                    "published_at": "2024-08-01T13:30:00Z",
                    "source_type": "press_release",
                },
            },
            {
                "title": "Apple Investor Relations",
                "content": (
                    "iPhone revenue was down slightly year-over-year while Services set "
                    "a new record."
                ),
            },
        ],
    },
)

evaluation_id = resp.json()["payload"]["evaluation_id"]

The fields

These are the fields this endpoint reads; anything else in the body is rejected.

Built for organization-bound API keys. The organization and user are taken from your key. The evaluation is filed under that organization, and RMS verification uses that user’s document visibility.

Platform-wide keys are not a supported configuration for this endpoint — it has not been designed or tested against them, and the behaviour you get is whatever the shared authentication layer does rather than something this endpoint guarantees. If you hold a platform key, talk to your LinqAlpha contact before integrating.

Field	Required	Notes
prompt	No	Your framing for the judge. Delivered verbatim as the system message when supplied — steers emphasis; does not change the scoring dimensions. Omit the field to fall back to Linq’s default judge prompt. If supplied it must contain non-whitespace text; an empty string is rejected.
query	Yes	The original user prompt that produced the answer. Lets the judge check whether the answer followed the instructions.
answer	Yes	The text being judged. Must contain non-whitespace text.
time_window	Yes	Inclusive search window. end_time is required and start_time is optional. Both use RFC 3339 timestamps with a timezone offset. start_time must be earlier than or equal to end_time; a future end_time is rejected.
references	No	Defaults to []. Max 50 items. Each element is { "title"?, "content", "url"?, "metadata"? }. content is required; the rest are optional. url must be HTTP(S). metadata accepts any JSON object, including nested objects and arrays. A bare URL string and undeclared reference-level fields are not accepted.
search_type	No	Verifier source scope: rms, external, or all. Defaults to external; use all when the answer combines RMS and external sources.

Fixed rubric. Every LLM Judge run is scored on the same four dimensions (Factuality / Completeness / Relevance / Grounding, each 1–5), with a server-computed overall_score (arithmetic mean). Previously supported scoring_rubric and response_schema fields were removed so that scores from any two runs are directly comparable; sending either now returns a 400.

time_window controls the selected search scope

start_time and end_time are inclusive bounds applied to the selected search_type. When search_type is all, the same window applies to both RMS and external search. Omit start_time to search everything up to and including end_time.

The offset is required, and that is deliberate. 2026-08-19T14:32:11 without one is ambiguous, and reading it as UTC would move a Seoul timestamp nine hours. The result of that is not an error you would see: it is a plausible verdict judged against the wrong instant. Send Z or your own offset — both name the same instant and both are accepted.

start_time must be earlier than or equal to end_time. A future end_time is rejected; a few minutes of clock skew is tolerated.

references carry the excerpt, and optionally a link

Each element is { "title"?, "content", "url"?, "metadata"? }. The reference object only accepts these four fields; any other undeclared field is rejected with 400.

content is the passage itself. The verifier checks the answer against that text, so a link alone would give it nothing to read; paste the passage you are relying on.

url is an optional HTTP(S) locator for that passage’s source. It is a pointer used to inspect the original, not evidence — neither the URL nor its domain is trusted as proof that the excerpt, publisher, or date is correct.

metadata is an optional free-form JSON object. It can contain source dates such as published_at, identifiers, tags, and arbitrary nested JSON without a predefined field list. The metadata value itself must be an object.

How the judgement is produced

1. Verify references

   The verifier runs each reference against Linq’s tool set for the selected search_type, producing a per-reference verdict (supported / contradicted / unresolved) with evidence. When references: [] this step is skipped and no cost is incurred here.

2. Grade with the fixed rubric

   A deterministic model (temperature=0, seed=7) reads your prompt, the query, answer, references, and verification verdicts, and returns integer scores for each of the four fixed dimensions plus a short reasoning. The server then computes overall_score as the arithmetic mean of the four scores.

Partial verification is preserved, not hidden. If a verifier chunk failed, only the surviving verdicts reach the judge, and the response reports verified_reference_count less than input_reference_count so you can tell — see Get LLM Judge Evaluation.

Idempotency

Send an Idempotency-Key header to make retries safe. Within your organization:

• Same key, same body → the original evaluation_id, no second judge run.
• Same key, different body → 409 Conflict.

Use it whenever a network error leaves you unsure whether a submission landed. Without it, a retry starts a second run and you are billed for both.

Idempotency is scoped per endpoint. Using the same key on Execute Agent Judge and this endpoint returns two distinct evaluations — the two endpoints do not share the key space.

A retry still answers 202, but the status it carries is the original evaluation’s current status — not necessarily pending. If that evaluation already finished, you get completed or failed straight from the retry and there is nothing left to poll.

Limits

	Max
prompt	50,000 UTF-16 code units
answer	200,000 UTF-16 code units
query	10,000 UTF-16 code units
references	50 items
Whole body, serialized	1,000,000 bytes (UTF-8)
Whole body, tokenized	~100,000 tokens

Oversized submissions are rejected with 400 at submission time — nothing is queued and nothing is billed, so a request that is too large costs only the round trip.

Two of these are easy to trip without noticing.

The character counts are UTF-16 code units, which is what "…".length returns in JavaScript. Characters outside the Basic Multilingual Plane — emoji, some rarer CJK — count as two. If your text is plain prose the distinction never comes up.

The token cap is separate from the byte cap, and applies to the request as a whole. Dense CJK text can pass 200,000 code units and still exceed 100,000 tokens, so a long Korean or Japanese answer may be refused while a longer English one is not.

These may be raised as we see real usage. A raise never breaks a client that was within the old figure, so code against them as minimums. If you are running close to one, tell us rather than splitting a submission.

Authorizations

X-API-KEY · string · header · required

Headers

Idempotency-Key · string

Makes a retry safe. Same key + same body returns the original evaluation_id; same key + different body is a 409. Scope is per-endpoint — the same key on POST /v2/judge/agent yields a distinct evaluation.

Maximum string length: 255

Body

application/json

Body for POST /v2/judge/llm. Unknown top-level fields are rejected (public-api zod is .strict()). The previously supported scoring_rubric and response_schema fields were removed to keep every run comparable on the fixed default rubric; a caller still sending either now gets a 400. Whole serialized body must be at most 1,000,000 bytes (UTF-8).

answer · string · required

The text being judged.

Required string length: 1 - 200000

query · string · required

The original user prompt that produced the answer. Lets the judge check whether the answer followed the instructions.

Required string length: 1 - 10000

time_window · object · required

Inclusive bounds applied to the selected search_type. For all, the same window applies to both RMS and external search. A bare date, missing timezone offset, or nonexistent instant is rejected.

Show child attributes

prompt · string

Optional. Delivered verbatim as the judge's system message when supplied. Omit the field to fall back to Linq's default judge prompt. If supplied it must contain non-whitespace text; an empty string is rejected.

Required string length: 1 - 50000

references · object[]

Structured references the caller wants verified. At most 50 items. Each element is { title?, content, url?, metadata? }; content is required, while title, an HTTP(S) url, and free-form JSON-object metadata are optional. Bare URL strings and undeclared reference-level fields are not accepted. references: [] is allowed — the verifier is skipped and the judge grades on prompt / query / answer only.

Maximum array length: 50

Show child attributes

search_type · enum<string> · default: external

Verifier source scope: rms, external, or all. Defaults to external; use all when the answer combines RMS and external sources.

Available options: rms, external, all

Response

202

application/json

Accepted and durably recorded. Not evaluated yet — unless this was an Idempotency-Key retry naming an evaluation that has since finished, in which case status reports that evaluation's current state. Read status; do not assume pending.

error · unknown

payload · object

Show child attributes