IntentSharpby C2Consultants Read the quick start
Menu
Documentation
Table of Contents

Verification

Code: src/IntentSharp/Verification/DeterministicVerifier.cs, ModelJudge.cs, src/IntentSharp/Results/AIResult.cs.

A result succeeds only when an attempt passes deterministic checks and, if enabled, the judge. Every failure is a ResultIssue with a message that is fed back to the model on the next attempt.

Deterministic checks

DeterministicVerifier.Verify<T>(AIResponse, PromptPlan, AIRequestOptions?, int attempt) runs, in order:

  1. Refusal. AIResponse.IsRefusal adds a Provider issue and stops.
  2. Truncation. AIResponse.IsTruncated adds a Provider issue. The attempt always fails.
  3. Empty. After StripFences and trimming, empty content is a Verification issue.
  4. String targets. For T == string the stripped text is the value. No JSON checks.
  5. JSON. Parsed with comments skipped and trailing commas allowed. If parsing fails, the verifier tries to read a bare primitive (int, long, double, decimal, bool, Guid, DateTime, DateOnly, TimeSpan, enums). An enum is read by name only: a number, or a combination of names that is no member of an enum without [Flags], is not read. Otherwise a Verification issue: "The response was not valid JSON".
  6. Envelope. An object with exactly one key value is unwrapped. An object that has value together with other keys is a Verification issue that names the unexpected keys ("Reply must be a single envelope object {"value": } and nothing else; unexpected keys: note."), because read as a bare value it would lose the result: every member of a typed target at its default, or the whole object for a runtime shape. The exceptions are targets whose bare value may hold a value key itself, a dictionary or a type with its own member serialized as value (any case): such an object is read as a bare T. Any other reply is treated as the bare value.
  7. Deserialization to T. A JsonException or NotSupportedException becomes a Verification issue that names the expected type. The deserializer sets RespectNullableAnnotations and RespectRequiredConstructorParameters on every target framework (net8.0, net9.0 and net10.0), and a presence check reports non-nullable members that are missing from the JSON. Enums are read by name only (JsonStringEnumConverter with allowIntegerValues: false), so {"value": 7} fails. When T is an enum without [Flags] and a reply such as "Neutral, Negative" combines names into a value that is no member, the issue is ""3" is not a member of Sentiment; answer with exactly one of: Positive, Neutral, Negative."
  8. Null. A null payload ({"value": null} or a bare null) passes only when T is Nullable<>, such as int?. For every other T, reference types included, it is a Verification issue: "The value was null but a non-null {T} is required." A successful result therefore never carries a null Value unless T is Nullable<>, and GetValueOrThrow never returns null for a class target.
  9. Validators. Each function in AIRequestOptions.Validators runs on the parsed value. Every string it returns is a Validator issue. A validator that throws becomes "Validator threw: ...".

The outcome passes only when there are no issues other than provider issues, and the response was not truncated.

Runtime shapes

A runtime shape is a JsonNode target, usually from TransformAsync(input, intent, JsonNode outputSchema) (prompting-and-schemas.md). Steps 1 to 5 and 9 are the same as for any other target. Steps 6 to 8 differ:

  • Envelope. As in step 6, but a runtime shape has no members, so the exception never applies: an object that has value together with other keys always fails. Any other reply, an object without a value key included, is accepted as the bare value. A result whose own top level has a value property therefore has to come back inside the envelope.
  • No deserialization. The value is returned as a detached copy (DeepClone) of the parsed JSON, so the result never shares nodes with the parsed reply.
  • Null. {"value": null} and a bare null fail with "The value was null but a JSON value is required."
  • Type. When T is a subtype such as JsonObject and the value is another kind of JSON, the issue names the kind that came back.

The verifier does not check the value against the schema. Validators receive the JsonNode, and the intent catalog adds one that validates schema shapes against the full schema (intent-catalog.md).

Validators

A validator is Func<object?, IEnumerable<string>>. An empty sequence means acceptable. Add them with:

var options = new AIRequestOptions()
    .Validate<List<int>>(xs => xs.Count == 3 ? Array.Empty<string>() : new[] { "Return exactly 3 numbers." })
    .Require<List<int>>(xs => xs.All(x => x > 0), "Every number must be positive.");

Validate<T> reports "Value was not a T." when the value has a different type. The attribute route is [AIGuardrail(GuardrailType.CustomValidator, typeof(MyValidator))], where MyValidator implements IGuardrailValidator and has a parameterless constructor. The collection helpers in AIExtensions and the database runtime add their own validators the same way.

The judge

IResultJudge.JudgeAsync(PromptPlan, string candidateOutput, string? model, CancellationToken) returns a JudgeVerdict (Passes, Problems) and the raw AIResponse.

ModelJudge sends the intent, the input, the candidate output and any extra output requirements to its provider, with a strict schema for {passes, problems} and MaxOutputTokens = 1024.

  • A judge reply that is not valid JSON counts as a pass, so a broken judge cannot block results. It is logged at Warning ("The provider judge reply for intent ... was not JSON; the attempt was accepted without a verdict.") through the logger given to ModelJudge(provider, logger); the builder's default judge gets one from WithLoggerFactory. Nothing in the AIResult marks it.
  • A reply that deserializes to null fails with "Judge returned no verdict."
  • A failing verdict with no problems gets "The judge rejected the output without listing problems."

When it runs:

IntentClientOptions.JudgeMode Judge runs
Never never; the builder creates no judge
OnRequest (default) when AIRequestOptions.UseJudge is true
Always on every attempt that passed deterministic checks

IntentClientBuilder.Build creates new ModelJudge(primary) unless JudgeMode is Never or WithJudge(...) supplied one. The judge model is IntentClientOptions.JudgeModel, or the provider default. Each judge call counts toward MaxModelCalls, AIResult.Usage, IntentClient.TotalModelCalls and the client token totals. When a rate limiter is configured, the judge call first takes a token from the primary provider's bucket.

Issue sources

IssueSource Meaning
Provider The provider refused, truncated, or failed (HTTP error, timeout, transport).
Verification The output was empty, not JSON, an envelope with extra keys, did not deserialize to T (an enum value that names no member included), or, for a runtime shape, was null.
Validator A caller-supplied or built-in validator rejected the parsed value.
Judge The judge listed a problem.
Guardrail The model-call ceiling or the time budget stopped the request.

ResultIssue.Attempt is the attempt number. Issues raised when the loop ends early use the next attempt number. AIResultBase.FailureSummary joins the distinct messages; GetValueOrThrow throws an IntentSharpException with that summary.

Why there is no confidence score

Version 1 reported confidence scores that were constants or heuristics, so they carried no information (see ../context/history.md). IntentSharp reports only facts that were checked: success, concrete issues, every attempt's raw output, usage and timing. Code that needs a stronger guarantee adds a validator or turns on the judge (GuardrailType.RequireJudge on an intent method). The obsolete GuardrailType.MinConfidenceScore alias, which only switched the judge on, was removed in 1.0.0-rc.1. See ADR 0004.