Compiled mappings

A compiled mapping turns records of one shape into records of another. The model compiles it once, from your intent and the two shapes, into a list of steps from a fixed set of operations. IntentSharp then runs those steps on every later input with the same shapes and intent, in your process and with no model call, and checks each output against your type and your validators. A row the steps cannot map fails with a category: the steps never guess. Classifying, extracting and summarizing free text still call the model every time.

A mapping store

A client that maps keeps its compiled mappings in a mapping store, under a namespace:

// builder: an IntentClientBuilder with your providers. The store keeps each compiled mapping and its revisions,
// under the namespace "customers".
var client = builder
    .WithMappingStore(new InMemoryMappingStore(), "customers")
    .Build();

Under dependency injection, services.AddMappingStore(store, "customers"). On a client with no mapping store, a mapping call throws InvalidOperationException, naming WithMappingStore.

Map by shape

MapByShapeAsync<TIn, TOut> maps one object. Its JSON form maps a JsonObject to the shape a JSON schema describes, with MappingOptions.SourceSchema describing the input. MapManyByShapeAsync maps a collection: the whole batch runs on one revision, so a batch never mixes two. When it compiles a replacement, every row runs again on it, so no output comes from the revision it replaced.

const string intent = "Map the legacy customer to the customer model. Since is dd/MM/yyyy, or yyyy-MM-dd in newer rows. " +
                      "Balance is a decimal with a point. Tier G is Gold, S is Silver, B is Bronze.";
var options = new MappingOptions();
options.SourceAllowedValues["/tier"] = ["G", "S", "B"];   // a member mapped through a table needs a declared list
options.SourceFormats["/balance"] = new SourceFormat { DecimalSeparator = "." };
// Every compile, and every replacement, must give exactly this output for this input.
options.AddSample(
    new LegacyCustomer { Id = "G-7", Name = "Grace Lin", Since = "28/02/2018", Balance = "1999.50", Tier = "S" },
    new Customer { Id = "G-7", Name = "Grace Lin", Since = new DateOnly(2018, 2, 28), Balance = 1999.50m, Tier = CustomerTier.Silver });

var customers = new List<AIResult<Customer>>();
foreach (LegacyCustomer row in rows)
{
    AIResult<Customer> result = await client.MapByShapeAsync<LegacyCustomer, Customer>(row, intent, options);
    // Compiled: this call asked the model for the mapping. Otherwise a stored mapping ran, with no model call.
    Console.WriteLine($"{row.Id}: {(!result.Success ? "failed" : result.Compiled ? "compiled" : "stored mapping")}");
    customers.Add(result);
}

On five scripted rows, where the first compile's first answer copies the name into the id:

C-1001: compiled, after 2 attempts
  Attempt 1: Golden sample 1: the output differs from the expected output at /id. Reproduce every example and golden sample exactly.
  C-1001: Ada Moreno, since 2019-04-03, balance 1250.50, Gold
C-1002: the stored mapping, no model call
  C-1002: Ben Ortiz, since 2021-11-17, balance 80.00, Silver
C-1003: the stored mapping, no model call
  C-1003: Chen Wu, since 2024-02-01, balance 0.00, Bronze
C-1004, since 2025-06-30: did not parse in the stored formats, so a replacement was compiled
  C-1004: Dana Fox, since 2025-06-30, balance 410.75, Silver
C-1005, tier P: failed, no model call
  [Mapping] attempt 1: Step 5 lookup at /tier: a value shaped A is not in the declared set of /tier. Add the value to MappingOptions.SourceAllowedValues, which compiles the mapping again.
Model calls for 5 rows: 3
Compiled by main (scripted-1) in 2 attempt(s)
Revision 1: Retired (ParseFailed at /since)
  Copy /id from /id
  Copy /name from /name
  ParseDate /since from /since
  ParseDecimal /balance from /balance
  Lookup /tier from /tier
Revision 2: Active
  Copy /id from /id
  Copy /name from /name
  ParseDate /since from /since
  ParseDecimal /balance from /balance
  Lookup /tier from /tier

Compiled is true on the call that asked the model. A later call with the same shapes and intent runs the stored mapping: its result has FromMapping true, no provider, no model and no tokens.
The mapping's key covers the two shapes, the intent, the declarations, the golden samples, KeyMaterial, the request's expected output description, system instructions and examples, the converters you register, and the provider and model unless ShareAcrossModels is true. A change of any of them compiles the mapping again. The request's validators are not in the key: they run on every output.

The operations

A mapping is written in a closed set of operations: copy, constant, coalesce, defaultIfNull, nullIf, trim, upper, lower, toString, concat, split, take, parseInt, parseLong, parseDecimal, parseDouble, parseBool, parseDate, parseDateTime, formatDate, formatNumber, scale, round, lookup, switch, whenEquals, each, add, multiply, sum, count, and convert, which calls a converter you register with WithMappingConverters. No operation evaluates an expression, runs a regular expression or reads meaning from text, and a stored mapping that names an operation the library does not know is refused.

What you declare bounds what a step may do:

A member mapped through a table needs a closed set: an enum, a boolean, or a list in SourceAllowedValues. A value outside a declared list fails with NotInSet and compiles nothing.
A date step reads exact formats, which the intent or SourceFormats states. SourceFormats also declares number separators, a time zone, and fixed formats with their masks.
A string a step writes as a constant must appear in the intent, the expected output description, the system instructions, a declared list or the target enum.

Golden samples

GoldenSamples, or AddSample, are inputs with the outputs a mapping must give for them. Every compile and every replacement must reproduce each one exactly. An answer that does not is sent back to the model with the finding, which names the sample and the path and never a value, as in the first row above.

A key's first compile sends the triggering input to the model, and no edit writer sees that compile, so give a mapping fed by untrusted data golden samples. CrossCheck checks a compile with one direct answer from the model on the triggering input, one more model call: it is the same model checking itself, not independent proof. RequireDistinctSamples refuses a compile whose samples cannot tell two members of one type apart.

When a stored mapping fails a row

OnReuseFailure says what happens when a step of the stored mapping fails on an input. With Recompile, the default, a value that does not parse in the stored formats compiles one replacement, as the row with an ISO date above did; any other failure, such as the tier outside the declared list, fails the call. With Fail, the call fails with the step failures and makes no model call. Replacements are capped by MaxRecompilesPerHour, 3 per key an hour unless set, shared by every process on the store; at the cap, the call fails and the stored mapping keeps serving.

A failure has a category: ParseFailed, LookupMiss, NotInSet, MaskMismatch, NullRequired, Overflow, ConverterFailed, ZoneUnresolved, StoredMappingRefused, or Other. Its message names the step, the operation and the output path, and describes a value only by its mask: digits as 9, letters as A, other characters as they are.

Recompile = true on a call compiles a replacement after a person reviewed the reason, and MappingTtl reuses a stored mapping for a limited time.

The store and its revisions

InMemoryMappingStore ships with the core. A store of your own implements IMappingStore, every member of it, TryTakeRecompileAsync included, which caps replacements across processes, and keeps the revisions where your application keeps its data. Each key has a sequence of revisions: one is active, and a replaced one is retired with a reason.

When the client has a document store, a replacement goes through the edit seam as an edit of kind Mapping, so a writer of your own can hold it as a revision that is not active, which runs once the store's ActivateAsync makes it active. Without a document store, a replacement is stored directly. A key's first compile is stored directly either way.

The store is a trust boundary. Before a stored revision runs, IntentSharp checks its namespace, key, status, versions, shapes, operations, converters and the fit of its steps to the shapes, but not that the steps do what the intent says, and it does not run the golden samples again. Limit write access to the store.

Provenance

// Which provider and model compiled the mapping a result came from. Compile time only: a stored mapping that runs
// on a row sends it to no model.
if (result.CompiledBy is { } by)
    Console.WriteLine($"Compiled by {by.Provider} ({by.Model}) in {by.Attempts} attempt(s)");

// The revisions in the store, and the steps of each, from the closed set of operations.
foreach (CompiledMapping mapping in await store.ListAsync("customers"))
{
    Console.WriteLine($"Revision {mapping.Revision}: {mapping.Status}{(mapping.RetiredReason is { Category: { } category } why ? $" ({category} at {why.Path})" : "")}");
    foreach (MappingStep step in mapping.Steps)
        Console.WriteLine($"  {step.Op} {step.Target} from {string.Join(", ", step.Sources)}");
}

CompiledBy records how a mapping was compiled: the provider and the model, the attempts, the tokens, when the compile finished and, when the providers come from a provider file, that file's version and content hash. It is compile-time provenance only: a stored mapping that runs on a row sends the row to no model. A mapping compiled under another version of the provider file is due for one replacement, within the cap. The end of the output above lists the two revisions and their steps.

The Reply checkpoint

A mapping's value is returned as a model's reply is. On a client with guardrails, every value a mapping call returns goes through the Reply checkpoint under the client's default sets and the GuardrailSets of MappingOptions.Request, whether the mapping was reused or compiled on the call. reply.text is the output's JSON text; reply.value and reply.strings are the value's. A value not allowed fails the result with a Guardrail issue that names the rule, and the call is not retried, since the stored mapping is not at fault. In a batch, the first output not allowed fails the batch.

intentsharp: rules/1
set: customers-out
checkpoint: reply
rules:
  - id: plain-text
    subject: reply.strings
    pattern: '^[^<>]*$'
    message: Write plain text, with no markup.

The setting mapping.replyCheck chooses what the checkpoint does with a mapping's values:

Value	What a mapping call does
`check`	The default. Every value is checked, and a value not allowed fails the call with no compile.
`recompileOnce`	As `check`, but a value enforced rules do not allow compiles the mapping again once, within `MaxRecompilesPerHour`. The new mapping replaces the active revision only when the checkpoint allows its values; otherwise it is dropped and the call fails. With `OnReuseFailure` set to `Fail`, nothing is compiled.
`skip`	No value is checked, and the result's warnings say so and where `skip` was set.

It is a security setting, read under the request's sets, then the settings file, then code. When a set or the settings file cannot be read, it is check. A file that sets skip is flagged as widening; recompileOnce is not, since every value it returns is checked. When several sets give it, check wins over recompileOnce, and recompileOnce over skip.

// customers-out.rules.yaml checks every value a mapping returns, reused or compiled. mapping.replyCheck set in
// code applies where no rule set and no settings file sets it.
var client = builder
    .WithMappingStore(mappings, "customers")
    .WithGuardrailFiles(store, o => o.CodeSettings["mapping.replyCheck"] = "recompileOnce", "customers-out")
    .Build();

A row with markup in its name, under customers-out, with each value of the setting:

A name with markup, with mapping.replyCheck check (the default):
  Success: false
  [Guardrail] attempt 1: The mapping's output was not allowed at the Reply checkpoint: Write plain text, with no markup (rule plain-text in guardrail set "customers-out").
  Model calls for this row: 0
The same row, with recompileOnce:
  Success: false
  Model calls: 1
  Warning: The Reply checkpoint did not allow the output of mapping 29fbb21e39d2 revision 1 (Write plain text, with no markup (rule plain-text in guardrail set "customers-out")), so with mapping.replyCheck set to recompileOnce a new mapping was compiled, and the checkpoint did not allow its output either; it was not stored, so revision 1 was not replaced.
The same row, with skip:
  Success: true
  Warning: The mapping's output was not checked at the Reply checkpoint: mapping.replyCheck is skip, set in code.

A new compile under recompileOnce is shown the refused input as masks only, with each rule's message, id and set.

Drift and the shadow check

Drift is a difference between what a stored mapping expects and what the input now holds. It is reported once per member and kind for a call or a batch, never once per row, and it changes no value. mapping.driftWarnings set to oncePerRevision adds the warning to the first call that measures it on a revision instead of every call. A drift raises a MappingShape change once per revision (Change detection), which no shipped handler acts on.

ShadowCheckEvery has the model answer every Nth reuse directly as well, and records any difference with the stored mapping's answer; the stored mapping's value is the one returned. The shadow check is experimental, INTENTSHARP002.

Limits

A mapping does not rethink each row. A field whose meaning changes while its shape stays the same is not detected unless a drift warning or a shadow check catches it, and a pass means the checks passed, not that the mapping is right.
A key's first compile sends the triggering input to the model and goes to no edit writer.
The stored revisions are trusted application data, and they are not signed.
A converter is your code: a mapping that calls one is as deterministic as the converter.
Time zone rules come from the host's time zone data, which differs by operating system and version.

Table of Contents