Long chains of calls

An application often runs several calls in a row: it extracts the facts of a request, estimates from them, reviews the estimate and revises it. Each step is an ordinary call with its own AIResult<T>. IntentSharp limits each call; the chain, its progress and what survives a restart are your application's code. This page shows one chain of four calls, with a budget on each, a deadline on the whole, a checkpoint per step and a report of what each step cost.

The budgets of one call

Limit	On the client	On one call	Counts	When it is reached
Time budget	`DefaultTimeout`, 5 minutes	`AIRequestOptions.Timeout`	The whole call: every attempt, retry wait, fallback and judge call	`Success` false, with a `Constraint` issue
Attempts	`MaxAttempts`, 2	`AIRequestOptions.MaxAttempts`	One request and its checks	`Success` false, with each attempt's findings
Model calls	`MaxModelCalls`, 6	`AIRequestOptions.MaxModelCalls`	Attempts and judge calls	`Success` false, with a `Constraint` issue

The client's values apply to every call that sets none of its own:

// The client's limits apply to every call that sets none of its own.
var client = new IntentClientBuilder()
    .UseProviderOptions(store)
    .Configure(options =>
    {
        options.DefaultTimeout = TimeSpan.FromMinutes(2);   // one call: every attempt, retry and fallback
        options.MaxAttempts = 2;
        options.MaxModelCalls = 4;                          // attempts and judge calls
    })
    .Build();

A settings file that sets client.timeout, client.maxAttempts or client.maxModelCalls wins over both (Settings reference). Each limit counts one call, not the chain: to bound a chain's spend, add up its calls' Usage and stop when the total passes your limit.

A chain, step by step

/// <summary>
/// Four calls in a row: the facts of a job request, an estimate from them, a review of the estimate, and a revision when
/// the review found problems. Each call has its own budget, the caller's token stops the chain, every finished stage is
/// saved, and each stage reports what it cost.
/// </summary>
public sealed class EstimateChain(IntentClient client, IChainCheckpoints checkpoints, IProgress<StageReport> progress, TimeProvider clock)
{
    public async Task<HoursEstimate?> RunAsync(JobRequest job, CancellationToken ct)
    {
        JobFacts? facts = await StageAsync(job, "extract", () => job.Description.AIAsync<JobFacts>(client,
            "List the facts this job request states, one sentence each, and what it leaves out.", Budget(30), ct), ct);
        if (facts is null) return null;

        HoursEstimate? estimate = await StageAsync(job, "estimate", () => facts.AIAsync<HoursEstimate>(client,
            "Estimate the labor hours from these facts, and name what each part of the estimate rests on.", Budget(60), ct), ct);
        if (estimate is null) return null;

        EstimateReview? review = await StageAsync(job, "review", () => new { facts, estimate }.AIAsync<EstimateReview>(client,
            "Check this estimate against the facts. List every problem, and accept it only when there is none.", Budget(30), ct), ct);
        if (review is null) return null;
        if (review.Accepted) return estimate;

        return await StageAsync(job, "revise", () => new { facts, estimate, review.Problems }.AIAsync<HoursEstimate>(client,
            "Revise this estimate so that it answers every problem the review found.", Budget(60), ct), ct);
    }

    // Each call's own budget: the time for all of its attempts, the attempts, and the model calls.
    private static AIRequestOptions Budget(int seconds) => new()
    {
        Timeout = TimeSpan.FromSeconds(seconds),
        MaxAttempts = 2,
        MaxModelCalls = 2,
    };

    private async Task<T?> StageAsync<T>(JobRequest job, string stage, Func<Task<AIResult<T>>> call, CancellationToken ct) where T : class
    {
        // A stage that an earlier run finished is read back, with no model call.
        if (await checkpoints.LoadAsync<T>(job.JobId, stage, ct) is { } saved)
        {
            progress.Report(new StageReport(stage, true, Resumed: true, 0, TokenUsage.Zero, TimeSpan.Zero, null));
            return saved;
        }

        long started = clock.GetTimestamp();
        AIResult<T> result = await call();   // once the caller's token is cancelled, this throws OperationCanceledException
        progress.Report(new StageReport(stage, result.Success, Resumed: false, result.Attempts.Count, result.Usage,
            clock.GetElapsedTime(started), result.FailureSummary));
        if (!result.Success) return null;

        await checkpoints.SaveAsync(job.JobId, stage, result.Value!, ct);
        return result.Value;
    }
}

Each step's input is what the step before it returned, so a later step sees only what an earlier one passed on.
Budget gives each call its own time and limits. A step that needs longer gets a longer budget of its own.
A step whose result fails stops the chain, and its report says why. IntentSharp does not run a failed step again; your code decides whether to.

What each step cost

/// <summary>What one stage of a chain did, read from its result and timed on the application's clock.</summary>
public sealed record StageReport(string Stage, bool Success, bool Resumed, int Attempts, TokenUsage Usage, TimeSpan Elapsed, string? Problem);

Every result carries Attempts, one record per attempt that finished its checks, with its provider, model, tokens, time and issues; Usage, the tokens of the call's model calls, judge calls included; and Elapsed, the call's time. The chain above times each step on a TimeProvider of its own, so a test can move the clock. In production pass TimeProvider.System, or read Elapsed. The client's TotalUsage and TotalModelCalls add up every call since it was built.

Progress and checkpoints are yours

The chain reports each step to an IProgress<StageReport>: a Progress<StageReport> that updates a page or writes a log line, or a list in a test. IntentSharp keeps no record of where a chain is between calls, and a call that stops is not resumed. To survive a restart, save each step's result where your application keeps its state, and read it back before calling the model again:

/// <summary>Where a chain keeps each stage it finished, so that a run that stops can resume: a table, a blob or a file of yours.</summary>
public interface IChainCheckpoints
{
    ValueTask<T?> LoadAsync<T>(string jobId, string stage, CancellationToken ct) where T : class;

    ValueTask SaveAsync<T>(string jobId, string stage, T value, CancellationToken ct) where T : class;
}

/// <summary>Checkpoints as JSON files in a folder, one per job and stage.</summary>
public sealed class FolderCheckpoints(string folder) : IChainCheckpoints
{
    public async ValueTask<T?> LoadAsync<T>(string jobId, string stage, CancellationToken ct) where T : class
    {
        var path = PathOf(jobId, stage);
        if (!File.Exists(path)) return null;
        await using var file = File.OpenRead(path);
        return await JsonSerializer.DeserializeAsync<T>(file, cancellationToken: ct);
    }

    public async ValueTask SaveAsync<T>(string jobId, string stage, T value, CancellationToken ct) where T : class
    {
        Directory.CreateDirectory(folder);
        var path = PathOf(jobId, stage);
        await File.WriteAllTextAsync(path + ".tmp", JsonSerializer.Serialize(value), ct);
        File.Move(path + ".tmp", path, overwrite: true);   // a stage is saved whole, or not at all
    }

    private string PathOf(string jobId, string stage) => Path.Combine(folder, $"{jobId}.{stage}.json");
}

A deadline for the whole chain

// The chain's own deadline. Each call keeps its own budget inside it.
using var deadline = new CancellationTokenSource(limit);
try
{
    return await chain.RunAsync(job, deadline.Token);
}
catch (OperationCanceledException) when (deadline.IsCancellationRequested)
{
    // Every stage that finished is saved: the next run of this job starts at the stage that was running.
    return null;
}

Your token is the chain's deadline. When it fires, the call that is running throws OperationCanceledException, and the steps saved before it stay saved. A call's own time budget is different: when it runs out, the call returns a failed result with a Constraint issue and throws nothing (Testing has a test of each).

A run on a scripted model

The chain, run on ScriptedProvider. The scripted model moves the test clock forward by a fixed time per call: 4, 9, 3 and 8 seconds. So the timings below are the same on every run; a live model's are its own. Then the same job when the review's provider fails once with a 503, and the next run, which reads the first two steps back; and a run whose two-second deadline passes while the estimate is still waiting for its model:

A run of job J-1042, on a scripted model:
  extract   passed  1 attempt    333 tokens   4.0 s
  estimate  passed  1 attempt    335 tokens   9.0 s
  review    passed  1 attempt    390 tokens   3.0 s
  revise    passed  1 attempt    413 tokens   8.0 s
  The chain: 4 stages, 4 model calls, 1471 tokens, 24.0 s. Estimate: 9 hours.

The same job when the review's provider fails on the first run:
  Run 1:
    extract   passed  1 attempt    333 tokens   4.0 s
    estimate  passed  1 attempt    335 tokens   9.0 s
    review    failed  0 attempts     0 tokens   0.0 s  Service unavailable.
  Run 2:
    extract   read back from its checkpoint, no model call
    estimate  read back from its checkpoint, no model call
    review    passed  1 attempt    390 tokens   3.0 s
    revise    passed  1 attempt    413 tokens   8.0 s
  Model calls in run 2: 2

A run whose deadline passes while the estimate is running:
  extract   passed  1 attempt    333 tokens   4.0 s
  Stopped during estimate: the chain's token was cancelled, and RunAsync threw OperationCanceledException.
  Saved: J-1042.extract.json

Limits

Retries happen inside one call, within its limits. Nothing retries a whole step or a whole chain.
A call that your token stops returns no result. Save what you need from each step as soon as it finishes.
Elapsed and the time budget are measured on the machine's clock. A test clock of your own times only your code.
A cached answer adds no tokens, but a judge, when on, still makes its call (Memory, cache and rate limiting).

Table of Contents