IntentSharpby C2Consultants Read the quick start
Menu
Documentation
Table of Contents

Testing

ScriptedProvider, in the IntentSharp.Testing namespace of the IntentSharp package, is a provider that answers with replies you script, inside your process. A client built on it needs no model, no network and no API key, and costs nothing to call. Each call still goes through the same checks, retries and limits as a call to a real model, so a test sees what your code would see for that reply.

The examples on this page are xUnit tests. They use the namespaces IntentSharp, IntentSharp.Extensions, IntentSharp.Providers, IntentSharp.Results and IntentSharp.Testing, IntentSharp.Catalog for intent files, and IntentSharp.DependencyInjection with Microsoft.Extensions.DependencyInjection for the test of the service registration. Any test framework works the same way.

A client on scripted replies

private static IntentClient ScriptedClient(ScriptedProvider provider)
    => new IntentClientBuilder()
        .UseProvider(provider)
        .WithMemory(null)
        .Build();

WithMemory(null) turns off the memory, which would otherwise record every request, so that tests do not share state through it. Without a license key a test client works as any other client does; Your license key covers setting it on build servers.

Scripting a reply

Script each reply the way a model answers: plain text for a string result, and for any other result a JSON object whose value property holds the result, such as {"value": 4}. The provider hands out the replies in order, one per model call.

[Fact]
public async Task Urgency_is_read_from_the_reply()
{
    var provider = new ScriptedProvider("""{"value": 4}""");
    using var client = ScriptedClient(provider);

    var result = await "The line stops on Thursday without the belts.".AIAsync<int>(client,
        "Rate the urgency of this order email from 1 (routine) to 5 (critical).");

    Assert.True(result.Success);
    Assert.Equal(4, result.Value);
    Assert.Equal(1, provider.CallCount);
}

When the replies run out, the call throws InvalidOperationException. In a test, that means the code made more model calls than you scripted. To compute each reply from the request instead, pass a method: new ScriptedProvider(request => ...).

Retries and failed results

A reply that fails a check is retried with the finding, as with a real model, so script one reply per attempt:

[Fact]
public async Task A_reply_that_breaks_a_rule_is_retried_with_the_finding()
{
    var provider = new ScriptedProvider("""{"value": 9}""", """{"value": 5}""");
    using var client = ScriptedClient(provider);
    var options = new AIRequestOptions()
        .Require<int>(u => u is >= 1 and <= 5, "Urgency must be between 1 and 5.");

    var result = await "Ship today.".AIAsync<int>(client, "Rate the urgency from 1 to 5.", options);

    Assert.True(result.Success);
    Assert.Equal(5, result.Value);
    Assert.Equal(2, result.Attempts.Count);
    var issue = Assert.Single(result.Issues);
    Assert.Equal(IssueSource.Validator, issue.Source);
    Assert.Equal("Urgency must be between 1 and 5.", issue.Message);
}

Script replies that never pass to test the path your code takes when a call fails:

[Fact]
public async Task The_caller_handles_a_result_that_never_passes()
{
    var provider = new ScriptedProvider("""{"value": 9}""", """{"value": 9}""");
    using var client = ScriptedClient(provider);
    var options = new AIRequestOptions()
        .Require<int>(u => u is >= 1 and <= 5, "Urgency must be between 1 and 5.");

    var result = await "Ship today.".AIAsync<int>(client, "Rate the urgency from 1 to 5.", options);

    Assert.False(result.Success);
    Assert.Equal(2, result.Attempts.Count);   // MaxAttempts is 2 by default
    Assert.Equal(0, result.Value);
}

Provider errors and the fallback

To test a provider error, throw one of IntentSharp's provider exceptions from the method that computes the reply: AIRateLimitException (429), AIProviderInternalException (5xx), AIAuthenticationException (401 or 403) or AIValidationException (400 or 422). The call does not throw: the result fails with a provider issue.

[Fact]
public async Task A_provider_error_is_an_issue_not_an_exception()
{
    var provider = new ScriptedProvider(_ => throw new AIRateLimitException("Rate limit exceeded.", providerName: "Scripted"));
    using var client = ScriptedClient(provider);

    var result = await "Ship today.".AIAsync<int>(client, "Rate the urgency from 1 to 5.");

    Assert.False(result.Success);
    Assert.Equal(IssueSource.Provider, Assert.Single(result.Issues).Source);
}

A transient error on the primary (429, 5xx, a transport error or a timeout) sends the request to the fallback provider. Give the second ScriptedProvider its own name, since the two must differ:

[Fact]
public async Task A_transient_error_on_the_primary_falls_back()
{
    var primary = new ScriptedProvider(_ => throw new AIProviderInternalException("Service unavailable.", 503, "Scripted"));
    var backup = new ScriptedProvider("""{"value": 3}""") { Name = "Backup" };
    using var client = new IntentClientBuilder()
        .UseProvider(primary)
        .UseFallbackProvider(backup)
        .WithMemory(null)
        .Build();

    var result = await "Ship this week.".AIAsync<int>(client, "Rate the urgency from 1 to 5.");

    Assert.True(result.Success);
    Assert.Equal("Backup", result.Provider);
}

ScriptedProvider never reports a refused reply or one cut off at the token limit. To test those, write a small IAIProvider of your own that sets IsRefusal or IsTruncated on the AIResponse it returns.

What was sent

provider.Requests holds every request the provider received, in order: the messages, the model, the output token ceiling, the temperature, the reply format and schema, and the tags. Assert on what your code controls (your input, your options and your tags) rather than on the full text of the messages, whose wording is not part of the public API.

[Fact]
public async Task The_input_and_the_tags_are_on_the_request()
{
    var provider = new ScriptedProvider("""{"value": 2}""");
    using var client = ScriptedClient(provider);
    var options = new AIRequestOptions { MaxOutputTokens = 200 };
    options.Tags["tenant"] = "northwind";

    await "Restock when convenient.".AIAsync<int>(client, "Rate the urgency from 1 to 5.", options);

    var request = Assert.Single(provider.Requests);
    Assert.Contains(request.Messages, m => m.Content.Contains("Restock when convenient."));
    Assert.Equal(200, request.MaxOutputTokens);
    Assert.Equal("northwind", request.Tags["tenant"]);
}

Intent files

A test can load the same intent files the application ships, and fail when one of them has a problem. Copy the folder into the test project's output, with the same <None Update="intents\**\*" CopyToOutputDirectory="PreserveNewest" /> item the quick start adds, and turn off watching, which a test does not need:

[Fact]
public void Every_intent_file_loads_without_problems()
{
    using var catalog = new FileIntentCatalog(Path.Combine(AppContext.BaseDirectory, "intents"),
        new FileIntentCatalogOptions { WatchForChanges = false });

    Assert.Empty(catalog.Current.Problems);
    Assert.True(catalog.Current.TryGet("Orders.ClassifyUrgency", out _));
}

[Fact]
public async Task The_rules_in_the_file_are_checked()
{
    using var catalog = new FileIntentCatalog(Path.Combine(AppContext.BaseDirectory, "intents"),
        new FileIntentCatalogOptions { WatchForChanges = false });
    var provider = new ScriptedProvider("""{"value": 7}""", """{"value": 5}""");
    using var client = ScriptedClient(provider);
    var runtime = new IntentRuntime(client, catalog);

    var result = await runtime.RunAsync<int>("Orders.ClassifyUrgency", "Line 3 is down. Ship today.");

    Assert.True(result.Success);
    Assert.Equal(5, result.Value);
    Assert.Equal(2, result.Attempts.Count);   // 7 broke the file's maximum of 5
}

The second test runs the intent with the rules the file declares, so a change to the file's rules is tested too.

With dependency injection

To test code that receives the client from the container, register a client on ScriptedProvider after the application's own registration. The container resolves the last registration, so the code under test gets the scripted client, and no provider key is needed:

[Fact]
public async Task The_application_gets_the_scripted_client_from_the_container()
{
    var services = new ServiceCollection();
    services.AddIntentSharp(settings => { });   // as the application registers IntentSharp
    services.AddSingleton(ScriptedClient(new ScriptedProvider("""{"value": 4}""")));   // the last registration wins
    services.AddSingleton<OrderTriage>();
    await using var container = services.BuildServiceProvider();

    var urgency = await container.GetRequiredService<OrderTriage>().UrgencyAsync("Ship today.", CancellationToken.None);

    Assert.Equal(4, urgency);
}

What a scripted test does not cover

  • The model's own answers. A scripted test shows how your code handles a reply, not whether a real model gives that reply. Check that with a few real calls, run apart from the unit tests, with your provider key; they are billed.
  • Token counts. ScriptedProvider estimates them at one token per four characters.
  • Embeddings. It has none unless you pass a method that computes them, as the second argument of new ScriptedProvider(request => ..., text => ...).