Memory, cache and rate limiting

Three parts sit around every call, each behind an interface you can replace:

Part	Interface	Built in	Builder default	`AddIntentSharp` default
The operation log	`ISemanticMemory`	`InMemorySemanticMemory`, `SqliteSemanticMemory`	in process, the last 10,000 calls	in process, the last 10,000 calls
The request cache	`IRequestCache`	`MemoryRequestCache`	none	in process, 1,000 entries
The rate limiter	`IRateLimiter`	`TokenBucketRateLimiter`	none	none

The cache and the rate limiter live in one process, and the in-process log does too; a SQLite log is a file, so processes given the same path share it. None of them is static: each belongs to the client it is given to. All three are set on the builder:

var client = new IntentClientBuilder()
    .UseAnthropic(anthropicKey)
    .WithMemory(new SqliteSemanticMemory("intentsharp-log.db"))
    .WithCache(new MemoryRequestCache(capacity: 5_000))
    .WithRateLimiter(new TokenBucketRateLimiter(requestsPerSecond: 2, burst: 5))
    .Configure(options => options.CacheTtl = TimeSpan.FromMinutes(30))
    .Build();

WithMemory(null) turns the log off, and WithCache(null) and WithRateLimiter(null) leave those out. The client does not dispose any of the three.

The operation log records every call

After each call, the client records one entry, whether the call succeeded or not:

Field	What it holds
`Intent`	The intent
`InputPreview`	The serialized input, cut to its first 2,000 characters
`Output`	The model's last reply, in full
`TargetType`	The name of the result type
`Success`, `Attempts`	Whether the call succeeded, and how many attempts it made
`Provider`, `Model`	Who answered
`InputTokens`, `OutputTokens`	The call's token counts
`Timestamp`	When it was recorded
`Tags`	The call's tags; for a catalog intent, its name, version and file

Nothing is recorded for a call answered from the cache, for a call with AIRequestOptions.DoNotRecord, or when IntentClientOptions.RecordInMemory is false. An error while recording is logged at Warning and never fails the call. The library never reads the log back into a request: it is for you.

Because the log holds inputs and replies, it holds whatever your inputs hold. Keep sensitive calls out of it with DoNotRecord, per call or with the doNotRecord field of an intent file:

// Ask the model every time, and keep this input out of the operation log.
var options = new AIRequestOptions { BypassCache = true, DoNotRecord = true };
var summary = await patientNote.AIAsync(client, "Summarize this note for the billing team.", options);

Where the log lives

InMemorySemanticMemory(maxEntries) keeps entries in the process and drops the oldest past maxEntries, 10,000 by default (0 means no limit). It is what a client gets when you do not call WithMemory.
SqliteSemanticMemory(path) keeps them in a SQLite file, created when missing, so they survive a restart.

Reading it

GetStatsAsync() counts the entries, the successful ones, the tokens and the oldest and newest times. RecallAsync(query, maxResults) returns the entries whose intent is most like the query, most similar first. ClearAsync() removes every entry.

if (client.Memory is { } log)
{
    MemoryStats stats = await log.GetStatsAsync();
    Console.WriteLine($"{stats.TotalEntries} calls, {stats.SuccessRate:P0} succeeded, {stats.TotalTokens} tokens");

    foreach (var match in await log.RecallAsync("order urgency", maxResults: 3))
        Console.WriteLine($"{match.Score:0.00} {match.Entry.Timestamp:u} {match.Entry.Intent} -> {match.Entry.Output}");
}

Recall compares words by default: the share of words of three or more characters that the query and an intent have in common. Entries with nothing in common are left out. SqliteSemanticMemory compares the most recent 2,000 entries (RecallCandidateLimit).

To recall by meaning instead, give the log a provider that supports embeddings:

// Recall by meaning: each recorded call costs one embeddings call to OpenAI.
var client = new IntentClientBuilder()
    .UseAnthropic(anthropicKey)
    .WithMemory(new InMemorySemanticMemory(maxEntries: 10_000, embeddingProvider: new OpenAIProvider(openAIKey)))
    .Build();

Each recorded call then costs one embeddings call, and so does each recall. These calls are not counted in TotalModelCalls, not rate-limited and not cached. When an embedding fails, the entry is kept without one and recall falls back to words for it; pass a logger to the log's constructor to see such failures at Warning.

The request cache answers identical requests without a model call

With a cache, an identical request to the same provider is answered from it while the entry lasts. Only the first attempt of a call reads the cache.

A cached answer is checked like a fresh one, against the current call's rules and checks. One that fails is discarded, the call asks the provider instead, and the fresh reply replaces the entry, so an answer cached under older rules is never served unchecked.

A call answered from the cache has FromCache true, adds no tokens to Usage, does not count as a model call, does not wait on the rate limiter and is not recorded in the log. A judge, when on, still makes its own call. Refusals and replies cut off by the output limit are never stored.

Setting	Default	What it does
`MemoryRequestCache(capacity)`	1,000 entries	Past the capacity, the least recently used entry goes.
`IntentClientOptions.CacheTtl`	1 hour	How long an entry lasts.
`IntentClientOptions.UseCache`	`true`	False stops this client using its cache.
`AIRequestOptions.BypassCache`	`false`	True keeps one call out of the cache: no read, no write.

GetStats() reports hits, misses, entries and the capacity, and HitRate:

if (client.Cache is { } cache)
{
    CacheStats stats = cache.GetStats();
    Console.WriteLine($"Cache: {stats.Hits} hits, {stats.Misses} misses, {stats.HitRate:P0}, {stats.Entries}/{stats.Capacity} entries");
}
if (client.RateLimiter is { } limiter)
{
    RateLimitSnapshot bucket = limiter.GetSnapshot(client.PrimaryProvider.Name);
    Console.WriteLine($"{bucket.Provider}: {bucket.AvailableTokens:0.0}/{bucket.Capacity} free, {bucket.Allowed} allowed, {bucket.Denied} refused");
}

To share a cache between processes, implement IRequestCache: GetAsync(key), SetAsync(key, response, ttl), ClearAsync() and GetStats(). The key is an opaque string the client computes; store the AIResponse under it as it is.

The rate limiter spaces out calls to each provider

TokenBucketRateLimiter(requestsPerSecond, burst) keeps one allowance per provider name. It starts full with burst calls (10 by default), refills at requestsPerSecond (5 by default), and each model call takes one.

A call waits for its turn for up to 30 seconds. When its turn is further away, it fails at once with AIRateLimitException, a transient error: the call falls back when a fallback is set, and otherwise ends with a Provider issue.
Judge calls take their turn from the primary provider's allowance.
Embedding calls do not wait on it.
It limits this process only. Several instances of an application each have their own.

GetSnapshot(provider) reports the calls that may start now, the burst size, and the calls allowed and refused so far.

With dependency injection

AddIntentSharp sets up the three from the IntentSharp section: MemoryPath, EnableRequestCache, CacheCapacity, RateLimitPerSecond and RateLimitBurst, which Configuration and DI lists with their defaults:

services.AddIntentSharp(settings =>
{
    settings.MemoryPath = "intentsharp-log.db";  // SQLite; "" turns the log off
    settings.CacheCapacity = 5_000;               // the cache is on by default
    settings.RateLimitPerSecond = 2;              // 0, the default, means no limiter
    settings.RateLimitBurst = 5;
    settings.Options.CacheTtl = TimeSpan.FromMinutes(30);
});

An implementation of your own that you register in the container is used in place of the built-in one, but only while its setting is on: to use your IRateLimiter, register it and set RateLimitPerSecond above 0.

Limits

The built-in cache and limiter live in one process. They are not shared between instances unless you supply implementations that are.
The log's InputPreview holds the first 2,000 characters of the input; Output holds the whole reply.
Recall compares the query with each entry's intent, not with its input or its output.

Table of Contents