Memory, cache and rate limiting

Three pluggable interfaces, each with a default implementation. All three are used by IntentClient; none is static.

Interface	Implementations	Builder default	DI default
ISemanticMemory	InMemorySemanticMemory, SqliteSemanticMemory	InMemorySemanticMemory(10_000), no embedder	in-process memory (see below)
IRequestCache	MemoryRequestCache	none	MemoryRequestCache(1000)
IRateLimiter	TokenBucketRateLimiter	none	none unless RateLimitPerSecond > 0

Memory

Code: src/IntentSharp/Memory/.

Task RecordAsync(MemoryEntry entry, CancellationToken ct = default);
Task<IReadOnlyList<MemoryMatch>> RecallAsync(string query, int maxResults = 5, CancellationToken ct = default);
Task<MemoryStats> GetStatsAsync(CancellationToken ct = default);
Task ClearAsync(CancellationToken ct = default);

The client records one MemoryEntry per completed request (see request-pipeline.md). An entry holds the intent, an input preview of at most 2000 characters, the full raw output, the target type name, success, attempt count, provider, model, token counts, timestamp, tags and an optional embedding. Nothing in the library calls RecallAsync; it is for applications.

Recall ranks by cosine similarity of embeddings when both the query and the entry have one, and otherwise by Jaccard overlap of lower-cased words of three or more characters (Similarity.TokenOverlap). Matches with a score of zero are dropped; ties go to the newest entry.

InMemorySemanticMemory(maxEntries = 10_000, embeddingProvider = null, logger = null) keeps entries in a locked linked list and evicts the oldest past the cap (0 means unlimited).

SqliteSemanticMemory(path, embeddingProvider = null, logger = null) uses Microsoft.Data.Sqlite. It creates the memory_entries table on construction, stores embeddings as float32 blobs, serializes writes with a semaphore, and ranks the most recent RecallCandidateLimit (2000) entries in process.

Both embed the intent on record when given a provider whose SupportsEmbeddings is true. An embedding error does not fail the call: the entry is stored without an embedding (or the recall query goes without one) and recall falls back to word overlap for it. When a logger was given, each such error is logged at Warning with the exception: "Embedding the recorded intent failed; word overlap is used for it instead of semantic similarity." ("the recall query" when recall embeds the query). A cancellation of the caller's own token is not swallowed; it propagates. The builder's default memory has no embedding provider, so it never makes an embeddings call and recall uses word overlap. Embeddings are opt-in: pass WithMemory(new InMemorySemanticMemory(10_000, provider)) (or a SqliteSemanticMemory with a provider), and each recorded request then costs one embeddings call that is not counted in IntentClient.TotalModelCalls, not rate-limited and not cached. WithMemory(null) disables memory.

Opt out per request with AIRequestOptions.DoNotRecord, or globally with IntentClientOptions.RecordInMemory.

Request cache

Code: src/IntentSharp/Resilience/RequestCache.cs.

RequestCacheKey.For(providerName, request) is a SHA-256 over the provider name, model, system prompt, messages, max output tokens, temperature, format and schema. Tags are not part of the key.

MemoryRequestCache(capacity = 1000) is an in-process dictionary with a TTL per entry and least-recently- used eviction at capacity. GetStats() reports hits, misses, entries and capacity.

The client consults the cache only on the first attempt, never stores refusals or truncated replies, and uses IntentClientOptions.CacheTtl (default one hour). A cache hit sets AIResult.FromCache, adds no tokens to AIResult.Usage (they were spent by the request that filled the cache) and is not recorded in memory. A judge call made for a cached reply still counts toward Usage.

A cached reply is checked like a fresh one, against this request's validators and judge. When it fails, the attempt is discarded (it is not recorded and does not use up an attempt), the rest of the request stops reading the cache, and the attempt is re-run against the provider. The fresh reply overwrites the cache entry, so the failing reply is not served to later identical requests.

Rate limiter

Code: src/IntentSharp/Resilience/RateLimiter.cs.

TokenBucketRateLimiter(requestsPerSecond = 5, burst = 10) keeps one bucket per provider name. A bucket starts full, refills continuously, and each call takes one token. AcquireAsync(provider, maxWait) waits for a token, or throws AIRateLimitException immediately when the next token is further away than maxWait. The client always passes 30 seconds. GetSnapshot(provider) reports available tokens, capacity and allowed and denied counts.

The limiter is local to one process. It throttles calls made through IntentClient.InvokeAsync and judge calls; a judge call takes a token from the primary provider's bucket, whichever provider the judge uses. Memory embedding calls (opt-in, see above) go directly to the provider and are not throttled.

DI settings

IntentSharpSettings (section IntentSharp) controls these in DI:

• MemoryPath: null uses a registered ISemanticMemory if there is one, else the builder default; an empty string disables memory; a path uses a registered ISemanticMemory or a new SqliteSemanticMemory (without an embedder).
• EnableRequestCache (true) and CacheCapacity (1000), or a registered IRequestCache.
• RateLimitPerSecond (0 = off) and RateLimitBurst (10), or a registered IRateLimiter.