Add extensible chat client routing by joshuajyue · Pull Request #7662 · dotnet/extensions

joshuajyue · Jul 27, 2026

Summary

Adds experimental chat-routing primitives for selecting configured IChatClient instances without imposing a particular routing policy.

RoutingChatClient supports one-shot client selection.
FailoverChatClient retries uncanceled failures that occur before output is exposed.
OrderedFailoverChatClient provides ordered fallback.
SemanticRoutingChatClient selects clients using cached profile embeddings.
RoutingContext exposes mutable request messages and options.
FailoverChatClientAttempt reports invocation outcome, duration, time to first update, completion, and output commitment.

Related to #7647.

API shape

Routing policies have two seams:

protected override ValueTask<IChatClient> SelectClientAsync(
    RoutingContext context,
    CancellationToken cancellationToken);

protected override ValueTask OnRoutingUpdateAsync(
    RoutingContext context,
    FailoverChatClientAttempt? attempt,
    bool isTerminal,
    CancellationToken cancellationToken);

SelectClientAsync runs before every invocation. OnRoutingUpdateAsync reports each attempt so policy state can influence the next selection.

isTerminal means the base will not select another client after the callback completes. A null attempt represents selection terminating before a client was invoked.

Failover behavior

Cancellation never triggers reselection.
Streaming fallback occurs only before the first update is exposed.
Mid-stream failures and caller abandonment are terminal.
Enumerator disposal failures are included in the attempt.
Attempt limits are request-local.
Null selector results are treated as terminal selection failures.
Ordered failover retains request state only between a nonterminal update and the immediately following selection; it holds no state while a provider invocation or stream is active.
Streaming callers must dispose active enumerators for terminal observation and inner-resource cleanup.

Semantic routing

SemanticRoutingChatClient:

Lazily embeds and caches profile utterances.
Embeds the final user message for selection.
Supports global top-K profile matching.
Aggregates matches per client using mean or sum.
Preserves the existing single-best-match behavior with topK: 1.
Uses a required default client when no profile clears the threshold.
Supports caller-controlled ownership through leaveOpen.

Tests

Coverage includes selection failures, retries, cancellation, attempt limits, streaming commitment, early disposal, enumerator failures, concurrent requests, state cleanup, semantic thresholds, top-K aggregation, caching, and ownership.

753 tests passed on net10.0
net472 test project builds successfully

Microsoft Reviewers: Open in CodeFlow

Introduce one-shot and failover routing primitives, ordered and semantic routing implementations, lifecycle reporting, streaming safeguards, and focused routing tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

joshuajyue · Jul 27, 2026

@dotnet-policy-service agree

Copilot

Pull request overview

Adds new experimental chat-routing primitives to the Microsoft.Extensions.AI.* stack to enable one-shot client selection, failover with attempt tracking, ordered fallback, and semantic (embedding-based) routing, along with extensive unit test coverage.

Changes:

Introduces RoutingChatClient / RoutingContext in AI.Abstractions for pluggable per-request client selection.
Adds FailoverChatClient, FailoverChatClientAttempt, OrderedFailoverChatClient, and SemanticRoutingChatClient in Microsoft.Extensions.AI.
Updates experimental diagnostic IDs and public API JSON manifests; adds comprehensive routing/failover/semantic tests.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/Libraries/Microsoft.Extensions.AI.Tests/ChatRouting/RoutingChatClientTests.cs	Adds broad unit coverage for routing, failover, ordered fallback, streaming behaviors, and semantic routing.
src/Shared/DiagnosticIds/DiagnosticIds.cs	Adds a new experimental diagnostic ID for routing chat APIs.
src/Libraries/Microsoft.Extensions.AI/Microsoft.Extensions.AI.json	Updates public API manifest for new experimental routing/failover types.
src/Libraries/Microsoft.Extensions.AI/ChatRouting/SemanticRoutingChatClient.cs	Implements embedding-based client selection with cached profile embeddings and top-K aggregation.
src/Libraries/Microsoft.Extensions.AI/ChatRouting/OrderedFailoverChatClient.cs	Implements ordered client failover policy as a `FailoverChatClient`.
src/Libraries/Microsoft.Extensions.AI/ChatRouting/FailoverChatClientAttempt.cs	Adds an attempt record type capturing duration/commitment/completion/exception details.
src/Libraries/Microsoft.Extensions.AI/ChatRouting/FailoverChatClient.cs	Adds failover orchestration for non-streaming + streaming, including commitment rules and attempt limits.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Microsoft.Extensions.AI.Abstractions.json	Updates public API manifest for new experimental routing primitives.
src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatRouting/RoutingContext.cs	Introduces mutable per-request routing context (messages/options).
src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatRouting/RoutingChatClient.cs	Introduces the base routing client (and callback factory) for one-shot selection and dispatch.

Comments suppressed due to low confidence (2)

src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatRouting/RoutingChatClient.cs:71

SelectClientAsync returning null will currently cause a null-dereference when attempting to stream from the selected client. Consider throwing a clear InvalidOperationException instead.

        var context = new RoutingContext(messages, options);
        IChatClient client = await SelectClientAsync(context, cancellationToken).ConfigureAwait(false);
        await foreach (ChatResponseUpdate update in
            client.GetStreamingResponseAsync(context.Messages, context.ChatOptions, cancellationToken)
                .WithCancellation(cancellationToken)
                .ConfigureAwait(false))

src/Libraries/Microsoft.Extensions.AI/ChatRouting/FailoverChatClient.cs:220

Streaming pre-output failures treat cancellation as terminal only when cancellationToken.IsCancellationRequested is true. If the failure is an OperationCanceledException not tied to the caller token, failover may incorrectly reselect another client.

                bool isTerminal = cancellationToken.IsCancellationRequested || reachedAttemptLimit;

                await OnRoutingUpdateAsync(context, attempt, isTerminal, cancellationToken);

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Treat provider-thrown cancellation as terminal, materialize semantic routing messages before inspection, and enforce non-null selection consistently for streaming. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/Libraries/Microsoft.Extensions.AI/ChatRouting/FailoverChatClient.cs:247

In streaming failover, outputCommitted is set to true before accessing enumerator.Current. If Current throws (possible for a faulty async enumerator), the attempt will incorrectly record output as committed and will not capture the exception in FailoverChatClientAttempt.Exception, and routing may become terminal when it should be eligible for failover before any update is exposed.

                while (hasCurrent)
                {
                    outputCommitted = true;
                    yield return enumerator.Current;

Copilot AI review requested due to automatic review settings July 27, 2026 22:31

joshuajyue requested review from a team as code owners July 27, 2026 22:31

Copilot started reviewing on behalf of joshuajyue July 27, 2026 22:31 View session

dotnet-policy-service Bot assigned joshuajyue Jul 27, 2026

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatRouting/RoutingChatClient.cs

Comment thread src/Libraries/Microsoft.Extensions.AI/ChatRouting/FailoverChatClient.cs

Comment thread src/Libraries/Microsoft.Extensions.AI/ChatRouting/SemanticRoutingChatClient.cs

joshuajyue and others added 3 commits July 27, 2026 15:43

Potential fix for pull request finding

665abe4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Credit semantic-router inspiration

26d7a5c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

joshuajyue requested a review from Copilot July 28, 2026 05:15

Copilot started reviewing on behalf of joshuajyue July 28, 2026 05:15 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add extensible chat client routing - #7662

Add extensible chat client routing#7662
joshuajyue wants to merge 4 commits into
dotnet:maindotnet/extensions:mainfrom
joshuajyue:routing-behaviorsjoshuajyue/extensions:routing-behaviorsCopy head branch name to clipboard

joshuajyue commented Jul 27, 2026 •

edited by dotnet-policy-service Bot

Loading

Uh oh!

joshuajyue commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Search code, repositories, users, issues, pull requests...

Uh oh!

Conversation

joshuajyue commented Jul 27, 2026 • edited by dotnet-policy-service Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

API shape

Failover behavior

Semantic routing

Tests

Microsoft Reviewers: Open in CodeFlow

Uh oh!

joshuajyue commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshuajyue commented Jul 27, 2026 •

edited by dotnet-policy-service Bot

Loading