fix(agents): virtual-provider budget cost is priced against the agent-level (provider, model), not the selected backend #159

New issue

Open

opened 2026-10-27 06:58:37 +00:00 by hurui200320 · 0 comments

hurui200320 commented

2026-10-27 06:58:37 +00:00

Member

Metadata

Commit Message: fix(agents): attribute virtual-provider cost to the selected backend
Branch: fix/virtual-provider-cost-attribution

Background and context

ADR-2038 virtual-provider pooling lets each backends entry override provider/model (spec §4.4.7), and each backend is constructed and credential-validated independently. The in-run budget machinery, however, anchors on the agent's own attributes: LLMAgent._accumulate_cost_after_invoke (agents/llm.py) resolves rates via _compute_cost(current_pricing, self.provider, self.model, …), and the #150 zero-cost carve-out in _check_budget_before_invoke consults the same pair via _is_zero_rated_invocation(current_pricing, str(self.provider), str(self.model)). For a pooled agent those agent-level values are inheritance defaults, not the backend that actually served the invocation.

Current behavior

Verified from the platform side during the cleverthis-webapp#549 E2E oneshot (2026-10-27, cleveractors @ 4b98ce00; platform semantics per #473: max_cost_usd is always set from the payer's wallet balance):

Pooled agent with per-backend overrides deepseek/deepseek-v4.1-flash + zai/glm-5.3-flash and agent-level anchor zai/glm-5.1: runs succeed, but accumulated cost is computed at the anchor model's rates — the rates of the backend that actually served never enter the gate, so cost accounting for pooled agents is wrong whenever backends differ from the agent-level pair.
If the agent-level (provider, model) is absent from the pricing table while the backends themselves are priced, _accumulate_cost_after_invoke raises ExecutionError(kind="cost", reason="missing_pricing_entry") after a successful model call — the run fails even though the invoked backend has valid pricing. With #473's always-set limit there is no configuration that avoids this short of adding the anchor model to the pricing table.
Symmetrically, _check_budget_before_invoke's zero-cost carve-out consults the agent-level pair: a provably-free backend cannot pass the gate when the agent-level pair is unresolvable or non-zero-rated ("anything unresolvable still blocks").

Expected behavior

Cost attribution for virtual-provider invocations should use the selected backend's provider/model — VirtualChatModel knows which backend served each call, so per-call token counts can carry that attribution back to LLMAgent. The agent-level pair remains the fallback for non-virtual agents. At minimum, an unpriced agent-level pair must not fail runs whose actually-selected backends are priced.

Acceptance criteria

Unit: pooled agent with backends from distinct priced providers → each invocation's accumulated cost uses the selected backend's rates.
Unit: agent-level model unpriced + selected backend priced → run completes; no missing_pricing_entry.
The zero-cost carve-out evaluates the selected backend's rates (a provably-free backend passes the gate even when the agent-level pair is non-zero/unresolvable).
Non-virtual agents behave exactly as before.
nox -s typecheck, nox -s unit_tests, and the coverage threshold stay green.

Supporting information

ADR-2038 (selection strategies, credential isolation) and spec §4.4.7; platform context: cleverthis-webapp#549 (integration issue), #473 (always-set max_cost_usd), #150/#151 (zero-cost carve-out precedent, "anything unresolvable still blocks").
Suggested direction: surface the selected backend identity from VirtualChatModel (response metadata or a selection callback) so LLMAgent can attribute cost; alternatively accept a per-backend pricing-resolution hook from the platform.

Subtasks

Decide the attribution mechanism (backend-identity propagation vs per-backend pricing hook).
Wire selected-backend attribution into _accumulate_cost_after_invoke and _check_budget_before_invoke.
Unit tests for the matrix above, including the unpriced-anchor case.
Update docs/index.md §4.4.7, ADR-2038 cost note, and CHANGELOG; revisit benchmarks/virtual_provider_benchmark.py if the hot path changes.

Definition of Done

All subtasks completed and checked off.
A commit whose first line matches the Metadata Commit Message is pushed on the Metadata Branch and merged via PR.

## Metadata - **Commit Message**: `fix(agents): attribute virtual-provider cost to the selected backend` - **Branch**: `fix/virtual-provider-cost-attribution` ## Background and context ADR-2038 virtual-provider pooling lets each `backends` entry override `provider`/`model` (spec §4.4.7), and each backend is constructed and credential-validated independently. The in-run budget machinery, however, anchors on the **agent's own attributes**: `LLMAgent._accumulate_cost_after_invoke` (agents/llm.py) resolves rates via `_compute_cost(current_pricing, self.provider, self.model, …)`, and the #150 zero-cost carve-out in `_check_budget_before_invoke` consults the same pair via `_is_zero_rated_invocation(current_pricing, str(self.provider), str(self.model))`. For a pooled agent those agent-level values are inheritance defaults, not the backend that actually served the invocation. ## Current behavior Verified from the platform side during the cleverthis-webapp#549 E2E oneshot (2026-10-27, cleveractors @ 4b98ce00; platform semantics per #473: `max_cost_usd` is **always** set from the payer's wallet balance): 1. Pooled agent with per-backend overrides `deepseek/deepseek-v4.1-flash` + `zai/glm-5.3-flash` and agent-level anchor `zai/glm-5.1`: runs succeed, but accumulated cost is computed at the **anchor** model's rates — the rates of the backend that actually served never enter the gate, so cost accounting for pooled agents is wrong whenever backends differ from the agent-level pair. 2. If the agent-level `(provider, model)` is absent from the pricing table while the backends themselves are priced, `_accumulate_cost_after_invoke` raises `ExecutionError(kind="cost", reason="missing_pricing_entry")` **after a successful model call** — the run fails even though the invoked backend has valid pricing. With #473's always-set limit there is no configuration that avoids this short of adding the anchor model to the pricing table. 3. Symmetrically, `_check_budget_before_invoke`'s zero-cost carve-out consults the agent-level pair: a provably-free backend cannot pass the gate when the agent-level pair is unresolvable or non-zero-rated ("anything unresolvable still blocks"). ## Expected behavior Cost attribution for virtual-provider invocations should use the **selected backend's** provider/model — `VirtualChatModel` knows which backend served each call, so per-call token counts can carry that attribution back to `LLMAgent`. The agent-level pair remains the fallback for non-virtual agents. At minimum, an unpriced agent-level pair must not fail runs whose actually-selected backends are priced. ## Acceptance criteria 1. Unit: pooled agent with backends from distinct priced providers → each invocation's accumulated cost uses the **selected backend's** rates. 2. Unit: agent-level model unpriced + selected backend priced → run completes; no `missing_pricing_entry`. 3. The zero-cost carve-out evaluates the selected backend's rates (a provably-free backend passes the gate even when the agent-level pair is non-zero/unresolvable). 4. Non-virtual agents behave exactly as before. 5. `nox -s typecheck`, `nox -s unit_tests`, and the coverage threshold stay green. ## Supporting information - ADR-2038 (selection strategies, credential isolation) and spec §4.4.7; platform context: cleverthis-webapp#549 (integration issue), #473 (always-set `max_cost_usd`), #150/#151 (zero-cost carve-out precedent, "anything unresolvable still blocks"). - Suggested direction: surface the selected backend identity from `VirtualChatModel` (response metadata or a selection callback) so `LLMAgent` can attribute cost; alternatively accept a per-backend pricing-resolution hook from the platform. ## Subtasks - [ ] Decide the attribution mechanism (backend-identity propagation vs per-backend pricing hook). - [ ] Wire selected-backend attribution into `_accumulate_cost_after_invoke` and `_check_budget_before_invoke`. - [ ] Unit tests for the matrix above, including the unpriced-anchor case. - [ ] Update `docs/index.md` §4.4.7, ADR-2038 cost note, and CHANGELOG; revisit `benchmarks/virtual_provider_benchmark.py` if the hot path changes. ## Definition of Done - All subtasks completed and checked off. - A commit whose first line matches the Metadata Commit Message is pushed on the Metadata Branch and merged via PR.

hurui200320 added the

labels

2026-10-27 06:58:37 +00:00

hurui200320 added reference fix/virtual-provider-cost-attribution

2026-10-27 06:59:53 +00:00