AI agents do not interact with a database or payment system through natural language alone. They call named tools with structured arguments, interpret returned objects and sometimes retry when the network fails. A small change to a tool schema can therefore become a production incident even when the model itself is unchanged. Consider a tool that once accepted a plain customer identifier but now requires an account scope and authorization intent. Yesterday’s successful agent trace may fail today, or worse, succeed with a different meaning. This article proposes a contract-first release process that tests schema evolution as a system behavior rather than treating it as a documentation edit.

Separate structural compatibility from semantic compatibility

JSON Schema Draft 2020-12, described in the official specification published 16 June 2022, provides vocabulary for types, required properties, validation and conditional structures. A new optional field can be structurally backward compatible for requests. Yet semantic compatibility asks a harder question: what happens when the field is absent? If a missing currency changes from USD to “infer from user location,” old calls can quietly acquire new behavior without triggering a validator.

Maintain separate compatibility decisions for request shape, response shape and side effects. A field rename is obviously breaking for a caller that sends the old name. A field addition can also be breaking if the consumer rejects unknown response keys. Schema linting is necessary but insufficient. A migration review should state old behavior, proposed behavior, defaulting rules and whether both versions can coexist during a transition window.

Inventory tool callers rather than counting endpoints

One endpoint may serve different AI models, mobile application versions, scheduled workflows and partner integrations. List the actual consumers and the schema versions each can produce. Capture anonymized input samples, excluding credentials and personal information, so a regression suite can replay representative call shapes. For tools with financial or account-changing effects, add explicit approval scope to the inventory. The most important caller may be an old background workflow nobody remembers during a release review.

Use a small contract registry with tool name, immutable schema revision, owner, deprecation date and rollback identifier. A breaking release should get a versioned endpoint or dispatch path, not a silent mutation of a shared schema. Include response-field changes in consumer tests. When a service returns additional object properties, check whether the client treats them as harmless annotations or blindly converts the entire response into a user-visible instruction.

Design a six-cell compatibility matrix

Test old clients against old servers, old clients against new servers, and new clients against new servers for both success and deliberate failure cases. That gives six useful cells before adding environmental variants. Use contract fixtures for valid input, omitted optional fields, unexpected properties, wrong types, expired permissions and partial network failures. Record both validation outcome and the exact side effect. A request that produces a different billing record but still returns HTTP 200 is not a passing compatibility test.

The JSON Schema validation vocabulary can describe structural assertions, but it cannot decide whether a payment amount is commercially authorized. Pair schema validation with business invariants. For example, a tool marked “read invoice” must never change invoice status. A parameter called dry_run must be supported by actual execution logic, not simply appear in the request contract. This is where an integration test should verify state before and after a call.

Make retries safe when tools change

Tool retries are often triggered by timeouts precisely when engineers have the least certainty about whether the original request executed. RFC 9110, published June 2022, distinguishes safe and idempotent HTTP method semantics and cautions against blindly retrying non-idempotent requests. An AI tool that creates an order should use a business idempotency key and a stable operation identity, even if its transport happens to be an HTTP POST. The key must survive agent retries and avoid collisions across tenants.

When an API changes argument naming, preserve idempotency across both schema revisions. Otherwise an old caller might submit the same purchase with an old field and a new caller might submit a second purchase using a renamed field. A robust receipt binds the normalized business action, tenant, version and idempotency key. If the system cannot prove that a retry is harmless, it should require human review rather than guess from a model-generated explanation.

Observe version drift in production

After deployment, monitor the distribution of callers by schema version, rejected fields, fallback-default usage and side-effect discrepancies. Instrument traces with tool identifier, contract version, authorization result, validation outcome and idempotency decision. Exclude sensitive arguments from generic logs. The purpose is to detect the early warning pattern: a cluster of retries and unexpected defaults from one older client, before customers experience a large outage.

Do not use raw success rate as the only launch metric. If a tool now accepts more calls by silently coercing malformed input, success may rise while correctness declines. Compare the same fixture population across versions and sample human-reviewed outcomes. A deployment gate can require no increase in unauthorized side effects, no change in normalized billing identity and a stable p95 tool latency budget under representative concurrency. These thresholds should come from the team’s service objectives.

A practical staged rollout and rollback plan

First, publish the candidate schema and compatibility matrix for review. Second, deploy the new server behind a small traffic cohort or shadow evaluator without executing duplicate writes. Third, replay sanitized traces, explicitly checking old-client behavior. Fourth, expand rollout while keeping the old contract available for a documented deprecation period. Finally, remove the old path only after telemetry shows that remaining callers have either migrated or been intentionally retired.

Rollbacks must restore behavior, not merely restore JSON field names. Keep a record of any side effects created while the newer contract was active, because reverting server code cannot reverse a completed payment or notification. The engineering lesson is simple: for autonomous tools, schemas are execution contracts. Their evolution deserves the same change control as a database migration or a financial ledger, supported by tests that examine meaning as well as syntax.

Sources

JSON Schema, Draft 2020-12, published 16 June 2022; JSON Schema Validation specification, 2022; IETF RFC 9110, HTTP Semantics, June 2022.