In this post, I am going to walk through what AI gateways are, the three types you will see in the market, and why you should not force your existing API gateway into this role. I will also cover a common question at the end. If your API needs are simple, can one AI gateway do both jobs?
What is an AI gateway?
An AI gateway is a middleware layer. It sits between an AI agent and whatever the agent is calling. It enforces identity, policy, cost control, and safety at that boundary.
The market terminology is confusing. Different vendors use different names for the same thing. The simple way I think about it is to classify each gateway by its target:
- Model/LLM gateway: when the target is a model or LLM
- MCP gateway: when the target is a tool exposed over MCP
- Agent gateway: when the target is another agent
"AI gateway" is just a term for all above three. The common idea is this, wherever governance changes are needed, you need a control point.
Model/LLM gateways
This is the most mature category. A model gateway usually supports two integration modes:
- Unified API: The agent sends an OpenAI-compatible request. It passes the target model as a parameter. The gateway translates the request and response for each provider.
- Passthrough mode: The agent sends the provider's native payload. The gateway forwards it as-is. You still get governance, without translation.
On top of that, a good model gateway gives you four things:
Authentication and credentials: Inbound, it validates the caller against our enterprise IdP using OAuth/OIDC. It applies access control like RBAC. Outbound, it injects provider credentials at runtime. So, your application code never holds LLM API keys. This is a big win on its own.
Security guardrails: The gateway sees every prompt and every response. So it is the natural place for prompt injection defence, Personally Identifiable Information (PII) detection and redaction, and content moderation. Some gateways do this internally. Others call an external guardrails service.
Cost management: AI cost is measured in tokens, not requests. The gateway meters input and output tokens. It attributes usage by app, model, and environment. It enforces rate limits, quotas, and budget limits.
Routing: This is where things get interesting. You can route by model alias, by metadata like user tier or region, or by traffic split for A/B. Some gateways can even pick a model based on the topic or complexity of the request. A simple question goes to the cheaper model and a hard question goes to the more capable model.
Some gateways also add prompt-level features. Prompt caching, often using a vector store. Prompt decoration to inject mandatory governance context. Prompt rewriting to compress or normalise prompts.
MCP gateways
MCP is becoming the standard way to connect agents to tools. So a second gateway category has appeared for the agent-to-tool path. An MCP gateway can act as:
- A proxy in front of an existing MCP server
- A bridge that wraps your normal REST APIs as MCP tools
- A composite (virtual) MCP server that aggregates multiple backends behind one MCP endpoint
MCP has two core operations. The gateway applies policy to both:
Tool discovery. When an agent asks for the tool list, the gateway filters it. Each agent sees only the tools it is allowed to use. On the security side, it can defend against tool poisoning. It can also pin tool definitions, so a compromised server cannot silently change a tool description.
Tool invocation. When an agent calls a tool, the gateway validates user permissions. It applies per-tool rate limits. It screens the payload for indirect prompt injection, DLP, and PII before forwarding the call.
One more useful feature is semantic tool search. Instead of returning all n tools and filling up the agent's context, the agent can invoke a search meta tool like search("bsb validation"). It gets only the relevant tools back.
Agent gateways
This is the newest category. It governs agent-to-agent communication, usually based on the A2A protocol. An A2A-aware agent gateway handles:
- Transport mediation (JSON-RPC, HTTP+JSON/REST)
- Agent discovery, including agent card rewriting and registry integration
- Identity, with per-agent authentication using OAuth
- Policy enforcement like rate limiting, token budgets, and guardrails
- Observability, with telemetry that carries A2A-specific attributes
Will your agents ever delegate work to an agent you do not operate? A partner's agent, or a SaaS vendor's agent? If so, this is the boundary you need to control.
Why not just use your existing API gateway?
This is the question I hear the most. You already invested in an API gateway. It works well for REST traffic. But AI traffic breaks several assumptions those products were built on.
1. The unit of cost is the token, not the request.
API gateways meter and throttle by request count. But two AI requests can differ in cost by 1000x. It depends on prompt and output size. A rate limit of "100 requests per minute" does almost nothing against a runaway agent generating huge completions. You need token metering, token attribution, and token budgets.
2. The risk in the payload, not the headers.
Traditional gateways make decisions on URLs, methods, headers, and keys. They rarely look deep into the body. With AI traffic, it is the opposite. Prompt injection, PII leakage, and poisoned tool descriptions are all inside the payload, in natural language. You need payload-aware inspection on both the request and the response.
3. Routing is semantic, not path-based.
An API gateway routes on the URL path. An AI gateway may need to route based on what the request means. Its topic or complexity. It may also need model aliases, fallback chains, and A/B splits across providers. There is no /v1/hard-questions endpoint to match on.
4. The protocols are different.
MCP tool discovery and invocation. A2A agent cards and task execution. Streaming responses. None of these map cleanly to the stateless request/response model. A policy like "this agent can discover tools A and B but not C" needs the gateway to understand MCP semantics. Not just proxy JSON.
5. Credential handling works in both directions.
API gateways mostly authenticate inbound callers. AI gateways also need to broker outbound trust. They hold provider credentials centrally. They inject them per request based on policy. And they do this across many model providers, MCP servers, and remote agents.
6. Reliable model failover, not just retries.
With a normal API, a 200 response is a good response. With models, you also care about provider outages. Those need automatic failover to a different model. You also want progressive rollout of new model versions, with quality comparison.
7. AI traffic is stateful.
This one deserves a closer look. "Stateful" here means three things:
- Streaming: LLM responses stream token by token, usually over SSE. A connection can stay open for a long time.
- Sessions: MCP's Streamable HTTP transport carries a session ID. Every request in a session must reach the same server instance, or shared session state.
- Long-running tasks: A2A tasks can run for minutes or even hours. The client polls or receives push updates.
An AI gateway must support all three. It must even apply policy mid-stream. For example, counting output tokens as they arrive. Or running guardrails on partially streamed content.
Now, can traditional API gateways handle stateful traffic? Technically, most can proxy WebSockets and SSE. But their defaults work against you. They treat every request as independent. They load-balance per request, which breaks MCP session affinity. They buffer responses, which kills token streaming. The user sees nothing until the full response completes. And their timeouts are tuned for sub second APIs. So long running agent connections get terminated.
We can reconfigure a lot of this. Disable buffering and extend timeouts. Add sticky routing on the session header. But even then, the gateway cannot apply meaningful policy to the stream itself. An AI gateway treats streaming and sessions as a first class citizens. AI gateways must support stateful connections and API gateways do not support statefulness well by default.
Can one AI gateway replace both?
Here is another question I get. If an organisation has simple API management requirements plus an AI gateway use case, can it just run one AI gateway for both?
Often, yes. Most AI gateways are built on standard proxy engines. Many are extensions of Envoy, Kong, or similar. So they already cover the basics of API management. OAuth/OIDC authentication. Rate limiting. Routing. TLS termination. Observability. If your API needs are a handful of internal REST APIs with auth, throttling, and logging, one AI gateway can cover both. You get one control point, one policy model, and one set of dashboards.
But be honest about the word "simple". Traditional API management platforms bring things AI gateways typically lack:
- A developer portal with self-service API keys
- API productisation and monetisation, like tiered plans and billing
- API lifecycle management, such as versioning and deprecation workflows
- Partner and external developer onboarding
- Mature support for protocols like SOAP or gRPC transcoding
If any of those are on your roadmap, you will outgrow the AI gateway on the API side.
There is also a middle path. As the vendor landscape is changing fast. Established API management vendors are adding firs class AI gateway capabilities into their product stack. My advice is to audit API requirements first. Internal-only traffic with basic controls? Consolidate on the AI gateway. External consumers, portals, or monetisation? Keep them separate, or pick a blended platform.
Final thoughts
AI gateways are not a rebranding of API gateways. They solve separate problems your existing gateway cannot solve. Token-based cost, payload-level risk, semantic routing, new protocols, and stateful traffic.
No comments:
Post a Comment