Kong AI Gateway (US) brings model traffic into Kong Gateway, the company’s API management platform. Instead of every application talking straight to a provider, the gateway normalizes calls, applies policies and records usage.
What it does
- Proxies LLM calls in OpenAI-compatible format or native passthrough
- Routes by model name, header, parameter or semantic similarity
- Fails over between providers with retries, circuit breaking and priority
- Balances with round robin, consistent hashing, least-busy or semantic strategies
- Applies token quotas and semantic caching to control cost
- Adds prompt guardrails, sensitive-data sanitization and cloud moderation integrations
- Reports token, latency and time-to-first-token metrics
How it works
- AI plugins attach to Kong Gateway routes and services
- Provider credentials stay in the gateway, out of application code
- The 2.x entity model declares providers, models, MCP servers and agents
- The control plane can be managed in Konnect with a self-hosted data plane
- Audit logs and metrics flow to the configured observability destinations
Models and providers
- Documented providers include OpenAI, Azure AI, Bedrock, SageMaker, Gemini, Anthropic, Cohere, Mistral, xAI and DeepSeek
- Open models served locally by vLLM or Ollama join as a provider
- MCP servers and agents can be exposed through the same gateway
Availability and license
- Kong Gateway is open source under Apache-2.0, with the ai-proxy and ai-prompt-guard plugins in the public repository
- Advanced plugins and the 2.x entity model are part of the paid offerings
- Plans charge per gateway and per model above the included limit
Points to note
- The newest model requires a managed Konnect control plane
- Advanced AI features are enterprise plugins
- Per-model charging means watching catalog growth