Apache APISIX AI Gateway (US, Apache Software Foundation) is the AI layer of APISIX, an open-source API gateway used in production for HTTP and streaming traffic. The AI plugins transform each request into the provider’s format and normalize the response.
What it does
- Exposes a single endpoint for different providers, in OpenAI-compatible format
- Routes between models with simple, weighted or semantic-similarity rules
- Falls back automatically when a provider returns an error, rate limit or health failure
- Balances with weighted round robin, consistent hashing and priority across instances
- Enforces token-based limits and exact or semantic caching
- Adds prompt guardrails, content moderation and external security integrations
- Reports token, model, latency and time-to-first-token metrics through Prometheus or OpenTelemetry
- Supports RAG flows through the context-retrieval plugin
How it works
- The ai-proxy and ai-proxy-multi plugins attach to APISIX routes
- Provider credentials stay in the gateway, not in the application
- Routing follows rules declared in configuration or in the control plane
- The gateway runs in a container, on Kubernetes or on your own machine
- Detailed cost governance such as billing and chargeback stays outside the gateway
Models and providers
- Named providers include OpenAI, Azure OpenAI, Anthropic, DeepSeek, Gemini, Vertex AI, Bedrock and OpenRouter
- Any OpenAI-compatible endpoint can be used
- The APISIX plugin catalog covers authentication, limits, transformation and observability
Availability and license
- Apache License 2.0, with no license fee
- Self-hosted install on Docker, Kubernetes or bare metal
- No SaaS offering maintained by the project itself
Points to note
- Cost control is partial: pricing, budgets and chargeback rely on other tools
- It requires operating and maintaining an API gateway
- Advanced features depend on the version and enabled plugins