LLM Gateway vs LLM Proxy

The short version

An LLM proxy forwards requests from your app to LLM providers. It sits in the request path and can modify, log, cache, or reroute traffic in real time. Think of it as a smart relay.

An LLM gateway is a proxy plus a control plane. It adds authentication, rate limiting, billing, multi-tenant isolation, and policy enforcement on top of the basic routing. Think of it as the proxy plus everything around it.

Every gateway contains a proxy. Not every proxy is a gateway.

When you need just a proxy

You have one team, one app, and one or two providers. You want to swap between OpenAI and Anthropic without changing application code. You want cost tracking and maybe caching. You do not need user management, billing, or multi-tenant isolation.

A proxy-only setup gets you model aliasing, provider failover, request logging, and basic cost tracking with minimal overhead. Stockyard calls this proxy-only mode.

When you need a gateway

Multiple teams share the same LLM infrastructure. You need per-team API keys, spend caps, rate limits, and audit trails. You are building a platform where internal or external users consume LLM capabilities through your API.

A gateway gives you the proxy layer plus identity, authorization, metering, and policy. Stockyard ships both modes in the same binary. Start with proxy-only, toggle on gateway features when you need them.

Architecture comparison

The Stockyard approach

Stockyard ships both in a single binary with embedded SQLite. There is no separate "gateway edition." You start with the proxy, and every gateway feature is a module you toggle on at runtime. No config file changes, no restart, no redeploy.

This means you do not have to predict your architecture up front. Start routing traffic through the proxy today. Add team isolation, spend caps, and audit logs when the need arises. The binary is the same either way.