Are You Testing Your LLM Gateway Like Any Other Critical Application?

Richard Tan

Jul 9, 2026 — 5 min read

As organisations continue adopting AI, one challenge quickly emerges: how do you manage access to multiple large language models (LLMs) in a secure and controlled way?

For many organisations, the answer is an LLM gateway.

If you're unfamiliar with the term, think of an LLM gateway as a central control point between users, applications, and AI models. Rather than every application connecting directly to OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, or self-hosted models, requests are routed through the gateway first.

This allows organisations to centralise authentication, access control, model routing, logging, rate limiting, cost management, and policy enforcement across different teams and systems. In some cases, the gateway also sits behind customer-facing products, helping organisations control how AI features are delivered at scale.

0:00

/0:12

How an LLM gateway sits between users and AI providers - and what it's responsible for

From a security and governance perspective, this approach has clear advantages.

Rather than managing controls across multiple AI platforms, everything can be managed from a single location.

But centralising access raises an important question:

Are organisations testing these gateways with the same level of scrutiny they apply to other critical applications?

In many cases, the answer is often no.

Most security reviews focus on the obvious things: authentication, API keys, user permissions, and logging. These controls are still essential, but LLM gateways introduce risks that don't typically exist in traditional applications.

The reason is simple.

The model itself becomes part of the attack surface.

Not All Models Behave the Same Way

One of the biggest differences between an LLM gateway and a traditional API gateway is that the underlying models may behave very differently from one another.

Many gateways expose multiple models through a single interface. Some may have strong safety controls. Others may be open-source models with fewer restrictions. Some may be intended for specific teams or internal use only.

To the user, they may all appear to be accessible through the same platform.

Behind the scenes, however, they can produce very different outcomes.

We've observed environments where users could switch between available models and receive completely different responses to the same prompt.

For example, in one trading-related environment, internal policies prohibited providing direct buy or sell recommendations. Some models correctly refused these requests or redirected users toward general educational information.

Other accessible models were far less restrictive and would provide explicit trading recommendations when asked.

Nothing was hacked. No vulnerability was exploited.

The issue was that security and policy controls were assumed to apply consistently across all available models when, in reality, the models behaved very differently.

This is an important distinction. If users can freely switch between models, the same restriction may be enforced consistently in one model but not reliably in another due to differences in model behaviour.

0:00

/0:10

Two tests. Four models. Inconsistent results

Testing should verify not only that users can access the correct models, but also that organisational policies are applied consistently across every model available through the gateway.

Model Enumeration and Access Controls

Another area worth examining is model access.

In many environments, not every user should have access to every available model. Some models may be reserved for specific teams, contain experimental capabilities, or incur significantly higher costs.

If access controls are not properly enforced, users may be able to discover and invoke models they were never intended to use.

At first glance this may seem like a straightforward permissions issue, but the impact can extend beyond security.

In some cases, a user could switch to a higher-cost or unintended model simply by changing the model identifier in their request.

Again, nothing needs to be “compromised” in the traditional sense. The gateway may still accept the request, even if the selection was not intended under policy.

In some cases, the bigger concern isn't cost.

Different models often respond differently to the same prompt. In regulated environments, those differences can determine whether internal policies are enforced or bypassed entirely.

What starts as an access control issue can quickly become a governance problem.

Traditional Rate Limiting Isn’t Enough

Most organisations already understand the importance of rate limiting.

The challenge with LLMs is that not all requests are equal.

A hundred short prompts may consume fewer resources than a single request containing a large amount of context and a high output token limit.

While model providers do enforce hard limits such as maximum context size and output length, these are technical constraints rather than usage or cost controls.

From a gateway’s perspective, requests may still be treated as a single unit, even though their underlying token usage and cost can vary significantly.

This means traditional request-based limits are not always sufficient on their own.

Testing should verify whether the gateway considers token usage, model-specific restrictions, and overall cost controls, rather than simply counting requests.

Without these controls, a small number of requests can still lead to unexpectedly high usage and cost.

Information Disclosure Through Error Messages

Error handling is another area that is often overlooked.

Like any application, LLM gateways can unintentionally reveal information through verbose error messages.

Depending on how the gateway is configured, error responses may expose internal model names, provider details, deployment information, configuration settings, or backend architecture.

Individually, these disclosures may not seem significant.

However, when combined, they can provide useful intelligence about the environment and help an attacker better understand the systems they're interacting with.

Error responses should be reviewed with the same scrutiny applied to any internet-facing application.

Visibility Matters

One challenge we frequently see in AI environments is limited visibility.

Security teams are accustomed to reviewing authentication logs, application logs, and audit trails when investigating incidents.

Many LLM deployments provide far less visibility into prompts, model selection, token consumption, policy violations, or generated outputs.

When something goes wrong, organisations often discover they don't have enough information to understand what happened.

Good logging isn't just useful for compliance.

It helps identify misuse, investigate incidents, understand spending patterns, and detect attempts to access models that users shouldn't be interacting with.

You can't effectively secure what you can't see.

What Should Organisations Be Testing?

If you're deploying an LLM gateway, it's worth remembering that it isn't just a simple API proxy.

It's a security and governance layer sitting in front of multiple models with different capabilities, behaviours, costs, and risk profiles.

At a minimum, security testing should include:

Authentication and authorisation controls
Model enumeration and access restrictions
Policy consistency across different models
Token-aware rate limiting and cost controls
Information disclosure through error handling
Logging, monitoring, and audit capabilities

Many of these checks won't appear in a traditional web application assessment, but they're becoming increasingly important as AI adoption grows.

Final Thoughts

LLM gateways solve a real problem. They give organisations a single place to manage who can access what, at what cost, under what controls.

But standing one up doesn't mean it's secure.

Unlike a traditional API, an LLM gateway isn't just exposing endpoints. It's exposing access to models that can behave very differently from one another. That makes the model itself part of the attack surface.

Most organisations haven't started testing these gateways properly yet. As AI becomes more embedded in how businesses operate, that gap is going to matter more.

The question isn't just whether your users can reach the platform. It's whether they can reach the right models, under the right controls, with the right safeguards in place.