“A chatbot” and “production enterprise chat” are not the same thing

Wiring a chat box to a language model is a weekend project. You can have something that looks impressive in a demo by Friday afternoon.

Production enterprise chat — the kind you can put in front of real users and an auditor — is a full distributed system. The model is one box on the diagram; almost all of the work, the cost, and the trust lives in the architecture around it. Here's roughly what the real version looks like on Azure.

One public front door

There is exactly one way in: an Application Gateway with a Web Application Firewall and DDoS protection. Everything behind it is private. That single, hardened entry point is what lets you reason about the attack surface at all — you're defending one door, not a dozen accidental ones.

The app tier, locked down

The chat UI runs on App Service and calls the agent over a private endpoint — the traffic never traverses the public internet. It authenticates with a managed identity, so there are no keys or secrets floating around in config to leak or rotate. Identity, not a shared key, is what one component uses to talk to the next.

The agent that orchestrates

Behind the UI is an agent service that does the actual orchestration: it reasons over the prompt, calls the model, and invokes tools as needed. This is the part people picture when they say "the AI," but on the diagram it's a coordinator — it decides what to do and then calls the pieces that do it.

Grounding — and controlled egress

Answers are grounded through Azure AI Search (RAG), so the system responds from your actual content rather than the model's memory. And any call that needs to leave the network is forced through a firewall for egress control — you decide exactly what the system is allowed to reach on the way out, which is as important as controlling what gets in.

State, secrets, and safety

Conversation history lives in Cosmos DB, files in Storage, and document chunks in the search index — each with a clear home rather than smeared across the app. Secrets sit in Key Vault, content safety screens inputs and outputs, and observability is wired across every component so you can actually see what happened when someone asks.

Wrapped in a private network

All of it sits inside a virtual network with Private Link. Private by default. Zone-redundant. Identity on every hop. None of these are features a user sees — they're the properties that let a regulated business run the thing without holding its breath.

The value is in the “boring” part

The gap between the demo and this is exactly the gap between "cool prototype" and "something a regulated business can actually run." Most of the cost and most of the value is in the boring infrastructure around the model — the front door, the private network, the grounding, the audit trail.

So when someone shows you a slick AI chat demo, the useful question isn't about the model. It's: what does the production architecture behind it look like?