BlogHow Do Large AI Models Like Claude, ChatGPT and Gemini Actually Run in Data Centers?
DevOps & Hosting8 min read

How Do Large AI Models Like Claude, ChatGPT and Gemini Actually Run in Data Centers?

A local AI model runs on one graphics card in your computer. Claude, ChatGPT and Gemini run on thousands of interconnected chips in data centers around the world. Here is what happens to your prompt from the moment you hit "Send" until the answer streams back word by word, and how one model serves millions of people at the same time.

Bahaa Esmail
ISMS & DevOps Lead
Data center hall of GPU racks receiving prompts from phones and laptops around a glowing globe

If you have ever run an AI model on your own machine with tools like Ollama or LM Studio, you know the picture: a weights file of a few gigabytes, one graphics card, and one user, which is you. So what happens when you type a question into Claude, ChatGPT or Gemini? Where does it go? Is there a "copy" of the model reserved for you? And if someone in Riyadh and someone in Cairo ask a question at the same second, are they talking to the same model?

The short answer: there is one model, but it runs as many identical copies on large clusters of chips in several data centers. Your request is routed to one of those copies, and that copy serves you and hundreds of other people at the same time without mixing up your conversations. This article walks through how that works, step by step.

Why one computer is not enough to run a model like Claude

At its core, a language model is a huge file of numbers called weights or parameters. Every parameter takes up memory. An open model like Llama 3 with 70 billion parameters needs about 140 GB at FP16 precision (two bytes per parameter), while the most powerful consumer graphics cards hold only 24 to 32 GB. That is why local models usually run as compressed (quantized), smaller versions.

Frontier models like Claude, GPT and Gemini are estimated at hundreds of billions, possibly trillions, of parameters (the companies do not publish exact figures). For comparison, the open Llama 3.1 405B model needs around 810 GB at FP16, and DeepSeek-V3 has 671 billion parameters. Even a top data center chip such as the NVIDIA H100 holds only 80 GB. The result: one model does not fit on one chip. It has to be split across several chips that work together as if they were one machine.

A local model running on a single GPU compared with a frontier model split across many GPUs
A local model running on a single GPU compared with a frontier model split across many GPUs

The hardware: not ordinary servers, but racks that act as one processor

In modern AI data centers the basic unit is no longer a single server but an entire rack. Three families of chips dominate:

  • NVIDIA GPUs: such as the H100 and B200. The GB200 NVL72 system, for example, packs 72 Blackwell GPUs into one liquid-cooled rack with roughly 13.8 TB of combined memory, all working as one giant accelerator.
  • Google TPUs: chips designed specifically for Transformer math, grouped into pods of hundreds to thousands of chips linked by Google's own optical network. Gemini runs primarily on these.
  • AWS Trainium: Amazon's own AI chips. Anthropic has stated in a published engineering post that it serves Claude across three platforms: AWS Trainium, NVIDIA GPUs and Google TPUs.

High-bandwidth memory (HBM): the real bottleneck

Speed is not only about raw compute. It is about how fast the weights can be read from memory. To generate a single word, the chip has to read almost all of the model's weights. That is why HBM memory sits directly next to the processor, delivering several terabytes per second, dozens of times faster than the memory in a regular PC.

The network: how the chips talk to each other

Once a model is split across chips, those chips must exchange results thousands of times per second. Inside a rack they use ultra-fast NVLink connections (up to 1.8 TB/s per GPU in the latest generation). Between racks, data centers use high-speed InfiniBand or Ethernet fabrics.

Scroll horizontally to see all columns

Comparison table
PlatformChips per unitMemory per chipInterconnect
NVIDIA HGX H1008 per server80 GB HBM3NVLink 4 (900 GB/s)
NVIDIA GB200 NVL7272 per rack192 GB HBM3eNVLink 5 (1.8 TB/s)
Google TPU v6e256 per pod32 GB HBMOptical ICI network

How a model is split across chips

There are three main techniques, and production systems usually combine them:

  1. Tensor parallelism: every layer of the model is sliced so each chip computes part of the same operation, then the results are combined. It needs extremely fast links, so it is used inside a single rack.
  2. Pipeline parallelism: the model's layers are distributed in sequence. The first layers live on one group of chips, the next layers on another, like a factory assembly line.
  3. Expert parallelism: many modern models use a Mixture of Experts design. The model contains dozens of specialized sub-networks, and only a few of them run for each token. In DeepSeek-V3, for example, only 37 billion of its 671 billion parameters are active per token. The experts are spread across different chips, and each token is sent to the experts it needs.

Scroll horizontally to see all columns

Comparison table
TechniqueWhat is splitWhere it usually runsMain challenge
Tensor parallelismInside each layerWithin a rack (NVLink)Interconnect speed
Pipeline parallelismLayers in sequenceAcross serversIdle gaps between stages
Expert parallelismSpecialized sub-networksWithin and across racksBalancing load across experts

The group of chips that holds one complete copy of the model is called a replica. And that brings us to your real question.

Your prompt's journey: from "Send" to the answer on your screen

A request travelling from the user's laptop through an API gateway and load balancer to one model replica, with the answer streamed back
A request travelling from the user's laptop through an API gateway and load balancer to one model replica, with the answer streamed back
  1. Encrypted connection and authentication: your prompt leaves your device over an encrypted HTTPS connection to the provider's nearest entry point, where your account or API key is verified.
  2. Usage limits: the API gateway checks that you have not exceeded your quota, which is usually measured in tokens, not just requests. Under heavy load you may see an "overloaded" message or a 429 error.
  3. Tokenization: your text is cut into small units called tokens, and each token becomes a number. The model never sees letters, only these numbers.
  4. Routing to a replica: a load balancer picks one of the identical model replicas based on region, how busy each replica is, the length of the conversation, and sometimes which replica already has parts of your conversation cached.
  5. Prefill (reading): the replica reads your entire prompt, plus the conversation history, in one parallel pass. This stage is compute-heavy, and its duration decides how long you wait for the first word.
  6. Decode (generating): the model generates the answer one token at a time, each new word depending on the ones before it. This stage is limited by memory speed more than compute.
  7. Streaming: every generated token is turned back into text and sent to you immediately. That is why you see the answer being typed out rather than waiting for all of it.

Key point: the model does not "remember" you between messages

The model itself has no lasting memory between requests. When you send a new message, the app sends the whole conversation (or a summary of it) along with your new message, and the model reads it again. So your second message may land on a completely different replica than the first, and you would never notice. To speed this up, systems temporarily keep the computations for the start of a conversation (prefix caching) and try to route your follow-up messages to the same replica to reuse them.

Are you and everyone else talking to the same model?

Yes and no. The weights are shared: everyone uses the same model with the same numbers, and they are read-only, so your conversation does not change them. The copies are many: providers run hundreds or thousands of identical replicas spread across racks and data centers in different regions. When demand rises, more replicas are brought online. When it drops, the chips are reassigned to other work.

World map with data centers in different regions, each running an identical copy of the model
World map with data centers in different regions, each running an identical copy of the model

Replicas can also be configured for different jobs. In an engineering postmortem published in September 2025 about three bugs that affected Claude's output quality, Anthropic explained that some short requests were mistakenly routed to servers configured for a one-million-token context window, and that routing was "sticky": a user whose request landed on the wrong pool tended to stay there for follow-up messages. It is a real example of the multiple server pools and complex routing that sit behind a simple chat box.

How one replica serves hundreds of users at the same moment

If every replica served only one user, the cost would be absurd. The answer is batching: requests from dozens or hundreds of users are merged into one batch, because reading the weights from memory once to serve a hundred users is far more efficient than reading them a hundred times.

Requests from many users merged into one batch inside a server, with separate memory per conversation
Requests from many users merged into one batch inside a server, with separate memory per conversation
  • Continuous batching: instead of waiting for every request in a batch to finish, a finished request leaves immediately and a new one takes its place at the next step. Engines such as vLLM, SGLang and TensorRT-LLM are built around this.
  • Per-conversation memory (KV cache): the only thing inside the replica that belongs to you is your conversation's working memory, stored results that save the model from re-reading everything for every new word. It is large: for a 70-billion-parameter model, one 8,000-token conversation needs about 2.7 GB, so serving 64 users at once takes more than 170 GB for this memory alone.
  • Paged memory (PagedAttention): an idea borrowed from operating systems. Conversation memory is split into small pages instead of reserving one large contiguous block per user, so each chip fits many more conversations.
  • Isolation between users: each conversation's memory is kept separate. Even when several requests share an identical fixed part (such as the same system instructions), the shared part is read-only, and any new text that belongs to you is written to your own space.

Separating reading from generating

Because prefill needs raw compute and decode needs memory speed, newer systems split them: one pool of chips reads the prompts, then ships the conversation memory over the network to another pool that specializes in generating. This cuts waiting time and raises how many users each rack can serve.

Power and cooling: the hidden side of every answer

A single GB200 NVL72 rack draws about 120 kilowatts, the equivalent of dozens of racks in an older data center. Air cooling can no longer keep up, so liquid cold plates are mounted directly on the chips, with closed loops that carry the heat to the building's water systems. That is why AI companies look for data center sites close to power sources, and why the cost of each answer depends on the model's size and the length of the prompt and the response.

Scroll horizontally to see all columns

Comparison table
On your computerIn the data center
A compressed model on one graphics cardA full model split across 8 to 72 chips or more
One userHundreds of users per replica, thousands of replicas
No networkNetworks moving terabytes per second
A cooling fanLiquid cooling and power systems in the hundreds of kilowatts
Full privacy on your deviceYour data passes through the provider's servers under its policies

What this means for business owners

Understanding this architecture helps you make better decisions when you bring AI into your product or website:

  • Using a ready-made API from Anthropic, OpenAI or Google gives you the most capable models with no hardware investment, paid per use.
  • Response speed depends on the data center closest to your customers and on how much content you send, so trimming unnecessary context and caching fixed instructions both help.
  • Rate limits are real: your application should handle "overloaded" responses and retries gracefully.
  • Local or self-hosted models make sense when data privacy is the top priority, as long as you accept a smaller model or higher hardware costs.

At Webdivs we help companies in Egypt and the Gulf bring AI into their websites and apps in a practical way: from choosing the right model and connecting it through an API, to building Arabic-speaking assistants that serve your customers around the clock.

Sources

Frequently Asked Questions

Quick answers about this topic

No. There are many identical copies of the model, and each one serves hundreds of users at the same time. The only thing that is yours is your conversation's temporary memory.

The model's weights do not change during a conversation. It reads the conversation history that is sent with each message. Whether conversations are used to train future versions depends on each company's policies and your settings.

Because the model generates text one token at a time, and each token is sent to you as soon as it is generated through streaming, instead of waiting for the full answer.

Because the number of chips available at any moment is limited. When demand exceeds capacity, systems protect service quality by temporarily rejecting or delaying some requests.

No. Frontier models like Claude, GPT and Gemini are not available to run locally, and their size requires dozens of specialized chips. You can, however, run smaller open models such as Llama, Qwen or Gemma on a powerful machine.

Want this for your product?

Send a short note about your project. We will review it and explain the next useful step.

Contact Our Team