A hospital in Leeds wants to screen 40,000 chest X-rays a month with a model that catches early signs of pneumonia. Buying the GPUs to run it would blow the radiology department’s equipment budget several times over. Renting equivalent capacity costs a few thousand pounds a month and drops to near zero on quiet weekends. That swap, someone else’s hardware for your model, is roughly what the AI cloud is.
It sounds simple. In practice the term covers everything from bare metal chips rented by the minute to a single API call that returns a paragraph of text. Knowing which layer you actually need saves more money than any discount negotiation.
What the AI cloud actually means
Ordinary cloud computing grew up around web servers: lots of small machines handling lots of small requests. AI workloads break that mould. Training a frontier model means keeping tens of thousands of chips fed with data at the same instant, and one stalled worker can waste hours of expensive compute. Inference is the opposite problem, huge spikes of demand that go almost silent at 3am.
Three layers have formed to serve those needs.
- Raw infrastructure. GPU or TPU instances you configure yourself, usually with high-speed networking and parallel file systems attached. Maximum control, maximum responsibility.
- Managed AI platforms. Vertex AI, Amazon Bedrock and Azure AI Foundry sit on top of the hardware and handle scaling, monitoring, model versioning and deployment pipelines.
- Model APIs. Per-token access to a hosted model. No clusters, no driver updates, no capacity planning. You pay for what you generate.
Most companies land somewhere in the middle. A couple of engineers spin up raw instances for fine-tuning while the product team calls a hosted model for the customer-facing feature, and nobody writes a single CUDA kernel.
Why renting beat buying
An Nvidia H100 card costs somewhere between $25,000 and $40,000, and a serious training cluster uses thousands of them. Add power, cooling, the building and the switching gear, and you’re into nine figures before anyone writes a line of model code. The hardware is also obsolete in 18 to 24 months whether you used it or not.
Renting shifts three risks off your balance sheet: the capital outlay, the depreciation curve and the utilisation problem. A team training one model a quarter might need 512 accelerators for six weeks and nothing for the rest of the year. On a cloud contract that is six weeks of billing. On owned hardware it is 512 chips sitting idle for ten months.
Three ways teams use AI cloud capacity
Training and fine-tuning
Long, expensive, bursty jobs. Reserved capacity or multi-year commitments cut the hourly rate by 40 to 60%, but only if you have a genuine pipeline of work to fill it. Spot instances cost far less and get reclaimed with a few seconds’ notice, which is workable if your framework checkpoints every few minutes and resumes cleanly.
Inference in production
This is where most of the money now goes. A customer support assistant handling two million conversations a month can burn more compute than the training run that produced it. Batching, quantisation and prompt caching are the levers that decide whether that bill is $4,000 or $40,000, and they sit squarely in engineering territory, as this breakdown of how AI programming tools and agents have changed makes clear.
Generative media
Image and video generation is the heaviest per-request workload most teams touch. A single high-resolution render can hold a GPU for several seconds, and users rarely stop at one. Guides such as this practical guide to creating art with generative tools spend as much time on batching and selection as on prompts, for exactly that reason.
Document and language pipelines
Quieter, but often the highest-return use of hosted AI services: contracts, invoices, scans and recordings pushed through cloud vision and speech models. It’s the same plumbing behind tools that read, summarise and speak documents aloud, and it rarely needs a single dedicated GPU.
How the major providers actually differ
The marketing says they’re interchangeable. The details disagree. Google’s edge has traditionally been TPUs and data-heavy pipelines, AWS has the widest catalogue of model choices through Bedrock, and Microsoft leans on its OpenAI relationship plus tight Azure integration. For a grounded view of one of those stacks, Google Cloud AI explained without the hype is a better starting point than any sales deck.
Three practical differences matter more than brand:
- Which accelerators are available in your region, and at what price per hour.
- Whether your data is allowed to leave the country, which rules out certain providers for healthcare and public sector work.
- How painful it would be to leave. Containerised workloads travel well. Proprietary pipeline tools do not.
The costs that never make the pricing page
The headline rate per GPU-hour is the least interesting number on the invoice. Data egress runs $0.08 to $0.12 per gigabyte on most providers, and dataset-heavy training jobs move terabytes. Checkpoint storage accumulates quietly at a few dollars per hundred gigabytes per month. An always-on inference endpoint bills around the clock even at zero traffic, so a staging environment left running over a long weekend can cost more than the feature it was testing.
The sneakiest charge is often the human one. Teams that adopt three platforms, two model gateways and a custom orchestration layer end up paying two engineers to maintain glue code that a single managed service would have replaced.
Where this is heading next
Two shifts are already visible. First, inference is moving closer to the user, onto edge devices and factory floors, because round-tripping every camera frame to a distant data centre adds latency and raises privacy questions. The same dynamic is pushing intelligence into physical machines, as covered in this look at why AI robots are finally doing real work.
Second, the accelerator market is fragmenting. Frontier training still needs the biggest chips available, but a growing share of inference runs perfectly well on cheaper, lower-power silicon. Providers that let you move a workload between chip types without rewriting it will win the next round of contracts.
How to choose without overthinking it
Match the layer to your team’s appetite for infrastructure work. If nobody wants to debug NCCL timeouts on a Friday night, stay on a managed platform or a model API and accept the premium. If you’re training regularly and have platform engineers on staff, reserved raw capacity will pay for itself within a quarter.
Then pressure-test three things before signing anything: a realistic monthly bill that includes egress and idle endpoints, a migration plan that assumes you’ll want to leave in two years, and a latency test run from the regions your users actually occupy. Providers are remarkably similar on the slide and remarkably different at 2am when your endpoint is cold and the queue is backing up.