CMP-02Compare · Infrastructure economics
On-premises AI infrastructure means owning the accelerators and the facility that powers and cools them; cloud AI infrastructure means renting that capacity by the hour inside someone else's facility. The choice is decided mostly by one number — sustained utilization — with data residency, latency, and burst profile acting as overrides that can flip the answer regardless of what the economics say.
PUBLISHED LAST VERIFIED BY JOSEF ELIMELECHREVIEWED PODOS AI ENGINEERING
The honest verdict
The deciding variable
Measuring that number honestly is harder than it looks, because allocated capacity is not consumed capacity. The PagedAttention work behind vLLM found that serving systems without paged memory management used roughly 20–38% of allocated KV-cache memory for actual token state (2023 measurements).[8] A cluster whose dashboards report full allocation can still be doing a fraction of the work its memory footprint implies. Size an ownership decision against measured throughput, not against an allocation graph.
The macro backdrop pushes in the same direction. The IEA projects data-centre electricity demand roughly doubling as a share of global consumption — from about 1.5% in 2025 toward about 3% by 2030 — with AI the dominant driver (Apr 2025).[1] LBNL put US data centers at 4.4% of national electricity in 2023 and projected 6.7–12% by 2028 (Dec 2024).[2] That growth is what makes accelerator capacity scarce and priced accordingly in both models; it does not by itself favour either one.
Table 1 · The comparison matrix
Cloud wins more rows than infrastructure vendors usually admit. The rows on-prem wins are the ones that tend to be non-negotiable when they apply.
| # | Dimension | Cloud | On-prem | Structural advantage |
|---|---|---|---|---|
| OC-01 | Utilization economics | Pay for hours consumed; idle capacity costs nothing. | Cost is fixed regardless of duty cycle; unit economics improve as utilization rises. | Cloud below your break-even duty cycle, on-prem above it. |
| OC-02 | Cost predictability | Variable, with committed-use discounts that reintroduce fixed cost by another name. | Known capital and facility cost across the asset life. | On-prem. |
| OC-03 | Burst capacity | Effectively unbounded within quota and region limits, available in minutes. | Hard-bounded by installed capacity; more capacity means procurement lead time. | Cloud, decisively. |
| OC-04 | Time to first capacity | Minutes to hours. | Weeks to years, depending entirely on whether power and space already exist. | Cloud. |
| OC-05 | Data residency and custody | Jurisdiction is selectable by region; physical custody stays with the provider. | Legal and physical custody stay inside your own perimeter. | On-prem where the requirement is custody rather than jurisdiction. |
| OC-06 | Latency and data gravity | Adequate for most inference; always a network round trip from the data source. | Compute can sit beside the dataset or the process it controls. | On-prem for control-loop and very large local datasets. |
| OC-07 | Facility efficiency | Hyperscale plants report best-in-class overhead — a 1.09 fleet trailing-twelve-month PUE at Google and a 1.12 design PUE at Microsoft.[5][6] | Entirely dependent on the facility; industry-average PUE has been essentially flat for about six years (Uptime, 2025).[3] | Cloud, unless the on-prem facility is purpose-built for the load. |
| OC-08 | Team requirements | Provider owns power, cooling, and hardware lifecycle; you still own the platform layer. | You own the facility, the hardware, and the platform layer. | Cloud. |
| OC-09 | Hardware choice and refresh risk | New accelerator generations appear without a purchase decision; you never hold the depreciating asset. | You pick the exact configuration and keep it as long as it earns. | Split: cloud for optionality, on-prem for control. |
Both directions
Read down each column before running any arithmetic. Two of these four cards decide the question on their own, whatever the break-even says.
Variable and bursty workloads are the clearest case. If demand swings by an order of magnitude between a quiet week and an evaluation sweep, owned capacity is either idle most of the time or too small when it matters — and cloud absorbs that swing in minutes.
Data residency is two requirements wearing one name. Jurisdictional residency — data must remain inside a legal boundary — is solved by choosing a cloud region. Custody — the equipment, the physical access, and the operators must be yours — is not. Defence work, some clinical and biometric data, certain industrial process data, and contracts that forbid third-party physical access all fall in the second category, and no region selector satisfies them. Read the obligation carefully before assuming it forces an on-prem build; read it just as carefully before assuming a region satisfies it.
Small scale is the second case: below roughly a rack of accelerators, the fixed overheads of power, cooling, spares, and staffing dominate the hardware cost, and no utilization rate rescues the arithmetic. Cloud also wins on optionality and on facility efficiency. Teams still deciding which accelerator generation suits their models should not be holding a depreciating asset while they find out. And a hyperscale plant runs at overheads a typical enterprise room does not reach: Google reports a fleet-wide trailing-twelve-month PUE of 1.09 and Microsoft a design PUE of 1.12 with a water-use effectiveness of 0.30 L/kWh, while Uptime's 2025 survey of more than 800 operators found industry-average PUE essentially flat for about six years.[5][6][3] If the on-prem option is a converted server room, the efficiency comparison is not close.
Latency splits the same way. For most inference, a network round trip is irrelevant next to model execution time. It stops being irrelevant when the model sits inside a control loop — a production line, a robot, a diagnostic instrument, a trading path — where the round trip is a hard budget rather than a nuisance. Data gravity is the quieter constraint: when a dataset is large enough, or regenerated often enough, moving compute to the data is cheaper than moving the data to the compute, and that calculation lands on-prem more often as datasets grow. The use-case breakdown walks through which workloads sit on which side of that line.
The reasoning
The capex-versus-opex framing flatters both sides when it is done loosely. On-prem comparisons understate cost by pricing hardware and forgetting the facility layer underneath it — power, cooling, floor space, spares, and staff — which is why BE-03 and BE-04 exist in the worksheet below. Cloud comparisons understate cost by pricing on-demand rates while ignoring egress, storage, idle reservations, and the committed-use contracts most large consumers sign.
The second honest adjustment is on the utilization side. Software that raises the work extracted per accelerator changes the break-even point for both models at once — memory-efficient serving, batching, and compression each move the same lever, which is why Syntropic sits alongside the hardware rather than after it.
Cloud outsources the facility, not the platform. Someone on your side still owns scheduling, images, drivers, networking, observability, and cost control. On-prem adds a second discipline on top: electrical and mechanical systems, hardware RMA cycles, spares inventory, and a maintenance calendar. Both models need enough depth that no single person is the only one who understands the system — Uptime's 2025 survey reports staffing and skills among operators' standing concerns alongside outage experience, with roughly half of respondents reporting an impactful outage within three years.[3] Resilience is bought deliberately in either model; neither includes it for free.
A three-year committed spend is not elastic. It is a fixed obligation recorded in a different place on the income statement, and it should be compared against owned capacity as such.
Table 2 · What this comparison assumes
Published cost comparisons rarely survive contact with a real workload because they omit one of these seven inputs. Fill them in with your own numbers before reading any vendor's. The right-hand column is the error we see most often in each row.
| # | Input | How to measure it honestly | Common error |
|---|---|---|---|
| BE-01 | Sustained utilization | Accelerator-hours doing useful work divided by accelerator-hours owned, averaged over a full quarter including holidays and refactors. | Quoting a peak week, or counting allocated capacity as consumed capacity. |
| BE-02 | Competitive life of the hardware | The number of years the accelerator stays economically useful for your workload — which is a market question, not a tax-schedule question. | Borrowing a general-purpose server depreciation schedule for accelerators. |
| BE-03 | All-in facility overhead | Power, cooling, floor space, and the PUE multiplier applied to every IT watt, plus spares and maintenance contracts. | Comparing a rented GPU-hour against a bare hardware purchase price with no facility layer. |
| BE-04 | Fully loaded staffing | Facility, hardware, and platform skills at loaded cost, including on-call coverage and the cost of not having a second person who knows the system. | Assuming the existing IT team absorbs an AI facility at zero marginal cost. |
| BE-05 | Shape of current cloud spend | The share of cloud spend already locked into reserved or committed-use contracts versus true on-demand. | Treating committed cloud spend as elastic when it behaves like a fixed obligation. |
| BE-06 | Data movement | Egress volume, dataset staging time, and cross-region replication for every training and evaluation cycle. | Modelling storage cost but not the repeated cost and latency of moving data to the compute. |
| BE-07 | Burst profile | Peak-to-median demand ratio, how often peaks occur, and how long they last. | Sizing owned capacity to the peak instead of to the median with a burst path attached. |
Honest limits
If any of these describe your situation, the honest recommendation is cloud — or a hybrid that keeps the owned footprint small.
The usual answer
Most organisations past the experimentation stage end up owning the predictable baseline and renting the peaks. Size owned capacity to sustained median demand, keep a cloud path for burst training runs, evaluation sweeps, and unexpected inference spikes, and re-measure the split whenever the workload or the hardware generation changes. That structure captures the utilization advantage of ownership without paying for capacity that exists only to survive a peak — and it keeps the residency-sensitive fraction of the data inside a perimeter you control while everything else stays elastic.
Two of those constraints have external evidence worth reading directly: the IEA's reporting on grid-connection bottlenecks driving the current scramble for capacity (2025)[4], and ASHRAE TC 9.9's white paper on why liquid cooling is displacing air at high rack densities[9], alongside NVIDIA's own specification for a 72-GPU liquid-cooled rack acting as one NVLink domain.[7] Our direct-to-chip liquid cooling explainer covers the second constraint end to end, and the power architecture page covers the first.
PODOS is on the on-prem side of the line, aimed specifically at the objections above that are about facilities rather than economics. Each PODOS Pod is designed as a standardized 1 MW building block and designed for 128 GPUs, with power and closed-loop liquid cooling integrated in the factory rather than built on site — so the "the room cannot host it" and "nobody owns the facility" constraints are answered by the product instead of by a construction project. PODOS targets a 90-day window from order to commissioning for a standard unit, which is the lever that matters against OC-04: shortening time-to-capacity is how owned infrastructure stops conceding that row by default.
It does not change the arithmetic in BE-01. If sustained utilization is low, renting is still the right call, and we would rather say so than sell a unit that runs idle. For the adjacent comparison — owning capacity through factory-built units versus a conventional facility — see modular vs traditional AI data centers. The deployment model explains how a unit reaches a site, the platform overview shows how units compose, and unfamiliar terms are defined in the AI infrastructure glossary.
QUESTIONS
Only above a break-even duty cycle. Owning hardware converts a variable cost into a fixed one, so on-prem economics improve as sustained utilization rises and get worse as it falls. Below that threshold — and for workloads that are bursty, seasonal, or still being explored — cloud is usually the cheaper and lower-risk option.
It satisfies a jurisdiction requirement, not necessarily a custody requirement. Selecting a region keeps data inside a legal boundary, but the equipment, the physical access controls, and the operational staff remain the provider's. Where a regulation, contract, or classification demands physical custody, on-premises infrastructure is the only architecture that provides it.
The common pattern is to own the predictable baseline and rent the peaks. Size owned capacity to sustained median demand, keep a cloud path for burst training runs, evaluation sweeps, and unexpected inference spikes, and re-measure the split each time the workload or the hardware generation changes.
Usually not without significant work. Rack-scale AI systems such as NVIDIA's GB200 NVL72 ship liquid-cooled and behave as a single NVLink domain, and ASHRAE's TC 9.9 has documented why air cooling stops being economic at those densities. Power capacity, floor loading, and a liquid-cooling path are the constraints that most often rule a legacy room out. Both points are sourced in the reference list at the foot of this page.
If sustained utilization does not clear the break-even, the honest answer is cloud — and you will hear it from us.