Syntropic: the inference-efficiency software layer

Large language model inference is frequently bound by memory, not arithmetic. During generation, a transformer stores key and value tensors — the KV cache — for every token, in every attention layer, so earlier tokens do not have to be recomputed at each decoding step. That cache grows linearly with sequence length and with the number of concurrent requests, and it competes for the same GPU memory that holds the model weights.^[1]

The result is a throughput ceiling that has little to do with raw compute. Research on production serving systems measured that pre-paging LLM serving frameworks used as little as roughly 20 to 40 percent of allocated KV-cache memory for actual token states, with the remainder lost to fragmentation and over-reservation (Kwon et al., 2023).^[1] The same work introduced paged memory management for the KV cache — handling GPU memory the way an operating system pages RAM — precisely because contiguous per-request allocation wastes so much of it.

Compression is the second active front. Quantization research indicates the KV cache can be stored at roughly 3.5 bits per channel while remaining close to baseline output quality, and at around 2.5 bits with marginal loss (TurboQuant, 2025).^[2] Follow-on work concentrates on how to measure the quality impact of KV-cache quantization rigorously rather than anecdotally.^[3] The direction of the public literature is consistent: substantial headroom exists between how inference systems store attention state today and what models actually need to preserve their behavior.

Memory efficiency compounds into infrastructure economics. Data centres consumed around 1.5 percent of global electricity in 2025, a share the IEA projects to roughly double to about 3 percent — around 945 TWh — by 2030, with AI as the main driver (IEA, 2025).^[4] In the United States, data centres drew 4.4 percent of national electricity in 2023, projected by Lawrence Berkeley National Laboratory to reach 6.7 to 12 percent by 2028 (LBNL, 2024).^[5]The hardware serving inference keeps densifying in parallel: NVIDIA’s GB200 NVL72 packages 72 GPUs into a single liquid-cooled, rack-scale NVLink domain built for exactly these workloads.^[6] Every point of memory utilization recovered in software is inference capacity that does not have to be built, powered, and cooled as new floor space.

PODOS AI treats AI infrastructure as one integrated modular AI data center platform: hardware that compresses how fast compute can be deployed, and software that raises how much useful work that compute does. The hardware unit is the PODOS Pod, designed as a standardized 1-MW building block for AI infrastructure and designed for 128 GPUs per pod. Its thermal and electrical systems are covered in the direct-to-chip liquid cooling explainer and the data center power architecture explainer, with the wider engineering approach on the engineering hub.

On the deployment side, PODOS targets a 90-day window from order to commissioning for a standard unit — the reasoning behind that target is laid out on the deployment model page. Syntropic is the software counterpart to that hardware thesis: where the pod is designed to compress the time between deciding to own compute and operating it, Syntropic is designed to raise the useful work extracted from the GPUs inside. The intended beneficiaries are the same organizations described across the platform use cases — teams running inference on infrastructure they control.

For the vocabulary used on this page — KV cache, quantization, NVLink domain, PUE and related terms — see the AI infrastructure glossary.