Monitoring and controls, sensor to alarm

ENG-06·Engineering

Why the instrumentation layer decides uptime

High-density AI hardware fails gradually before it fails suddenly. Thermal margin erodes, a pump curve shifts, a coolant filter loads up, a phase drifts out of balance — and the visible symptom is a throttled GPU or a tripped breaker weeks later. The Uptime Institute's 2025 survey of more than 800 operators found that half of respondents had experienced an impactful outage in the previous three years, with power and cooling failures among the recurring causes.^[2] Instrumentation is what converts those slow-moving physical trends into something a human or a control loop can act on while the margin still exists.

The design question is therefore not "how many sensors" but "which measurements close which loop". Reliability practice for continuous power systems treats monitoring points as part of the reliability model, not as an accessory bolted on after commissioning.^[9] A point that no alarm, no control loop, and no maintenance task depends on is storage cost with a dashboard attached.

Telemetry in a compute site is not one stream. It is five domains with different physics, different sample rates, and different owners — and most integration pain comes from treating them as one. Machine telemetry leaves the server through management interfaces; the DMTF's Redfish API and its metric-report model are the vendor-neutral way to pull it without a per-vendor agent.^[3]Fluid telemetry comes off the cooling distribution unit, whose expected instrumentation is being standardised through the Open Compute Project's cooling work.^[6] And life-safety signals report on their own supervised paths under fire-protection requirements for IT equipment areas.^[10]

Thermal

Temperature is the symptom, delta-T is the diagnosis

An air-cooled room is monitored at the inlet, because ASHRAE's thermal guidelines define equipment environmental classes by inlet air conditions — the number the hardware warranty is written against. A liquid-cooled system moves the primary measurement into the loop, and the same guidelines extend the logic by naming facility water classes according to supply temperature.^[1]

The diagnostic value sits in the relationships, not the absolute readings. Return-minus-supply temperature against known heat load tells you whether flow has fallen. Rising differential pressure at constant flow tells you a filter is loading or a channel is fouling. A supply temperature drifting toward the dew point of the enclosure air warns of condensation on the piping before a leak sensor ever wets. That is why the same four measurements — supply, return, flow, differential pressure — appear on every well-instrumented direct-to-chip cooling loop, and why leak detection is treated as a continuous, always-on channel rather than a polled point.^[6]

Metering answers how much energy moved. Power-quality monitoring answers whether the waveform delivering it stayed inside tolerance. IEC 61000-4-30 defines the measurement methods and aggregation intervals for the parameters that matter — power frequency, supply voltage magnitude, dips and swells, interruptions, transients, unbalance, harmonics and interharmonics — precisely so that two instruments from two vendors produce comparable results from the same event.^[4]Without that discipline, a "voltage sag" on one dashboard and on another are not the same measurement.

AI compute makes this concrete in two directions. The load is a large population of switched-mode rectifiers, which draws non-sinusoidal current and injects harmonic distortion back toward the point of common coupling — the condition IEEE 519 exists to bound.^[5] And the load swings hard: synchronised training and inference workloads step a large fraction of site power in seconds, which shows up as voltage deviation and stresses upstream equipment. Both effects are invisible to an energy meter and obvious to a class-A analyser. The mitigation choices sit in the power architecture, but the evidence that they work only ever comes from this instrument class.

Good electrical telemetry also unlocks capability, not just protection. NREL's demonstration of a 70 MW data center as a flexible grid asset dispatched 35 MW of battery storage in under five seconds while keeping workload service levels intact — a control action that is only possible when load, storage state, and grid signals are measured fast enough to close the loop.^[7]

The slow layer

From telemetry to maintenance work

Maintenance data is the slow layer of the same instrumentation. It is not sampled in seconds and it does not raise alarms; it accumulates into a service record. Pump and fan runtime hours drive bearing replacement intervals. Filter differential-pressure trends set change-out dates instead of a calendar guess. Coolant chemistry — sampled, not sensed — tracks inhibitor depletion and biological growth in the technology loop. Firmware and configuration versions per node close the loop between a behaviour change and the change that caused it.

Efficiency reporting belongs to this slow layer too, and its credibility is entirely a methodology question. Published hyperscale efficiency figures are stated as trailing-twelve-month, all-season averages including every overhead inside the measurement boundary — which is why they are comparable and why an instantaneous best-hour number is not.^[8] Any site publishing an efficiency ratio should state the boundary, the window, and whether the inputs were measured or derived.

This distinction is a house rule, and it is worth stating plainly. Interactive visualisations, configurators, and animated dashboards published on a website render illustrative values produced by a model. They exist to explain an architecture. They are not measurements, and PODOS does not describe them as live, real-time, or operational data anywhere on this site.

Live telemetry means something specific: a value sampled from an identified instrument on operating hardware, carrying a timestamp and a known measurement uncertainty. Three properties, all three required. The checklist below is the test we apply to our own numbers before publishing them, and it works equally well on anyone else's monitoring claims.

In the product

How PODOS approaches monitoring in a factory-built unit

A factory-integrated unit changes where the instrumentation problem is solved. Each PODOS Pod is designed as a standardized 1 MW building block and designed for 128 GPUs, so the sensor set, the controller, the alarm list, and the acceptance tests are defined once for a repeated product rather than negotiated per building. Sensor placement, wiring, and point naming are part of the enclosure design; commissioning checks run on the factory floor against a known configuration. That repeatability is part of why PODOS targets a 90-day window from order to commissioning for a standard unit — a site is not also a bespoke instrumentation project.

The instrumentation layer connects the rest of the system: the thermal points that prove the cooling loop is doing its job, the electrical points that make the power architecture observable, and the acceptance data handed over at the end of the deployment process. For the wider engineering context see the engineering index and how this differs from a conventional build in modular vs traditional AI data centers; unfamiliar terms are defined in the AI infrastructure glossary.