How do you accept a GPU cluster before handover?

Define what is being accepted

“The cluster passed burn-in” is too vague to close a purchase. A server can pass a component diagnostic while the rack has a wrong cable, the fabric has a slow rail, or the site cannot sustain the configured load. The acceptance plan should say whether the supplier is delivering individual L10 servers, an integrated L11 rack, a connected multi-rack cluster or a complete service at the destination. Each boundary needs a different witness and a different set of evidence.

The L10 versus L11 guide defines the manufacturing handoff. This page starts where that guide ends: it turns the promised scope into acceptance tests. Put the stage, location, owner and sign-off authority on every test row. Factory acceptance and site acceptance may both be needed; passing one does not automatically waive the other.

Freeze the as-built baseline first

A test result is only meaningful when it can be tied to the hardware that will ship. Capture serial numbers and physical positions for servers, GPUs, NICs, switches, power supplies and storage devices. Record CPU, memory and accelerator population; PCIe and GPU topology; firmware, BIOS, BMC, driver, CUDA and fabric software versions. Keep a port-to-port map for every production and management link, including optics and cable IDs where those matter to support.

• Configuration: compare the installed BOM with the approved quote and change record; identify every substitution.
• Topology: record which GPU, NIC and switch port form each rail or path, plus rack and host labels.
• Software: pin the image, driver, NCCL or communication library, firmware and test-tool revisions.
• Management: verify BMC reachability, time sync, credentials handoff, telemetry export and alert destinations.
• Exceptions: state which differences were approved, by whom, and whether they require retesting.

NVIDIA's deployment validation checklist records inventory, serial numbers, network and management checks for its rack systems. Use the equivalent inventory for the actual platform being purchased. A screen capture of a passing test without serial numbers and versions is difficult to use when a replacement part arrives six months later.

Validate every node before aggregate tests

Start with a repeatable node-level check so a cluster failure is not blamed on the fabric when one accelerator or host is unhealthy. Confirm all expected devices enumerate at the correct width and speed, then check accelerator, host memory, storage and NIC health. Capture thermal, power and error telemetry at idle and under a representative sustained load. Log every failed and rerun test; a final green summary should not erase the path taken to get there.

For NVIDIA platforms, DCGM Diagnostics provides active deployment and hardware tests; DCGM health monitoring is a passive view of retained telemetry. They answer different questions. A passive health check does not prove the GPU can sustain the agreed workload, and a one-time diagnostic does not replace ongoing monitoring. Use the vendor-supported equivalents for other accelerators.

1. Confirm the inventory and GPU/NIC topology against the approved configuration.
2. Run supported active diagnostics on each accelerator and capture tool output with timestamps.
3. Check host memory, local storage, NIC link state, firmware alignment and operating-system errors.
4. Hold the node at the agreed load while recording power, temperatures, clocks, throttling and errors.
5. Repeat failed checks after remediation and preserve the original failure plus the passing rerun.

Driver Xid events are GPU error reports in system logs; their presence, context and recovery action deserve review during a node test. Do not turn an arbitrary count into a universal pass rule. Decide in advance which errors trigger investigation, a replacement or a rerun, using the platform maker's guidance and the buyer's reliability requirement.

Prove the fabric path, then the collective

A GPU cluster has several paths that can fail independently: GPU-to-GPU links within a node, GPU-to-NIC placement, physical NIC links, the inter-node fabric and the management network. Validate link inventory and speed first. Then run path-level tests between representative and worst-case node pairs, followed by collectives across the intended rank set. Save per-pair and per-rank results so one slow path cannot hide inside an average.

NVIDIA's nccl-tests checks correctness as well as performance. Its performance guide distinguishes algorithm bandwidth from bus bandwidth; both depend on operation and rank count. A single large all-reduce number does not identify every path problem. Compare the same message sizes, collective, rank count, placement and software settings to an agreed reference. The GPU bandwidth guide explains common interpretation traps in more detail.

Treat a slow rank as an investigation lead, not proof of a bad switch. Check topology, NIC placement, PCIe width, link counters, routing, congestion and host activity before replacing hardware. Keep the test environment stable enough that a rerun can confirm the change. If the cluster spans several racks, include cross-rack pairs rather than inferring their health from within-rack tests.

Test the rack at its real load and fault boundary

Power-on proves little about a dense GPU rack. The acceptance plan should name the configured load, feed topology, rack-PDU and circuit mapping, cooling design and the allowed failure event. Capture readings from the same boundary used in the site agreement, with both normal and post-fault load visible. Verify that every server, switch and management device has the intended power path. Ask qualified facilities staff to conduct any feed-loss test under the approved procedure.

NVIDIA's DGX H100 electrical guide shows why a generic “dual-feed” label can be misleading: the number of energized PSUs and distribution paths matters for that exact system. Apply the manufacturer design for the actual hardware, then compare it with the site single-line diagram. The GPU rack power guide provides the detailed site-readiness checklist.

• Record rack and branch power while the agreed workload runs; correlate with GPU clocks and thermal telemetry.
• Check inlet conditions, cooling alarms and any liquid-cooling leak or flow monitoring included in scope.
• Witness the permitted feed or component loss at an agreed load, then verify workload and management behavior.
• Restore the path and confirm alarms, event logs, and the as-built circuit and PSU map agree.
• Repeat the site test after a material rack BOM or power-path change; a factory result cannot prove the destination circuit.

Use a representative workload to close the gap

Synthetic diagnostics isolate components and paths; they do not show that the buyer’s application works as expected. Select a representative training step, inference trace, rendering job or simulation before the build. Define its model or dataset, job size, concurrency, storage access, checkpoint behavior and expected completion or latency target. If the real workload cannot be shared, agree on a substitute that stresses the same limiting paths and document what it leaves untested.

Measure both a steady-state period and startup or transition behavior. A cluster can look healthy after jobs settle yet fail during image distribution, simultaneous GPU initialization, checkpoint writes or rolling restarts. Compare the full job time or requests completed within a service level, then inspect node and fabric variance. Record utilization, errors, throttling and scheduler events during the run so performance can be explained rather than merely scored.

Avoid accepting a workload result that changes several variables at once. If the supplier adjusts clocks, rank placement, model precision or network settings after a failed run, preserve the old result, record the change and rerun the same workload. This protects both sides: the buyer can understand the final configuration, and the supplier can show exactly how a defect was resolved.

Agree what happens when something fails

A production cluster will eventually lose a component. Acceptance cannot simulate every incident, but it can prove the recovery behavior the contract relies on. Pick the fault cases that correspond to the purchased design: a node reboot, a planned link disable, a PDU or feed loss where permitted, or a failed storage path. State whether the expected result is uninterrupted work, a clean job restart, a checkpoint recovery or a controlled service degradation.

Capture detection time, alert destination, affected jobs, recovery steps, operator intervention and data integrity. A workload surviving one failed component does not prove the entire system is concurrently maintainable; the upstream paths may still converge. Use the as-built diagrams to choose a meaningful failure domain and let the authorized facilities and operations teams run the test.

If a test fails, classify the issue before sign-off. A missing document, a repairable component failure and a missed workload threshold have different consequences. Record the owner, fix, retest and buyer waiver if one is granted. A waiver should name the exact exception and any operating limitation; an unsigned punch list should not become a silent acceptance.

Deliver evidence that survives the handover

The useful output is an evidence package tied to the as-built system. It should let the buyer reproduce a test, diagnose a later regression and decide whether a failure belongs to the hardware, fabric, software or site boundary. Request raw machine-readable output where available alongside a short signed summary. A slide with green check marks is a report of opinion unless the configuration and underlying results can be retrieved.

Hold a joint review with the integrator, site operator and workload owner. Read unresolved exceptions aloud, assign their disposition and record who signs at factory and who signs after installation. If software or firmware is changed between those events, state which tests must be repeated. The result is a baseline for future incident comparison, not a promise that no component will ever fail.

Use one sequence from purchase order to sign-off

1. Before purchase: freeze the deliverable boundary, reference configuration, test methods, pass thresholds and evidence list.
2. Before factory testing: receive the installed BOM, topology and software baseline; resolve substitutions.
3. At factory: run node diagnostics, rack power and fabric checks, sustained load and agreed workload tests.
4. At destination: verify shipping condition, site power and cooling, upstream connections and cluster-level behavior.
5. At sign-off: review raw results and exceptions, archive the as-built package and transfer support ownership.

This sequence can be adapted to a single server or a multi-rack deployment. The ODM GPU server sourcing guide helps define the purchase, while the L11 guide explains the rack build. Bring the actual workload, site and commercial boundary together in the acceptance schedule instead of assuming a vendor's standard burn-in covers them all.

Frequently asked questions

What is a GPU cluster acceptance test?

It is the set of agreed checks that proves the delivered cluster matches its specification and can run the intended work at the defined site boundary. It starts with inventory and node health, then checks fabric, rack power and cooling, a representative workload, and selected fault cases. The plan should name the test location, method, pass criteria, duration, witness, evidence file and remediation owner. A factory test may prove an integrated rack before shipment; a site test proves it after connection to destination power, cooling and upstream networks. Both may be required.

Is GPU burn-in the same as cluster acceptance?

Burn-in applies sustained load to expose hardware or integration faults over an agreed period. Cluster acceptance is broader: it also verifies the installed BOM, firmware, topology, cross-node fabric, rack power, cooling, workload behavior, recovery and handover records. A node can survive burn-in while a rail is cabled incorrectly or a site feed is undersized. Agree the duration and pass rule for burn-in, then place that result inside a larger matrix. No fixed number of hours is a substitute for testing the exact configuration and fault boundary being purchased.

Which GPU diagnostics should run before shipment?

Use diagnostics supported by the accelerator and platform vendor on every configured node, then preserve the detailed output. For NVIDIA systems, DCGM offers active diagnostic suites and separate passive health monitoring. Also inspect device enumeration, PCIe and GPU links, host memory, storage, NIC state, firmware alignment and system error logs. Run a sustained node workload with power and thermal telemetry when the contract calls for it. Record failed tests and the remedy before rerunning; a final pass without the original failure history leaves the buyer unable to assess a recurring fault.

What should an NCCL acceptance test prove?

It should show that the agreed ranks can complete selected collective operations correctly and at a performance level appropriate for the configured topology. Save operation, message sizes, rank count and placement, software versions, raw output and per-rank or per-pair outliers. NVIDIA nccl-tests reports algorithm and bus bandwidth, which are interpreted differently across collectives. A single all-reduce headline cannot prove every rail or cross-rack path. Pair-level link and fabric evidence should sit beside collective results, and the pass threshold should be set from the actual platform and signed reference configuration.

How do we set a fair pass threshold without a universal benchmark?

Choose a platform-specific reference from the manufacturer, a buyer-approved known-good configuration or a jointly witnessed baseline, then fix the test method and environment before the build. State the exact tool, version, rank map, message size, workload, duration and allowable variation. Compare like with like; changing a collective algorithm or node placement can change the result. For a business workload, define completed work and latency or job-time targets. For each threshold, name what happens if it is missed: investigate, repair, rerun, accept with a documented waiver or reject.

Do we need to retest at the destination after factory acceptance?

Usually yes for the boundaries that changed during shipment and installation. Factory acceptance can validate the assembled rack and its local tests. At the destination, verify shipping condition, circuit and PSU mapping, cooling, upstream network, storage, management access and the final multi-node workload. Retest any component or setting changed after the factory run. The buyer and supplier should agree which factory results carry forward and which require a site witness. This prevents a site power or network issue from being treated as proof that factory testing was wrong—or vice versa.

What should a GPU rack power acceptance test include?

Trace the actual source, circuit, rack PDU, outlet and PSU path for every server, switch and management device. Measure the configured rack under the agreed load, including feed balance and thermal conditions. Where the manufacturer and site design permit it, have qualified facilities staff witness a planned feed or component loss and restoration. Record what happened to workload and management access, not only whether the lights stayed on. Compare the result with the exact PSU architecture; a generic dual-feed label does not establish redundancy for every GPU system.

How long should the cluster run before sign-off?

Set duration according to the fault modes and workload the test is meant to expose. A quick inventory or link check needs a different window from a thermal soak, repeated checkpoint cycle or production-like inference trace. The contract should state the period, load, allowed interruptions and evidence sampling interval before testing begins. If a failure occurs, record the original run, repair and agreed rerun window rather than resetting the clock without explanation. A universal burn-in number can create false confidence when the relevant failure appears only at a particular load or transition.

What belongs in the final handover package?

Include the approved and as-built BOM, serial numbers, rack positions, firmware and image versions, topology and port maps, power and cooling diagrams, test matrix, raw outputs, workload results, alarms, exceptions and retests. Add monitoring access, spares, warranty terms, escalation contacts and the party responsible for each support boundary. Make each result traceable to the delivered equipment and the software version used. A signed summary is helpful, but the buyer should also receive enough underlying evidence to reproduce a test or compare a later incident with the original baseline.

What happens if one acceptance test fails?

Record the failed condition, its impact and the responsible party. Determine whether the issue is a component fault, an integration or site problem, a software setting, or a disputed threshold. Preserve raw evidence, agree the remedy and rerun the affected test plus any dependent checks. If the buyer chooses to accept an exception, the waiver should name the exact limitation, owner, deadline and support consequences. Do not let a green summary overwrite a failed run or treat an unsigned punch list as acceptance. The contract should state whether payment or handover depends on the affected test.