First identify the memory bottleneck
A large model, long-context inference service, graph workload and in-memory database can all ask for more memory, but they do not need the same access path. Measure working-set size, read/write ratio, access locality, tail-latency sensitivity, concurrency and the cost of a miss or data movement.
- Capacity shortage: a single host runs out of addressable memory.
- Stranded capacity: one node owns idle memory while another spills or is overprovisioned.
- Data movement: model state, KV cache or datasets repeatedly cross slow storage or network paths.
- Sharing: several jobs need access to the same state, with clear ownership and consistency rules.
For the underlying architecture, start with what memory disaggregation means. This page focuses on the product and integration decision.
Compare the product categories
| Approach | What the buyer gets | Key dependency |
|---|---|---|
| More local DDR or HBM | Capacity inside each server or GPU | Physical slots or GPU choice; capacity remains tied to a node |
| CXL memory expansion or pool | External memory reached through a CXL-capable host and device path | Platform, firmware, switch or device support and software policy |
| RDMA-accessed memory appliance | Memory in another system reached across a qualified network | NIC and fabric behavior plus software that places and retrieves data |
| Software-managed cache or tier | Placement between fast and slower memory tiers | Application or runtime integration and measurable eviction behavior |
The CXL Consortium specification defines the protocol family, while NVIDIA documents RoCE as a network RDMA transport. Neither transport alone supplies an application-level sharing policy. Ask who owns allocation, isolation, failover and observability.
The programming model is the deciding difference
The phrase “remote memory” hides two materially different experiences. A CXL-capable platform can expose device memory through a host memory path; an RDMA appliance moves data over a network path under software control. Do not assume the same application can switch between them unchanged.
- Visibility: Is capacity addressable by the OS, a driver, a runtime, a library or the application?
- Consistency: Can two hosts safely access the same bytes, and under what ownership rules?
- Placement: Who decides which data stays in HBM, local DDR or remote memory?
- Failure: What happens to a running job if the appliance, switch or link fails?
- Operations: Which counters reveal pool occupancy, page movement, queueing and tail latency?
See CXL versus RDMA over Ethernet for a deeper technical comparison. TORmem M1000 is the product path for TORmem's RDMA-based memory appliance.
Match the architecture to the workload
| Workload | First question to test | Evidence to request |
|---|---|---|
| Long-context inference | Does the working set or KV cache exceed economical local capacity? | Latency and throughput under real context lengths and concurrency |
| Training or model serving | Is the problem capacity, bandwidth, placement or repeated transfers? | End-to-end job time and stall profile, not a link benchmark alone |
| Graph or in-memory analytics | Are accesses random and tail-latency sensitive? | P95/P99 access time and workload completion time |
| Shared services | Is data genuinely shared, or only replicated across nodes? | Consistency, isolation and failure-recovery tests |
The inference memory sizing worksheet turns the KV-cache question into a capacity estimate. The memory overprovisioning cost article explores another motivation. A capacity estimate does not prove a specific remote-memory design will improve a production workload; insist on an application-level benchmark.
Ask for a workload proof, not a bandwidth slogan
- Record the current baseline: dataset, model, concurrency, local-memory configuration and observed bottleneck.
- Specify the candidate host, memory device, fabric, software version and data placement policy.
- Measure end-to-end job time or request latency alongside pool occupancy, network utilization and host CPU load.
- Repeat under contention and after a host, link or appliance fault; document the recovery behavior.
- Compare total system cost at the same service-level target, including the network and software operations.
Use TORmem memory architecture for the system context and the product catalog for available appliance and platform paths.
Frequently asked questions
- What products can provide shared memory for AI workloads?
- The main product paths are local memory upgrades, CXL-capable expansion or pooling devices, network-attached memory appliances and software-managed memory tiers. Local upgrades add capacity to one machine; external devices and pools require supported hosts, fabric and software. For a shared-memory claim, ask whether hosts share the same bytes or receive separately managed copies, and how access is isolated.
- Is disaggregated memory the same as GPU HBM?
- No. HBM sits on or near the GPU and serves its highest-bandwidth working set. Disaggregated memory adds a separate capacity tier or pool, reached through another interconnect and software path. The relevant test is whether placing less frequently used state in that tier improves the application at the same service target; remote capacity is not a drop-in HBM replacement.
- Is CXL memory pooling the same as RDMA memory pooling?
- No. CXL uses a host and device interconnect with memory-specific protocol support. RDMA moves data between systems over a network. The software-visible model, coherency, topology and failure modes differ, even when both are described as pools. Ask the supplier to diagram the host path and name the OS, driver or runtime that allocates and accesses the capacity.
- Can multiple GPU servers share one memory appliance?
- They can access capacity in an appliance if the hardware, fabric and host software support it. Whether the same bytes are concurrently shared, safely isolated or copied between hosts depends on the software’s ownership and consistency design. Test the intended concurrency level, per-tenant access control and behavior when one host or link fails.
- Will remote memory be as fast as local DDR or HBM?
- Do not assume so. Remote access adds a device or network path, and behavior depends on access size, locality, contention, software placement and fabric health. Compare application-level latency and throughput against the local baseline at the target concurrency, and report tail latency rather than a peak link-bandwidth figure alone.
- What workloads should be tested first?
- Start with workloads that demonstrably spill, overprovision memory per node or repeatedly move large state. Long-context inference, some graph and in-memory analytics workloads, and shared services are candidates. Select one representative dataset and a fixed service target, then compare end-to-end performance and cost with and without the external tier.
- What should a memory appliance proof of concept measure?
- Measure job time or request latency, P95/P99 behavior, memory-pool occupancy, network utilization, CPU overhead, contention and recovery from a link or appliance fault. Record the baseline, placement policy, data size, concurrency and all software versions so the result is reproducible. Include the appliance and network in the cost comparison.
- Does RoCE alone make memory disaggregated?
- No. RoCE provides an RDMA transport over Ethernet. A usable disaggregated-memory product also needs memory capacity, host integration, allocation and placement software, access control, monitoring and a failure model. Verify all of those in the proposed stack; a fast NIC benchmark does not establish an application-ready shared-memory service.
Built to order
Test a shared-memory design against your workload
Bring the working-set size, concurrency and current bottleneck. We can scope a measurable proof of concept around the application.
Related
Sources
Product and company names identify hardware and published sources. They don't imply any partnership or endorsement.