RoCE
π Short definition
Protocol that brings RDMA benefits on suitable Ethernet networks.
π Detailed definition
RoCE brings RDMA benefits (low latency, low CPU load) on standard Ethernet switches, without requiring a dedicated InfiniBand fabric.
RoCE v2 (the current version) uses UDP/IP, which simplifies routing but requires a lossless network so RDMA performance does not degrade:
- PFC (Priority Flow Control): avoids packet loss by pausing traffic per priority.
- ECN (Explicit Congestion Notification): manages congestion before buffers saturate.
Without this configuration, a lost packet forces retransmission that can degrade throughput by 10Γ or more.
RoCE vs InfiniBand:
- RoCE: reuses Ethernet infrastructure, lower cost, demanding network configuration.
- InfiniBand: dedicated fabric, natively lossless, lowest latency, high cost.
π‘ Why it matters for on-prem AI
Credible alternative to InfiniBand for on-prem AI clusters when budget does not allow a dedicated IB fabric. Backbone of GPU clusters at 25/100/400 GbE.
β οΈ Common pitfalls
- RoCE without correctly configured PFC/ECN performs worse than optimized TCP.
- A single misconfigured switch on the path breaks lossless behavior for the whole flow.
π See also
Network technique that reads and writes remote memory without going through the CPU, with low latency; it comes in InfiniBand and RoCE implementations.Dedicated high-performance network fabric for GPU clusters, HPC and AI datacenter standard.Priority Flow Control β Ethernet per-priority pause mechanism for a lossless network required by RoCE.Explicit Congestion Notification β network congestion signaling used with RoCE to avoid packet loss.Mechanism letting GPUs exchange data directly with network devices without CPU copies.