Skip to content

RoCE

πŸ“ Short definition

Protocol that brings RDMA benefits on suitable Ethernet networks.

πŸ“– Detailed definition

RoCE brings RDMA benefits (low latency, low CPU load) on standard Ethernet switches, without requiring a dedicated InfiniBand fabric.

RoCE v2 (the current version) uses UDP/IP, which simplifies routing but requires a lossless network so RDMA performance does not degrade:

  • PFC (Priority Flow Control): avoids packet loss by pausing traffic per priority.
  • ECN (Explicit Congestion Notification): manages congestion before buffers saturate.

Without this configuration, a lost packet forces retransmission that can degrade throughput by 10Γ— or more.

RoCE vs InfiniBand:

  • RoCE: reuses Ethernet infrastructure, lower cost, demanding network configuration.
  • InfiniBand: dedicated fabric, natively lossless, lowest latency, high cost.

πŸ’‘ Why it matters for on-prem AI

Credible alternative to InfiniBand for on-prem AI clusters when budget does not allow a dedicated IB fabric. Backbone of GPU clusters at 25/100/400 GbE.

⚠️ Common pitfalls

  • RoCE without correctly configured PFC/ECN performs worse than optimized TCP.
  • A single misconfigured switch on the path breaks lossless behavior for the whole flow.

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.