RDMA protocol overview
A series of protocols are layered to perform RDMA over TCP (Figure 3). The top three layers form the
iWARP family of protocols that provide high-speed internet operability. The RDMA protocol converts
RDMA write, RDMA read, and sends into Direct Data Placement (DDP) messages. The DDP protocol
segments outbound DDP messages into one or more DDP segments, and reassembles one or more
DDP segments into a DDP message. The marker-based, protocol-data-unit-aligned (MPA) protocol
adds a backward marker at a fixed interval to DDP segments and also adds a length and cyclical
redundancy check (CRC) to each MPA segment. TCP schedules outbound TCP segments and satisfies
delivery guarantees. IP adds necessary network routing information.
Figure 3. Protocol layers for RDMA over TCP
TCP uses a stream of 8-bit fields (octets) to communicate data, while DDP uses fixed protocol data
units (PDUs). To enable RDMA, DDP needs a framing mechanism for the TCP transport protocol. MPA
is a marker-based PDU-aligned framing mechanism for the TCP protocol.
MPA provides a generalized framing mechanism that enables a network adapter using DDP to locate
the DDP header. The adapter can then place the data directly in the application buffer, based on the
control information carried in the header. MPA enables this capability even when the packets arrive
out of order. By enabling DDP, MPA avoids the memory copy overhead and reduces the memory
requirement for handling out-of-order packets and dropped packets.
MPA creates a framed PDU (FPDU) by prefixing a header, inserting markers, and appending a CRC
after the DDP segment. MPA delivers the FPDU to TCP. The MPA-aware TCP sender puts the FPDUs
into the TCP stream and segments the TCP stream so that each TCP segment contains a single FPDU.
The MPA receiver locates and assembles complete FPDUs within the stream, verifies their integrity,
and removes information that is no longer necessary. MPA then provides the complete DDP segment
Converts RDMA write, RDMA read, and sends into DDP
Segments outbound (TX) DDP messages into one or more
DDP segments; reassembles one or more DDP segments
into a DDP message
Adds a backward marker at a fixed interval to DDP
segments; also adds a length and CRC to each MPA
Schedules outbound TCP segments and satisfies delivery
Adds necessary network routing information