Infiniband Specifications; Pcie Generation 3 - IBM z13s Technical Manual

4.1.3 InfiniBand specifications

The InfiniBand specification defines a point-to-point, bidirectional serial communications
between two nodes. It is intended for the connection of processors with high-speed peripheral
devices, such as disks. InfiniBand supports various signaling rates on a physical lane, and as
with PCI Express, lanes can be bonded together to create a wider link. Connection with only
one physical lane pair is defined as a 1x link width connection. The specification also
supports 4x and 12x link width connections by aggregating 4 or 12 pairs of physical lanes.
The base signaling rate on one physical lane is 2.5 Gbps, which is called the single data rate
(SDR). InfiniBand can also support enhanced signaling rates like 5.0 Gbps for the double
data rate (DDR) and 10 Gbps for the quad data rate (QDR)
While operating at SDR, DDR or QDR, data transmission of physical lane is 8b/10b encoding,
which means every 10 bits carry 8 bits of data. The following are the data rates for different
link widths operating at different signaling rates:
250 MB/s for 1x SDR
500 MB/s for 1x DDR
1 GB/s for 4x SDR
2 GB/s for 4x DDR
3 GB/s for 12x SDR
6 GB/s for 12x DDR
Link performance: The link speeds do not represent the actual performance of the link.
The actual performance is dependent upon many factors that include latency through the
adapters, cable lengths, and the type of workload.
z13s servers use InfiniBand 12x DDR connections between the CPC fanouts and I/O
drawers.
For details and the standard for InfiniBand, see the InfiniBand website at:
http://www.infinibandta.org

4.1.4 PCIe Generation 3

The z13 and z13s servers are the first generation of z Systems servers to support the PCIe
Generation 3 (Gen3) protocol.
PCIe Generation 3 uses 128b/130b encoding for data transmission. This encoding reduces
the encoding effort by approximately 1.54% when compared to the PCIe Generation 2, which
has an encoding effort of 20% using 8b/10b encoding.
The PCIe standard uses a low voltage differential serial bus. Two wires are used for signal
transmission, and a total of four wires (two for transmit and two for receive) become a lane of
a PCIe link, which is full duplex. Multiple lanes can be aggregated for larger link width. PCIe
supports link widths of one lane (x1), x2, x4, x8, x12, x16, and x32.
The data transmission rate of a PCIe link is determined by the link width(numbers of lanes),
the signaling rate of each lane, and the signal encoding rule. The signaling rate of a PCIe
Generation 3 lane is 8 giga transmits per second (GT/s), which means 8 giga bits are
transmitted per second (Gbps).
1
Higher link speeds might be available for other applications. For more information, see the InfiniBand Trade
Association website at:
2
z Systems does not support 4x link width
2
http://www.infinibandta.org
Chapter 4. Central processor complex I/O system structure
1
.
139