Redundant I/O Interconnect - IBM z13s Technical Manual

Host Channel Adapter (HCA3-O LR (1xIFB)): This optical long reach fanout provides
1x InfiniBand coupling link connectivity up to a 10 km (6.2 miles) unrepeated, or 100 km
(62 miles) when extended by using z Systems qualified DWDM equipment) distance to
z13s, z13, zEC12, zBC12, z196, and z114 servers.
When you are configuring for availability, balance the channels, coupling links, and OSAs
across drawers. In a system that is configured for maximum availability, alternative paths
maintain access to critical I/O devices, such as disks and networks.
Note: Fanout plugging rules for z13s servers are different than previous EC12 and z114
servers.
Preferred plugging for PCIe Generation 3 fanouts will always be in CPC Drawer 1 (bottom
drawer) alternating between the two nodes. Previous systems EC12 and z114 were
plugged across two drawers. This configuration is for performance reasons and to
maintain RAS characteristics.
If the configuration contains two CPC drawers, two PCIe I/O drawers, and PCIe ICA
Coupling adapters. All PCIe Generation 3 fanouts are plugged in CPC Drawer 1, and all
PCIe ICA coupling adapters are plugged in CPC Drawer 0 (top), alternating between the
internal node affinity.
A fanout can be repaired concurrently with the use of redundant I/O interconnect.

2.6.1 Redundant I/O interconnect

Redundancy is provided for both InfiniBand I/O and for PCIe I/O interconnects.
InfiniBand I/O connection
Redundant I/O interconnect is accomplished by the facilities of the InfiniBand I/O connections
to the InfiniBand Multiplexer (IFB-MP) card. Each IFB-MP card is connected to a jack in the
InfiniBand fanout of a CPC drawer. IFB-MP cards are half-high cards and are interconnected
through the I/O drawer backplane. This configuration allows redundant I/O interconnect if the
connection coming from a CPC drawer ceases to function. This situation can happen when,
for example, a CPC drawer connection to the I/O Drawer is broken for maintenance.
Chapter 2. Central processor complex hardware components
67