I/O Connectivity: Pcie And Infiniband; I/O Subsystem - IBM z13s Technical Manual

One memory port for each PU chip.
Increased bandwidth between memory and I/O
Asymmetrical memory size and DRAM technology across CPC drawers
Large memory pages (1 MB and 2 GB)
Key storage
Storage protection key array that is kept in physical memory
Storage protection (memory) key that is also kept in every L2 and L3 cache directory entry
A larger (40 GB) fixed-size HSA that eliminates having to plan for HSA
PCIe fanout hot-plug
PCIe fanout
The
the PCIe 16 GBps bus and cables. The PCIe fanout is hot-pluggable. During an outage, a
redundant I/O interconnect allows a PCIe fanout to be concurrently repaired without loss of
access to its associated I/O domains. Up to four PCIe fanouts per drawer are available on the
N10 and up to eight PCIe fanouts per drawer on the N20. The PCIe fanout can also be used
for the ICA SR. If redundancy in coupling link connectivity is ensured, the PCIe fanout can be
concurrently repaired.
Host channel adapter fanout hot-plug
The HCA fanout provides the path for data between memory and the I/O cards in an I/O
drawer through 6 GBps IFB cables. The HCA fanout is hot-pluggable. During an outage, an
HCA fanout can be concurrently repaired without the loss of access to its associated I/O
features by using redundant I/O interconnect to the I/O drawer. Up to four HCA fanouts are
available per CPC drawer.

1.3.5 I/O connectivity: PCIe and InfiniBand

z13s servers offer various improved I/O features and uses technologies such as PCIe and
InfiniBand. This section briefly reviews the most relevant I/O capabilities.
z13s servers take advantage of PCIe Generation 3 to implement the following features:
PCIe Generation 3 (Gen3) fanouts that provide 16 GBps connections to the PCIe I/O
features in the PCIe I/O drawers.
PCIe Gen3 fanouts that provide 8 GBps coupling link connections through the new IBM
ICA SR. Up to eight ICA SR fanouts and up to 16 ICA SR ports are supported on an
IBM z13s server, enabling improved connectivity for short distance coupling.
z13s servers take advantage of InfiniBand to implement the following features:
A 6 GBps I/O bus that includes the InfiniBand infrastructure (HCA2-C) for the I/O drawer
for non-PCIe I/O features.
Parallel Sysplex coupling links using IFB include 12x InfiniBand coupling links (HCA3-O)
for local connections and 1x InfiniBand coupling links (HCA3-O LR) for extended distance
connections between any two zEnterprise CPCs. The 12x IFB link (HCA3-O) has a
bandwidth of 6 GBps and the HCA3-O LR 1X InfiniBand links have a bandwidth of 5 Gbps.

1.3.6 I/O subsystem

The z13s I/O subsystem is similar to z13 servers and includes a new PCIe Gen3
infrastructure. The I/O subsystem is supported by both a PCIe bus and an I/O bus similar to
16
IBM z13s Technical Guide
provides the path for data between memory and the PCIe features through