D.3 Shared Memory Communications - Direct Memory Access - IBM z13s Technical Manual

D.3 Shared Memory Communications - Direct Memory Access

This section describes the new Shared Memory Communication - Direct Memory Access
(SMC-D) functions implemented in IBM z13 and z13s Driver Level 27 servers.
Concepts
Co-location of multiple tiers of a workload onto a single z Systems physical server allows for
the exploitation of HiperSockets, an internal LAN technology that provides low-latency
communication between virtual machines within a physical z Systems CPC. HiperSockets is
implemented fully within z Systems firmware, so it requires no physical cabling or external
network connection to purchase, maintain, or replace. The lack of external components also
provides for a secure and low latency network connection because data transfer occurs much
like a cross-address-space memory move.
With the z13 (Driver 27) and z13s servers, IBM introduces SMC-D. SMC-D maintains the
socket-API transparency aspect of SMC-R so that applications that use TCP/IP
communications can benefit immediately without requiring any application software or IP
topology changes. SMC-D completes the overall Shared Memory Communications solution,
providing synergy with SMC-R. Both protocols use shared memory architectural concepts,
eliminating TCP/IP processing in the data path, yet preserving TCP/IP Qualities of Service for
connection management purposes.
From an operations standpoint, SMC-D is similar to SMC-R. The objective is to provide
consistent operations and management tasks for both SMC-D and SMC-R. SMC-D uses a
new virtual PCI adapter called Internal Shared Memory (ISM). The ISM Interfaces are
associated with IP interfaces (for example HiperSockets or OSA, ISM interfaces do not exist
without an IP interface). ISM interfaces are not defined in software. Instead, ISM interfaces
are dynamically defined and created, and automatically started and stopped. You do not need
to operate (Start or Stop) ISM interfaces. Unlike RoCE, ISM FIDs (PFIDs) are not defined in
software. Instead, they are auto-discovered based on their PNet ID.
SMC-R uses RDMA (RoCE), which is based on Queue Pair (QP) technology:
RC-QPs represent SMC Links (logical point-to-point connection).
RC-QPs over unique RNICs are logically bound together to form Link Groups (used for HA
and load balancing).
Link Groups (LGs) and Links are provided in many Netstat displays (for operational and
various network management tasks).
SMC-D over ISM does not use QPs:
Links and LGs based on QPs (or other hardware constructs) are not applicable to ISM. So
the SMC-D information in the Netstat command displays are related to ISM link
information rather than LGs.
SMC-D protocol (like SMC-R) has a design concept of a "logical point-to-point connection"
and therefore preserves the concept of an SMC-D Link (for various reasons that include
network administrative purposes).
Note: The SMC-D information in the Netstat command displays is related to ISM link
information (not LGs).
Appendix D. Shared Memory Communications
489