Nvlink - IBM Power Systems S822LC Technical Overview And Introduction

For a list of supported CAPI adapters, see 1.13.5, "CAPI enabled Infiniband adapters" on
page 27.

1.7.9 NVLink

NVLink is NVIDIA's new high-speed interconnect technology for GPU-accelerated computing.
Supported on SXM2 based Tesla P100 accelerator boards, NVLink significantly increases
performance for both GPU-to-GPU communications, and for GPU access to system memory.
Today, multiple GPUs are common in workstations as well as the nodes of HPC computing
clusters and deep learning training systems. A powerful interconnect is extremely valuable in
multiprocessing systems. Our vision for NVLink was to create an interconnect for GPUs that
would offer much higher bandwidth than PCI Express Gen3 (PCIe), and be compatible with
the GPU ISA to support shared memory multiprocessing workloads.
Support for the GPU ISA means that programs running on NVLink-connected GPUs can
execute directly on data in the memory of another GPU as well as on local memory. GPUs
can also perform atomic memory operations on remote GPU memory addresses, enabling
much tighter data sharing and improved application scaling.
NVLink uses NVIDIA's new High-Speed Signaling interconnect (NVHS). NVHS transmits data
over a differential pair running at up to 20 Gb/sec. Eight of these differential connections form
Sub-Link
a that sends data in one direction, and two sub-links - one for each direction - form a
Link
that connects two processors (GPU-to-GPU or GPU-to-CPU). A single Link supports up
to 40 GB/sec of bidirectional bandwidth between the endpoints. Multiple Links can be
Gangs
combined to form for even higher-bandwidth connectivity between processors. The
NVLink implementation in Tesla P100 supports up to four Links, allowing for a gang with an
aggregate maximum theoretical bandwidth of 160 GB/sec bidirectional bandwidth.
17
Chapter 1. Architecture and technical description