Related Manuals for Nvidia BlueField-2 InfiniBand/VPI DPU
Summary of Contents for Nvidia BlueField-2 InfiniBand/VPI DPU
- Page 1 NVIDIA BlueField-2 InfiniBand/VPI DPU User Guide Exported on Dec/12/2022 02:20 PM Arrow.com. Downloaded from...
-
Page 2: Table Of Contents
Table of Contents About This Manual ................5 Ordering Part Numbers................5 Half-Height Half-Length (HHHL) DPUs ............5 Full-Height Half-Length (FHHL) DPUs - P-Series / 2.75GHz ....... 5 Full-Height Half-Length (FHHL) DPUs - E-Series / 2.0GHz......... 6 Intended Audience ...................6 Related Documentation ................6 Document Conventions................7 Revision History ..................7 Introduction.................. - Page 3 Safety Warnings ..................24 Installation Procedure Overview..............25 System Requirements ................25 Hardware Requirements ..............25 Airflow Requirements ................. 26 Software Requirements ............... 26 Safety Precautions ................. 26 Unpacking ................... 26 Pre-Installation Checklist ................. 26 Installation Instructions................27 Cables and Modules................28 Networking Cable Installation ...............
- Page 4 NC-SI Management Interface ..............46 Table A - NC-SI Connector Pins ............... 47 Table B - NC-SI Connector Pins............... 50 Table C - NC-SI Connector Pins............... 54 Engineering Samples Pinouts ..............58 30-pin NC-SI Connector ............... 58 Document Revision History ..............60 ...
-
Page 5: About This Manual
About This Manual This user manual describes NVIDIA® BlueField®-2 InfiniBand/VPI Programmable DPU (data processing unit). It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and a step-by-step plan of how to bring up the BlueField-2 DPUs. -
Page 6: Full-Height Half-Length (Fhhl) Dpus - E-Series / 2.0Ghz
This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with InfiniBand/VPI network and architecture specifications. Technical Support Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: •... -
Page 7: Document Conventions
NVIDIA LinkX The NVIDIA® LinkX® product family of cables and transceivers provides the industry’s Interconnect most complete line of products for Cloud, HPC, Web 2.0, Enterprise, telco, storage and artificial intelligence, data center applications. Read more at https:// Solutions www.nvidia.com/en-us/networking/interconnect/ BlueField DPU OS NVIDIA®... -
Page 8: Introduction
Introduction The NVIDIA® BlueField®-2 data processing unit (DPU) is a data center infrastructure on a chip optimized for traditional enterprise, high-performance computing (HPC), and modern cloud workloads, delivering a broad set of accelerated software-defined networking, storage, security, and management services. BlueField-2 DPU enables organizations to transform their IT infrastructures into state-of-the-art data centers that are accelerated, fully programmable and armed with “zero trust”... -
Page 9: Package Contents
Package Contents Before unpacking your DPU, it is important to ensure you meet all the system requirements listed above for a smooth installation. Be sure to inspect each piece of equipment shipped in the packing box. If anything is missing or damaged, contact your reseller. Card Package Item Description... - Page 10 Feature Description PCI Express Uses PCIe Gen 4.0 (16GT/s) through an x16 edge connector. Gen 1.1, 2.0, and 3.0 (PCIe) compatible. InfiniBand BlueField-3 DPU delivers low latency, high bandwidth, and computing efficiency for high- Architecture performance computing (HPC), artificial intelligence (AI), and hyperscale cloud data Specification center applications.
- Page 11 64bit + 8bit ECC. BlueField-2 DPU The NVIDIA BlueField-2 DPU integrates eight 64-bit Armv8 A72 cores interconnected by a coherent mesh network, one DRAM controller, and an RDMA intelligent network adapter supporting up to 200Gb/s, an embedded PCIe switch with endpoint and root complex functionality, and up to 16 lanes of PCIe Gen 3.0/4.0.
- Page 12 NVIDIA DPU effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol. RDMA and RDMA NVIDIA DPU, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over over Converged Converged InfiniBand/VPI) technology, delivers low-latency and high-performance over InfiniBand/VPI InfiniBand/VPI networks.
- Page 13 DPU boards. PPS IN/OUT NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related features called “5T”. NVIDIA PTP and 5T software solutions, are designed to meet the most demanding PTP profiles. BlueField-2 incorporates an integrated Hardware Clock (PHC) that allows BlueField-2 to achieve sub-20u Sec accuracy and offers many timings- related functions such as time-triggered scheduling or time-based SND accelerations.
-
Page 14: Supported Interfaces
Supported Interfaces This section describes the DPU-supported interfaces. Each numbered interface that is referenced in the figures is described in the following table with a link to detailed information. The below figures are for illustration purposes only and might not reflect the current revision of the DPU. -
Page 15: Fhhl Dpu Layout And Interface Information
FHHL DPU Layout and Interface Information Interfaces of MBF2H516A-EEEOT, MBF2H516A-EENOT, MBF2M516A-EECOT, MBF2M516A-EEEOT and MBF2M516A-EENOT Comp Print Side onent Side Item Interface Description DPU IC 8 cores PCI Express Interface PCIe Gen 4.0 through an x16 edge connector Networking Ports The network traffic is transmitted through the DPU QSFP56 connectors. - Page 16 Comp Print Side onent Side Interface Description DPU IC 8 cores PCI Express PCIe Gen 4.0 through an x16 edge connector Interface Networking Ports The network traffic is transmitted through the DPU QSFP56 connectors. The QSFP56 connectors allow the use of modules, optical and passive cable interconnect solutions. ...
-
Page 17: Interfaces Detailed Description
Interfaces Detailed Description DPU IC NVIDIA® BlueField®-2 DPU is a family of advanced DPU IC solutions that integrate a coherent mesh of 64-bit Arm v8 A72 cores, an NVIDIA® ConnectX®-6 Dx network adapter front-end, and a PCI Express switch into a single chip. The powerful DPU IC architecture includes an Armv8 multicore processor array and enables customers to develop sophisticated applications and highly differentiated feature sets. -
Page 18: Pci Express Interface
State Bi-Color LED (Yellow/Green) Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: Error Type Description LED Behavior Blinks until C access to the error is fixed networking ports fails Over-current Blinks until Over- condition of the... - Page 19 DPU OPN Maximum Trace Length on the Board MBF2M516C-EECOT 5 inch in 30-pin connector MBF2M516C-EESOT 10 inch in 20-pin connector MBF2H516C-EECOT MBF2H516C-EESOT MBF2M345A-HECOT 3.3inch MBF2M345A-HESOT MBF2H516A-EEEOT 6inch MBF2H516A-EENOT MBF2M516A-EECOT MBF2M516A-EEEOT MBF2M516A-EENOT UART Interface Connectivity A UART debug interface is available on the DPU cards via the NC-SI connector. The below table describes the UART interface location and connectivity per the NC-SI connector type on the DPU you have purchased.
-
Page 20: Usb Interfaces
NC-SI UART Interface Location and Connectivity Connector Type 30-pin For DPUs with onboard BMC, the UART interface is that of the BlueField-2 device. For DPUs without onboard BMC, the UART interface is that of the NIC BMC device. NC-SI Connector Pin # The signal on DPU without The signal on DPU with BF_UART0_RX... -
Page 21: Bmc Interface
USB Interface USB Cable Type USB 4-pin MBF2M345A- Use a 4-pin male connector to a male Type-A cable to connect to the vertical HECOT board. The cable is not included in the shipped DPU card box and should connector MBF2M345A- be ordered separately as part of the accessories kit (P/N: MBF25-DKIT). -
Page 22: External Pcie Power Supply Connector
For DPUs with integrated BMC: 1GbE OOB Management can be performed via the BlueField-2 device or the integrated BMC. 1GbE OOB Management LEDs Interface There are 2 OOB management LEDs, one green and one amber/yellow. The following table describes LED behavior for DPUs with or with onboard BMC. -
Page 23: Emmc Interface
The DPU PTP solution allows you to run any PTP stack on your host. With respect to testing and measurements, selected NVIDIA DPUs allow you to use the PPS-out signal from the onboard MMCX RA connectors; the DPU also allows measuring PTP in scale, with the PPS-In signal. -
Page 24: Hardware Installation
Hardware Installation Installation and initialization of the DPU require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. Please observe all safety warnings to avoid injury and prevent damage to system components. Note that not all warnings are relevant to all models. -
Page 25: Installation Procedure Overview
Installation Procedure Overview The installation procedure of DPU involves the following steps: Step Procedure Direct Link Check the system’s requirements. Refer to System Requirements Pay attention to the airflow consideration within Refer to Airflow Requirements the host system Follow the safety precautions Refer to Safety Precautions Unpack the package... -
Page 26: Airflow Requirements
Airflow Requirements DPU is offered with one airflow pattern: from the heatsink to the network ports. It is prohibited to use port-to-heatsink airflow as it may cause damage to the DPU. Please refer to the Specifications section for airflow numbers per card model. Software Requirements •... -
Page 27: Installation Instructions
Installation Instructions This section provides detailed instructions on installing your DPU in a system. Please note that the following figures are for illustration purposes only. To power up the FHHL P-Series DPUs, you need to connect a PCIe external power cable. The PCIe external power cable should be supplied by the customer. -
Page 28: Cables And Modules
When the DPU is properly seated, the port connectors are aligned with the slot opening, and the DPU faceplate is visible against the system chassis. Secure the DPU with the screw. For the FHHL 100Gb/s P-Series DPUs, install the PCIe external power cable. Connect the 6-pin power connector from the power supply to the power connector on the top edge of the DPU. Note that the connector and socket on the graphics card have a unique shape and connect one way only. ... -
Page 29: Uart Cable Installation
Support the weight of the cable before connecting the cable to the DPU card. Do this by using a cable holder or tying the cable to the rack. Determine the correct orientation of the connector to the card before inserting the connector. -
Page 30: Identify The Dpu In Your System
connector orientation on the board interface. Carefully connect the 30 pin connector on the board to the mating cable. Connect the USB cable to the desired interface master, which can be a motherboard or any desired server. Do not leave the USB side unattended. Identify the DPU in your System On Linux Get the device location on the PCI bus by running lspci and locating lines with the string:... - Page 31 5. When the port connectors reach the top of the chassis window, gently pull the DPU in parallel to the motherboard. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
-
Page 32: Driver Installation And Update
Application developers who see to create applications and services on top of DPU should use the DOCA SDK. An introduction to DOCA and how to get started with it is available at https:// developer.nvidia.com/networking/doca/getting-started. The latest DOCA SDK documentation is available at https://docs.nvidia.com/doca/sdk/. Arrow.com. -
Page 33: Troubleshooting
Check that both the DPU and its link are set to established the same speed and duplex settings Refer to the latest version of BlueField DPU SW Manual Forgot password needed to install/upgrade the and follow instructions under "Upgrading NVIDIA BlueField-2 DPU image BlueField DPU Software" section. Arrow.com. Arrow.com. Arrow.com. -
Page 34: Specifications
Failure to meet the environmental requirements listed in this user manual may void the warranty. Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Specifications document, which is available through the customer portal following login. ... -
Page 35: Mbf2M516A-Eecot / Mbf2M516A-Eeeot / Mbf2M516A-Eenot Specifications
Voltage: 12V Supply, Maximum power available through QSFP56 port: 6W Consumption and Airflow Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Specifications document, which is available through the customer portal following login. Temperature Operational 0°C to 45°C Environmental Non-operational -40°C to 70°C... -
Page 36: Mbf2H516A-Eenot / Mbf2H516A-Eeeot Specifications
Voltage: 12V Supply, Consumption Maximum power available through QSFP56 port: 2.5W (per port) and Airflow Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Specifications document, which is available through the customer portal following login. Environmental Temperature Operational 0°C to 45°C... -
Page 37: Mbf2M516C-Eecot / Mbf2M516C-Eesot Specifications
Voltage: 12V Supply, Maximum power available through QSFP56 port: 2.5W (per port) Consumption and Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Airflow Specifications document, which is available through the customer portal following login. Environmental... -
Page 38: Mbf2H516C-Eecot/ Mbf2H516C-Eesot Specifications
DPU Power Supply, Maximum power available through QSFP56 port: 2.5W (per port) Consumption and Airflow Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Specifications document, which is available through the customer portal following login. Temperature Operational 0°C to 45°C... -
Page 39: Dpu Mechanical Drawing And Dimensions
Non-operational -40°C to 70°C Humidity 90% relative humidity Safety CB / cTUVus / CE CE / FCC / VCCI / ICES / RCM Regulatory RoHS RoHS compliant a. For both operational and non-operational states. DPU Mechanical Drawing and Dimensions All dimensions are in millimeters. - Page 40 Group # Bracket MBF2H516A-EEEOT MBF2H516A-EENOT MBF2M516A-EECOT MBF2M516A-EEEOT MBF2M516A-EENOT MBF2H516C-EECOT MBF2H516C-EESOT MBF2M516C-EECOT MBF2M516C-EESOT Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
-
Page 41: Finding The Guid/Mac On The Dpu
Finding the GUID/MAC on the DPU Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID). -
Page 42: Supported Servers And Power Cords
Supported Servers and Power Cords Supported Servers Server support depends on the particular setup being used. The following is a partial list of servers with which the DPUs have been tested. For more information, please contact your NVIDIA representative. Vendor Name... - Page 43 Vendor Name Part Number Description 755742-001 HP GPU Power Cable for hp dl380 gen9 805123-001 HP GPU Power Cable for hp dl380 gen9 687955-001 HP GPU Power Cable for hp dl380 gen9 Dell TR5TP Dell R740R740xd GPU Riser to GPU Power Cable J30DG Dell R720 R720XD GPU Power Cable Arrow.com.
-
Page 44: Pin Description
Pin Description PCI Express Interface The below tables lists the PCI Express pins description. For further details, please refer to PCI Express Interface. Pin # Signal Name Description Pin # Signal Name Description PRSNT1# Mechanical Present JTAG - Not Connected B5 SMCLK Host SMBus JTAG - Not Connected B6... - Page 45 Pin # Signal Name Description Pin # Signal Name Description RSVD RSVD PETP4 PETN4 PERP4 PERN4 PETP5 PETN5 PERP5 PERN5 PETP6 PETN6 PERP6 PERN6 PETP7 PETN7 PERP7 PERN7 RSVD RSVD PETP8 PETN8 PERP8 PERN8 PETP9 PETN9 PERP9 PERN9 PETP10 PETN10 PERP10 PERN10 PETP11...
-
Page 46: External Power Supply Connector
Pin # Signal Name Description Pin # Signal Name Description PERN12 PETP13 PETN13 PERP13 PERN13 PETP14 PETN14 PERP14 PERN14 PETP15 PETN15 PERP15 PERN15 PRSNT2# Mechanical Present External Power Supply Connector The below table provides the External Power Supply pins of the external power supply interfaces on the DPU. - Page 47 Link to NC-SI connector mapping MBF2H516C-EECOT Group 1 Table A - NC-SI Connector Pins MBF2H516C-EESOT MBF2M516C-EECOT MBF2M516C-EESOT Table B - NC-SI Connector Pins Group 2 MBF2M345A-HECOT MBF2M345A-HESOT MBF2M516A-EECOT Group 3 MBF2M516A-EEEOT MBF2M516A-EENOT MBF2H516A-EECOT Table C - NC-SI Connector Pins MBF2H516A-EEEOT MBF2H516A-EENOT Table A - NC-SI Connector Pins ...
- Page 48 Pin# Pin Name Description/Comments RBT_RXD0 Output Receive data Out 0. Data signals from the network controller to the BMC. For baseboards, this pin should be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ pull- down resistor to GND on the baseboard between the BMC and the RBT isolator to prevent the signal from floating when no card is installed.
- Page 49 Pin# Pin Name Description/Comments RBT_ISOLATE_N Input Signal used to indicate the SmartNIC has powered and is ready for NC-SI physical layer connection to be present. When low the baseboard circuitry will isolate the NC-SI connection to the SmartNIC. When high normal NC-SI RBT connectivity is available.
- Page 50 Pin# Pin Name Description/Comments NCSI_TX_D1 Input Transmit Data In 0. Data signals from the BMC to the network controller. For baseboards, this pin should be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ pull- down resistor to GND on the baseboard between the RBT isolator and the DPUcable connector to prevent the card-side signals from floating when...
- Page 51 Pin # Pin Name Description / Comments NCSI_REF_CLK Input RBT Reference clock. Synchronous clock reference for receive, transmit and control interface. The clock shall have a typical frequency of 50MHz ±50 ppm. For baseboards, this pin shall be connected between the baseboard NC-SI over RBT PHY and the DPU cable connector.
- Page 52 Pin # Pin Name Description / Comments PACK_ID1, connected to Input See description above table. NC-SI PKG_ID BlueField-2 NIC_GPIO[47] Shall be connected to the Primary controller NC-SI PKG_ID pins to set the appropriate package ID. PKG_ID0 shall be connected to the endpoint device GPIO associated with Package ID[0].
- Page 53 Pin # Pin Name Description / Comments NCSI_TX_D0 Input Transmit Data In 0. Data signals from the BMC to the network controller. For baseboards, this pin shall be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ...
- Page 54 Pin # Pin Name Description / Comments ARM_NSRST# Output Open-drain signal This signal enables hard reset of the Arm Array from an external entity ARM_I2C1_SCL Input/Output Open-drain signal PACK_ID2, connected to Input See description above table. NC-SI PKG_ID BlueField-2 device Shall be connected to the Primary controller pin NIC_GPIO[46] NC-SI PKG_ID pins to set the appropriate package ID.
- Page 55 Pin# Signal Name Description Comments Ground ARB_IN Input NCSI hardware NC-SI hardware arbitration input arbitration input If the baseboard supports multiple DPU cards connected to the same RBT interface, it shall implement logic that connects the RBT_ARB_IN pin of the first populated DPU card to its RBT_ARB_OUT pin if it is the only card present or to the RBT_ARB_OUT pin of the next populated...
- Page 56 Pin# Signal Name Description Comments RX_D1 Output Receive data Output for SoC Receive data. Data signals from the network controller to the BMC. For baseboards, this pin shall be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ...
- Page 57 Pin# Signal Name Description Comments TX_D1 Input Transmit data Input for SoC. Data signals from the BMC to the network controller. For baseboards, this pin shall be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ pull down resistor to GND on the baseboard between the RBT isolator and the DPUcable connector to prevent the card-...
- Page 58 Engineering Samples Pinouts 30-pin NC-SI Connector The below table applies to the engineering samples of the following OPNs: MBF2H516C-EECOT, MBF2H516C-EESOT, MBF2M516C-EECOT and MBF2M516C-EESOT. Pin # Pinout Description / Comments NCSI_REF_CLK Input 50MHz REF CLK for NC-SI BUS Ground NCSI_ARB_IN Input NC-SI hardware arbitration input...
- Page 59 Pin # Pinout Description / Comments Ground NIC_BMC_CTRL0, connected to BMC device pin Y3 (Open Drain) Not Connected BMC_TX5 Output UART transmit serial data output Not Connected BMC_RX5 Input UART transmit serial data input Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
- Page 60 Document Revision History Date Description Nov. 2022 Updated Driver Installation and Update Oct. 2022 Added a warning to the following sections: • System Requirements • Hardware Requirements • Specifications Sep. 2022 Updated the following sections: • Driver Installation and Update • Specifications Jul.
- Page 61 Date Description Dec. 2020 • Updated the NCSI interface pinouts • Removed the UART interface from "Supported Interfaces" • Updated "Package Contents" Dec. 2020 • Updated the NCSI interface pinouts • Added "Supported Servers and Power Cords" chapter Oct. 2020 First release Arrow.com.
- Page 62 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
- Page 63 Copyright © 2022 NVIDIA Corporation & affiliates. All Rights Reserved. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.