About This Manual This user manual describes NVIDIA® BlueField®-2 InfiniBand/VPI Programmable DPU (data processing unit). It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and a step-by-step plan of how to bring up the BlueField-2 DPUs.
This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with InfiniBand/VPI network and architecture specifications. Technical Support Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: •...
NVIDIA LinkX The NVIDIA® LinkX® product family of cables and transceivers provides the industry’s Interconnect most complete line of products for Cloud, HPC, Web 2.0, Enterprise, telco, storage and artificial intelligence, data center applications. Read more at https:// Solutions www.nvidia.com/en-us/networking/interconnect/ BlueField DPU OS NVIDIA®...
Introduction The NVIDIA® BlueField®-2 data processing unit (DPU) is a data center infrastructure on a chip optimized for traditional enterprise, high-performance computing (HPC), and modern cloud workloads, delivering a broad set of accelerated software-defined networking, storage, security, and management services. BlueField-2 DPU enables organizations to transform their IT infrastructures into state-of-the-art data centers that are accelerated, fully programmable and armed with “zero trust”...
Package Contents Before unpacking your DPU, it is important to ensure you meet all the system requirements listed above for a smooth installation. Be sure to inspect each piece of equipment shipped in the packing box. If anything is missing or damaged, contact your reseller. Card Package Item Description...
Page 10
Feature Description PCI Express Uses PCIe Gen 4.0 (16GT/s) through an x16 edge connector. Gen 1.1, 2.0, and 3.0 (PCIe) compatible. InfiniBand BlueField-3 DPU delivers low latency, high bandwidth, and computing efficiency for high- Architecture performance computing (HPC), artificial intelligence (AI), and hyperscale cloud data Specification center applications.
Page 11
64bit + 8bit ECC. BlueField-2 DPU The NVIDIA BlueField-2 DPU integrates eight 64-bit Armv8 A72 cores interconnected by a coherent mesh network, one DRAM controller, and an RDMA intelligent network adapter supporting up to 200Gb/s, an embedded PCIe switch with endpoint and root complex functionality, and up to 16 lanes of PCIe Gen 3.0/4.0.
Page 12
NVIDIA DPU effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol. RDMA and RDMA NVIDIA DPU, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over over Converged Converged InfiniBand/VPI) technology, delivers low-latency and high-performance over InfiniBand/VPI InfiniBand/VPI networks.
Page 13
DPU boards. PPS IN/OUT NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related features called “5T”. NVIDIA PTP and 5T software solutions, are designed to meet the most demanding PTP profiles. BlueField-2 incorporates an integrated Hardware Clock (PHC) that allows BlueField-2 to achieve sub-20u Sec accuracy and offers many timings- related functions such as time-triggered scheduling or time-based SND accelerations.
Supported Interfaces This section describes the DPU-supported interfaces. Each numbered interface that is referenced in the figures is described in the following table with a link to detailed information. The below figures are for illustration purposes only and might not reflect the current revision of the DPU.
FHHL DPU Layout and Interface Information Interfaces of MBF2H516A-EEEOT, MBF2H516A-EENOT, MBF2M516A-EECOT, MBF2M516A-EEEOT and MBF2M516A-EENOT Comp Print Side onent Side Item Interface Description DPU IC 8 cores PCI Express Interface PCIe Gen 4.0 through an x16 edge connector Networking Ports The network traffic is transmitted through the DPU QSFP56 connectors.
Page 16
Comp Print Side onent Side Interface Description DPU IC 8 cores PCI Express PCIe Gen 4.0 through an x16 edge connector Interface Networking Ports The network traffic is transmitted through the DPU QSFP56 connectors. The QSFP56 connectors allow the use of modules, optical and passive cable interconnect solutions. ...
Interfaces Detailed Description DPU IC NVIDIA® BlueField®-2 DPU is a family of advanced DPU IC solutions that integrate a coherent mesh of 64-bit Arm v8 A72 cores, an NVIDIA® ConnectX®-6 Dx network adapter front-end, and a PCI Express switch into a single chip. The powerful DPU IC architecture includes an Armv8 multicore processor array and enables customers to develop sophisticated applications and highly differentiated feature sets.
State Bi-Color LED (Yellow/Green) Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: Error Type Description LED Behavior Blinks until C access to the error is fixed networking ports fails Over-current Blinks until Over- condition of the...
Page 19
DPU OPN Maximum Trace Length on the Board MBF2M516C-EECOT 5 inch in 30-pin connector MBF2M516C-EESOT 10 inch in 20-pin connector MBF2H516C-EECOT MBF2H516C-EESOT MBF2M345A-HECOT 3.3inch MBF2M345A-HESOT MBF2H516A-EEEOT 6inch MBF2H516A-EENOT MBF2M516A-EECOT MBF2M516A-EEEOT MBF2M516A-EENOT UART Interface Connectivity A UART debug interface is available on the DPU cards via the NC-SI connector. The below table describes the UART interface location and connectivity per the NC-SI connector type on the DPU you have purchased.
NC-SI UART Interface Location and Connectivity Connector Type 30-pin For DPUs with onboard BMC, the UART interface is that of the BlueField-2 device. For DPUs without onboard BMC, the UART interface is that of the NIC BMC device. NC-SI Connector Pin # The signal on DPU without The signal on DPU with BF_UART0_RX...
USB Interface USB Cable Type USB 4-pin MBF2M345A- Use a 4-pin male connector to a male Type-A cable to connect to the vertical HECOT board. The cable is not included in the shipped DPU card box and should connector MBF2M345A- be ordered separately as part of the accessories kit (P/N: MBF25-DKIT).
For DPUs with integrated BMC: 1GbE OOB Management can be performed via the BlueField-2 device or the integrated BMC. 1GbE OOB Management LEDs Interface There are 2 OOB management LEDs, one green and one amber/yellow. The following table describes LED behavior for DPUs with or with onboard BMC.
The DPU PTP solution allows you to run any PTP stack on your host. With respect to testing and measurements, selected NVIDIA DPUs allow you to use the PPS-out signal from the onboard MMCX RA connectors; the DPU also allows measuring PTP in scale, with the PPS-In signal.
Hardware Installation Installation and initialization of the DPU require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. Please observe all safety warnings to avoid injury and prevent damage to system components. Note that not all warnings are relevant to all models.
Installation Procedure Overview The installation procedure of DPU involves the following steps: Step Procedure Direct Link Check the system’s requirements. Refer to System Requirements Pay attention to the airflow consideration within Refer to Airflow Requirements the host system Follow the safety precautions Refer to Safety Precautions Unpack the package...
Airflow Requirements DPU is offered with one airflow pattern: from the heatsink to the network ports. It is prohibited to use port-to-heatsink airflow as it may cause damage to the DPU. Please refer to the Specifications section for airflow numbers per card model. Software Requirements •...
Installation Instructions This section provides detailed instructions on installing your DPU in a system. Please note that the following figures are for illustration purposes only. To power up the FHHL P-Series DPUs, you need to connect a PCIe external power cable. The PCIe external power cable should be supplied by the customer.
When the DPU is properly seated, the port connectors are aligned with the slot opening, and the DPU faceplate is visible against the system chassis. Secure the DPU with the screw. For the FHHL 100Gb/s P-Series DPUs, install the PCIe external power cable. Connect the 6-pin power connector from the power supply to the power connector on the top edge of the DPU. Note that the connector and socket on the graphics card have a unique shape and connect one way only. ...
Support the weight of the cable before connecting the cable to the DPU card. Do this by using a cable holder or tying the cable to the rack. Determine the correct orientation of the connector to the card before inserting the connector.
connector orientation on the board interface. Carefully connect the 30 pin connector on the board to the mating cable. Connect the USB cable to the desired interface master, which can be a motherboard or any desired server. Do not leave the USB side unattended. Identify the DPU in your System On Linux Get the device location on the PCI bus by running lspci and locating lines with the string:...
Page 31
5. When the port connectors reach the top of the chassis window, gently pull the DPU in parallel to the motherboard. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
Application developers who see to create applications and services on top of DPU should use the DOCA SDK. An introduction to DOCA and how to get started with it is available at https:// developer.nvidia.com/networking/doca/getting-started. The latest DOCA SDK documentation is available at https://docs.nvidia.com/doca/sdk/. Arrow.com.
Check that both the DPU and its link are set to established the same speed and duplex settings Refer to the latest version of BlueField DPU SW Manual Forgot password needed to install/upgrade the and follow instructions under "Upgrading NVIDIA BlueField-2 DPU image BlueField DPU Software" section. Arrow.com. Arrow.com. Arrow.com.
Failure to meet the environmental requirements listed in this user manual may void the warranty. Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Specifications document, which is available through the customer portal following login. ...
Voltage: 12V Supply, Maximum power available through QSFP56 port: 6W Consumption and Airflow Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Specifications document, which is available through the customer portal following login. Temperature Operational 0°C to 45°C Environmental Non-operational -40°C to 70°C...
Voltage: 12V Supply, Consumption Maximum power available through QSFP56 port: 2.5W (per port) and Airflow Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Specifications document, which is available through the customer portal following login. Environmental Temperature Operational 0°C to 45°C...
Voltage: 12V Supply, Maximum power available through QSFP56 port: 2.5W (per port) Consumption and Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Airflow Specifications document, which is available through the customer portal following login. Environmental...
DPU Power Supply, Maximum power available through QSFP56 port: 2.5W (per port) Consumption and Airflow Power and airflow specifications are provided in NVIDIA BlueField-2 DPUs Power and Airflow Specifications document, which is available through the customer portal following login. Temperature Operational 0°C to 45°C...
Non-operational -40°C to 70°C Humidity 90% relative humidity Safety CB / cTUVus / CE CE / FCC / VCCI / ICES / RCM Regulatory RoHS RoHS compliant a. For both operational and non-operational states. DPU Mechanical Drawing and Dimensions All dimensions are in millimeters.
Finding the GUID/MAC on the DPU Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID).
Supported Servers and Power Cords Supported Servers Server support depends on the particular setup being used. The following is a partial list of servers with which the DPUs have been tested. For more information, please contact your NVIDIA representative. Vendor Name...
Page 43
Vendor Name Part Number Description 755742-001 HP GPU Power Cable for hp dl380 gen9 805123-001 HP GPU Power Cable for hp dl380 gen9 687955-001 HP GPU Power Cable for hp dl380 gen9 Dell TR5TP Dell R740R740xd GPU Riser to GPU Power Cable J30DG Dell R720 R720XD GPU Power Cable Arrow.com.
Pin Description PCI Express Interface The below tables lists the PCI Express pins description. For further details, please refer to PCI Express Interface. Pin # Signal Name Description Pin # Signal Name Description PRSNT1# Mechanical Present JTAG - Not Connected B5 SMCLK Host SMBus JTAG - Not Connected B6...
Pin # Signal Name Description Pin # Signal Name Description PERN12 PETP13 PETN13 PERP13 PERN13 PETP14 PETN14 PERP14 PERN14 PETP15 PETN15 PERP15 PERN15 PRSNT2# Mechanical Present External Power Supply Connector The below table provides the External Power Supply pins of the external power supply interfaces on the DPU.
Page 47
Link to NC-SI connector mapping MBF2H516C-EECOT Group 1 Table A - NC-SI Connector Pins MBF2H516C-EESOT MBF2M516C-EECOT MBF2M516C-EESOT Table B - NC-SI Connector Pins Group 2 MBF2M345A-HECOT MBF2M345A-HESOT MBF2M516A-EECOT Group 3 MBF2M516A-EEEOT MBF2M516A-EENOT MBF2H516A-EECOT Table C - NC-SI Connector Pins MBF2H516A-EEEOT MBF2H516A-EENOT Table A - NC-SI Connector Pins ...
Page 48
Pin# Pin Name Description/Comments RBT_RXD0 Output Receive data Out 0. Data signals from the network controller to the BMC. For baseboards, this pin should be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ pull- down resistor to GND on the baseboard between the BMC and the RBT isolator to prevent the signal from floating when no card is installed.
Page 49
Pin# Pin Name Description/Comments RBT_ISOLATE_N Input Signal used to indicate the SmartNIC has powered and is ready for NC-SI physical layer connection to be present. When low the baseboard circuitry will isolate the NC-SI connection to the SmartNIC. When high normal NC-SI RBT connectivity is available.
Page 50
Pin# Pin Name Description/Comments NCSI_TX_D1 Input Transmit Data In 0. Data signals from the BMC to the network controller. For baseboards, this pin should be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ pull- down resistor to GND on the baseboard between the RBT isolator and the DPUcable connector to prevent the card-side signals from floating when...
Page 51
Pin # Pin Name Description / Comments NCSI_REF_CLK Input RBT Reference clock. Synchronous clock reference for receive, transmit and control interface. The clock shall have a typical frequency of 50MHz ±50 ppm. For baseboards, this pin shall be connected between the baseboard NC-SI over RBT PHY and the DPU cable connector.
Page 52
Pin # Pin Name Description / Comments PACK_ID1, connected to Input See description above table. NC-SI PKG_ID BlueField-2 NIC_GPIO[47] Shall be connected to the Primary controller NC-SI PKG_ID pins to set the appropriate package ID. PKG_ID0 shall be connected to the endpoint device GPIO associated with Package ID[0].
Page 53
Pin # Pin Name Description / Comments NCSI_TX_D0 Input Transmit Data In 0. Data signals from the BMC to the network controller. For baseboards, this pin shall be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ...
Page 54
Pin # Pin Name Description / Comments ARM_NSRST# Output Open-drain signal This signal enables hard reset of the Arm Array from an external entity ARM_I2C1_SCL Input/Output Open-drain signal PACK_ID2, connected to Input See description above table. NC-SI PKG_ID BlueField-2 device Shall be connected to the Primary controller pin NIC_GPIO[46] NC-SI PKG_ID pins to set the appropriate package ID.
Page 55
Pin# Signal Name Description Comments Ground ARB_IN Input NCSI hardware NC-SI hardware arbitration input arbitration input If the baseboard supports multiple DPU cards connected to the same RBT interface, it shall implement logic that connects the RBT_ARB_IN pin of the first populated DPU card to its RBT_ARB_OUT pin if it is the only card present or to the RBT_ARB_OUT pin of the next populated...
Page 56
Pin# Signal Name Description Comments RX_D1 Output Receive data Output for SoC Receive data. Data signals from the network controller to the BMC. For baseboards, this pin shall be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ...
Page 57
Pin# Signal Name Description Comments TX_D1 Input Transmit data Input for SoC. Data signals from the BMC to the network controller. For baseboards, this pin shall be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ pull down resistor to GND on the baseboard between the RBT isolator and the DPUcable connector to prevent the card-...
Page 58
Engineering Samples Pinouts 30-pin NC-SI Connector The below table applies to the engineering samples of the following OPNs: MBF2H516C-EECOT, MBF2H516C-EESOT, MBF2M516C-EECOT and MBF2M516C-EESOT. Pin # Pinout Description / Comments NCSI_REF_CLK Input 50MHz REF CLK for NC-SI BUS Ground NCSI_ARB_IN Input NC-SI hardware arbitration input...
Page 59
Pin # Pinout Description / Comments Ground NIC_BMC_CTRL0, connected to BMC device pin Y3 (Open Drain) Not Connected BMC_TX5 Output UART transmit serial data output Not Connected BMC_RX5 Input UART transmit serial data input Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
Page 60
Document Revision History Date Description Nov. 2022 Updated Driver Installation and Update Oct. 2022 Added a warning to the following sections: • System Requirements • Hardware Requirements • Specifications Sep. 2022 Updated the following sections: • Driver Installation and Update • Specifications Jul.
Page 61
Date Description Dec. 2020 • Updated the NCSI interface pinouts • Removed the UART interface from "Supported Interfaces" • Updated "Package Contents" Dec. 2020 • Updated the NCSI interface pinouts • Added "Supported Servers and Power Cords" chapter Oct. 2020 First release Arrow.com.
Page 62
NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.