Nvidia BlueField-3 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Network Hardware
  5. BlueField-3
  6. User manual
Nvidia BlueField-3 User Manual

Nvidia BlueField-3 User Manual

Networking platform
Hide thumbs Also See for BlueField-3:
Table of Contents

Advertisement

Quick Links

Download this manual
NVIDIA BlueField-3 Networking Platform
User Guide

Advertisement

Table of Contents
loading

Related Manuals for Nvidia BlueField-3

Summary of Contents for Nvidia BlueField-3

  • Page 1 NVIDIA BlueField-3 Networking Platform User Guide...

  • Page 2: Table Of Contents

    BlueField DPU Administrator Quick Start Guide........15 Prerequisites for Initial BlueField DPU Deployment ........15 First-time Installation Procedure ............15 Supported Interfaces ..............16 BlueField-3 SuperNICs Layout and Interface Information......16 BlueField-3 DPUs Layout and Interface Information........18 Interfaces Detailed Description............19 3.3.1 DPU System-on-Chip (SoC).............
  • Page 3 External Power Supply Connector ............29 NC-SI Management Interface .............. 30 Cabline CA-II Plus Connectors Pinouts ........... 32 4.4.1 Component Side................. 32 4.4.2 Print Side ................. 34 Hardware Installation and PCIe Bifurcation ........36 Safety Warnings ................36 Installation Procedure Overview............37 System Requirements ..............
  • Page 4 Specifications................57 B3140H SuperNICs Specifications ............57 B3140L SuperNICs Specifications ............58 B3220L SuperNICs Specifications ............59 B3210L SuperNICs Specifications ............61 B3240 DPUs Specifications..............62 B3210 DPUs Specifications..............63 B3210E DPUs Specifications ............... 64 B3220 DPUs Specifications..............65 DPUs Mechanical Drawing and Dimensions..........66 8.10 Bracket Mechanical Drawings .............
  • Page 5 This User Manual describes NVIDIA BlueField-3 DPUs (Data Processing Unit) and SuperNICs. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and a step-by-step plan of how to bring up the BlueField-3 DPUs and SuperNICs.
  • Page 6 Mass 3B6-00 Series / Ports 00001 Produc SC-EA0 16 Arm- HDR100 QSFP1 117  tion cores 100Gb/s  Ethernet : 100GbE (Default) BlueField-3 SuperNICs Model NVIDIA Series/ Data No. of PCIe Integr PSID Lifecy Cores Transmis Ports Suppo Boar ated Form sion...
  • Page 7 Model NVIDIA Series/ Data No. of PCIe Integr PSID Lifecy Cores Transmis Ports Suppo Boar ated Form sion Factor Rate ✔ ✔ B3220L 900-9D3B InfiniBand: 2-Ports PCIe 16GB MT_000 Mass Single- 4-00CV- Series / NDR200 QSFP11 Gen 5.0 0001093 Producti...
  • Page 8 Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: • URL: www.nvidia.com → Support • E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA M-1 Global Support Services, please see your contract for details regarding Technical Support.

  • Page 9: Introduction

    The NVIDIA® BlueField®-3 networking platform is designed to accelerate data center infrastructure workloads and usher in the era of accelerated computing and AI. Supporting both Ethernet and InfiniBand connectivity, BlueField-3 offers speeds up to 400 gigabits per second (Gb/s). It combines powerful computing with software-defined hardware accelerators for networking, storage, and cybersecurity—all fully programmable through the NVIDIA DOCA™...

  • Page 10: Package Contents

    The cable is not included in the package. For further details, please refer External PCIe Power Supply Connector. Item Description Card 1x BlueField-3 DPU Accessories 1x tall bracket (shipped assembled on the card) 1.2.2 Accessories Kit  This is an optional accessories kit used for debugging purposes and can be ordered separately. ...

  • Page 11: Features And Benefits

    This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability. Feature Description InfiniBand BlueField-3 DPU delivers low latency, high bandwidth, and computing efficiency for high- Architecture performance computing (HPC), artificial intelligence (AI), and hyperscale cloud data Specification centers applications.
  • Page 12 Solder down on-board. 128bit + 16bit ECC BlueField-3 IC The NVIDIA BlueField-3 DPU integrates x8 / x16 Armv8.2+ A78 Hercules cores (64-bit) is interconnected by a coherent mesh network, one DRAM controller, an RDMA intelligent network adapter supporting up to 400Gb/s, an embedded PCIe switch with endpoint and...
  • Page 13 NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. NVIDIA DPU effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.
  • Page 14 TCO and power consumption Out-of-Band The NVIDIA BlueField-3 DPU incorporates a 1GbE RJ45 out-of-band port that allows the Management network operator to establish trust boundaries in accessing the management function to apply it to network resources.

  • Page 15: Bluefield Dpu Administrator Quick Start Guide

    Guide. Not sure which guide to follow? For more details on the different BlueField user types, please refer to the NVIDIA BlueField and DOCA User Types document. 2.1 Prerequisites for Initial BlueField DPU Deployment Unable to render include or excerpt-include. Could not retrieve page.

  • Page 16: Supported Interfaces

    The below figures are for illustration purposes only and might not reflect the current revision of the DPU. 3.1 BlueField-3 SuperNICs Layout and Interface Information SuperNIC Component Side...
  • Page 17 Interface Description RTC Battery Battery holder for RTC eMMC  x8 NAND flash SuperNIC Component Side SuperNIC Print Side FHHL Single-Slot Dual-Port DPUs Model: B3220L 900-9D3B4-00CV-EA0 900-9D3B4-00SV-EA0 Model:  B3210L 900-9D3B4-00CC-EA0 900-9D3B4-00SC-EA0 FHHL Single-Slot Single-Port DPUs Model: B3140L 900-9D3B4-00EN-EA0 900-9D3B4-00PN-EA0 Interface Description DPU SoC DPU SoC 8/16 Arm-Cores ...

  • Page 18: Bluefield-3 Dpus Layout And Interface Information

    DPU BMC  SSD Interface  128GB RTC Battery Battery holder for RTC eMMC  x8 NAND flash 3.2 BlueField-3 DPUs Layout and Interface Information DPU Component Side DPU Print Side FHHL Single-Slot Dual-Port DPUs with PCIe Extension Option Model: B3210E 900-9D3B6-00CC-EA0    ...

  • Page 19: Interfaces Detailed Description

    Leverages the rich Arm software ecosystem and introduces the ability to offload the x86 software stack. At the heart of BlueField-3, the ConnectX-7 network offload controller with RDMA and RDMA over Converged Ethernet (RoCE) technology delivers cutting-edge performance for networking and storage applications such as NVMe over Fabrics. Advanced features include an embedded virtual...

  • Page 20: Networking Interface

    3.3.1.1 Encryption Applies to Crypto enabled OPNs. The DPU and SuperNICs addresses the concerns of modern data centers by combining hardware encryption accelerators with embedded software and fully integrated advanced network capabilities, making it an ideal platform for developing proprietary security applications. It enables a distributed security architecture by isolating and protecting each workload and providing flexible control and visibility at the server and workload level;...

  • Page 21: Pci Express Interface

    2.5, 5.0, or 8.0, 16.0 or 32.0 GT/s link rate x16 lanes • Auto-negotiates to x16, x8, x4, x2, or x1 3.3.5 DDR5 SDRAM On-Board Memory  The BlueField-3 incorporate 10 or 20 units of DDR5 SDRAM. See the following table for DDR5 SDRAM memory specifications per ordering part number. Model OPNs...

  • Page 22: Nc-Si Management Interface

    Interface for pins. 3.3.7 UART Interface Connectivity A UART debug interface is available on BlueField-3 boards via a 20-pin NC-SI connector. For DPUs without onboard BMC hardware, the UART interface is that of the BlueField-3 device. For DPUs with onboard BMC hardware, the UART interface is that of the NIC BMC device. The connectivity for both...

  • Page 23: 1Gbe Oob Management Interface

    (P/N: MBF35-DKIT). 3.3.9 1GbE OOB Management Interface BlueField-3 incorporates a 1GbE RJ45 out-of-band port that allows the network operator to establish trust boundaries in accessing the management function to apply it to network resources. It can also be used to ensure management connectivity (including the ability to determine the status of any network component) independent of other in-band network components' status.

  • Page 24: External Pcie Power Supply Connector

    • Do not link the CPU power cable to the BlueField-3 DPU's PCIe Aux power, as their pin configurations differ. Utilizing the CPU power cable in this manner is strictly prohibited and can potentially damage the BlueField-3 DPU. Please refer to...

  • Page 25: Cabline Ca-Ii Plus Connectors

    3.3.12 Cabline CA-II Plus Connectors Applies to the following OPNs: B3220 DPUs: 900-9D3B6-00CV-AA0 and 900-9D3B6-00SV-AA0 B3240 DPUs: 900-9D3B6-00CN-AB0 and 900-9D3B6-00SN-AB0 B3210 DPUs: 900-9D3B6-00CC-AA0 and 900-9D3B6-00SC-AA0 B3210E DPUs: 900-9D3B6-00CC-EA0 and 900-9D3B6-00SC-EA0 The Cabline CA-II connectors on the DPU enable connectivity to an additional PCIe x16 bus in addition to the PCIe x16 bus available through the golden-fingers.

  • Page 26: Rtc Battery

    3.3.15 RTC Battery  BlueField-3 incorporates a Coin type Lithium battery CR621 for RTC (Real Time Clock). 3.3.16 eMMC Interface The eMMC is an x8 NAND flash used for Arm boot and operating system storage. Memory size is 128GB, where it is effectively pSLC 40GB.

  • Page 27: Pinouts Description

    4 Pinouts Description 4.1 PCI Express Interface The following table lists the PCI Express pins description. For further details, please refer to PCI Express Interface. DPU PCI Express x16 Pin Description Signal Description Signal Description Name Name PRSNT1# Mechanical Present JTAG - Not Connected SMCLK Host SMBus JTAG - Not Connected...
  • Page 28 Signal Description Signal Description Name Name PERN2 PETP3 PETN3 PERP3 PERN3 RSVD RSVD RSVD RSVD PETP4 PETN4 PERP4 PERN4 PETP5 PETN5 PERP5 PERN5 PETP6 PETN6 PERP6 PERN6 PETP7 PETN7 PERP7 PERN7 RSVD RSVD PETP8 PETN8 PERP8 PERN8 PETP9 PETN9 PERP9 PERN9 PETP10 PETN10...

  • Page 29: External Power Supply Connector

    Signal Description Signal Description Name Name PETP11 PETN11 PERP11 PERN11 PETP12 PETN12 PERP12 PERN12 PETP13 PETN13 PERP13 PERN13 PETP14 PETN14 PERP14 PERN14 PETP15 PETN15 PERP15 PERN15 PRSNT2# Mechanical Present 4.2 External Power Supply Connector The following table provides the External Power Supply pins of the external power supply interfaces on the DPU.

  • Page 30: Nc-Si Management Interface

    Pin Number Description Sense0 4.3 NC-SI Management Interface The following table list the NC-SI management interface pinout descriptions. For further details, please refer to NC-SI Management Interface. Signal Signal Description Name Ground PKG_ID1 Input (to NC-SI PKG_ID BlueField- Shoul be connected to the Primary controller NC-SI PKG_ID pins to set the appropriate package ID.
  • Page 31 Signal Signal Description Name RBT_CRS_D Output Carrier sense/receive data valid. This signal is used to indicate to the baseboard that the carrier sense/receive data is valid. For baseboards, this pin should be connected between the baseboard NC-SI over RBT PHY and the connector. This signal requires a 100 kΩ pull down resistor on the baseboard between the BMC and the RBT isolator to prevent the signal from floating when no DPU is installed.

  • Page 32: Cabline Ca-Ii Plus Connectors Pinouts

    Signal Signal Description Name Ground RBT_ARB_O Input NC-SI hardware arbitration output. If the baseboard supports multiple DPUs cards connected to the same RBT interface, it should implement logic that connects the RBT_ARB_OUT pin of the first populated DPU card to its RBT_ARB_IN pin if it is the only card present or to the RBT_ARB_IN pin of the next populated card and so on sequentially for all cards on the specified RBT bus to ensure the arbitration ring is complete.
  • Page 33 Pin# Signal Name Wire Type AWG# Pin# on other PCIE_CPU_CX_11P Micro coax GND  GND BAR PCIE_CPU_CX_10N Micro coax PCIE_CPU_CX_10P Micro coax GND  GND BAR PCIE_CPU_CX_9N  Micro coax PCIE_CPU_CX_9P Micro coax GND  GND BAR PCIE_CPU_CX_8N Micro coax PCIE_CPU_CX_8P Micro coax GND  GND BAR PCIE_CPU_CX_7N ...

  • Page 34: Print Side

    Pin# Signal Name Wire Type AWG# Pin# on other I2C_DPU_BMC_SCL Micro coax AUX_PGOOD Micro coax No wire Micro coax I2C_AUX_SCL Micro coax I2C_AUX_SDA Micro coax S_PRSNT1_L  Micro coax No wire 4.4.2 Print Side Pin# Signal Name Wire Type AWG# Pin# on other SER_CLK Micro coax SER_CAPTURE...
  • Page 35 Pin# Signal Name Wire Type AWG# Pin# on other PCIE_CPU_CX_5P Micro coax PCIE_CPU_CX_5N Micro coax GND  GND BAR PCIE_CPU_CX_6P Micro coax PCIE_CPU_CX_6N Micro coax GND  GND BAR PCIE_CPU_CX_7P Micro coax PCIE_CPU_CX_7N Micro coax GND  GND BAR PCIE_CPU_CX_8P Micro coax PCIE_CPU_CX_8N Micro coax GND ...

  • Page 36: Hardware Installation And Pcie Bifurcation

    5 Hardware Installation and PCIe Bifurcation Installation and initialization of the DPU/SuperNIC require attention to the mechanical attributes, power specification, and precautions for electronic equipment. 5.1 Safety Warnings Safety warnings are provided here in the English language.  Please observe all safety warnings to avoid injury and prevent damage to system components. Note that not all warnings are relevant to all models.

  • Page 37: Installation Procedure Overview

    5.2 Installation Procedure Overview The installation procedure of DPU involves the following steps: Step Procedure Direct Link Check the system’s requirements. Refer to System Requirements Pay attention to the airflow consideration within Refer to Airflow Requirements the host system Follow the safety precautions Refer to Safety Unpack the package...

  • Page 38: Airflow Requirements

    • Software Stacks -BlueField-3 DPU/SuperNIC is shipped with Ubuntu – a Linux commercial operating system – which includes the NVIDIA OFED stack (MLNX_OFED), and is capable of running all customer-based Linux applications seamlessly. For more information, please refer to the DOCA SDK documentation or NVIDIA BlueField DPU BSP. ...

  • Page 39: Installation Instructions

    documentation for instructions. Before you install the DPU, make sure that the system is disconnected from power. 5.7 Installation Instructions This section provides detailed instructions on how to install your DPU/SuperNIC in a system. Choose the installation instructions according to the DPU configuration you would like to use. OPNs Installation Instructions All DPUs...

  • Page 40: Dpu Power-Up And Power-Down Sequences

    After plugging in a cable, lock the connector using the latching mechanism particular to the cable vendor. When data is being transferred the Green LED will blink. Make sure not to impede the air exhaust flow through the ventilation holes. Use cable lengths that allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack.

  • Page 41: Power-Down Sequence

    The 12V_ATX can be powered up after the 12V_PCIE, with a maximum delay of 2 seconds. The below graph illustrates the delay between the 12V_ATX and 12V_PCIE voltages at power-up. 5.9.2 Power-Down Sequence The 12V_PCIE voltage can be powered down simultaneously with the 12V_ATX voltage, or within a maximum delay of 200usec.

  • Page 42: Pcie X16 Dpu/Supernic Installation Instructions

    5.10 PCIe x16 DPU/SuperNIC Installation Instructions 5.10.1 Installation Instructions This section provides detailed instructions on how to install your DPU in a system.  Please note that the following figures are for illustration purposes only. Open the system case. Locate an available PCI Express slot. Hold the card by its edges and bracket, avoiding contact with the top surface.

  • Page 43: Uninstalling The Dpu/Supernic

    When the DPU is properly seated, the port connectors are aligned with the slot opening, and the DPU faceplate is visible against the system chassis. Secure the DPU with the screw.  Close the system case. Install the networking cables. For instructions, please refer to Networking Cable Installation.

  • Page 44: Pcie Extension Option (2X Pcie X16) Installation Instructions

    The DPU/SuperNIC is installed in a system that operates with voltages that can be lethal. Before uninstalling the board, please observe the following precautions to avoid injury and prevent damage to system components. Remove any metallic objects from your hands and wrists. It is strongly recommended to use an ESD strap or other antistatic devices.

  • Page 45: Installing The Dpu

    5.11.1 Installing the DPU The installation instructions include steps that involve a retention clip to be used while connecting the Cabline harnesses to the DPUs. Please note that this is an optional accessory. Please make sure to install the DPU cards in a PCIe slot that is capable of supplying the required power and airflow as stated in Specifications.
  • Page 46 Step 4: Slide the retention clip latches through the cutouts on the PCB. The latches should face the annotation on the PCB.  Step 4: Clamp the retention clip. Verify both latches are firmly locked.
  • Page 47 Step 5: Slide the Cabline CA-II Plus harnesses through the retention clip. Make sure that the clip opening is facing the plugs. ...
  • Page 48 Step 6: Plug the Cabline CA-II Plus harnesses on the PCIe Auxiliary Card. As indicated on both sides of the Auxiliary connection card; plug the black harness to the component side and the white harness to the print side. Step 7: Verify the plugs are locked. Step 8: Slide the retention clip through the cutouts on the PCB.

  • Page 49: Uninstalling The Cards

    Connect the DPU and PCIe Auxiliary Connection cards in available PCI Express x16 slots in the chassis. Step 1: Locate two available PCI Express x16 slots. Step 2: Applying even pressure at both corners of the cards, insert the DPU in the PCI Express slots until firmly seated.

  • Page 50: Pcie Bifurcation Configuration Options

    This section describes the various connectivity options for peripherals on the PCIe, including scenarios where the BlueField-3 DPU acts as the PCIe switch with NVMe SSDs as PCIe endpoints. While this list of scenarios is not exhaustive, it highlights the tested and verified options.
  • Page 51 The various configuration scenarios listed in this section include a diagram and instructions on how to bifurcate the PCIe interface using the mlxconfig tool. For more information on the mlxconfig tool, please refer to mlxconfig – Changing Device Configuration Tool. Before setting the desired configuration, take note of the following warnings: •...

  • Page 52: Host As Root Port On X4 Pcie Lane Peripherals

    5.12.1 Host as Root Port on x4 PCIe Lane Peripherals In this scenario, the x16 PCIe Goldfingers of the BlueField-3 DPU serve as endpoints to the host server (default), while the additional x16 PCIe lanes are accessible via the Cabline CA-II Plus connector, bifurcated into four PCIe links, where each link comprises x4 PCIe lanes.

  • Page 53: Dpu Arms As Root Port On Peripherals

    5.12.2 DPU ARMs as Root Port on Peripherals In this scenario, the x16 PCIe Gold Fingers of the BlueField-3 DPU serve as an endpoint to the host server (default), while the additional x16 PCIe lanes are accessible via the Cabline CA-II Plus connector, bifurcated into four PCIe links, where each link comprises x4 PCIe lanes.
  • Page 54 mlxconfig -d <device> s PCI_BUS10_HIERARCHY_TYPE=2 mlxconfig -d <device> s PCI_BUS10_WIDTH=3 mlxconfig -d <device> s PCI_BUS10_SPEED=4 mlxconfig -d <device> s PCI_BUS12_HIERARCHY_TYPE=2 mlxconfig -d <device> s PCI_BUS12_WIDTH=3 mlxconfig -d <device> s PCI_BUS12_SPEED=4 mlxconfig -d <device> s PCI_BUS14_HIERARCHY_TYPE=2 mlxconfig -d <device> s PCI_BUS14_WIDTH=3 mlxconfig -d <device>...

  • Page 55: Setting High-Speed-Port Link Type

    6 Setting High-Speed-Port Link Type The following table lists the BlueField-3 supported speeds and the default networking port link type OPN. Model Data Transmission Rate Default Protocol and Rate B3140L 900-9D3B4-00EN-EA0 / InfiniBand: NDR 400Gb/s InfiniBand NDR 400Gb/s B3240 900-9D3B4-00PN-EA0...

  • Page 56: Troubleshooting

    Check that both the DPU and its link are set to the established same speed and duplex settings Refer to the latest version of BlueField DPU SW Manual Forgot password needed to install/upgrade and follow instructions under "Upgrading NVIDIA the DPU image BlueField DPU Software" section.

  • Page 57: Specifications

    OPNs • 900-9D3D4-00NN-HA0: Crypto Disabled with integrated BMC BlueField-3 E-Series -8 Arm-Cores - 505MHz/2000MHz BlueField-3 SoC Single-Slot HHHL Card Dimensions: 68.90mm x 167.65mm Physical Tall Bracket Dimensions: 120.5 x 21.95mm Short Bracket Dimensions: 80.22mm x 22.83mm  Heatsink Dimensions (Length, Width, Height): 139.6mm x 92.7mm x 10.2mm Supported Interfaces Interfaces ...

  • Page 58: B3140L Supernics Specifications

    RoHS compliant Regulatory Model Number D3D4  Notes:   The BlueField-3 SuperNICs supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.  The non-operational storage temperature specifications apply to the product without its package.

  • Page 59: B3220L Supernics Specifications

    RoHS compliant Regulatory Model Number D3B4  Notes:   The BlueField-3 SuperNICs supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.  The non-operational storage temperature specifications apply to the product without its package.
  • Page 60 RoHS compliant Regulatory Model Number D3B4  Notes:   The BlueField-3 SuperNICs supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.  The non-operational storage temperature specifications apply to the product without its...

  • Page 61: B3210L Supernics Specifications

    RoHS compliant Regulatory Model Number D3B4  Notes:   The BlueField-3 SuperNICs supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.  The non-operational storage temperature specifications apply to the product without its...

  • Page 62: B3240 Dpus Specifications

    Electrical and Thermal Specifications Electrical and thermal specifications are provided in "NVIDIA BlueField-3 DPUs and SuperNICs Electrical and Thermal Specifications" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative. Temperature Operational 0°C to 55°C...

  • Page 63: B3210 Dpus Specifications

    RoHS compliant Regulatory Model Number D3B6  Notes:   The BlueField-3 DPU supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.  The non-operational storage temperature specifications apply to the product without its package.

  • Page 64: B3210E Dpus Specifications

    RoHS compliant Regulatory Model Number D3B6 Notes:   The BlueField-3 DPU supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.  The non-operational storage temperature specifications apply to the product without its package.

  • Page 65: B3220 Dpus Specifications

    RoHS compliant Regulatory Model Number D3B6  Notes:   The BlueField-3 DPU supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.  The non-operational storage temperature specifications apply to the product without its package.

  • Page 66: Dpus Mechanical Drawing And Dimensions

    RoHS compliant Regulatory Model Number D3B6  Notes:   The BlueField-3 DPU supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.  The non-operational storage temperature specifications apply to the product without its package.
  • Page 67 HHHL Single-Slot SuperNICs Model: B3140H  900-9D3D4-00EN-HA0 900-9D3D4-00NN-HA0      FHHL Single-Slot Single-Port SuperNICs Model: B3140L 900-9D3B4-00EN-EA0 900-9D3B4-00PN-EA0      FHHL Single-Slot Dual-Port SuperNICs Model: B3220L 900-9D3B4-00CV-EA0 900-9D3B4-00SV-EA0 Model: B3210L 900-9D3B4-00CC-EA0 900-9D3B4-00SC-EA0 FHHL Single-Slot Dual-Port DPUs with PCIe Extension Option Model: B3220  900-9D3B6-00CV-AA0 900-9D3B6-00SV-AA0 Model: B3210...

  • Page 68: Bracket Mechanical Drawings

    8.10 Bracket Mechanical Drawings All dimensions are in millimeters. DPU Configuration Model/OPNs Tall Bracket Short Bracket HHHL Single-Slot  Model: B3140H  Dual-Port SuperNICs 900-9D3D4-00EN-HA0 900-9D3D4-00NN-HA0 FHHL Single-Slot  Model: B3140L Single-port SuperNICs 900-9D3B4-00EN-EA0 900-9D3B4-00PN-EA0  FHHL Single-Slot  Model: B3210L Dual-Port SuperNICs 900-9D3B4-00CC-EA0 900-9D3B4-00SC-EA0 Model: B3220L 900-9D3B4-00CV-EA0 900-9D3B4-00SV-EA0...

  • Page 69: Monitoring

    9 Monitoring Thermal Sensors The thermal threshold definitions are described in the "NVIDIA BlueField-3 DPU and SuperNIC Electrical and Thermal Specifications" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative. 9.2 Heatsink The heatsink is attached to the DPU by three screws to dissipate the heat from the SoC. ...

  • Page 70: Finding The Guid/Mac On The Dpu

    10 Finding the GUID/MAC on the DPU Each BlueField-3 DPU is uniquely identified to facilitate efficient management and configuration. This section aims to guide you through the process of identifying your BlueField-3 DPU by using the information provided on the DPU label.
  • Page 71 Item on Label Description Example/Additional Information GUID (Global Unique GUIDs is a unique 16 byte identifier The GUID on the label is assigned to the Identifier) of the BlueField in an InfiniBand System GUID of the BlueField HCA. network, the GUID never changes. The System GUID represents the HCA hardware with all its ports.

  • Page 72: Supernics Board Label Example

    Item on Label Description Example/Additional Information DPU BMC (Baseboard Represents the MAC address for the Management Controller) 1GbE OOB management port of the DPU BMC. Note: The base MAC address for DPU BMC is +1 form the OOB MAC address. S/N (Serial Number) A unique serial number that distinguishes your DPU from others.
  • Page 73 Item on Label Description Example/Additional Information HOST (Base MAC) Represents the base MAC address for DPUs with more than one high-speed high-speed data port 0 port will use incremental (+1) MAC addresses against the base HOST MAC on the label. For instance: From the label - HOST: 94 6D AE F5 A1 Port 0: 94 6D AE F5 A1 CC (same as...

  • Page 74: Pcie Auxiliary Card Kit

    11 PCIe Auxiliary Card Kit This section applies to the following DPUs when used as Socket Direct cards in dual-socket servers.   • 900-9D3B6-00CN-AB0     • 900-9D3B6-00SN-AB0     • 900-9D3B6-00CV-AA0     • 900-9D3B6-00SV-AA0     Socket Direct network cards, which cost-effectively integrate a single network adapter silicon on a primary board, and an auxiliary PCIe connection card and Cabline SA-II Plus Harness connecting the...

  • Page 75: Channel Insertion Loss

    The PCI-SIG Gen5 SPEC also defines the total loss for AIC (bump to GF) to be 9.5dB @16Ghz. Please note that the BlueField-3 AIC, together with a 15cm Cabline CA-II Plus harnesses and the Passive PCIe Auxiliary Card loss is 10.3dB, which is approved by NVIDIA.

  • Page 76: Cabline Ca-Ii Plus Harness Pinouts

    11.3 Cabline CA-II Plus Harness Pinouts 11.3.1 Cabline CA-II Plus Harness - Component Side Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end GND BAR PCIE_REFCLK1_P Micro coax   Primary PCIe clock from the motherboard to the BlueField DPU Main card, to be used for the x16 Cabline...
  • Page 77 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end PCIE_CPU_CX_14N  Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point PCIE_CPU_CX_14P  Micro coax   Connect from the Black Cabline CA-II Plus cable through...
  • Page 78 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end PCIE_CPU_CX_11P  Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point GND BAR PCIE_CPU_CX_10N  Micro coax  ...
  • Page 79 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end GND BAR PCIE_CPU_CX_7N  Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point PCIE_CPU_CX_7P  Micro coax  ...
  • Page 80 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end PCIE_CPU_CX_4N  Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point PCIE_CPU_CX_4P  Micro coax   Connect from the Black Cabline CA-II Plus cable through...
  • Page 81 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end PCIE_CPU_CX_1P Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point GND BAR PCIE_CPU_CX_0N  Micro coax  ...
  • Page 82 Cabline CA-II Plus interface parameters to the main-card BlueField DPU silicon, like Cabline CA-II Plus cables length (contact NVIDIA for the format of this EEPROM). If additional optional I2C slave devices need to be managed by the main-card BlueField DPU...

  • Page 83: Cabline Ca-Ii Plus Harness - Print Side

    Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end S_PRSNT1_L Micro coax   Connect this pin to No wires are connected to these pins to ensure they do not interfere with the operation of S_PRSNT2_L for the detection when the two Cabline harnesses are...
  • Page 84 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end S_PERST2_CONN_L     Micro coax   Optional: PCIe compliant PERST_L (active low PCI Reset) signal for the Cabline CA-II Plus PCIe interface. To be used as the PERST_L signal for the control of PCIe lane 15:8, when a bifurcation...
  • Page 85 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end PCIE_CX_CPU_0P Micro coax   Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the BlueField DPU. ...
  • Page 86 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end PCIE_CX_CPU_2N Micro coax   Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the BlueField DPU. ...
  • Page 87 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end PCIE_CX_CPU_5P Micro coax   Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the BlueField DPU. ...
  • Page 88 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end PCIE_CX_CPU_7N Micro coax   Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the BlueField DPU. ...
  • Page 89 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end PCIE_CX_CPU_10P Micro coax   Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the BlueField DPU. ...
  • Page 90 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end PCIE_CX_CPU_12N Micro coax   Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the BlueField DPU. ...
  • Page 91 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end PCIE_CX_CPU_15P Micro coax   Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the BlueField DPU. ...

  • Page 92: Technical Specifications

    Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end S_PRSNT2_L Micro coax   Connect to a 4.7K pull-up resistor to 3.3V on the motherboard side, to detect if both the Cabline harnesses are connected or not. This signal is connected to S_PRSNT1_L on the BlueField DPU.

  • Page 93: Pcie Auxiliary Card Mechanical Drawings And Dimensions

    PCIe Auxiliary Card Size: 5.09 in. x 2.32 in. (129.30mm x 59.00mm) Physical Two Cabline CA-II Plus harnesses (white and black) Length: 15, 25 or 35cm Voltage: 12V, 3.3V_AUX  Power Maximum current: 100mA for the 3.3V_AUX voltage rail Consumption EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant 11.4.1 PCIe Auxiliary Card Mechanical Drawings and Dimensions...

  • Page 94: Cabline Ca-Ii Plus Harnesses Mechanical Drawing

    11.4.3 Cabline CA-II Plus Harnesses Mechanical Drawing...

  • Page 95: Supported Servers And Power Cords

    12 Supported Servers and Power Cords 12.1 Supported Servers Server support depends on the particular setup being used. The following is a partial list of servers with which the DPUs have been tested. For more information, please contact your NVIDIA representative. Dell...

  • Page 96: Document Revision History

    Description Jan 2023 Fixed a typo in Specifications Dec 2023 • Updated the document to include BlueField-3 SuperNICs upon reaching the "Mass Production" milestone. OPNs: B3210L, B3220L and B3210E • Updated the eMMC interface in Features and Benefits • Added PCIe Bifurcation Configuration Options •...
  • Page 97 PCI Express Interface pinouts to reflect changes in pins BB81 and B882.   • Added heatsink dimensions in Introduction and Specifications. Jun 2022 Renamed the document from "NVIDIA BleuField-3 InfiniBand/VPI DPU User Guide" to "NVIDIA BlueField-3 DPU User Guide" May 2022 First release...
  • Page 98 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
  • Page 99 Mellanox Technologies Ltd. in the U.S. and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated. Copyright © 2024 NVIDIA Corporation & affiliates. All Rights Reserved.

Table of Contents