1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. PCI Card
  5. BlueField BF1600
  6. User manual

Nvidia BlueField BF1600 User Manual

Infiniband/ethernet controller card
Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
NVIDIA BlueField BF1600 InfiniBand/Ethernet Controller Card User
Manual
Exported on Oct/31/2023 01:39 PM

Advertisement

Table of Contents
loading

Related Manuals for Nvidia BlueField BF1600

Summary of Contents for Nvidia BlueField BF1600

  • Page 1 NVIDIA BlueField BF1600 InfiniBand/Ethernet Controller Card User Manual Exported on Oct/31/2023 01:39 PM...

  • Page 2: Table Of Contents

    BF1600 Controller Card with Auxiliary Connection Card....................14 BF1600 Controller Card without Auxiliary Connection Card..................15 Supported Interfaces ..........................16 Layout and Connector Information...........................16 BlueField BF1600 Controller Card ........................16 PCIe Auxiliary Connection Card ........................... 20 PCIe Auxiliary Connection Card - Print Side ......................22 Interfaces ................................23 BlueField System On Chip (SoC) ..........................
  • Page 3 PCIe Gen 3.0 through Two CABLINE-CA II PLUS Connectors..................24 NCSI Management Interface ..........................25 USB Connector............................... 26 UART0 Interface............................. 26 I²C0 Interface ............................... 27 JTAG CoreSight 10 ............................27 External PCIe Power Supply Interfaces ........................28 External PCIe Power Supply Interface on the BF1600 Controller Card ................. 28 External PCIe Power Supply Interface on the Auxiliary PCIe Connection Card ..............
  • Page 4 PCIe x16 Edge Connector Pins Description - Print Side.................... 35 PCIe x16 Edge Connector Pinouts Description - Component Side................41 PCIe Gen 3.0 through Two CABLINE-CA II PLUS Connectors..................47 CABLINE-CA II PLUS Connector - Component Side ....................48 CABLINE-CA II PLUS Connector - Print Side......................52 Power Sequencing............................
  • Page 5 Thermal Sensors............................72 Hardware Installation ..........................73 Installation Procedure Overview..........................73 System Requirements ............................74 Hardware Requirements ........................... 74 Airflow Requirements ..........................74 Software Requirements ............................ 75 Pre-Installation Checklist ............................75 Bracket Replacement Instructions ...........................76 Removing the Existing Bracket..........................76 Installing the New Bracket ..........................76 Installation Instructions............................76 BF1600 Controller Card Installation Instructions - PCIe x16..................
  • Page 6 Networking Cables ............................84 Bring-Up and Driver Installation ........................86 Troubleshooting ............................87 General Troubleshooting ............................87 Specifications............................88 Dual-slot Controller Cards Specifications ........................88 Single-slot Controller Card Specifications........................89 LED Interfaces ..............................91 Networking LED Interfaces ..........................91 CPLD Revision LEDs Interface ..........................91 CABLINE-CA II PLUS Connector LED Interfaces ......................
  • Page 7 About This Manual This User Manual describes NVIDIA®  BlueField® BF1600 Controller Card. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the card, hardware installation, driver installation and bring-up instructions. Overview of Document Content...
  • Page 8 • E-mail: enterprisesupport@nvidia.com Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller. Related Documentation IEEE Ethernet specification at http://standards.ieee.org...

  • Page 9: Introduction

    Product Overview The BlueField BF1600 Dual Port 100Gb/s Controller Card is the perfect solution for managing NVMe storage drives in storage and hyper-converged systems. BlueField BF1600 Dual Port 100Gb/s Controller Card delivers the highest NVMe-oF target performance, reducing TCO and increasing ROI. The BlueField BF1600 Controller Cards can transform existing JBOF systems into NVMe-oF compliant solutions, simply by plugging the card into an existing PCIe slot.

  • Page 10: Bluefield Multicore System-On-Chip (Soc)

     The BlueField BF1600 Controller Card should be installed only in a JBOF System as it functions as a PCIe root-complex (RC) initiating PCIe bus operations. Installing it in a regular host system may damage the card. BlueField Multicore System-on-chip (SoC) BlueField SoC is a highly integrated and efficient controller, optimized for NVMe storage systems, Network Functions Virtualization (NFV), Cloud and AI workloads.

  • Page 11: Features And Benefits

    PCIe Gen 3.0 through x16 PCIe Gen 3.0 SERDES @ 8.0GT/s Auxiliary Card 2x 8GB 2x 16GB 2x 8GB 2x 16GB On-board DDR4 Memory 1GbE OOB Management √ √ √ √ RoHS RoHS Compliant BlueField SoC BlueField E-Series 16 cores a. Note: Refer to BlueField Software and Firmware release notes for the availability of PCIe Gen 4.0 capabilities.
  • Page 12 NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. BlueField BF1600 Controller Card effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.

  • Page 13: Operating Systems/Distributions

    The latest advancement in GPU-GPU communications is GPUDirect RDMA. This new technology provides a direct P2P (Peer-to-Peer) data path between the GPU Memory directly to/from the NVIDIA HCA devices. This provides a significant decrease in GPU-GPU communication latency and completely offloads the CPU, removing it from all GPU-GPU communications across the network.

  • Page 14: Package Contents

    Before installing your new system, unpack it and check against the below tables that all the parts have been sent according to the OPN you purchased. Check the parts for visible damage that may have occurred during shipping.  If anything is damaged or missing, contact your sales representative at support@NVIDIA.com. BF1600 Controller Card with Auxiliary Connection Card ...

  • Page 15: Bf1600 Controller Card Without Auxiliary Connection Card

    Category Qty. Item Accessories Tall bracket for the PCIe Auxiliary card (shipped assembled) Short bracket for the PCIe Auxiliary card BF1600 Controller Card without Auxiliary Connection Card  Applies to MBF1M606A-CSNAT, MBF1M646A-CSNAT and MBF1M616A-CSNAT. Category Qty. Item Cards BF1600 Controller Card Cables USB 2.0 Cable Type A to Type B USB 2.0 Cable to UART...

  • Page 16: Supported Interfaces

    Supported Interfaces Layout and Connector Information BlueField BF1600 Controller Card The BF1600 Controller card is a 9.13 in. x 4.37 in. (232.0 mm x 111.15mm) card. The component height on the top and bottom of the card complies with the PCIe specification. The below figures show the connector and LED designations. ...
  • Page 18 BF1600 Controller Card - Print Side Callout Interface Description BlueField System On Chip (SoC) BlueField SoC with heatsink PCIe x16 Gen 3.0/4.0 Edge Connector The interface between the BlueField Controller and the JBOF system. PCIe Gen 3.0 through Two CABLINE-CA II PLUS The interface between the PCIe Auxiliary Card and the BlueField Controller Card.
  • Page 19  4 Ethernet QSFP56 Interfaces Ethernet traffic is transmitted through the controller’s QSFP28 connectors. The QSFP28 connectors allow for the use of Optical modules and cable interconnect solutions. USB Connector USB Type-B Bracket mounted on card I²C0 Interface For ConnectX-5 Debug UART0 Interface BlueField UART0 (Console): connected to a transceiver and header JTAG CoreSight 10...

  • Page 20: Pcie Auxiliary Connection Card

    PCIe Auxiliary Connection Card  The PCIe Auxiliary Connection card is supplied with the following OPNs: MBF1M626A-CSNAT, MBF1M656A-CSNAT and MBF1M636A-CSNAT. The PCIe Auxiliary Card is a 5.09in. x 2.32in. (129.30mm x 59.00mm) card. The below figures provide the connector designations.  The below figures are for illustration purposes only and might not reflect the current revision of the BF1600 card.
  • Page 21  ...

  • Page 22: Pcie Auxiliary Connection Card - Print Side

    PCIe Auxiliary Connection Card - Print Side   Callout Interface Description PCIe x16 Gen 3.0/4.0 Edge Connector The interface between the PCIe Auxiliary Card and the JBOF system. PCIe Gen 3.0 through Two CABLINE-CA II PLUS Connectors The interface between the PCIe Auxiliary Card and the BlueField Controller Card.

  • Page 23: Interfaces

    32 lanes of external PCIe lanes supporting Gen3 and Gen 4, along with an embedded PCIe switch. Ethernet QSFP56 Interfaces The network ports of the BlueField BF1600 Controller Card are compliant with the IEEE 802.3 Ethernet standards listed in  Ethernet traffic is transmitted through the cards' QSFP28 connectors.

  • Page 24: Pci Express Interfaces

    The PCI Express bus is a high-speed interface used to connect the Controller Card to other peripheral devices (the BMC, PCIe switch or directly to the NVMe SSDs backplane), and operates as a Root-Complex (RC) initiating PCIe bus operations. The BlueField BF1600 Controller Card supports PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible) through the following connectors:...

  • Page 25: Ncsi Management Interface

    The BlueField BF1600 Controller Card supports PCI Express Gen 3.0 (1.1 and 2.0 compatible) through two on-card x8 CABLINE-CA II PLUS connectors that together provide an additional x16 PCIe interface. The belly to belly CABLINE-CA II PLUS connectors are located on the component and print side of the card and they are connected to the BlueField SoC.

  • Page 26: Usb Connector

    USB Connector The BlueField BF1600 Controller Card uses a USB 2.0 Type B device. The USB connector is mounted on the Controller Card bracket. The purpose of the USB connector is debugging and loading new versions of the operating systems and firmware.

  • Page 27: I²C0 Interface

    The BlueField BF1600 Controller Card provides one I²C0 interface. The I²C0 Interface is populated for ConnectX-5 debug and bring-up purposes.  JTAG CoreSight 10 The BlueField BF1600 Controller Card uses JTAG CoreSight 10-Pin Header. The purpose of the JTAG CoreSight 10 is for debugging the Arm with Arm DSTREAM debug tool. For JTAG CoreSight 10 pins, please refer JTAG CoreSight 10...

  • Page 28: External Pcie Power Supply Interfaces

    The BF1600 Controller Card requires the +3.3V AUX to come up prior to the +12V at the PCIe connector. The BlueField BF1600 Controller Card consumes power from two power sources; an external +12V through an 8-pin ATX connector (reference #9) and an additional +12V through the PCIe x16 edge connector.

  • Page 29: External Pcie Power Supply Interface On The Auxiliary Pcie Connection Card

     The use of an external power supply interface is restricted to the ATX 8-pin connector only, as shown in the above picture. External PCIe Power Supply Interface on the Auxiliary PCIe Connection Card  The PCIe Auxiliary Connection card is supplied with the following OPNs: MBF1M626A-CSNAT, MBF1M656A-CSNAT and MBF1M636A-CSNAT The Auxiliary PCIe card contains the same external +12V power connector as on the BF1600 Controller Card (reference #4).

  • Page 30: Cpld Interface

      CPLD Interface The BlueField BF1600 Controller Card incorporates a CPLD device. It drives all BlueField chip configuration pins, all system and device reset, miscellaneous board configuration setups, power monitoring, and power sequence control. eMMC Interface The BlueField BF1600 Controller Card incorporates one eMMC (embedded Multi-media Card) interface (eMMC0) which is utilized as a boot source for the...

  • Page 31: Ddr4 Sodimm Memory

    Applies to MBF1M616A-CSNAT, MBF1M636A-CSNAT, MBF1M646A-CSNAT, and MBF1M656A-CSNAT. The BlueField BF1600 Controller Card incorporates two DDR4 small outline dual-inline memory modules (SODIMM) memory interfaces. Each interface is 16GB, 64-bit data and 8-bit Error Correcting Code (ECC) memory. The connections between the DDR4 SODIMMs and the BlueField SoC is listed in...

  • Page 32: Spi Flash Memory

    Interfaces. CPLD Revision LEDs Interface There are four green I/O LEDs to indicate CPLD revision located on the BlueField BF1600 Controller Card. See the below figure for the CPLD LEDs location on the card. For LEDs specifications, please refer to CPLD Revision LEDs...

  • Page 33: Cabline-Ca Ii Plus Connector Led Interfaces

      CABLINE-CA II PLUS Connector LED Interfaces There is one green I/O LED near each CABLINE-CA II PLUS connector (four LEDs in total) on both on the Controller card and PCIe Auxiliary Connection card indicating the successful connection of the CABLINE-CA II PLUS cables. See the below figure for the green I/O LED location on the cards. For LED specifications, please refer to CABLINE-CA II PLUS Connector LED Interfaces.

  • Page 34: Power-Good Signal Led Interface

    Power-Good Signal LED Interface There is one green I/O LED to indicate power-good signal. See the below figure for LED locations on the card. For the power-good LED specifications, please refer to Power-Good Signal LED Interface.    PCI Link Status LEDs ...

  • Page 35: Pin Description

    Pin Description PCI Express Interface PCIe x16 Gen 3.0/4.0 Edge Connector The BF1600 Controller Card uses a PCI Express x16 connector (component and print sides) according to the PCI Express 3.0/4.0 specifications. The following two tables provide component side pins description and print side pins description. For further details, please refer to PCIe x16 Gen 3.0/4.0 Edge Interface.
  • Page 36 JTAG_HOST_TDO Output No connected JTAG_HOST_TMS Input No connected +3.3v Input Power +3.3v Input Power PERST# Output PCIe Interface Reset# For the PCIe interface via the card PCIe Edge connector For System NIC mode, it's direction will turn as input. Mechanical Key Ground Ground REFCLK+...
  • Page 37 Ground HSIp(1)     BlueField Card PCIe TX to Carrier Board PCIe RX Output HSIn(1) Ground Ground HSIp(2)     BlueField Card PCIe TX to Carrier Board PCIe RX Output HSIn(2) Ground Ground HSIp(3)     BlueField Card PCIE TX to Carrier Board PCIe RX Output HSIn(3) Ground...
  • Page 38 HSIp(4)  Output  BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(4) Ground Ground HSIp(5)  Output  BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(5) Ground Ground HSIp(6)  Output  BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(6) Ground Ground HSIp(7)
  • Page 39 HSIp(8)  Output  BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(8) Ground Ground HSIp(9)  Output  BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(9) Ground Ground HSIp(10) Output BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(10) Ground Ground HSIp(11)
  • Page 40 BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(13) Ground Ground HSIp(14)  Output BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(14) Ground Ground HSIp(15)  Output  BlueField Card PCIe TX to Carrier Board PCIe RX HSIn(15) Ground a. From BlueField BF1600 Controller Card Perspective.

  • Page 41: Pcie X16 Edge Connector Pinouts Description - Component Side

    PCIe x16 Edge Connector Pinouts Description - Component Side...
  • Page 42 Pin# Name PCIe Description Description +12v Input Power +12v Input Power +12v Input Power Ground SMCLK Input SMBUS_SCL - connected to BlueField SoC SMB2 Inter- face SCL SMDAT bidirectional SMBUS_SDA - connected to BlueField SoC SMB2 Inter- face SDA Ground +3.3v Input Power...
  • Page 43 HSOp(0)     BlueField Card PCIe RX to Carrier Board PCIe TX Input HSOn(0) Ground PRSNT#2 Not Connected. Assembly option for on-board CPLD connectivity Ground HSOp(1)     BlueField Card PCIe RX to Carrier Board PCIe TX Input HSOn(1) Ground Ground HSOp(2)  ...
  • Page 44 RSVD For testing purposes only. Keep it floating. PRSNT#2 For testing purposes only. Keep it floating. Ground HSOp(4)     BlueField Card PCIe RX to Carrier Board PCIe TX Input HSOn(4) Ground Ground HSOp(5)     Input BlueField Card PCIe RX to Carrier Board PCIe TX HSOn(5) Ground Ground...
  • Page 45 HSOp(7)     BlueField Card PCIe RX to Carrier Board PCIe TX Input HSOn(7) Ground PRSNT#2 Not Connected. Assembly option for on-board CPLD connectivity Ground HSOp(8)   BlueField Card PCIe RX to Carrier Board PCIe TX Input HSOn(8) Ground Ground HSOp(9)  ...
  • Page 46 HSOp(10)     BlueField Card PCIE RX to Carrier Board PCIE TX Input HSOn(10) Ground Ground HSOp(11)     BlueField Card PCIe RX to Carrier Board PCIe TX Input HSOn(11) Ground Ground HSOp(12)     BlueField Card PCIe RX to Carrier Board PCIe TX Input HSOn(12) Ground...

  • Page 47: Pcie Gen 3.0 Through Two Cabline-Ca Ii Plus Connectors

    PCIe Gen 3.0 through Two CABLINE-CA II PLUS Connectors The BlueField BF1600 Controller Card uses two x8 CABLINE-CA II PLUS connectors (Component Side and Print Side Connectors) according to the PCI Express 3.0 specifications. The below two tables provide the component side pins description and the print side pins description. For further details, please refer to PCIe Gen 3.0 through Two CABLINE-CA II PLUS...

  • Page 48: Cabline-Ca Ii Plus Connector - Component Side

    CABLINE-CA II PLUS Connector - Component Side The below table provides pins description of the CABLINE-CA II PLUS connector mounted on the controller’s component side. Pin # Pin Name Pin Direction Description Ground PCIE_REF- CLK1_P PCIe Reference Differential Clock. Not Connected (default) PCIe separated clocking default scheme PCIE_REF- CLK1_N Assembly option to support PCIe shared clocking scheme, then:...
  • Page 49 Ground HSO_12N     BlueField Card PCIe RX to Carrier Board PCIe TX Input HSO_12P Ground HSO_11N     Input BlueField Card PCIe RX to Carrier Board PCIe TX HSO_11P Ground HSO_10N Input BlueField Card PCIe RX to Carrier Board PCIe TX HSO_10P Ground HSO_9N...
  • Page 50 HSO_7N Input BlueField Card PCIe RX to Carrier Board PCIe TX HSO_7P Ground HSO_6N Input BlueField Card PCIe RX to Carrier Board PCIe TX HSO_6P Ground HSO_5N Input BlueField Card PCIe RX to Carrier Board PCIe TX HSO_5P Ground HSO_4N Input BlueField Card PCIe RX to Carrier Board PCIe TX HSO_4P...
  • Page 51 Ground HSO_1N Input BlueField Card PCIe RX to Carrier Board PCIe TX HSO_1P Ground HSO_0N Input BlueField Card PCIe RX to Carrier Board PCIe TX HSO_0P Ground I2C_SMDAT Not Connected. Optional assembly connectivity to BlueField SoC I2C GW4 interface I2C_SMCLK Not Connected.

  • Page 52: Cabline-Ca Ii Plus Connector - Print Side

    RSVD Not Connected. Keep it floating. CABLINE-CA II PLUS Connector - Print Side The below table provides pins description of the CABLINE-CA II PLUS connector mounted on the con- troller’s print side. Pin # Pin Name Pin Direction Description RSVD Not Connected.
  • Page 53 RSVD Not Connected. Assembly option for on-board CPLD connectivity RSVD Not Connected. Assembly option for on-board CPLD connectivity Ground HSI_0P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_0N Ground HSI_1P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_1N Ground HSI_2P...
  • Page 54 HSI_3N Ground HSI_4P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_4N Ground HSI_5P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_5N Ground HSI_6P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_6N Ground HSI_7P Output...
  • Page 55 HSI_8P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_8N Ground HSI_9P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_9N Ground HSI_10P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_10N Ground HSI_11P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_11N...
  • Page 56 HSI_12N Ground HSI_13P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_13N Ground HSI_14P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_14N Ground HSI_15P Output BlueField Card PCIE TX to Carrier Board PCIE RX HSI_15N Ground PERST# Output...

  • Page 57: Power Sequencing

    S_PRSNT2_L Input Presence Indication Should be grounded and shorted to IPEX Component Side pin #58. Power Sequencing The BF1600 Controller Card requires the +3.3V to come up prior to the +12V at the PCIe connector. NCSI Management Interface The below table lists the NCSI Management Interface pins description. For further details, please refer to NCSI Management Interface.
  • Page 58 RXD1 Receive data Connected to Board Ground CRS_DV Carrier Sense/Receive Data Valid Connected to Board Ground TXD0 Transmit data Connected to Board Ground TXD1 Transmit data Connected to Board Ground TX_EN Transmit enable Connected to Board Ground RX_ER Receive Error, keep float at system side Connected to Board Ground I2C_SDA Used for testing.

  • Page 59: External Power Supply Interfaces

    Connected to Board Ground 3.3V_AUX Used for testing. Keep float at system side. 3.3V_AUX Used for testing. Keep float at system side. 3.3V_AUX Used for testing. Keep float at system side. 3.3V_AUX Used for testing. Keep float at system side. External Power Supply Interfaces The below table lists the External Power Supply pins of the external power supply interfaces on the BF1600 Controller card and PCIe auxiliary card.

  • Page 60: External +12V Mechanical Pinout Diagram

    +12V ATX Supplied 12V +12V ATX Supplied 12V  External +12V Mechanical Pinout Diagram   UART0 Interface The below table lists the UART0 interface pins from the BlueField Controller Card perspective. For further details, please refer to UART0 Interface.  The electrical levels of the UART0 interface is CMOS (3.3V) UART0 Pinouts Description Pin # Signal Name...

  • Page 61: Jtag Coresight 10 Interface

    JTAG CoreSight 10 Interface The below table lists the JTAG CoreSight 10 pins description. For further details, please refer to JTAG CoreSight 10 Interface.  Pin Signal Voltage  Description Name Domain  1  VTREF  A The Voltage Target Reference pin supplies DSTREAM with the debug rail voltage of the target to match its I/O logic levels. VTREF can be tied HIGH on the target. If VTREF is pulled HIGH by a resistor, its value must be no greater than 100Ω.

  • Page 62: Ddr4 Dram Memory Interface

    DDR4 DRAM Memory Interface The below table lists the DDR4 DRAM pins description. For further details, please refer to DDR4 SODIMM Memory. Pin Name Description Pin Name Description A0–A16 SDRAM address bus I2C serial bus clock for SPD/TS BA0, BA1 SDRAM bank select I2C serial bus data line for SPD/TS BG0, BG1 SDRAM bank group select...
  • Page 63 DQS0_t– SDRAM data strobes RESET_n Set SDRAMs to a Known State DQS8_t (positive line of differential pair) DQS0_c– SDRAM data strobes EVENT_ SPD signals a thermal event has occurred. DQS8_c (negative line of differential pair) DM0_n–DM8_n, DBI0_n-DBI8_n SDRAM data masks/data bus inversion  ...

  • Page 64: Cables And Cabling Configurations

    Cables and Cabling Configurations UART Interface Cabling Debugging or bringing up the card required the use of a USB-to-UART cable that is included in the package contents. Upon inspection, there is a USB connector with USB 2.0 full speed compatibility and 3 single pole 0.1m pitch header that connects to a serial port on the Controller Card. The following figure shows an example of a USB-to-UART Cable, and the table provides the cable signal description.

  • Page 65: Usb Interface Cabling

    Color Signal Description Orange Transmit Asynchronous Data output. Connects to RXD input on Controller Card UART Cable Connected to BF1600 Card USB Interface Cabling Debugging and loading new versions of the operating systems and firmware requires the use of Type A to Type B USB 2.0 Cable. The following figure shows an example of Type A to Type V USB Cable.

  • Page 67: Configuration Scenarios

    Configuration Scenarios Mellanox BlueField BF1600 Controller Cards offer a variety of options to serve the requirements of high-density and high-performance storage environments. The following section describes various connectivity options for NVMe SSDs. This list of scenarios is not exhaustive (represents the maximum utilization of PCIe connectivity), but it shows some different available options.

  • Page 69: Configuration 2: Direct Connectivity To 8-16 Nvme Ssds With (No External Pcie Switch)

    Configuration 2: Direct Connectivity to 8-16 NVMe SSDs with (No External PCIe Switch) For direct connectivity to 8-16 SSDs configuration, the first BlueField PCIe Gen 4.0 x16 inter- face is routed through the x16 PCIe Gen 3.0/4.0 Edge Connector towards the first group of four x4 SSDs or eight x2 SSDs. The second BlueField PCIe Gen 4.0 x16 interface is routed through the x16 PCIe Gen 3.0 only CABLINE-CA II PLUS connector towards the second group of four x4 SSDs or eight x2 SSDs.

  • Page 70: Configuration 3: Connectivity To 8-16 Nvme Ssds (With External Pcie Switch)

    Configuration 3: Connectivity to 8-16 NVMe SSDs (with External PCIe Switch) For 18-6 SSDs configuration with an external PCIe switch, the first BlueField PCIe Gen 3./4.0 x16 interface is routed through the x16 PCIe Gen 3.0/4.0 Edge Connector toward the first x16 port of the PCIe switch. The second BlueField PCIe Gen 3.0 x16 interface is routed through the PCIe Gen 3.0 CABLINE-CA II PLUS connector and the PCIe Auxiliary Card towards the sec- ond x16 port of the PCIe Switch.

  • Page 71: Clocks Distribution

    Clocks Distribution The BlueField BF1600 Controller Card implements a “separate clocking scheme”. The Controller Card has a dedicated 100MHz oscillator clock source for feeding its PCIe reference clock. This clock is not driven out of the card and it is expected that the downstream PCIe entities will have their own clock...

  • Page 72: Thermal Sensors

    Thermal Sensors The BlueField BF1600 Controller Card incorporates the BlueField SoC which operates in the range of temperatures between 0C and 120C. There are three thermal threshold definitions for the BlueField SoC which impact the overall system operation state: •...

  • Page 73: Hardware Installation

    Hardware Installation Installation and initialization of the BlueField BF1600 Controller Card require attention to the mechanical, power, and precautions for rack-mounted equipment. Safety Warnings Please observe all safety warnings to avoid injury and prevent damage to system components. Note that not all warnings are relevant to all models.

  • Page 74: System Requirements

    Hardware Requirements  Unless otherwise specified, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operating environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level.

  • Page 75: Software Requirements

     All systems in the same rack should be planned with the same airflow direction. All components need to have the same airflow direction. Please refer to the Specifications section for airflow numbers for each specific card model. Software Requirements • See Operating Systems/Distributions section under the Introduction section.

  • Page 76: Bracket Replacement Instructions

    Bracket Replacement Instructions The BF1600 Controller Card and auxiliary connection card are shipped with assembled high-profile brackets. If this form factor is suitable for your requirements, you can skip the remainder of this section and move to Installation Instructions. If you need to replace the Auxiliary Connection Card with the short bracket that is included in the shipping box, please follow the instructions in this section.

  • Page 77: Bf1600 Controller Card Installation Instructions - Pcie X16

     The BlueField BF1600 Controller Card should be installed only in a JBOF System as it functions as a PCIe root-complex (RC) initiating PCIe bus operations. Installing it in a regular host system may damage the card.  Please note that the following figures are for illustration purposes only.
  • Page 78   Step 2.      Applying even pressure at both corners of the card, insert the BF1600 Controller Card in a PCI Express slot until firmly seated.

  • Page 79: Secure The Bf1600 Controller Card To The Jbof System

     Do not use excessive force when seating the card, as this may damage the system or the auxiliary PCIe connection card.   Secure the BF1600 Controller Card to the JBOF system. Step 1.      Secure the bracket to the JBOF system with the bracket screw. BF1600 Controller and Auxiliary Connection Cards Installation Instructions - PCIe x32 ...

  • Page 80: Connect The Bf1600 Controller Card With The Auxiliary Connection Card Using The Supplied Cabline-Caii Plus Harnesses

     Connect the BF1600 Controller Card with the Auxiliary connection card using the supplied CABLINE-CAII PLUS harnesses. Step 1. Plug the CABLINE-CAII PLUS harness on the BF1600 Controller Card while paying attention to the color-coding. As indicated on both sides of the card;...

  • Page 81: Connect The Bf1600 Controller Card In An Available Pci Express Slot On The Jbof System

    Step 3.      Plug the CABLINE-CAII PLUS harness on the PCIe Auxiliary Card. As indicated on both sides of the Auxiliary connection card; plug the black harness to the component side and the white harness to the print side.  Connect the BF1600 Controller Card in an available PCI Express slot on the JBOF system. Step 1.     ...

  • Page 82: Connect The Pcie Auxiliary Connection Card In An Available Pci Express Slot On The Jbof System

    Step 2.      Applying even pressure at both corners of the card, insert the BF1600 Controller Card in a PCI Express slot until firmly seated.  Do not use excessive force when seating the card, as this may damage the system or the auxiliary PCIe connection card.  Connect the PCIe Auxiliary Connection Card in an available PCI Express slot on the JBOF system.

  • Page 83: Secure The Bf1600 Controller And Pcie Auxiliary Connection Cards To The Jbof System

     Secure the BF1600 Controller and PCIe Auxiliary Connection Cards to the JBOF system. Step 1.      Secure the bracket to the JBOF system with the bracket screw.

  • Page 84: Cables And Modules

     Do not use excessive force when seating the card, as this may damage the system or the adapter. Cables and Modules Networking Cables All networking cables can be inserted or removed with the unit powered on. To insert a cable, press the connector into the port receptacle until the connector is firmly seated. Support the weight of the cable before connecting the cable to the adapter card.
  • Page 85 Verify that the connector locks in place.   When installing cables make sure that the latches engage.  Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card. After inserting a cable into a port, the Amber LED indicator will light when the physical connection is established (that is, when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port).

  • Page 86: Bring-Up And Driver Installation

    Bring-Up and Driver Installation  It is recommended to upgrade your BlueField product to the latest software and firmware versions in order to enjoy the latest features and bug fixes. Instructions for the BlueField Controller card bring-up and driver installation are located in the BlueField Software documentation at Controller Card Bring- Up and Driver Installation.

  • Page 87: Troubleshooting

    Verify that your switch and BlueField BF1600 Controller Card port are compatible Link light is on, but with no communication • Check that the latest driver is loaded established • Check that both the BlueField BF1600 Controller Card and its link are set to the same speed and duplex settings...

  • Page 88: Specifications

    Specifications Dual-slot Controller Cards Specifications        Size: 9.13 in. x 4.37 in. (232.0 mm x 111.15mm)  Physical Connector: Dual Ethernet QSFP28 (copper and optical) Ethernet: 100GBASE-CR4, 100GBASE-KR4, 100GBASE-SR4, 56GBASE-R4, 550G Ethernet   Consortium, 40GBASE-CR4, 40GBASE-SR4, 40GBASE-LR4, 40GBASE-ER4, 25GBASE-CR/  Protocol CR-S, 25GBASE-SR, 25GBASE-LR, 25G Ethernet Consortium, 10GBASE-SR, 10GBASE-LR, 10GBASE-ER, 10GBASE-CX4, SGMII, 1000BASE-X...

  • Page 89: Single-Slot Controller Card Specifications

    3.5W Active Cables 114W 800LFM 900LFM Maximum power available through QSFP28 port: 3.5W Temperature Operational 0°C to 55°C Environmental Non-operational -40°C to 70°C Humidity Operational 10% to 85% relative humidity  Non-operational 10% to 90% relative humidity  Safety: CB / cTUVus / CE Regulatory EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS Compliant...
  • Page 90 PCI Express Gen 3.0 / 4.0: SERDES @ 8.0GT/s / 16GT/s, 32 lanes (2.0 and 1.1 compatible) Voltage: 12V     Power Airflow (LFM)     @40°C @45°C   Typical Power Passive Cables 700LFM 800LFM Power   Passive Cables 700LFM 800LFM  ...

  • Page 91: Led Interfaces

     1Hz Blinking Amber occurs due to running a beacon command for locating the BF1600 Controller Card CPLD Revision LEDs Interface There are four green I/O LEDs to indicate the CPLD revision. The LEDs are located on the BlueField BF1600 Controller Card.  See the below table for Revision LEDs.

  • Page 92: Cabline-Ca Ii Plus Connector Led Interfaces

    LED Number CPLD Revision Rev 0 Solid Green Rev 1 Solid Green Rev 2 Solid Green Solid Green Rev 3 Solid Green Rev 4 Solid Green Solid Green Rev 5 Solid Green Solid Green Rev 6 Solid Green Solid Green Solid Green Rev 7 Solid Green...

  • Page 93: Bf1600 Controller Card Cabline-Ca Ii Plus Connector Led

    BF1600 Controller Card Cabline-CA II Plus Connector LED LED Number LED state Description The CABLINE-CA II PLUS cable is properly connected The CABLINE-CA II PLUS cable is not properly connected PCIe Auxiliary Connection Card Cabline-CA II Plus Connector LED  LED Number LED state Description The CABLINE-CA II PLUS cable is properly connected...

  • Page 94: Finding The Mac On The Controller Card

    Finding the MAC on the Controller Card Each BF1600 Controller Card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol.  The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards. MBF1M606A-CSNAT Board Label (Example)  ...
  • Page 95   MBF1M626A-CSNAT Board Label (Example)  ...
  • Page 96 MBF1M636A-CSNAT Board Label (Example)   MBF1M646A-CSNAT Board Label (Example)
  • Page 97 MBF1M656A-CSNAT Board Label (Example) PCIe Auxiliary Card Board Label...
  • Page 98  ...

  • Page 99: Document Revision History

    Document Revision History Date Comments/Changes May. 2023 Updated Specifications - added non-operational storage temperature specifications Apr. 2021 Added MBF1M606A-ESNAT Nov. 2019 Added a note to Introduction Nov. 2019 Updated the list of OPNs. Updated Specifications Aug. 2019 Updated Configuration Scenarios Jul. 2019 Mellanox BlueField was modified from trademark to registered trademark. Updated the following s due to re-spin of the cards: •...
  • Page 100 Date Comments/Changes • Bring-Up and Driver Installation May. 2018 Updated the following sections: • Introduction • Interfaces • Pin Description • Hardware Installation • Bring-Up and Driver Installation Jan. 2018 First release...
  • Page 101 NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
  • Page 102 INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in...

This manual is also suitable for:

Mbf1m606a-csnatMbf1m606a-esnatMbf1m616a-csnatMbf1m626a-csnatMbf1m636a-csnatMbf1m646a-csnat ... Show all

Table of Contents