Page 1 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 2: Table Of Contents
Speci cations Speci cations PCIe Auxiliary Card Kit PCIe Auxiliary Card Kit Channel Insertion Loss Channel Insertion Loss MCIO Harness Pinouts MCIO Harness Pinouts PCIe Auxiliary Kit Technical Speci cations PCIe Auxiliary Kit Technical Speci cations NVIDIA ConnectX-8 SuperNIC User Manual...
Page 3 Document Revision History Document Revision History NVIDIA ConnectX-8 SuperNIC User Manual...
Page 4 About This Manual This User Manual describes NVIDIA® ConnectX®-8 SuperNICs. It provides details as to the interfaces of the board, speci cations, required software and rmware for operating the board, and relevant documentation. Intended Audience This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with In niBand and Ethernet network and architecture speci cations.
Page 5 Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: URL: https://www.nvidia.com > Support E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased NVIDIA products through an NVIDIA-approved reseller should rst seek assistance through their reseller.
Page 6 Linux diagnostic, tools content and con guration. DOCA User Manual describing DOCA Documentation SDK platform development. User Manual describing WinOF- WinOF-2 for 2 features, performance, Windows Ethernet diagnostic, tools Documentation content and con guration. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 7 (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km. In addition to meeting Ethernet and IBTA standards, NVIDIA tests every product in an end-to-end environment ensuring a Bit Error Rate of less than 1E-15 . NVIDIA ConnectX-8 SuperNIC User Manual...
Page 8 MegaBytes. The use of Mb or Mbits (small b) indicates size in MegaBits. IB is used in this document to mean In niBand. In this document, PCIe is used to mean PCI Express. Revision History A list of the changes made to this document is provided in Document Revision History. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 9: Introduction
Powerful Networking for Generative AI Central to NVIDIA’s AI networking portfolio, ConnectX-8 SuperNICs fuel the next wave of innovation in forming accelerated, massive-scale AI compute fabrics. They seamlessly integrate with next-gen NVIDIA networking platforms, providing end- to-end 800Gb/s connectivity.
Page 10: System Requirements
PCIe interface. Utilizing the Socket-Direct or the Multi-Host capability, the PCIe extension card is connected to the SuperNIC and is used as an end-point extension. NVIDIA o ers ConnectX-8 Socket Direct, which enables 800Gb/s or 400Gb/s connectivity for servers with PCIe Gen5 or Gen4 capability, respectively. The SuperNIC's 32-lane PCIe bus is split into two 16-lane buses, with one bus accessible through a PCIe x16 edge connector and the other bus through an x16 Auxiliary PCIe Connection card.
Page 11 In Socket Direct/Multi-Host Con guration (2x PCIe x16) PCIe Gen5 SERDES @32GT/s through edge connector PCIe Gen5 SERDES @32GT/s through PCIe Auxiliary Connection Card or SFF-TA-1016 MCIO In PCIe x16 Extension Option - Switch DSP (Data Stream Port) NVIDIA ConnectX-8 SuperNIC User Manual...
Page 12: Package Contents
Powered connectors for optical and active cable support Package Contents Category Item Cards ConnectX-8 SuperNIC Short bracket Accessories Tall bracket (shipped assembled on the SuperNIC) Optional Accessories, not included in the package, can be purchased separately: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 13 Accessories assembled on the SuperNIC) NVIDIA SocketDirect/MultiH ost Auxiliary Kit for 930-9XAX6- Additional PCIe 0025-000 Gen6x16 Connection, 250mm MCIO Harness NVIDIA ConnectX-8 200mm Cable 930-9XCBL- Extender for Low- 000A-000 Speed Signals Over 30p Debug Connector NVIDIA ConnectX-8 SuperNIC User Manual...
Page 14: Features And Bene Ts
PCIe Gen6/5 ( SERDES @ 64GT/s / 32GT/s) through x16 edge connector In niBand Architecture ConnectX-8 delivers low latency, high bandwidth, and Speci cation v1.7 compliant computing e ciency for high-performance computing (HPC), arti cial intelligence (AI), and hyperscale cloud NVIDIA ConnectX-8 SuperNIC User Manual...
Page 15 IBTA Vol1 IBTA Vol2 212.5 HDR / IBTA Vol2 HDR1 212.5 106.25 Up to 400 Gigabit Ethernet ConnectX-8 SuperNICs comply with the following IEEE 802.3 standards: 400GbE / 200GbE / 100GbE / 25GbE / 10GbE NVIDIA ConnectX-8 SuperNIC User Manual...
Page 16 Prot Standa Rate (Gb/s) ocol IBTA Vol1 1.7 IBTA 212.5 Vol2 1.5 IBTA 212.5 106.25 Vol2 1.4 IEEE 100/20 802. 0/400G Gigabit Etherne NVIDIA ConnectX-8 SuperNIC User Manual...
Page 17 Gigabit IEEE Etherne 802. (Include IEEE 802. enhanc ement) IEEE 802. IEEE 802. Gigabit IEEE Etherne 802. IEEE 802. Ethe 25/50 rnet Gigabit Tech Etherne nolo Cons ortiu IEEE Gigabit 802. Etherne IEEE 802. Gigabit NVIDIA ConnectX-8 SuperNIC User Manual...
Page 18 IEEE 802. (For 2.5: support only 2.5 x1000B ASE-X) Based IEEE auto- 802. negotia tion and KR startup IEEE 802. Link Aggreg IEEE ation 802. IEEE 802. IEEE 802. tags priori IEEE 802. 1Qau (QCN NVIDIA ConnectX-8 SuperNIC User Manual...
Page 19 128Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus. (Note: Address 0x58 is reserved.) Overlay Networks In order to better scale their networks, data center operators often create overlay networks that carry NVIDIA ConnectX-8 SuperNIC User Manual...
Page 20 For further information, refer to the MFT User Manual. Host Management ConnectX-8 technology maintains support for host manageability through a BMC. ConnectX-8 PCIe stand- up adapter can be connected to a BMC using MCTP over NVIDIA ConnectX-8 SuperNIC User Manual...
Page 21 SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up SuperNIC. For con guring the adapter for the speci c manageability solution in use by the server, please contact NVIDIA Support. Protocols: PLDM, NCSI Transport layer – RBT, MCTP over SMBus, and MCTP over PCIe Physical layer: SMBus 2.0 / I2C interface for device...
Page 22 ConnectX-8 supports IPSec, MACSec, and PSP cryptography acceleration. Connectx-8 SuperNIC Cryptography Accelerations hardware-based accelerations o oad the crypto operations and free up the CPU, reducing latency and enabling scalable crypto solutions. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 23: Supported Interfaces
Supported Interfaces This section describes the ConnectX-8 SuperNIC supported interfaces. Each numbered interface referenced in the gures is described in the following table with a link to detailed information. Note The below gures are for illustration purposes only and might not re ect the current revision of the SuperNIC.
Page 24 900- 9X81Q- 00CN- ConnectX-8 ConnectX-8 IC Integrated Circuit PCIe Gen6 Host Interface through x16 edge connector Networking Network tra c Interfaces is transmitted through the networking connectors. The networking connectors allow for the use of NVIDIA ConnectX-8 SuperNIC User Manual...
Page 25 Management remote Interface management Interfaces Detailed Description ConnectX-8 IC The ConnectX-8 family of IC devices delivers In niBand and Ethernet connectivity paired with best-in-class hardware capabilities that accelerate and secure cloud and data-center workloads. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 26: Host Interface
Host Interface ConnectX-8 SuperNIC supports PCI Express Gen6 (5.0 and 4.0 compatible) through an x16 edge connector. The following lists host interface features: PCIe Gen6 or Gen5 (up to x32 PCIe lanes) NVIDIA Multi-Host™ (up to 4 hosts) PCIe switch downstream port containment (DPC) - Applies to 900-9X81E-00EX-DT0...
Page 27 In niBand Architecture Speci cation, Release 1.7 . Networking Ports LEDs Speci cations For the description of the networking ports LEDs, follow the table below, depending on the ConnectX-8 SuperNIC you have purchased. Protocol Speci cations LEDs Scheme The network ports...
Page 28 C8180 SuperNICs 900- 9X81E- 00EX- Scheme 2: Two LEDs 900- 9X81E- 00EX- Scheme 1: One Bi-Color LED There is one bi-color (Yellow and Green) I/O LED per port that indicate port speed and link status. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 29 Ethernet standards in Features and Bene ts. The network ports are compliant with the In niBand In niBand Architecture Speci cation, Release 1.7 . C8240 SuperNICs 900- Scheme 1: One Bi- 9X81Q- Color LED 00CN- NVIDIA ConnectX-8 SuperNIC User Manual...
Page 30 Indicates an error with the link. The error can be one of the following: Descri Beha ption vior Blink acces s to until netwo error rking ports fails Over- Blink curren until condit ion of error netwo NVIDIA ConnectX-8 SuperNIC User Manual...
Page 31 Descri Beha ption vior rking ports Physical The Green LED will Activity blink. The Green LED will Link Up be solid. Physical Up The Yellow LED will (IB Only) be solid. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 32 Scheme 2: Two LEDs There are two I/O LEDs per port that indicate port speed and link status. LED1 is a bi-color LED (Yellow and Green) LED2 is a single-color LED (Green) NVIDIA ConnectX-8 SuperNIC User Manual...
Page 33 LEDs Scheme The network ports comply with the IEEE 802.3 Ethernet Ethernet standards in Features and Bene ts. The network ports are compliant with the In niBand In niBand Architecture Speci cation, Release 1.7 . NVIDIA ConnectX-8 SuperNIC User Manual...
Page 34 1Hz blinking Yellow SuperNIC Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: Descri Beha ption vior Blink access until to the netwo error rking NVIDIA ConnectX-8 SuperNIC User Manual...
Page 35 Physical Up The Yellow LED will (IB Only) be solid. Beacon command for locating 1Hz blinking Yellow SuperNIC Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 36 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 37 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 38 Physical The Green LED will Blinking Activity blink. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 39 The sideband management interface (a 30-pin IPEX connector) in ConnectX-8 SuperNICs enables out-of-band management, allowing administrators to monitor and control the NVIDIA ConnectX-8 SuperNIC User Manual...
Page 40 The table below speci es the maximum trace lengths per board type. Please consider the maximum trace length on the board in your design. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 41 IEEE 802.3 Ethernet Ethernet standards in Features and Bene ts. The network ports are compliant with the In niBand In niBand Architecture Speci cation, Release 1.7 . C8240 Scheme 1: One Bi- SuperNICs Color LED NVIDIA ConnectX-8 SuperNIC User Manual...
Page 42 1Hz blinking Yellow locating SuperNIC Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: Descri Beha ption vior Blink acces until s to error netwo rking NVIDIA ConnectX-8 SuperNIC User Manual...
Page 43 Physical The Yellow LED will Up (IB be solid. Only) Beacon command 1Hz blinking Yellow locating SuperNIC Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 44 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 45 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 46 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 47 Extender board provides access to the MTUSB, PPS, NCSI, EN_INB_REC, and FNP interfaces, enabling debugging. The kit includes the extender board and a 200mm IPEX cable ( micro-coax, pin-to-pin, lock-to-lock ) that can be connected via the 30-pin connector on the ConnectX-8 SuperNIC. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 48 IEEE 802.3 Ethernet Ethernet standards in Features and Bene ts. The network ports are compliant with the In niBand In niBand Architecture Speci cation, Release 1.7 . C8240 Scheme 1: One Bi- SuperNIC Color LED NVIDIA ConnectX-8 SuperNIC User Manual...
Page 49 900- Beacon command 1Hz blinking Yellow locating SuperNIC Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: Descri Beha ption vior Blink acces until s to NVIDIA ConnectX-8 SuperNIC User Manual...
Page 50 Physical The Yellow LED will Up (IB be solid. Only) Beacon command 1Hz blinking Yellow locating SuperNIC Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 51 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 52 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 53 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 54 In full port speed: the Green LED is solid Link Up In less than full port speed: the Yellow LED is solid C8240 SuperNIC 900- 140mm (5.51 inch) 9X81Q- 00CN-ST0 C8180 75mm (2.95 inch) SuperNIC NVIDIA ConnectX-8 SuperNIC User Manual...
Page 55: Pinouts Description
Extender for Low- 9XCBL- Speed Signals Over 000A-000 30p Debug Connector Pinouts Description NC-SI Sideband Interface The following table lists the NC-SI management interface pinout descriptions. For further details, please refer to NC-SI Sideband Interface. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 56 ConnectX Flash is NOT present enable_inband_recovery PCIe operatiin live sh mode PPS IN PPS input for timing applications PPS OUT PPS output for timing applications RSVD3 Reserved RSVD4 Reserved RSVD5 Reserved RSVD6 Reserved NVIDIA ConnectX-8 SuperNIC User Manual...
Page 57 Reserved MCIO Interface The tables below list the MCIO connector pinout descriptions, per SuperNIC OPN. For further details, please refer to MCIO Connector. Default (Socket-Direct/Multi-Host) Pinouts for 900- 9X81Q-00CN-ST0 and 900-9X81E-00EX-ST0 SuperNICs Description PERp0 PERn0 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 58 Description PERp1 PERn1 SMBUS_SCL SMBUS_SDA NVIDIA ConnectX-8 SuperNIC User Manual...
Page 59 Description PERST_A_N CBL_PRES_A PERp2 PERn2 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 60 Description PERp3 PERn3 PERp4 PERn4 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 61 Description PERp5 PERn5 FLEXIO3_A/I2C_M_ SDA_A (NC) NVIDIA ConnectX-8 SuperNIC User Manual...
Page 62 Description FLEXIO4_A/I2C_M_ SCL_A (NC) PERST_B_N NVIDIA ConnectX-8 SuperNIC User Manual...
Page 63 Description CBL_PRES_B PERp6 PERn6 PERp7 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 64 Description PERn7 PERp8 PERn8 PERp9 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 65 Description PERn9 PERp10 PERn10 PERp11 PERn11 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 66 Description PERp12 PERn12 PERp13 PERn13 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 67 Description PERp14 PERn14 PERp15 PERn15 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 68 Description Switch Mode Pinouts for 900-9X81E-00EX-DT0 SuperNICs Description PERp0 PERn0 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 69 Description PERp1 PERn1 SMBUS_SCL SMBUS_SDA NVIDIA ConnectX-8 SuperNIC User Manual...
Page 70 Description PERST_A_N CBL_PRES_A PERp2 PERn2 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 71 Description PERp3 PERn3 PERp4 PERn4 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 72 Description PERp5 PERn5 FLEXIO3_A/I2C _M_SDA_A (NC) NVIDIA ConnectX-8 SuperNIC User Manual...
Page 73 Description FLEXIO4_A/I2C _M_SCL_A (NC) PERST_B_N CBL_PRES_B NVIDIA ConnectX-8 SuperNIC User Manual...
Page 74 Description PERp6 PERn6 PERp7 PERn7 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 75 Description PERp8 PERn8 PERp9 PERn9 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 76 Description PERp10 PERn10 PERp11 PERn11 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 77 Description PERp12 PERn12 PERp13 PERn13 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 78 Description PERp14 PERn14 PERp15 PERn15 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 79 Description PERp0 PERn0 PERp1 PERn1 3V3_AUX NVIDIA ConnectX-8 SuperNIC User Manual...
Page 80 Description VPP_INT_N REFCLK_A_Dp REFCLK_A_Dn NVIDIA ConnectX-8 SuperNIC User Manual...
Page 81 Description PERp2 PERn2 PERp3 PERn3 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 82 Description PERp4 PERn4 PERp5 PERn5 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 83 Description VPP_SDA_A VPP_SCL_A NVIDIA ConnectX-8 SuperNIC User Manual...
Page 84 Description FLEXIO7_A/US B2_A_Dp /SER_DO(NC) FLEXIO8_A/US B2_A_Dn/SER_ DI(NC) NVIDIA ConnectX-8 SuperNIC User Manual...
Page 85 Description PERp6 PERn6 PERp7 PERn7 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 86 Description PERp8 PERn8 PERp9 PERn9 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 87 Description PERp10 PERn10 PERp11 PERn11 PERp12 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 88 Description PERn12 PERp13 PERn13 PERp14 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 89 Description PERn14 PERp15 PERn15 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 90: Hardware Installation
Hardware Installation Installation and initialization of ConnectX-8 SuperNIC require attention to the mechanical attributes, power speci cation, and precautions for electronic equipment. Safety Warnings Info Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the SuperNIC Installation Safety Instructions.
Page 91: Installation Procedure Overview
IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20 Installation Procedure Overview The installation procedure of ConnectX-8 SuperNICs involves the following steps: Step Procedure Direct Link Check the system’s System Requirements requirements. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 92 Identify the SuperNIC in the Identifying Your SuperNIC system System Requirements To successfully integrate the ConnectX-8 SuperNIC into a chassis, ensure that the following system requirements are met: Warning Unless otherwise speci ed, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination.
Page 93 Power Supply: Ensure the system’s power supply unit (PSU) can provide su cient power for the ConnectX-8 SuperNIC, typically requiring 75W from the PCIe slot. If an additional power connector is required (depending on the system con guration), ensure the PSU has the necessary connectors for auxiliary power.
Page 94 Safety Precautions Before Installing ConnectX-8 SuperNIC Pre-Installation Checklist Unpack the ConnectX-8 SuperNIC. Check against the package contents list that all the parts have been sent. Check the parts for visible damage that may have occurred during shipping. Please note that the cards must be placed on an antistatic surface.
Page 95: Bracket Replacement Instructions
Refer to Bracket Replacement Instructions . Bracket Replacement Instructions The ConnectX-8 SuperNIC and PCIe Auxiliary Connection card are usually shipped with an assembled high-pro le bracket. If this form factor is suitable for your requirements, you can skip the remainder of this section and move to Installation Instructions. If you need to replace the high-pro le bracket with the short bracket that is included in the shipping box, please follow the instructions in this section.
Page 96: Installation Instructions
Removing the Existing Bracket 1. Using a torque driver, remove the two screws holding the bracket in place. 2. Separate the bracket from the ConnectX-8 SuperNIC. Warning Be careful not to put stress on the LEDs on the SuperNIC. 3. Save the two screws.
Page 97: Cables And Modules
Do not apply any torque, up or down, to the connector cage in the SuperNIC. 4. Make sure that the connector locks in place. Note When installing cables make sure that the latches engage. Warning NVIDIA ConnectX-8 SuperNIC User Manual...
Page 98 “Mellanox Technologies”: ConnectX-8 SuperNIC Output Example Con guration Single-port Socket Direct SuperNIC (2x PCIe x16) [root@mftqa-009 ~]# lspci |grep mellanox -i a3:00.0 Infiniband controller: Mellanox Technologies CX8 Family [ConnectX-8] NVIDIA ConnectX-8 SuperNIC User Manual...
Page 99 PCIe slots, each SuperNIC gets a unique PCI Bus and Device number. Each of the PCIe x16 busses sees two network ports; in e ect, the two physical ports of the ConnectX-8 Socket Direct SuperNIC are viewed as four net devices by the system. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 100 1. Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. 2. Expand System Devices and locate your ConnectX-8 SuperNIC. 3. Right-click the mouse on your SuperNIC's row and select Properties to display the SuperNIC properties window.
Page 101 Note If the PCI device does not have a SuperNIC ID, return to Step 2 to check another device. Note The list of NVIDIA PCI Device IDs can be found at the PCI ID repository. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 102: Connectx-8 Pcie X16 Installation Instructions
(depending on the chassis model). Set the screws aside in a safe place. Locate an available PCIe slot that is compatible with the ConnectX-8 SuperNIC. Refer to the chassis and motherboard manual to ensure the PCIe slot supports the NIC's speci cations (PCIe Gen6 x16).
Page 103 Avoid excessive force. Secure the SuperNIC: Use a screw (often provided with the chassis or card) to secure the card's metal bracket to the chassis. This prevents the card from moving and ensures a stable connection. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 104 Ensure the connections are secure before powering on the chassis. Power On and Verify the Installation Power on the chassis. Verify that the ConnectX-8 SuperNIC is properly recognized by the system. Depending on the operating system: lspci | grep Mellanox On Linux, run to ensure the NIC is detected.
Page 105: Open The Chassis
Using a screwdriver, unscrew and remove the chassis side panel (or top cover, depending on the chassis design). Keep the screws in a safe place for reassembly later. Identify the ConnectX-8 SuperNIC you wish to remove. It will be seated in one of the PCIe slots. Disconnect the Networking Cable To remove the networking cable, disengage the locks and slowly pull the connector away from the port receptacle.
Page 106 Do not twist or force it out. (Optional) Inspect and Clean the Components If needed, inspect the PCIe slot and the NIC for dust or debris. Use compressed air to clean them, but avoid touching the connectors directly. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 107: Close The Chassis
Windows, you can check the Device Manager under "Network Adapters." ConnectX-8 Extension Option (2x PCIe x16) Installation Instructions Note This section applies to SuperNICs when used with the extension option. The images below are for illustration purposes only. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 108 Speci cations. Connect the ConnectX-8 SuperNIC in an available PCI Express x16 slot in the chassis. Refer to ConnectX-8 PCIe x16 Installation Instructions. Connecting the MCIO Harness Identify Connectors: Locate the MCIO connectors on both the ConnectX-8 SuperNIC and the PCIe Auxiliary Board.
Page 109 MCIO harness length (250mm) is taken into account. Position the PCIe Auxiliary Board: Align the PCIe Auxiliary Board with an available PCIe slot. Insert the Board: Gently insert the board into the slot, applying even pressure until it is fully seated. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 110 Secure the Board: Use the chassis retention mechanisms (e.g., screws or latches) to secure the board in place. Installing the Networking Cable For more information, refer to Networking Cable Installation. Uninstallation Instructions Preparation NVIDIA ConnectX-8 SuperNIC User Manual...
Page 111 Using a screwdriver, unscrew and remove the chassis side panel (or top cover, depending on the chassis design). Keep the screws in a safe place for reassembly later. Identify the ConnectX-8 SuperNIC you wish to remove. It will be seated in one of the PCIe slots. Disconnect the Networking Cable To remove the networking cable, disengage the locks and slowly pull the connector away from the port receptacle.
Page 112 2. Gently disconnect the MCIO harness from the SuperNIC MCIO connector. 3. Similarly, disconnect the other end of the MCIO harness from the PCIe Auxiliary card. 4. Once both ends are disconnected, carefully remove the MCIO harness from the system. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 113 3. Release the Retention Mechanism: Disengage the hockey stick on the PCIe Auxiliary card bracket. 4. Remove PCIe Card: Carefully pull the PCIe Auxiliary card out of the PCIe slot, making sure to hold it by its edges. Avoid touching the connectors. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 114 2. Release the SuperNIC from the Slot: If the SuperNIC is secured in place with screws or latches (Hocket Stick), remove or release them. 3. Remove the SuperNIC: Gently pull the SuperNIC from its PCIe slot, again holding it by the edges to avoid damage. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 115: Final Steps
2. Inspect for Damage: Examine the components for any signs of physical damage that might have occurred during installation or removal. Dispose of Components Properly If you plan on disposing of any of the hardware, make sure to follow your organization’s disposal guidelines for electronic waste (e-waste). NVIDIA ConnectX-8 SuperNIC User Manual...
Page 116: Driver Installation
The installation requires administrator (root) Installer Privileges privileges on the target machine. Downloading MLNX_OFED 1. Verify that the system has a network adapter installed by running lspci command. The below table provides output examples per ConnectX-7 card con guration. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 117 1. The other card is installed in a PCI slot with PCI Bus address 82 (hexadecimal), PCI Device number 00 and PCI Function number 0 and 1.Since the two PCIe cards are installed in two PCIe slots, each card gets a unique PCI Bus NVIDIA ConnectX-8 SuperNIC User Manual...
Page 118 2. Download the ISO image to your host. The image’s name has the format MLNX_OFED_LINUX-<ver>-<OS label><CPU arch>.iso You can download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the NVIDIA web site at nvidia.com/en-us/networking Products Software In niBand Drivers NVIDIA MLNX_OFED 1.
Page 119 Note: To perform a rmware upgrade using customized rmware binaries, a path can be provided to the folder that contains the rmware binary les, by running --fw-image-dir. Using this option, the rmware version embedded in the MLNX_OFED package will be ignored. Example: ./mlnxofedinstall --fw-image-dir /tmp/my_fw_bin_files NVIDIA ConnectX-8 SuperNIC User Manual...
Page 120 (suchaspdsh). If your kernel version does not match with any of the o ered pre-built RPMs, you can add your kernel version by using the “mlnx_add_kernel_support.sh” script located inside the MLNX_OFED package. Note NVIDIA ConnectX-8 SuperNIC User Manual...
Page 121 'mdev' already exists with different path The “mlnx_add_kernel_support.sh” script can be executed directly from the mlnxofedinstall script. For further information, please see '--add-kernel- support' option below. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 122 Building OFED RPMs. Please wait... Removing OFED RPMs... Created /tmp/MLNX_OFED_LINUX-x.x-x-rhel7. -x86_64-ext.tgz The script adds the following lines to /etc/security/limits.conf for the userspace components such as MPI: * soft memlock unlimited * hard memlock unlimited NVIDIA ConnectX-8 SuperNIC User Manual...
Page 123 For the list of installation options, run: ./mlnxofedinstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards. 1. Log in to the installation machine as root. 2. Mount the ISO image on your machine.
Page 124 ./mlnxofedinstall --without-dkms --add-kernel-support -- kernel --without-fw-update --force For example: ./mlnxofedinstall --without-dkms --add-kernel-support -- kernel 3.13.0-85-generic --without-fw-update --force Note that the path to kernel sources (--kernel-sources) should be added if the sources are not in their default location. Note NVIDIA ConnectX-8 SuperNIC User Manual...
Page 125 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: ---------- Device Type: ConnectX-X Part Number: MCXXXX-XXX NVIDIA ConnectX-8 SuperNIC User Manual...
Page 126 Refer to the table below to nd the appropriate action for your speci c card. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 127 ConnectX- 7 Card Action \ Output Examples Driver Restart Con gurati Adapter Single- port [root@mft Socket qa-009 Direct ~]# lspci Card (2x PCIe x16) |grep mellanox a3:00.0 Infiniban controlle Mellanox Technolog NVIDIA ConnectX-8 SuperNIC User Manual...
Page 128 MT2910 Family [ConnectX e3:00.0 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX Dual-port Socket [root@mft Direct qa-009 Card (2x ~]# lspci PCIe x16) |grep mellanox 05:00.0 Infiniban controlle Mellanox Technolog MT2910 Family NVIDIA ConnectX-8 SuperNIC User Manual...
Page 129 [ConnectX 05:00.1 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX 82:00.0 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX 82:00.1 Infiniban controlle Mellanox Technolog MT2910 Family NVIDIA ConnectX-8 SuperNIC User Manual...
Page 130 Device number. Each of the PCIe x16 busses sees two network ports; in e ect, the two physical ports of the ConnectX-7 Socket Direct adapter are viewed as four net devices by the system. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 131 [root@mft qa-009 ~]# lspci |grep mellanox 3:00.0 Infiniban Single- port PCIe controlle x16 Card Mellanox Technolog MT2910 Family [ConnectX Dual-port PCIe x16 [root@mft Card qa-009 ~]# lspci |grep mellanox 86:00.0 Network controlle Mellanox Technolog MT2910 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 132 5. (In niBand only) Run the hca_self_test.ofed utility to verify whether or not the In niBand link is up. The utility also checks for and displays additional information such as: HCA rmware version Kernel architecture NVIDIA ConnectX-8 SuperNIC User Manual...
Page 133 The list of the modules that will be loaded automatically upon boot can be found in the /etc/in niband/openib.conf le. Note Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. Installation Results NVIDIA ConnectX-8 SuperNIC User Manual...
Page 134 ConnectX-7 Card Action \ Output Examples Driver Restart Con gurati Adapter Single-port Socket [root@mftqa Direct Card -009 ~]# (2x PCIe lspci |grep x16) mellanox -i a3:00.0 Infiniband controller: Mellanox Technologie s MT2910 Family [ConnectX- NVIDIA ConnectX-8 SuperNIC User Manual...
Page 135 [ConnectX- Dual-port Socket [root@mftqa Direct Card -009 ~]# (2x PCIe lspci |grep x16) mellanox -i 05:00.0 Infiniband controller: Mellanox Technologie s MT2910 Family [ConnectX- 05:00.1 Infiniband controller: Mellanox Technologie s MT2910 Family [ConnectX- 82:00.0 Infiniband NVIDIA ConnectX-8 SuperNIC User Manual...
Page 136 The other card is installed in a PCI slot with PCI Bus address 82 (hexadecimal), PCI Device number 00 and PCI Function number 0 and 1.Since the two PCIe cards are installed in two PCIe slots, each NVIDIA ConnectX-8 SuperNIC User Manual...
Page 137 [root@mftqa -009 ~]# lspci |grep mellanox - 3:00.0 Single-port Infiniband PCIe x16 controller: Card Mellanox Technologie s MT2910 Family [ConnectX- Dual-port PCIe x16 [root@mftqa Card -009 ~]# lspci |grep mellanox - 86:00.0 Network NVIDIA ConnectX-8 SuperNIC User Manual...
Page 138 MT2910 Family [ConnectX- Standard ConnectX- 4/ConnectX -4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Software Most of MLNX_OFED packages are installed under the “/usr” directory except for the following packages NVIDIA ConnectX-8 SuperNIC User Manual...
Page 139 The kernel modules are installed under /lib/mod ules/`un ame - r`/updat es on SLES Fedora Distribut ions /lib/mod ules/`un ame - r`/extra/ mlnx- ofa_kern el on RHEL other RedHat like Distribut ions NVIDIA ConnectX-8 SuperNIC User Manual...
Page 140 ‘-- without- update’ rmware version of the adapter device is older than the rmware version included with the OFED image NVIDIA ConnectX-8 SuperNIC User Manual...
Page 141 In case your machine has unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM21501100 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 142 .IBMM2150110033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically. To prevent the automatic load of the NVIDIA drivers upon system boot: 1. Add the following lines to the "/etc/modprobe.d/mlnx.conf" le. blacklist mlx5_core blacklist mlx5_ib 2.
Page 143 The table below lists the mlnxofedinstall script return codes and their meanings. ConnectX- 7 Card Action \ Output Examples Driver Restart Con gurat Adapter Single- port [root@mftq Socket a-009 ~]# Direct lspci Card (2x PCIe x16) |grep mellanox - a3:00.0 Infiniband NVIDIA ConnectX-8 SuperNIC User Manual...
Page 144 Infiniband controller : Mellanox Technologi es MT2910 Family [ConnectX- Dual-port Socket [root@mftq Direct a-009 ~]# Card (2x lspci PCIe x16) |grep mellanox - 05:00.0 Infiniband controller : Mellanox Technologi es MT2910 Family [ConnectX- 05:00.1 Infiniband NVIDIA ConnectX-8 SuperNIC User Manual...
Page 145 PCI slot with PCI Bus address 05 (hexadecimal), PCI Device number 00 and PCI Function number 0 and 1. The other card is NVIDIA ConnectX-8 SuperNIC User Manual...
Page 146 ConnectX-7 Socket Direct adapter are viewed as four net devices by the system. Single- port PCIe [root@mftq x16 Card a-009 ~]# lspci |grep mellanox - 3:00.0 Infiniband controller : Mellanox Technologi es MT2910 Family NVIDIA ConnectX-8 SuperNIC User Manual...
Page 147 86:00.0 Network controller : Mellanox Technologi Dual-port es MT2910 PCIe x16 Card Family [ConnectX- 7] 86:00.1 Network controller : Mellanox Technologi es MT2910 Family [ConnectX- Standard ConnectX- 4/Connect X-4 Lx or higher Adapters with Multi- NVIDIA ConnectX-8 SuperNIC User Manual...
Page 148 “/opt” directory: fca and ibutils iproute 2 (rdma tool) - installe d under /opt/Me llanox/i proute2 /sbin/rd The kernel modules are installed under /lib/mod ules/`u name - r`/upda tes on SLES Fedora NVIDIA ConnectX-8 SuperNIC User Manual...
Page 149 Ubuntu Firmware The rmware of existing network adapter devices will be updated if the following conditions are ful lled: installat script is run in default mode; that is, without option ‘- NVIDIA ConnectX-8 SuperNIC User Manual...
Page 150 OFED image Note: If adapter ’s Flash originall progra mmed with an Expansi on ROM image, automa rmwar update will also burn an Expansi NVIDIA ConnectX-8 SuperNIC User Manual...
Page 151 NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor." The Installation ended successfully The installation failed No rmware was found for the adapter device Invalid parameter NVIDIA ConnectX-8 SuperNIC User Manual...
Page 152 Prerequisites are not met. For example, missing the required software installed or the hardware is not con gured correctly Failed to start the mst driver NVIDIA ConnectX-8 SuperNIC User Manual...
Page 153 7 Card Action \ Output Examples Driver Restart Con gurat Adapter Single- port [root@mftq Socket a-009 ~]# Direct lspci Card (2x PCIe x16) |grep mellanox - a3:00.0 Infiniband controller : Mellanox Technologi es MT2910 Family NVIDIA ConnectX-8 SuperNIC User Manual...
Page 154 Family [ConnectX- Dual-port Socket [root@mftq Direct a-009 ~]# Card (2x lspci PCIe x16) |grep mellanox - 05:00.0 Infiniband controller : Mellanox Technologi es MT2910 Family [ConnectX- 05:00.1 Infiniband controller : Mellanox Technologi es MT2910 Family NVIDIA ConnectX-8 SuperNIC User Manual...
Page 155 Bus address 05 (hexadecimal), PCI Device number 00 and PCI Function number 0 and 1. The other card is installed in a PCI slot with PCI Bus address 82 (hexadecimal), PCI Device number 00 and PCI Function NVIDIA ConnectX-8 SuperNIC User Manual...
Page 156 [root@mftq a-009 ~]# lspci |grep mellanox - 3:00.0 Single- port PCIe Infiniband x16 Card controller : Mellanox Technologi es MT2910 Family [ConnectX- Dual-port PCIe x16 [root@mftq Card a-009 ~]# NVIDIA ConnectX-8 SuperNIC User Manual...
Page 157 MT2910 Family [ConnectX- 7] 86:00.1 Network controller : Mellanox Technologi es MT2910 Family [ConnectX- Standard ConnectX- 4/Connect X-4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Software Most of MLNX_OFED packages are NVIDIA ConnectX-8 SuperNIC User Manual...
Page 158 - installe d under /opt/Me llanox/i proute2 /sbin/rd The kernel modules are installed under /lib/mod ules/`u name - r`/upda tes on SLES Fedora Distribu tions /lib/mod ules/`u name - r`/extra /mlnx- ofa_ker nel on NVIDIA ConnectX-8 SuperNIC User Manual...
Page 159 ‘- without - fw- update’ rmwar version of the NVIDIA ConnectX-8 SuperNIC User Manual...
Page 160 Note: If adapter ’s Flash originall progra mmed with an Expansi on ROM image, automa rmwar update will also burn an Expansi on ROM image. In case your machine has unsupported network adapter device, no NVIDIA ConnectX-8 SuperNIC User Manual...
Page 161 The installation failed No rmware was found for the adapter device Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware is not NVIDIA ConnectX-8 SuperNIC User Manual...
Page 162 “/usr” directory except for the following packages which are installed under the “/opt” directory: fca and ibutils iproute 2 (rdma tool) - installe d under /opt/Me llanox/i proute2 /sbin/rd The kernel modules are installed under NVIDIA ConnectX-8 SuperNIC User Manual...
Page 163 RHEL other RedHat like Distribu tions /lib/mod ules/`u name - r`/upda tes/dk ms/ on Ubuntu Firmware The rmware of existing network adapter devices will be updated if the following conditions are ful lled: installat NVIDIA ConnectX-8 SuperNIC User Manual...
Page 164 ‘- without - fw- update’ rmwar version of the adapter device is older than rmwar version include d with OFED image Note: If adapter ’s Flash originall progra mmed with an Expansi on ROM image, NVIDIA ConnectX-8 SuperNIC User Manual...
Page 165 "The rmware for this device is distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor." Installation Logging NVIDIA ConnectX-8 SuperNIC User Manual...
Page 166 2. Mount the ISO image on your machine and copy its content to a shared location in your network. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt 3. Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link: http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox NVIDIA ConnectX-8 SuperNIC User Manual...
Page 167 Retrieving key from file: ///repos/MLNX_OFED/<MLNX_OFED ﬁle>/RPM-GPG- KEY-Mellanox Importing GPG key 0x6224C050 Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mellanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY- Mellanox ok [y/N]: this 5. Check that the key was successfully imported. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 168 Loaded plugins: product-id, security, subscription- manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. repo id repo name status mlnx_ofed MLNX_OFED Repository rpmforge RHEL 6Server - RPMforge.net - dag repolist: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 169 WARNING: that it contains unsigned rpms, therefore, you need to disable the gpgcheck WARNING: by setting in the repository conf file. 'gpgcheck=0' Created /tmp/MLNX_OFED_LINUX- -rhel7. -x86_64- ext.tgz 4. Open the tarball. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 170 This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. repo id repo name status mlnx_ofed MLNX_OFED Repository rpmforge RHEL 6Server - RPMforge.net - dag repolist: Installing MLNX_OFED Using the YUM Tool NVIDIA ConnectX-8 SuperNIC User Manual...
Page 171 (User Space packages only) package mlnx-ofed-eth-only-user-only.noarch : MLNX_OFED eth- only-user-only installer (User Space packages package only) mlnx-ofed-guest.noarch : MLNX_OFED guest installer package (with KMP support) mlnx-ofed-guest-user-only.noarch : MLNX_OFED guest-user- only installer (User Space packages only) package NVIDIA ConnectX-8 SuperNIC User Manual...
Page 172 (User Space packages only) package mlnx-ofed-vma-user-only.noarch : MLNX_OFED vma-user-only installer (User Space packages only) package mlnx-ofed-vma-vpi.noarch : MLNX_OFED vma-vpi installer (with KMP support) package mlnx-ofed-vma-vpi-user-only.noarch : MLNX_OFED vma-vpi- user-only installer (User Space packages only package where: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 173 ConnectX -7 Card Action \ Output Examples Driver Restart Con gura Adapter tion Single- port [root@mft Socket qa-009 Direct ~]# lspci Card (2x PCIe x16) |grep mellanox a3:00.0 Infiniban controlle Mellanox Technolog NVIDIA ConnectX-8 SuperNIC User Manual...
Page 174 MT2910 Family [ConnectX e3:00.0 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX Dual- port [root@mft Socket qa-009 Direct ~]# lspci Card (2x PCIe x16) |grep mellanox 05:00.0 Infiniban controlle Mellanox Technolog MT2910 Family NVIDIA ConnectX-8 SuperNIC User Manual...
Page 175 [ConnectX 05:00.1 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX 82:00.0 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX 82:00.1 Infiniban controlle Mellanox Technolog MT2910 Family NVIDIA ConnectX-8 SuperNIC User Manual...
Page 176 Device number. Each of the PCIe x16 busses sees two network ports; in e ect, the two physical ports of the ConnectX-7 Socket Direct adapter are viewed as four net devices by the system. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 177 ~]# lspci |grep mellanox 3:00.0 Infiniban Single- port PCIe controlle x16 Card Mellanox Technolog MT2910 Family [ConnectX Dual- port PCIe [root@mft x16 Card qa-009 ~]# lspci |grep mellanox 86:00.0 Network controlle Mellanox Technolog MT2910 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 178 Mellanox Technolog MT2910 Family [ConnectX Standard ConnectX 4/Connec tX-4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Software Most of MLNX_OFE D packages are installed under the “/usr” directory except for following NVIDIA ConnectX-8 SuperNIC User Manual...
Page 179 (rdma tool) - install under /opt/M ellano x/iprou te2/sbi n/rdm The kernel modules are installed under /lib/mo dules/ `una me - r`/upd ates SLES Fedora Distrib utions /lib/mo dules/ `una me - r`/extr a/mlnx NVIDIA ConnectX-8 SuperNIC User Manual...
Page 180 - r`/upd ates/d kms/ Ubunt Firmware rmware of existing network adapter devices will be updated if the following conditions are ful lled: install ation script is run defaul mode; that is, withou t the option NVIDIA ConnectX-8 SuperNIC User Manual...
Page 181 ‘-- withou t- fw- updat e’ rmwa versio n of adapte device older than rmwa versio includ with OFED image Note: If an adapte r’s Flash origina progra mmed with Expan sion image, NVIDIA ConnectX-8 SuperNIC User Manual...
Page 182 "The rmware for this device is distributed inside NVIDIA driver: 0000:01:00. 0 (PSID: IBM215011 0033) To obtain rmware for this device, NVIDIA ConnectX-8 SuperNIC User Manual...
Page 183 Prerequisites are not met. For example, missing the required software installed or the hardware is not con gured correctly Failed to start the mst driver Software Most of MLNX_OFE D packages are installed under the NVIDIA ConnectX-8 SuperNIC User Manual...
Page 184 “/opt” directory: ibutils iprout (rdma tool) - install under /opt/M ellano x/iprou te2/sbi n/rdm The kernel modules are installed under /lib/mo dules/ `una me - r`/upd ates SLES Fedora Distrib utions /lib/mo dules/ NVIDIA ConnectX-8 SuperNIC User Manual...
Page 185 Distrib utions /lib/mo dules/ `una me - r`/upd ates/d kms/ Ubunt Firmware rmware of existing network adapter devices will be updated if the following conditions are ful lled: install ation script is run defaul NVIDIA ConnectX-8 SuperNIC User Manual...
Page 186 ‘-- withou t- fw- updat e’ rmwa versio n of adapte device older than rmwa versio includ with OFED image Note: If an adapte r’s Flash origina progra mmed with NVIDIA ConnectX-8 SuperNIC User Manual...
Page 187 In case your machine has an unsupporte d network adapter device, no rmware update will occur and the error message below will be printed. "The rmware for this device is distributed inside NVIDIA driver: 0000:01:00. 0 (PSID: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 188 Installs packages ofed-vma required by VMA Installs packages mlnx- required by VMA ofed- to work over vma-eth Ethernet mlnx- Installs packages ofed- required by VMA vma-vpi to support VPI Installs packages blue eld required for BlueField NVIDIA ConnectX-8 SuperNIC User Manual...
Page 189 .fc21.x86_64 (without package 3.17 KMP support) mlnx-ofed-hypervisor- .fc21.x86_64.noarch : 3.17 MLNX_OFED hypervisor installer kernel package 3.17 .fc21.x86_64 (without KMP support) mlnx-ofed-vma- .fc21.x86_64.noarch : MLNX_OFED vma 3.17 installer kernel .fc21.x86_64 (without package 3.17 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 190 Please note that the resulting MLNX_OFED repository will contain unsigned RPMs, therefore, you should set 'gpgcheck=0' in the repository con guration 2. Install the desired group. # yum install mlnx-ofed-all Loaded plugins: langpacks, product-id, subscription- manager NVIDIA ConnectX-8 SuperNIC User Manual...
Page 191 # yum install mlnx-fw-updater Installing MLNX_OFED Using apt-get This type of installation is applicable to Debian and Ubuntu operating systems. Setting up MLNX_OFED apt-get Repository 1. Log into the installation machine as root. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 192 3. Create an apt-get repository con guration le called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: deb file:/<path to extracted MLNX_OFED >/DEBS ./ package 4. Download and install NVIDIA's Technologies GPG-KEY. # wget -qO - http: //www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - 5. Verify that the key was successfully imported.
Page 193 WARNING: that it contains unsigned rpms, therefore, you need to disable the gpgcheck WARNING: by setting in the repository conf file. 'gpgcheck=0' Created /tmp/MLNX_OFED_LINUX- -rhel7. -x86_64- ext.tgz 4. Open the tarball. # cd /tmp/ # tar -xvf /tmp/MLNX_OFED_LINUX- -rhel7. -x86_64- ext.tgz NVIDIA ConnectX-8 SuperNIC User Manual...
Page 194 - MLNX_OFED vma-vpi installer package (with DKMS support) mlnx-ofed-kernel-only - MLNX_OFED kernel-only installer (with DKMS support) package mlnx-ofed-bluefield - MLNX_OFED bluefield installer (with DKMS support) package mlnx-ofed-hpc-user-only - MLNX_OFED hpc-user-only installer (User Space packages only) package NVIDIA ConnectX-8 SuperNIC User Manual...
Page 195 (User Space package packages only) mlnx-ofed-basic-user-only - MLNX_OFED basic-user-only installer (User Space packages only) package mlnx-ofed-basic-exact - MLNX_OFED basic installer package (with DKMS support) (exact) mlnx-ofed-basic - MLNX_OFED basic installer (with package DKMS support) NVIDIA ConnectX-8 SuperNIC User Manual...
Page 196 (with DKMS support) package where: ConnectX -7 Card Action \ Output Examples Driver Restart Con gura Adapter tion Single- port [root@mft Socket qa-009 Direct ~]# lspci Card (2x PCIe x16) |grep mellanox a3:00.0 Infiniban NVIDIA ConnectX-8 SuperNIC User Manual...
Page 197 Mellanox Technolog MT2910 Family [ConnectX e3:00.0 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX Dual- port [root@mft Socket qa-009 Direct Card (2x ~]# lspci PCIe x16) |grep mellanox 05:00.0 Infiniban controlle Mellanox NVIDIA ConnectX-8 SuperNIC User Manual...
Page 198 Technolog MT2910 Family [ConnectX 05:00.1 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX 82:00.0 Infiniban controlle Mellanox Technolog MT2910 Family [ConnectX 82:00.1 Infiniban controlle Mellanox NVIDIA ConnectX-8 SuperNIC User Manual...
Page 199 PCIe slots, each card gets a unique PCI Bus and Device number. Each of the PCIe x16 busses sees two network ports; in e ect, the two physical ports of the ConnectX-7 Socket Direct NVIDIA ConnectX-8 SuperNIC User Manual...
Page 200 [root@mft qa-009 ~]# lspci |grep mellanox 3:00.0 Infiniban Single- port PCIe controlle x16 Card Mellanox Technolog MT2910 Family [ConnectX Dual- port PCIe [root@mft x16 Card qa-009 ~]# lspci |grep mellanox 86:00.0 Network controlle NVIDIA ConnectX-8 SuperNIC User Manual...
Page 201 Family [ConnectX 86:00.1 Network controlle Mellanox Technolog MT2910 Family [ConnectX Standard ConnectX 4/Connec tX-4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Software Most of MLNX_OFE D packages are installed under the NVIDIA ConnectX-8 SuperNIC User Manual...
Page 202 “/opt” directory: ibutils iprout (rdma tool) - install under /opt/M ellano x/iprou te2/sbi n/rdm The kernel modules are installed under /lib/mo dules/ `una me - r`/upd ates SLES Fedora Distrib utions /lib/mo dules/ NVIDIA ConnectX-8 SuperNIC User Manual...
Page 203 Distrib utions /lib/mo dules/ `una me - r`/upd ates/d kms/ Ubunt Firmware rmware of existing network adapter devices will be updated if the following conditions are ful lled: install ation script is run defaul NVIDIA ConnectX-8 SuperNIC User Manual...
Page 204 ‘-- withou t- fw- updat e’ rmwa versio n of adapte device older than rmwa versio includ with OFED image Note: If an adapte r’s Flash origina progra mmed with NVIDIA ConnectX-8 SuperNIC User Manual...
Page 205 In case your machine has an unsupporte d network adapter device, no rmware update will occur and the error message below will be printed. "The rmware for this device is distributed inside NVIDIA driver: 0000:01:00. 0 (PSID: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 206 This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required software installed or the hardware is not con gured correctly NVIDIA ConnectX-8 SuperNIC User Manual...
Page 207 “/opt” directory: ibutils iprout (rdma tool) - install under /opt/M ellano x/iprou te2/sbi n/rdm The kernel modules are installed under /lib/mo dules/ `una me - r`/upd ates NVIDIA ConnectX-8 SuperNIC User Manual...
Page 208 RHEL other RedHa t like Distrib utions /lib/mo dules/ `una me - r`/upd ates/d kms/ Ubunt Firmware rmware of existing network adapter devices will be updated if the following conditions are ful lled: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 209 ‘-- withou t- fw- updat e’ rmwa versio n of adapte device older than rmwa versio includ with OFED image Note: If an adapte NVIDIA ConnectX-8 SuperNIC User Manual...
Page 210 Expan sion image. In case your machine has an unsupporte d network adapter device, no rmware update will occur and the error message below will be printed. "The rmware for this device is NVIDIA ConnectX-8 SuperNIC User Manual...
Page 211 Installs packages ofed-hpc required for HPC mlnx- Installs packages ofed- required by hyperviso hypervisor OS mlnx- Installs packages ofed-vma required by VMA Installs packages mlnx- required by VMA ofed- to work over vma-eth Ethernet NVIDIA ConnectX-8 SuperNIC User Manual...
Page 212 MLNX_OFED vma ofed-vma installer package MLNX_OFED mlnx- HPC installer ofed-hpc package mlnx- MLNX_OFED ofed- vma-eth installer vma-eth package mlnx- MLNX_OFED ofed- vma-vpi installer vma-vpi package MLNX_OFED knem- DKMS support dkms for mlnx-ofed kernel modules NVIDIA ConnectX-8 SuperNIC User Manual...
Page 213 MLNX_OFED dpdk dpdk installer package mlnx- MLNX_OFED ofed- mlnx-ofed-basic- basic- exact installer exact package MLNX_OFED dpdk- dpdk-upstream- upstream libs installer -libs package 2. Install the desired group. apt-get install '<group name>' Example: apt-get install mlnx-ofed-all NVIDIA ConnectX-8 SuperNIC User Manual...
Page 214: Windows Driver Installation
Windows Driver Installation For Windows, download and install the latest WinOF-2 for Windows software package available via the NVIDIA website at: WinOF-2 webpage. Follow the installation instructions included in the download package (also available from the download page). The snapshots in the following sections are presented for illustration purposes only. The installation interface may slightly vary, depending on the operating system in use.
Page 215 On an x64 (64-bit) machine, the output will be “AMD64”. 2. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > In niBand Drivers (Learn More) > Nvidia WinOF-2. 3. Download the .exe image according to the architecture of your machine (see Step 1).
Page 216: Attended Installation
The following is an example of an installation session. 1. Double click the .exe and follow the GUI instructions to install MLNX_WinOF2. 2. [Optional] Manually con gure your setup to contain the logs option (replace “LogFile” with the relevant directory). MLNX_WinOF2_<revision_version>_All_Arch.exe /v"/l*vx [LogFile]" NVIDIA ConnectX-8 SuperNIC User Manual...
Page 217 Please remove the following oem inf les from driver store: <oem inf list>" 5. [Optional] If you want to skip the check for unsupported devices, run. MLNX_WinOF2_<revision_version>_All_Arch.exe /v " SKIPUNSUPPORTEDDEVCHECK=1" 6. Click Next in the Welcome screen. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 218 7. Read and accept the license agreement and click Next. 8. Select the target folder for the installation. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 219 If the user has an OEM card. In this case, the rmware will not be displayed. If the user has a standard NVIDIA® card with an older rmware version, the rmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA®...
Page 220 Documentation - contains the User Manual and Release Notes Management tools - installation tools used for management, such as mlxstat Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd 2. Click Next to install the desired tools. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 221 11. Click Install to start the installation. 12. In case rmware upgrade option was checked in Step 7, you will be noti ed if a rmware upgrade is required (see NVIDIA ConnectX-8 SuperNIC User Manual...
Page 222: Unattended Installation
13. Click Finish to complete the installation. Unattended Installation NVIDIA ConnectX-8 SuperNIC User Manual...
Page 223 3. [Optional] Manually con gure your setup to contain the logs option: MLNX_WinOF2-[Driver/Version]_<revision_version>_All_-Arch.exe /S /v/qn /v”/l*vx [LogFile]" 4. [Optional] if you wish to control whether to install ND provider or not (i.e., MT_NDPROPERTY default value is True ). MLNX_WinOF2-[Driver/Version]_<revision_version>_All_Arch.exe /vMT_NDPROPERTY= NVIDIA ConnectX-8 SuperNIC User Manual...
Page 224 <oem inf list>" 7. [Optional] If you want to enable the default con guration for Rivermax, run. MLNX_WinOF2_<revision_version>_All_Arch.exe /v "MT_RIVERMAX=1 /l*vx C:\Users\<user>\log.txt " 8. [Optional] If you want to skip the check for unsupported devices, run/ NVIDIA ConnectX-8 SuperNIC User Manual...
Page 225 SKIPUNSUPPORTEDDEVCHECK=1" Firmware Upgrade If the machine has a standard NVIDIA® card with an older rmware version, the rmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade rmware manually, please refer to MFT User Manual.
Page 226: Firmware Upgrade
Each SuperNIC is shipped with the latest version of quali ed rmware at the time of manufacturing. However, NVIDIA issues rmware updates occasionally that provide new features and bug xes. To check that your card is programmed with the latest available rmware version, download the mlxup rmware update and query utility.
Page 227 28.33.0800 28.33.1000 Status: Up to date Perform FW update? [y/N]: y Device #1: Up to date Device #2: Updating FW ... Done Restart needed for updates to take effect. Log File: /var/log/mlxup/mlxup-yyyymmdd.log NVIDIA ConnectX-8 SuperNIC User Manual...
Page 228: Supernic Con Gurations
IP address and network parameters and the source location of the kernel/OS from which to boot. The DHCP server instructs the PreBoot drivers to access the kernel/OS through a TFTP server, an iSCSI target, or some other service. For more information and instructions, refer to UEFI. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 229: Pcie Bifurcation Con Guration Options
PCIe switch with NVMe SSDs as PCIe endpoints. While this list of scenarios is not exhaustive, it highlights the tested and veri ed options. Customers seeking to support unlisted con gurations should contact NVIDIA Support. The SuperNIC exposes two x16 PCIe interfaces, with internal PCIe switch architecture.
Page 230 To change back to the INI default con guration: 1x PCIe x16 supporting Gen6 1. Execute the following commands to con gure the PCIe interface: mlxconfig -d /dev/mst/mt4131_pciconf0 reset 2. Reboot or power cycle the system. Port Splitting Con gurations NVIDIA ConnectX-8 SuperNIC User Manual...
Page 231: Port Splitting Con Gurations
While the con gurations listed here are not exhaustive, they represent the tested and veri ed options. Customers seeking to support unlisted con gurations should contact NVIDIA Support. The Port Splitting feature allows a single physical networking module (QSFP112 or OSFP) to be split into multiple network ports.
Page 232 IP address and network parameters and obtain the source location of the kernel/OS to boot from. The DHCP server instructs the PreBoot drivers to access the kernel/OS through a TFTP server, an iSCSI target, or some other service. For more information and instructions, refer to UEFI. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 233 Miscon guration can potentially damage the system. It is recommended to establish out-of-band connectivity to the ConnectX-8 SuperNIC before setting any con guration for the rst time. This allows resetting the NVCon g parameters to their default values in case of miscon guration.
Page 234 C8180 Port-Splitting Con gurations The C8180 ConnectX-8 SuperNIC, featuring a single-port OSFP module, supports up to 800Gb/s. For the recon guration to take e ect, please perform power cycle. C8180 Con guration #1 (Default): One In niBand Port, Supporting XDR 800Gb/s As the default con guration, the device is con gured with a single In niBand port, providing an In niBand XDR rate of 800Gb/s that operates as four times 200G/s .
Page 235 The con guration is illustrated in the diagram below. To apply this con guration, use the following mlxcon g commands: mlxconfig -d <device> set LINK_TYPE_P1= mlxconfig -d <device> set NUM_OF_PLANES_P1 = mlxconfig -d <device> set MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ ]=FF NVIDIA ConnectX-8 SuperNIC User Manual...
Page 236 -d <device> set NUM_OF_PF = NVIDIA ConnectX-8 SuperNIC User Manual...
Page 237 The con guration is illustrated in the diagram below. To enable this con guration, use the following mlxcon g commands: mlxconfig -d <device> set LINK_TYPE_P1= mlxconfig -d <device> set NUM_OF_PLANES_P1 = mlxconfig -d <device> set MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ NVIDIA ConnectX-8 SuperNIC User Manual...
Page 238 MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ ]=FF mlxconfig -d <device> set NUM_OF_PF = NVIDIA ConnectX-8 SuperNIC User Manual...
Page 239 C8240 Port-Splitting Con gurations The C8240 ConnectX-8 SuperNIC, featuring dual QSFP modules, supports up to 400Gb/s per port. For the recon guration to take e ect, please perform power cycle. C8240 Con guration #1 (Default): Two Ethernet Ports, Each Supporting 400GbE By default, the device is con gured with two Ethernet ports, each supporting a data rate of 400Gb/s.
Page 240 The con guration is illustrated in the diagram below. To apply this con guration, use the following mlxcon g commands: mlxconfig -d <device> set MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ MODULE_SPLIT_M0[ ]=FF mlxconfig -d <device> set MODULE_SPLIT_M1[ MODULE_SPLIT_M1[ MODULE_SPLIT_M1[ MODULE_SPLIT_M1[ NVIDIA ConnectX-8 SuperNIC User Manual...
Page 241: Monitoring
The heatsink is attached to the SuperNIC IC to dissipate the heat from the IC. It is attached by spring-loaded screws. The SuperNIC IC has a thermal shutdown safety mechanism that automatically shuts down the SuperNICcard in cases of high-temperature events, improper thermal coupling, or heatsink removal. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 242 For the required air ow (LFM) per OPN, please refer to the Speci cations chapter. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 243: Troubleshooting
Verify that your switch and SuperNIC port are compatible Check that the latest driver is loaded Link light is on, but with no Check that both the SuperNIC and its link are communication established set to the same speed and duplex settings NVIDIA ConnectX-8 SuperNIC User Manual...
Page 244: Windows Troubleshooting
Export option from the File menu. Assign a le name and save. Mellanox Firmware Tool (MFT) Download and install MFT: MFT Documentation Refer to the User Manual for installation instructions. Once installed, open a CMD window and run: NVIDIA ConnectX-8 SuperNIC User Manual...
Page 245 PSID/board ID from here. int –d <mst_device> –i < rmware_bin_ le> b Event log viewer MST device logs: mst start mst status Collect Log File int –d <mst_device> dc > dump_con guration.log mstdump <mst_device> dc > mstdump.log NVIDIA ConnectX-8 SuperNIC User Manual...
Page 246: Finding The Guid/Mac On The Supernic
Note The product revisions indicated on the labels in the following gures do not necessarily represent the latest revisions of the SuperNICs. Board Label Example To be updated in a future version of this document NVIDIA ConnectX-8 SuperNIC User Manual...
Page 247: Speci Cations
Failure to meet the environmental requirements listed in this user manual may void the warranty. Note Please install the ConnectX-8 SuperNIC in a PCIe slot capable of supplying the required power and air ow. NVIDIA ConnectX-8 SuperNIC User Manual...
Page 248 CR4, 100GBASE-CR2, 100GBASE-CR1, 50GAUI-2 C2M, 50GAUI-1 C2M, 50GBASE-CR, 50GBASE-R2 , 40GBASE-CR4, 40GBASE-R2, 25GBASE-R, 10GBASE-R, 10GBASE-CX4, 1000BASE-CX, CAUI-4 C2M, 25GAUI C2M, XLAUI C2M , XLPPI, SFI Capabilities 900-9X81Q-00CN- Secure Boot Enabled, Crypto Enabled with x16 PCIe extension option NVIDIA ConnectX-8 SuperNIC User Manual...
Page 249 Electrical and The complete electrical and thermal speci cations are provided in Thermal " NVIDIA ConnectX-8 SuperNIC Product Speci cations " document. You Speci cation can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
Page 250 Voltage: 12V, 3.3V, 3.3VAUX Electrical and The complete electrical and thermal speci cations are provided in Thermal " NVIDIA ConnectX-8 SuperNIC Product Speci cations" document. You Speci cation can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
Page 251 The ConnectX-8 SuperNICs supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product. b. The non-operational storage temperature speci cations apply to the product without its package.
Page 252 C8280 SuperNICs SuperNIC Dimensions: PCIe Half Height, Half Physical Length 2.61 in. x 6.62 in. (66.40mm x 168.40mm) Interfaces See Supported Interfaces PCI Express Interface T/s, lane (Gen pati ble) Opti onal: Addi tion PCIe NVIDIA ConnectX-8 SuperNIC User Manual...
Page 253 NDR200 (2 lanes x 100Gb/s per lane) port, HDR (50Gb/s per lane) port, HDR100 (2 lane x 50Gb/s per lane), EDR (25Gb/s per lane) port, FDR (14.0625Gb/s per lane), 1X/2X/4X SDR (2.5Gb/s per lane). NVIDIA ConnectX-8 SuperNIC User Manual...
Page 254 Cryp Enab Voltage: 12V, 3.3V, 3.3VAUX The complete electrical and thermal speci cations are provided in " NVIDIA Electrical and Thermal Speci cations ConnectX-8 SuperNIC Product Speci cations" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
Page 255 C8180 SuperNICs C8280 SuperNICs ° ° ° Humidity NVIDIA ConnectX-8 SuperNIC User Manual...
Page 256 The ConnectX-8 SuperNICs supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product. b. The non-operational storage temperature speci cations apply to the product without its package.
Page 257 C8180 SuperNICs C8280 SuperNICs NVIDIA ConnectX-8 SuperNIC User Manual...
Page 258 Brackets Mechanical Drawings and Dimensions Note All dimensions are in millimeters. The PCB mechanical tolerance is +/- 0.13mm. OPNs Tall Bracket Short Bracket C8180 SuperNICs 900-9X81E- 00EX-ST0 900-9X81E- 00EX-DT0 C8240 SuperNICs 900-9X81Q-00CN- Auxiliary PCIe Connection Card NVIDIA ConnectX-8 SuperNIC User Manual...
Page 259: Pcie Auxiliary Card Kit
Auxiliary card) Channel Insertion Loss Channel insertion loss refers to the reduction in signal power due to the insertion of a device in a transmission line or optical ber. It is typically measured in decibels (dB). NVIDIA ConnectX-8 SuperNIC User Manual...
Page 260: Mcio Harness Pinouts
The table below outlines the channel insertion loss budget for the ConnectX-8 SuperNIC in a PCIe Gen5 architecture (32 GT/s). The listed values are based on measured data. It is advisable to add a 0.5 dB margin to your system, as channel loss may vary.
Page 261 Description PETp1 PETn1 3V3_AUX FLEXIO0_A NVIDIA ConnectX-8 SuperNIC User Manual...
Page 262 Description REFCLK_A_Dp REFCLK_A_Dn PETp2 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 263 Description PETn2 PETp3 PETn3 PETp4 PETn4 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 264 Description PETp5 PETn5 FLEXIO1_A/REFCLK _B_Dp NVIDIA ConnectX-8 SuperNIC User Manual...
Page 265 Description FLEXIO2_A/REFCLK _B_Dn NVIDIA ConnectX-8 SuperNIC User Manual...
Page 266 Description FLEXIO7_A FLEXIO8_A PETp6 PETn6 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 267 Description PETp7 PETn7 PETp8 PETn8 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 268 Description PETp9 PETn9 PETp10 PETn10 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 269 Description PETp11 PETn11 PETp12 PETn12 PETp13 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 270 Description PETn13 PETp14 PETn14 PETp15 NVIDIA ConnectX-8 SuperNIC User Manual...
Page 271: Pcie Auxiliary Kit Technical Speci Cations
PCI Express Gen5: SERDES @ 16 GT/s, x16 lanes (Gen 3.0 compatible) Connectivity Power Voltage: 12V, 3.3V_PCIe, 3.3V_AUX Consumption Environment Operational 0°C to 55°C Temperature Non-operational -40°C to 70°C Humidity 10% to 85% relative Operational humidity NVIDIA ConnectX-8 SuperNIC User Manual...
Page 272 PCIe Auxiliary Card Mechanical Drawings and Dimensions Note All dimensions are in millimeters. The PCB mechanical tolerance is +/- 0.13mm. Bracket Mechanical Drawings and Dimensions Auxiliary PCIe Connection Card Tall Auxiliary PCIe Connection Card Short Bracket Bracket NVIDIA ConnectX-8 SuperNIC User Manual...
Page 273 Auxiliary PCIe Connection Card Tall Auxiliary PCIe Connection Card Short Bracket Bracket MCIO Cable Mechanical Drawing NVIDIA ConnectX-8 SuperNIC User Manual...
Page 274: Document Revision History
NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright,...
Page 275 Terms of Sale for the product. Trademarks NVIDIA and the NVIDIA logo are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. ...

This manual is also suitable for:

C8240 C8180 900-9x81q-00cn-st0 900-9x81e-00ex-st0 900-9x81e-00ex-dt0Connectx-8 supernic c8240 ... Show all

Nvidia ConnectX-8 SuperNIC User Manual

Quick Links

Troubleshooting

Need help?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Nvidia ConnectX-8 SuperNIC

Summary of Contents for Nvidia ConnectX-8 SuperNIC

This manual is also suitable for: