Related Manuals for Nvidia ConnectX-4 Lx
Summary of Contents for Nvidia ConnectX-4 Lx
- Page 1 NVIDIA ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0 User Manual ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0 Exported on Dec/05/2023 11:54 AM...
-
Page 2: Table Of Contents
Installation Instructions................18 Cables and Modules................19 Identifying the Card in Your System............. 20 Uninstalling the Card ................21 Safety Precautions ................17 Card Removal ................... 21 Driver Installation................24 Linux Driver Installation ................24 Prerequisites..................24 Downloading NVIDIA OFED ..............24... - Page 3 Firmware Upgrade ................46 VMware Driver Installation ............... 46 Hardware and Software Requirements ............. 46 Installing NATIVE ESXi Driver for VMware vSphere ........46 Removing Earlier NVIDIA Drivers ............. 47 Firmware Programming ............... 47 Updating Adapter Firmware ..............48 Troubleshooting ................49 General Troubleshooting ................
- Page 4 MCX4411A-ACUN Board Label (Example) ............61 MCX4421A-ACUN Board Label (Example) ............61 MCX4431A-GCAN Board Label (Example) ............62 MCX4431M-GCAN Board Label (Example)............62 MCX4431A-GCUN Board Label (Example)............63 Document Revision History ..............64 ...
- Page 5 Project, spec 2.0. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. EOL'd (End of Life) Ordering Part Numbers The table below provides the ordering part numbers (OPN) for ConnectX-4 Lx Ethernet adapter cards for OCP Spec 2.0. NVIDIA SKU...
- Page 6 • URL: https://www.nvidia.com > Support • E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller. Related Documentation...
- Page 7 When discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to mean PCI Express.
-
Page 8: Introduction
The following provides the ordering part number, port speed, number of ports, and PCI Express speed. Each adapter is shipped with two bracket heights - short and tall (assembled). ConnectX-4 Lx 10Gb/s Ethernet Cards for OCP Spec 2.0 ConnectX-4 Lx 10GbE for OCP Spec 2.0... -
Page 9: Features And Benefits
ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0 RoHS RoHS Compliant IC Part Number MT27712A0-FDCF-AE Device ID 4117 for Physical Function (PF) 4118 for Virtual Function (VF) ConnectX-4 Lx 40/50 Gb/s Ethernet Adapter Cards Model ConnectX-4 Lx 40/50 Gb/s Ethernet Adapter Cards... - Page 10 VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-4 Lx effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.
-
Page 11: Operating Systems/Distributions
UEFI UEFI is a standard firmware interface designed to replace BIOS. NVIDIA UEFI Network driver allows boot over network via PXE (Preboot eXecution Environment). This network driver allows remote boot over InfiniBand or Ethernet, or Boot over iSCSI (Bo-iSCSI) in UEFI mode, and also supports the SecureBoot standard. -
Page 12: Connectivity
Connectivity • Interoperable with 1/10/25/40/50 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support... -
Page 13: Interfaces
Interfaces Ethernet SFP28 and QSFP28 Interfaces The network ports of the ConnectX®-4 Lx adapter card for OCP Spec 2.0 are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28 connectors on the adapter card. ... -
Page 14: Heatsink Interface
If LED1 is off, then there is no activity Heatsink Interface A heatsink is attached to the ConnectX-4 Lx IC in order to dissipate the heat from the ConnectX-4 Lx IC. It is attached either by using four spring-loaded push pins that insert into four mounting holes or by screws. -
Page 15: Hardware Installation
Hardware Installation Installation and initialization of ConnectX-4 Lx adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. For safety warnings in other... -
Page 16: Installation Procedure Overview
Airflow Requirements ConnectX-4 Lx adapter cards are offered with two airflow patterns: from the heatsink to the network ports, and vice versa, as shown below. Please refer to the Specifications section for airflow numbers for each specific card model. -
Page 17: Software Requirements
Software Requirements • See Operating Systems/Distributions section under the Introduction section. • Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal. Before opening the case of the system, observe the following precautions to avoid injury and prevent damage to system components. -
Page 18: Installation Instructions
Installation Instructions This section provides detailed instructions on how to install your adapter card in a system. Please note that the following figures are for illustration purposes only. Before inserting the card in a system, make sure that the system is off and the power cord is not connected to the server. -
Page 19: Cables And Modules
Applying even pressure on four corners of the card (as shown in the below picture), insert the adapter card into the PCI Express slot until firmly seated. Secure the adapter with the adapter clips or screws. Close the system case. ... -
Page 20: Identifying The Card In Your System
[ConnectX-4 On Windows Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. Expand System Devices and locate your NVIDIA ConnectX-4 Lx adapter card. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window. Click the Details tab and select Hardware Ids (Windows 2012/R2/2016) from the Property pull-down menu. -
Page 21: Uninstalling The Card
In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA, and DEV is equal to 1018 (for ConnectX-4 Lx) – this is a valid NVIDIA PCI Device ID. - Page 22 To remove the card, disengage clip 1 and 2 on connector A side. To disconnect connector A, gently pull the adapter card upwards. Disengage clip 3 and clip 4 on the adapter card. ...
- Page 23 6. To remove the card, gently pull the adapter card upwards.
-
Page 24: Driver Installation
This section describes how to install and test the MLNX_OFED for Linux package on a single server with a ConnectX-5 adapter card installed. Prerequisites Requirements Description Platforms A server platform with aN installed ConnectX-4 Lx adapter card: • MT27710 ConnectX®-4 Lx (firmware: fw-ConnectX4-Lx) Required Disk Space for Installation Operating System Linux operating system. -
Page 25: Installing Mlnx_Ofed
3. Use the Hash utility to confirm the file integrity of your ISO image. Run the following command and compare the result to the value provided on the download page. SHA256 MLNX_OFED_LINUX-<ver>-<OS label>.iso Installing MLNX_OFED Installation Script The installation script, mlnxofedinstall, performs the following: •... - Page 26 • If your kernel version does not match with any of the offered pre-built RPMs, you can add your kernel version by using the “mlnx_add_kernel_support.sh” script located inside the MLNX_OFED package. On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_kernel_support.sh script.
- Page 27 For the list of installation options, run: ./mlnxofedinstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards. Log in to the installation machine as root. Mount the ISO image on your machine. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Run the installation script.
- Page 28 FW XX.XX.XXXX Status: No matching image found Error message #2: The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor. 4. Case A: If the installation script has performed a firmware update on your network adapter, you need to either restart the driver or reboot your system before the firmware update can take effect.
- Page 29 In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
-
Page 30: Driver Load Upon System Boot
Logs dir: /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0.IBMM2150110033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically. To prevent the automatic load of the NVIDIA drivers upon system boot: Add the following lines to the "/etc/modprobe.d/mlnx.conf" file. blacklist mlx5_core blacklist mlx5_ib Set “ONBOOT=no”... -
Page 31: Additional Installation Procedures
In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."... - Page 32 Mount the ISO image on your machine and copy its content to a shared location in your network. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link: http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox...
- Page 33 # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Build the packages with kernel support and create the tarball. # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7.6 under /tmp directory. Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log ...
- Page 34 (User Space packages only where: mlnx-ofed-all Installs all available packages in MLNX_OFED mlnx-ofed-basic Installs basic packages required for running NVIDIA cards mlnx-ofed-guest Installs packages required by guest OS mlnx-ofed-hpc Installs packages required for HPC mlnx-ofed-hypervisor Installs packages required by hypervisor OS...
- Page 35 Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: deb file:/<path to extracted MLNX_OFED package>/DEBS ./ Download and install NVIDIA's Technologies GPG-KEY. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Verify that the key was successfully imported. ...
- Page 36 # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Build the packages with kernel support and create the tarball. # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7.6 under /tmp directory. Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log ...
-
Page 37: Performance Tuning
Depending on the application of the user's system, it may be necessary to modify the default configuration of network adapters based on the ConnectX® adapters. In case that tuning is required, please refer to the Performance Tuning Guide for NVIDIA Network Adapters. Windows Driver Installation... -
Page 38: Software Requirements
On an x64 (64-bit) machine, the output will be “AMD64”. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > InfiniBand Drivers (Learn More) > Nvidia WinOF-2. Download the .exe image according to the architecture of your machine (see Step 1). ... - Page 39 • Attended Installation An installation procedure that requires frequent user intervention. • Unattended Installation An automated installation procedure that requires no user intervention. Attended Installation The following is an example of an installation session. Double click the .exe and follow the GUI instructions to install MLNX_WinOF2. [Optional] Manually configure your setup to contain the logs option (replace “LogFile”...
- Page 40 Read and accept the license agreement and click Next. Select the target folder for the installation.
- Page 41 • If the user has a standard NVIDIA® card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA® card, only the NVIDIA® card will be updated.
- Page 42 Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes •...
- Page 43 Click Next to install the desired tools. Click Install to start the installation. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see ). ...
- Page 44 13. Click Finish to complete the installation.
- Page 45 Unattended Installation If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD.
-
Page 46: Firmware Upgrade
Firmware Upgrade If the machine has a standard NVIDIA® card with an older firmware version, the firmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to MFT User Manual. ... -
Page 47: Removing Earlier Nvidia Drivers
PartnerSupported 2017-01-31 After the installation process, all kernel modules are loaded automatically upon boot. Removing Earlier NVIDIA Drivers Please unload the previously installed drivers before removing them. To remove all the drivers: Log into the ESXi server with root permissions. -
Page 48: Updating Adapter Firmware
To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility. The utility can query for available NVIDIA adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images. -
Page 49: Troubleshooting
Troubleshooting General Troubleshooting • Ensure that the adapter is placed correctly Server unable to find the adapter • Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •... -
Page 50: Linux Troubleshooting
-d <mst_device> q ibstat Ports Information ibv_devinfo To download the latest firmware version, refer to Firmware Version Upgrade the NVIDIA Update and Query Utility. cat /var/log/messages Collect Log File dmesg >> system.log journalctl (Applicable on new operating systems) cat /var/log/syslog Windows Troubleshooting... -
Page 51: Specifications
Specifications MCX4421A-XCQN Specifications Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Physical Connector: Dual SFP28 (copper and optical) Ethernet: 10GBASE-SR, 10GBASE-LR, 10GBASE-ER, 10GBASE-CX4, SGMII, 1000BASE-X Protocol Support Data Rate: 1/10 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and Airflow Power... -
Page 52: Mcx4411A-Acan/Mcx4411A-Acun Specifications
MCX4411A-ACAN/MCX4411A-ACUN Specifications Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Physical Connector: Single SFP28 (copper and optical) Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, Protocol Support 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and Airflow Power... -
Page 53: Mcx4421A-Acan/Mcx4421A-Acun Specifications
MCX4421A-ACAN/MCX4421A-ACUN Specifications Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Physical Connector: Single SFP28 (copper and optical) Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, Protocol Support 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and Airflow Power... -
Page 54: Mcx4411A-Acqn Specifications
MCX4411A-ACQN Specifications Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Physical Connector: Single SFP28 (copper and optical) Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, Protocol Support 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and Airflow Power... -
Page 55: Mcx4421A-Acqn Specifications
MCX4421A-ACQN Specifications Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Physical Connector: Dual SFP28 (copper and optical) Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, Protocol Support 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and Airflow Power... -
Page 56: Mcx4431A-Gcan/Mcx4431A-Gcun Specifications
MCX4431A-GCAN/MCX4431A-GCUN Specifications Size: 2.67 in. x 4.33 in (68.00 mm x 110.05 mm) Physical Connector: Single QSFP28 (copper and optical) Ethernet: 50GBASE-R4, 50G-R2 (EC), 40GBASE-CR4, 40GBASE-KR4, 40GBASE-SR4, Protocol Support 40GBASE-LR4, 25GBASE-CR/CR-S, 25GBASE-SR, 25GBASE-LR, 25G-R (EC), 10GBASE-SR, 10GBASE-LR, 10GBASE-ER, 10GBASE-CX4, SGMII / 1000BASE-X Data Rate: 1/10/25/40/50 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX, 3.3V_AUX... -
Page 57: Mcx4431M-Gcan Specifications
MCX4431M-GCAN Specifications Size: 3.07 in. x 4.33 in (78.00 mm x 110.05 mm) Physical Connector: Single QSFP28 (copper and optical) Ethernet: 50GBASE-R4, 50G-R2 (EC), 40GBASE-CR4, 40GBASE-KR4, 40GBASE-SR4, Protocol Support 40GBASE-LR4, 25GBASE-CR/CR-S, 25GBASE-SR, 25GBASE-LR, 25G-R (EC), 10GBASE-SR, 10GBASE-LR, 10GBASE-ER, 10GBASE-CX4, SGMII / 1000BASE-X Data Rate: 1/10/25/40/50 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX, 3.3V_AUX... -
Page 58: Board Mechanical Drawing And Dimensions
Board Mechanical Drawing and Dimensions For the 3D Model of the card, please refer to 3D Models at http://www.nvidia.com/page/ 3d_models. All dimensions are in millimeters. The PCB mechanical tolerances is +/- 0.13mm. Single-port 10/25 Gb/s Cards Dual-port 10/25 Gb/s Cards... -
Page 59: Finding The Mac On The Adapter Card
Finding the MAC on the Adapter Card Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards. -
Page 60: Mcx4421A-Acan Board Label (Example)
MCX4421A-ACAN Board Label (Example) MCX4411A-ACQN Board Label (Example) MCX4421A-ACQN Board Label (Example) -
Page 61: Mcx4411A-Acun Board Label (Example)
MCX4411A-ACUN Board Label (Example) MCX4421A-ACUN Board Label (Example) -
Page 62: Mcx4431A-Gcan Board Label (Example)
MCX4431A-GCAN Board Label (Example) MCX4431M-GCAN Board Label (Example) -
Page 63: Mcx4431A-Gcun Board Label (Example)
MCX4431A-GCUN Board Label (Example) MCX4431A-GCUN ... -
Page 64: Document Revision History
Document Revision History Date Description of Changes May. 2023 Added non-operational storage temperature specifications. Nov. 2022 Updated Specifications. Feb. 2021 Added Standby Mode power numbers for passive cables for OPNs MCX4421A-XCQN. Nov. 2020 Added airflow for standby modes. Feb. 2020 Added altitude criteria to "Specifications". Jan. - Page 65 Date Description of Changes Dec. 2015 • Updated Product Overview • Updated speed specifications in the following sections: • Features and Benefits • Connectivity • Updated Specifications Dec. 2015 • Added mlxup to Documents List and Updating Adapter Card Firmware •...
- Page 66 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
- Page 67 Copyright © 2023 NVIDIA Corporation & affiliates. All Rights Reserved.