Page 5
About This Manual This User Manual describes NVIDIA® ConnectX®-4 Lx Ethernet adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. EOL'd (End of Life) Ordering Part Numbers The table below provides the ordering part numbers (OPN) for ConnectX-4 Lx Ethernet adapter cards.
Page 6
• URL: https://www.nvidia.com > Support • E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller. Related Documentation...
Introduction Product Overview This is the User Guide for Ethernet adapter cards based on the ConnectX®-4 Lx integrated circuit device. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0 servers used in Enterprise Data Centers and High- Performance Computing environments.
Uses PCIe Gen 3.0 (8GT/s) through an x8 edge connector. Gen 1.1 and 2.0 compatible. Up to 50 Gigabit Ethernet NVIDIA adapters comply with the following IEEE 802.3 standards: • 50GbE / 40GbE / 25GbE / 10GbE / 1GbE • IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes •...
Page 9
UEFI UEFI is a standard firmware interface designed to replace BIOS. NVIDIA UEFI Network driver allows boot over the network via PXE (Preboot eXecution Environment). This network driver allows remote boot over Ethernet, or Boot over iSCSI (Bo-iSCSI) in UEFI mode, and also supports the Secure Boot standard.
Feature Description Secure Firmware A “Secure firmware update” is the ability of a device to verify digital signatures of new firmware binaries, in order to assure that only officially approved versions can be installed from the host, the network or a Board Management Controller (BMC). The firmware of devices with “secure firmware update”...
Interfaces Ethernet SFP28 and QSFP28 Interfaces The network ports of the ConnectX®-4 Lx adapter card are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28 connectors on the adapter card. The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables.
BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.
Identify the ConnectX-4 Lx adapter card in the Refer to Identifying the Card in Your system. System System Requirements Hardware Requirements Unless otherwise specified, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination.
All cards in the system should be planned with the same airflow direction. Software Requirements • Operating Systems/Distributions section under the Introduction section. • Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. Unpacking the Product ...
Shut down your system if active: Turn off the power to the system, and disconnect the power cord. Refer to the system documentation for instructions. Before you install the ConnectX-4 Lx card, make sure that the system is disconnected from power. Bracket Replacement Instructions The card is usually shipped with an assembled high-profile bracket.
Place the bracket onto the card until the screw holes line up. Do not force the bracket onto the adapter card. Screw on the bracket using the screws saved from the bracket removal procedure above. Use a torque driver to apply up to 2 lbs-in torque on the screws. Installation Instructions This section provides detailed instructions on how to install your adapter card in a system.
Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. When the adapter is properly seated, the port connectors are aligned with the slot opening, and the adapter faceplate is visible against the system chassis. Secure the adapter with the adapter clip or screw.
[ConnectX-4 On Windows Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. Expand System Devices and locate your NVIDIA ConnectX-4 Lx adapter card. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window. Click the Details tab and select Hardware Ids (Windows 2012/R2/2016) from...
0x15B3 – this is the Vendor ID; and DEV is equal to 1018 (for ConnectX-4 Lx) – this is a valid PCI Device ID. If the PCI device does not have a NVIDIA adapter ID, return to Step 2 to check another device. ...
Page 20
Please note that the following images are for illustration purposes only. Verify that the system is powered off and unplugged. Wait 30 seconds. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). Holding the adapter card from its center, gently pull the adapter card out of the PCI Express slot.
The image’s name has the format . MLNX_OFED_LINUX-<ver>-<OS label><CPU arch>.iso You can download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the NVIDIA web site at nvidia.com/en-us/networking → Products → Software → InfiniBand Drivers → NVIDIA MLNX_OFED Scroll down to the Download wizard, and click the Download tab.
3. Use the Hash utility to confirm the file integrity of your ISO image. Run the following command and compare the result to the value provided on the download page. SHA256 MLNX_OFED_LINUX-<ver>-<OS label>.iso Installing MLNX_OFED Installation Script The installation script, mlnxofedinstall, performs the following: •...
Page 23
• If your kernel version does not match with any of the offered pre-built RPMs, you can add your kernel version by using the “mlnx_add_kernel_support.sh” script located inside the MLNX_OFED package. On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_kernel_support.sh script.
Page 24
For the list of installation options, run: ./mlnxofedinstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards. Log in to the installation machine as root. Mount the ISO image on your machine. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Run the installation script.
Page 25
FW XX.XX.XXXX Status: No matching image found Error message #2: The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor. 4. Case A: If the installation script has performed a firmware update on your network adapter, you need to either restart the driver or reboot your system before the firmware update can take effect.
Page 26
In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
Logs dir: /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0.IBMM2150110033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically. To prevent the automatic load of the NVIDIA drivers upon system boot: Add the following lines to the "/etc/modprobe.d/mlnx.conf" file. blacklist mlx5_core blacklist mlx5_ib Set “ONBOOT=no”...
In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
Page 29
Mount the ISO image on your machine and copy its content to a shared location in your network. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link: http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox...
Page 30
# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Build the packages with kernel support and create the tarball. # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7.6 under /tmp directory. Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log ...
Page 31
(User Space packages only where: mlnx-ofed-all Installs all available packages in MLNX_OFED mlnx-ofed-basic Installs basic packages required for running NVIDIA cards mlnx-ofed-guest Installs packages required by guest OS mlnx-ofed-hpc Installs packages required for HPC mlnx-ofed-hypervisor Installs packages required by hypervisor OS...
Page 32
Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: deb file:/<path to extracted MLNX_OFED package>/DEBS ./ Download and install NVIDIA's Technologies GPG-KEY. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Verify that the key was successfully imported. ...
Page 33
# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Build the packages with kernel support and create the tarball. # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7.6 under /tmp directory. Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log ...
Depending on the application of the user's system, it may be necessary to modify the default configuration of network adapters based on the ConnectX® adapters. In case that tuning is required, please refer to the Performance Tuning Guide for NVIDIA Network Adapters. Windows Driver Installation...
On an x64 (64-bit) machine, the output will be “AMD64”. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > InfiniBand Drivers (Learn More) > Nvidia WinOF-2. Download the .exe image according to the architecture of your machine (see Step 1). ...
Page 36
• Attended Installation An installation procedure that requires frequent user intervention. • Unattended Installation An automated installation procedure that requires no user intervention. Attended Installation The following is an example of an installation session. Double click the .exe and follow the GUI instructions to install MLNX_WinOF2. [Optional] Manually configure your setup to contain the logs option (replace “LogFile”...
Page 37
Read and accept the license agreement and click Next. Select the target folder for the installation.
Page 38
• If the user has a standard NVIDIA® card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA® card, only the NVIDIA® card will be updated.
Page 39
Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes •...
Page 40
Click Next to install the desired tools. Click Install to start the installation. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see ). ...
Page 41
13. Click Finish to complete the installation.
Page 42
Unattended Installation If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD.
Firmware Upgrade If the machine has a standard NVIDIA® card with an older firmware version, the firmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to MFT User Manual. ...
PartnerSupported 2017-01-31 After the installation process, all kernel modules are loaded automatically upon boot. Removing Earlier NVIDIA Drivers Please unload the previously installed drivers before removing them. To remove all the drivers: Log into the ESXi server with root permissions.
To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility. The utility can query for available NVIDIA adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images.
Troubleshooting General Troubleshooting • Ensure that the adapter is placed correctly Server unable to find the adapter • Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...
-d <mst_device> q ibstat Ports Information ibv_devinfo To download the latest firmware version, refer to Firmware Version Upgrade the NVIDIA Update and Query Utility. cat /var/log/messages Collect Log File dmesg >> system.log journalctl (Applicable on new operating systems) cat /var/log/syslog Windows Troubleshooting...
Specifications MCX4111A-XCAT Specifications Size: 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Physical Connector: Single SFP28 (copper and optical) Ethernet: 10GBASE-SR, 10GBASE-LR, 10GBASE-ER, 10GBASE-CX4, SGMII, 1000BASE-X Protocol Support Data Rate: 1/10 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V, 3.3V Power and Airflow...
Voltage: 12V, 3.3V Power and Airflow Power Cable Type Passive Cables 10.6W Typical Power Maximum Power Passive Cables 12.6W 1.5W Active Cables 14.2W Maximum power available through SFP28 port: 1.5W Airflow Airflow Specifications. Temperature Operational 0°C to 55°C Environmental Non-operational -40°C to 70°C Humidity Operational 10% to 85% relative...
Safety CB / cTUVus / CE Regulatory CE / FCC / VCCI / ICES / RCM RoHS RoHS compliant Notes: a. Typical power for ATIS traffic load. b. The non-operational storage temperature specifications apply to the product without its package. Airflow Specifications 10Gb/s Adapter Cards Airflow Airflow (LFM)
Board Mechanical Drawing and Dimensions All PSB dimensions are in millimeters. All the mechanical tolerances are +/- 0.13mm. Single-port 10/25 Gb/s Dual-port 10/25 Gb/s Single-port 40/50 Gb/s Bracket Mechanical Drawing All bracket dimensions are in millimeters. All the mechanical tolerances are +/- 0.25mm. Single-port Tall Bracket Dual-port Tall Bracket...
Finding the MAC on the Adapter Card Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. The product revisions indicated on the labels in the following figures do not necessarily...
Page 60
Document Revision History Date Description of Changes May. 2023 Added non-operational storage temperature specifications. Nov. 2020 Added MCX4121A-ACHT support across the document. Feb. 2020 Added altitude criteria to "Specifications". Oct. 2019 Updated Ethernet supported protocols in "Specifications". Sep. 2019 Added MCX4121A-XCHT support across the document. May.
Page 61
Date Description of Changes Dec. 2015 • Added mlxup to Documents List on page 9 and Updating Adapter Card Firmware • Updated Device ID in Product Overview • Added IEEE 802.3by support. See “Features and Benefits” • Updated PCI Device •...
Page 62
NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
Need help?
Do you have a question about the 900-9X4B0-0012-0T1 and is the answer not in the manual?
Questions and answers