Page 4
Mechanical Drawing of Single-port 50Gb/s Adapter Cards ......63 Mechanical Drawing of Multi-host Single-port 50Gb/s Adapter Cards ....64 Finding the MAC on the Adapter Card .............65 MCX4421A-XCQN Board Label (Example) ............65 MCX4411A-ACAN Board Label (Example) ............65 MCX4421A-ACAN Board Label (Example) ............66 MCX4411A-ACQN Board Label (Example) ............
Page 5
About This Manual This User Manual describes NVIDIA® ConnectX®-4 Lx Ethernet adapter cards for Open Compute Project, spec 2.0. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.
Page 6
• URL: https://www.nvidia.com > Support • E-mail: Network-support@nvidia.com Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller. Related Documentation...
Page 7
Document Conventions When discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to mean PCI Express.
Each adapter is shipped with two bracket heights - short and tall (assembled). ConnectX-4 Lx 10Gb/s Ethernet Cards for OCP Spec 2.0 ConnectX-4 Lx 10GbE for OCP Spec 2.0 Part Number MCX4421A-XCQN OCP Spec 2.0 Stacking Height Complying with OCP Type1 stacking height Network Connector Type...
Page 9
ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0 Ethernet Data Rate 1/10/25 Gb/s PCI Express Connectors PCIe Gen 3.0 SERDES @ 8.0GT/s Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Host √ √ Management UEFI √...
Uses PCIe Gen 3.0 (8GT/s) through an x8 edge connector. Gen 1.1 and 2.0 compatible. Up to 50 Gigabit Ethernet NVIDIA adapters comply with the following IEEE 802.3 standards: • 50GbE / 40GbE / 25GbE / 10GbE / 1GbE • IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes •...
Page 11
UEFI UEFI is a standard firmware interface designed to replace BIOS. NVIDIA UEFI Network driver allows boot over network via PXE (Preboot eXecution Environment). This network driver allows remote boot over InfiniBand or Ethernet, or Boot over iSCSI (Bo-iSCSI) in UEFI mode, and also supports the SecureBoot standard.
Feature Description Secure Firmware A “Secure firmware update” is the ability of a device to verify digital signatures of new firmware binaries, in order to assure that only officially approved versions can be installed from the host, the network or a Board Management Controller (BMC). The firmware of devices with “secure firmware update”...
Interfaces Ethernet SFP28 and QSFP28 Interfaces The network ports of the ConnectX®-4 Lx adapter card for OCP Spec 2.0 are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28 connectors on the adapter card. ...
BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.
Voltage Regulators The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These voltage supply pins feed onboard regulators that provide the necessary power to the various components on the card.
Hardware Installation Installation and initialization of ConnectX-4 Lx adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions.
Hardware Requirements Unless otherwise specified, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operation environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level.
Software Requirements • See Operating Systems/Distributions section under the Introduction section. • Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal. Before opening the case of the system, observe the following precautions to avoid injury and prevent damage to system components.
Installation Instructions This section provides detailed instructions on how to install your adapter card in a system. Please note that the following figures are for illustration purposes only. Before inserting the card in a system, make sure that the system is off and the power cord is not connected to the server.
To uninstall the adapter card, see Uninstalling the Card. Cables and Modules To obtain the list of supported NVIDIA cables for your adapter, please refer to the Cables Reference Table at http://www.nvidia.com/products/interconnect/cables-configurator.php. Cable Installation All cables can be inserted or removed with the unit powered on.
[ConnectX-4 On Windows Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. Expand System Devices and locate your NVIDIA ConnectX-4 Lx adapter card. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window. Click the Details tab and select Hardware Ids (Windows 2012/R2/2016) from the Property pull-down menu.
In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA, and DEV is equal to 1018 (for ConnectX-4 Lx) – this is a valid NVIDIA PCI Device ID.
Turn off the system and disconnect the power cord from the server. Card Removal Please note that the following images are for illustration purposes only. Verify that the system is powered off and unplugged. Wait 30 seconds. To remove the card, disengage clip 1 and 2 on connector A side. To disconnect connector A, gently pull the adapter card upwards. ...
Page 24
Disengage clip 3 and clip 4 on the adapter card. To remove the card, gently pull the adapter card upwards.
• Windows Driver Installation • VMware Driver Installation Linux Driver Installation This section describes how to install and test the NVIDIA OFED for Linux package on a single server with a NVIDIA ConnectX-5 adapter card installed. Prerequisites Requirements Description Platforms...
You will be prompted to acknowledge the deletion of the old packages. • If you need to install NVIDIA OFED on an entire (homogeneous) cluster, a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system...
Page 27
RAM. For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be running on one of the fabric nodes. At this point, NVIDIA OFED for Linux has already installed the OpenSM Subnet Manager on your machine.
support on RedHat or any other distribution, the initramfs will not be changed. Therefore, the inbox drivers may be loaded on boot. In this case, openibd service script will automatically unload them and load the new drivers that come with MLNX_OFED. Installation Procedure Login to the installation machine as root. ...
/etc/infiniband/info. Most of the NVIDIA OFED components can be configured or reconfigured after the installation, by modifying the relevant configuration files. See the relevant chapters in this manual for details.
OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh An example of OPENIBD_POST_START script for activating all interfaces is provided in the MLNX_OFED package under the docs/scripts/openibd-post-start-configure-interfaces/ folder. Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically.
Failed to start the mst driver Uninstalling MLNX_OFED Use the script /usr/sbin/ofed_uninstall.sh to uninstall the NVIDIA OFED package. The script is part of the ofed-scripts RPM. Installing MLNX_OFED Using YUM This type of installation is applicable to RedHat/OL, Fedora, XenServer Operating Systems.
Page 32
# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Download and install NVIDIA GPG-KEY: The key can be downloaded via the following link: http://www.nvidia.com/downloads/ofed/ RPM-GPG-KEY-Mellanox # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox --2014-04-20 13:52:30-- http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox Resolving www.mellanox.com... 72.3.194.0 Connecting to www.mellanox.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1354 (1.3K) [text/plain]...
Page 33
........qperf.x86_64 0:0.4.9-9 rds-devel.x86_64 0:2.0.7-1.12 rds-tools.x86_64 0:2.0.7-1.12 sdpnetstat.x86_64 0:1.60-26 srptools.x86_64 0:1.0.2-12 Complete! Uninstalling MLNX_OFED Using the YUM Tool Use the script /usr/sbin/ofed_uninstall.sh to uninstall the NVIDIA OFED package. The script is part of the ofed-scripts RPM.
Setting up MLNX_OFED apt-get Repository Log into the installation machine as root. Extract the MLNX_OFED pacakge on a shared location in your network. You can download it from http://www.nvidia.com > Products > Software> Ethernet Drivers. Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: # deb file:/<path to extracted MLNX_OFED package>/DEBS ./...
The firmware can be updated either manually or automatically (upon system boot), as described in the sections below. Updating the Device Online To update the device online on the machine from the NVIDIA site, use the following command line: mlxfwmanager --online -u -d <device> Example: mlxfwmanager --online -u -d 0000:09:00.0...
Support > Firmware Download). Get the device’s PSID. mlxfwmanager_pci | grep PSID PSID: MT_1210110019 Download the firmware BIN file from the NVIDIA website or the OEM website. Burn the firmware. mlxfwmanager_pci -i <fw_file.bin> Reboot your machine after the firmware burning is completed.
In order to support loading MLNX_OFED drivers when an OS supporting Secure Boot boots on a UEFI- based system with Secure Boot enabled, the NVIDIA x.509 public key should be added to the UEFI Secure Boot key database and loaded onto the system key ring by the kernel.
Windows Driver Installation For Windows, download and install the latest NVIDIA WinOF-2 for Windows software package available via the NVIDIA website at: http://www.nvidia.com > Products > Software > Ethernet Drivers > Download. Follow the installation instructions included in the download package (also available from the download page).
On an x64 (64-bit) machine, the output will be “AMD64”. Go to the NVIDIA WinOF-2 web page at http://www.nvidia.com > Products > Ethernet Drivers > Windows SW/Drivers. Download the exe image according to the architecture of your machine (see Step 1). The name of the .exe is in the following format: MLNX_WinOF2-<version>_x<arch>.exe.
WinOF-2 supports adapter cards based on the NVIDIA ConnectX®-4 and above family of adapter IC devices only. If you have ConnectX-3 and ConnectX-3 Pro on your server, you will need to install WinOF driver. For details on how to install WinOF driver, please refer to WinOF User Manual.
Page 41
If the user has an OEM card. In this case, the firmware will not be displayed. • If the user has a standard NVIDIA card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM...
Page 42
Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat...
Page 43
• Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd Click Next to install the desired tools. 9. Click Install to start the installation.
Page 44
10. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see ). 11. Click Finish to complete the installation.
Unattended Installation If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session.
Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. The inf files can be located at: %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\ To see the NVIDIA network adapters, display the Device Manager and pull down the “Network adapters” menu. Uninstalling NVIDIA WinOF-2 Driver Attended Uninstallation To uninstall MLNX_WinOF2 on a single node: ...
Unattended Uninstallation To uninstall MLNX_WinOF2 in unattended mode: Open a CMD console. (Click Task Manager > File > Run new task, and enter CMD.) To uninstall the driver, run: MLNX_WinOF2-2_0_All_x64.exe /S /x /v"/qn" Extracting Files Without Running Installation To extract the files without running installation, perform the following steps: Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter CMD.
Page 48
Click Change and specify the location in which the files are extracted to. Click Install to extract this folder, or click Change to install to a different folder. ...
Page 49
6. To complete the extraction, click Finish.
Firmware Upgrade If the machine has a standard NVIDIA card with an older firmware version, the firmware will be automatically updated as part of the WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to the MFT User Manual at www.nvidia.com >Products >...
2017-01-31 After the installation process, all kernel modules are loaded automatically upon boot. Removing Earlier NVIDIA Drivers Please unload the previously installed drivers before removing them. To remove all the drivers: Log into the ESXi server with root permissions.
To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility. The utility can query for available NVIDIA adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images.
Troubleshooting General Troubleshooting • Ensure that the adapter is placed correctly Server unable to find • Make sure the adapter slot and the adapter are compatible the adapter Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...
To export system information to a text file, choose the Export option from the File menu. Assign a file name and save. Download and install MFT: http://www.nvidia.com/content/ NVIDIA Firmware Tool (MFT) pages.php?pg=management_tools&menu_section=34 Refer to the MFT User Manual for installation instructions. Once installed, open a CMD window and run:...
MCX4411A-AC[A/U]N Specifications MCX4411A-ACAN/MCX4411A-ACUN Specifications Physical Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Connector: Single SFP28 (copper and optical) Protocol Support Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and Environmental...
MCX4421A-AC[A/U]N Specifications MCX4421A-ACAN/MCX4421A-ACUN Specifications Physical Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Connector: Single SFP28 (copper and optical) Protocol Support Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and...
MCX4411A-ACQN Specifications Physical Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Connector: Single SFP28 (copper and optical) Protocol Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, 10GBASE- Support LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and Environmental...
MCX4421A-ACQN Specifications Physical Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Connector: Dual SFP28 (copper and optical) Protocol Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, 10GBASE- Support LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 5V_AUX and 3.3V_AUX Power and Environmental...
Board Mechanical Drawing and Dimensions For the 3D Model of the card, please refer to 3D Models at http://www.nvidia.com/page/ 3d_models. All dimensions are in millimeters. The PCB mechanical tolerances is +/- 0.13mm. Mechanical Drawing of Single-port 10/25 Gb/s Adapter Cards...
Finding the MAC on the Adapter Card Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards.
Document Revision History Date Revision Description of Changes Feb. 2021 Added Standby Mode power numbers for passive cables for OPNs MCX4421A-XCQN. Nov. 2020 Added airflow for standby modes. Feb. 2020 Added altitude criteria to "Specifications". Jan. 2020 Updated the card voltage in "Specifications".
Page 71
Date Revision Description of Changes Mar. 2016 • Removed I2C-compatible Interface from Interfaces • Updated Features and Benefits Feb. 2016 Updated Bracket Installation Instructions Feb. 2016 • Updated Linux Driver • Updated Windows Driver • Updated VMware Driver • Added a note to Finding the MAC and Serial Number on the Adapter Card Dec.
Page 72
NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
Need help?
Do you have a question about the MCX4421A-XCQN and is the answer not in the manual?
Questions and answers