Nvidia Mellanox ConnectX-4 Lx User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Adapter
  5. Mellanox ConnectX-4 Lx
  6. User manual

Nvidia Mellanox ConnectX-4 Lx User Manual

Ethernet adapter cards
Hide thumbs Also See for Mellanox ConnectX-4 Lx:
Table of Contents

Advertisement

Quick Links

Download this manual
NVIDIA Mellanox ConnectX-4 Lx Ethernet
Adapter Cards User Manual
Exported on Dec/24/2020 12:18 PM
https://docs.mellanox.com/x/mYaO

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Mellanox ConnectX-4 Lx and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia Mellanox ConnectX-4 Lx

Summary of Contents for Nvidia Mellanox ConnectX-4 Lx

  • Page 1 NVIDIA Mellanox ConnectX-4 Lx Ethernet Adapter Cards User Manual Exported on Dec/24/2020 12:18 PM https://docs.mellanox.com/x/mYaO...

  • Page 2: Table Of Contents

    Table of Contents Introduction ......................... 6 Product Overview ........................ 6 Features and Benefits ......................7 Operating Systems/Distributions ..................9 Connectivity ........................10 Interfaces ........................11 Ethernet SFP28 and QSFP28 Interfaces ................11 PCI Express Interface ....................... 11 LED Interface ........................11 Heatsink Interface ......................
  • Page 3 Installing MLNX_OFED Using YUM ................27 Installing MLNX_OFED Using apt-get Tool ..............29 Updating Firmware After Installation ................31 UEFI Secure Boot ......................32 Performance Tuning ..................... 33 Windows Driver Installation ....................34 Software Requirements ....................34 Downloading Mellanox WinOF-2 Driver ................ 34 Installing Mellanox WinOF-2 Driver ................
  • Page 4 About This Manual This User Manual describes NVIDIA® Mellanox® ConnectX®-4 Lx Ethernet adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Ordering Part Numbers The table below provides the ordering part numbers (OPN) for the available ConnectX-4 Lx Ethernet adapter cards.
  • Page 5 Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller. Related Documentation User Manual describing OFED features, performance, band diagnostic, tools content Mellanox OFED for and configuration. See Mellanox OFED for Linux Documentation. Linux User Manual and Release Notes User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools...

  • Page 6: Introduction

    Introduction Product Overview This is the User Guide for Ethernet adapter cards based on the ConnectX®-4 Lx integrated circuit device. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0 servers used in Enterprise Data Centers and High- Performance Computing environments.

  • Page 7: Features And Benefits

    Model ConnectX-4 Lx 25Gb/s Ethernet Adapter Cards Secure Firmware Host Management RoHS RoHS Compliant IC Part MT27712A0-FDCF-AE Number Device ID 4117 for Physical Function (PF) 4118 for Virtual Function (VF) For more detailed information see Specifications. Features and Benefits This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability.
  • Page 8 Feature Description Up to 25 Gigabit Ethernet Mellanox adapters comply with the following IEEE 802.3 standards: • 25GbE / 10GbE / 1GbE • IEEE 802.3by, Ethernet Consortium25 Gigabit Ethernet, supporting all FEC modes • IEEE 802.3by 25 Gigabit Ethernet • IEEE 802.3ae 10 Gigabit Ethernet •...

  • Page 9: Operating Systems/Distributions

    Feature Description Storage Acceleration A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can leverage RDMA for high- performance storage access. • NVMe over Fabric offloads for the target machine ConnectX-4 Lx SR-IOV technology provides dedicated adapter SR-IOV resources and guaranteed isolation and protection for virtual...

  • Page 10: Connectivity

    Connectivity • Interoperable with 1/10/25 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support...

  • Page 11: Interfaces

    Interfaces Ethernet SFP28 and QSFP28 Interfaces The network ports of the ConnectX®-4 Lx adapter card are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28 connectors on the adapter card. The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables.

  • Page 12: Smbus Interface

    ConnectX-4 Lx IC has a thermal shutdown safety mechanism which automatically shuts down the ConnectX-4 Lx card in cases of high-temperature event, improper thermal coupling or heatsink removal. SMBus Interface ConnectX-4 Lx technology maintains support for manageability through a BMC. ConnectX-4 Lx PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand-up adapter.

  • Page 13: Hardware Installation

    Hardware Installation Installation and initialization of ConnectX-4 Lx adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions document available on Mellanox's website.

  • Page 14: System Requirements

    System Requirements Hardware Requirements Unless otherwise specified, Mellanox products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operation environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level.

  • Page 15: Bracket Replacement Instructions

    1. Unpack the adapter package and place them on an antistatic surface, and verify you have received the following items: a. ConnectX-4 Lx Adapter Card, with a tall bracket assembled on the card. b. Short bracket (accessory) 2. Check the parts for visible damage that may have occurred during shipping. Please note that if the card is removed hastily from the antistatic bag, the plastic ziplock may harm the EMI fingers on the networking connector.

  • Page 16: Installation Instructions

    2. Separate the bracket from the ConnectX-4 Lx card. Be careful not to put stress on the LEDs on the adapter card. 3. Save the two screws. Installing the New Bracket 1. Place the bracket onto the card until the screw holes line up. Do not force the bracket onto the adapter card.

  • Page 17: Cables And Modules

    4. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. 5. When the adapter is properly seated, the port connectors are aligned with the slot opening, and the adapter faceplate is visible against the system chassis. 6.

  • Page 18: Identifying The Card In Your System

    1. Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. 2. Expand System Devices and locate your Mellanox ConnectX-4 Lx adapter card. 3. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window.

  • Page 19: Uninstalling The Card

    PCI Device (Example) 5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1018 (for ConnectX-4 Lx) –...
  • Page 20 Please note that the following images are for illustration purposes only. 1. Verify that the system is powered off and unplugged. 2. Wait 30 seconds. 3. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). 4.

  • Page 21: Driver Installation

    Driver Installation Please use the relevant driver installation section. • Linux Driver Installation • Windows Driver Installation • VMware Driver Installation Linux Driver Installation This section describes how to install and test the Mellanox OFED for Linux package on a single server with a Mellanox ConnectX-5 adapter card installed.

  • Page 22: Installing Mellanox Ofed

    d. To obtain the download link, accept the End User License Agreement (EULA). 3. Use the md5sum utility to confirm the file integrity of your ISO image. Run the following command and compare the result to the value provided on the download page. md5sum MLNX_OFED_LINUX-<ver>-<OS label>.iso Installing Mellanox OFED Installation Script...
  • Page 23 On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_kernel_support.sh" is irrelevant on Ubuntu and Debian distributions. Example The following command will create a MLNX_OFED_LINUX ISO image for RedHat 6.3 under the / tmp directory.
  • Page 24 /mnt/mlnxofedinstall Logs dir: /tmp/MLNX_OFED_LINUX-x.x-x.logs This program will install the MLNX_OFED_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them. Starting MLNX_OFED_LINUX-x.x.x installation ...
  • Page 25 In case your machine has an unsupported network adapter device, no firmware update will occur and one of the following error messages below will be printed. Please contact your hardware vendor for help on firmware updates. Error message 1: Device #1: Device Type: ConnectX4-Lx Part Number:...
  • Page 26 Firmware • The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled: • The installation script is run in default mode; that is, without the option ‘--without-fw-update’ • The firmware version of the adapter device is older than the firmware version included with the Mellanox OFED ISO image Note: If an adapter’s flash was originally programmed with an Expansion ROM image, the automatic firmware update will also...

  • Page 27: Installing Mlnx_Ofed Using Yum

    blacklist mlx4_core blacklist mlx4_en blacklist mlx5_core blacklist mlx5_ib 2. Set “ONBOOT=no” in the "/etc/infiniband/openib.conf" file. 3. If the modules exist in the initramfs file, they can automatically be loaded by the kernel. To prevent this behavior, update the initramfs using the operating systems’ standard tools. Note: The process of updating the initramfs will add the blacklists from step 1, and will prevent the kernel from loading the modules automatically.
  • Page 28 # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox --2014-04-20 13:52:30-- http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox Resolving www.mellanox.com... 72.3.194.0 Connecting to www.mellanox.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1354 (1.3K) [text/plain] Saving to: ?RPM-GPG-KEY-Mellanox? 100%[=================================================>] 1,354 --.-K/s in 0s 2014-04-20 13:52:30 (247 MB/s) - ?RPM-GPG-KEY-Mellanox? saved [1354/1354] 4.

  • Page 29: Installing Mlnx_Ofed Using Apt-Get Tool

    mlnx-ofed-all Installs all available packages in MLNX_OFED. mlnx-ofed-basic Installs basic packages required running Mellanox cards. mlnx-ofed-guest Installs packages required by guest OS. mlnx-ofed-hpc Installs packages required HPC. mlnx-ofed-hypervisor Installs packages required by hypervisor OS. mlnx-ofed-vma Installs packages required by VMA. mlnx-ofed-vma-eth Installs packages required by VMA to work over Ethernet.
  • Page 30 Setting up MLNX_OFED apt-get Repository 1. Log into the installation machine as root. 2. Extract the MLNX_OFED pacakge on a shared location in your network. You can download it from http://www.mellanox.com > Products > Software> Ethernet Drivers. 3. Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: # deb file:/<path to extracted MLNX_OFED package>/DEBS ./ 4.

  • Page 31: Updating Firmware After Installation

    Installing MLNX_OFED using the “apt-get” tool does not automatically update the firmware. To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Or, update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Updating Adapter Firmware.

  • Page 32: Uefi Secure Boot

    2. Download the firmware BIN file from the Mellanox website or the OEM website. 3. Burn the firmware. mlxfwmanager_pci -i <fw_file.bin> 4. Reboot your machine after the firmware burning is completed. Updating the Device Firmware Automatically upon System Boot As of MLNX_OFED v3.1-x.x.x, firmware can be automatically updated upon system boot. The firmware update package (mlnx-fw-updater) is installed in the “/opt/mellanox/mlnx-fw-updater”...

  • Page 33: Performance Tuning

    2. Add the public key to the MOK list using the mokutil utility. # mokutil --import mlnx_signing_key_pub.der 3. Reboot the system. The pending MOK key enrollment request will be noticed by shim.efi and it will launch MokManager.efi to allow you to complete the enrollment from the UEFI console. You will need to enter the password you previously associated with this request and confirm the enrollment.

  • Page 34: Windows Driver Installation

    please refer to the Performance Tuning Guide for Mellanox Network Adapters https:// community.mellanox.com/docs/DOC-2489. Windows Driver Installation For Windows, download and install the latest Mellanox WinOF-2 for Windows software package available via the Mellanox web site at: http://www.mellanox.com > Products > Software > Ethernet Drivers >...

  • Page 35: Installing Mellanox Winof-2 Driver

    Installing the incorrect exe file is prohibited. If you do so, an error message will be displayed. For example, if you try to install a 64-bit executable file on a 32-bit machine, the wizard will display the following (or a similar) error message: “The installation package is not supported by this processor type.
  • Page 36 4. Click Next in the Welcome screen. 5. Read and accept the license agreement and click Next. 6. Select the target folder for the installation.
  • Page 37 7. The firmware upgrade screen will be displayed in the following cases: • If the user has an OEM card. In this case, the firmware will not be displayed. • If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly.
  • Page 38 8. Select a Complete or Custom installation, follow Step a onward. a. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes •...
  • Page 39 9. Click Install to start the installation. 10. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see 11. Click Finish to complete the installation.
  • Page 40 Unattended Installation If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. 1.

  • Page 41: Uninstalling Mellanox Winof-2 Driver

    Installation Results Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. The inf files can be located at: %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\ To see the Mellanox network adapters, display the Device Manager and pull down the “Network adapters”...

  • Page 42: Extracting Files Without Running Installation

    1. Open a CMD console. (Click Task Manager > File > Run new task, and enter CMD.) 2. To uninstall the driver, run: MLNX_WinOF2-2_0_All_x64.exe /S /x /v"/qn" Extracting Files Without Running Installation To extract the files without running installation, perform the following steps: 1.
  • Page 43 5. Click Install to extract this folder, or click Change to install to a different folder.

  • Page 44: Firmware Upgrade

    6. To complete the extraction, click Finish. Firmware Upgrade If the machine has a standard Mellanox card with an older firmware version, the firmware will be automatically updated as part of the WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to the MFT User Manual at www.m ellanox.com >Products >...

  • Page 45: Installing Mellanox Native Esxi Driver For Vmware Vsphere

    Installing Mellanox NATIVE ESXi Driver for VMware vSphere Please uninstall all previous Mellanox driver packages prior to installing the new version. See Removing Earlier Mellanox Drivers for further information. To install the driver: 1. Log into the ESXi server with root permissions. 2.

  • Page 46: Firmware Programming

    Firmware Programming 1. Download the VMware bootable binary images v4.6.0 from the Mellanox Firmware Tools (MFT) site. a. ESXi 6.5 File: mft-4.6.0.48-10EM-650.0.0.4598673.x86_64.vib b. MD5SUM: 0804cffe30913a7b4017445a0f0adbe1 2. Install the image according to the steps described in the MFT User Manual. The following procedure requires custom boot image downloading, mounting and booting from a USB device.

  • Page 47: Updating Adapter Firmware

    Updating Adapter Firmware Each adapter card is shipped with the latest version of qualified firmware at the time of manufacturing. However, Mellanox issues firmware updates occasionally that provide new features and bug fixes. To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility.

  • Page 48: Troubleshooting

    Troubleshooting General Troubleshooting • Ensure that the adapter is placed correctly Server unable to find • Make sure the adapter slot and the adapter are compatible the adapter Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...

  • Page 49: Windows Troubleshooting

    ibstat Ports Information ibv_devinfo To download the latest firmware version refer to http:// Firmware Version Upgrade www.mellanox.com/supportdownloader cat /var/log/messages Collect Log File dmesg >> system.log journalctl (Applicable on new operating systems) cat /var/log/syslog Windows Troubleshooting From the Windows desktop choose the Start menu and run: Environment Information msinfo32 To export system information to a text file, choose the Export...

  • Page 50: Specifications

    Specifications MCX4121A-XCAT Specifications Size: 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Physical Connector: Single SFP28 (copper and optical) Ethernet: 10GBASE-SR, 10GBASE-LR, 10GBASE-ER, 10GBASE-CX4, SGMII, Protocol Support 1000BASE-X Data Rate: 1/10 Gb/s Ethernet...

  • Page 51: Mcx4121A-Acat Specifications

    PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V, 3.3V Power Power Cable Type Passive Cables 9.3W Typical Power Maximum Power Passive Cables 11.1W 1W Active Cables 13.3W Maximum power available through SFP28 port: 1.5W Temperature Operational 0°C to 55°C...
  • Page 52 Airflow Airflow Specifications. Safety CB / cTUVus / CE Regulatory CE / FCC / VCCI / ICES / RCM RoHS RoHS compliant Notes: a. Typical power for ATIS traffic load. b. For both operational and non-operational states.

  • Page 53: Airflow Specifications

    Airflow Specifications 10Gb/s Adapter Cards Airflow Airflow (LFM) Airflow Direction Heatsink to Port Port to Heatsink Passive Active 1W Passive Active 1W MCX4121A-XCAT 25Gb/s Adapter Cards Airflow Airflow (LFM) Airflow Direction Heatsink to Port Port to Heatsink Passive Active 1.5W Passive Active 1.5W MCX4121A-ACAT...

  • Page 54: Board Mechanical Drawing And Dimensions

    Board Mechanical Drawing and Dimensions All PSB dimensions are in millimeters. All the mechanical tolerances are +/- 0.13mm. Mechanical Drawing of Dual-port 10/25 Gb/s Adapter Card...

  • Page 55: Bracket Mechanical Drawing

    For the 3D Model of the card, please refer to 3D Models at http://www.mellanox.com/page/ 3d_models. Bracket Mechanical Drawing All bracket dimensions are in millimeters. All the mechanical tolerances are +/- 0.25mm. Dimensions in millimeters.

  • Page 56: Dual-Port Tall Bracket

    Dual-port Tall Bracket...

  • Page 57: Finding The Mac On The Adapter Card

    Finding the MAC on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards.

  • Page 58: Mcx4121A-Acat Board Label (Example)

    MCX4121A-ACAT Board Label (Example)

  • Page 59: Document Revision History

    Document Revision History Date Revisi Description of Changes Feb. 2020 2.7 Added altitude criteria to "Specifications". Oct. 2019 Updated Ethernet supported protocols in "Specifications". May. 2019 2.4 Updated bracket and PCB tolerance in "Specifications". May. 2019 2.3 Updated board labels and added a note to "Finding the MAC and Serial Number on the Adapter Card"...
  • Page 60 Date Revisi Description of Changes Dec. 2015 1.1 • Added mlxup to Documents List on page 9 and Updating Adapter Card Firmware • Updated Device ID in Product Overview • Added IEEE 802.3by support. See “Features and Benefits” • Updated PCI Device •...
  • Page 61 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
  • Page 62 NVIDIA, the NVIDIA logo, and Mellanox are trademarks and/or registered trademarks of Mellanox Technologies Ltd. and/or NVIDIA Corporation in the U.S. and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated.

This manual is also suitable for:

Mcx4121a-xcatMcx4121a-acat

Table of Contents