Subscribe to Our Youtube Channel
Related Manuals for Mellanox Technologies ConnectX-4 Lx
Summary of Contents for Mellanox Technologies ConnectX-4 Lx
- Page 1 ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0 User Manual Exported on Apr/02/2019 03:29 PM Mellanox Technologies https://docs.mellanox.com/x/pI6O...
- Page 2 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT.
-
Page 3: Table Of Contents
Table of Contents Introduction to ConnectX-4 Lx Ethernet for OCP Spec 2.0 Cards ............9 Product Overview ........................... 9 Features and Benefits ........................11 Operating Systems/Distributions ....................13 Connectivity..........................13 Interfaces............................14 Ethernet SFP28 and QSFP28 Interfaces ..................14 PCI Express Interface ........................14 FRU EEPROM .......................... - Page 4 VMware Driver Installation ......................49 4.3.1 Hardware and Software Requirements ................... 49 4.3.2 Installing Mellanox NATIVE ESXi Driver for VMware vSphere..........50 4.3.3 Removing Earlier Mellanox Drivers ..................50 4.3.4 Firmware Programming......................50 Updating Adapter Firmware........................52 Troubleshooting..........................53 General Troubleshooting ......................53 Mellanox Technologies...
- Page 5 MCX4421A-ACAN Board Label (Example) .................. 65 MCX4411A-ACQN Board Label (Example).................. 65 MCX4421A-ACQN Board Label (Example).................. 65 MCX4411A-ACUN Board Label (Example) .................. 66 MCX4421A-ACUN Board Label (Example).................. 66 MCX4431A-GCAN Board Label (Example).................. 67 MCX4431M-GCAN Board Label (Example) ................. 67 Document Revision History ........................68 Mellanox Technologies...
- Page 6 Project, spec 2.0. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Ordering Part Numbers The table below provides the ordering part numbers (OPN) for the available ConnectX-4 Lx Ethernet adapter cards for OCP Spec 2.0. Marketing Description...
- Page 7 Mellanox Firmware Tools (MFT) User Manual User Manual describing the set of MFT firmware management tools for a single node. See http:// www.mellanox.com > Products > Software > Firmware Tools IEEE Std 802.3 Specification IEEE Ethernet specification at https://standards.ieee.org/ Mellanox Technologies...
- Page 8 When discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to mean PCI Express. Mellanox Technologies...
-
Page 9: Introduction To Connectx-4 Lx Ethernet For Ocp Spec 2.0 Cards
1 Introduction to ConnectX-4 Lx Ethernet for OCP Spec 2.0 Cards 1.1 Product Overview This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the ConnectX®-4 Lx integrated circuit device. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0 servers used in Enterprise Data Centers and High-Performance Computing... - Page 10 ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0 Ethernet 1/10/25 Gb/s Data Rate PCI Express PCIe Gen 3.0 SERDES @ 8.0GT/s Connectors Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Host √ √ Management UEFI √...
-
Page 11: Features And Benefits
VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-4 Lx effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol. - Page 12 PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-4 Lx advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes.
-
Page 13: Operating Systems/Distributions
• FreeBSD • VMware • OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) 1.4 Connectivity • Interoperable with 1/10/25/40/50 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Mellanox Technologies... -
Page 14: Interfaces
The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables. 2.2 PCI Express Interface The table below describes the supported PCIe interface in ConnectX-4 Lx and ConnectX-4 Lx Ex adapter cards. Supported PCIe Interface Features PCIe Gen 3.0 (1.1 and 2.0 compatible) through x8 edge connectors... -
Page 15: Heat Sink Interface
2.5 Heat Sink Interface A heatsink is attached to the ConnectX-4 Lx IC in order to dissipate the heat from the ConnectX-4 Lx IC. It is attached either by using four spring-loaded push pins that insert into four mounting holes or by screws. -
Page 16: Hardware Installation
3 Hardware Installation Installation and initialization of ConnectX-4 Lx adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions document available on mellanox.com. -
Page 17: Installation Procedure Overview
Refer to the Specifications section of the manual for more power requirements. 3.2.1.1 Airflow Requirements ConnectX-4 Lx adapter cards are offered with two airflow patterns: from the heatsink to the network ports, and vice versa, as shown below. Mellanox Technologies... -
Page 18: Software Requirements
• Shut down your system if active; Turn off the power to the system, and disconnect the power cord. Refer to the system documentation for instructions. Before you install the ConnectX-4 Lx card, make sure that the system is disconnected from power. -
Page 19: Cables And Modules
To obtain the list of supported Mellanox cables for your adapter, please refer to the Cables Reference Table at http:// www.mellanox.com/products/interconnect/cables-configurator.php. Cable Installation 1. All cables can be inserted or removed with the unit powered on. 2. To insert a cable, press the connector into the port receptacle until the connector is firmly seated. Mellanox Technologies... -
Page 20: Identifying The Card In Your System
LED indicator will turn off when the cable is unseated. 3.7 Identifying the Card in Your System On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mellanox Technologies”: lspci |grep -i Mellanox Network controller: Mellanox Technologies MT27710 Family... -
Page 21: Uninstalling The Card
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies, and DEV is equal to 1018 (for ConnectX-4 Lx) – this is a valid Mellanox Technologies PCI Device ID. - Page 22 3. To remove the card, disengage clip 1 and 2 on connector A side. 4. To disconnect connector A, gently pull the adapter card upwards. 5. Disengage clip 3 and clip 4 on the adapter card. Mellanox Technologies...
- Page 23 6. To remove the card, gently pull the adapter card upwards. Mellanox Technologies...
-
Page 24: Driver Installation
ConnectX-5 adapter card installed. 4.1.1 Prerequisites Requirements Description Platforms A server platform with a ConnectX-4 Lx adapter card installed • MT27710 ConnectX®-4 Lx (firmware: fw-ConnectX4-Lx) Required Disk Space for Installation Device ID For the latest list of device IDs, please visit the Mellanox website at http://www.mellanox.com/page/... -
Page 25: Installing Mellanox Ofed
The regular installation can be performed and weak updates mechanism will create symbolic links to the MLNX_OFED kernel modules. The “mlnx_add_kernel_support.sh” script can be executed directly from the mlnxofedinstall script. For further information, please see '--add-kernel-support' option below. Mellanox Technologies... -
Page 26: Installation Procedure
In this case, openibd service script will automatically unload them and load the new drivers that come with MLNX_OFED. 4.1.3.2 Installation Procedure 1. Login to the installation machine as root. 2. Mount the ISO image on your machine. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt 3. Run the installation script. Mellanox Technologies... - Page 27 ConnectX-6 Part Number: MCX653105A-HDAT Description: ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE, single-port QSFP56, PCIe4.0 x16, tall bracket PSID: MT_2190110032 PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d5cf810 Versions: Current Available 16.22.0228 16.22.0228 Status: Up to date Mellanox Technologies...
-
Page 28: Installation Results
“/opt” directory: • fca and ibutils • The kernel modules are installed under • /lib/modules/`uname -r`/updates on SLES and Fedora Distributions • /lib/modules/`uname -r`/extra/mlnx-ofa_kernel on RHEL and other Red Hat like Distributions Mellanox Technologies... -
Page 29: Installation Logs
An example of OPENIBD_POST_START script for activating all interfaces is provided in the MLNX_OFED package under the docs/scripts/openibd-post-start-configure-interfaces/ folder. 4.1.3.6 Driver Load Upon System Boot Upon system boot, the Mellanox drivers will be loaded automatically. To prevent automatic load of the Mellanox drivers upon system boot: Mellanox Technologies... -
Page 30: Mlnxofedinstall Return Codes
This type of installation is applicable to RedHat/OL, Fedora, XenServer Operating Systems. 4.1.4.1 Setting up MLNX_OFED YUM Repository 1. Log into the installation machine as root. 2. Mount the ISO image on your machine and copy its content to a shared location in your network. Mellanox Technologies... - Page 31 Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY- Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mellanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Is this ok [y/N]: 5. Check that the key was successfully imported.
-
Page 32: Installing Mlnx_Ofed Using The Yum Tool
Note: MLNX_OFED provides kernel module RPM packages with KMP support for RHEL and SLES. For other operating systems, kernel module RPM packages are provided only for the operating systems' default kernel. In this case, the group RPM packages have the supported kernel version in their package's name. Example: Mellanox Technologies... -
Page 33: Uninstalling Mlnx_Ofed Using The Yum Tool
4.1.4.3 Uninstalling MLNX_OFED Using the YUM Tool Use the script /usr/sbin/ofed_uninstall.sh to uninstall the Mellanox OFED package. The script is part of the ofed-scripts RPM. 4.1.5 Installing MLNX_OFED Using apt-get Tool This type of installation is applicable to Debian and Ubuntu operating systems. Mellanox Technologies... -
Page 34: Setting Up Mlnx_Ofed Apt-Get Repository
> Products > Software> Ethernet Drivers. 3. Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: # deb file:/<path to extracted MLNX_OFED package>/DEBS ./ 4. Download and install Mellanox Technologies GPG-KEY. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - 5. -
Page 35: Uninstalling Mlnx_Ofed Using The Apt-Get Tool
Installing MLNX_OFED using the “apt-get” tool does not automatically update the firmware. To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Or, update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Updating Adapter Firmware. -
Page 36: Updating The Device Manually
(manually) update the firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case that you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http://www.mellanox.com >... -
Page 37: Uefi Secure Boot
# rpm -qa | grep -E "kernel-ib|mlnx-ofa_kernel|iser|srp|knem|mlnx-rds|mlnx- nfsrdma|mlnx-nvme|mlnx-rdma-rxe" | xargs rpm -ql | grep "\.ko$" | xargs strip - After the signature has been removed, a message as the below will no longer be presented upon module loading: Mellanox Technologies... -
Page 38: Performance Tuning
"Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" However, please note that a similar message as the following will still be presented: "my_module: module verification failed: signature and/or required key missing - tainting kernel" This message is only presented once, upon first module boot that either has no signature or whose key is not in the kernel key ring. -
Page 39: Downloading Mellanox Winof-2 Driver
The following is an example of an installation session. 1. Double click the .exe and follow the GUI instructions to install MLNX_WinOF2. 2. [Optional] Manually configure your setup to contain the logs option (replace “LogFile” with the relevant directory): Mellanox Technologies... - Page 40 3. [Optional] If you do not want to upgrade your firmware version. (Note: MT_SKIPFWUPGRD default value is False.) MLNX_WinOF2-2_10_50000_All_x64.exe /v"/l*vx MyLog.txt=1" 4. Click Next in the Welcome screen. 5. Read and accept the license agreement and click Next. Mellanox Technologies...
- Page 41 • If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a Mellanox card, only the Mellanox card will be updated. Mellanox Technologies...
- Page 42 • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat • Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd Mellanox Technologies...
- Page 43 Click Next to install the desired tools. 9. Click Install to start the installation. 10. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see Mellanox Technologies...
-
Page 44: Unattended Installation
If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. Mellanox Technologies... -
Page 45: Installation Results
5. [Optional] If you do not wish to upgrade your firmware version (i.e., MT_SKIPFWUPGRD default value is False ). MLNX_WinOF2-[Driver/Version]_<revision_version>_All_Arch.exe /vMT_SKIPFWUPGRD=1 4.2.3.3 Installation Results Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. The inf files can be located at: %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\ Mellanox Technologies... -
Page 46: Uninstalling Mellanox Winof-2 Driver
1. Open a CMD console. (Click Task Manager > File > Run new task, and enter CMD.) 2. To uninstall the driver, run: MLNX_WinOF2-2_0_All_x64.exe /S /x /v"/qn" 4.2.5 Extracting Files Without Running Installation To extract the files without running installation, perform the following steps: Mellanox Technologies... - Page 47 2. Extract the driver and the tools: MLNX_WinOF2-2_0_<revision_version>_All_x64 /a To extract only the driver file MLNX_WinOF2-2_0_<revision_version>_All_x64 /a /vMT_DRIVERS_ONLY=1 3. Click Next to create a server image. 4. Click Change and specify the location in which the files are extracted to. Mellanox Technologies...
- Page 48 5. Click Install to extract this folder, or click Change to install to a different folder. Mellanox Technologies...
-
Page 49: Firmware Upgrade
4.3.1 Hardware and Software Requirements Requirement Description Platforms A server platform with an adapter card based on one of the following Mellanox Technologies’ devices: • ConnectX®-4 Lx (EN) (firmware:fw-ConnectX4Lx) Device ID For the latest list of device IDs, please visit Mellanox website. Operating System ESXi 6.5... -
Page 50: Installing Mellanox Native Esxi Driver For Vmware Vsphere
To remove the modules, you must run the command in the same order as shown in the example above. d. Reboot the server. 4.3.4 Firmware Programming a. Download the VMware bootable binary images v4.6.0 from the Mellanox Firmware Tools (MFT) site. Mellanox Technologies... - Page 51 ESXi 6.5 File: mft-4.6.0.48-10EM-650.0.0.4598673.x86_64.vib ii. MD5SUM: 0804cffe30913a7b4017445a0f0adbe1 b. Install the image according to the steps described in the MFT User Manual. The following procedure requires custom boot image downloading, mounting and booting from a USB device. Mellanox Technologies...
-
Page 52: Updating Adapter Firmware
Firmware Update Example [server1]# ./mlxup Querying Mellanox devices firmware ... Device Type: ConnectX-4 Lx Part Number: MCX4431A-GCAN Description: ConnectX-4 Lx EN adapter for OCP; 50GbE single-port QSFP28; PCIe3.0 x8; no PSID: MT_2190110032 PCI Device Name: 0000:06:00.0 Base GUID: e41d2d0300fd8b8a Versions:... -
Page 53: Troubleshooting
6.2 Linux Troubleshooting Environment Information cat /etc/issue uname -a cat /proc/cupinfo | grep ‘model name’ | uniq ofed_info -s ifconfig -a ip link show ethtool <interface> ethtool -i <interface_of_Mellanox_port_num> ibdev2netdev Card Detection lspci | grep -i Mellanox Mellanox Technologies... -
Page 54: Windows Troubleshooting
Download the latest firmware version using the PSID/board http://www.mellanox.com/supportdownloader/ flint –d <mst_device> –i <firmware_bin_file> b Collect Log File • Event log viewer • MST device logs: • mst start • mst status • flint –d <mst_device> dc > dump_configuration.log • mstdump <mst_device> dc > mstdump.log Mellanox Technologies... -
Page 55: Specifications
For both operational and non-operational states. c. Simulated inside an air tunnel (15.2 mm high and 73 mm wide). Airflow direction - port to heatsink. 7.2 MCX4411A-AC[A/U]N Specifications Physical Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Mellanox Technologies... -
Page 56: Mcx4421A-Ac[A/U]N Specifications
Simulated inside an air tunnel (15.2 mm high and 73 mm wide). Airflow direction - port to heatsink. 7.3 MCX4421A-AC[A/U]N Specifications Physical Size: 2.67 in. x 4.33 in. (68.00 mm x 110.05 mm) Connector: Single SFP28 (copper and optical) Protocol Support Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE- SR, 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-KX4, 10GBASE-CR, 10GBASE-KR, SGMII Mellanox Technologies... -
Page 57: Mcx4411A-Acqn Specifications
Connector: Single SFP28 (copper and optical) Protocol Support Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE- SR, 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-KX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Mellanox Technologies... -
Page 58: Mcx4421A-Acqn Specifications
SR, 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-KX4, 10GBASE-CR, 10GBASE-KR, SGMII Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Power and Environmental Voltage: 12V, 3.3V Power Cable Type Passive Cables 9.01W Typical Power Mellanox Technologies... -
Page 59: Mcx4431A-Gcan Specifications
PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Power and Voltage: 5V_AUX, 3.3V_AUX Environmental Power Cable Type Passive Cables 9.07W Typical Power Maximum Power Passive Cables 10.82W 1.5W Active Cables 14.32W Maximum power available through QSFP28 port: 3.5W Mellanox Technologies... -
Page 60: Mcx4431M-Gcan Specifications
Power and Voltage: 5V_AUX, 3.3V_AUX Environmental Power Cable Type Passive Cables 11.2W Typical Power Maximum Power Passive Cables 13.08W 1.5W Active Cables 16.58W Maximum power available through QSFP28 port: 3.5W Temperature Operational 0°C to 35°C Non-operational -40°C to 70°C Mellanox Technologies... -
Page 61: Board Mechanical Drawing And Dimensions
7.8 Board Mechanical Drawing and Dimensions For the 3D Model of the card, please refer to 3D Models at http://www.mellanox.com/page/3d_models. All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. 7.8.1 Mechanical Drawing of Single-port 10/25 Gb/s Adapter Cards Mellanox Technologies... -
Page 62: Mechanical Drawing Of Dual-Port 10/25 Gb/S Adapter Cards
7.8.2 Mechanical Drawing of Dual-port 10/25 Gb/s Adapter Cards 7.8.3 Mechanical Drawing of Single-port 50Gb/s Adapter Cards Mellanox Technologies... -
Page 63: Mechanical Drawing Of Multi-Host Single-Port 50Gb/S Adapter Cards
7.8.4 Mechanical Drawing of Multi-host Single-port 50Gb/s Adapter Cards Mellanox Technologies... -
Page 64: Finding The Mac On The Adapter Card
Each Mellanox adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards. 8.1 MCX4421A-XCQN Board Label (Example) 8.2 MCX4411A-ACAN Board Label (Example) Mellanox Technologies... -
Page 65: Mcx4421A-Acan Board Label (Example)
8.3 MCX4421A-ACAN Board Label (Example) 8.4 MCX4411A-ACQN Board Label (Example) 8.5 MCX4421A-ACQN Board Label (Example) Mellanox Technologies... -
Page 66: Mcx4411A-Acun Board Label (Example)
8.6 MCX4411A-ACUN Board Label (Example) 8.7 MCX4421A-ACUN Board Label (Example) Mellanox Technologies... -
Page 67: Mcx4431A-Gcan Board Label (Example)
8.8 MCX4431A-GCAN Board Label (Example) 8.9 MCX4431M-GCAN Board Label (Example) Mellanox Technologies... -
Page 68: Document Revision History
• Updated Features and Benefits Feb. Updated Bracket Installation Instructions 2016 Feb. • Updated Linux Driver 2016 • Updated Windows Driver • Updated VMware Driver • Added a note to Finding the MAC and Serial Number on the Adapter Card Mellanox Technologies... - Page 69 • Added mlxup to Documents List and Updating Adapter Card Firmware 2015 • Updated Device ID in Product Overview • Added IEEE 802.3by support. See “Features and Benefits” • Updated PCI Device • Added VMware support • Updated Specifications • Added Single-port Bracket Nov. First release 2015 Mellanox Technologies...
Need help?
Do you have a question about the ConnectX-4 Lx and is the answer not in the manual?
Questions and answers