Nvidia Mellanox ConnectX-5 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Adapter
  5. Mellanox ConnectX-5
  6. User manual

Nvidia Mellanox ConnectX-5 User Manual

Ethernet adapter cards
Hide thumbs Also See for Mellanox ConnectX-5:
Table of Contents

Advertisement

Quick Links

NVIDIA Mellanox ConnectX-5 Ethernet Adapter
Cards User Manual
100Gb/s Ethernet Adapter Cards; Intelligent RDMA (RoCE) enabled NICs. Supports 1, 10, 25, 50, and
100 Gigabit Ethernet speeds.
Exported on Dec/23/2020 12:46 PM
https://docs.mellanox.com/x/SQFp

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Mellanox ConnectX-5 and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia Mellanox ConnectX-5

Summary of Contents for Nvidia Mellanox ConnectX-5

  • Page 1 NVIDIA Mellanox ConnectX-5 Ethernet Adapter Cards User Manual 100Gb/s Ethernet Adapter Cards; Intelligent RDMA (RoCE) enabled NICs. Supports 1, 10, 25, 50, and 100 Gigabit Ethernet speeds. Exported on Dec/23/2020 12:46 PM https://docs.mellanox.com/x/SQFp...

  • Page 2: Table Of Contents

    Table of Contents Introduction .......................... 8 Product Overview ......................... 8 Operating Systems/Distributions ....................11 Connectivity ..........................11 Interfaces ..........................13 Ethernet SFP28 and QSFP28 Interfaces ..................13 PCI Express Interface ........................ 13 LED Interface ..........................13 Heat Sink Interface ........................14 SMBus Interface ........................
  • Page 3 Installing MLNX_OFED Using YUM ..................31 Installing MLNX_OFED Using apt-get Tool ................33 Updating Firmware After Installation ..................35 UEFI Secure Boot ......................... 36 Performance Tuning ......................37 Windows Driver Installation ......................38 Software Requirements ......................38 Downloading Mellanox WinOF-2 Driver ................. 38 Installing Mellanox WinOF-2 Driver ..................
  • Page 4 Document Revision History ....................63...
  • Page 5 About This Manual This User Manual describes NVIDIA® Mellanox® ConnectX®-5 and ConnectX®-5 Ex Ethernet Single and Dual SFP28 and QSFP28 port PCI Express x8/x16 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.
  • Page 6 Intended Audience This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with Ethernet network and architecture specifications. Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods: •...
  • Page 7 Mellanox LinkX Ethernet cables and transceivers are designed to maximize the Mellanox LinkX performance of High-Performance Computing networks, requiring high- Interconnect bandwidth, low-latency connections between compute nodes and switch nodes. Solutions Mellanox offers one of the industry’s broadest portfolios of 40Gb/s, 56Gb/s and 100Gb/s cables, including Direct Attach Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km.

  • Page 8: Introduction

    Introduction Product Overview This is the User Guide for Ethernet adapter cards based on the ConnectX®-5 and ConnectX®-5 Ex integrated circuit device. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Centers and High-Performance Computing environments.
  • Page 9 Model ConnectX-5 Ethernet Adapter Cards UEFI Enabled (x86/ Arm) RoHS RoHS Compliant IC Part Number MT27808A0-FCCF-EV ConnectX-5 Ethernet 100Gb/s Adapter Cards Model ConnectX-5 Ethernet Adapter Cards Part Number MCX515A-CCAT MCX516A-CCAT Ethernet Data Rate 10/25/40/50/100 Gb/s Network Connector Single-port QSFP28 Dual-port QSFP28 Type PCI Express PCIe Gen 3.0 x16...
  • Page 10 Features and Benefits This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability. Feature Description Uses PCIe Gen 3.0 (8GT/s) through an x8 or x16 edge connector. Gen PCI Express (PCIe) 1.1 and 2.0 compatible.

  • Page 11: Operating Systems/Distributions

    Feature Description Mellanox PeerDirect™ PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-5 advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes.
  • Page 12 • Passive copper cable with ESD protection • Powered connectors for optical and active cable support...

  • Page 13: Interfaces

    Interfaces Ethernet SFP28 and QSFP28 Interfaces The network ports of the ConnectX®-5 adapter card are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28 connectors on the adapter card. The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables.

  • Page 14: Heat Sink Interface

    State Bi-color LED (Yellow/Green) - Link speed Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: Error Type Description LED Behavior Blinks until error C access to the is fixed networking ports fails Over-current Over-current Blinks until error...

  • Page 15: Thermal Sensors

    Thermal Sensors The adapter card incorporates the ConnectX IC which operates in the range of temperatures between 0C and 105C. There are three thermal threshold definitions which impact the overall system operation state: • Warning – 105°C: On managed systems only: When the device crosses the 100°C threshold, a Warning Threshold message will be issued by the management SW, indicating to system administration that the card has crossed the Warning threshold.

  • Page 16: Hardware Installation

    Hardware Installation Installation and initialization of ConnectX-5 adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions document available on mellanox.com.

  • Page 17: Installation Procedure Overview

    Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Hazardous Radiation Exposure • Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure.For products with optical ports.

  • Page 18: Airflow Requirements

    Airflow Requirements ConnectX-5 adapter cards are offered with two airflow patterns: from the heatsink to the network ports, and vice versa, as shown below. Please refer to the Specifications section for airflow numbers for each specific card model. Airflow from the heatsink to the network ports: Airflow from the network ports to the heatsink: All cards in the system should be planned with the same airflow direction.

  • Page 19: Unpacking The Product

    Unpacking the Product The adapter is being installed in a system that operates with voltages that can be lethal. Before opening the case of the system, observe the following precautions to avoid injury and prevent damage to system components. • Remove any metallic objects from your hands and wrists. •...

  • Page 20: Bracket Replacement Instructions

    Bracket Replacement Instructions The card is usually shipped with an assembled high-profile bracket. If this form factor is suitable for your requirements, you can skip the remainder of this section and move to Installation Instructions. If you need to replace the high-profile bracket with the short bracket that is included in the shipping box, please follow the instructions in this section.
  • Page 21 3. Place the adapter in an available PCI Express slot. 4. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. 5. When the adapter is properly seated, the port connectors are aligned with the slot opening, and the adapter faceplate is visible against the system chassis.

  • Page 22: Cables And Modules

    To uninstall the adapter card, see Uninstalling the Card. Cables and Modules To obtain the list of supported Mellanox cables for your adapter, please refer to the Cables Reference Table at http://www.mellanox.com/products/interconnect/cables-configurator.php. Cable Installation 1. All cables can be inserted or removed with the unit powered on. 2.

  • Page 23: Uninstalling The Card

    1. Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. 2. Expand System Devices and locate your Mellanox ConnectX-5 adapter card. 3. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window.
  • Page 24 1. Remove any metallic objects from your hands and wrists. 2. It is strongly recommended to use an ESD strap or other antistatic devices. 3. Turn off the system and disconnect the power cord from the server. Card Removal Please note that the following images are for illustration purposes only. Please note that the following images are for illustration purposes only.

  • Page 25: Driver Installation

    • VMware Driver Installation Linux Driver Installation This section describes how to install and test the Mellanox OFED for Linux package on a single server with a Mellanox ConnectX-5 adapter card installed. Prerequisites Requirements Description A server platform with one of the following Mellanox...

  • Page 26: Installing Mellanox Ofed

    c. Click the desired ISO/tgz package. d. To obtain the download link, accept the End User License Agreement (EULA). 3. Use the md5sum utility to confirm the file integrity of your ISO image. Run the following command and compare the result to the value provided on the download page. md5sum MLNX_OFED_LINUX-<ver>-<OS label>.iso Installing Mellanox OFED Installation Script...
  • Page 27 On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_kernel_support.sh" is irrelevant on Ubuntu and Debian distributions. Example The following command will create a MLNX_OFED_LINUX ISO image for RedHat 6.3 under the / tmp directory.
  • Page 28 /mnt/mlnxofedinstall Logs dir: /tmp/MLNX_OFED_LINUX-x.x-x.logs This program will install the MLNX_OFED_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them. Starting MLNX_OFED_LINUX-x.x.x installation ...
  • Page 29 In case your machine has an unsupported network adapter device, no firmware update will occur and one of the following error messages below will be printed. Please contact your hardware vendor for help on firmware updates. Error message 1: Device #1: Device Type: ConnectX-5 Part Number:...
  • Page 30 Firmware • The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled: • The installation script is run in default mode; that is, without the option ‘--without-fw-update’ • The firmware version of the adapter device is older than the firmware version included with the Mellanox OFED ISO image Note: If an adapter’s flash was originally programmed with an Expansion ROM image, the automatic firmware update will also...

  • Page 31: Installing Mlnx_Ofed Using Yum

    blacklist mlx4_core blacklist mlx4_en blacklist mlx5_core blacklist mlx5_ib 2. Set “ONBOOT=no” in the "/etc/infiniband/openib.conf" file. 3. If the modules exist in the initramfs file, they can automatically be loaded by the kernel. To prevent this behavior, update the initramfs using the operating systems’ standard tools. Note: The process of updating the initramfs will add the blacklists from step 1, and will prevent the kernel from loading the modules automatically.
  • Page 32 # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox --2014-04-20 13:52:30-- http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox Resolving www.mellanox.com... 72.3.194.0 Connecting to www.mellanox.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1354 (1.3K) [text/plain] Saving to: ?RPM-GPG-KEY-Mellanox? 100%[=================================================>] 1,354 --.-K/s in 0s 2014-04-20 13:52:30 (247 MB/s) - ?RPM-GPG-KEY-Mellanox? saved [1354/1354] 4.

  • Page 33: Installing Mlnx_Ofed Using Apt-Get Tool

    mlnx-ofed-all Installs all available packages in MLNX_OFED. mlnx-ofed-basic Installs basic packages required running Mellanox cards. mlnx-ofed-guest Installs packages required by guest OS. mlnx-ofed-hpc Installs packages required HPC. mlnx-ofed-hypervisor Installs packages required by hypervisor OS. mlnx-ofed-vma Installs packages required by VMA. mlnx-ofed-vma-eth Installs packages required by VMA to work over Ethernet.
  • Page 34 Setting up MLNX_OFED apt-get Repository 1. Log into the installation machine as root. 2. Extract the MLNX_OFED pacakge on a shared location in your network. You can download it from http://www.mellanox.com > Products > Software> Ethernet Drivers. 3. Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: # deb file:/<path to extracted MLNX_OFED package>/DEBS ./ 4.

  • Page 35: Updating Firmware After Installation

    Installing MLNX_OFED using the “apt-get” tool does not automatically update the firmware. To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Or, update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Updating Adapter Firmware.

  • Page 36: Uefi Secure Boot

    2. Download the firmware BIN file from the Mellanox website or the OEM website. 3. Burn the firmware. mlxfwmanager_pci -i <fw_file.bin> 4. Reboot your machine after the firmware burning is completed. Updating the Device Firmware Automatically upon System Boot As of MLNX_OFED v3.1-x.x.x, firmware can be automatically updated upon system boot. The firmware update package (mlnx-fw-updater) is installed in the “/opt/mellanox/mlnx-fw-updater”...

  • Page 37: Performance Tuning

    2. Add the public key to the MOK list using the mokutil utility. # mokutil --import mlnx_signing_key_pub.der 3. Reboot the system. The pending MOK key enrollment request will be noticed by shim.efi and it will launch MokManager.efi to allow you to complete the enrollment from the UEFI console. You will need to enter the password you previously associated with this request and confirm the enrollment.

  • Page 38: Windows Driver Installation

    please refer to the Performance Tuning Guide for Mellanox Network Adapters https:// community.mellanox.com/docs/DOC-2489. Windows Driver Installation For Windows, download and install the latest Mellanox WinOF-2 for Windows software package available via the Mellanox web site at: http://www.mellanox.com > Products > Software > Ethernet Drivers >...

  • Page 39: Installing Mellanox Winof-2 Driver

    Installing the incorrect exe file is prohibited. If you do so, an error message will be displayed. For example, if you try to install a 64-bit executable file on a 32-bit machine, the wizard will display the following (or a similar) error message: “The installation package is not supported by this processor type.
  • Page 40 4. Click Next in the Welcome screen. 5. Read and accept the license agreement and click Next. 6. Select the target folder for the installation.
  • Page 41 7. The firmware upgrade screen will be displayed in the following cases: • If the user has an OEM card. In this case, the firmware will not be displayed. • If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly.
  • Page 42 8. Select a Complete or Custom installation, follow Step a onward. a. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes •...
  • Page 43 9. Click Install to start the installation. 10. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see 11. Click Finish to complete the installation.
  • Page 44 Unattended Installation If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. 1.

  • Page 45: Uninstalling Mellanox Winof-2 Driver

    Installation Results Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. The inf files can be located at: %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\ To see the Mellanox network adapters, display the Device Manager and pull down the “Network adapters”...

  • Page 46: Extracting Files Without Running Installation

    1. Open a CMD console. (Click Task Manager > File > Run new task, and enter CMD.) 2. To uninstall the driver, run: MLNX_WinOF2-2_0_All_x64.exe /S /x /v"/qn" Extracting Files Without Running Installation To extract the files without running installation, perform the following steps: 1.
  • Page 47 5. Click Install to extract this folder, or click Change to install to a different folder.

  • Page 48: Firmware Upgrade

    6. To complete the extraction, click Finish. Firmware Upgrade If the machine has a standard Mellanox card with an older firmware version, the firmware will be automatically updated as part of the WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to the MFT User Manual at www.m ellanox.com >Products >...

  • Page 49: Installing Mellanox Native Esxi Driver For Vmware Vsphere

    Installing Mellanox NATIVE ESXi Driver for VMware vSphere Please uninstall all previous Mellanox driver packages prior to installing the new version. See Removing Earlier Mellanox Drivers for further information. To install the driver: 1. Log into the ESXi server with root permissions. 2.

  • Page 50: Firmware Programming

    Firmware Programming 1. Download the VMware bootable binary images v4.6.0 from the Mellanox Firmware Tools (MFT) site. a. ESXi 6.5 File: mft-4.6.0.48-10EM-650.0.0.4598673.x86_64.vib b. MD5SUM: 0804cffe30913a7b4017445a0f0adbe1 2. Install the image according to the steps described in the MFT User Manual. The following procedure requires custom boot image downloading, mounting and booting from a USB device.

  • Page 51: Updating Adapter Firmware

    Updating Adapter Firmware Each adapter card is shipped with the latest version of qualified firmware at the time of manufacturing. However, Mellanox issues firmware updates occasionally that provide new features and bug fixes. To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility.

  • Page 52: Troubleshooting

    Troubleshooting General Troubleshooting • Ensure that the adapter is placed correctly Server unable to find the adapter • Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...

  • Page 53: Windows Troubleshooting

    Mellanox Firmware Tool (MFT) Download and install MFT: http://www.mellanox.com/ content/pages.php?pg=management_tools&menu_section=34 Refer to the User Manual for installation instructions. Once installed, run: mst start mst status flint -d <mst_device> q ibstat Ports Information ibv_devinfo To download the latest firmware version refer to http:// Firmware Version Upgrade www.mellanox.com/supportdownloader...

  • Page 54: Specifications

    Specifications MCX512A-ACAT Specifications Size: 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Physical Connector: Dual SFP28 Ethernet (copper and optical) Ethernet: 25GBASE-CR/CR-S, 25GBASE-SR, 25GBASE-LR, 25G Ethernet Protocol Support Consortium, 10GBASE-SR, 10GBASE-LR, 10GBASE-ER, SGMII / 1000BASE-X Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V Power...
  • Page 55 Non-operational -40°C to 70°C Humidity 90% relative humidity Altitude (Operational) 3050m Airflow Airflow Specifications. Safety CB / cTUVus / CE Regulatory CE / FCC / VCCI / ICES / RCM RoHS RoHS-compliant Notes: a. Typical power for ATIS traffic load. b.
  • Page 56 Voltage: 12V Power and Environmental Power Cable Type Passive Cable 14.2W Typical Power Maximum Power: Passive Cable 17.0W 1.5W Active Cable 18.6W 2.5W Active Cable 19.7W Maximum power available through QSFP28 port: 5W Temperature: Operational 0°C to 55°C Non-operational -40°C to 70°C Humidity: 90% relative humidity Altitude (Operational)

  • Page 57: Airflow Specifications

    1.5W Active Cable 22.3W 2.5W Active Cable 24.6W Maximum power available through QSFP28 port: 5W Temperature: Operational 0°C to 55°C Non-operational -40°C to 70°C Humidity: 90% relative humidity Altitude (Operational) 3050m Airflow: Airflow Specifications. Safety: CB / cTUVus / CE Regulatory EMC: CE / FCC / VCCI / ICES / RCM...

  • Page 58: Board Mechanical Drawing And Dimensions

    Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Mechanical Drawing of a Dual-Port x16 Card Mechanical Drawing of a Dual-Port x8 Card For the 3D Model of the card, please refer to 3D Models at http://www.mellanox.com/page/ 3d_models.

  • Page 59: Bracket Mechanical Drawing

    Bracket Mechanical Drawing Single-port Tall Bracket Dimensions in millimeters. Dual-port Tall Bracket Dimensions in millimeters.
  • Page 60 Environmental...

  • Page 61: Finding The Mac On The Adapter Card

    Finding the MAC on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards.

  • Page 62: Mcx515A-Ccat Board Label (Example)

    MCX515A-CCAT Board Label (Example) MCX516A-CCAT Board Label (Example)
  • Page 63 Document Revision History Date Revision Description of Changes Oct. 2020 Updated board labels. Feb. 2020 Added altitude criteria to "Specifications". Oct. 2019 Updated the Ethernet supported protocols in "Specifications". Dec. 2018 Migrated to online format; minor reorganization. • Added MCX512A-ACAT support across the Nov.
  • Page 64 Date Revision Description of Changes • Updated the following specifications tables: July 2017 • MCX515A-CCAT Specifications • MCX516A-CCAT Specifications • Updated Airflow Specifications • Updated Product Overview Apr. 2017 • Updated Installation Instructions Mar. 2017 First release...
  • Page 65 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
  • Page 66 NVIDIA, the NVIDIA logo, and Mellanox are trademarks and/or registered trademarks of Mellanox Technologies Ltd. and/or NVIDIA Corporation in the U.S. and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated.

This manual is also suitable for:

Mellanox connectx-5 ex

Table of Contents

Save PDF