Nvidia ConnectX-5 User Manual
Infiniband/vpi adapter card
Hide thumbs
Also See for ConnectX-5:
- User manual (71 pages) ,
- Hardware installation (7 pages) ,
- User manual (64 pages)
Related Manuals for Nvidia ConnectX-5
Summary of Contents for Nvidia ConnectX-5
- Page 1 NVIDIA ConnectX-5 InfiniBand/VPI Adapter Cards User Manual Exported on Sep/19/2022 09:36 AM...
-
Page 2: Table Of Contents
Identifying the Card in Your System............. 18 Uninstalling the Card ................19 Safety Precautions ................19 Card Removal ................... 19 Driver Installation................21 Linux Driver Installation ................21 Prerequisites..................21 Downloading NVIDIA OFED ..............21 Installing NVIDIA OFED ................ 22 Installing MLNX_OFED Using YUM ............27... - Page 3 UEFI Secure Boot................32 Performance Tuning ................34 Windows Driver Installation ..............34 Software Requirements ............... 34 Downloading NVIDIA WinOF-2 Driver ............35 Installing NVIDIA WinOF-2 Driver ............35 Uninstalling NVIDIA WinOF-2 Driver ............42 Extracting Files Without Running Installation ..........43 Firmware Upgrade ................
- Page 4 PCI Express x16 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Ordering Part Numbers The table below provides the ordering part numbers (OPN) for the available ConnectX-5 VPI adapter cards. IC in Use Marketing Description ConnectX®-...
- Page 5 NVIDIA ConnectX® NATIVE ESXi stack. See VMware® ESXi Drivers Documentation. NVIDIA Firmware Utility (mlxup) User NVIDIA firmware update and query utility used to update the Manual and Release Notes firmware. Refer to Firmware Utility (mlxup) Documentation. NVIDIA Firmware Tools (MFT) User Manual User Manual describing the set of MFT firmware management tools for a single node.
-
Page 6: Introduction
Embedded environments. The following table provides the ordering part number, port speed, number of ports, and PCI Express speed. Each adapter comes with two bracket heights - short and tall. ConnectX-5 Ex InfiniBand/VPI Adapter Cards Model ConnectX-5 Ex InfiniBand/VPI Adapter Cards... -
Page 7: Features And Benefits
A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 25.78125Gb/s InfiniBand with a 64b/66b encoding, resulting in an effective bandwidth of 100Gb/s. 100Gb/s ConnectX-5 offers the highest throughput VPI adapter, supporting EDR 100Gb/s InfiniBand and Virtual 100Gb/s Ethernet and enabling any standard networking, clustering, or storage to operate Protocol In seamlessly over any converged network leveraging a consolidated software stack. - Page 8 NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.
-
Page 9: Operating Systems/Distributions
Accelerati UEFI UEFI is a standard firmware interface designed to replace BIOS. NVIDIA UEFI Network driver allows boot over network via PXE (Preboot eXecution Environment). This network driver allows remote boot over InfiniBand or Ethernet, or Boot over iSCSI (Bo-iSCSI) in UEFI mode, and also supports the SecureBoot standard. -
Page 10: Interfaces
Interfaces InfiniBand Interface The network ports of the ConnectX®-5 adapter cards are compliant with the InfiniBand Architecture Specification, Release 1.3. InfiniBand traffic is transmitted through the cards' QSFP28 connectors. Ethernet QSFP28 Interface The network ports of the ConnectX®-5 adapter card are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits. Ethernet traffic is transmitted through the cards'... -
Page 11: Thermal Sensors
Physical and Logical Link Indications - InfiniBand Mode: State LED Behavior Beacon command for locating the card Blinks yellow at 1Hz Error Blinks yellow at 4Hz until error is resolved. The reason for the error may be: • Failure of I C access to the port •... -
Page 12: Hardware Installation
Hardware Installation Installation and initialization of ConnectX-5 adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the... -
Page 13: Airflow Requirements
Specifications section of the manual for more power requirements. Airflow Requirements ConnectX-5 adapter cards are offered with two airflow patterns: from the heatsink to the network ports, and vice versa, as shown below. Please refer to the "Specifications" chapter for airflow numbers for each specific card model. -
Page 14: Software Requirements
Software Requirements • See Operating Systems/Distributions section under the Introduction section. • Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. Unpacking the Product The adapter is being installed in a system that operates with voltages that can be lethal. -
Page 15: Bracket Replacement Instructions
Shut down your system if active: Turn off the power to the system, and disconnect the power cord. Refer to the system documentation for instructions. Before you install the ConnectX-5 card, make sure that the system is disconnected from power. - Page 16 Open the system case. Place the adapter in an available PCI Express slot. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. When the adapter is properly seated, the port connectors are aligned with the slot opening, and the adapter faceplate is visible against the system chassis.
-
Page 17: Cables And Modules
To uninstall the adapter card, see Uninstalling the Card. Cables and Modules To obtain the list of supported NVIDIA cables for your adapter, please refer to the Cables Reference Table at http://www.nvidia.com/products/interconnect/cables-configurator.php. Cable Installation All cables can be inserted or removed with the unit powered on. -
Page 18: Identifying The Card In Your System
In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA; and DEV is equal to 1018 (for ConnectX-5) – this is a valid NVIDIA PCI Device ID. -
Page 19: Uninstalling The Card
If the PCI device does not have a NVIDIA adapter ID, return to Step 2 to check another device. The list of NVIDIA PCI Device IDs can be found in the PCI ID repository at http:// pci-ids.ucw.cz/read/PC/15b3. Uninstalling the Card Safety Precautions The adapter is installed in a system that operates with voltages that can be lethal. - Page 20 5. When the port connectors reach the top of the chassis window, gently pull the adapter card in parallel to the motherboard. ...
-
Page 21: Driver Installation
Driver Installation Please use the relevant driver installation section. Linux Driver Installation This section describes how to install and test the NVIDIA OFED for Linux package on a single server with a NVIDIA ConnectX-5 adapter card installed. Prerequisites Requirements Description Platforms A server platform with one of the following NVIDIA adapters installed: •... -
Page 22: Installing Nvidia Ofed
You will be prompted to acknowledge the deletion of the old packages. • If you need to install NVIDIA OFED on an entire (homogeneous) cluster, a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system such as NFS. - Page 23 RAM. For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be running on one of the fabric nodes. At this point, NVIDIA OFED for Linux has already installed the OpenSM Subnet Manager on your machine. For the list of installation options, run: ./mlnxofedinstall --h...
- Page 24 Device #1: ---------- Device Type: ConnectX-5 Part Number: MCX555A-ECAT Description: ConnectX®-5 VPI adapter card, EDR IB (100Gb/s) and 100GbE, single-port QSFP28, PCIe3.0 x16, tall bracket...
- Page 25 /etc/infiniband/info. Most of the NVIDIA OFED components can be configured or reconfigured after the installation, by modifying the relevant configuration files. See the relevant chapters in this manual for details.
- Page 26 OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh An example of OPENIBD_POST_START script for activating all interfaces is provided in the MLNX_OFED package under the docs/scripts/openibd-post-start-configure-interfaces/ folder. Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically.
-
Page 27: Installing Mlnx_Ofed Using Yum
Failed to start the mstdriver Uninstalling MLNX_OFED Use the script /usr/sbin/ofed_uninstall.sh to uninstall the NVIDIA OFED package. The script is part of the ofed-scripts RPM. Installing MLNX_OFED Using YUM This type of installation is applicable to RedHat/OL, Fedora, XenServer Operating Systems. - Page 28 # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Download and install NVIDIA GPG-KEY: The key can be downloaded via the following link: http://www.nvidia.com/downloads/ofed/ RPM-GPG-KEY-Mellanox # wget http://www.nvidia.com/downloads/ofed/RPM-GPG-KEY-Mellanox --2014-04-20 13:52:30-- http://www.nvidia.com/downloads/ofed/RPM-GPG-KEY-Mellanox Resolving www.nvidia.com... 72.3.194.0 Connecting to www.nvidia.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1354 (1.3K) [text/plain]...
- Page 29 --> Processing Dependency: kmod-isert = 1.0-OFED.3.1.0.1.2.1.g832a737.rhel7u1 package: mlnx-ofed-all-3.1-0.1.2.noarch ........qperf.x86_64 0:0.4.9-9 rds-devel.x86_64 0:2.0.7-1.12 rds-tools.x86_64 0:2.0.7-1.12 sdpnetstat.x86_64 0:1.60-26 srptools.x86_64 0:1.0.2-12 Complete! Uninstalling MLNX_OFED Using the YUM Tool Use the script /usr/sbin/ofed_uninstall.sh to uninstall the NVIDIA OFED package. The script is part of the ofed-scripts RPM.
-
Page 30: Installing Mlnx_Ofed Using Apt-Get Tool
Setting up MLNX_OFED apt-get Repository Log into the installation machine as root. Extract the MLNX_OFED package on a shared location in your network. You can download it from http://www.nvidia.com > Products > Software> Ethernet Drivers. Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: # deb file:/<path to extracted MLNX_OFED package>/DEBS ./... -
Page 31: Updating Firmware After Installation
The firmware can be updated either manually or automatically (upon system boot), as described in the sections below. Updating the Device Online To update the device online on the machine from the NVIDIA site, use the following command line: mlxfwmanager --online -u -d <device> Example: mlxfwmanager --online -u -d 0000:09:00.0... -
Page 32: Uefi Secure Boot
Support > Firmware Download). Get the device’s PSID. mlxfwmanager_pci | grep PSID PSID: MT_1210110019 Download the firmware BIN file from the NVIDIA website or the OEM website. Burn the firmware. mlxfwmanager_pci -i <fw_file.bin> Reboot your machine after the firmware burning is completed. - Page 33 In order to support loading MLNX_OFED drivers when an OS supporting Secure Boot boots on a UEFI- based system with Secure Boot enabled, the NVIDIA x.509 public key should be added to the UEFI Secure Boot key database and loaded onto the system key ring by the kernel.
-
Page 34: Performance Tuning
Windows Driver Installation For Windows, download and install the latest NVIDIA WinOF-2 for Windows software package available via the NVIDIA web site at: http://www.nvidia.com > Products > Software > Ethernet Drivers > Download. Follow the installation instructions included in the download package (also available from the download page). -
Page 35: Downloading Nvidia Winof-2 Driver
%PROCESSOR_ARCHITECTURE% On an x64 (64-bit) machine, the output will be “AMD64”. Go to the NVIDIA WinOF-2 web page at http://www.nvidia.com > Products > Ethernet Drivers > Windows SW/Drivers. Download the exe image according to the architecture of your machine (see Step 1). The name of the .exe is in the following format: MLNX_WinOF2-<version>_x<arch>.exe. - Page 36 [Optional] Manually configure your setup to contain the logs option (replace “LogFile” with the relevant directory): MLNX_WinOF2-[Driver/Version]_<revision_version>_All_Arch.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version. (Note: MT_SKIPFWUPGRD default value is False.) MLNX_WinOF2-2_10_50000_All_x64.exe /v"/l*vx MyLog.txt=1" Click Next in the Welcome screen. Read and accept the license agreement and click Next.
- Page 37 If the user has an OEM card. In this case, the firmware will not be displayed. • If the user has a standard NVIDIA card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM...
- Page 38 Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat...
- Page 39 • Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd Click Next to install the desired tools. 9. Click Install to start the installation.
- Page 40 10. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see ). 11. Click Finish to complete the installation.
- Page 41 Unattended Installation If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session.
-
Page 42: Uninstalling Nvidia Winof-2 Driver
Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. The inf files can be located at: %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\ To see the NVIDIA network adapters, display the Device Manager and pull down the “Network adapters” menu. Uninstalling NVIDIA WinOF-2 Driver Attended Uninstallation To uninstall MLNX_WinOF2 on a single node: ... -
Page 43: Extracting Files Without Running Installation
Unattended Uninstallation To uninstall MLNX_WinOF2 in unattended mode: Open a CMD console. (Click Task Manager > File > Run new task, and enter CMD.) To uninstall the driver, run: MLNX_WinOF2-2_0_All_x64.exe /S /x /v"/qn" Extracting Files Without Running Installation To extract the files without running installation, perform the following steps: Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter CMD. - Page 44 Click Change and specify the location in which the files are extracted to. Click Install to extract this folder, or click Change to install to a different folder. ...
-
Page 45: Firmware Upgrade
6. To complete the extraction, click Finish. Firmware Upgrade If the machine has a standard NVIDIA card with an older firmware version, the firmware will be automatically updated as part of the WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to the MFT User Manual... -
Page 46: Troubleshooting
Troubleshooting General Tips Server unable to find the adapter • Ensure that the adapter is placed correctly. • Make sure the adapter slot and the adapter are compatible. • Install the adapter in a different PCI Express slot. • Use the drivers that came with the adapter or download the latest. -
Page 47: Gathering Information - Linux Environment
Firmware queries You can use the "mlxup --query" command to verify the card is installed. For expanded firmware query and debug capabilities, download and install MFT: http://www.nvidia.com/content/ pages.php?pg=management_tools&menu_section=34. Refer to the MFT User Manual for installation instructions. Once installed, run:... -
Page 48: Updating Adapter Firmware
To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility. The utility can query for available NVIDIA adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images. -
Page 49: Specifications
Specifications MCX555A-ECAT Specifications Physical Size: 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Connector: Single QSFP28 InfiniBand and Ethernet (copper and optical) Protocol Support InfiniBand: IBTA v1.3 a Auto-Negotiation: 1X/2X/4X SDR (2.5Gb/s per lane), DDR (5Gb/s per lane), QDR (10Gb/s per lane), FDR10 (10.3125Gb/s per lane), FDR (14.0625Gb/s per lane), EDR (25Gb/s per lane) port Ethernet: 100GBASE-CR4, 100GBASE-KR4, 100GBASE-SR4, 50GBASE-R2,... -
Page 50: Mcx556A-Ec[A/U]T Specifications
EMC: CE / FCC / VCCI / ICES / RCM RoHS compliant The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product. -
Page 51: Mcx556A-Edat Specifications
EMC: CE / FCC / VCCI / ICES / RCM RoHS compliant Notes: a. The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product. ... - Page 52 Protoco InfiniBand: IBTA v1.3 a Support Auto-Negotiation: 1X/2X/4X SDR (2.5Gb/s per lane), DDR (5Gb/s per lane), QDR (10Gb/ s per lane), FDR10 (10.3125Gb/s per lane), FDR (14.0625Gb/s per lane), EDR (25Gb/s per lane) port Ethernet: 100GBASE-CR4, 100GBASE-KR4, 100GBASE-SR4, 50GBASE-R2, 50GBASE-R4, 40GBASE-CR4, 40GBASE-KR4, 40GBASE-SR4, 40GBASE-LR4, 40GBASE-ER4, 40GBASE-R2, 25GBASE-R, 20GBASE-KR2, 10GBASE-LR,10GBASE-ER, 10GBASE-CX4,10GBASE-CR, 10GBASE-KR, SGMII, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR Data Rate...
-
Page 53: Airflow Specifications
The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get a. better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. -
Page 54: Adapter Card Mechanical Drawing (Component Side)
Adapter Card Mechanical Drawing (Component Side) Bracket Mechanical Drawing All dimensions are in millimeters. The bracket mechanical tolerance is +/- 0.25mm. - Page 55 ...
-
Page 56: Finding The Guid/Mac And Serial Number On The Adapter Card
Finding the GUID/MAC and Serial Number on the Adapter Card Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID). -
Page 57: Mcx556A-Ecut Board Label Example
MCX556A-ECUT board label example: MCX556A-EDAT board label example:... -
Page 58: Document Revision History
Document Revision History Date Revision Description of Changes Sep. 2022 Added a note concerning FRU EEPROM memory component under the Features and Benefits table. Jan. 2021 • Updated PCIe spec fields under "Specifications" page • Updated LED Interface Feb. 2020 Added altitude criteria to "Specifications"... - Page 59 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
- Page 60 Copyright © 2022 NVIDIA Corporation & affiliates. All Rights Reserved.