Related Manuals for Nvidia Mellanox
Summary of Contents for Nvidia Mellanox
- Page 1 NVIDIA Mellanox ConnectX-6 InfiniBand/VPI Adapter Cards for OCP Spec 3.0 User Manual Exported on Oct/23/2020 12:05 PM https://docs.mellanox.com/x/TwvTAQ...
-
Page 2: Table Of Contents
Table of Contents Table of Contents .......................2 Introduction........................8 Products Overview........................8 Features and Benefits......................9 Operating Systems/Distributions ..................11 Connectivity ........................... 11 Interfaces .........................12 InfiniBand Interface....................... 12 Ethernet QSFP56 Interfaces ....................12 PCI Express Interface ......................12 LED Interface......................... 12 SMBus Interface........................ - Page 3 Cards with Ejector Latch Extraction Instructions............23 Driver Installation ....................25 Windows Driver Installation....................25 Hardware and Software Requirements................25 Downloading Mellanox WinOF-2 Driver ................25 Installing Mellanox WinOF-2 Driver ................26 Attended Installation ....................26 Unattended Installation....................30 Installation Results ......................31 Extracting Files Without Running Installation ..............
- Page 4 Updating the Device Manually ..................45 Updating the Device Firmware Automatically upon System Boot ......45 UEFI Secure Boot ......................46 Enrolling Mellanox's x.509 Public Key on Your Systems........... 46 Removing Signature from kernel Modules ..............46 Performance Tuning ......................47 VMware Driver Installation ....................
- Page 5 MCX653435A-HDAI Board Labels (Example) - TBD ............61 MCX653436A-HDAI Board Labels (Example) - TBD ............61 MCX653435A-EDAI Board Labels (Example) - TBD............. 61 MCX653435A-HDAE Board Labels (Example) - TBD ............61 Document Revision History ..................62...
- Page 6 About This Manual This User Manual describes NVIDIA® Mellanox® ConnectX®-6 VPI adapter cards for Open Compute Project (OCP), Spec 3.0. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.
- Page 7 Mellanox firmware update and query utility used to update the firmware. See h Mellanox Firmware Utility ttp://www.mellanox.com > Products > Software > Firmware Tools >mlxup (mlxup) User Manual and Firmware Utility Release Notes User Manual describing the set of MFT firmware management tools for a Mellanox Firmware Tools single node.
-
Page 8: Introduction
XXX, as the OCP spec 3.0 available at that time did not contain any S&V definitions. A newer version of the OCP spec 3.0 has defined S&V specifications and Mellanox is in the midst of retesting these cards to comply with OCP spec 3.0. -
Page 9: Features And Benefits
53.125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of 100Gb/ Up to 200 Gigabit Ethernet Mellanox adapters comply with the following IEEE 802.3 standards: • 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE •... - Page 10 ConnectX-6 advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks. Mellanox PeerDirect™ PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time.
-
Page 11: Operating Systems/Distributions
Operating Systems/Distributions • RHEL/CentOS • Windows • FreeBSD • VMware • OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Connectivity • Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support... -
Page 12: Interfaces
Interfaces InfiniBand Interface The network ports of the ConnectX®-6 adapter cards are compliant with the InfiniBand Architecture Specification, Release 1. 3. InfiniBand traffic is transmitted through the cards' QSFP56 connectors. Ethernet QSFP56 Interfaces The network ports of the ConnectX®-6 adapter card are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits. -
Page 13: Smbus Interface
BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox OCP 3.0 adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact Mellanox Support. -
Page 14: Cpld Interface
CPLD Interface The adapter card incorporates a CPLD device that controls the networking port LEDs and the scan chain. It draws its power supply from 3.3V_EDGE. -
Page 15: Hardware Installation
System Requirements Unless otherwise specified, Mellanox products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operating environment should meet severity level G1 as per ISA 71.04 for gaseous... -
Page 16: Hardware Requirements
Cold Aisle Cooling Heatsink-to-port Airflow Direction Port-to-heatsink Airflow Direction Software Requirements • See Operating Systems/Distributions section under the Introduction section. • Software Stacks - Mellanox OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. -
Page 17: Safety Precautions
Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal. Before opening the case of the system, observe the following precautions to avoid injury and prevent damage to system components. • Remove any metallic objects from your hands and wrists. -
Page 18: Cards With Thumbscrew (Pull-Tab) Brackets
Retention Mechanism Installation Instructions Internal-Lock Bracket Installation Instructions for Cards with Internal Lock Ejector-Latch Bracket Installation Instructions for Cards with Internal-Lock Bracket Please note that the following figures are for illustration purposes only. Cards with Thumbscrew (Pull-tab) Brackets Please note that the following figures are for illustration purposes only. -
Page 19: Cards With Ejector Latch
Cards with Ejector Latch This section applies to MCX653435A-HDAE. Before installing the card, make sure that the system is off and the power cord is not connected to the server. Please follow proper electrical grounding procedures. Open the system case. Align the card with the system rails while making sure the ejector latch is open. ... -
Page 20: Cards With Internal Lock
To uninstall the adapter card, see Uninstalling the Card. Cables and Modules To obtain the list of supported Mellanox cables for your adapter, please refer to the Cables Reference Table at http://www.mellanox.com/products/interconnect/cables-configurator.php. Cable Installation All cables can be inserted or removed with the unit powered on. -
Page 21: Identifying The Card In Your System
LED indicator will turn off when the cable is unseated. Identifying the Card in Your System On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mellanox Technologies”: lspci |grep -i Mellanox... -
Page 22: Adapter Cards Extraction Instructions
In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1018 (for ConnectX-6) – this is a valid Mellanox Technologies PCI Device ID. -
Page 23: Cards Extraction Instructions
Cards Extraction Instructions Please note that the following images are for illustration purposes only. Cards with Thumbscrew (Pull-tab) Bracket Extraction Instructions Verify that the system is powered off and unplugged. Wait 30 seconds. Rotate the captive screw counterclockwise. While holding the tab, carefully pull out the adapter card. Cards with Internal-Lock Bracket Extraction Instructions ... - Page 24 Verify that the system is powered off and unplugged. Wait 30 seconds. Open the ejector latch. Gently extract the adapter card from the server.
-
Page 25: Driver Installation
VMware Driver Installation Windows Driver Installation For Windows, download and install the latest Mellanox WinOF-2 for Windows software package available via the Mellanox web site at: http://www.mellanox.com > Products > Software > Ethernet Drivers > Download. Follow the installation instructions included in the download package (also available from the download page). -
Page 26: Installing Mellanox Winof-2 Driver
Both Attended and Unattended installations require administrator privileges. WinOF-2 supports adapter cards based on the Mellanox ConnectX®-4 and above family of adapter IC devices only. If you have ConnectX-3 and ConnectX-3 Pro on your server, you will need to install WinOF driver. For details on how to install WinOF driver, please refer to WinOF User Manual . - Page 27 If the user has an OEM card. In this case, the firmware will not be displayed. If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a Mellanox...
- Page 28 Mellanox card will be updated. Select a Complete or Custom installation, follow Step a and on. Select the desired feature to install: Performance tools - install the performance tools that are used to measure performance in user environment Documentation - contains the User Manual and Release Notes iii.
- Page 29 Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd Click Next to install the desired tools. Click Install to start the installation. In case that firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required. See TBD.
-
Page 30: Unattended Installation
Click Finish to complete the installation. Unattended Installation If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. Use the /norestart or /forcerestart standard command-line options to control reboots. The following is an example of an unattended installation session. -
Page 31: Installation Results
Device Manager. Upon installation completion, the inf files can be located at: %ProgramFiles% \Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the Network adapters menu. Extracting Files Without Running Installation To extract the files without running installation, perform the following steps. - Page 32 Click Next to create a server image. Click Change and specify the location in which the files are extracted to. Click Install to extract this folder, or click Change to install to a different folder.
-
Page 33: Uninstalling Mellanox Winof-2 Driver
/v"/qn" Firmware Upgrade If the machine has a standard Mellanox card with an older firmware version, the firmware will be automatically updated as part of the WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to the MFT User Manual at www.m... -
Page 34: Online Update
-I -a <Driver_Folder>\mlx5.inf When upgrading the driver on a server where the remote connection was done over Mellanox device, there will be a loss of connectivity. To upgrade, it is recommended to run a script that will execute all the required upgrade commands. -
Page 35: Linux Driver Installation
Linux Driver Installation This section describes how to install and test the Mellanox OFED for Linux package on a single server with a Mellanox ConnectX-6 adapter card installed. Prerequisites Requirements Description Platforms A server platform with a ConnectX-6 InfiniBand/VPI adapter card installed •... -
Page 36: Installing Mellanox Ofed
You will be prompted to acknowledge the deletion of the old packages. • If you need to install Mellanox OFED on an entire (homogeneous) cluster, a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system such as NFS. -
Page 37: Installation Procedure
If desired, tune the value unlimited to a specific amount of RAM. For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be running on one of the fabric nodes. At this point, Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine. - Page 38 Status: No matching image found Error message 2: The firmware for this device is not distributed inside Mellanox driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor. 4. If the installation script has performed a firmware update on your network adapter, complete the step relevant to your adapter card type to load the firmware:...
-
Page 39: Installation Results
/etc/infiniband/info. Most of the Mellanox OFED components can be configured or reconfigured after the installation, by modifying the relevant configuration files. See the relevant chapters in this manual for details. -
Page 40: Driver Load Upon System Boot
MLNX_OFED package under the docs/scripts/openibd-post-start-configure-interfaces/ folder. Driver Load Upon System Boot Upon system boot, the Mellanox drivers will be loaded automatically. To prevent automatic load of the Mellanox drivers upon system boot: Add the following lines to the "/etc/modprobe.d/mlnx.conf" file. blacklist mlx4_core... -
Page 41: Uninstalling Mlnx_Ofed
Uninstalling MLNX_OFED Use the script /usr/sbin/ofed_uninstall.sh to uninstall the Mellanox OFED package. The script is part of the ofed-scripts RPM. Installing MLNX_OFED Using YUM This type of installation is applicable to RedHat/OL, Fedora, XenServer Operating Systems. Setting up MLNX_OFED YUM Repository Log into the installation machine as root. -
Page 42: Installing Mlnx_Ofed Using The Yum Tool
: MLNX_OFED vma-vpi installer package (with KMP support) Where: mlnx-ofed-all Installs all available packages in MLNX_OFED. mlnx-ofed-basic Installs basic packages required running Mellanox cards. mlnx-ofed-guest Installs packages required by guest OS. mlnx-ofed-hpc Installs packages required HPC. mlnx-ofed-hypervisor Installs packages required by hypervisor OS. -
Page 43: Uninstalling Mlnx_Ofed Using The Yum Tool
0:1.0.2-12 Complete! Uninstalling MLNX_OFED Using the YUM Tool Use the script /usr/sbin/ofed_uninstall.sh to uninstall the Mellanox OFED package. The script is part of the ofed-scripts RPM. Installing MLNX_OFED Using apt-get Tool This type of installation is applicable to Debian and Ubuntu operating systems. -
Page 44: Uninstalling Mlnx_Ofed Using The Apt-Get Tool
The firmware can be updated either manually or automatically (upon system boot), as described in the sections below. Updating the Device Online To update the device online on the machine from the Mellanox site, use the following command line: mlxfwmanager --online -u -d <device> Example:... -
Page 45: Updating The Device Manually
Support > Firmware Download). Get the device’s PSID. mlxfwmanager_pci | grep PSID PSID: MT_1210110019 Download the firmware BIN file from the Mellanox website or the OEM website. Burn the firmware. mlxfwmanager_pci -i <fw_file.bin> Reboot your machine after the firmware burning is completed. -
Page 46: Uefi Secure Boot
In order to support loading MLNX_OFED drivers when an OS supporting Secure Boot boots on a UEFI- based system with Secure Boot enabled, the Mellanox x.509 public key should be added to the UEFI Secure Boot key database and loaded onto the system key ring by the kernel. -
Page 47: Performance Tuning
A server platform with an adapter card based on one of the following Mellanox Technologies’ devices: • ConnectX®-6 (VPI, IB, EN) (firmware: fw-ConnectX6) Device ID For the latest list of device IDs, please visit Mellanox's website. Operating System ESXi 6.5 Installer Privileges The installation requires administrator privileges on the target... -
Page 48: Installing Mellanox Native Esxi Driver For Vmware Vsphere
Installing Mellanox NATIVE ESXi Driver for VMware vSphere Please uninstall all previous Mellanox driver packages prior to installing the new version. See Removing Earlier Mellanox Drivers for further information. To install the driver: Log into the ESXi server with root permissions. -
Page 49: Firmware Programming
Firmware Programming Download the VMware bootable binary images v4.6.0 from the Mellanox Firmware Tools (MFT) site. ESXi 6.5 File: mft-4.6.0.48-10EM-650.0.0.4598673.x86_64.vib MD5SUM: 0804cffe30913a7b4017445a0f0adbe1 Install the image according to the steps described in the MFT User Manual. The following procedure requires custom boot image downloading, mounting and... -
Page 50: Updating Adapter Firmware
The utility can query for available Mellanox adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images. -
Page 51: Troubleshooting
Linux Troubleshooting cat /etc/issue Environment uname -a Information cat /proc/cupinfo | grep ‘model name’ | uniq ofed_info -s ifconfig -a ip link show ethtool <interface> ethtool -i <interface_of_Mellanox_port_num> ibdev2netdev lspci | grep -i Mellanox Card Detection... -
Page 52: Windows Troubleshooting
To export system information to a text file, choose the Export option from the File menu. Assign a file name and save. Download and install MFT: http://www.mellanox.com/content/ Mellanox Firmware Tool (MFT) pages.php?pg=management_tools&menu_section=34 Refer to the User Manual for installation instructions. Once installed, open a CMD window and run: WinMFT... -
Page 53: Specifications
Specifications MCX653435A-HDAI Specifications Please make sure to install the ConnectX-6 OCP 3.0 card in a PCIe slot that is capable of supplying 35W. Size: 2.99 in. x 4.52 in (76.00mm x 115.00mm) Physical Connector: Single QSFP56 InfiniBand and Ethernet (copper and optical) Retention Mechanism: Internal Lock Data Rate: Ethernet... -
Page 54: Mcx653436A-Hdai Specifications
Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow numbers are measured while using Mellanox HDR optic cable. The maximum allowed temperature (internal sensor) for Mellanox HDR optic cable is 75C. MCX653436A-HDAI Specifications ... - Page 55 Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow numbers are measured while using Mellanox HDR optic cable. The maximum allowed temperature (internal sensor) for Mellanox HDR optic cable is 75C. ...
-
Page 56: Mcx653435A-Edai Specifications
Specifications MCX653435A-EDAI Please make sure to install the ConnectX-6 OCP 3.0 card in a PCIe slot that is capable of supplying 35W. Size: 2.99 in. x 4.52 in (76.00mm x 115.00mm) Physi Connector: Single QSFP56 InfiniBand and Ethernet (copper and optical) Retention Mechanism: Internal Lock Ethernet: 200GBASE-CR4, 200GBASE-KR4, 200GBASE-SR4, 100GBASE-CR4, 100GBASE-KR4, Proto... -
Page 57: Mcx653435A-Hdae Specifications
Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Mellanox 100Gb/s Cable. MCX653435A-HDAE Specifications Please make sure to install the ConnectX-6 OCP 3.0 card in a PCIe slot that is capable of supplying 35W. -
Page 58: Board Mechanical Drawing And Dimensions
IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. -
Page 59: Brackets Mechanical Drawings And Dimensions
Brackets Mechanical Drawings and Dimensions All dimensions are in millimeters. The brackets mechanical tolerance is +/- 0.25mm. Cards with Ejector Latch Bracket ... -
Page 60: Cards With Internal Lock Bracket
Cards with Internal Lock Bracket... -
Page 61: Finding The Guid/Mac And Serial Number On The Adapter Card
Finding the GUID/MAC and Serial Number on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID). - Page 62 Document Revision History Date Revision Description of Changes Oct. 2020 • Updated airflow numbers in "Specifications". • Updated networking ports LEDs indications in "Interfaces" Sep. 2020 Added OCP 3.0 bracket replacement instructions. Nov. 2020 Updated "Specifications" with airflow numbers. Oct. 2019 Added a note to "Introduction".
- Page 63 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
- Page 64 Terms of Sale for the product. Trademarks NVIDIA, the NVIDIA logo, and Mellanox are trademarks and/or registered trademarks of Mellanox Technologies Ltd. and/or NVIDIA Corporation in the U.S. and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated.
Need help?
Do you have a question about the Mellanox and is the answer not in the manual?
Questions and answers