Related Manuals for Mellanox Technologies ConnectX-5
Summary of Contents for Mellanox Technologies ConnectX-5
- Page 1 Mellanox ConnectX®-5 Ex 100Gb/s VPI Single and Dual Adapter Cards User Manual P/N: MCX555A-ECAT, MCX556A-ECAT, MCX556A-EDAT Rev 1.5 www.mellanox.com Mellanox Technologies...
- Page 2 ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT (S) AND/OR THE SYSTEM USING IT . THEREFORE , MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY .
-
Page 3: Table Of Contents
3.9.1 On Windows ..........19 Rev 1.5 Mellanox Technologies... - Page 4 4.2.7 Firmware Upgrade ..........49 Rev 1.5 Mellanox Technologies...
- Page 5 Appendix C Avertissements de sécurité d’installation (Warnings in French) 68 Appendix D Sicherheitshinweise (Warnings in German) ....70 Appendix E Advertencias de seguridad para la instalación (Warnings in Spanish) 72 Rev 1.5 Mellanox Technologies...
-
Page 6: List Of Tables
Table 19: Physical and Logical Link Indications (Ethernet Mode)..... . 61 Table 20: Physical and Logical Link Indications (InfiniBand Mode) ....61 Rev 1.5 Mellanox Technologies... -
Page 7: List Of Figures
MCX556A-EDAT Board Label (Example) ........65 Rev 1.5 Mellanox Technologies... -
Page 8: Revision History
• MCX556A-ECAT Specification Table on page 56 • MCX556A-EDAT Specification Table on page 57 • Updated Linux on page 22 April 2017 • Updated Product Overview on page 9 • Updated Card Installation Instructions on page 15 March 2017 First release Mellanox Technologies... -
Page 9: About This Manual
About This Manual This User Manual describes Mellanox Technologies ConnectX®-5 Ex VPI Single and Dual QSFP28 port PCI Express x16 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documenta- tion. - Page 10 The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support Or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Rev 1.5 Mellanox Technologies...
-
Page 11: Chapter 1 Introduction
Introduction Introduction This is the User Guide for Mellanox Technologies VPI adapter cards based on the ConnectX®-5 integrated circuit device. These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Cen- ters, High-Performance Computing, and Embedded environments. -
Page 12: Features And Benefits
Overlay Networks work scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5 effectively addresses this by providing advanced NVGRE and VXLAN hardware offload- ing engines that encapsulate and de-capsulate the overlay protocol. -
Page 13: Operating Systems/Distributions
Introduction Table 4 - Features ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high- RDMA and RDMA over performance over Band and Ethernet networks. Leveraging data center bridging Converged Ethernet (RoCE) -
Page 14: Connectivity
Introduction Connectivity • Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Rev 1.5 Mellanox Technologies... -
Page 15: Chapter 2 Interfaces
• Auto-negotiates to x16, x8, x4, x2, or x1 • Support for MSI/MSI-X mechanisms LED Interface There is one bi-color I/O LED per port located on the adapter card.For LED specifications, please refer to Section 7.8, “Adapter Card LED Operations”, on page Rev 1.5 Mellanox Technologies... -
Page 16: Chapter 3 Hardware Installation
3. After shutting down the system, turn off power and unplug the cord. Remove the card from its package. Please note that the card must be placed on an antistatic surface.Check the card for visible signs of damage. Do not attempt to install the card if damaged. Rev 1.5 Mellanox Technologies... -
Page 17: Bracket Installation Instructions
Please note that the following figures are for illustration purposes only. 1. Before installing the card, make sure that the system is off and the power cord is not con- nected to the server. Please follow proper electrical grounding procedures. Rev 1.5 Mellanox Technologies... -
Page 18: Cables And Modules
Do not use excessive force when seating the card, as this may damage the system or the adapter. 5. Close the system case. Cables and Modules To obtain the list of supported Mellanox cables for your adapter, please refer to the Cables Refer- ence Table. Rev 1.5 Mellanox Technologies... -
Page 19: Cable Installation
6. To remove a cable, disengage the locks and slowly pull the connector away from the port receptacle. LED indicator will turn off when the cable is unseated. Rev 1.5 Mellanox Technologies... -
Page 20: Adapter Card Un-Installation Instructions
2. Wait 30 seconds. 3. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). 4. Holding the adapter card from its center, gently pull the adapter card from the PCI Express slot. Rev 1.5 Mellanox Technologies... -
Page 21: Identify The Card In Your System
1. Open Device Manager on the server. Click Start => Run, and then enter “devmgmt.msc”. 2. Expand System Devices and locate your Mellanox ConnectX-5 adapter card. 3. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window. -
Page 22: On Linux
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1018 (for ConnectX-5) – this is a valid Mellanox Technologies PCI Device ID. - Page 23 Hardware Installation Network controller: Mellanox Technologies MT28800 Family [ConnectX-5] Rev 1.5 Mellanox Technologies...
-
Page 24: Chapter 4 Driver Installation
Step 1. The following example shows a system with an installed Mellanox HCA: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 Download the ISO image to your host. -
Page 25: Installing Mellanox Ofed
NFS. To install on all the cluster nodes, use cluster-aware tools (such as pdsh). 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Rev 1.5 Mellanox Technologies... - Page 26 For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be running on one of the fabric nodes. At this point, Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine. For the list of installation options, run: ./mlnxofedinstall --h Rev 1.5 Mellanox Technologies...
-
Page 27: Installation Procedure
./mlnxofedinstall --without-dkms --add-kernel-support --kernel <kernel version in chroot> --without-fw-update --force For example: ./mlnxofedinstall --without-dkms --add-kernel-support --kernel 3.13.0-85-generic --without-fw-update --force Note that the path to kernel sources (--kernel-sources) should be added if the sources are not in their default location. Rev 1.5 Mellanox Technologies... - Page 28 If the installation script has performed a firmware update on your network adapter, com- Step 4. plete the step relevant to your adapter card type to load the firmware: • ConnectX-5/ConnectX-5 Ex - perform a standard reboot Otherwise, restart the driver by running: "/etc/init.d/openibd restart" Rev 1.5 Mellanox Technologies...
-
Page 29: Installation Results
0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW ven- dor. 4.1.3.4 Installation Logging While installing MLNX_OFED, the install log for each selected package will be saved in a sepa- rate log file. Rev 1.5 Mellanox Technologies... -
Page 30: Openibd Script
To prevent this behavior, update the initramfs using the operating systems’ standard tools. Note: The process of updating the initramfs will add the blacklists from step 1, and will pre- vent the kernel from loading the modules automatically. Rev 1.5 Mellanox Technologies... -
Page 31: Mlnxofedinstall Return Codes
Mount the ISO image on your machine and copy its content to a shared location in your net- Step 2. work. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Download and install Mellanox Technologies GPG-KEY: Step 3. The key can be downloaded via the following link: Rev 1.5... -
Page 32: Installing Mlnx_Ofed Using The Yum Tool
# sudo rpm --import RPM-GPG-KEY-Mellanox warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mel- lanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Is this ok [y/N]: Check that the key was successfully imported. - Page 33 MLNX_OFED, you can use the mlnx_add_ker- script to build MLNX_OFED for your kernel. nel_support.sh The script will automatically build the matching group RPM packages for your kernel so that you can still install MLNX_OFED via yum. Rev 1.5 Mellanox Technologies...
-
Page 34: Uninstalling Mellanox Ofed Using The Yum Tool
To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in 4.1.5.3 Uninstalling Mellanox OFED Using the YUM Tool Use the script to uninstall the Mellanox OFED package. -
Page 35: Installing Mlnx_Ofed Using The Apt-Get Tool
“apt-get” To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Rev 1.5 Mellanox Technologies... -
Page 36: Uninstalling Mellanox Ofed Using The Apt-Get Tool
OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case that you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http://www.mellanox.com > Support > Firmware Download). -
Page 37: Updating The Device Firmware Automatically Upon System Boot
The 'mokutil' package is installed on your system • The system is booted in UEFI mode Download the x.509 public key. Step 1. # wget http://www.mellanox.com/downloads/ofed/mlnx_signing_key_pub.der Add the public key to the MOK list using the mokutil utility. Step 2. Rev 1.5 Mellanox Technologies... -
Page 38: Removing Signature From Kernel Modules
After the signature has been removed, a massage as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" However, please note that a similar message as the following will still be presented: "my_module: module verification failed: signature and/or required key missing - taint-... -
Page 39: Performance Tuning
Depending on the application of the user's system, it may be necessary to modify the default con- figuration of network adapters based on the ConnectX® adapters. In case that tuning is required, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Rev 1.5 Mellanox Technologies... -
Page 40: Windows Driver
64-bit .exe on a 32-bit machine, the wizard will display the follow- ing (or a similar) error message: “The installation package is not supported by this processor type. Contact your vendor” Rev 1.5 Mellanox Technologies... -
Page 41: Installing Mellanox Winof-2 Driver
[Optional] Manually configure your setup to contain the logs option (replace “LogFile” with Step 2. the relevant directory):. MLNX_WinOF2-1_10_<revision_version>_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. MLNX_WinOF2-1_10_<revision_version>_All_x64.exe /v" MT_SKIPFWUPGRD=1" 1. MT_SKIPFWUPGRD default value is False Rev 1.5 Mellanox Technologies... - Page 42 Driver Installation Click Next in the Welcome screen. Step 4. Read then accept the license agreement and click Next. Step 5. Rev 1.5 Mellanox Technologies...
- Page 43 Driver Installation Select the target folder for the installation. Step 6. Rev 1.5 Mellanox Technologies...
- Page 44 • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat Rev 1.5 Mellanox Technologies...
- Page 45 Driver Installation • Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd b. Click Next to install the desired tools. Click Install to start the installation. Step 9. Rev 1.5 Mellanox Technologies...
- Page 46 Driver Installation In case that firmware upgrade option was checked in Step 7, you will be notified if a firm- Step 10. ware upgrade is required (See Click Finish to complete the installation. Step 11. Rev 1.5 Mellanox Technologies...
-
Page 47: Unattended Installation
Upon installation completion, the inf files can be located at: • %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the “Network adapters” menu. 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Rev 1.5 Mellanox Technologies... -
Page 48: Extracting Files Without Running Installation
Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. MLNX_WinOF2-1_10_All_x64 /a To extract only the driver files. MLNX_WinOF2-1_10_All_x64 /a /vMT_DRIVERS_ONLY=1 Rev 1.5 Mellanox Technologies... - Page 49 Driver Installation Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Rev 1.5 Mellanox Technologies...
- Page 50 Driver Installation Click Install to extract this folder, or click Change to install to a different folder. Step 5. To complete the extraction, click Finish. Step 6. Rev 1.5 Mellanox Technologies...
-
Page 51: Uninstalling Mellanox Winof-2 Driver
Download the driver (MLNX_WinOF2_MLNX_WinOF2-1_64_mlx5_All_win2016_x64_- Step 2. fre_1_64_15407.exe). Extract the driver to a local directory (see Section 4.2.5, “Extracting Files Without Running Step 3. Installation”, on page 46). Copy the contents of this directory to C:\WS2016TP5_Drivers. Step 4. Rev 1.5 Mellanox Technologies... -
Page 52: Online Update
When upgrading the driver on a server where the remote connection was done over Mella- nox device, there will be a loss of connectivity. To upgrade, it is recommended to run a script that will execute all the required upgrade commands. Rev 1.5 Mellanox Technologies... -
Page 53: Chapter 5 Updating Adapter Card Firmware
=> Products => Software => Firmware Tools. Firmware Update Example [server1]# ./mlxup Querying Mellanox devices firmware ... Device Type: ConnectX-5 Part Number: MCX556A-EDAT Description: ConnectX®-5 Ex VPI adapter card, EDR IB (100Gb/s) and 100GbE, dual-port QSFP28, PCIe4.0 x16, tall bracket, ROHS R6... -
Page 54: Chapter 6 Troubleshooting
Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but with no • Check that both the adapter and its link are set to the same speed and communication established duplex settings Rev 1.5 Mellanox Technologies... -
Page 55: Linux
-d <mst_device> q ibstat Ports Information ibv_devinfo To download the latest firmware version refer to Firmware Version Upgrade http://www.mellanox.com/supportdownloader cat /var/log/messages dmesg >> system.log Collect Log File journalctl (Applicable on new operating systems) cat /var/log/syslog Rev 1.5 Mellanox Technologies... -
Page 56: Windows
–d <mst_device> –i <firmware_bin_file> b • Event log viewer • MST device logs: • mst start Collect Log File • mst status • flint –d <mst_device> dc > dump_configuration.log • mstdump <mst_device> dc > mstdump.log Rev 1.5 Mellanox Technologies... -
Page 57: Chapter 7 Specifications
EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox Infini- Band product. -
Page 58: Mcx556A-Ecat Specifications
EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. -
Page 59: Mcx556A-Edat Specifications
EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. -
Page 60: Mcx555A-Ecat Specifications
EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox Infini- Band product. -
Page 61: Mcx556A-Ecat Specifications
EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. -
Page 62: Mcx556A-Edat Specifications
EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. -
Page 63: Airflow Specifications
Indicates a valid logical (data activity) link with no active traffic Blinking Green Indicates a valid logical link with active traffic a. 1 Hz Blinking Amber occurs due to running a beacon command for locating the adapter card. Rev 1.5 Mellanox Technologies... -
Page 64: Board Mechanical Drawing And Dimensions
Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Fore the 3D Model of the card, please refer to Models. Figure 3: Mechanical Drawing of the Dual-port Adapter Cards 142.24 68.9 Rev 1.5 Mellanox Technologies... -
Page 65: Bracket Mechanical Drawing
Specifications 7.10 Bracket Mechanical Drawing Figure 4: Dual-port Tall Bracket 21.59 121.05 Rev 1.5 Mellanox Technologies... -
Page 66: Appendix A Finding The Guid/Mac And Serial Number On The Adapter Card
GUID and a MAC (derived from the GUID). The revision indicated on the labels in the following figures do not necessarily represent the latest revision of the card. Figure 5: MCX555A-ECAT Board Label (Example) Figure 6: MCX556A-ECAT Board Label (Example) Rev 1.5 Mellanox Technologies... -
Page 67: Figure 7: Mcx556A-Edat Board Label (Example)
Figure 7: MCX556A-EDAT Board Label (Example) Rev 1.5 Mellanox Technologies... -
Page 68: Appendix B Safety Warnings
6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Rev 1.5 Mellanox Technologies... - Page 69 Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Rev 1.5 Mellanox Technologies...
-
Page 70: Appendix C Avertissements De Sécurité D'installation (Warnings In French)
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 1.5 Mellanox Technologies... - Page 71 Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Rev 1.5 Mellanox Technologies...
-
Page 72: Appendix D Sicherheitshinweise (Warnings In German)
6. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. Rev 1.5 Mellanox Technologies... - Page 73 8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1 Rev 1.5 Mellanox Technologies...
-
Page 74: Appendix E Advertencias De Seguridad Para La Instalación (Warnings In Spanish)
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Rev 1.5 Mellanox Technologies... - Page 75 Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Rev 1.5 Mellanox Technologies...