Summary of Contents for Mellanox Technologies ConnectX-5 MCX515A-CCAT
Page 1
ConnectX®-5 Ethernet Single and Dual Port Adapter Card User Manual P/N: MCX515A-CCAT, MCX516A-CCAT, MCX516A-CDAT Rev 1.1 www.mellanox.com Mellanox Technologies...
Page 2
ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT (S) AND/OR THE SYSTEM USING IT . THEREFORE , MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY .
Page 5
Appendix D Avertissements de sécurité d’installation (Warnings in French) 73 Appendix E Sicherheitshinweise (Warnings in German) ....75 Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 77 Rev 1.1 Mellanox Technologies...
This document was printed on May 16, 2017. Table 1 - Revision History Table Date Comments/Changes April 2017 • Updated Product Overview on page 10. • Updated Card Installation Instructions on page 18 March 2017 First release Rev 1.1 Mellanox Technologies...
About this Manual This User Manual describes Mellanox Technologies ConnectX®-5 Ethernet Single and Dual QSFP28 port PCI Express x16 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documenta- tion.
The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support Or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Rev 1.1 Mellanox Technologies...
Introduction Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con- nectX®-5 integrated circuit device. These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen4 servers used in Enterprise Data Centers, High-Performance Computing, and Embedded environments.
Band and Ethernet networks. Converged Ethernet Leveraging data center bridging (DCB) capabilities as well as Con- (RoCE) nectX-5 advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks. Rev 1.1 Mellanox Technologies...
Page 13
• Adaptive Routing on Reliable Transport Accelerations • Burst Buffer Offloads for Background Checkpointing a. This section describes hardware features and capabilities. Please refer to the driver release notes for feature availabil- ity. See Related Documentation. Rev 1.1 Mellanox Technologies...
• VMware • OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Connectivity • Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Rev 1.1 Mellanox Technologies...
• Auto-negotiates to x16, x8, x4, x2, or x1 • Support for MSI/MSI-X mechanisms LED Interface There is one bi-color I/O LED per port located on the adapter card For LED specifications, please refer to Section 7.3, “Adapter LED Operation”, on page Rev 1.1 Mellanox Technologies...
Section 3.5, “Card Installa- tion Instructions”, on page 18. If you need to replace it with the short bracket that is included in the shipping box, please follow the instructions in this section. Rev 1.1 Mellanox Technologies...
2. Screw on the bracket using the screws and washers saved from the bracket removal procedure above. 3. Make sure that the LEDs are aligned onto the bracket holes. 4. Use a torque driver to apply up to 2 lbs-in torque on the screws. Rev 1.1 Mellanox Technologies...
2. Open the system case. 3. Place the adapter in an available PCI Express slot. 4. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. Rev 1.1 Mellanox Technologies...
Insert the connector into the adapter card. Be careful to insert the connector straight into the cage. Do not apply any torque, up or down, to the connector cage in the adapter card. d. Make sure that the connector locks in place. When installing cables make sure that the latches engage. Rev 1.1 Mellanox Technologies...
Please note that the following images are for illustration purposes only. 1. Verify that the system is powered off and unplugged. 2. Wait 30 seconds. 3. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). Rev 1.1 Mellanox Technologies...
Page 21
4. Holding the adapter card from its center, gently pull the adapter card from the PCI Express slot. 5. When the port connectors reach the top of the chassis window, gently pull the adapter card in parallel to the motherboard. Rev 1.1 Mellanox Technologies...
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
Hardware Installation 3.8.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mellanox Technologies”: # lspci |grep -i Mellanox Network controller: Mellanox Technologies MT28800 Family [ConnectX-5] Rev 1.1 Mellanox Technologies...
Step 1. The following example shows a system with an installed Mellanox HCA: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 Download the ISO image to your host.
The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Rev 1.1 Mellanox Technologies...
At this point, Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine. 4.1.3.1 Installation Procedure Login to the installation machine as root. Step 1. Mount the ISO image on your machine. Step 2. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Rev 1.1 Mellanox Technologies...
Page 27
ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; ROHS R6 PSID: MT_2190110032 PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d5cf810 Versions: Current Available 12.14.0114 12.14.0114 FW (Running) 12.14.0094 Status: Up to date Rev 1.1 Mellanox Technologies...
Page 28
Most of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files. See the relevant chapters in this manual for details. The list of the modules that will be loaded automatically upon boot can be found in the /etc/ file. infiniband/openib.conf Rev 1.1 Mellanox Technologies...
Logs dir: /tmp/MLNX_OFED_LINUX-x.x-x.logs 4.1.3.4 openibd Script As of MLNX_OFED v2.2-1.0.0 the script supports pre/post start/stop scripts: openibd This can be controlled by setting the variables below in the /etc/infiniband/openibd.conf file. OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh Rev 1.1 Mellanox Technologies...
Failed to start the mst driver 4.1.4 Uninstalling Mellanox OFED Use the script to uninstall the Mellanox OFED package. /usr/sbin/ofed_uninstall.sh The script is part of the RPM. ofed-scripts Rev 1.1 Mellanox Technologies...
To update the firmware to the version included in MLNX_OFED package, you can either: • Run # yum install mlnx-fw-updater • Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in section Section 4.1.7, “Updating Firmware After Installation”, on page 35.
Create an apt-get repository configuration file called Step 3. with the following content: "/etc/apt/sources.list.d/mlnx_ofed.list" deb file:/<path to extracted MLNX_OFED package>/DEBS / Download and install Mellanox Technologies GPG-KEY. Step 4. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Check that the key was successfully imported.
OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http:// www.mellanox.com > Support > Firmware Download).
"\.ko$" | xargs strip -g After the signature has been removed, a massage as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" Rev 1.1 Mellanox Technologies...
Depending on the application of the user's system, it may be necessary to modify the default con- figuration of network adapters based on the ConnectX® adapters In case tuning is required, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Rev 1.1 Mellanox Technologies...
Installing the incorrect .exe file is prohibited. If you do so, an error message will be displayed. For example, if you try to install a 64-bit .exe on a 32-bit machine, the wizard will display the follow- ing (or a similar) error message: Rev 1.1 Mellanox Technologies...
[Optional] Manually configure your setup to contain the logs option. Step 2. MLNX_WinOF2-1_21_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. MLNX_WinOF2-1_21_All_x64.exe /v" MT_SKIPFWUPGRD=1" 1. MT_SKIPFWUPGRD default value is False Rev 1.1 Mellanox Technologies...
Page 41
Driver Installation Click Next in the Welcome screen. Step 4. Read then accept the license agreement and click Next. Step 5. Rev 1.1 Mellanox Technologies...
Page 42
Driver Installation Select the target folder for the installation. Step 6. Rev 1.1 Mellanox Technologies...
Page 43
If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a Mellanox card, only the Mellanox card will be updated. Rev 1.1 Mellanox Technologies...
Page 44
• Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat Rev 1.1 Mellanox Technologies...
Page 45
Driver Installation • Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd b. Click Install to start the installation. Rev 1.1 Mellanox Technologies...
Page 46
Driver Installation In case firmware upgrade option was checked in <Xref>Step 7, you will be notified if a Step 9. firmware upgrade is required (see Click Finish to complete the installation. Step 10. Rev 1.1 Mellanox Technologies...
Upon installation completion, the inf files can be located at: • %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the “Network adapters” menu. 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Rev 1.1 Mellanox Technologies...
Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. MLNX_WinOF2-1_21_All_x64 /a • To extract only the driver files. MLNX_WinOF2-1_21_All_x64 /a /vMT_DRIVERS_ONLY=1 Rev 1.1 Mellanox Technologies...
Page 49
Driver Installation Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Rev 1.1 Mellanox Technologies...
Page 50
Driver Installation Click Install to extract this folder, or click Change to install to a different folder. Step 5. To complete the extraction, click Finish. Step 6. Rev 1.1 Mellanox Technologies...
Download the driver (MLNX_WinOF2_MLNX_WinOF2-1_60_mlx5_All_win2016_x64_- Step 2. fre_1_60_15407.exe). Extract the driver to a local directory (see Section 4.2.5, “Extracting Files Without Running Step 3. Installation”, on page 48). Copy the contents of this directory to C:\WS2016TP5_Drivers. Step 4. Rev 1.1 Mellanox Technologies...
When upgrading the driver on a server where the remote connection was done over Mella- nox device, there will be a loss of connectivity. To upgrade, it is recommended to run a script that will execute all the required upgrade commands. Rev 1.1 Mellanox Technologies...
6. Reboot the machine. 7. Verify the driver was installed successfully. esxcli software vib list | grep nmlx nmlx5-core 4.16.8.8-1OEM.650.0.0.4240417 PartnerSupported 2017-01-31 nmlx5-rdma 4.16.8.8-1OEM.650.0.0.4240417 PartnerSupported 2017-01-31 After the installation process, all kernel modules are loaded automatically upon boot. Rev 1.1 Mellanox Technologies...
• ESXi 6.5 File: mft-4.6.0.48-10EM-650.0.0.4598673.x86_64.vib MD5SUM: 0804cffe30913a7b4017445a0f0adbe1 2. Install the image according to the steps described in the MFT User Manual. The following procedure requires custom boot image downloading, mounting and boot- ing from a USB device. Rev 1.1 Mellanox Technologies...
Base MAC: 0000e41d2da206d4 Versions: Current Available 12.12.0780 12.12.1100 Status: Update required Perform FW update? [y/N]: y Device #1: Up to date Device #2: Updating FW ... Done Restart needed for updates to take effect. Log File: /var/log/mlxup/mlxup-yyyymmdd.log Rev 1.1 Mellanox Technologies...
Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but with no • Check that both the adapter and its link are set to the same speed communication established and duplex settings Rev 1.1 Mellanox Technologies...
(MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo To download the latest firmware version refer to Firmware Version Upgrade http://www.mellanox.com/supportdownloader cat /var/log/messages Collect Log File dmesg > system.log Rev 1.1 Mellanox Technologies...
Airflow Specifications on page 61 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. T ypical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.1 Mellanox Technologies...
Airflow Specifications on page 61 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.1 Mellanox Technologies...
Airflow Specifications on page 61 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.1 Mellanox Technologies...
Indicates an active physical link Blinking Amber Indicates a problem with the physical link Solid Green Indicates a valid logical (data activity) link with no active traffic Blinking Green Indicates a valid logical link with active traffic Rev 1.1 Mellanox Technologies...
Specifications Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 3: Mechanical Drawing of the Dual-port Adapter Cards 142.24 68.9 Rev 1.1 Mellanox Technologies...
Figure 5: Connector and Cage Views Table 18 - Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name TXN_2 Tx2n TXP_2 Tx2p TXN_4 Tx4n TXP_4 Tx4p ModSelL_Port0 ModSelL ResetL_Port0 ResetL VccRx Rev 1.1 Mellanox Technologies...
Page 66
Port A Signal Name RXP_3 Rx3p RXN_3 Rx3n RXP_1 Rx1p RXN_1 Rx1n RXN_2 Rx2n RXP_2 Rx2p RXN_4 Rx4n RXP_4 Rx4p ModPrsl_Port0 Mod PrsL IntL IntL VccTx Vcc1 LPMode_Port0 LPMode TXP_3 Tx3p TXN_3 Tx3n TXP_1 Tx1p TXN_1 Tx1n Rev 1.1 Mellanox Technologies...
PCI Express x16 Connector Pinout The adapter cards use a standard PCI Express x16 edge connector and the PCI Express x16 stan- dard pinout according to the PCI Express 3.0/4.0 specification. Figure 6: PCIe x16 Connector Pinout. Rev 1.1 Mellanox Technologies...
MAC for the Ethernet protocol The revision indicated on the labels in the following figures do not necessarily represent the latest revision of the card. Figure 7: MCX515A-CCAT Board Label (Example) Figure 8: MCX516A-CCAT Board Label Example) Rev 1.1 Mellanox Technologies...
6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Rev 1.1 Mellanox Technologies...
Page 71
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Rev 1.1 Mellanox Technologies...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 1.1 Mellanox Technologies...
Page 73
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Rev 1.1 Mellanox Technologies...
6. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. Rev 1.1 Mellanox Technologies...
Page 75
8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1 Rev 1.1 Mellanox Technologies...
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Rev 1.1 Mellanox Technologies...
Page 77
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Rev 1.1 Mellanox Technologies...
Need help?
Do you have a question about the ConnectX-5 MCX515A-CCAT and is the answer not in the manual?
Questions and answers