Summary of Contents for Mellanox Technologies ConnectX-4 MCX413A-BCAT
Page 1
ConnectX®-4 Ethernet Single and Dual QSFP28 Port Adapter Card User Manual P/N: MCX413A-BCAT, MCX414A-BCAT, MCX415A-BCAT, MCX416A-BCAT, MCX413A- GCAT, MCX414A-GCAT, MCX415A-GCAT, MCX416A-GCAT, MCX415A-CCAT, MCX416A-CCAT Rev 2.6 www.mellanox.com Mellanox Technologies...
Page 2
ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT (S) AND/OR THE SYSTEM USING IT . THEREFORE , MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY .
Page 5
Appendix C Avertissements de sécurité d’installation (Warnings in French) 78 Appendix D Advertencias de seguridad para la instalación (Warnings in Spanish) 80 Appendix E Sicherheitshinweise (Warnings in German) ....82 Rev 2.6 Mellanox Technologies...
Interfaces on page 17 March 2016 • Updated Product Overview on page 13 • Updated Features and Benefits on page 14 • Updated the following tables: • MCX415A-CCAT Specifications on page 66 • MCX416A-CCAT Specifications on page 67 Rev 2.6 Mellanox Technologies...
Page 9
MCX415A-GCAT Specifications on page 64 • MCX416A-GCAT Specifications on page 65 • MCX415A-CCAT Specifications on page 66 • MCX416A-CCAT Specifications on page 67 • Updated Features and Benefits on page 14 • Updated Section 3.8.1, “On Windows,” on page 22 Rev 2.6 Mellanox Technologies...
Page 10
MCX413A-GCAT Specifications on page 62 • MCX416A-GCAT Specifications on page 65 Airflow Specifications on page 58 • Added • Added VMware Driver on page 50 • Updated Updating Adapter Card Firmware on page 54 November First Release 2015 Rev 2.6 Mellanox Technologies...
About this Manual This User Manual describes Mellanox Technologies ConnectX®-4 Ethernet Single and Dual QSFP28 port PCI Express x8 or x16 adapter cards. The User Manual provides details as to the interfaces of the adapter card, specifications, required software and firmware for operating the adapter card, and relevant documentation.
The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support Or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Rev 2.6 Mellanox Technologies...
Introduction Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con- nectX®-4 integrated circuit device. These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers, High-Performance Computing, and Embedded environments.
PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run Mellanox PeerDirect™ time. ConnectX-4 advanced acceleration technology enables higher cluster effi- ciency and scalability to tens of thousands of nodes. Rev 2.6 Mellanox Technologies...
• VMware • OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Connectivity • Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Rev 2.6 Mellanox Technologies...
• Auto-negotiates tox16, x8, x4, x2, or x1 • Support for MSI/MSI-X mechanisms LED Interface There is a one bi-color I/O LED per port located on the adapter card. For LED specifications, please refer to Section 7.12, “Adapter Card LED Operations,” on page Rev 2.6 Mellanox Technologies...
Bracket Installation Instructions The adapter card is usually shipped with a tall bracket installed. If this form factor is suitable for your requirements, you can skip the remainder of this section and move to Section 3.6, “Adapter Rev 2.6 Mellanox Technologies...
Please note that the following figures are for illustration purposes only. 1. Before installing the card, make sure that the system is off and the power cord is not con- nected to the server. Please follow proper electrical grounding procedures. 2. Open the system case. Rev 2.6 Mellanox Technologies...
Page 19
Hardware Installation 3. Locate an available PCI Express slot for the adapter card. 4. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. Rev 2.6 Mellanox Technologies...
Make sure that the connector locks in place. When installing cables make sure that the latches engage. Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card. Rev 2.6 Mellanox Technologies...
Please note that the following images are for illustration purposes only. 1. Verify that the system is powered off and unplugged. 2. Wait 30 seconds. 3. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). Rev 2.6 Mellanox Technologies...
3. Right click the mouse on your adapter's row and select properties to display the adapter card properties window. 4. Click the Details tab and select Hardware Ids (Windows 2012/R2) from the Properties pull- down menu. Rev 2.6 Mellanox Technologies...
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
Step 1. The following example shows a system with an installed Mellanox HCA: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 Download the ISO image to your host.
The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Rev 2.6 Mellanox Technologies...
At this point, Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine. 4.1.3.1 Installation Procedure Login to the installation machine as root. Step 1. Mount the ISO image on your machine. Step 2. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Rev 2.6 Mellanox Technologies...
Page 27
InfiniBand link is up. The utility also checks for and displays additional information such • HCA firmware version • Kernel architecture • Driver version • Number of active HCA ports along with their states • Node GUID Rev 2.6 Mellanox Technologies...
The path to the directory containing the log files will be displayed after running the installation script in the following format: "Logs dir: /tmp/MLNX_OFED_LINUX-<version>.<PID>.logs" Example: Logs dir: /tmp/MLNX_OFED_LINUX-x.x-x.logs 4.1.3.4 openibd Script As of MLNX_OFED v2.2-1.0.0 the script supports pre/post start/stop scripts: openibd Rev 2.6 Mellanox Technologies...
4.1.4 Uninstalling Mellanox OFED Use the script to uninstall the Mellanox OFED package. /usr/sbin/ofed_uninstall.sh The script is part of the RPM. ofed-scripts 4.1.5 Installing MLNX_OFED using YUM This type of installation is applicable to RedHat/OEL, Fedora, XenServer Operating Systems. Rev 2.6 Mellanox Technologies...
• Run # yum install mlnx-fw-updater • Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Section 4.1.7, “Updating Firmware After Installation,” on page 34. 4.1.5.3 Uninstalling Mellanox OFED using the YUM Tool Use the script to uninstall the Mellanox OFED package.
Step 3. "/etc/apt/sources.list.d/mlnx- with the following content: _ofed.list" deb file:/<path to extracted MLNX_OFED package> ./ Download and install Mellanox Technologies GPG-KEY. Step 4. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Check that the key was successfully imported.
OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http:// www.mellanox.com...
"\.ko$" | xargs strip -g After the signature has been removed, a message as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" Rev 2.6 Mellanox Technologies...
Obtain the machine architecture. Step 1. For Windows Server 2012 / 2012 R2 / 8.1 Client To go to the Start menu, position your mouse in the bottom-right corner of the Remote Desktop of your screen. Rev 2.6 Mellanox Technologies...
[Optional] Manually configure your setup to contain the logs option. Step 2. > MLNX_WinOF2-1_35_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. > MLNX_WinOF2-1_35_All_x64.exe /v" MT_SKIPFWUPGRD=1" 1. MT_SKIPFWUPGRD default value is False Rev 2.6 Mellanox Technologies...
Page 39
Driver Installation Click Next in the Welcome screen. Step 4. Read then accept the license agreement and click Next. Step 5. Rev 2.6 Mellanox Technologies...
Page 40
Driver Installation Select the target folder for the installation. Step 6. Rev 2.6 Mellanox Technologies...
Page 41
If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a Mellanox card, only the Mellanox card will be updated. Rev 2.6 Mellanox Technologies...
Page 42
Analyze tools - install the tools that can be used either to diagnose or analyze the InfiniBand envi- ronment. • SDK - contains the libraries and DLLs for developing InfiniBand application over IBAL. • Documentation - contains the User Manual and Release Notes Rev 2.6 Mellanox Technologies...
Page 43
Driver Installation • Management tools - installation tools used for management, such as mlxstat. b. Click Install to start the installation. Rev 2.6 Mellanox Technologies...
Page 44
Driver Installation In case firmware upgrade option was checked in step Step 7., you will be notified if a firm- Step 9. ware upgrade is required. Click Finish to complete the installation. Step 10. Rev 2.6 Mellanox Technologies...
Upon installation completion, the inf files can be located at: • %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the “Net- work adapters” menu. 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Rev 2.6 Mellanox Technologies...
Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. > MLNX_WinOF2-1_35_All_x64 /a • To extract only the driver files. > MLNX_WinOF2-1_35_All_x64 /a /vMT_DRIVERS_ONLY=1 Rev 2.6 Mellanox Technologies...
Page 47
Driver Installation Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Rev 2.6 Mellanox Technologies...
To complete the extraction, click Finish. Step 6. 4.2.6 Uninstalling Mellanox WinOF-2 Driver 4.2.6.1 Attended Uninstallation To uninstall MLNX_WinOF2 on a single node: Click Start -> Control Panel -> Programs and Features -> MLNX_WinOF2 -> Uninstall. Rev 2.6 Mellanox Technologies...
Extract the driver to a local directory (see Step 3. Section 4.2.5, “Extracting Files Without Running Installa- tion,” on page 46 Copy the contents of this directory to C:\WS2016TP5_Drivers. The directory contents should appear as follows: Rev 2.6 Mellanox Technologies...
To upgrade, it is recommended to run a script that will execute all the required upgrade commands. 4.2.9 Performance Tuning For further information on WinOF-2 performance, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Rev 2.6 Mellanox Technologies...
Page 53
Driver Installation Install the image according to the steps described in the MFT User Manual The following procedure requires custom boot image downloading, mounting and boot- ing from a USB device. Rev 2.6 Mellanox Technologies...
Device #2: Updating FW ... Done Perform cold reboot on the machine if the installation script performed firmware updates to your network adapter hardware. Otherwise, restart the driver by running: "/ etc/init.d/openibd restart. Log File: /var/log/mlxup/mlxup-yyyymmdd.log Rev 2.6 Mellanox Technologies...
• Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but with no communication • Check that both the adapter and its link are set to the same established speed and duplex settings Rev 2.6 Mellanox Technologies...
Once installed, run: (MFT) mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo To download the latest firmware version refer to Firmware Version http://www.mellanox.com/supportdownloader Upgrade cat /var/log/messages Collect Log File dmesg > system.log Rev 2.6 Mellanox Technologies...
Airflow Specifications on page 68 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Airflow Specifications on page 68 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Power and Environmental Non-operational -40°C to 70°C Humidity: 90% relative humidity Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Airflow Specifications on page 68 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Airflow Specifications on page 68 Safety: CB / cTUVus / CE Regulatory EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Airflow Specifications on page 68 Safety: CB / cTUVus / CE Regulatory EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Temperature Environmental Non-operational -40°C to 70°C Humidity: 90% relative humidity Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Airflow Specifications on page 68 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Airflow Specifications on page 68 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Airflow Specifications on page 68 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 2.6 Mellanox Technologies...
Solid Green Indicates a valid link with no active traffic Blinking Green Indicates a valid link with active traffic a. 1 Hz Blinking Amber occurs due to running a beacon command for locating the adapter card. Rev 2.6 Mellanox Technologies...
All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 3: Mechanical Drawing of the Single-port x8 Adapter Cards 142.24 68.90 Figure 4: Mechanical Drawing of the Dual-port x8 Adapter Cards 142.24 68.90 Rev 2.6 Mellanox Technologies...
MAC for the Ethernet protocol The revisions indicated on the labels in the following figure do not necessarily repre- sent the latest revisions of the card. Figure 9: MCX413A-BCAT Board Label (Example) Figure 10: MCX414A-BCAT Board Label (Example) Rev 2.6 Mellanox Technologies...
6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Rev 2.6 Mellanox Technologies...
Page 77
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Rev 2.6 Mellanox Technologies...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 2.6 Mellanox Technologies...
Page 79
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposi- tion au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Rev 2.6 Mellanox Technologies...
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Rev 2.6 Mellanox Technologies...
Page 81
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Rev 2.6 Mellanox Technologies...
6. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. Rev 2.6 Mellanox Technologies...
Page 83
8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1 Rev 2.6 Mellanox Technologies...
Need help?
Do you have a question about the ConnectX-4 MCX413A-BCAT and is the answer not in the manual?
Questions and answers