Connectx-3 ethernet single and dual qsfp+ port adapter card (65 pages)
Summary of Contents for Mellanox Technologies ConnectX-5 Ex
Page 1
Mellanox ConnectX®-5 and ConnectX®-5 Ex Ethernet SFP28 and QSFP28 Ports Adapter Cards User Manual P/N: MCX512A-ACAT, MCX511F-ACAT, MCX512F-ACAT,MCX516A-BDAT, MCX515A-GCAT, MCX516A-GCAT, MCX515A-CCAT, MCX516A-CCAT, MCX516A-CDAT Rev 1.7 www.mellanox.com Mellanox Technologies...
Page 2
ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT (S) AND/OR THE SYSTEM USING IT . THEREFORE , MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY .
Page 5
Sicherheitshinweise (Warnings in German) ..... 83 Advertencias de seguridad para la instalación (Warnings in Spanish) 85 Rev 1.7 Mellanox Technologies...
About This Manual This User Manual describes Mellanox Technologies ConnectX®-5 and ConnectX®-5 Ex Ether- net Single and Dual SFP28 and QSFP28 port PCI Express x8/x16 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operat- ing the board, and relevant documentation.
Page 10
• Tel: +1.408.916.0055 Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller. Rev 1.7 Mellanox Technologies...
Introduction Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con- nectX®-5 integrated circuit device. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Centers and High-Performance Computing environments.
The following section provides the ordering part number, port speed, number of ports, and PCI Express speed. Each adapter comes with two bracket heights - short and tall. Table 4 - ConnectX-5 Ex Ethernet Adapter Cards Model ConnectX-5 Ex Ethernet Adapter Cards...
Adapter functionality enabling reduced CPU overhead allowing more available CPU for computation tasks. 2(TM) CPU Offload Open VSwitch (OVS) offload using ASAP • Flexible match-action flow tables • Tunneling encapsulation / decapsulation Rev 1.7 Mellanox Technologies...
• VMware • OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Connectivity • Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Rev 1.7 Mellanox Technologies...
7. Ethernet traffic is transmitted through the cards' SFP28 and QSFP28 connectors. PCI Express Interface Table 8 describes the supported PCIe interface in ConnectX-5 and ConnectX-5 Ex adapter cards. Table 8 - PCI Express Interface ConnectX-5 Supported PCIe Interface Features PCIe Gen 3.0 (1.1 and 2.0 compatible)
Voltage Regulators The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These voltage supply pins feed on-board regulators that provide the necessary power to the vari- ous components on the card. Rev 1.7 Mellanox Technologies...
Install ConnectX-5 adapter card in the system Refer to “Installation Instructions” Connect cables or modules to the card Refer to “Cables and Modules” Identify ConnectX-5 adapter card in the system Refer to “Identify the Card in Your System” Rev 1.7 Mellanox Technologies...
Please refer to “Specifications” airflow numbers per OPN. All cards in the system should be planned with the same airflow direction. Table 11 - Airflow Color Legend Direction Description From the heatsink to the network ports Rev 1.7 Mellanox Technologies...
Check the parts for visible damage that may have occurred during ship- ping. Please note that the cards must be placed on an antistatic surface. For package contents please refer to Section 1.2, “Package Content”, on page 12. Rev 1.7 Mellanox Technologies...
Due to risk of damaging the EMI fingers, it is not recommended to replace the bracket more than three times. To replace the brackets you will need the following parts: Rev 1.7 Mellanox Technologies...
Please note that the following figures are for illustration purposes only. 1. Before installing the card, make sure that the system is off and the power cord is not con- nected to the server. Please follow proper electrical grounding procedures. 2. Open the system case. Rev 1.7 Mellanox Technologies...
Page 22
Hardware Installation 3. Place the adapter in an available PCI Express slot. 4. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. Rev 1.7 Mellanox Technologies...
Make sure that the connector locks in place. When installing cables make sure that the latches engage. Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card. Rev 1.7 Mellanox Technologies...
Please note that the following images are for illustration purposes only. 1. Verify that the system is powered off and unplugged. 2. Wait 30 seconds. 3. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). Rev 1.7 Mellanox Technologies...
1. Open Device Manager on the server. Click Start => Run, and then enter “devmgmt.msc”. 2. Expand System Devices and locate your Mellanox ConnectX-5 adapter card. 3. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window. Rev 1.7 Mellanox Technologies...
“PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1018 (for ConnectX-5) or 1019 (for ConnectX-5 Ex) – this is a valid Mellanox Technologies PCI Device ID.
Hardware Installation 3.8.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mellanox Technologies”: lspci |grep -i Mellanox Network controller: Mellanox Technologies MT28800 Family [ConnectX-5] Rev 1.7 Mellanox Technologies...
Step 1. The following example shows a system with an installed Mellanox adapter cards: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 Download the ISO image to your host.
NFS. To install on all the cluster nodes, use cluster-aware tools (such as pdsh). 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Rev 1.7 Mellanox Technologies...
Page 30
For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be running on one of the fabric nodes. At this point, Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine. For the list of installation options, run: ./mlnxofedinstall --h Rev 1.7 Mellanox Technologies...
./mlnxofedinstall --without-dkms --add-kernel-support --kernel <kernel version in chroot> --without-fw-update --force For example: ./mlnxofedinstall --without-dkms --add-kernel-support --kernel 3.13.0-85-generic --without-fw-update --force Note that the path to kernel sources (--kernel-sources) should be added if the sources are not in their default location. Rev 1.7 Mellanox Technologies...
Page 32
• ConnectX-5/ConnectX-5 Ex - perform a standard reboot Otherwise, restart the driver by running: "/etc/init.d/openibd restart" After installation completion, information about the Mellanox OFED installation, such as prefix,...
"Logs dir: /tmp/MLNX_OFED_LINUX-<version>.<PSID>.logs" Example: Logs dir: /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0.IBMM2150110033.logs 4.1.3.5 openibd Script As of MLNX_OFED v2.2-1.0.0 the script supports pre/post start/stop scripts: openibd This can be controlled by setting the variables below in the /etc/infiniband/openibd.conf file. OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Rev 1.7 Mellanox Technologies...
Not applicable to this system configuration. This can occur when the required hardware is not present on the system. Prerequisites are not met. For example, missing the required software installed or the hardware is not configured correctly. Failed to start the mst driver Rev 1.7 Mellanox Technologies...
RHEL and SLES. For other operating systems, kernel module RPM packages are provided only for the operating systems' default kernel. In this case, the group RPM packages have the supported kernel version in their package's name. Rev 1.7 Mellanox Technologies...
0:0.4.9-9 rds-devel.x86_64 0:2.0.7-1.12 rds-tools.x86_64 0:2.0.7-1.12 sdpnetstat.x86_64 0:1.60-26 srptools.x86_64 0:1.0.2-12 Complete! 4.1.5.3 Uninstalling Mellanox OFED Using the YUM Tool Use the script to uninstall the Mellanox OFED package. The /usr/sbin/ofed_uninstall.sh script is part of the ofed-scripts RPM. Rev 1.7 Mellanox Technologies...
Step 3. with the following content: "/etc/apt/sources.list.d/mlnx_ofed.list" # deb file:/<path to extracted MLNX_OFED package>/DEBS ./ Download and install Mellanox Technologies GPG-KEY. Step 4. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Check that the key was successfully imported.
To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Section 5, “Updating Adapter Card Firmware”, on page 58.
OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case that you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http://www.mellanox.com > Support > Firmware Download).
| xargs rpm -ql | grep "\.ko$" | xargs strip -g After the signature has been removed, a massage as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" Rev 1.7...
Depending on the application of the user's system, it may be necessary to modify the default con- figuration of network adapters based on the ConnectX® adapters. In case that tuning is required, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Rev 1.7 Mellanox Technologies...
Step 2. http://www.mellanox.com => Products =>Ethernet Drivers => Windows SW/Drivers. Download the .exe image according to the architecture of your machine (see Step 1). The Step 3. name of the .exe is in the following format MLNX_WinOF2-<version>_x<arch>.exe. Rev 1.7 Mellanox Technologies...
[Optional] Manually configure your setup to contain the logs option (replace “LogFile” with Step 2. the relevant directory): MLNX_WinOF2-1_10_<revision_version>_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. MLNX_WinOF2-1_10_<revision_version>_All_x64.exe /v" MT_SKIPFWUPGRD=1" 1. MT_SKIPFWUPGRD default value is False Rev 1.7 Mellanox Technologies...
Page 45
Driver Installation Click Next in the Welcome screen. Step 4. Read then accept the license agreement and click Next. Step 5. Rev 1.7 Mellanox Technologies...
Page 46
Driver Installation Select the target folder for the installation. Step 6. Rev 1.7 Mellanox Technologies...
Page 47
• Performance tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat Rev 1.7 Mellanox Technologies...
Page 48
Driver Installation • Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd b. Click Next to install the desired tools. Click Install to start the installation. Step 9. Rev 1.7 Mellanox Technologies...
Page 49
Driver Installation In case that firmware upgrade option was checked in Step 7, you will be notified if a firm- Step 10. ware upgrade is required. Click Finish to complete the installation. Step 11. Rev 1.7 Mellanox Technologies...
Upon installation completion, the inf files can be located at: • %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the “Network adapters” menu. 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Rev 1.7 Mellanox Technologies...
Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. MLNX_WinOF2-1_10_All_x64 /a To extract only the driver files. MLNX_WinOF2-1_10_All_x64 /a /vMT_DRIVERS_ONLY=1 Rev 1.7 Mellanox Technologies...
Page 52
Driver Installation Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Rev 1.7 Mellanox Technologies...
Page 53
Driver Installation Click Install to extract this folder, or click Change to install to a different folder. Step 5. To complete the extraction, click Finish. Step 6. Rev 1.7 Mellanox Technologies...
Download the driver (MLNX_WinOF2_MLNX_WinOF2-1_64_mlx5_All_win2016_x64_- Step 2. fre_1_64_15407.exe). Extract the driver to a local directory (see Section 4.2.5, “Extracting Files Without Running Step 3. Installation”, on page 51). Copy the contents of this directory to C:\WS2016TP5_Drivers. Step 4. Rev 1.7 Mellanox Technologies...
When upgrading the driver on a server where the remote connection was done over Mella- nox device, there will be a loss of connectivity. To upgrade, it is recommended to run a script that will execute all the required upgrade commands. Rev 1.7 Mellanox Technologies...
3. Reboot the machine. 4. Verify the driver was installed successfully. esxcli software vib list | grep nmlx nmlx5-core 4.16.8.8-1OEM.650.0.0.4240417 PartnerSupported 2017-01-31 nmlx5-rdma 4.16.8.8-1OEM.650.0.0.4240417 PartnerSupported 2017-01-31 After the installation process, all kernel modules are loaded automatically upon boot. Rev 1.7 Mellanox Technologies...
• ESXi 6.5 File: mft-4.6.0.48-10EM-650.0.0.4598673.x86_64.vib MD5SUM: 0804cffe30913a7b4017445a0f0adbe1 2. Install the image according to the steps described in the MFT User Manual. The following procedure requires custom boot image downloading, mounting and boot- ing from a USB device. Rev 1.7 Mellanox Technologies...
0000e41d2da206d4 Versions: Current Available 16.24.1000 16.24.1000 Status: Up to date Perform FW update? [y/N]: y Device #1: Up to date Device #2: Updating FW ... Done Restart needed for updates to take effect. Log File: /var/log/mlxup/mlxup-yyyymmdd.log Rev 1.7 Mellanox Technologies...
Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but with no • Check that both the adapter and its link are set to the same speed and communication established duplex settings Rev 1.7 Mellanox Technologies...
-d <mst_device> q ibstat Ports Information ibv_devinfo To download the latest firmware version refer to Firmware Version Upgrade http://www.mellanox.com/supportdownloader cat /var/log/messages dmesg >> system.log Collect Log File journalctl (Applicable on new operating systems) cat /var/log/syslog Rev 1.7 Mellanox Technologies...
Specifications Specifications Table 20 - Direct links for Specifications Table per OPN Marketing Description MCX516A-BDAT MCX516A-BDAT Specifications on page 63 ConnectX-5 Ex MCX516A-CDAT MCX516A-CDAT Specifications on page 64 MCX512A-ACAT MCX512A-ACAT Specifications on page 65 MCX512F-ACAT MCX512F-ACAT Specifications on page 66...
Airflow Specifications on page 71 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS Compliant a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.7 Mellanox Technologies...
Airflow Specifications on page 71 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS Compliant a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.7 Mellanox Technologies...
Airflow Specifications on page 71 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS Compliant a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.7 Mellanox Technologies...
Airflow Specifications on page 71 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS Compliant a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.7 Mellanox Technologies...
Airflow Specifications on page 71 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS Compliant a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.7 Mellanox Technologies...
Airflow Specifications on page 71 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS Compliant a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.7 Mellanox Technologies...
Airflow Specifications on page 71 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS Compliant a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.7 Mellanox Technologies...
Airflow Specifications on page 71 Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM Regulatory RoHS: RoHS Compliant a. Typical power for ATIS traffic load. b. For both operational and non-operational states. Rev 1.7 Mellanox Technologies...
Passive Active 1.5W Active 2.5W Active 3.5W Active 5W MCX516A-BDAT – MCX516A-CDAT 1000 1400 MCX512A-ACAT – – – MCX512F-ACAT – – MCX515A-GCAT – MCX516A-GCAT – MCX515A-CCAT MCX516A-CCAT 1300 a. Cells marked with (–) are not applicable Rev 1.7 Mellanox Technologies...
7.10 Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. For the 3D Model of the card, please refer to Models. Figure 5: Mechanical Drawing of Dual-port x16 Adapter Cards 142.24mm 68.9mm Rev 1.7 Mellanox Technologies...
MAC for the Ethernet protocol . The revision indicated on the labels in the following figures do not necessarily represent the latest revision of the card. Figure 9: MCX512A-ACAT Board Label (Example) Figure 10: MCX512F-ACAT Board Label (Example) Rev 1.7 Mellanox Technologies...
6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Rev 1.7 Mellanox Technologies...
Page 80
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Rev 1.7 Mellanox Technologies...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 1.7 Mellanox Technologies...
Page 82
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Rev 1.7 Mellanox Technologies...
6. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. Rev 1.7 Mellanox Technologies...
Page 84
8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1 Rev 1.7 Mellanox Technologies...
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Rev 1.7 Mellanox Technologies...
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Rev 1.7 Mellanox Technologies...
Need help?
Do you have a question about the ConnectX-5 Ex and is the answer not in the manual?
Questions and answers