Connectx-3 pro vpi single and dual qsfp+ port adapter card (79 pages)
Summary of Contents for Mellanox Technologies Mellanox ConnectX MCX556M-ECAT-S25
Page 1
Mellanox ConnectX®-5 VPI Dual Port Socket Direct Adapter Card User Manual for Dual-Socket Servers P/N: MCX556M-ECAT-S25 Rev 1.6 www.mellanox.com Mellanox Technologies...
Page 2
ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT (S) AND/OR THE SYSTEM USING IT . THEREFORE , MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY .
Page 5
Appendix D Avertissements de sécurité d’installation (Warnings in French) 78 Appendix E Sicherheitshinweise (Warnings in German) ....80 Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 82 Rev 1.6 Mellanox Technologies...
Table 18: Slim-Line SAS Pinout Connector on the Adapter Card ..... . 69 Table 19: Slim-Line SAS Pinout Connector on the Auxiliary PCIe Connection Card ..71 Rev 1.6 Mellanox Technologies...
• Updated Product Overview on page 10 • Updated Hardware on page 17 • Updated Cable Installation on page 21 April 2017 • Updated Linux on page • Updated Windows Driver on page 44 March 2017 First release Mellanox Technologies...
About This Manual This User Manual describes Mellanox Technologies ConnectX®-5 VPI adapter card supporting Dual-Socket Servers. The kit includes an adapter card with dual QSFP28 ports with PCI Express x8 edge connector, an auxiliary PCIe connection card with PCI Express x8 edge connector and a Slim-Line SAS cable which connects both cards.
Page 10
The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support Or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Rev 1.6 Mellanox Technologies...
Introduction Introduction This is the User Guide for Mellanox Technologies VPI adapter cards based on the ConnectX®-5 integrated circuit device. Mellanox offers an alternate ConnectX-5 Socket Direct™ card for serv- ers without x16 PCIe slots. The adapter’s 16-lane PCIe bus is split into two 8-lane buses, with one bus accessible through a PCIe x8 edge connector and the other bus through an x8 parallel connector to an Auxiliary PCIe Connection Card.
Full performance of the adapter card (100Gb/s) is possible only when installing both the adapter card and the auxiliary PCIe connection card in the dual PCIe x8 connectors. Please refer to the performance optimization document for further guidance: Performance Optimization. Rev 1.6 Mellanox Technologies...
ConnectX-5 Socket Direct adapter card package contents. For MCX556M-ECAT-S25, use the 25cm Slim-line SAS cable (HAR000629) and its Auxil- iary PCIe connection card (SA002142). Note that the Adapter and Auxiliary PCIe connection cards are shipped with assembled tall brackets. Rev 1.6 Mellanox Technologies...
VXLAN. While this solves network scalability issues, it hides the TCP Overlay Networks packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol. Rev 1.6 Mellanox Technologies...
Page 15
• Adaptive Routing on Reliable Transport Accelerations • Burst Buffer Offloads for Background Checkpointing a. This section describes hardware features and capabilities. Please refer to the driver release notes for feature availabil- ity. See Related Documentation. Rev 1.6 Mellanox Technologies...
Direct PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact Mellanox Sup- port. Rev 1.6 Mellanox Technologies...
• 2.5, 5.0, or 8.0 GT/s link rate x168 • Support for MSI/MSI-X mechanisms LED Interface There is one bi-color I/O LED per port located on the adapter card.For LED specifications, please refer to Section 7.3, “Adapter Card LED Operations”, on page Rev 1.6 Mellanox Technologies...
3. Verify that the system is powered off and is unplugged. 4. It is strongly recommended to use an ESD strap or other antistatic devices. Pre-Installation Checklist 1. Verify that your system meets the hardware and software requirements stated above. Rev 1.6 Mellanox Technologies...
2. Screw on the bracket using the screws saved from the bracket removal procedure above. 3. Make sure that the LEDs on the adapter card are aligned onto the bracket holes. 4. Use a torque driver to apply up to 2 lbs-in torque on the screws Rev 1.6 Mellanox Technologies...
4. Locate two available PCI Express slots on the server, one for the adapter card and one for the auxiliary PCIe connection card. For optimal thermal performance, it is preferable to place the auxiliary PCIe connec- tion card component-side facing the adapter card’s print-side. Rev 1.6 Mellanox Technologies...
Page 21
PCI Express slot until firmly seated. Do not use excessive force when seating the card, as this may damage the system or the auxiliary PCIe connection card. 6. Secure the auxiliary PCIe connection card using the server's retention mechanisms. Rev 1.6 Mellanox Technologies...
Page 22
Do not use excessive force when seating the card, as this may damage the system or the adapter. 8. Secure the adapter card using the server’s retention mechanisms. 9. Connect the slim line-SAS cable to the connector on the adapter card. Rev 1.6 Mellanox Technologies...
Make sure that the connector locks in place. When installing cables make sure that the latches engage. Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card. Rev 1.6 Mellanox Technologies...
2. It is strongly recommended to use an ESD strap or other antistatic devices. 3. Turn off the system and disconnect the power cord from the server. 3.7.2 Un-Installing the Cards 1. Verify that the system is powered off and unplugged. 2. Wait 30 seconds. Rev 1.6 Mellanox Technologies...
Page 25
4. Disengage the retention mechanisms (clips or screws) on adapter card. 5. Holding the card from its center, gently pull the adapter card from the PCI Express slot. 6. Disengage the retention mechanisms (clips or screws) on the auxiliary PCIe connection card. Rev 1.6 Mellanox Technologies...
Page 26
Hardware Installation 7. Holding the card from its center, gently pull the auxiliary PCIe connection card from the PCI Express slot. 8. Disconnect the slim-line SAS cable from the connector on the auxiliary PCIe connection card. Rev 1.6 Mellanox Technologies...
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1018 (for ConnectX-5) – this is a valid Mellanox Technologies PCI Device ID.
You will be prompted to acknowledge the deletion of the old packages. Pre-existing configuration files will be saved with the extension “.conf.rpmsave”. 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Rev 1.6 Mellanox Technologies...
Page 30
For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be running on one of the fabric nodes. At this point, Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine. For the list of installation options, run: ./mlnxofedinstall --h Rev 1.6 Mellanox Technologies...
./mlnxofedinstall --without-dkms --add-kernel-support --kernel <kernel version in chroot> --without-fw-update --force For example: ./mlnxofedinstall --without-dkms --add-kernel-support --kernel 3.13.0-85-generic --without-fw-update --force Note that the path to kernel sources (--kernel-sources) should be added if the sources are not in their default location. Rev 1.6 Mellanox Technologies...
Page 32
Step 4. plete the step relevant to your adapter card type to load the firmware: • Socket Direct or Multi-Host cards - perform a cold reboot (power cycle) Otherwise, restart the driver by running: "/etc/init.d/openibd restart" Rev 1.6 Mellanox Technologies...
Page 33
Most of the Mellanox OFED components can be configured or reconfigured after the installation, by modifying the relevant configuration files. See the relevant chapters in this manual for details. The list of the modules that will be loaded automatically upon boot can be found in the /etc/ file. infiniband/openib.conf Rev 1.6 Mellanox Technologies...
Logs dir: /tmp/MLNX_OFED_LINUX-x.x-x.logs 4.1.3.5 openibd Script As of MLNX_OFED v2.2-1.0.0 the script supports pre/post start/stop scripts: openibd This can be controlled by setting the variables below in the /etc/infiniband/openibd.conf file. OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh Rev 1.6 Mellanox Technologies...
To prevent this behavior, update the initramfs using the operating systems’ standard tools. Note: The process of updating the initramfs will add the blacklists from step 1, and will pre- vent the kernel from loading the modules automatically. Rev 1.6 Mellanox Technologies...
PCI function 0000:84:00.0. MT4119 is the PCI Device ID of the Mellanox ConnectX-5 adapters family. Check to which NUMA each netdevice is connected to: Step 2. [root@gen-l-vrt-203 ~]# cat /sys/class/net/p2p1/device/numa_node [root@gen-l-vrt-203 ~]# cat /sys/class/net/p5p1/device/numa_node Rev 1.6 Mellanox Technologies...
The script is part of the RPM. ofed-scripts 4.1.5 Installing MLNX_OFED Using YUM This type of installation is applicable to RedHat/OL, Fedora, XenServer Operating Systems. 4.1.5.1 Setting up MLNX_OFED YUM Repository Log into the installation machine as root. Step 1. Rev 1.6 Mellanox Technologies...
Page 39
# sudo rpm --import RPM-GPG-KEY-Mellanox warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mel- lanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Is this ok [y/N]: Check that the key was successfully imported.
To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in 4.1.5.3 Uninstalling Mellanox OFED Using the YUM Tool Use the script to uninstall the Mellanox OFED package.
Step 3. with the following content: "/etc/apt/sources.list.d/mlnx_ofed.list" # deb file:/<path to extracted MLNX_OFED package>/DEBS ./ Download and install Mellanox Technologies GPG-KEY. Step 4. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Check that the key was successfully imported.
To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in 4.1.5.7 Uninstalling Mellanox OFED Using the apt-get Tool Use the script to uninstall the Mellanox OFED package.
Manager.efi enter the password you previously associated with this request and confirm the enrollment. Once done, the public key is added to the MOK list, which is persistent. Once a key is in the MOK list, Rev 1.6 Mellanox Technologies...
After the signature has been removed, a massage as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" However, please note that a similar message as the following will still be presented: "my_module: module verification failed: signature and/or required key missing - taint-...
=> Products => InfiniBand/VPI Drivers => Windows SW/Driv- ers. Download the image according to the architecture of your machine (see Step 1). The name of Step 3. the .exe is in the following format MLNX_WinOF2-<version>_x.exe. Rev 1.6 Mellanox Technologies...
[Optional] Manually configure your setup to contain the logs option (replace “LogFile” with Step 2. the relevant directory):. MLNX_WinOF2-1_10_<revision_version>_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. MLNX_WinOF2-1_10_<revision_version>_All_x64.exe /v" MT_SKIPFWUPGRD=1" 1. MT_SKIPFWUPGRD default value is False Rev 1.6 Mellanox Technologies...
Page 48
Driver Installation Click Next in the Welcome screen. Step 4. Read then accept the license agreement and click Next. Step 5. Rev 1.6 Mellanox Technologies...
Page 49
Driver Installation Select the target folder for the installation. Step 6. Rev 1.6 Mellanox Technologies...
Page 50
• Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat Rev 1.6 Mellanox Technologies...
Page 51
Driver Installation • Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd b. Click Next to install the desired tools. Click Install to start the installation. Step 9. Rev 1.6 Mellanox Technologies...
Page 52
Driver Installation In case that firmware upgrade option was checked in Step 7, you will be notified if a firm- Step 10. ware upgrade is required (See Click Finish to complete the installation. Step 11. Rev 1.6 Mellanox Technologies...
Upon installation completion, the inf files can be located at: • %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the “Network adapters” menu. 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Rev 1.6 Mellanox Technologies...
Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. MLNX_WinOF2-1_10_All_x64 /a To extract only the driver files. MLNX_WinOF2-1_10_All_x64 /a /vMT_DRIVERS_ONLY=1 Rev 1.6 Mellanox Technologies...
Page 55
Driver Installation Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Rev 1.6 Mellanox Technologies...
Page 56
Driver Installation Click Install to extract this folder, or click Change to install to a different folder. Step 5. To complete the extraction, click Finish. Step 6. Rev 1.6 Mellanox Technologies...
Download the driver (MLNX_WinOF2_MLNX_WinOF2-1_64_mlx5_All_win2016_x64_- Step 2. fre_1_64_15407.exe). Extract the driver to a local directory (see Section 4.2.5, “Extracting Files Without Running Step 3. Installation”, on page 52). Copy the contents of this directory to C:\WS2016TP5_Drivers. Step 4. Rev 1.6 Mellanox Technologies...
When upgrading the driver on a server where the remote connection was done over Mella- nox device, there will be a loss of connectivity. To upgrade, it is recommended to run a script that will execute all the required upgrade commands. Rev 1.6 Mellanox Technologies...
Base MAC: 0000e41d2da206d4 Versions: Current Available 16.00.0000 16.00.0000 Status: Update required Perform FW update? [y/N]: y Device #1: Up to date Device #2: Updating FW ... Done Restart needed for updates to take effect. Log File: /var/log/mlxup/mlxup-yyyymmdd.log Rev 1.6 Mellanox Technologies...
Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but with no • Check that both the adapter and its link are set to the same speed and communication established duplex settings Rev 1.6 Mellanox Technologies...
-d <mst_device> q ibstat Ports Information ibv_devinfo To download the latest firmware version refer to Firmware Version Upgrade http://www.mellanox.com/supportdownloader cat /var/log/messages dmesg >> system.log Collect Log File journalctl (Applicable on new operating systems) cat /var/log/syslog Rev 1.6 Mellanox Technologies...
The ConnectX-5 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox Infini- Band product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. Rev 1.6 Mellanox Technologies...
Indicates a valid logical (data activity) link with no active traffic Blinking Green Indicates a valid logical link with active traffic a. 1 Hz Blinking Amber occurs due to running a beacon command for locating the adapter card. Rev 1.6 Mellanox Technologies...
All the mechanical tolerances are +/- 0.1mm. Fore the 3D Model of the card, please refer to Models. Figure 3: Mechanical Drawing of the Dual-port Adapter Card 167.65 68.9 Figure 4: Mechanical Drawing of the Auxiliary PCIe Connection Card 113.00 40.00 Rev 1.6 Mellanox Technologies...
Transmitter Non-Inverted Data Input Ground ModSelL Module Select ResetL Module Reset Vcc Rx +3.3 V Power supply receiver 2-wire serial interface clock 2-wire serial interface data Ground Rx3p Receiver Non-Inverted Data Output Rx3n Receiver Inverted Data Output Rev 1.6 Mellanox Technologies...
Page 69
+3.3 V Power supply transmitter Vcc1 +3.3 V Power Supply LPMode Low Power Mode Ground Tx3p Receiver Inverted Data Output Tx3n Receiver Non-Inverted Data Output Ground Tx1p Receiver Inverted Data Output Tx1n Receiver Non-Inverted Data Output Ground Rev 1.6 Mellanox Technologies...
The adapter cards support PCI Express 3.0 two PCIe x8 connectors; a standard PCI Express x8 edge connector and Slim-Line SAS connector pinout according to SFF/9401 standards.PCIe x16 Connector Pinout. Figure 9: PCIe x8 Edge Connector Pinout Applies to both to the adapter card and the auxiliary PCIe connection card. Rev 1.6 Mellanox Technologies...
GUID and a MAC (derived from the GUID).. The revision indicated on the labels in the following figures do not necessarily represent the latest revision of the card. Figure 10: MCX556M-ECAT-S25 Board Label (Example) Figure 11: Auxiliary PCIe Connection Card Board Label Rev 1.6 Mellanox Technologies...
6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Rev 1.6 Mellanox Technologies...
Page 79
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Rev 1.6 Mellanox Technologies...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 1.6 Mellanox Technologies...
Page 81
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Rev 1.6 Mellanox Technologies...
6. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. Rev 1.6 Mellanox Technologies...
Page 83
8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1 Rev 1.6 Mellanox Technologies...
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Rev 1.6 Mellanox Technologies...
Page 85
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Rev 1.6 Mellanox Technologies...
Need help?
Do you have a question about the Mellanox ConnectX MCX556M-ECAT-S25 and is the answer not in the manual?
Questions and answers