Connectx-3 pro vpi single and dual qsfp+ port adapter card (79 pages)
Summary of Contents for Mellanox Technologies MCX456M-ECAT
Page 1
ConnectX®-4 VPI Dual Port QSFP28 Socket-Direct Adapter Card User Manual for Dual-Socket Servers P/N: MCX456M-ECAT Rev 1.0 www.mellanox.com Mellanox Technologies...
Page 2
NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT (S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS -IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS.
About this Manual This User Manual describes Mellanox Technologies ConnectX®-4 VPI adapter card supporting Dual-Socket Servers. The kit includes an adapter card with dual QSFP28 ports with PCI Express x8 edge connector, an auxiliary PCIe connection card with PCI Express x8 edge connector and a Slim-Line SAS cable which connects both cards.
The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support Or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Mellanox Technologies Rev 1.0...
Introduction Introduction This is the User Guide for Mellanox Technologies VPI adapter cards based on the ConnectX®-4 integrated circuit device. Mellanox offers an alternate ConnectX-4 Socket-Direct™ card for serv- ers without x16 PCIe slots. The adapter’s 16-lane PCIe bus is split into two 8-lane buses, with one bus accessible through a PCIe x8 edge connector and the other bus through an x8 parallel connector to an Auxiliary PCIe Connection Card.
The following section provides the ordering part number, port speed, number of ports, and PCI Express speed. The adapter card and auxiliary PCIe connection card come with tall brackets. Table 3 - Dual Port EDR Adapter Card MCX456M-ECAT includes the following components: • Low Profile Adapter Card - Dual-port Ordering Part Number (OPN) •...
TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-4 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and decapsulate the overlay protocol. Rev 1.0 Mellanox Technologies...
• OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Connectivity • Interoperable with InfiniBand or 10/25/40/50/100 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Mellanox Technologies Rev 1.0...
Direct PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact Mellanox sup- port. Rev 1.0 Mellanox Technologies...
• Auto-negotiates to x8, x4, x2, or x1 • Support for MSI/MSI-X mechanisms LED Interface There is a one bi-color I/O LED per port located on the adapter card. For LED specifications, please refer to Section 7.3, “Adapter Card LED Operations,” on page Mellanox Technologies Rev 1.0...
4. It is strongly recommended to use an ESD strap or other antistatic devices. Pre-installation Checklist 1. Verify that your system meets the hardware and software requirements stated above. 2. Shut down your system if active. 3. After shutting down the system, turn off power and unplug the cord. Rev 1.0 Mellanox Technologies...
2. Screw on the bracket using the screws saved from the bracket removal procedure above. 3. Make sure that the LEDs are aligned onto the bracket holes. This only applies to the adapter card. 4. Use a torque driver to apply up to 2.9 lbf-in torque on the screws. Mellanox Technologies Rev 1.0...
4. Locate two available PCI Express slots on the server, one for the adapter card and one for the auxiliary PCIe connection card. For optimal thermal performance, it is preferable to place the auxiliary PCIe connec- tion card component-side facing the adapter card’s print-side. Rev 1.0 Mellanox Technologies...
Page 20
PCI Express slot until firmly seated. Do not use excessive force when seating the card, as this may damage the system or the auxiliary PCIe connection card. 6. Secure the auxiliary PCIe connection card using server's retention mechanisms. Mellanox Technologies Rev 1.0...
Page 21
Do not use excessive force when seating the card, as this may damage the system or the adapter. 8. Secure the adapter card using the server’s retention mechanisms. 9. Connect the slim line-SAS cable to the connector on the adapter card. Rev 1.0 Mellanox Technologies...
3. After inserting a cable into a port, the Yellow LED indicator on the adapter card will light when the physical connection is established (that is, when the unit is powered on and a cable Mellanox Technologies Rev 1.0...
3. Turn off the system and disconnect the power cord from the server. 3.8.2 Un-Installing the Cards Please note that the following figures are for illustration purposes only. 1. Verify that the system is powered off and unplugged. 2. Wait 30 seconds. Rev 1.0 Mellanox Technologies...
Page 24
4. Disengage the retention mechanisms (clips or screws) on adapter card. 5. Holding the card from its center, gently pull the adapter card from the PCI Express slot. 6. Disengage the retention mechanisms (clips or screws) on the auxiliary PCIe connection card. Mellanox Technologies Rev 1.0...
Page 25
Hardware Installation 7. Holding the card from its center, gently pull the auxiliary PCIe connection card from the PCI Express slot. 8. Disconnect the slim-line SAS cable from the connector on the auxiliary PCIe connection card. Rev 1.0 Mellanox Technologies...
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
Hardware Installation The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi- http://pci-ids.ucw.cz/read/PC/15b3 tory at 3.9.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella- nox Technologies”:...
Step 1. The following example shows a system with an installed Mellanox HCA: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 Download the ISO image to your host.
The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Rev 1.0 Mellanox Technologies...
Page 30
On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using is irrelevant on Ubuntu and "mlnx_add_kernel_support.sh" Debian distributions. Mellanox Technologies Rev 1.0...
Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed. Uninstalling the previous version of MLNX_OFED_LINUX Starting MLNX_OFED_LINUX-x.x.x installation ....Installation finished successfully. Attempting to perform Firmware update... Querying Mellanox devices firmware ... Rev 1.0 Mellanox Technologies...
This can be controlled by setting the variables below in the /etc/infiniband/openibd.conf file. OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh 4.1.3.5 Driver Load Upon System Boot Upon system boot, the Mellanox drivers will be loaded automatically. Rev 1.0 Mellanox Technologies...
# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt You can download it from http://www.mellanox.com => Products => Software => Infini- Band/VPI Drivers. Download and install Mellanox Technologies GPG-KEY: Step 3. The key can be downloaded via the following link: Mellanox Technologies Rev 1.0...
# sudo rpm --import RPM-GPG-KEY-Mellanox warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mel- lanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Is this ok [y/N]: Check that the key was successfully imported.
Page 36
If you have Operating Systems different than RHEL or SLES, or you have installed a ker- nel that is not supported by default in MLNX_OFED, you can use the mlnx_add_ker- script to build MLNX_OFED for your kernel. nel_support.sh Mellanox Technologies Rev 1.0...
To update the firmware to the version included in MLNX_OFED package, you can either: • Run # yum install mlnx-fw-updater • Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Section 4.1.7, “Updating Firmware After Installation,” on page 4.1.5.3 Uninstalling Mellanox OFED using the YUM Tool...
Step 3. "/etc/apt/sources.list.d/mlnx- with the following content: _ofed.list" deb file:/<path to extracted MLNX_OFED package> ./ Download and install Mellanox Technologies GPG-KEY. Step 4. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Check that the key was successfully imported.
OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http:// www.mellanox.com...
'mokutil' package is installed on your system • the system is booted in UEFI mode Download the x.509 public key. Step 1. # wget http://www.mellanox.com/downloads/ofed/mlnx_signing_key_pub.der Add the public key to the MOK list using the mokutil utility. Step 2. Mellanox Technologies Rev 1.0...
After the signature has been removed, a message as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" However, please note that a similar message as the following will still be presented: "my_module: module verification failed: signature and/or required key missing - taint-...
Enter the following command. > echo %PROCESSOR_ARCHITECTURE% On an x64 (64-bit) machine, the output will be “AMD64”. Go to the Mellanox WinOF-2 web page at: Step 2. http://www.mellanox.com => Products => InfiniBand/VPI Drivers => Windows SW/Drivers. Mellanox Technologies Rev 1.0...
[Optional] Manually configure your setup to contain the logs option. Step 2. > MLNX_WinOF2-1_35_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. > MLNX_WinOF2-1_35_All_x64.exe /v" MT_SKIPFWUPGRD=1" 1. MT_SKIPFWUPGRD default value is False Rev 1.0 Mellanox Technologies...
Page 44
Click Next in the Welcome screen. Step 4. Read then accept the license agreement and click Next. Step 5. Mellanox Technologies Rev 1.0...
Page 45
Driver Installation Select the target folder for the installation. Step 6. Rev 1.0 Mellanox Technologies...
Page 46
If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a Mellanox card, only the Mellanox card will be updated. Mellanox Technologies Rev 1.0...
Page 47
Analyze tools - install the tools that can be used either to diagnose or analyze the InfiniBand envi- ronment. • SDK - contains the libraries and DLLs for developing InfiniBand application over IBAL. • Documentation - contains the User Manual and Release Notes Rev 1.0 Mellanox Technologies...
Page 48
• Management tools - installation tools used for management, such as mlxstat. b. Click Install to start the installation. Mellanox Technologies Rev 1.0...
Page 49
Driver Installation In case firmware upgrade option was checked in Step 7., you will be notified if a firmware Step 9. upgrade is required. Click Finish to complete the installation. Step 10. Rev 1.0 Mellanox Technologies...
Upon installation completion, the inf files can be located at: • %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the “Net- work adapters” menu. 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Mellanox Technologies Rev 1.0...
Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. > MLNX_WinOF2-1_35_All_x64 /a • To extract only the driver files. > MLNX_WinOF2-1_35_All_x64 /a /vMT_DRIVERS_ONLY=1 Rev 1.0 Mellanox Technologies...
Page 52
Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Mellanox Technologies Rev 1.0...
To complete the extraction, click Finish. Step 6. 4.2.6 Uninstalling Mellanox WinOF-2 Driver 4.2.6.1 Attended Uninstallation To uninstall MLNX_WinOF2 on a single node: Click Start -> Control Panel -> Programs and Features -> MLNX_WinOF2 -> Uninstall. Rev 1.0 Mellanox Technologies...
Extract the driver to a local directory (see Section 4.2.5, “Extracting Files Without Running Step 3. Installation,” on page 51). Copy the contents of this directory to C:\WS2016TP5_Drivers. The directory contents should appear as follows: Mellanox Technologies Rev 1.0...
To upgrade, it is recommended to run a script that will execute all the required upgrade commands. 4.2.9 Performance Tuning For further information on WinOF-2 performance, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Rev 1.0 Mellanox Technologies...
If the user confirms, mlxup upgrades the firmware using embedded images. The latest mlxup executable and documentation are available from http://www.mellanox.com => Products => Software => Firmware Tools. Mellanox Technologies Rev 1.0...
• Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but with no • Check that both the adapter and its link are set to the same communication speed and duplex settings established Mellanox Technologies Rev 1.0...
Tool (MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo To download the latest firmware version refer to Firmware Version http://www.mellanox.com/supportdownloader Upgrade /var/log/messages Collect Log File dmesg > system.logF Rev 1.0 Mellanox Technologies...
Specifications Specifications MCX456M-ECAT Specifications Table 7 - MCX456M-ECAT Specification Table Low Profile Adapter Card Size: 6.6 in. x 2.71 in. (167.65mm x 68.90mm) Auxiliary PCIe Connection Card Size: 4.44 in. x 1.57 in. (113.0mm x 40.00mm) Physical Slim-Line SAS Cable: 5.9 in. x 1.02 in. (150mm x 25.95mm) Connector: Dual QSFP28 located on the adapter card (Copper and optical) InfiniBand: IBTA v1.3...
Passive Active 3.5W Passive Active 3.5W MCX456M-ECAT a. Cells marked with (–) are not applicable Adapter Card LED Operations There is a one bi-color I/O LED per port located on the adapter card to indicate physical and log- ical link status. See...
All the mechanical tolerances are +/- 0.1mm. Figure 3: Mechanical Drawing of the Dual-port Adapter Card 167.65 68.9 Figure 4: Mechanical Drawing of the Auxiliary PCIe Connection Card 113.00 40.00 Figure 5: Mechanical Drawing of the Slim-Line SAS Cable 150.00 25.95 Rev 1.0 Mellanox Technologies...
Table 9 - Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name TXN_2 Tx2n TXP_2 Tx2p TXN_4 Tx4n TXP_4 Tx4p ModSelL_Port0 ModSelL ResetL_Port0 ResetL VccRx RXP_3 Rx3p Rev 1.0 Mellanox Technologies...
Tx1n PCI Express x8 Edge Connector Pinout The adapter cards support PCI Express 3.0 through two PCIe x8 connectors; a standard PCI Express x8 edge connector and Slim-Line SAS connector pinout according to SFF/9401 stan- dards. Mellanox Technologies Rev 1.0...
GUID and a MAC (derived from the GUID). The revisions indicated on the labels in the following figure do not necessarily repre- sent the latest revisions of the card. Figure 10: MCX456M-ECAT Board Label (Example) Mellanox Technologies Rev 1.0...
6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Rev 1.0 Mellanox Technologies...
Page 74
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Mellanox Technologies Rev 1.0...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 1.0 Mellanox Technologies...
Page 76
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Mellanox Technologies Rev 1.0...
6. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. Rev 1.0 Mellanox Technologies...
Page 78
8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1 Mellanox Technologies Rev 1.0...
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Rev 1.0 Mellanox Technologies...
Page 80
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Mellanox Technologies Rev 1.0...
Need help?
Do you have a question about the MCX456M-ECAT and is the answer not in the manual?
Questions and answers