Connectx-3 pro vpi single and dual qsfp+ port adapter card (79 pages)
Summary of Contents for Mellanox Technologies MCX453A-FCAT
Page 1
ConnectX®-4 VPI Single and Dual Port QSFP28 Adapter Card User Manual P/N: MCX453A-FCAT, MCX454A-FCAT, MCX455A-FCAT, MCX456A-FCAT, MCX455A-ECAT, MCX456A-ECAT Rev 2.0 www.mellanox.com Mellanox Technologies...
Page 2
ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT (S) AND/OR THE SYSTEM USING IT . THEREFORE , MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY .
February 2016 • Updated speed specifications in the following sections: • Product Overview on page 12. Features on page 14 • MCX453A-FCAT Specifications on page 56 • MCX454A-FCAT Specifications on page 57 • MCX455A-FCAT Specifications on page 58 • MCX456A-FCAT Specifications on page 59 •...
Page 9
Card on page 72. • Updated the following Board Labels: • MCX453A-FCAT Board Label (Example) on page 72. • MCX454A-FCAT Board Label (Example) on page 72. • MCX455A-FCAT Board Label (Example) on page 73. • MCX456A-FCAT Board Label (Example) on page 73.
About this Manual This User Manual describes Mellanox Technologies ConnectX®-4 Single and Dual QSFP28 port PCI Express x16 adapter cards. The User Manual provides details as to the interfaces of the adapter card, specifications, required software and firmware for operating the adapter card, and relevant documentation.
The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support Or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Rev 2.4 Mellanox Technologies...
Introduction Introduction This is the User Guide for Mellanox Technologies VPI adapter cards based on the ConnectX®-4 integrated circuit device. These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers, High-Performance Computing, and Embedded environments.
Ethernet: 1/10/25/40/50/100 Gb/s Network Connector Types Single or Dual-port QSFP28 PCI Express (PCIe) SerDes Speed PCIe 3.0 x16 8GT/s RoHS Adapter IC Part Number MT27704A0-FDCF-EV MT27708A0-FDCF-EV Device ID 4115 for Physical Function (PF) 4116 for Virtual Function (VF) Rev 2.4 Mellanox Technologies...
TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-4 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and decapsulate the overlay protocol. Rev 2.4 Mellanox Technologies...
• OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Connectivity • Interoperable with EDR InfiniBand or 1/10/25/40/50/100 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Rev 2.4 Mellanox Technologies...
• Auto-negotiates to x16, x8, x4, x2, or x1 • Support for MSI/MSI-X mechanisms LED Interface There is a one bi-color I/O LED per port located on the adapter card. For LED specifications, please refer to Section 7.8, “Adapter Card LED Operations,” on page Rev 2.4 Mellanox Technologies...
Section 3.5, “Card Installation Instructions,” on page 18. If you need to replace it with the short bracket that is included in the shipping box, please follow the instructions in this section. Rev 2.4 Mellanox Technologies...
Please note that the following figures are for illustration purposes only. 1. Before installing the card, make sure that the system is off and the power cord is not con- nected to the server. Please follow proper electrical grounding procedures. 2. Open the system case. Rev 2.4 Mellanox Technologies...
Page 19
(x16 in a x8). Align the adapter connector edge with the PCI Express connector slot. 4. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. Rev 2.4 Mellanox Technologies...
Insert the connector into the adapter card. Be careful to insert the connector straight into the cage. Do not apply any torque, up or down, to the connector cage in the adapter card. d. Make sure that the connector locks in place. Rev 2.4 Mellanox Technologies...
3. Turn off the system and disconnect the power cord from the server. 3.7.2 Card Un-installation Please note that the following images are for illustration purposes only. 1. Verify that the system is powered off and unplugged. 2. Wait 30 seconds. Rev 2.4 Mellanox Technologies...
3. Right click the mouse on your adapter's row and select properties to display the adapter card properties window. 4. Click the Details tab and select Hardware Ids (Windows 2012/R2) from the Properties pull- down menu. Rev 2.4 Mellanox Technologies...
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
Step 1. The following example shows a system with an installed Mellanox HCA: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 Download the ISO image to your host.
The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Rev 2.4 Mellanox Technologies...
At this point, Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine. 4.1.3.1 Installation Procedure Login to the installation machine as root. Step 1. Mount the ISO image on your machine. Step 2. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Rev 2.4 Mellanox Technologies...
The path to the directory containing the log files will be displayed after running the installation script in the following format: "Logs dir: /tmp/MLNX_OFED_LINUX-<version>.<PID>.logs" Example: Logs dir: /tmp/MLNX_OFED_LINUX-x.x-x.logs 4.1.3.4 openibd Script As of MLNX_OFED v2.2-1.0.0 the script supports pre/post start/stop scripts: openibd Rev 2.4 Mellanox Technologies...
4.1.4 Uninstalling Mellanox OFED Use the script to uninstall the Mellanox OFED package. /usr/sbin/ofed_uninstall.sh The script is part of the RPM. ofed-scripts 4.1.5 Installing MLNX_OFED using YUM This type of installation is applicable to RedHat/OEL, Fedora, XenServer Operating Systems. Rev 2.4 Mellanox Technologies...
To update the firmware to the version included in MLNX_OFED package, you can either: • Run # yum install mlnx-fw-updater • Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Section 4.1.7, “Updating Firmware After Installation,” on page 4.1.5.3 Uninstalling Mellanox OFED using the YUM Tool...
Step 3. "/etc/apt/sources.list.d/mlnx- with the following content: _ofed.list" deb file:/<path to extracted MLNX_OFED package> ./ Download and install Mellanox Technologies GPG-KEY. Step 4. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Check that the key was successfully imported.
OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http:// www.mellanox.com...
"\.ko$" | xargs strip -g After the signature has been removed, a message as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" Rev 2.4 Mellanox Technologies...
Obtain the machine architecture. Step 1. For Windows Server 2012 / 2012 R2 / 8.1 Client To go to the Start menu, position your mouse in the bottom-right corner of the Remote Desktop of your screen. Rev 2.4 Mellanox Technologies...
[Optional] Manually configure your setup to contain the logs option. Step 2. > MLNX_WinOF2-1_35_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. > MLNX_WinOF2-1_35_All_x64.exe /v" MT_SKIPFWUPGRD=1" 1. MT_SKIPFWUPGRD default value is False Rev 2.4 Mellanox Technologies...
Page 39
Driver Installation Click Next in the Welcome screen. Step 4. Read then accept the license agreement and click Next. Step 5. Rev 2.4 Mellanox Technologies...
Page 40
Driver Installation Select the target folder for the installation. Step 6. Rev 2.4 Mellanox Technologies...
Page 41
If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a Mellanox card, only the Mellanox card will be updated. Rev 2.4 Mellanox Technologies...
Page 42
Analyze tools - install the tools that can be used either to diagnose or analyze the InfiniBand envi- ronment. • SDK - contains the libraries and DLLs for developing InfiniBand application over IBAL. • Documentation - contains the User Manual and Release Notes Rev 2.4 Mellanox Technologies...
Page 43
Driver Installation • Management tools - installation tools used for management, such as mlxstat. b. Click Install to start the installation. Rev 2.4 Mellanox Technologies...
Page 44
Driver Installation In case firmware upgrade option was checked in Step 7., you will be notified if a firmware Step 9. upgrade is required. Click Finish to complete the installation. Step 10. Rev 2.4 Mellanox Technologies...
Upon installation completion, the inf files can be located at: • %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the “Net- work adapters” menu. 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Rev 2.4 Mellanox Technologies...
Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. > MLNX_WinOF2-1_35_All_x64 /a • To extract only the driver files. > MLNX_WinOF2-1_35_All_x64 /a /vMT_DRIVERS_ONLY=1 Rev 2.4 Mellanox Technologies...
Page 47
Driver Installation Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Rev 2.4 Mellanox Technologies...
To complete the extraction, click Finish. Step 6. 4.2.6 Uninstalling Mellanox WinOF-2 Driver 4.2.6.1 Attended Uninstallation To uninstall MLNX_WinOF2 on a single node: Click Start -> Control Panel -> Programs and Features -> MLNX_WinOF2 -> Uninstall. Rev 2.4 Mellanox Technologies...
Extract the driver to a local directory (see Section 4.2.5, “Extracting Files Without Running Step 3. Installation,” on page 46). Copy the contents of this directory to C:\WS2016TP5_Drivers. The directory contents should appear as follows: Rev 2.4 Mellanox Technologies...
To upgrade, it is recommended to run a script that will execute all the required upgrade commands. 4.2.9 Performance Tuning For further information on WinOF-2 performance, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Rev 2.4 Mellanox Technologies...
• Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but with no • Check that both the adapter and its link are set to the same communication speed and duplex settings established Rev 2.4 Mellanox Technologies...
The ConnectX-4 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. Rev 2.4 Mellanox Technologies...
The ConnectX-4 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. Rev 2.4 Mellanox Technologies...
The ConnectX-4 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. Rev 2.4 Mellanox Technologies...
The ConnectX-4 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. Rev 2.4 Mellanox Technologies...
The ConnectX-4 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. Rev 2.4 Mellanox Technologies...
Air Flow (LFM) Air Flow Direction Heat Sink to Port Port to Heat Sink Active Active Active Active Cable Type Passive Passive 1.5W 3.5W 1.5W 3.5W MCX453A-FCAT – – MCX454A-FCAT – – MCX455A-FCAT – – MCX456A-FCAT – – MCX455A-ECAT – –...
All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 3: Mechanical Drawing of the Single-port x8 Adapter Cards 142.24 68.90 Figure 4: Mechanical Drawing of the Dual-port x8 Adapter Cards 142.24 68.90 Rev 2.4 Mellanox Technologies...
Table 15 - Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name TXN_2 Tx2n TXP_2 Tx2p TXN_4 Tx4n TXP_4 Tx4p ModSelL_Port0 ModSelL ResetL_Port0 ResetL VccRx RXP_3 Rx3p RXN_3 Rx3n RXP_1 Rx1p Rev 2.4 Mellanox Technologies...
Page 67
Connector Pin Number Connector Pin Name Port A Signal Name RXN_1 Rx1n RXN_2 Rx2n RXP_2 Rx2p RXN_4 Rx4n RXP_4 Rx4p ModPrsl_Port0 Mod PrsL IntL IntL VccTx Vcc1 LPMode_Port0 LPMode TXP_3 Tx3p TXN_3 Tx3n TXP_1 Tx1p TXN_1 Tx1n Rev 2.4 Mellanox Technologies...
Edge Connector Pinout The adapter cards support PCI Express 3.0 x8 or x16 edge connector and the PCI Expressx8 or x816 standard pinout according to the PCI Express 3.0 specification. Figure 10: PCIe x8 Connector Pinout Rev 2.4 Mellanox Technologies...
GUID and a MAC (derived from the GUID). The revisions indicated on the labels in the following figure do not necessarily repre- sent the latest revisions of the card. Figure 12: MCX453A-FCAT Board Label (Example) Figure 13: MCX454A-FCAT Board Label (Example) Rev 2.4...
6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Rev 2.4 Mellanox Technologies...
Page 74
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Rev 2.4 Mellanox Technologies...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 1.9 Mellanox Technologies...
Page 76
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Rev 1.9 Mellanox Technologies...
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Rev 2.4 Mellanox Technologies...
Page 78
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Rev 2.4 Mellanox Technologies...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 2.4 Mellanox Technologies...
Page 80
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Rev 2.4 Mellanox Technologies...
Need help?
Do you have a question about the MCX453A-FCAT and is the answer not in the manual?
Questions and answers