Mellanox Technologies MCX453A-FCAT User Manual
  1. Manuals
  2. Brands
  3. Mellanox Technologies Manuals
  4. Computer Hardware
  5. MCX453A-FCAT
  6. User manual

Mellanox Technologies MCX453A-FCAT User Manual

4 vpi single and dual qsfp28 port adapter card
Hide thumbs Also See for MCX453A-FCAT:
Table of Contents

Advertisement

Quick Links

Download this manual See also: User Manual
ConnectX®-4 VPI Single and Dual QSFP28 Port
Adapter Card User Manual
P/N:
MCX453A-FCAT, MCX454A-FCAT, MCX455A-FCAT, MCX456A-FCAT, MCX455A-ECAT, MCX456A-
ECAT
Rev 1.0
www.mellanox.com

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the MCX453A-FCAT and is the answer not in the manual?

Questions and answers

Related Manuals for Mellanox Technologies MCX453A-FCAT

Summary of Contents for Mellanox Technologies MCX453A-FCAT

  • Page 1 ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual P/N: MCX453A-FCAT, MCX454A-FCAT, MCX455A-FCAT, MCX456A-FCAT, MCX455A-ECAT, MCX456A- ECAT Rev 1.0 www.mellanox.com...
  • Page 2 KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.

  • Page 3: Table Of Contents

    4.1.2 Downloading Mellanox OFED ........22 Mellanox Technologies...
  • Page 4 7.1 MCX453A-FCAT Specifications ........
  • Page 5 Appendix E Sicherheitshinweise (Warnings in German) ..... . . 66 Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) . 68 Mellanox Technologies...

  • Page 6: List Of Tables

    MCX453A-FCAT Specifications Table ........
  • Page 7 Compatible Connector Plug and Pinout ........60 Mellanox Technologies...

  • Page 8: Revision History

    Rev 1.0 Revision History This document was printed on June 23, 2015. Table 1 - Revision History Table Date Comments/Changes June 2015 First Release Mellanox Technologies...

  • Page 9: About This Manual

    Rev 1.0 About this Manual This User Manual describes Mellanox Technologies ConnectX®-4 VPI Single and Dual QSFP28 port PCI Express x8 and x16 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant docu- mentation.

  • Page 10: Firmware And Software Updates

    The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Mellanox Technologies...

  • Page 11: Product Overview

    Rev 1.0 Introduction This is the User Guide for Mellanox Technologies adapter cards based on the ConnectX®-4 inte- grated circuit device. These adapters connectivity provide the highest performing and most flexi- ble interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers, High- Performance Computing, and Embedded environments.

  • Page 12: Table 4: Single And Dual-Port Edr Adapter Cards

    Ordering Part Number (OPN) MCX456A-ECAT - dual-port card InfiniBand: EDR (100 Gb/s) Data Transmission Rate Ethernet: 40/56/100Gb/s single or dual-port QSFP28 Network Connector Types PCIe 3.0 x16 8GT/s PCI Express (PCIe) SerDes Speed RoHS MT27704A0-FDCF-EV Adapter IC Part Number MT27708A0-FDCF-EV Mellanox Technologies...

  • Page 13: Features And Benefits

    PCI Express - stores and accesses InfiniBand and/or Ethernet fabric connection information and packet data SPI - includes one16MB SPI Flash device (M25PX16-VMN6P device by ST Memory Microelectronics) EEPROM - accessible through the I C-compatible interface. The EEPROM capacity is 128Kb. Mellanox Technologies...

  • Page 14: Operating Systems/Distributions

    InfiniBand RDMA for high-performance stor- age access. Operating Systems/Distributions • RHEL/CentOS • Windows • FreeBSD • VMware • OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Connectivity • Interoperable with InfiniBand or 10/25/40/50/100Gb Ethernet switches Mellanox Technologies...
  • Page 15 ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual Rev 1.0 • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Mellanox Technologies...

  • Page 16: Chapter 2 Interfaces

    “Mechanical Drawing of the Dual-port MCX456A-[E/F]CAT Adapter Card,” on page 54 for the location on the board. LED Interface There is a one bi-color I/O LED per port. For LED specifications please refer to Section 7.7, “Adapter LED Operation,” on page Mellanox Technologies...

  • Page 17: Chapter 3 Hardware Installation

    Note: Due to risk of damaging the EMI gasket, it is not recommended to replace the bracket more than three times. To replace the bracket you will need the following parts: Mellanox Technologies...

  • Page 18: Removing The Existing Bracket

    When the adapter is properly seated, the adapter port connectors are aligned with the slot opening, and the adapter faceplate is visible against the system chassis. Do not use excessive force when seating the card, as this may damage the system or the adapter. Mellanox Technologies...

  • Page 19: Cables And Modules

    6. To remove a cable, disengage the locks and slowly pull the connector away from the port receptacle. LED indicator will turn off when the cable is unseated. Mellanox Technologies...

  • Page 20: Identify The Card In Your System

    5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.

  • Page 21: On Linux

    ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual Rev 1.0 3.7.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella- nox Technologies”: > lspci |grep -i Mellanox Network controller: Mellanox Technologies MT4115 Family [ConnectX-4] Mellanox Technologies...

  • Page 22: Chapter 4 Driver Installation

    Step 1. The following example shows a system with an installed Mellanox HCA: # lspci -v | grep Mellanox 06:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] Subsystem: Mellanox Technologies Device 0024 Download the ISO image to your host. Step 2.

  • Page 23: Installing Mellanox Ofed

    Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using is irrelevant on Ubuntu and "mlnx_add_kernel_support.sh" Debian distributions. 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Mellanox Technologies...

  • Page 24: Installation Procedure

    Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed. Uninstalling the previous version of MLNX_OFED_LINUX Starting MLNX_OFED_LINUX-x.x.x installation ....Installation finished successfully. Attempting to perform Firmware update... Querying Mellanox devices firmware ... Mellanox Technologies...
  • Page 25 Most of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files. See the relevant chapters in this manual for details. The list of the modules that will be loaded automatically upon boot can be found in the /etc/ file. infiniband/openib.conf Mellanox Technologies...

  • Page 26: Installation Results

    This can be controlled by setting the variables below in the /etc/infiniband/openibd.conf file. OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh 4.1.3.5 Driver Load Upon System Boot Upon system boot, the Mellanox drivers will be loaded automatically. Mellanox Technologies...

  • Page 27: Mlnxofedinstall Return Codes

    Uninstalling Mellanox OFED Use the script to uninstall the Mellanox OFED package. /usr/sbin/ofed_uninstall.sh The script is part of the RPM. ofed-scripts 4.1.6 Installing MLNX_OFED using YUM This type of installation is applicable to RedHat/OEL, Fedora, XenServer Operating Systems. Mellanox Technologies...

  • Page 28: Setting Up Mlnx_Ofed Yum Repository

    # sudo rpm --import RPM-GPG-KEY-Mellanox warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mel- lanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Is this ok [y/N]: Check that the key was successfully imported.

  • Page 29: Installing Mlnx_Ofed Using The Yum Tool

    This system is not registered to Red Hat Subscription Management. You can use subscrip- tion-manager to register. Setting up Group Process Resolving Dependencies --> Running transaction check ---> Package ar_mgr.x86_64 0:1.0-0.11.g22fff4a will be installed ......rds-devel.x86_64 0:2.0.6mlnx-1 rds-tools.x86_64 0:2.0.6mlnx-1 srptools.x86_64 0:0.0.4mlnx3-OFED.2.0.2.6.7.11.ge863cb7 Complete! Mellanox Technologies...

  • Page 30: Uninstalling Mellanox Ofed Using The Yum Tool

    "--fw-update-only" • Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Chapter 5,“Updating Adapter Card Firmware” on page 43. 4.1.6.3 Uninstalling Mellanox OFED using the YUM Tool If MLNX_OFED was installed using the yum tool, then it can be uninstalled as follow: yum groupremove '<group name>'...

  • Page 31: Uninstalling Mellanox Ofed Using The Apt-Get Tool

    Updating Firmware After Installation The firmware can be updated in one of the following methods. 4.1.8.1 Updating the Device Online To update the device online on the machine from Mellanox site, use the following command line: mlxfwmanager --online -u -d <device> Mellanox Technologies...

  • Page 32: Updating The Device Manually

    OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http:// www.mellanox.com > Support > Firmware Download).

  • Page 33: Removing Signature From Kernel Modules

    After the signature has been removed, a massage as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" However, please note that a similar message as the following will still be presented: "my_module: module verification failed: signature and/or required key missing - taint-...

  • Page 34: Performance Tuning

    This section provides instructions for two types of installation procedures: • “Attended Installation” An installation procedure that requires frequent user intervention. • “Unattended Installation” An automated installation procedure that requires no user intervention. Both Attended and Unattended installations require administrator privileges. Mellanox Technologies...
  • Page 35 Double click the .exe and follow the GUI instructions to install MLNX_WinOF2. Step 1. [Optional] Manually configure your setup to contain the logs option. Step 2. > MLNX_WinOF2-1_10_All_x64.exe /v"/l*vx [LogFile]" Click Next in the Welcome screen. Step 3. Mellanox Technologies...
  • Page 36 Rev 1.0 Driver Installation Read then accept the license agreement and click Next. Step 4. Select the target folder for the installation. Step 5. Mellanox Technologies...
  • Page 37 Select a Complete or Custom installation, follow Step a and on, on page Step 6. a.Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user envi- ronment. • Documentation - contains the User Manual and Release Notes. Mellanox Technologies...
  • Page 38 Rev 1.0 Driver Installation b. Click Install to start the installation. Click Finish to complete the installation. Step 7. Mellanox Technologies...

  • Page 39: Unattended Installation

    • %ProgramFiles%\Mellanox\MLNX_WinOF2\ETH To see the Mellanox network adapter device, and the Ethernet or IPoIB network device (depending on the used card) for each port, display the Device Manager and expand “Network adapters”. 1. MT_NDPROPERTY default value is True Mellanox Technologies...

  • Page 40: Extracting Files Without Running Installation

    Run new task --> and enter CMD. Extract the driver and the tools: Step 2. > MLNX_WinOF2-1_10_All_x64 /a • To extract only the driver files. > MLNX_WinOF2-1_10_All_x64 /a /vMT_DRIVERS_ONLY=1 Click Next to create a server image. Step 3. Mellanox Technologies...
  • Page 41 ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual Rev 1.0 Click Change and specify the location in which the files are extracted to. Step 4. Click Install to extract this folder, or click Change to install to a different folder. Step 5. Mellanox Technologies...

  • Page 42: Uninstalling Mellanox Winof-2 Driver

    For information on how to upgrade firmware manually please refer to MFT User Manual: www.mellanox.com ->Products -> InfiniBand/VPI Drivers -> Firmware Tools The adapter card may not have been shipped with the latest firmware version. The section below describes how to update firmware. Mellanox Technologies...
  • Page 43 ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual Rev 1.0 Mellanox Technologies...

  • Page 44: Chapter 5 Updating Adapter Card Firmware

    Enter your card PSID to display the latest firmware file. The file name of the binary is composed by combin- ing the firmware name, the firmware release version, and the card part number. Note: Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card. Mellanox Technologies...

  • Page 45: Figure 3: Support Download Assistant

    > flint -d /dev/mst/mt4099_pci_cr0 -i <binary image> burn a. Reboot the computer. b. Enter: mst start. c. Verify that the card firmware was updated successfully. > flint -d /dev/mst/mt4103_pci_cr0 q Image type: ConnectX-4 FW Version: 2.9.4100 Device ID: 4103 Mellanox Technologies...

  • Page 46: Chapter 6 Troubleshooting

    • Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but • Check that both the adapter and its link are set to the same speed with no and duplex settings communication established Mellanox Technologies...

  • Page 47: Linux

    Assign a file name and save. Download and install MFT: http://www.mellanox.com/con- tent/pages.php?pg=management_tools&menu_section=34 Refer to the User Manual for installation instructions. Mellanox Firmware Once installed, open a CMD window and run: Tool (MFT) cd C:\Program Files\Mellanox\WinMFT mst start mst status flint –d <mst_device> q Mellanox Technologies...
  • Page 48 –d <mst_device> –i <firmware_bin_file> b • Event log viewer • MST device logs: • mst start Collect log file • mst status • C:\Users\Administrator> flint –d <mst_device> dc > dump_configura- tion.log • C:\Users\Administrator> mstdump <mst_device> dc > mstdump.log Mellanox Technologies...

  • Page 49: Chapter 7 Specifications

    ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual Rev 1.0 Specifications MCX453A-FCAT Specifications Table 9 - MCX453A-FCAT Specifications Table Size: 2.71 in. x5.6 in. (68.90mm x 142.24 mm) Connector: QSFP28 InfiniBand (Copper and optical) Physical InfiniBand: IBTA v1.3 Auto-Negotiation : 1X/2X/4X SDR (2.5Gb/s per lane), DDR (5Gb/s per lane),...

  • Page 50: Mcx454A-Fcat Specifications

    This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. For both operational and non-operational states. c. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 51: Mcx455A-Fcat Specifications

    This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. For both operational and non-operational states. c. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 52: Mcx456A-Fcat Specifications

    This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. For both operational and non-operational states. c. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 53: Mcx455A-Ecat Specifications

    This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. For both operational and non-operational states. c. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 54: Mcx456A-Ecat Specifications

    This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. For both operational and non-operational states. c. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 55: Adapter Led Operation

    If only the green LED is lit and the yellow LED is off, then the logical link has not been established Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 4: Mechanical Drawing of the Dual-port MCX456A-[E/F]CAT Adapter Card 142.24 68.90 Mellanox Technologies...

  • Page 56: Figure 5: Dual-Port Bracket

    Rev 1.0 Specifications Figure 5: Dual-port Bracket Mellanox Technologies...

  • Page 57: Appendix A Interface Connectors Pinout

    Figure 6: Connector and Cage Views Table 16 - Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name TXN_2 Tx2n TXP_2 Tx2p TXN_4 Tx4n TXP_4 Tx4p ModSelL_Port0 ModSelL ResetL_Port0 ResetL VccRx Mellanox Technologies...

  • Page 58: Pci Express X16/X16 Or X8 Connector Pinout

    PCI Express x16/x16 or x8 Connector Pinout The adapter cards use a standard PCI Express x16 or x16 or x8 edge connector and the PCI Express x16 or x8 standard pinout according to the PCI Express 3.0 specification. Mellanox Technologies...

  • Page 59: Figure 7: Pcie X8 Connector Pinout

    Rev 1.0 Figure 7: PCIe x8 Connector Pinout Mellanox Technologies...

  • Page 60: Figure 8: Pcie X16 Connector Pinout

    ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual Rev 1.0 Figure 8: PCIe x16 Connector Pinout Mellanox Technologies...

  • Page 61: Figure 9: Compatible Connector Plug And Pinout

    Rev 1.0 C-compatible Connector Pinout Figure 9: Compatible Connector Plug and Pinout Connector Pin Signal Name Number Mellanox Technologies...

  • Page 62: Appendix B Finding The Guid/ Mac And Serial Number On The Adapter Card

    Each Mellanox adapter card has a different identifier printed on the label: serial number the card MAC for the Ethernet protocol. and the card GUID for the InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID). Mellanox Technologies...

  • Page 63: Appendix C Safety Warnings

    This equipment should be installed in compliance with local and national electrical codes. 8. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. Mellanox Technologies...
  • Page 64 ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual Rev 1.0 CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Mellanox Technologies...

  • Page 65: Appendix D Avertissements De Sécurité D'installation (Warnings In French)

    L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Mellanox Technologies...
  • Page 66 PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 67: Appendix E Sicherheitshinweise (Warnings In German)

    Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. This equipment should be installed in compliance with local and national elec- trical codes. Mellanox Technologies...
  • Page 68 Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen.. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 69: Appendix F Advertencias De Seguridad Para La Instalación (Warnings In Spanish)

    Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. 8. Exposición a niveles de radiación peligrosos Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. Mellanox Technologies...
  • Page 70 ConnectX®-4 VPI Single and Dual QSFP28 Port Adapter Card User Manual Rev 1.0 PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Mellanox Technologies...

This manual is also suitable for:

Mcx454a-fcatMcx455a-ecatMcx455a-fcatMcx456aecatMcx456a-fcat

Table of Contents