Download
Share
Print this page
Nvidia Mellanox ConnectX-6 MCX654106A-ECAT User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Computer Hardware
  5. Mellanox ConnectX-6 MCX654106A-ECAT
  6. User manual

Nvidia Mellanox ConnectX-6 MCX654106A-ECAT User Manual

Infiniband/vpi adapter cards
Hide thumbs Also See for Mellanox ConnectX-6 MCX654106A-ECAT:

Advertisement

Quick Links

Download this manual
NVIDIA Mellanox ConnectX-6 InfiniBand/VPI
Adapter Cards User Manual
Exported on Oct/22/2020 07:16 PM
https://docs.mellanox.com/x/NCQuAQ

Advertisement

loading

Related Manuals for Nvidia Mellanox ConnectX-6 MCX654106A-ECAT

Summary of Contents for Nvidia Mellanox ConnectX-6 MCX654106A-ECAT

  • Page 1 NVIDIA Mellanox ConnectX-6 InfiniBand/VPI Adapter Cards User Manual Exported on Oct/22/2020 07:16 PM https://docs.mellanox.com/x/NCQuAQ...
  • Page 2 Table of Contents Table of Contents .......................2 Introduction........................9 Product Overview ........................9 ConnectX-6 PCIe x8 Card...................... 10 ConnectX-6 PCIe x16 Card....................11 ConnectX-6 Socket Direct™ Cards ..................11 ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16) ..........12 ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a row) ....... 13 Package Contents .........................
  • Page 3 Bracket Replacement Instructions ..................24 Installation Instructions ......................25 Cables and Modules......................25 Identifying the Card in Your System ..................26 ConnectX-6 PCIe x8/16 Installation Instructions ..............27 Installing the Card......................27 Uninstalling the Card ....................... 29 ConnectX-6 Socket Direct (2x PCIe x16) Installation Instructions ........30 Installing the Card......................
  • Page 4 Updating the Device Firmware Automatically upon System Boot ......50 UEFI Secure Boot ......................51 Enrolling Mellanox's x.509 Public Key on Your Systems........... 51 Removing Signature from kernel Modules ..............51 Performance Tuning ......................52 Windows Driver Installation....................52 Software Requirements ....................53 Downloading Mellanox WinOF-2 Driver ................
  • Page 5 MCX654106A-ECAT Specifications ..................78 MCX653105A-EFAT Specifications..................80 MCX653106A-EFAT Specifications..................81 Adapter Card and Bracket Mechanical Drawings and Dimensions........82 ConnectX-6 PCIe x16 Adapter Card ................83 ConnectX-6 PCIe x8 Adapter Card .................. 83 Auxiliary PCIe Connection Card ..................84 Tall Bracket ........................84 Short Bracket ........................
  • Page 6 About This Manual This User Manual describes NVIDIA® Mellanox® ConnectX®-6 InfiniBand/VPI adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Ordering Part Numbers The table below provides the ordering part numbers (OPN) for the available ConnectX-6 InfiniBand/VPI adapter cards.
  • Page 7 • Tel: +1.408.916.0055 Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller. Related Documentation User Manual describing OFED features, performance, band diagnostic, tools content and Mellanox OFED configuration.
  • Page 8 Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the Mellanox LinkX performance of High-Performance Computing networks, requiring high-bandwidth, low- Interconnect latency connections between compute nodes and switch nodes. Mellanox offers one of Solutions industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/HDR100 (100Gb/s) and HDR (200Gb/s) cables, including Direct Attach Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km.
  • Page 9 Introduction Product Overview This is the user guide for Mellanox technologies VPI adapter cards based on the ConnectX®-6 integrated circuit device. ConnectX-6 connectivity provides the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in enterprise datacenters and high-performance computing environments.
  • Page 10 Configuration Marketing Description MCX654106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR InfiniBand and 100GbE), dual- port QSFP56, Socket Direct 2x PCIe3.0/4.0 x16, tall bracket ConnectX-6 Single-slot Socket Direct MCX653105A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s Cards (2x PCIe x8 in a row) (HDR100, EDR IB and 100GbE), single-port QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in a row, tall bracket...
  • Page 11 ConnectX-6 PCIe x16 Card ConnectX-6 with a single PCIe x16 slot can support a bandwidth of up to 100Gb/s in a PCIe Gen 3.0 slot, or up to 200Gb/s in a PCIe Gen 4.0 slot. Part Number MCX653105A- MCX653106A- MCX653105A- MCX653106A- ECAT ECAT...
  • Page 12 ConnectX-6 Socket Direct cards are available in two configurations: Dual-slot Configuration (2x PCIe x16) and Single-slot Configuration (2x PCIe x8).  ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16) In order to obtain 200Gb/s speed, Mellanox offers ConnectX-6 Socket Direct that enable 200Gb/s connectivity also for servers with PCIe Gen 3.0 capability.
  • Page 13 ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a row) The PCIe x16 interface comprises two PCIe x8 in a row, such that each of the PCIe x8 lanes can be connected to a dedicated CPU in a dual-socket server. In such a configuration, Socket Direct brings lower latency and lower CPU utilization as the direct connection from each CPU to the network means the interconnect can bypass a QPI (UPI) and the other CPU, optimizing performance and improving latency.
  • Page 14 Package Contents ConnectX-6 PCIe x8/x16 Adapter Cards  Applies to MCX651105A-EDAT, MCX653105A-ECAT, MCX653106A-ECAT, MCX653105A-HDAT, MCX653106A-HDAT, MCX653105A-EFAT, MCX653106A-EFAT. Category Item Cards ConnectX-6 adapter card Accessories Adapter card short bracket Adapter card tall bracket (shipped assembled on the card) ConnectX-6 Socket Direct Cards (2x PCIe x16) ...
  • Page 15 Features and Benefits  Make sure to use a PCIe slot that is capable of supplying the required power and airflow to the ConnectX-6 cards as stated in Specifications. Uses the following PCIe interfaces: Express • PCIe x8/x16 configurations: (PCIe) PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x8/x16 edge connector.
  • Page 16 Overlay In order to better scale their networks, datacenter operators often create overlay networks that Networks carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU.
  • Page 17 Operating Systems/Distributions  ConnectX-6 Socket Direct cards 2x PCIe x16 (OPNs: MCX654105A-HCAT, MCX654106A-HCAT and MCX654106A-ECAT) are not supported in Windows and WinOF-2. • OpenFabrics Enterprise Distribution (OFED) • RHEL/CentOS • Windows  • FreeBSD • VMware • OpenFabrics Enterprise Distribution (OFED) •...
  • Page 18 Interfaces InfiniBand Interface The network ports of the ConnectX®-6 adapter cards are compliant with the InfiniBand Architecture Specification, Release 1.3. InfiniBand traffic is transmitted through the cards' QSFP56 connectors. Ethernet QSFP56 Interfaces  The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables.
  • Page 19 • LED 3 and 4: Reserved for future use. LED1 and LED2 Link Status Indications (Physical and Logical) - Ethernet Protocole: LED Color and State Description A link has not been established Blinking amber 1 Hz Blinking amber occurs due to running a beacon command for locating the adapter card 4 Hz blinking amber indicates a problem with the physical link Solid green...
  • Page 20 SMBus Interface ConnectX-6 technology maintains support for manageability through a BMC. ConnectX-6 PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact Mellanox Support.
  • Page 21 Hardware Installation Installation and initialization of ConnectX-6 adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings  Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions document available on mellanox.com.
  • Page 22 System Requirements Hardware Requirements  Unless otherwise specified, Mellanox products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operating environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level.
  • Page 23     Airflow  from the network ports  to the heatsink:    All cards in the system should be planned with the same airflow direction. Software Requirements • Operating Systems/Distributions section under the Introduction section. • Software Stacks - Mellanox OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware.
  • Page 24 Pre-Installation Checklist • Unpack the ConnectX-6 Card; Unpack and remove the ConnectX-6 card. Check against the package contents list that all the parts have been sent. Check the parts for visible damage that may have occurred during shipping. Please note that the cards must be placed on an antistatic surface.
  • Page 25  Use a torque driver to apply up to 2 lbs-in torque on the screws. Installation Instructions This section provides detailed instructions on how to install your adapter card in a system. Choose the installation instructions according to the ConnectX-6 configuration you have purchased. OPNs Installation Instructions MCX651105A-EDAT...
  • Page 26 After plugging in a cable, lock the connector using the latching mechanism particular to the cable vendor. When data is being transferred the Green LED will blink. See Interface under the Interfaces section.  Care should be taken as not to impede the air exhaust flow through the ventilation holes. Use cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack.
  • Page 27 PCI Device (Example) In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1018 (for ConnectX-6) –...
  • Page 28  Please make sure to install the ConnectX-6 cards in a PCIe slot that is capable of supplying the required power and airflow as stated in Specifications. Connect the adapter Card in an available PCI Express x16 slot in the chassis. Step 1: Locate an available PCI Express x16 slot and insert the adapter card to the chassis.
  • Page 29 Uninstalling the Card Safety Precautions The adapter is installed in a system that operates with voltages that can be lethal. Before uninstalling the adapter card, please observe the following precautions to avoid injury and prevent damage to system components. Remove any metallic objects from your hands and wrists. It is strongly recommended to use an ESD strap or other antistatic devices.
  • Page 30       4. Holding the adapter card from its center, gently pull the ConnectX-6 and Auxiliary Connections cards out of the PCI Express slot. ConnectX-6 Socket Direct (2x PCIe x16) Installation Instructions The hardware installation section uses the terminology of white and black harnesses to differentiate between the two supplied cables.
  • Page 31 The Cabline harness marked with "WHITE" label should be connected to the connector on the ConnectX-6 and PCIe card engraved with “White Cable”  while the one marked with "BLACK" label should be connected to the connector on the ConnectX-6 and PCIe card engraved with “Black Cable”. ...
  • Page 32 Step 2: Verify the plugs are locked. Step 3: Slide the retention clip latches through the cutouts on the PCB. The latches should face the annotation on the PCB. ...
  • Page 33 Step 4: Clamp the retention clip. Verify both latches are firmly locked. Step 5: Slide the Cabline CA-II Plus harnesses through the retention clip. Make sure that the clip opening is facing the plugs. ...
  • Page 34 Step 6: Plug the Cabline CA-II Plus harnesses on the PCIe Auxiliary Card. As indicated on both sides of the Auxiliary connection card; plug the black harness to the component side and the white harness to the print side. Step 7: Verify the plugs are locked. Step 8: Slide the retention clip through the cutouts on the PCB.
  • Page 35 Step 9: Clamp the retention clip. Verify both latches are firmly locked. Connect the ConnectX-6 adapter and PCIe Auxiliary Connection cards in available PCI Express x16 slots in the chassis. Step 1: Locate two available PCI Express x16 slots. Step 2: Applying even pressure at both corners of the cards, insert the adapter card in the PCI Express slots until firmly seated.
  • Page 36  Do not use excessive force when seating the cards, as this may damage the system or the cards. Step 3: Applying even pressure at both corners of the cards, insert the Auxiliary Connection card in the PCI Express slots until firmly seated. Secure the ConnectX-6 adapter and PCIe Auxiliary Connection Cards to the chassis.
  • Page 37 Uninstalling the Card Safety Precautions The adapter is installed in a system that operates with voltages that can be lethal. Before uninstalling the adapter card, please observe the following precautions to avoid injury and prevent damage to system components. Remove any metallic objects from your hands and wrists. It is strongly recommended to use an ESD strap or other antistatic devices.
  • Page 38       4. Holding the adapter card from its center, gently pull the ConnectX-6 and Auxiliary Connections cards out of the PCI Express slot.
  • Page 39 Driver Installation Please use the relevant driver installation section.  ConnectX-6 Socket Direct cards 2x PCIe x16 (OPNs: MCX654106A-HCAT and MCX654106A- ECAT) are not supported in Windows and WinOF-2. • Linux Driver Installation • Windows Driver Installation • VMware Driver Installation Linux Driver Installation This section describes how to install and test the Mellanox OFED for Linux package on a single server with a Mellanox ConnectX-5 adapter card installed.
  • Page 40 ConnectX-6 Card Configuration Single-port Socket Direct Card (2x PCIe [root@mftqa-009 ~]# lspci |grep mellanox -i x16) a3:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6] e3:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6] Dual-port Socket Direct Card (2x PCIe [root@mftqa-009 ~]# lspci |grep mellanox -i x16) 05:00.0 Infiniband controller: Mellanox Technologies MT28908A0 Family [ConnectX-6]...
  • Page 41 Installing Mellanox OFED Installation Script The installation script,  , performs the following: mlnxofedinstall • Discovers the currently installed kernel • Uninstalls any software stacks that are part of the standard operating system distribution or another vendor's commercial stack • Installs the MLNX_OFED_LINUX binary RPMs (if they are available for the current kernel) •...
  • Page 42 Example The following command will create a MLNX_OFED_LINUX ISO image for RedHat 6.3 under the / tmp directory. # ./MLNX_OFED_LINUX-x.x-x-rhel6.3-x86_64/mlnx_add_kernel_support.sh -m /tmp/MLNX_OFED_LINUX-x.x-x-rhel6.3- x86_64/ --make-tgz Note: This program will create MLNX_OFED_LINUX TGZ for rhel6.3 under /tmp directory. All Mellanox, OEM, OFED, or Distribution IB packages will be removed. Do you want to continue?[y/N]:y See log file /tmp/mlnx_ofed_iso.21642.log  ...
  • Page 43  For unattended installation, use the --force installation option while running the MLNX_OFED installation script: /mnt/mlnxofedinstall --force  MLNX_OFED for Ubuntu should be installed with the following flags in chroot environment: ./mlnxofedinstall --without-dkms --add-kernel-support --kernel <kernel version in chroot> --without-fw-update --force For example: ./mlnxofedinstall --without-dkms --add-kernel-support --kernel 3.13.0-85-generic --without-fw-update --force...
  • Page 44 After installation completion, information about the Mellanox OFED installation, such as prefix, kernel version, and installation parameters can be retrieved by running the command /etc/infiniband/info. Most of the Mellanox OFED components can be configured or reconfigured after the installation, by modifying the relevant configuration files.
  • Page 45 OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh  An example of OPENIBD_POST_START script for activating all interfaces is provided in the MLNX_OFED package under the docs/scripts/openibd-post-start-configure-interfaces/ folder. Driver Load Upon System Boot Upon system boot, the Mellanox drivers will be loaded automatically. To prevent automatic load of the Mellanox drivers upon system boot: Add the following lines to the "/etc/modprobe.d/mlnx.conf"...
  • Page 46 Uninstalling MLNX_OFED Use the script /usr/sbin/ofed_uninstall.sh to uninstall the Mellanox OFED package. The script is part of the ofed-scripts RPM. Installing MLNX_OFED Using YUM This type of installation is applicable to RedHat/OL, Fedora, XenServer Operating Systems. Setting up MLNX_OFED YUM Repository Log into the installation machine as root.
  • Page 47 # yum repolist Loaded plugins: product-id, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. repo id repo name status mlnx_ofed MLNX_OFED Repository rpmforge RHEL 6Server - RPMforge.net - dag 4,597  ...
  • Page 48 # yum install mlnx-ofed-all Loaded plugins: langpacks, product-id, subscription-manager Resolving Dependencies --> Running transaction check ---> Package mlnx-ofed-all.noarch 0:3.1-0.1.2 will be installed --> Processing Dependency: kmod-isert = 1.0-OFED.3.1.0.1.2.1.g832a737.rhel7u1 package: mlnx-ofed-all-3.1-0.1.2.noarch ........qperf.x86_64 0:0.4.9-9 rds-devel.x86_64 0:2.0.7-1.12 rds-tools.x86_64 0:2.0.7-1.12 sdpnetstat.x86_64 0:1.60-26 srptools.x86_64 0:1.0.2-12  ...
  • Page 49 <pre># apt-cache search mlnx-ofed- mlnx-ofed-vma-eth - MLNX_OFED vma-eth installer package (with DKMS support) mlnx-ofed-hpc - MLNX_OFED hpc installer package (with DKMS support) mlnx-ofed-vma-vpi - MLNX_OFED vma-vpi installer package (with DKMS support) mlnx-ofed-basic - MLNX_OFED basic installer package (with DKMS support) mlnx-ofed-vma - MLNX_OFED vma installer package (with DKMS support)
  • Page 50 mlxfwmanager --online -u -d 0000:09:00.0 Querying Mellanox devices firmware ... Device #1: ---------- Device Type: ConnectX-5 Part Number: Description: PSID: MT_1020120019 PCI Device Name: 0000:09:00.0 Port1 GUID: 0002c9000100d051 Port2 MAC: 0002c9000002 Versions: Current Available FW 2.32.5000 2.33.5000 Status: Update required --------- Found 1 device(s) requiring firmware update.
  • Page 51 separated list of PCI devices to exclude from the firmware update. Example: MLNX_EXCLUDE_DEVICES="00:05.0,00:07.0" UEFI Secure Boot All kernel modules included in MLNX_OFED for RHEL7 and SLES12 are signed with x.509 key to support loading the modules when Secure Boot is enabled. Enrolling Mellanox's x.509 Public Key on Your Systems In order to support loading MLNX_OFED drivers when an OS supporting Secure Boot boots on a UEFI- based system with Secure Boot enabled, the Mellanox x.509 public key should be added to the UEFI...
  • Page 52 # rpm -qa | grep -E "kernel-ib|mlnx-ofa_kernel|iser|srp|knem|mlnx-rds|mlnx-nfsrdma|mlnx-nvme|mlnx-rdma-rxe" | xargs rpm -ql | grep "\.ko$" | xargs strip -g After the signature has been removed, a massage as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11"...
  • Page 53 Software Requirements Description Package Windows Server 2012 R2 MLNX_WinOF2-2_10_All_x64.exe Windows Server 2012 Windows Server 2016 Windows Server 2019 Windows 8.1 Client (64 bit only) Windows 10 Client (64 bit only) Note: The Operating System listed above must run with administrator privileges. Downloading Mellanox WinOF-2 Driver To download the .exe file according to your Operating System, please follow the steps below: Obtain the machine architecture.
  • Page 54 • Attended Installation - An installation procedure that requires frequent user intervention. • Unattended Installation - An automated installation procedure that requires no user intervention.  Both Attended and Unattended installations require administrator privileges. Attended Installation The following is an example of an installation session. Double click the .exe and follow the GUI instructions to install MLNX_WinOF2.
  • Page 55 Select the target folder for the installation. The firmware upgrade screen will be displayed in the following cases: • If the user has an OEM card. In this case, the firmware will not be displayed. • If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly.
  • Page 56 Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat •...
  • Page 57 b. Click Next to install the desired tools.  9.  Click Install to start the installation.          10. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see  ). ...
  • Page 58          11.  Click Finish to complete the installation.         Unattended Installation...
  • Page 59  If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD.
  • Page 60 Uninstalling Mellanox WinOF-2 Driver Attended Uninstallation To uninstall MLNX_WinOF2 on a single node:  Click Start > Control Panel > Programs and Features > MLNX_WinOF2 > Uninstall. (NOTE: This requires elevated administrator privileges) Unattended Uninstallation To uninstall MLNX_WinOF2 in unattended mode: Open a CMD console. (Click Task Manager > File > Run new task, and enter CMD.) To uninstall the driver, run: MLNX_WinOF2-2_0_All_x64.exe /S /x /v"/qn"...
  • Page 61 Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter CMD. Extract the driver and the tools: MLNX_WinOF2-2_0_<revision_version>_All_x64 /a To extract only the driver file MLNX_WinOF2-2_0_<revision_version>_All_x64 /a /vMT_DRIVERS_ONLY=1 Click Next to create a server image.   Click Change and specify the location in which the files are extracted to. ...
  • Page 62         Click Install to extract this folder, or click Change to install to a different folder. ...
  • Page 63 To complete the extraction, click Finish. Firmware Upgrade If the machine has a standard Mellanox card with an older firmware version, the firmware will be automatically updated as part of the WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to the MFT User Manual at www.m ellanox.com >Products >...
  • Page 64 Installing Mellanox NATIVE ESXi Driver for VMware vSphere  Please uninstall all previous Mellanox driver packages prior to installing the new version. See Removing Earlier Mellanox Drivers for further information. To install the driver: Log into the ESXi server with root permissions. Install the driver. ...
  • Page 65 Firmware Programming Download the VMware bootable binary images v4.6.0 from the Mellanox Firmware Tools (MFT) site. ESXi 6.5 File: mft-4.6.0.48-10EM-650.0.0.4598673.x86_64.vib MD5SUM: 0804cffe30913a7b4017445a0f0adbe1 Install the image according to the steps described in the MFT User Manual.  The following procedure requires custom boot image downloading, mounting and booting from a USB device.
  • Page 66 Updating Adapter Firmware Each adapter card is shipped with the latest version of qualified firmware at the time of manufacturing. However, Mellanox issues firmware updates occasionally that provide new features and bug fixes. To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility.
  • Page 67 Troubleshooting General Troubleshooting Server unable to find the adapter • Ensure that the adapter is placed correctly • Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...
  • Page 68 Mellanox Firmware Tool (MFT) Download and install MFT: http://www.mellanox.com/ content/pages.php?pg=management_tools&menu_section=34 Refer to the User Manual for installation instructions. Once installed, run: mst start mst status flint -d <mst_device> q Ports Information ibstat ibv_devinfo Firmware Version Upgrade To download the latest firmware version refer to http:// www.mellanox.com/supportdownloader Collect Log File...
  • Page 69 Specifications MCX651105A-EDAT Specifications   Please make sure to install the ConnectX-6 card in a PCIe slot that is capable of supplying the required power and airflow as stated in the below table. Physical Adapter Card Size: 6.6 in. x 2.71 in. (167.65mm x 68.90mm) Connector: Single QSFP56 InfiniBand and Ethernet (copper and optical) Protocol InfiniBand: IBTA v1.3...
  • Page 70 Airflow Direction Cable Type Heatsink to Port Port to Heatsink Airflow (LFM) / Passive Cables Ambient Temperature Mellanox Active W Cables Regulatory Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant MCX653105A-HDAT Specifications ...
  • Page 71 Voltage: 3.3Aux Maximum current: 100mA Maximum power available through QSFP56 port: 5W Envi Temperature Operational 0°C to 55°C Non-operational -40°C to 70°C Humidity: 90% relative humidity Airflow Direction Cable Type Heatsink to Port Port to Heatsink Airflow (LFM) / Passive Cables 350 LFM / 55°C 250 LFM / 35°C Ambient...
  • Page 72 Ethernet: 200GBASE-CR4, 200GBASE-KR4, 200GBASE-SR4, 100GBASE-CR4, 100GBASE-KR4, 100GBASE-SR4, 50GBASE-R2, 50GBASE-R4, 40GBASE-CR4, 40GBASE-KR4, 40GBASE-SR4, 40GBASE-LR4, 40GBASE-ER4, 40GBASE-R2, 25GBASE-R, 20GBASE-KR2, 10GBASE-LR, 10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII, 1000BASE-CX, 1000BASE- KX, 10GBASE-SR Data Rate InfiniBand SDR/DDR/QDR/FDR/EDR/HDR100/HDR Ethernet 1/10/25/40/50/100/200 Gb/s PCI Express Gen3/4: SERDES @ 8.0GT/s/16GT/s, x16 lanes (2.0 and 1.1 compatible) Adapter Card Voltage: 12V, 3.3VAUX Power...
  • Page 73 MCX653105A-ECAT Specifications  Please make sure to install the ConnectX-6 card in a PCIe slot that is capable of supplying the required power and airflow as stated in the below table. Physical Adapter Card Size: 6.6 in. x 2.71 in. (167.65mm x 68.90mm) Connector: Single QSFP56 InfiniBand and Ethernet (copper and optical) Protocol InfiniBand: IBTA v1.3...
  • Page 74 Mellanox Active 300 LFM / 55°C 200 LFM / 35°C 2.7W Cables Regulatory Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-6 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches.
  • Page 75 Maximum Power Please refer to ConnectX-6 VPI Power Specifications (requires MyMellanox login credentials) Voltage: 3.3Aux Maximum current: 100mA Maximum power available through QSFP56 port: 5W Environmental Temperature Operational 0°C to 55°C Non-operational -40°C to 70°C Humidity: 90% relative humidity Airflow Direction Cable Type Heatsink to Port Port to Heatsink...
  • Page 76 Protocol Support InfiniBand: IBTA v1.3 Auto-Negotiation: 1X/2X/4X SDR (2.5Gb/s per lane), DDR (5Gb/s per lane), QDR (10Gb/ s per lane), FDR10 (10.3125Gb/s per lane), FDR (14.0625Gb/s per lane), EDR (25Gb/s per lane) port, HDR100 (2 lane x 50Gb/s per lane), HDR (50Gb/s per lane) port Ethernet: 200GBASE-CR4, 200GBASE-KR4, 200GBASE-SR4, 100GBASE-CR4, 100GBASE-KR4, 100GBASE-SR4, 50GBASE-R2, 50GBASE-R4, 40GBASE-CR4, 40GBASE-KR4, 40GBASE-SR4, 40GBASE-LR4, 40GBASE-ER4, 40GBASE-R2,...
  • Page 77 Notes: a. The ConnectX-6 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mellanox InfiniBand product. b. Typical power for ATIS traffic load. c. For both operational and non-operational states. d.
  • Page 78 Active Auxiliary PCIe Typical Power 3.0W Connection Card Power Maximum Power 4.0W Environmental Operational 0°C to 55°C Temperature Non-operational -40°C to 70°C Humidity: 90% relative humidityc Airflow Direction Cable Type Heatsink to Port Port to Heatsink Airflow (LFM) / Ambient Passive Cables 700 LFM / 55°C 400 LFM / 35°C...
  • Page 79 Ethernet: 100GBASE-CR4, 100GBASE-KR4, 100GBASE-SR4, 50GBASE-R2, 50GBASE- R4, 40GBASE-CR4, 40GBASE-KR4, 40GBASE-SR4, 40GBASE-LR4, 40GBASE-ER4, 40GBASE-R2, 25GBASE-R, 20GBASE-KR2, 10GBASE-LR,10GBASE-ER, 10GBASE- CX4, 10GBASE-CR, 10GBASE-KR, SGMII, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR Data Rate InfiniBand SDR/DDR/QDR/FDR/HDR100/EDR Ethernet 1/10/25/40/50/100 Gb/s Gen3: SERDES @ 8.0GT/s, x16 lanes (2.0 and 1.1 compatible) Adapter Card Power Voltage: 12V, 3.3VAUX Power...
  • Page 80 MCX653105A-EFAT Specifications  Please make sure to install the ConnectX-6 card in a PCIe slot that is capable of supplying the required power and airflow as stated in the below table. Physical Adapter Card Size: 6.6 in. x 2.71 in. (167.65mm x 68.90mm) Connector: Single QSFP56 InfiniBand and Ethernet (copper and optical) Protocol InfiniBand: IBTA v1.3...
  • Page 81 Airflow Direction Cable Type Heatsink to Port Port to Heatsink Airflow (LFM) / Ambient Passive Cables 300 / 55°C 200 / 35°C Temperature Mellanox Active 300 / 55°C 200 / 35°C 2.75W Cables Regulatory Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-6 adapters supplement the IBTA auto-negotiation specification to get better bit...
  • Page 82 Adapter Card Voltage: 12V, 3.3VAUX Power Power Cable Passive Cables 21.6W Typical Power Maximum Power Please refer to ConnectX-6 VPI Power Specifications (requires MyMellanox login credentials) Voltage: 3.3Aux Maximum current: 100mA Maximum power available through QSFP56 port: 5W Environmental Temperature Operational 0°C to 55°C Non-operational -40°C to 70°C...
  • Page 83 ConnectX-6 PCIe x16 Adapter Card  ConnectX-6 PCIe x8 Adapter Card ...
  • Page 84 Auxiliary PCIe Connection Card  Tall Bracket Short Bracket...
  • Page 85 PCIe Express Pinouts Description for Single- Slot Socket Direct Card  This section applies to ConnectX-6 single-slot cards (MCX653105A-EFAT and MCX653106A- EFAT). ConnectX-6 single-slot Socket Direct cards offer improved performance to dual-socket servers by enabling direct access from each CPU in a dual-socket server to the network through its dedicated PCIe interface.
  • Page 86 Finding the GUID/MAC on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID). ...
  • Page 87 MCX653106A-ECAT Board Label MCX654105A-HCAT Board Label MCX654106A-HCAT Board Label MCX654105A-ECAT Board Label MCX654106A-ECAT Board Label MCX653106A-EFAT Board Label...
  • Page 88 Document Revision History Date Revi Comments/Changes sion Mar. 2020 Added MCX651105A-EDAT support across the document. Sep. 2019 Added a note to the hardware installation instructions. Aug. 2019 Updated "Package Contents" and "Hardware Installation" Aug. 2019 Updated "PCI Express Pinouts Description" Aug. 2019 Updated "Hardware Installation"...
  • Page 89 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
  • Page 90 NVIDIA, the NVIDIA logo, and Mellanox are trademarks and/or registered trademarks of Mellanox Technologies Ltd. and/or NVIDIA Corporation in the U.S. and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated.

This manual is also suitable for:

Mellanox connectx-6 mcx653105a-efatMellanox connectx-6 mcx653106a-efatMellanox connectx-6 mcx651105a-edatMellanox connectx-6 mcx653105a-ecatMellanox connectx-6 mcx653106a-ecatMellanox connectx-6 mcx653105a-hdat ... Show all