Nvidia ConnectX-5 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Network Hardware
  5. Mellanox ConnectX-5
  6. User manual
Nvidia ConnectX-5 User Manual

Nvidia ConnectX-5 User Manual

Ethernet adapter cards
Hide thumbs Also See for ConnectX-5:
Table of Contents

Advertisement

Quick Links

Download this manual
NVIDIA ConnectX-5 Ethernet Adapter Cards User
Manual

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ConnectX-5 and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia ConnectX-5

Summary of Contents for Nvidia ConnectX-5

  • Page 1 NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 2: Table Of Contents

    VMware Driver Installation Updating Adapter Firmware Updating Adapter Firmware Troubleshooting Troubleshooting Specifications Specifications Monitoring Monitoring Finding the MAC on the Adapter Card Finding the MAC on the Adapter Card Document Revision History Document Revision History NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 3 Figure 14. Image2018 11 8 16 32 40 Figure 15. Image2018 10 2 16 16 3 Figure 16. Image2018 11 14 15 55 33 Figure 17. Image2018 11 8 16 41 39 Figure 18. Image2018 11 8 16 42 15 NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 4 Figure 35. Image2023 12 18 10 54 55 Figure 36. Image2018 11 8 19 13 49 Figure 37. Image2018 11 8 19 14 6 Figure 38. Image2018 11 8 19 14 49 Figure 39. Image2018 11 8 19 15 15 NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 5 Figure 40. Image2018 11 8 19 15 28 Figure 41. Image2018 11 8 19 15 43 Figure 42. Image2018 11 8 19 15 56 Figure 43. Image2020 11 10 10 20 48 Figure 44. Image2018 11 8 19 16 8 NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 6 The ConnectX-5 product line has moved to end-of-life. About This Manual This User Manual describes NVIDIA® ConnectX®-5 and ConnectX®-5 Ex Ethernet Single and Dual SFP28 and QSFP28 port PCI Express x8/x16 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.
  • Page 7 This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with Ethernet network and architecture specifications. Technical Support Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: URL: https://www.nvidia.com > Support E-mail: enterprisesupport@nvidia.com...
  • Page 8 Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support.Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller. Related Documentation MLNX_O FED for Linux User User Manual describing OFED features, performance, band diagnostic, tools Manual content, and configuration.
  • Page 9 Industry Standard PCI Express Base and Card Electromechanical Specifica Specifications. Refer to PCI-SIG Specifications. tions NVIDIA LinkX Ethernet cables and transceivers are designed to maximize the performance of High-Performance Computing networks, requiring high- bandwidth, low-latency connections between compute nodes and switch LinkX nodes.

  • Page 10: Introduction

    100Gb/s. b. The MCX516A-CDAT card has been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be tested when servers with Gen 4.0 support become available. ConnectX-5 Ethernet 25Gb/s Adapter Cards NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 11 SerDes @ 8.0GT/s Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Host Management UEFI Enabled (x86/Arm) RoHS RoHS Compliant IC Part Number MT27808A0-FCCF-EV ConnectX-5 Ethernet 100Gb/s Adapter Cards NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 12 firmware release notes for feature availability. Feature Description Uses PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x8 or x16 edge Express connector. Gen 1.1 and 2.0 compatible. (PCIe) NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 13 Network solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.
  • Page 14 PCIe bus (for example, from GPU to CPU), and therefore significantly reduces ect™ application run time. ConnectX-5 advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes. Adapter functionality enabling reduced CPU overhead allowing more available CPU for computation tasks.
  • Page 15 RHEL/CentOS Windows FreeBSD VMware OpenFabrics Enterprise Distribution (OFED) OpenFabrics Windows Distribution (WinOF-2) Connectivity Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches Passive copper cable with ESD protection Powered connectors for optical and active cable support NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 16: Interfaces

    The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables. PCI Express Interface The table below describes the supported PCIe interface in ConnectX-5 and ConnectX-5 Ex adapter cards. IC Model Supported PCIe Interface Features Link Rates: 2.5.
  • Page 17 PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.

  • Page 18: Hardware Installation

    Hardware Installation Installation and initialization of ConnectX-5 adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Note Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions.
  • Page 19 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20 Installation Procedure Overview The installation procedure of ConnectX-5 adapter cards involves the following steps: Step Procedure Check the system’s hardware and software requirements. Pay attention to the airflow consideration within the host system.
  • Page 20 System Requirements Hardware Requirements Important Unless otherwise specified, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operation environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for...
  • Page 21 Airflow Requirements ConnectX-5 adapter cards are offered with two airflow patterns: from the heatsink to the network ports, and vice versa, as shown below. Please refer to the Specifications section for airflow numbers for each specific card model.
  • Page 22 All cards in the system should be planned with the same airflow direction. Software Requirements See Operating Systems/Distributions section under the Introduction section. Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. Unpacking the Product Important The adapter is being installed in a system that operates with voltages that can be lethal.
  • Page 23 1. Unpack the adapter package and place them on an antistatic surface, and verify you have received the following items: 1. ConnectX-5 Adapter Card, with a tall bracket assembled on the card. 2. Short bracket (accessory) 2. Check the parts for visible damage that may have occurred during shipping.
  • Page 24 3. Shut down your system if active: Turn off the power to the system, and disconnect the power cord. Refer to the system documentation for instructions. Before you install the ConnectX-5 card, make sure that the system is disconnected from power.
  • Page 25 Please note that the following figures are for illustration purposes only. 1. Before installing the card, make sure that the system is off and the power cord is not connected to the server. Please follow proper electrical grounding procedures. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 26 2. Open the system case. 3. Place the adapter in an available PCI Express slot. 4. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 27 5. When the adapter is properly seated, the port connectors are aligned with the slot opening, and the adapter faceplate is visible against the system chassis. 6. Secure the adapter with the adapter clip or screw. Note NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 28 4. Make sure that the connector locks in place. Warning When installing cables make sure that the latches engage. Important Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 29 1. Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. 2. Expand System Devices and locate your NVIDIA ConnectX-5 adapter card. 3. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window.
  • Page 30 5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA; and DEV is equal to 1018 (for ConnectX-5) – this is a valid NVIDIA PCI Device ID.
  • Page 31 3. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). 4. Holding the adapter card from its center, gently pull the adapter card out of the PCI Express slot. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 32 5. When the port connectors reach the top of the chassis window, gently pull the adapter card in parallel to the motherboard. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 33: Driver Installation

    Please use the relevant driver installation section. Linux Driver Installation Windows Driver Installation VMware Driver Installation Linux Driver Installation This section describes how to install and test the NVIDIA OFED for Linux package on a single server with a ConnectX-5 adapter card installed. Prerequisites Requirements Description...
  • Page 34 1. Verify that the system has an NVIDIA network adapter installed. The following example shows a system with an installed NVIDIA adapter card: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family...
  • Page 35 flag during --skip-unsupported-devices-check installation. Usage ./mnt/mlnxofedinstall [OPTIONS] The installation script removes all previously installed OFED packages and re-installs from scratch. You will be prompted to acknowledge the deletion of the old packages. Warning NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 36 filesystem mounted over a ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the following command to update the initramfs: NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 37 /tmp directory. All Mellanox, OEM, OFED, or Distribution IB packages will be removed. Do you want to ?[y/N]:y continue See log file /tmp/mlnx_ofed_iso. .log 21642   Building OFED RPMs. Please wait... NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 38 For the list of installation options, run: ./mlnxofedinstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards. 1. Log in to the installation machine as root. 2. Mount the ISO image on your machine.
  • Page 39 --without-fw-update --forceFor example:./mlnxofedinstall --without-dkms --add-kernel- support --kernel 3.13.0-85-generic --without-fw-update -- forceNote that the path to kernel sources (--kernel- sources) should be added if the sources are not in their default location. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 40 firmware update can take effect. Refer to the table below to find the appropriate action for your specific card. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 41 files. See the relevant chapters in this manual for details. The list of the modules that will be loaded automatically upon boot can be found in the /etc/infiniband/openib.conf file. Warning NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 42 In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
  • Page 43 .IBMM2150110033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically. To prevent the automatic load of the NVIDIA drivers upon system boot: 1. Add the following lines to the "/etc/modprobe.d/mlnx.conf" file. blacklist mlx5_core blacklist mlx5_ib 2.
  • Page 44 In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
  • Page 45 2. Mount the ISO image on your machine and copy its content to a shared location in your network. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt 3. Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link: http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox # wget http: //www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 46 5. Check that the key was successfully imported. # rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n' grep Mellanox gpg-pubkey-a9e4b643-520791ba gpg(Mellanox Technologies <support .com>) @mellanox 6. Create a yum repository configuration file called "/etc/yum.repos.d/mlnx_ofed.repo" with the following content: NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 47 /mnt 3. Build the packages with kernel support and create the tarball. # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7. under /tmp directory. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 48 6. Check that the repository was successfully added. # yum repolist Loaded plugins: product-id, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. repo id repo name status mlnx_ofed MLNX_OFED Repository NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 49 (User Space packages only) package mlnx-ofed-eth-only-user-only.noarch : MLNX_OFED eth-only-user-only installer (User Space packages only) package mlnx-ofed-guest.noarch : MLNX_OFED guest installer (with KMP package support) mlnx-ofed-guest-user-only.noarch : MLNX_OFED guest-user-only installer (User Space packages only) package NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 50 Installs basic packages required for running NVIDIA mlnx-ofed-basic cards mlnx-ofed-guest Installs packages required by guest OS mlnx-ofed-hpc Installs packages required for HPC mlnx-ofed- Installs packages required by hypervisor OS hypervisor mlnx-ofed-vma Installs packages required by VMA NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 51 .fc21.x86_64 (without KMP support) package 3.17 mlnx-ofed-vma-vpi- .fc21.x86_64.noarch : MLNX_OFED vma-vpi 3.17 installer   kernel .fc21.x86_64 (without KMP support) package 3.17 mlnx-ofed-hypervisor- .fc21.x86_64.noarch : MLNX_OFED 3.17 hypervisor installer   kernel .fc21.x86_64 (without KMP package 3.17 NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 52 --> Running transaction check ---> Package mlnx-ofed-all.noarch will be installed --> Processing Dependency: kmod-isert = OFED. .g832a737.rhel7u1   : mlnx-ofed-all- .noarch package ........qperf.x86_64   rds-devel.x86_64 1.12 rds-tools.x86_64 1.12 sdpnetstat.x86_64 1.60 srptools.x86_64   Complete! NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 53 4. Download and install NVIDIA's Technologies GPG-KEY. # wget -qO - http: //www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - 5. Verify that the key was successfully imported. # apt-key list pub 1024D/A9E4B643 2013 uid Mellanox Technologies <support .com> @mellanox NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 54 WARNING: that it contains unsigned rpms, therefore, you need to disable the gpgcheck WARNING: by setting in the repository conf file. 'gpgcheck=0' Created /tmp/MLNX_OFED_LINUX- -rhel7. -x86_64-ext.tgz NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 55 - MLNX_OFED vma-vpi installer (with DKMS package support) mlnx-ofed-kernel-only - MLNX_OFED kernel-only installer (with package DKMS support) mlnx-ofed-bluefield - MLNX_OFED bluefield installer (with DKMS package support) mlnx-ofed-hpc-user-only - MLNX_OFED hpc-user-only installer package (User Space packages only) NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 56 - MLNX_OFED basic installer (with DKMS package support) (exact) mlnx-ofed-basic - MLNX_OFED basic installer (with DKMS support) package mlnx-ofed-dpdk-upstream-libs - MLNX_OFED dpdk-upstream-libs installer (with DKMS support) package where: mlnx-ofed-all MLNX_OFED all installer package NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 57 MLNX_OFED dpdk-upstream-libs installer package 2. Install the desired group. apt-get install '<group name>' Example: apt-get install mlnx-ofed-all Warning Installing MLNX_OFED using the “apt-get” tool does not automatically update the firmware.To update the firmware NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 58: Windows Driver Installation

    Windows Driver Installation For Windows, download and install the latest WinOF-2 for Windows software package available via the NVIDIA website at: WinOF-2 webpage. Follow the installation instructions included in the download package (also available from the download page). The snapshots in the following sections are presented for illustration purposes only. The installation interface may slightly vary, depending on the operating system in use.
  • Page 59 On an x64 (64-bit) machine, the output will be “AMD64”. 2. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > InfiniBand Drivers (Learn More) > Nvidia WinOF-2. 3. Download the .exe image according to the architecture of your machine (see Step The name of the .exe is in the following format: MLNX_WinOF2-...
  • Page 60 3. [Optional] If you do not want to upgrade your firmware version (i.e., MT_SKIPFWUPGRD default value is False). MLNX_WinOF2_<revision_version>_All_Arch.exe /v " MT_SKIPFWUPGRD=1" 4. [Optional] If you do not want to install the Rshim driver, run. MLNX_WinOF2_<revision_version>_All_Arch.exe /v " MT_DISABLE_RSHIM_INSTALL=1" NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 61 5. [Optional] If you want to skip the check for unsupported devices, run. MLNX_WinOF2_<revision_version>_All_Arch.exe /v " SKIPUNSUPPORTEDDEVCHECK=1" 6. Click Next in the Welcome screen. 7. Read and accept the license agreement and click Next. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 62 8. Select the target folder for the installation. 9. The firmware upgrade screen will be displayed in the following cases: If the user has an OEM card. In this case, the firmware will not be displayed. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 63 If the user has a standard NVIDIA® card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA® card, only the NVIDIA® card will be updated. 10. Select a Complete or Custom installation, follow Step a onward.
  • Page 64 Management tools - installation tools used for management, such as mlxstat Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd 2. Click Next to install the desired tools. 11. Click Install to start the installation. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 65 12. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 66 The following is an example of an unattended installation session. 1. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD. 2. Install the driver. Run: NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 67 The Rshim driver installanion will fail if a prior Rshim driver is already installed. The following fail message will be displayed in the log: "ERROR!!! Installation failed due to following errors: MlxRshim drivers installation disabled and MlxRshim drivers Installed, NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 68: Vmware Driver Installation

    " SKIPUNSUPPORTEDDEVCHECK=1" Firmware Upgrade If the machine has a standard NVIDIA® card with an older firmware version, the firmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to MFT User Manual.
  • Page 69 1. Log into the ESXi server with root permissions. 2. Install the driver. #> esxcli software vib install –d <path>/<bundle_file> Example: #> esxcli software vib install -d /tmp/MLNX-NATIVE-ESX-ConnectX- -5_4. 16.8 10EM- .zipesxcli 650.0 0.4240417 3. Reboot the machine. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 70 2. List all the existing NATIVE ESXi driver modules. (See Step 4 in Installing NATIVE ESXi Driver for VMware vSphere.) 3. Remove each module: #> esxcli software vib remove -n nmlx5-rdma #> esxcli software vib remove -n nmlx5-core NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 71 2. MD5SUM: 0804cffe30913a7b4017445a0f0adbe1 2. Install the image according to the steps described in the MFT User Manual. Warning The following procedure requires custom boot image downloading, mounting and booting from a USB device. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 72: Updating Adapter Firmware

    fixes. To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility. The utility can query for available NVIDIA adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images. The latest mlxup executable and documentation are available in mlxup - Update and Query Utility .
  • Page 73   Status: Up to date   Perform FW update? [y/N]: y Device #1: Up to date Device #2: Updating FW ... Done   Restart needed for updates to take effect. Log File: /var/log/mlxup/mlxup-yyyymmdd.log NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 74: Troubleshooting

    Verify that your switch and adapter port are compatible Link light is on, but with Check that the latest driver is loaded Check that both the adapter and its link are set to the same speed communicati and duplex settings established NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 75 Once installed, run: Tool (MFT) mst start mst status flint -d <mst_device> q ibstat Ports Information ibv_devinfo Firmware Version To download the latest firmware version, refer to the NVIDIA Upgrade Update and Query Utility. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 76 Upgrade flint –d <mst_device> –i <firmware_bin_file> b Event log viewer MST device logs: mst start Collect Log File mst status flint –d <mst_device> dc > dump_configuration.log mstdump <mst_device> dc > mstdump.log NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 77: Specifications

    Operational 0°C to 55°C Temperature Non-operational -40°C to 70°C 10% to 85% relative Operational Environment humidity Humidity 10% to 90% relative Non-operational humidity Altitude 3050m (Operational) Regulatory Safety CB / cTUVus / CE NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 78 Maximum power available through QSFP28 port: 3.5W Airflow Specifications. Environm Operational 0°C to 55°C Temperature ental Non-operational -40°C to 70°C Operational 10% to 85% relative humidity Humidity Non-operational 10% to 90% relative humidity NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 79 Voltage: 12V Airflow Power Cable Type Passive Cables 19.3W Typical Power Passive Cables 23.4W Maximum Power 1.5W Active Cable 26.8W 2.5W Active Cable 29.0W Maximum power available through QSFP28 port: 5W Airflow See Airflow NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 80 PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Power and Voltage: 12V Airflow Power Cable Type Passive Cable 9.5W Typical Power Passive Cable 11.8W Maximum Power 1.5W Active Cable 15.1W Maximum power available through SFP28 port: 1.5W NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 81 PCI Express Gen3: SERDES @ 8.0GT/s, 16 lanes (2.0 and 1.1 compatible) Voltage: 12V Power Cable Type Passive Cable 12.7W Typical Power Power and Airflow Passive Cable 15.1W Maximum Power 1.5W Active Cable 18.5W Maximum power available through SFP28 port: 2.5W NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 82 PCI Express Gen3: SERDES 8.0GT/s, 16 lanes (2.0 and 1.1 compatible) Power and Voltage: 12V Airflow Power Cable Type Passive Cable 12.7W Typical Power Passive Cable 15.1W Maximum Power 1.5W Active Cable 18.5W 2.5W Active Cable 20.7W NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 83 PCI Express Gen3: SERDES @ 8.0GT/s, 16 lanes (2.0 and 1.1 compatible) Power and Voltage: 12V Airflow Power Cable Type Passive Cable 14.7W Typical Power Maximum Power Passive Cable 17.2W 1.5W Active Cable 20.6W NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 84 CX4, SGMII, 1000BASE-X Data Rate: 1/10/25/40/50/100 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 16 lanes (2.0 and 1.1 compatible) Power and Voltage: 12V Airflow Power Cable Type Passive Cable 14.2W Typical Power NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 85 PCI Express Gen3: SERDES @ 8.0GT/s, 16 lanes (2.0 and 1.1 compatible) Note: PCIe 3.0 x16 bus can supply a maximum bandwidth of 128Gb/s only (=16 * 8GT/s, including overhead), and therefore cannot support 200Gb/s NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 86 The non-operational storage temperature specifications apply to the product without its package. Airflow Specifications Airflow (LFM) with airflow direction from heatsink to port. Cable Type Passive Active 1.5W Active 2.5W Active 3.5W Active 5W NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 87 MCX516A-CCAT / 350LF 500LFM 600LFM 650LFM 1300LFM MCX516A-CCHT Board Mechanical Drawing and Dimensions Warning All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Mechanical Drawing of a Dual-Port x16 Card NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 88 Mechanical Drawing of a Dual-Port x8 Card Bracket Mechanical Drawing Dimensions in millimeters. NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 89 Single-port Tall Bracket Dual-port Tall Bracket MCX512F-ACAT Tall Bracket NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 90: Monitoring

    IC thermal requirements. Adapter Heatsink A heatsink is attached to the ConnectX-5 IC in order to dissipate the heat from the ConnectX- 5 IC. It is attached either by using four spring-loaded push pins that insert into four mounting holes or by screws.

  • Page 91: Finding The Mac On The Adapter Card

    Finding the MAC on the Adapter Card Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. Warning The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards.
  • Page 92 MCX512F-ACHT Board Label (Example) MCX516A-BDAT Board Label (Example) MCX515A-GCAT Board Label (Example) MCX516A-GCAT Board Label (Example) NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 93 MCX515A-CCAT Board Label (Example) MCX516A-CCAT Board Label (Example) MCX516A-CCHT Board Label (Example) NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 94 MCX516A-CDAT Board Label (Example) NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

  • Page 95: Document Revision History

    Document Revision History Date Description of Changes Jan. Added a note that the ConnectX-5 product line has moved to end-of-life. 2024 Dec. Updated Specifications with MCX512F-ACAT bracket. 2023 May. Added non-operational storage temperature specifications. 2023 Nov. Updated Specifications tables. 2022 Aug.
  • Page 96 Added MCX515A-GCAT and MCX516A-GCAT support across the 2017 document and added the following: MCX516A-BDAT Specifications Table MCX516A-GCAT Specification Table Mechanical Drawing of the Single-port Adapter Cards Single-port Tall Bracket MCX512A-ACAT Board Label (Example) MCX516A-GCAT Board Label (Example) NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...
  • Page 97 Updated the following specifications tables: MCX515A-CCAT Specifications July MCX516A-CCAT Specifications 2017 MCX516A-CDAT Specifications Updated Airflow Specifications Updated Product Overview Apr. Updated Installation Instructions 2017 Mar. First release 2017 © Copyright 2023, NVIDIA. PDF Generated on 06/07/2024 NVIDIA ConnectX-5 Ethernet Adapter Cards User Manual...

This manual is also suitable for:

Connec tx-5 ex900-9x5az0053-st5900-9x5ad0056-st7900-9x5ad0054-st0900-9x5az0053-0t3900-9x5az0053-st4 ... Show all

Table of Contents