Nvidia ConnectX-7 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Computer Hardware
  5. ConnectX-7
  6. User manual

Nvidia ConnectX-7 User Manual

Adapter cards
Hide thumbs Also See for ConnectX-7:
Table of Contents

Advertisement

Quick Links

Download this manual
 
 
 
 
 
 
 
NVIDIA ConnectX-7 Adapter Cards User
Manual
 
 
Exported on Feb/16/2023 05:54 AM

Advertisement

Table of Contents
loading

Related Manuals for Nvidia ConnectX-7

Summary of Contents for Nvidia ConnectX-7

  • Page 1               NVIDIA ConnectX-7 Adapter Cards User Manual     Exported on Feb/16/2023 05:54 AM...

  • Page 2: Table Of Contents

    Intended Audience ................7 Ordering Part Numbers................ 8 ConnectX-7 PCIe x16 Stand-up Adapter Cards...........8 ConnectX-7 for Telecommunication Applications ........... 9 ConnectX-7 Socket Direct Ready Cards for Dual-Slot Servers ....... 10 Technical Support ................11 Related Documentation ..............12 Document Conventions...............13 Revision History ................14 Introduction..................15...
  • Page 3 Identifying the Card in Your System............. 29 ConnectX-7 PCIe x16 Installation Instructions..........31 Installing the Card ................31 Uninstalling the Card ................32 ConnectX-7 Socket Direct (2x PCIe x16) Installation Instructions ......34 Installing the Cards ................35 Uninstalling the Cards................. 40 Driver Installation................43 Linux Driver Installation ................
  • Page 4 mlxconfig.................... 69 UEFI ....................69 Troubleshooting ................70 General Troubleshooting ................70 Linux Troubleshooting ................70 Windows Troubleshooting................. 71 Specifications..................72 MCX75310AAC-NEAT / MCX75310AAS-NEAT Specifications........72 MCX75310AAS-HEAT Specifications .............. 73 MCX755106AC-HEAT / MCX755106AS-HEAT Specifications ........74 MCX715105AS-WEAT Specifications.............. 75 MCX75510AAS-HEAT Specifications .............. 76 MCX75510AAS-NEAT Specifications ..............
  • Page 5 Document Revision History ..............109  ...

  • Page 6: About This Manual

    About This Manual This User Manual describes NVIDIA® ConnectX®-7 InfiniBand and Ethernet adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.

  • Page 7: Intended Audience

    Intended Audience This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with InfiniBand and Ethernet network and architecture specifications.

  • Page 8: Ordering Part Numbers

    Ordering Part Numbers The table below lists the ordering part numbers (OPNs) for the available ConnectX-7 stand-up cards. For the Open Compute Project (OCP 3.0) cards, please refer to NVIDIA ConnectX-7 Adapter Cards for OCP 3.0 User Manual. ConnectX-7 PCIe x16 Stand-up Adapter Cards...

  • Page 9: Connectx-7 For Telecommunication Applications

    O-0003- Height, Half 50/25GbE SFP56 4.0 @ SERDES Bracket 04AS- Length 16GT/s ADAT 2.71 in. x 6.6 in. (68.90mm x 167.65 mm) ConnectX-7 for Telecommunication Applications NVIDIA Legacy Form Data No. of PCIe Secure Cryp Timing Bracke Factor Transmis Ports...

  • Page 10: Connectx-7 Socket Direct Ready Cards For Dual-Slot Servers

    ConnectX-7 Socket Direct Ready Cards for Dual-Slot Servers NVIDIA Form Legacy Data No. of PCIe Socket Secur Bracke Factor Transmissi Ports Support Direct e Boot t Type on Rate Ready - Type PCIe Extension Option ✔ 900-9X7A MCX715 PCIe Half...

  • Page 11: Technical Support

    • URL: https://www.nvidia.com > Support • E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller.

  • Page 12: Related Documentation

    Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km. In addition to meeting Ethernet and IBTA standards, NVIDIA tests every product in an end-to-end environment ensuring a Bit Error Rate of less than 1E-15. Read more at LinkX Cables and Transceivers.

  • Page 13: Document Conventions

    Document Conventions When discussing memory sizes, MB and MBytes are used in this document to mean size in MegaBytes. The use of Mb or Mbits (small b) indicates size in MegaBits. IB is used in this document to mean InfiniBand. In this document, PCIe is used to mean PCI Express.

  • Page 14: Revision History

    Revision History A list of the changes made to this document is provided in Document Revision History.

  • Page 15: Introduction

    AI, scientific computing, and hyperscale cloud data center workloads.  ConnectX-7 network adapters are offered in two form factors and various flavors: stand-up PCIe and Open Compute Project (OCP) Spec 3.0 cards. This user manual covers the PCIe stand-up cards, for the OCP 3.0 cards, please refer to...

  • Page 16: System Requirements

    EDR/HDR100/HDR/NDR200/NDR InfiniBand switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Package Contents Category Item Cards ConnectX-7 adapter card Accessories Adapter card short bracket Adapter card tall bracket (shipped assembled on the card)

  • Page 17: Features And Benefits

    PCIe Gen 4.0/5.0 ( SERDES @ 16GT/s / 32GT/s) through x16 edge connector PCIe Gen 4.0 SERDES @ 16GT/s through PCIe Auxiliary Connection Card InfiniBand ConnectX-7 delivers low latency, high bandwidth, and computing efficiency for high-performance computing (HPC), artificial intelligence (AI), and hyperscale cloud data center applications. Architect ConnectX-7 is InfiniBand Architecture Specification v1.5 compliant.
  • Page 18 NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-7 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.
  • Page 19 Secure on ConnectX-7. This is achieved using cryptographic primitives using asymmetric cryptography. Boot ConnectX-7 supports several cryptographic functions in its HW Root-of-Trust (RoT) that has its key stored in on-chip FUSES. Secure The Secure firmware update feature enables a device to verify digital signatures of new...
  • Page 20 PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe Managem protocols as if it is a standard NVIDIA PCIe stand-up adapter card. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.

  • Page 21: Supported Interfaces

    Supported Interfaces This section describes the ConnectX-7 supported interfaces. Each numbered interface that is referenced in the figures is described in the following table with a link to detailed information.  The below figures are for illustration purposes only and might not reflect the current revision of the adapter card.

  • Page 22: Interfaces Detailed Description

    PCI Express Interface ConnectX-7 adapter cards support PCI Express Gen 5.0 (4.0 and 3.0 compatible) through x16 edge connector. The following lists PCIe interface features: • PCIe Gen 5.0 compliant, 4.0, 3.0, 2.0 and 1.1 compatible •...

  • Page 23: Cabline Ca-Ii Plus Connectors

    • LED2 is a single-color LED (Green) State Bi-Color LED (Yellow/Green) Single Color LED (Green) Beacon command for locating the 1Hz blinking Yellow  adapter card Error 4Hz blinking Yellow Indicates an error with the link. The error can be one of the following: Error Description...

  • Page 24: Smbus Interface

    BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.

  • Page 25: Hardware Installation

    Hardware Installation Installation and initialization of ConnectX-7 adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings  Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions.

  • Page 26: Installation Procedure Overview

    Refer to the Specifications section of the manual for more power requirements.  Please make sure to install the ConnectX-7 cards in a PCIe slot that is capable of supplying the required power as stated in Specifications. ConnectX-7 Configuration...

  • Page 27: Airflow Requirements

    Airflow Requirements ConnectX-7 adapter cards are offered with two airflow patterns: from the heatsink to the network ports, and vice versa, as shown below. Please refer to the Specifications section for airflow numbers for each specific card model. Airflow from the heatsink to the network ports Airflow from the network ports to the heatsink...

  • Page 28: Bracket Replacement Instructions

    Bracket Replacement Instructions. Bracket Replacement Instructions The ConnectX-7 card and PCIe Auxiliary Connection card are usually shipped with an assembled high-profile bracket. If this form factor is suitable for your requirements, you can skip the remainder of this section and move to Installation...

  • Page 29: Cables And Modules

    OPNs Installation Instructions MCX755106AC-HEAT ConnectX-7 Socket Direct (2x PCIe x16) Adapter Card MCX755106AS-HEAT  MCX75510AAS-HEAT MCX75510AAS-NEAT Cables and Modules Cable Installation Before connecting a cable to the adapter card, ensure that the bracket is fastened to the server chassis using a screw to prevent movement or unplugging of the card when the cable is inserted or extracted.
  • Page 30 Single-port Socket Direct Card (2x PCIe [root@mftqa-009 ~]# lspci |grep mellanox -i x16) a3:00.0 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7] e3:00.0 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7] Dual-port Socket Direct Card (2x PCIe x16) [root@mftqa-009 ~]# lspci |grep mellanox -i 05:00.0 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7]...

  • Page 31: Connectx-7 Pcie X16 Installation Instructions

    MCX75510AAS-HEAT, MCX75510AAS-NEAT and MCX755106AS-HEAT, please refer to ConnectX-7 Socket Direct (2x PCIe x16) Adapter Card Installation instructions.  Please make sure to install the ConnectX-7 cards in a PCIe slot that is capable of supplying the required power and airflow as stated in Specifications.

  • Page 32: Uninstalling The Card

     The below images are for illustration purposes only.  Connect the adapter Card in an available PCI Express x16 slot in the chassis. Step 1: Locate an available PCI Express x16 slot and insert the adapter card to the chassis. Step 2: Applying even pressure at both corners of the card, insert the adapter card in a PCI Express slot until firmly seated.
  • Page 33 Verify that the system is powered off and unplugged. Wait 30 seconds. To remove the card, disengage the retention mechanisms on the bracket (clips or screws).       4. Holding the adapter card from its center, gently pull the ConnectX-7 card out of the PCI Express slot.

  • Page 34: Connectx-7 Socket Direct (2X Pcie X16) Installation Instructions

    The Cabline harness marked with the "WHITE" label should be connected to the connector on the ConnectX-7 and PCIe card engraved with “White Cable,”  while the one marked with the "BLACK" label should be connected to the connector on the ConnectX-7 and PCIe card engraved with “Black Cable”.

  • Page 35: Installing The Cards

       Please make sure to install the ConnectX-7 cards in a PCIe slot capable of supplying the required power and airflow as stated in the Specifications. Connect the adapter card with the Auxiliary connection card using the supplied Cabline CA-II Plus harnesses.
  • Page 36 Step 3: Slide the retention clip latches through the cutouts on the PCB. The latches should face the annotation on the PCB.  Step 4: Clamp the retention clip. Verify both latches are firmly locked.
  • Page 37 Step 5: Slide the Cabline CA-II Plus harnesses through the retention clip. Make sure that the clip opening is facing the plugs. ...
  • Page 38 Step 6: Plug the Cabline CA-II Plus harnesses into the PCIe Auxiliary Card. As indicated on both sides of the Auxiliary connection card, plug the black harness into the component side and the white harness into the print side. Step 7: Verify the plugs are locked. Step 8: Slide the retention clip through the cutouts on the PCB.
  • Page 39 Connect the ConnectX-7 adapter and PCIe Auxiliary Connection cards in available PCI Express x16 slots in the chassis. Step 1: Locate two available PCI Express x16 slots. Step 2: Applying even pressure at both corners of the cards, insert the adapter card in the PCI Express slots until firmly seated.

  • Page 40: Uninstalling The Cards

    Step 3: Applying even pressure at both corners of the cards, insert the Auxiliary Connection card in the PCI Express slots until firmly seated. Secure the ConnectX-7 adapter and PCIe Auxiliary Connection Cards to the chassis. Step 1: Secure the brackets to the chassis with the bracket screws.
  • Page 41 The adapter is installed in a system that operates with voltages that can be lethal. Before uninstalling the adapter card, please observe the following precautions to avoid injury and prevent damage to system components. Remove any metallic objects from your hands and wrists. Using an ESD strap or other antistatic devices is strongly recommended.
  • Page 42       4. Holding the adapter card from its center, gently pull the ConnectX-7 and Auxiliary Connections cards out of the PCI Express slot.

  • Page 43: Driver Installation

    The installation requires administrator (root) privileges on the target machine. Downloading MLNX_OFED Verify that the system has a network adapter installed by running lspci command. The below table provides output examples per ConnectX-7 card configuration. ConnectX-7 Card Output Examples Configuration...

  • Page 44: Installing Mlnx_Ofed

    Direct Card (2x PCIe [root@mftqa-009 ~]# lspci |grep mellanox -i x16) 05:00.0 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7] 05:00.1 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7] 82:00.0 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7] 82:00.1 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7] In the output example above, the first...
  • Page 45 • Uninstalls any software stacks that are part of the standard operating system distribution or another vendor's commercial stack • Installs the MLNX_OFED_LINUX binary RPMs (if they are available for the current kernel) • Identifies the currently installed InfiniBand and Ethernet network adapters and automatically upgrades the firmware Note: To perform a firmware upgrade using customized firmware binaries, a path can be provided to the folder that contains the firmware binary files, by running --fw-...
  • Page 46 At this point, OFED for Linux has already installed the OpenSM Subnet Manager on your machine. For the list of installation options, run: ./mlnxofedinstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards.  Log in to the installation machine as root.
  • Page 47 Mount the ISO image on your machine.  host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Run the installation script. /mnt/mlnxofedinstall Logs dir: /tmp/MLNX_OFED_LINUX-x.x-x.logs This program will install the MLNX_OFED_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to conflicts with MLNX_OFED_LINUX, not reinstall them.
  • Page 48 FW 12.14.0114 N/A Status: No matching image found Error message #2: The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor. Case A: If the installation script has performed a firmware update on your network adapter, you need to either restart the driver or reboot your system before the firmware update can take effect.

  • Page 49: Driver Load Upon System Boot

    Example: Logs dir: /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0.IBMM2150110033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically.  To prevent the automatic load of the NVIDIA drivers upon system boot: Add the following lines to the "/etc/modprobe.d/mlnx.conf" file. ...
  • Page 50 blacklist mlx5_core blacklist mlx5_ib Set “ONBOOT=no” in the "/etc/infiniband/openib.conf" file. If the modules exist in the initramfs file, they can automatically be loaded by the kernel. To prevent this behavior, update the initramfs using the operating systems’ standard tools. Note: The process of updating the initramfs will add the blacklists from step 1, and will prevent the kernel from loading the modules automatically.

  • Page 51: Additional Installation Procedures

    "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
  • Page 52 Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link:  http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox --2018-01-25 13:52:30-- http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox Resolving www.mellanox.com... 72.3.194.0 Connecting to www.mellanox.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... Length: 1354 (1.3K) [text/plain] Saving to: ?RPM-GPG-KEY-Mellanox?  ...
  • Page 53 Building MLNX_OFED_LINUX RPMS . Please wait... Creating metadata-rpms 3.10.0-957.21.3.el7.x86_64 ... WARNING: If you are going to configure this package as a repository, then please note WARNING: that it contains unsigned rpms, therefore, you need to disable the gpgcheck WARNING: by setting 'gpgcheck=0' in the repository conf file.
  • Page 54 Installs all available packages in MLNX_OFED mlnx-ofed-basic Installs basic packages required for running NVIDIA cards mlnx-ofed-guest Installs packages required by guest OS mlnx-ofed-hpc Installs packages required for HPC mlnx-ofed-hypervisor Installs packages required by hypervisor OS mlnx-ofed-vma Installs packages required by VMA...
  • Page 55 Create an apt-get repository configuration file called "/etc/apt/sources.list.d/ mlnx_ofed.list" with the following content:  deb file:/<path to extracted MLNX_OFED package>/DEBS ./ Download and install NVIDIA's Technologies GPG-KEY.  # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Verify that the key was successfully imported. ...
  • Page 56 # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Build the packages with kernel support and create the tarball.  # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7.6 under /tmp directory. Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log  ...

  • Page 57: Performance Tuning

    Depending on the application of the user's system, it may be necessary to modify the default configuration of network adapters based on the ConnectX® adapters. In case that tuning is required, please refer to the Performance Tuning Guide for NVIDIA Network Adapters. Windows Driver Installation...

  • Page 58: Software Requirements

    On an x64 (64-bit) machine, the output will be “AMD64”. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > InfiniBand Drivers (Learn More) > Nvidia WinOF-2. Download the .exe image according to the architecture of your machine (see Step 1). ...
  • Page 59 • Attended Installation An installation procedure that requires frequent user intervention. • Unattended Installation An automated installation procedure that requires no user intervention. Attended Installation The following is an example of an installation session. Double click the .exe and follow the GUI instructions to install MLNX_WinOF2. [Optional] Manually configure your setup to contain the logs option (replace “LogFile”...
  • Page 60 Read and accept the license agreement and click Next. Select the target folder for the installation.
  • Page 61 • If the user has a standard NVIDIA® card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA® card, only the NVIDIA® card will be updated.
  • Page 62 Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes •...
  • Page 63 Click Next to install the desired tools. Click Install to start the installation. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see  ). ...
  • Page 64 13. Click Finish to complete the installation.
  • Page 65 Unattended Installation  If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD.

  • Page 66: Firmware Upgrade

    Firmware Upgrade If the machine has a standard NVIDIA® card with an older firmware version, the firmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to MFT User Manual. ...

  • Page 67: Removing Earlier Nvidia Drivers

     After the installation process, all kernel modules are loaded automatically upon boot. Removing Earlier NVIDIA Drivers  Please unload the previously installed drivers before removing them. To remove all the drivers: Log into the ESXi server with root permissions.

  • Page 68: Updating Adapter Firmware

    Each adapter card is shipped with the latest version of qualified firmware at the time of manufacturing. However, NVIDIA issues firmware updates occasionally that provide new features and bug fixes. To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility.

  • Page 69: Setting High-Speed-Port Link Type

    Setting High-Speed-Port Link Type  This section applies to ConnectX-7 cards supporting both Ethernet and InfiniBand protocols - see the relevant OPNs in the following table.  The following table lists the ConnectX-7 cards supporting both Ethernet and InfiniBand protocols, the supported speeds and the default networking port link type.

  • Page 70: Troubleshooting

    Troubleshooting General Troubleshooting Server unable to find the adapter • Ensure that the adapter is placed correctly • Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...

  • Page 71: Windows Troubleshooting

    -d <mst_device> q Ports Information ibstat ibv_devinfo Firmware Version Upgrade To download the latest firmware version, refer to the NVIDIA Update and Query Utility. Collect Log File cat /var/log/messages dmesg >> system.log journalctl (Applicable on new operating systems) cat /var/log/syslog Windows Troubleshooting...

  • Page 72: Specifications

     Please make sure to install the ConnectX-7 card in a PCIe slot that is capable of supplying the required power and airflow as stated in the below table. MCX75310AAC-NEAT / MCX75310AAS-NEAT Specifications ...

  • Page 73: Mcx75310Aas-Heat Specifications

    EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-7 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.

  • Page 74: Mcx755106Ac-Heat / Mcx755106As-Heat Specifications

    EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-7 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.

  • Page 75: Mcx715105As-Weat Specifications

    EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-7 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.

  • Page 76: Mcx75510Aas-Heat Specifications

    EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-7 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.

  • Page 77: Mcx75510Aas-Neat Specifications

    EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-7 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.
  • Page 78 EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-7 adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA InfiniBand product.

  • Page 79: Mcx713106Ac-Ceat And Mcx713106As-Ceat Specifications

    Thermal Maximum power available through QSFP112 port: 11W per port (Not thermally supported), 2.5W Specificati per port (Thermally Supported) Electrical and thermal specifications are provided in NVIDIA ConnectX-7 Electrical and Thermal Specifications document, which is available at NVOnline following login. Environme Temperatur Operatio 0°C to 55°C...

  • Page 80: Mcx713106Ac-Veat And Mcx713106As-Veat Specifications

    Thermal Maximum power available through QSFP112 port: 11W per port (Not thermally supported), Specificatio 5.1W per port (Thermally supported) Electrical and thermal specifications are provided in NVIDIA ConnectX-7 Electrical and Thermal Specifications document, which is available at NVOnline following login. Environmen Temperature Operational 0°C to 55°C...

  • Page 81: Mcx713104Ac-Adat And Mcx713104As-Adat Specifications

    Voltage: 12V, 3.3VAUX Electrical and Maximum current: 100mA Thermal Specifications Maximum power available through Quad-port SFP56: TBD  Power and airflow specifications are provided in NVIDIA ConnectX-7 Electrical and Thermal Specifications document, which is available at NVOnline following login. Environmental Temperatur Operationa 0°C to 55°C Non- -40°C to 70°C...

  • Page 82: Cards Mechanical Drawings And Dimensions

    Interfaces See Supported Interfaces PCI Express Gen 4.0: SERDES @ 16GT/s, x16 lanes (4.0 and 3.0 compatible) Networking Port: Quad-port SFP56 Ethernet (copper and optical) Protocol Data Rate Ethernet 50/25 GbE Support Ethernet Protocols: 50GAUI-2 C2M, 50GAUI-1 C2M, 50GBASE-CR, 50GBASE-R2 , 40GBASE- CR4, 40GBASE-R2, 25GBASE-R, 10GBASE-R, 10GBASE-CX4, 1000BASE-CX, CAUI-4 C2M, 25GAUI C2M, XLAUI C2M , XLPPI, SFI Capabilities...

  • Page 83: Brackets Mechanical Drawings And Dimensions

    Single-port x16 QSFP112 Adapter Card   Quad-port SFP56 Adapter Card   Single-port x16 OSFP Adapter Card   Auxiliary PCIe Connection Card  Brackets Mechanical Drawings and Dimensions  All dimensions are in millimeters. The PCB mechanical tolerance is +/- 0.13mm. Dual-port QSFP112 Tall Bracket Dual-port QSFP112 Short Bracket Single-port QSFP112 Tall Bracket Single-port QSFP112 Short Bracket...
  • Page 84 Dual-port QSFP112 Tall Bracket Dual-port QSFP112 Short Bracket Quad-port SFP56 Tall Bracket Quad-port SFP56 Short Bracket Auxiliary PCIe Connection Card Tall Bracket Auxiliary PCIe Connection Card Short Bracket...

  • Page 85: Monitoring

    IC thermal requirements. Adapter Card Heatsink The heatsink is attached to the ConnectX-7 IC to dissipate the heat from the ConnectX-7 IC. It is attached by spring-loaded screws. ConnectX-7 IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX-7 card in cases of high-temperature events, improper thermal coupling, or heatsink removal. ...

  • Page 86: Finding The Guid/Mac On The Adapter Card

    Finding the GUID/MAC on the Adapter Card Each adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol. InfiniBand/Ethernet cards have both a GUID and a MAC (derived from the GUID). IB only cards have GUID for the InfiniBand protocol.

  • Page 87: Pcie Auxiliary Card Kit

    (top side) of the network card, while the White Cabline SA-II Plus harnesses mates with the pint side (bottom side) of the main network card. For hardware installation, please refer to ConnectX-7 Socket Direct (2x PCIe x16) Installation Instructions.

  • Page 88: Pcie Auxiliary Card Package Contents

    PCIe Auxiliary Card Package Contents Category Item Cards MTMK9100-T15: PCIe x16 Gen 5.0/4.0 Auxiliary Connection Card MTMK9100-T25 and MTMK9100-T35: PCIe x16 Gen 4.0 Auxiliary Connection Card Harnesses Cabline CA-II Plus harness (white) - Length according to kit OPN (15, 25 or 35cm) Cabline CA-II Plus harness (black) - Length according to kit OPN (15, 25 or 35cm) Retention Clip for Cabeline harness (shipped assembled on...
  • Page 89 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end PCIE_REFCLK1_N Micro coax   Primary PCIe clock from the motherboard to the Network Adapter Main card, to be used for the x16 Cabline harness PCIe interface. This clock must meet all the PCIe SIG spec requirements.
  • Page 90 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end PCIE_CPU_CX_13P  Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point GND BAR PCIE_CPU_CX_12N  Micro coax  ...
  • Page 91 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end GND BAR PCIE_CPU_CX_9N  Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point PCIE_CPU_CX_9P  Micro coax  ...
  • Page 92 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end PCIE_CPU_CX_6N  Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point PCIE_CPU_CX_6P  Micro coax   Connect from the Black Cabline CA-II Plus cable through...
  • Page 93 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end PCIE_CPU_CX_3P  Micro coax   Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 PCIe Tx lanes of the CPU/ GPU/End-Point GND BAR PCIE_CPU_CX_2N  Micro coax  ...
  • Page 94 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end GND BAR I2C_SMDAT Micro coax   PCIe compliant SMBus interface to the network adapter. The Network adapter silicon may be accessed through this SMBus interface after the Power-On sequence of the network adapter is completed.
  • Page 95 Pin# Signal Name Wire Type Detailed AWG# Pin# on the Description other end I2C_SMCLK Micro coax   PCIe compliant SMBus interface to the network adapter. The Network adapter silicon may be accessed through this SMBus interface after the Power-On sequence of the network adapter is completed.
  • Page 96 I2C_AUX_SCL Micro coax   The ConnectX-7 silicon serves as the I2C bus master on this bus. An I2C EEPROM at I2C address 0x57 needs to be mounted on...

  • Page 97: Print Side

    Signal Name Wire Type Detailed AWG# Pin# on the Description other end Micro coax   The ConnectX-7 I2C_AUX_SDA silicon serves as the I2C bus master on this bus. An I2C EEPROM at I2C address 0x57 needs to be mounted on the motherboard...
  • Page 98 Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card.  Micro coax   Connect from the PCIE_CX_CPU_0N White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point.
  • Page 99 Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card.  Micro coax   Connect from the PCIE_CX_CPU_1N White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point.
  • Page 100 White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card.  GND BAR PCIE_CX_CPU_4P Micro coax   Connect from the White Cabline CA-II Plus harness to the...
  • Page 101 Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card.  Micro coax   Connect from the PCIE_CX_CPU_6N White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point.
  • Page 102 White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card.  GND BAR PCIE_CX_CPU_9P Micro coax   Connect from the White Cabline CA-II Plus harness to the...
  • Page 103 Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card.  Micro coax   Connect from the PCIE_CX_CPU_11N White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point.
  • Page 104 White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card.  GND BAR PCIE_CX_CPU_14P Micro coax   Connect from the White Cabline CA-II Plus harness to the...
  • Page 105 Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end Micro coax   PCIe compliant S_PERST1_CONN_L PERST_L (active low PCI Reset) signal for the PCIe Cabline CA- II Plus Connectors. The direction of this PERST_L signal depends on the implementation: When connecting a CPU root complex to...

  • Page 106: Pcie Auxiliary Kit Technical Specifications

    Pin# Signal Name Wire Type AWG# Detailed Pin# on the Description other end S_PRSNT2_L Micro coax   Connect to a 4.7K pull-up resistor to 3.3V on the motherboard side, to detect if both the Cabline harnesses are connected or not. This signal is connected to S_PRSNT1_L on the network adapter...

  • Page 107: Pcie Auxiliary Card Mechanical Drawings And Dimensions

    Non-operational 10% to 90% relative humidity  Altitude (Operational) 3050m Regulatory Safety: CB / cTUVus / CE EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant PCIe Auxiliary Card Mechanical Drawings and Dimensions  All dimensions are in millimeters. The PCB mechanical tolerance is +/- 0.13mm. Bracket Mechanical Drawings and Dimensions Auxiliary PCIe Connection Card Tall Bracket Auxiliary PCIe Connection Card Short Bracket...

  • Page 108: Cabline Ca-Ii Plus Harnesses Mechanical Drawing

    Cabline CA-II Plus Harnesses Mechanical Drawing...
  • Page 109 Specifications Dec. 2022 • Modified ordering part numbers and product descriptions in NVIDIA ConnectX-7 Adapter Cards User Manual • Added PCIe Auxiliary Card Kit • Updated Monitoring Nov. 2022 First release of the consolidated user manual for all ConnectX-7 adapter cards...
  • Page 110 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
  • Page 111 Copyright © 2023 NVIDIA Corporation & affiliates. All Rights Reserved.

Table of Contents