Download
Share
Print this page
Nvidia ConnectX-7 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Adapter
  5. ConnectX-7
  6. User manual
Nvidia ConnectX-7 User Manual

Nvidia ConnectX-7 User Manual

Adapter cards
Hide thumbs Also See for ConnectX-7:

Advertisement

Quick Links

Download this manual
NVIDIA ConnectX-7 Adapter Cards User Manual

Advertisement

loading
Need help?

Need help?

Do you have a question about the ConnectX-7 and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia ConnectX-7

Summary of Contents for Nvidia ConnectX-7

  • Page 1 NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 2 Supported Interfaces Hardware Installation Hardware Installation ConnectX-7 PCIe x16 Installation Instructions ConnectX-7 PCIe x16 Installation Instructions ConnectX-7 Socket Direct (2x PCIe x16) Installation Instructions ConnectX-7 Socket Direct (2x PCIe x16) Installation Instructions Driver Installation Driver Installation Linux Driver Installation Linux Driver Installation...
  • Page 3 In niBand and Ethernet network and architecture speci cations. Ordering Part Numbers The table below lists the ordering part numbers (OPNs) for the available ConnectX-7 stand-up cards. For the Open Compute Project (OCP 3.0) cards, please refer to NVIDIA ConnectX-7 Adapter Cards for OCP 3.0 User Manual.
  • Page 4 (68.90mm x Etherne 167.65 mm) 400GbE 900- MCX75310AAS- PCIe Half In niBa Single-port 9X766 HEAT Height, Half OSFP -003N- Length NDR20 2.71 in. x 6.6 in. (68.90mm x 200Gb/ 167.65 mm) (Default speed) NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 5 Etherne 200GbE PCIe Half 900- Height, Half Etherne 9X7AO MCX713104AC- Length Quad-port -00C3- ADAT 2.71 in. x 6.6 in. 50/25G SFP56 (68.90mm x 167.65 mm) NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 6 Rate MCX71 PCIe Full Height, Half Ethe PCIe Tall 3114TC Length rnet: -GEAT 4.53 in. x 6.6 in. (115.15 mm 50/2 port /Out, x 167.65 mm) 5GbE 4.0 @ SERD Sycn 16GT/ NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 7 Data Timi Tran Port PCIe Legacy Form Factor smis Suppo Capa sion bilitie Rate ConnectX-7 Socket Direct Ready Cards for Dual-Slot Servers NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 8 NVIDI Legacy OPN A SKU 900- MCX715105AS-WEAT 9X7A 0039- NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 9 900- MCX75510AAS-NEAT 9X72 003N- NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 10 900- MCX75510AAS-HEAT 9X72 003N- NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 11 900- MCX755106AS-HEAT 9X7A 0078- NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 12 900- MCX755106AC-HEAT 9X7A 0079- NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 13 Data No. of NVIDI Timing Legacy Form Trans Ports PCIe Lifec Capabiliti Factor missio Support ycle n Rate Type 900- MCX7 PCIe Half Ethern Dual- PCIe x16 Tall 9X7A 13106 Height, port Gen 4.0/5.0 NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 14 900- MCX7 PCIe Half Ethern Dual- PCIe x16 Tall 9X7A 13106 Height, port Gen 4.0/5.0 Half 100Gb QSFP @ SERDES Prod 0076 CEAT Length 16GT/s/32G uctio -ST0 2.71 in. x 6.6 in. (68.90m 167.65 NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 15 167.65 S Out For more information, please refer to PCIe Auxiliary Card Kit. Technical Support Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: URL: https://www.nvidia.com > Support E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support.
  • Page 16 Solutions high-bandwidth, low-latency connections between compute nodes and switch nodes. NVIDIA o ers one of the industry’s most complete line of 10, 25, 40, 50, 100, 200, and 400GbE in Ethernet and EDR, HDR, and NDR, including Direct Attach Copper cables (DACs), copper...
  • Page 17 0.5m to 10km. In addition to meeting Ethernet and IBTA standards, NVIDIA tests every product in an end-to-end environment ensuring a Bit Error Rate of less than 1E-15 . Read more at LinkX Cables and Transceivers.
  • Page 18 AI, scienti c computing, and hyperscale cloud data center workloads. ConnectX-7 network adapters are o ered in two form factors and various avors : stand- up PCIe and Open Compute Project (OCP) Spec 3.0 cards. This user manual covers the PCIe stand-up cards, for the OCP 3.0 cards, please refer to NVIDIA ConnectX-7 Cards for...
  • Page 19 CPU in a dual-socket server to the network through its dedicated PCIe interface. NVIDIA o ers ConnectX-7 Socket Direct adapter cards, which enable 400Gb/s or 200Gb/s connectivity, and also for servers with PCIe Gen 4.0 capability. The adapter’s 32-lane PCIe bus is split into two 16-lane buses, with one bus accessible through a PCIe x16 edge connector and the other bus through an x16 Auxiliary PCIe Connection card.
  • Page 20 Single-port NDR200 200Gb/s OSFP Dual-port NDR200 200Gb/s and 200GbE QSFP112 Single-port NDR 400Gb/s and 400GbE QSFP112 For more information on the passive PCIe Auxiliary kit, please refer to PCIe Auxiliary Card Kit. System Requirements NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 21 Powered connectors for optical and active cable support Package Contents Category Item Cards ConnectX-7 adapter card Adapter card short bracket Accessories Adapter card tall bracket (shipped assembled on the card) Features and Bene ts NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 22 Note Make sure to use a PCIe slot capable of supplying the required power and air ow to the ConnectX-7 cards as stated in the Speci cations chapter. Note This section describes hardware features and capabilities. Please refer to the relevant driver and rmware release notes for feature availability.
  • Page 23 1.3.1 encodin 64b/66 IBTA Vol2 56.25 encodin Up to ConnectX-7 adapter cards comply with the following IEEE 802.3 standards: 400GbE / 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE Gigabit Etherne NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 24 25/50 Gigabit Ethernet Techno logy Consor tium IEEE 802.3b 40 Gigabit Ethernet IEEE 802.3a 10 Gigabit Ethernet IEEE 2.5/5 Gigabit Ethernet 802.3c (For 2.5: support only 2.5 x1000BASE-X) IEEE 802.3a Based on auto-negotiation and KR startup NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 25 MAC Rate IEEE 802.3a Link Aggregation IEEE 802.1A IEEE 802.1Q IEEE 802.1P VLAN tags priority IEEE 802.1Q (QCN) Conge stion Noti c ation IEEE 802.1Q (ETS) 802.1Q (PFC) IEEE 802.1Q IEEE 1588v IEEE 802.1A (MACS NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 26 Network network scalability issues, it hides the TCP packet from the hardware o oading engines, placing higher loads on the host CPU. ConnectX-7 e ectively addresses this by providing advanced NVGRE and VXLAN hardware o oading engines that encapsulate and de-capsulate the overlay protocol.
  • Page 27 BMC. ConnectX-7 PCIe stand-up adapter can be connected to a BMC using ment MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter card. For con guring the adapter for the speci c manageability solution in use by the server, please contact NVIDIA Support.
  • Page 28 Boot setting NVIDIA o ers a full IEEE 1588v2 PTP software solution, as well as time- sensitive related features called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most demanding PTP pro les. ConnectX-7 incorporates an integrated Hardware Clock (PHC) that allows ConnectX-7 to...
  • Page 29 (ORAN) compatible in high bandwidth. The PTP part supports the subordinate clock, master clock, and boundary clock. ConnectX-7 PTP solution allows you to run any PTP stack on your host. With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS-out signal from the onboard SMA connecter, ConnectX-7 also allows measuring PTP in scale, with PPS-In signal.
  • Page 30 Supported Interfaces This section describes the ConnectX-7 supported interfaces. Each numbered interface that is referenced in the gures is described in the following table with a link to detailed information. Note The below gures are for illustration purposes only and might not re ect the current revision of the adapter card.
  • Page 31 ConnectX-7 IC  ConnectX-7 Integrated Circuit PCI Express PCIe Gen 4.0/5.0 through x16 edge connector Interface Networking Network tra c is transmitted through the adapter card Interfaces networking connectors. The networking connectors allow for the NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 32 Applies to OPN: MCX713114TC-GEAT only. Interfaces Detailed Description ConnectX-7 IC The ConnectX-7 family of adapter IC devices delivers In niBand and Ethernet connectivity paired with best-in-class hardware capabilities that accelerate and secure cloud and data- center workloads. PCI Express Interface ConnectX-7 adapter cards support PCI Express Gen 5.0 (4.0 and 3.0 compatible) through...
  • Page 33 Architecture Speci cation, Release 1.5. In niBand tra c is transmitted through the cards' networking connectors. Networking Ports LEDs Speci cations For the networking ports LEDs description, follow the below table depending on the ConnectX-7 SKU you have purchased. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 34 Over- Blinks until error is current networking ports Physical Activity Green LED is blinking Link Up Green LED is solid Physical Up (IB Only) Yellow LED is solid Scheme 2: Two LEDs NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 35 In less than full port speed: the Yellow LED is Blinking solid In full port speed: the Green LED is solid Link Up In less than full port speed: the Yellow LED is solid Physical Up (IB Only) Yellow LED is solid NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 36 Pulse Per Second (PPS) is an out-of-band signal used in synchronized systems. 5T technology support PPS-in and PPS-out on selected devices. Selected ConnectX-7 adapter cards incorporate an integrated Hardware Clock (PHC) that allows the adapter to achieve sub-20u Sec accuracy and also o ers many timing-related functions such as time-triggered scheduling or time-based SND accelerations (time- based ASAP²).
  • Page 37 With respect to testing and measurements, selected ConnectX-7 adapters allow you to use the PPS-out signal from the onboard MMCX RA connecter. The adapter also allows measuring PTP in scale with the PPS-In signal. The PTP HW clock on the Network adapter is sampled on each PPS-In signal, and the timestamp is sent to the SW.
  • Page 38 PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For con guring the adapter for the speci c manageability solution in use by the server, please contact NVIDIA Support.
  • Page 39 Hardware Installation Installation and initialization of ConnectX-7 adapter cards require attention to the mechanical attributes, power speci cation, and precautions for electronic equipment. Safety Warnings Info Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions.
  • Page 40 CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Installation Procedure Overview The installation procedure of ConnectX-7 adapter cards involves the following steps: Step Procedure Direct Link Check the system’s hardware and System Requirements software requirements.
  • Page 41 System Requirements Hardware Requirements Warning Unless otherwise speci ed, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operating environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for...
  • Page 42 Air ow Requirements ConnectX-7 adapter cards are o ered with two air ow patterns: from the heatsink to the network ports, and vice versa, as shown below. Please refer to the Speci cations section for air ow numbers for each speci c card model.
  • Page 43 All cards in the system should be planned with the same air ow direction. Software Requirements See System Requirements section under the Introduction section. Software Stacks - NVIDIA® OpenFabrics Enterprise Distribution for Linux (MLNX_OFED), WinOF-2 for Windows, and VMware. See the Driver Installation section. Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal.
  • Page 44 Shut down your system if active; Turn o the power to the system, and disconnect the power cord. Refer to the system documentation for instructions. Before you install the ConnectX-7 card, make sure that the system is disconnected from power. (Optional) Check the mounting bracket on the ConnectX-7 or PCIe Auxiliary Connection Card;...
  • Page 45 Removing the Existing Bracket 1. Using a torque driver, remove the two screws holding the bracket in place. 2. Separate the bracket from the ConnectX-7 card. Warning Be careful not to put stress on the LEDs on the adapter card.
  • Page 46 Use a torque driver to apply up to 2 lbs-in torque on the screws. Installation Instructions This section provides detailed instructions on how to install your adapter card in a system. Choose the installation instructions according to the ConnectX-7 con guration you would like to use. OPNs...
  • Page 47 6. To remove a cable, disengage the locks and slowly pull the connector away from the port receptacle. The LED indicator will turn o when the cable is unseated. Identifying the Card in Your System On Linux NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 48 PCI Bus and Device number. Each of the PCIe x16 busses sees two network ports; in e ect, the two physical ports of the ConnectX-7 Socket Direct adapter are viewed as four net devices by the system.
  • Page 49 1. Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. 2. Expand System Devices and locate your ConnectX-7 adapter card. 3. Right-click the mouse on your adapter's row and select Properties to display the adapter card properties window.
  • Page 50 0x15B3 – this is the Vendor ID of Mellanox Technologies, and DEV is equal to 1021 (for ConnectX-7) – this is a valid NVIDIA PCI Device ID. Note If the PCI device does not have an NVIDIA adapter ID, return to Step 2 to check another device.
  • Page 51 Note Please make sure to install the ConnectX-7 cards in a PCIe slot that is capable of supplying the required power and air ow as stated in Speci cations. Note The below images are for illustration purposes only. Connect the adapter Card in an available PCI Express x16 slot in the chassis.
  • Page 52 1. Remove any metallic objects from your hands and wrists. 2. It is strongly recommended to use an ESD strap or other antistatic devices. 3. Turn o the system and disconnect the power cord from the server. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 53 3. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). 4. Holding the adapter card from its center, gently pull the ConnectX-7 card out of the PCI Express slot. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 54 ConnectX-7 Socket Direct (2x PCIe x16) Installation Instructions Note This section applies to the following adapter cards when used as Socket Direct cards in dual-socket servers. MCX755106AS-HEAT MCX755106AC-HEAT MCX75510AAS-NEAT MCX75510AAS-HEAT MCX715105AS-WEAT Note NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 55 The Cabline harness marked with the "WHITE" label should be connected to the connector on the ConnectX-7 and PCIe card engraved with “White Cable,” while the one marked with the "BLACK" label should be connected to the connector on the ConnectX-7 and PCIe card engraved with “Black Cable”.
  • Page 56 Step 2: Plug the Cabline CA-II Plus harnesses into the ConnectX-7 adapter card while paying attention to the color coding. As indicated on both sides of the card, plug the black harness into the component side and the white harness into the print side.
  • Page 57 Step 3: Slide the retention clip latches through the cutouts on the PCB. The latches should face the annotation on the PCB. Step 4: Clamp the retention clip. Verify both latches are rmly locked. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 58 Step 5: Slide the Cabline CA-II Plus harnesses through the retention clip. Make sure that the clip opening is facing the plugs. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 59 Step 7: Verify the plugs are locked. Step 8: Slide the retention clip through the cutouts on the PCB. Ensure latches are facing "Blthe ack Cable" annotation, as seen in the picture below. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 60 Step 9: Clamp the retention clip. Verify both latches are rmly locked. Step 1: Locate two available PCI Express x16 slots. Step 2: Applying even pressure at both corners of the cards, insert the adapter card in the PCI Express slots until rmly seated. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 61 Do not use excessive force when seating the cards, as this may damage the system or the cards. Step 3: Applying even pressure at both corners of the cards, insert the Auxiliary Connection card in the PCI Express slots until rmly seated. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 62 Step 1: Secure the brackets to the chassis with the bracket screws. Uninstalling the Cards Safety Precautions NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 63 Please note that the following images are for illustration purposes only. 1. Verify that the system is powered o and unplugged. 2. Wait 30 seconds. 3. To remove the card, disengage the retention mechanisms on the brackets (clips or screws). NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 64 4. Holding the adapter card from its center, gently pull the ConnectX-7 and Auxiliary Connections cards out of the PCI Express slot. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 65 The installation requires administrator (root) privileges on the Installer Privileges target machine. Downloading MLNX_OFED 1. Verify that the system has a network adapter installed by running lspci command. The below table provides output examples per ConnectX-7 card con guration. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 66 PCI Bus and Device number. Each of the PCIe x16 busses sees two network ports; in e ect, the two physical ports of the ConnectX-7 Socket Direct adapter are viewed as four net devices by the system. Single-port PCIe x16...
  • Page 67 Technologies MT2910 Family [ConnectX-7] 86:00.1 Network controller: Mellanox Technologies MT2910 Family [ConnectX-7] For Linux driver installation, please refer to NVIDIA DOCA Installation Guide for Linux. Windows Driver Installation For Windows, download and install the latest WinOF-2 for Windows software package available via the NVIDIA website at: WinOF-2 webpage.
  • Page 68 On an x64 (64-bit) machine, the output will be “AMD64”. 2. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > In niBand Drivers (Learn More) > Nvidia WinOF-2. 3. Download the .exe image according to the architecture of your machine (see Step 1).
  • Page 69 3. [Optional] If you do not want to upgrade your rmware version (i.e., MT_SKIPFWUPGRD default value is False). MLNX_WinOF2_<revision_version>_All_Arch.exe /v " MT_SKIPFWUPGRD=1" 4. [Optional] If you do not want to install the Rshim driver, run. MLNX_WinOF2_<revision_version>_All_Arch.exe /v " MT_DISABLE_RSHIM_INSTALL=1" NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 70 5. [Optional] If you want to skip the check for unsupported devices, run. MLNX_WinOF2_<revision_version>_All_Arch.exe /v " SKIPUNSUPPORTEDDEVCHECK=1" 6. Click Next in the Welcome screen. 7. Read and accept the license agreement and click Next. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 71 If the user has an OEM card. In this case, the rmware will not be displayed. If the user has a standard NVIDIA® card with an older rmware version, the rmware will be updated accordingly. However, if the user has both an OEM...
  • Page 72 NVIDIA® card, only the NVIDIA® card will be updated. 10. Select a Complete or Custom installation, follow Step a onward. 1. Select the desired feature to install: Performances tools - install the performance tools that are used to...
  • Page 73 Management tools - installation tools used for management, such as mlxstat Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd 2. Click Next to install the desired tools. 11. Click Install to start the installation. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 74 12. In case rmware upgrade option was checked in Step 7, you will be noti ed if a rmware upgrade is required (see 13. Click Finish to complete the installation. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 75 To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. 1. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD. 2. Install the driver. Run: NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 76 5. [Optional] If you do not wish to upgrade your rmware version (i.e., MT_SKIPFWUPGRD default value is False ). MLNX_WinOF2-[Driver/Version]_<revision_version>_All_Arch.exe /vMT_SKIPFWUPGRD= 6. [Optional] If you do not want to install the Rshim driver, run. MLNX_WinOF2_<revision_version>_All_Arch.exe /v " MT_DISABLE_RSHIM_INSTALL=1" Note NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 77 SKIPUNSUPPORTEDDEVCHECK=1" Firmware Upgrade If the machine has a standard NVIDIA® card with an older rmware version, the rmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade rmware manually, please refer to MFT User Manual.
  • Page 78 VMware Driver Installation This section describes VMware Driver Installation. Hardware and Software Requirements Requirement Description A server platform with an adapter card based on NVIDIA Platforms devices: ConnectX®-7 (In niBand/Ethernet) ( rmware: fw-ConnectX7) For the complete list of VMware supported operating systems,...
  • Page 79 After the installation process, all kernel modules are loaded automatically upon boot. Removing Earlier NVIDIA Drivers Note Please unload the previously installed drivers before removing them. To remove all the drivers: 1. Log into the ESXi server with root permissions. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 80 2. MD5SUM: 0804c e30913a7b4017445a0f0adbe1 2. Install the image according to the steps described in the MFT User Manual . Note The following procedure requires custom boot image downloading, mounting and booting from a USB device. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 81 Each adapter card is shipped with the latest version of quali ed rmware at the time of manufacturing. However, NVIDIA issues rmware updates occasionally that provide new features and bug xes. To check that your card is programmed with the latest available rmware version, download the mlxup rmware update and query utility.
  • Page 82 28.33.0800     28.33.1000 Status: Up to date   Perform FW update? [y/N]: y Device #1: Up to date Device #2: Updating FW ... Done   Restart needed for updates to take effect. Log File: /var/log/mlxup/mlxup-yyyymmdd.log NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 83 In niBand protocols - see the relevant OPNs in the following table. The following table lists the ConnectX-7 cards supporting both Ethernet and In niBand protocols, the supported speeds and the default networking port link type . Data Transmission Rate...
  • Page 84 The DHCP server instructs the PreBoot drivers to access the kernel/OS through a TFTP server, an iSCSI target, or some other service. For more information and instructions, refer to UEFI. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 85 Check that the latest driver is loaded Link light is on, but with Check that both the adapter and its link are set to the no communication same speed and duplex settings established Linux Troubleshooting NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 86 -d <mst_device> q ibstat Ports Information ibv_devinfo Firmware Version To download the latest rmware version, refer to the NVIDIA Upgrade Update and Query Utility. cat /var/log/messages dmesg >> system.log Collect Log File journalctl (Applicable on new operating systems)
  • Page 87 Upgrade int –d <mst_device> –i < rmware_bin_ le> b Event log viewer MST device logs: mst start Collect Log File mst status int –d <mst_device> dc > dump_con guration.log mstdump <mst_device> dc > mstdump.log NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 88 Note Please make sure to install the ConnectX-7 card in a PCIe slot that is capable of supplying the required power and air ow as stated in the below table.
  • Page 89 Thermal Gen 5.0 x16 Speci c The complete electrical and thermal speci cations are provided in " NVIDIA ations ConnectX-7 Adapter Card Product Speci cations" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
  • Page 90 EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes: a. The ConnectX-7 adapters supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product.
  • Page 91 Thermal Gen 5.0 x16 Speci c The complete electrical and thermal speci cations are provided in " NVIDIA ations ConnectX-7 Adapter Card Product Speci cations" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
  • Page 92 The ConnectX-7 adapters supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product. b. This table lists the electrical protocols. A wide range of optical protocols are delivered through the C2M interfaces listed in this table and optical modules.
  • Page 93 Thermal Gen 5.0 x16 Speci c The complete electrical and thermal speci cations are provided in " NVIDIA ations ConnectX-7 Adapter Card Product Speci cations" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
  • Page 94 RoHS: RoHS Compliant Notes : a. The ConnectX-7 adapters supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product. b. This table lists the electrical protocols. A wide range of optical protocols are delivered through the C2M interfaces listed in this table and optical modules.
  • Page 95 EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes : a. The ConnectX-7 adapters supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product.
  • Page 96 Auto-Negotiation: NDR200 ( 2 lanes x 100Gb/s per lane ) port, HDR (50Gb/s Support per lane) port, HDR100 (2 lane x 50Gb/s per lane), EDR (25Gb/s per lane) port, FDR (14.0625Gb/s per lane), 1X/2X/4X SDR (2.5Gb/s per lane) Capabilit Secure Boot Enabled, Crypto Disabled NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 97 EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes : a. The ConnectX-7 adapters supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product.
  • Page 98 Thermal Gen 5.0 x16 Speci c The complete electrical and thermal speci cations are provided in " NVIDIA ations ConnectX-7 Adapter Card Product Speci cations" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
  • Page 99 EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes : a. The ConnectX-7 adapters supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product.
  • Page 100 EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant Notes : a. The ConnectX-7 adapters supplement the IBTA auto-negotiation speci cation to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another NVIDIA In niBand product.
  • Page 101 Thermal QSFP112 cage Speci c The complete electrical and thermal speci cations are provided in " NVIDIA ations ConnectX-7 Adapter Card Product Speci cations" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
  • Page 102 Thermal Gen 4.0 x16 Speci c The complete electrical and thermal speci cations are provided in " NVIDIA ations ConnectX-7 Adapter Card Product Speci cations" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
  • Page 103 40GBASE-R2, 25GBASE-R, 10GBASE-R, 10GBASE-CX4, 1000BASE-CX, CAUI-4 C2M, 25GAUI C2M, XLAUI C2M , XLPPI, SFI Capabilit Enhanced-SyncE & PTP, PPS In/Out, Secure Boot, MCX713114TC-GEAT Crypto Enabled Electrica Voltage: 12V, 3.3VAUX l and Maximum current: 100mA Thermal NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 104 PCIe 15.8W Gen 4.0 x16 The complete electrical and thermal speci cations are provided in " NVIDIA ConnectX-7 Adapter Card Product Speci cations" document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
  • Page 105 Dual-port x16 QSFP112 Adapter Card Ready Adapter Card Single-port x16 QSFP112 Adapter Card HHHL Quad-port SFP56 Adapter Card HHHL Quad-port SFP56 with PPS IN/OUT Adapter Card Single-port x16 OSFP Adapter Card Auxiliary PCIe Connection Card NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 106 These di er slightly from the PCI CEM speci cation due to the mechanical constraint of the single quad-port SFP56 cage. It is recommended to use the associated 3D step le. Please contact your NVIDIA sales representative to get the mechanical simulation. Note All dimensions are in millimeters.
  • Page 107 S-HEAT, MCX755106A C-HEAT, MCX713106A C-CEAT, MCX713106A S-CEAT, MCX713106A C-VEAT, MCX713106A S-VEAT Single-port OSFP MCX75310AA S-NEAT, MCX75310AA C-NEAT, MCX75310AA S-HEAT, MCX75510AA S-NEAT, MCX75510AA S-HEAT Quad-port SFP56 MCX713104A C-ADAT, MCX713104A S-ADAT Quad-port SFP56 with PPS IN/OUT NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 108 OPNs Tall Bracket Short Bracket MCX713114T C-GEAT Auxiliary PCIe Connection Card NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 109 Adapter Card Heatsink The heatsink is attached to the ConnectX-7 IC to dissipate the heat from the ConnectX-7 IC. It is attached by spring-loaded screws. ConnectX-7 IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX-7 card in cases of high-temperature events, improper thermal coupling, or heatsink removal.
  • Page 110 In niBand/Ethernet cards have both a GUID and a MAC (derived from the GUID). IB only cards have GUID for the In niBand protocol. Note The product revisions indicated on the labels in the following gures do not necessarily represent the latest revisions of the cards. Board Label Example NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 111 Cabline SA-II Plus harnesses. Ordering Part Cabline SA-II Plus Harnesses Passive Auxiliary Connection Number Length MTMK9100- PCIe Gen 4.0/5.0 x16 connection card 2x 150mm harnesses MTMK9100- PCIe Gen 4.0 x16 connection card 2x 250mm harnesses NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 112 (top side) of the network card, while the White Cabline SA-II Plus harnesses mates with the pint side (bottom side) of the main network card. For hardware installation, please refer to ConnectX-7 Socket Direct (2x PCIe x16) Installation Instructions. PCIe Auxiliary Card Package Contents...
  • Page 113 GND BAR PCIE_REFCLK1_P Micro coax Primary PCIe clock from mother board to the Networ Adapter Main card, to be used for the Cabline harness PCIe interfac e. This clock must meet all NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 114 PCIE_REFCLK1_N Micro coax Primary PCIe clock from mother board to the Networ Adapter Main card, to be used for the Cabline harness PCIe interfac e. This clock must meet all PCIe NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 115 Black Cabline CA-II Plus cable through PCIE_CPU_CX_15N Micro coax 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_15P Micro coax Connec t from Black Cabline CA-II Plus NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 116 Black Cabline CA-II Plus cable through PCIE_CPU_CX_14N Micro coax 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_14P Micro coax Connec t from Black Cabline CA-II Plus cable NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 117 Black Cabline CA-II Plus cable through PCIE_CPU_CX_13N Micro coax 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_13P Micro coax Connec t from Black Cabline CA-II Plus cable through NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 118 Cabline CA-II Plus cable through PCIE_CPU_CX_12N Micro coax 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_12P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 119 CA-II Plus cable through PCIE_CPU_CX_11N Micro coax 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_11P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 120 PCIE_CPU_CX_10N Micro coax 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_10P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 121 PCIE_CPU_CX_9N Micro coax 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_9P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 122 Micro coax 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_8P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 123 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_7P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx lanes of NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 124 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_6P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx lanes of NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 125 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_5P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 126 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_4P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 127 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_3P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 128 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_2P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 129 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_1P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 130 PCIe Tx lanes of CPU/GP U/End- Point PCIE_CPU_CX_0P Micro coax Connec t from Black Cabline CA-II Plus cable through 220nF capacit ors to the x16 PCIe Tx lanes of CPU/GP NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 131 SMBus interfac e to the network adapter. Networ adapter silicon may be accesse through this SMBus interfac e after Power- sequen ce of network adapter complet ed. The mother board side serves as the NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 132 EEPRO M at address 0x50 is also present on this interfac e, to allow server manage ment entity to read identity of the auxiliary card. For a direct mother board interfac e to the NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 133 – for mother board manage ment purpose s of the Cabline CA-II Plus interfac e to the network adapter) I2C_SMCLK Micro coax PCIe complia SMBus interfac e to the network adapter. Networ NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 134 SMBus interfac e after Power- sequen ce of network adapter complet ed. The mother board side serves as the master on the SMBus interfac (Please note on board implem entatio n, an EEPRO NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 135 For a direct mother board interfac e to the Cabline CA-II Plus cables, mounti ng such EEPRO M on mother board should conside NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 136 Reserved_01 Micro coax S_WAKE1_CONN_L Micro coax setups that support operatio n of the network adapter card during standby operatin g mode, this optional PCIe complia nt Wake signal, may be used for network adapter NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 137 . Please consult NVIDA in case intentio n to use this signal, as it has been populat ed or used in designs so far. I2C_AUX_SCL Micro coax Connec tX-7 silicon serves NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 138 Cabline CA-II Plus interfac parame ters to main- card network adapter silicon, like Cabline CA-II Plus cables length (contact NVIDIA NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 139 I2C bus as well. I2C_AUX_SDA Micro coax Connec tX-7 silicon serves as the I2C bus master on this bus. An EEPRO M at NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 140 Cabline CA-II Plus interfac parame ters to main- card network adapter silicon, like Cabline CA-II Plus cables length (contact NVIDIA for the format of this EEPRO M). If addition optional NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 141 I2C bus as well. S_PRSNT1_L Micro coax Connec t this pin to wires connect ed to these pins to ensure they do interfer e with operatio n of S_PRSN T2_L for NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 142 Micro coax S_BIFFURCATE_AU Micro coax In speci c X_BRD Network Card Assemblies which support bifurcation of the PCIe x16 IPEX interface to two x8 PCIe interfaces, this pin is used for either the motherboard NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 143 Reserved 06 in the generic White IPEX Cable pinout) Reserved for Reserved_07 Micro coax future expansion Reserved for Reserved_08 Micro coax future expansion GND BAR PCIE_CX_CPU_0P Micro coax Connect from the White NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 144 ConnectX-7 card. GND BAR PCIE_CX_CPU_1P Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/En d-Point. 220nF caps NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 145 Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the PCIE_CX_CPU_2P Micro coax CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. PCIE_CX_CPU_2N Micro coax Connect from the NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 146 CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. PCIE_CX_CPU_3N Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/En d-Point. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 147 Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the PCIE_CX_CPU_4N Micro coax CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. GND BAR NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 148 Micro coax CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. GND BAR PCIE_CX_CPU_6P Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 149 ConnectX-7 card. GND BAR PCIE_X_CPU_7P Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/En d-Point. 220nF caps are required on this signal on the NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 150 PCIe x16 Rx lanes of the PCIE_CX_CPU_8P Micro coax CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. PCIE_CX_CPU_8N Micro coax Connect from the White Cabline CA-II Plus harness NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 151 ConnectX-7 card. PCIE_CX_CPU_9N Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/En d-Point. 220nF caps are required on this signal NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 152 Plus harness to the PCIe x16 Rx lanes of the PCIE_CX_CPU_10N Micro coax CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. GND BAR PCIE_CX_CPU_11P Micro coax Connect from the White NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 153 ConnectX-7 card. GND BAR PCIE_CX_CPU_12P Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/En d-Point. 220nF caps NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 154 Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the PCIE_CX_CPU_13P Micro coax CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. PCIE_CX_CPU_13N Micro coax Connect from the NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 155 CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. PCIE_CX_CPU_14N Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/En d-Point. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 156 Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the PCIE_CX_CPU_15N Micro coax CPU/GPU/En d-Point. 220nF caps are required on this signal on the ConnectX-7 card. GND BAR NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 157 (from the CPU), to the network adapter. When connecting a GPU or an end point to the PCIe Cabline CA-II Plus interface, this signal is driven from NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 158 This signal is connected to S_PRSNT1_L on the network adapter main card. In the motherboard side, read logic low if both Cabline harnesses connected. Read logic 1 (3.3V) if one or both the NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 159 PCI Express Gen 5.0/4.0: SERDES @ 16/32 GT/s, x16 lanes MTMK9100-T15 PCIe (Gen 3.0 compatible) Conne MTMK9100-T25 / PCI Express Gen 4.0: SERDES @ 16GT/s, x16 lanes (Gen ctivity MTMK9100-T35 3.0 compatible) Power Consu Voltage: 12V, 3.3V_AUX mptio Maximum current: 100mA NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 160 EMC: CE / FCC / VCCI / ICES / RCM / KC tory RoHS: RoHS Compliant PCIe Auxiliary Card Mechanical Drawings and Dimensions Note All dimensions are in millimeters. The PCB mechanical tolerance is +/- 0.13mm. Bracket Mechanical Drawings and Dimensions NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 161 Auxiliary PCIe Connection Card Short Auxiliary PCIe Connection Card Tall Bracket Bracket Cabline CA-II Plus Harnesses Mechanical Drawing NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 162 Updated VMware Driver Installation. Added support for OPN MCX715105AS-WEAT . Jan. 2023 Updated 400Gb/s Ethernet protocols in Speci cations. Dec. 2022 Modi ed ordering part numbers and product descriptions in NVIDIA ConnectX-7 Adapter Cards User Manual. NVIDIA ConnectX-7 Adapter Cards User Manual...
  • Page 163 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.<br/><br/>No license, either expressed or implied, is granted under any NVIDIA patent right, copyright,...

This manual is also suitable for:

900- 9x766 -003nsq01900-9x766-003n-sr0900-9x766-003n-st02900-9x7ao-00c3-stz900-9x7ao-0003-st0