Product Overview ..................9 Features and Benefits ................13 Operating Systems/Distributions ..............15 Connectivity ..................15 Interfaces ..................16 ConnectX-6 Dx IC Interface............... 16 Encryption ..................17 PCI Express Interface................17 Networking Ports LEDs Interface ..............17 Scheme 1: One Bi-Color LED ..............18 Scheme 2: One Bi-Color LED and one Single Color LED ........
Page 4
Finding the MAC on the Adapter Card .............68 Document Revision History ..............69 Arrow.com. Arrow.com. Arrow.com. Arrow.com. Downloaded from Downloaded from Downloaded from Downloaded from...
Page 5
Ordering Part Numbers The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet adapter cards. Configuration NVIDIA SKU...
Page 6
ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8, Crypto, No Secure Boot, Tall Bracket MCX623102AN-GDAT ConnectX-6 Dx EN adapter card, 50GbE , Dual-port SFP56, PCIe 4.0 x16, No Crypto, Tall Bracket MCX623102AE-GDAT ConnectX-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket...
Page 7
This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with Ethernet network and architecture specifications. Technical Support Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: •...
Page 8
Solutions bandwidth, low-latency connections between compute nodes and switch nodes. NVIDIA offers one of the industry’s broadest portfolio of 40GbE, 56GbE, 100GbE, 200GbE and 400GbE cables, including Direct Attach Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km.
This is the User Manual for Ethernet adapter cards based on the ConnectX®-6 Dx integrated circuit device. As the world's most advanced cloud SmartNIC, ConnectX-6 Dx provides up to two ports of 25, 50 or 100Gb/s or a single-port of 200Gb/s Ethernet connectivity, powered by 50Gb/s PAM4 SerDes technology and PCIe Gen 4.0 host connectivity. ConnectX-6 Dx continues among NVIDIA's innovation...
Page 10
6.01in. x 25/10/1 Dual-port PCIe Gen 4.0 Tall 2AC-ADAT 2.71in SFP28 SERDES @ Bracket (152.90mm x 16.0GT/s x8 68.90mm) ConnectX-6 Dx 50GbE Adapter Cards Form Factor/ Data No. of PCIe Secure Crypt Bracket Dimensions Transmissi Ports and Interface Boot Type...
Page 11
PCIe Gen 4.0 Tall AS-CDAT 2.71in 10/1 GbE QSFP56 SERDES @ Bracket (142.00mm x 16.0GT/s x16 68.90mm) ConnectX-6 Dx 100GbE Adapter Cards for Timing and Telecommunication Application SMAs Form Sycn Data No. of PCIe Brack Factor/ SS Transmi Ports Interface Dimensio &...
Page 12
5.59in. x 100/50/ Dual- PCIe Gen 106GC- 2.71in 25/10/1 port Bracke CDAT (142.00m QSFP56 SERDES @ 16.0GT/s 68.90mm) ConnectX-6 Dx 200GbE Adapter Cards Form Factor/ Data No. of PCIe Secure Crypt Bracket Dimensions Transmissi Ports and Interface Boot Type on Rate Type ✔...
NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-6 Dx effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.
Page 14
ITU-T G.8273.2 class C. PTP Enabled in MCX623106TN-CDAT, MCX623106TC-CDAT, MCX623106GN-CDAT, and MCX623106GC- CDAT. Grand NVIDIA offers ConnectX-6 Dx with an integrated GNSS receiver to allow a compact and efficient Master solution for a Grand Master at every server. (GNSS Enabled in MCX623106G[N/C]-CDAT Reciver) Arrow.com.
With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS- out signal from the onboard SMA connecter, ConnectX-6 Dx also allows measuring PTP in scale, with PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal, and the timestamp will be sent to the SW.
Interfaces The below figures show the component side of the ConnectX-6 Dx adapter card. Each numbered interface that is referenced in the figures is described in the following table with a link to detailed information. The below figures are for illustration purposes only and might not reflect the current revision of the adapter card.
Embedded hardware root-of-trust and support for RSA-based secure firmware update and secure boot, providing guaranteed integrity of the network adapter. Ethernet SFP28 / SFP56 / QSFP56 Interfaces The network ports of the ConnectX-6 Dx adapter card are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits.
Scheme 1: One Bi-Color LED There is one bicolor (Yellow and Green) I/O LED per port to indicate speed and link status. Link Indications State Bi-Color LED (Yellow/Green) Physical link speed Beacon command for locating the adapter card 1Hz blinking Yellow Error 4Hz blinking Yellow Indicates an error with the link.
BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.
Hardware Installation Installation and initialization of ConnectX-6 Dx adapter cards require attention to the mechanical attributes, power specification, and precautions for electronic equipment. Safety Warnings Safety warnings are provided here in the English language. For safety warnings in other...
Airflow Requirements ConnectX-6 Dx adapter cards are offered with two airflow patterns: from the heatsink to the network ports. Please refer to the Specifications section for airflow numbers for each specific card model.
Replacement Instructions. Bracket Replacement Instructions The ConnectX-6 Dx card is usually shipped with an assembled high-profile bracket. If this form factor is suitable for your requirements, you can skip the remainder of this section and move to Installation Instructions. If you need to replace the high-profile bracket with the short bracket that is included in the shipping box, please follow the instructions in this section.
Save the two screws. Installing the New Bracket Place the bracket onto the card until the screw holes line up. Do not force the bracket onto the adapter card. Screw on the bracket using the screws saved from the bracket removal procedure above. ...
Step 2: Applying even pressure at both corners of the card, insert the adapter card in a PCI Express slot until firmly seated. Do not use excessive force when seating the card, as this may damage the chassis. Secure the adapter card to the chassis. Step 1: Secure the bracket to the chassis with the bracket screw.
[ConnectX-6 On Windows Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. Expand System Devices and locate your ConnectX-6 Dx adapter card. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window. Click the Details tab and select Hardware Ids (Windows 2012/R2/2016) from the Property pull-down menu.
In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA; and DEV is equal to 1018 (for ConnectX-6 Dx) – this is a valid NVIDIA PCI Device ID.
Page 27
To remove the card, disengage the retention mechanisms on the bracket (clips or screws). Holding the adapter card from its center, gently pull the ConnectX-6 and Auxiliary Connections cards out of the PCI Express slot. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
ConnectX-6 Dx adapter card installed. Prerequisites Requirements Description Platforms A server platform with a ConnectX-6 Dx Ethernet adapter card installed. Required Disk Space for Installation Operating System Linux operating system. For the list of supported operating system distributions and kernels, please refer to the MLNX_OFED Release Notes.
SHA256 MLNX_OFED_LINUX-<ver>-<OS label>.iso Installing MLNX_OFED Installation Script The installation script, mlnxofedinstall, performs the following: • Discovers the currently installed kernel • Uninstalls any software stacks that are part of the standard operating system distribution or another vendor's commercial stack • Installs the MLNX_OFED_LINUX binary RPMs (if they are available for the current kernel) •...
Page 30
On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_kernel_support.sh script. The regular installation can be performed and weak-updates mechanism will create symbolic links to the MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add-kernel- ), the packages installation will not involve automatic regeneration of support...
Page 31
For the list of installation options, run: ./mlnxofedinstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards. Log in to the installation machine as root. Mount the ISO image on your machine. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Run the installation script.
Page 32
FW 12.14.0114 N/A Status: No matching image found Error message #2: The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor. Case A: If the installation script has performed a firmware update on your network adapter, you need to either restart the driver or reboot your system before the firmware update can take effect.
Page 33
"The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
Logs dir: /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0.IBMM2150110033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically. To prevent the automatic load of the NVIDIA drivers upon system boot: Add the following lines to the "/etc/modprobe.d/mlnx.conf" file. blacklist mlx5_core blacklist mlx5_ib Set “ONBOOT=no”...
Page 35
"The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
Mount the ISO image on your machine and copy its content to a shared location in your network. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link: http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox...
Page 37
rpmforge RHEL 6Server - RPMforge.net - dag 4,597 repolist: 8,351 Setting up MLNX_OFED YUM Repository Using --add-kernel-support Log into the installation machine as root. Mount the ISO image on your machine and copy its content to a shared location in your network.
Page 38
(User Space packages only where: mlnx-ofed-all Installs all available packages in MLNX_OFED mlnx-ofed-basic Installs basic packages required for running NVIDIA cards mlnx-ofed-guest Installs packages required by guest OS mlnx-ofed-hpc Installs packages required for HPC mlnx-ofed-hypervisor Installs packages required by hypervisor OS...
Page 39
→ Software→ InfiniBand Drivers. Create an apt-get repository configuration file called "/etc/apt/sources.list.d/ mlnx_ofed.list" with the following content: deb file:/<path to extracted MLNX_OFED package>/DEBS ./ Download and install NVIDIA's Technologies GPG-KEY. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Arrow.com. Arrow.com.
Page 40
Verify that the key was successfully imported. # apt-key list 1024D/A9E4B643 2013-08-11 Mellanox Technologies <support@mellanox.com> 1024g/09FCC269 2013-08-11 Update the apt-get cache. # sudo apt-get update Setting up MLNX_OFED apt-get Repository Using --add-kernel-support Log into the installation machine as root. Mount the ISO image on your machine and copy its content to a shared location in your network.
Enter the following command. echo %PROCESSOR_ARCHITECTURE% On an x64 (64-bit) machine, the output will be “AMD64”. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > InfiniBand Drivers (Learn More) > Nvidia WinOF-2. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
Download the .exe image according to the architecture of your machine (see Step 1). The name of the .exe is in the following format: MLNX_WinOF2-<version>_<arch>.exe. Installing the incorrect .exe file is prohibited. If you do so, an error message will be displayed.
Page 44
"ERROR!!! Installation failed due to following errors: MlxRshim drivers installation disabled and MlxRshim drivers Installed, Please remove the following oem inf files from driver store: <oem inf list>" [Optional] If you want to skip the check for unsupported devices, run. MLNX_WinOF2_<revision_version>_All_Arch.exe /v"...
Page 45
• If the user has a standard NVIDIA® card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA® card, only the NVIDIA® card will be updated.
Page 46
Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes Arrow.com.
Page 47
• Management tools - installation tools used for management, such as mlxstat • Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd Click Next to install the desired tools. Click Install to start the installation. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
Page 48
In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see ). Click Finish to complete the installation. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
Page 49
Unattended Installation If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD.
Firmware Upgrade If the machine has a standard NVIDIA® card with an older firmware version, the firmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to MFT User Manual. ...
After the installation process, all kernel modules are loaded automatically upon boot. Removing Earlier NVIDIA Drivers Please unload the previously installed drivers before removing them. To remove all the drivers: Log into the ESXi server with root permissions.
To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility. The utility can query for available NVIDIA adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images.
Troubleshooting General Troubleshooting Server unable to find the adapter • Ensure that the adapter is placed correctly • Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...
-d <mst_device> q ibstat Ports Information ibv_devinfo Firmware Version Upgrade To download the latest firmware version, refer to the NVIDIA Update and Query Utility. Collect Log File cat /var/log/messages dmesg >> system.log journalctl (Applicable on new operating systems) cat /var/log/syslog Windows Troubleshooting...
(efficiency factor) b. Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow is measured in wind tunnel. e. Contact NVIDIA for airflow numbers with other active modules' power levels. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
(efficiency factor) b. Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow is measured in wind tunnel. e. Contact NVIDIA for airflow numbers with other active modules' power levels. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
Safety CB / cTUVus / CE CE / FCC / VCCI / ICES / RCM Regulatory RoHS RoHS compliant a. Power numbers are provided for passive cables only. For board power numbers while using active cables, please add the outcome of the following formula to the passive cables power numbers stated above: Active_Module_Power x Number_of_Modules x 1.1 (efficiency factor) b.
Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow is measured in wind tunnel. e. Contact NVIDIA for airflow numbers with other active modules' power levels. MCX623105AN-CDAT / MCX623105AE-CDAT / MCX623105AC-CDAT Specifications Size: 5.59in. x 2.71in (142.00mm x 68.90mm)
Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow is measured in wind tunnel. e. Contact NVIDIA for airflow numbers with other active modules' power levels. MCX623106AS-CDAT / MCX623106AN-CDAT / MCX623106AC-CDAT / MCX623106AE-CDAT Specifications Size: 5.59in. x 2.71in (142.00mm x 68.90mm) Physical Connector: Dual QSFP56 Ethernet (copper and optical)
Airflow is measured in wind tunnel. e. Contact NVIDIA for airflow numbers with other active modules' power levels. MCX623106PN-CDAT / MCX623106PC-CDAT Specifications Size: 5.59in. x 2.71in (142.00mm x 68.90mm) Physical Connector: Dual QSFP56 Ethernet (copper and optical) Data Rate: Ethernet 1/10/25/40/50/100 Gb/s...
(efficiency factor) b. Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow is measured in wind tunnel. e. Contact NVIDIA for airflow numbers with other active modules' power levels. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow is measured in wind tunnel. e. Contact NVIDIA for airflow numbers with other active modules' power levels. MCX623105AC-VDAT / MCX623105AN-VDAT / MCX623105AS-VDAT / MCX623105AE-VDAT Specifications Size: 5.59in. x 2.71in (142.00mm x 68.90mm)
Typical power for ATIS traffic load. c. For both operational and non-operational states. d. Airflow is measured in wind tunnel. e. Contact NVIDIA for airflow numbers with other active modules' power levels. Board Mechanical Drawing and Dimensions 3D model of the cards are available through the customer portal following login.
IC thermal requirements. Adapter Card Heatsink The heatsink is attached to the ConnectX-6 Dx IC in order to dissipate the heat. It is attached either by using four spring-loaded push pins that insert into four mounting holes, or by screws. ConnectX-6 Dx IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX-6 Dx card in cases of high-temperature event, improper thermal coupling or heatsink removal.
Page 68
Finding the MAC on the Adapter Card Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards.
Page 69
Document Revision History Date Revision Description of Changes Aug. 2022 Updated the "Legacy (EOL) Ordering Part Numbers" table. Jun. 2022 Updated board and bracket mechanical drawings and mechanical tolerances. Mar. 2022 Added the following OPNs to relevant sections: • MCX621202AS-ADAT •...
Page 70
Date Revision Description of Changes Feb. 2020 Added the following OPNs to all relevant sections: • MCX623106AS-CDAT • MCX623105AS-VDAT • MCX623102AS-GDAT Nov. 2019 First release Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com. Arrow.com.
Page 71
NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
Need help?
Do you have a question about the ConnectX-6 Dx and is the answer not in the manual?
Questions and answers