Nvidia 900-9X662-00 53-ST1 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Adapter
  5. 900-9X662-00 53-ST1
  6. User manual
Nvidia 900-9X662-00 53-ST1 User Manual

Nvidia 900-9X662-00 53-ST1 User Manual

Nvidia connectx-6 lx pcie hhhl ethernet adapter cards
Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
 
 
 
 
 
 
 
NVIDIA ConnectX-6 Lx PCIe HHHL Ethernet
Adapter Cards User Manual
 
 
Exported on Aug/16/2023 07:20 PM

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the 900-9X662-00 53-ST1 and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia 900-9X662-00 53-ST1

Summary of Contents for Nvidia 900-9X662-00 53-ST1

  • Page 1               NVIDIA ConnectX-6 Lx PCIe HHHL Ethernet Adapter Cards User Manual     Exported on Aug/16/2023 07:20 PM...

  • Page 2: Table Of Contents

    Table of Contents Introduction..................6 Product Overview ..................6 Features and Benefits ................7 Operating Systems/Distributions ..............8 Connectivity ..................8 Manageability..................9 Interfaces ..................10 ConnectX-6 Lx IC Interface ............... 11 Encryption ..................11 PCI Express Interface................11 Networking Ports LEDs Interface ..............11 Voltage Regulators ................. 12 Hardware Installation ................13 Safety Warnings ..................
  • Page 3 Firmware Upgrade ................44 VMware Driver Installation ............... 44 Hardware and Software Requirements ............. 44 Installing NATIVE ESXi Driver for VMware vSphere ........44 Removing Earlier NVIDIA Drivers ............. 45 Firmware Programming ............... 45 Updating Adapter Firmware ..............46 Troubleshooting ................47 General Troubleshooting ................
  • Page 4 About This Manual This User Manual describes NVIDIA® ConnectX®-6 Lx Ethernet adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Ordering Part Numbers The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Lx Ethernet adapter cards.
  • Page 5 User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools WinOF-2 for content and configuration. See WinOF-2 for Windows Documentation. Windows User Manual and Release Notes User Manual describing the various components of the NVIDIA ConnectX® NATIVE NVIDIA VMware for ESXi stack. See VMware® ESXi Documentation. Ethernet User Manual and Release Notes NVIDIA firmware update and query utility used to update the firmware.

  • Page 6: Introduction

    Providing up to two ports of 25GbE or a single-port of 50GbE connectivity, and PCIe Gen 3.0/4.0 x8 host connectivity, ConnectX-6 Lx is a member of NVIDIA’s world-class, award-winning, ConnectX family of network adapters. Continuing NVIDIA’s consistent innovation in networking, ConnectX-6 Lx provides agility and efficiency at every scale.

  • Page 7: Features And Benefits

    NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. NVIDIA ConnectX-6 Lx effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.

  • Page 8: Operating Systems/Distributions

    Support for port-based Quality of Service enabling various application requirements for latency Service and SLA. (QoS) Hardware- NVIDIA ConnectX-6 Lx provides dedicated adapter resources and guaranteed isolation and based I/ protection for virtual machines within the server. O Virtualiz ation Storage...

  • Page 9: Manageability

    ConnectX-6 Lx PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.

  • Page 10: Interfaces

    Interfaces The below figure show the component side of the NVIDIA ConnectX-6 Lx adapter card. Each numbered interface that is referenced in the figures is described in the following table with a link to detailed information.  The below figures are for illustration purposes only and might not reflect the current revision of the adapter card.

  • Page 11: Connectx-6 Lx Ic Interface

    AES data-at-rest cryptographic operations. • Stateful firewall solution acceleration, powered by Open vSwitch connection tracking and NVIDIA’s ASAP2 technology. • Embedded hardware root-of-trust and support for RSA-based secure firmware update and secure boot, providing guaranteed integrity of the network adapter.

  • Page 12: Voltage Regulators

    BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.

  • Page 13: Hardware Installation

    Identify ConnectX-6 Lx in the system Refer to Identifying Your Card System Requirements Hardware Requirements  Unless otherwise specified, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination.

  • Page 14: Software Requirements

    Software Requirements • Operating Systems/Distributions section under the Introduction section. • Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal. Before opening the case of the system, observe the following precautions to avoid injury and prevent damage to system components.

  • Page 15: Pre-Installation Checklist

    Category Qty. Item Adapter card tall bracket (shipped assembled on the card)  Please note that if the card is removed hastily from the antistatic bag, the plastic ziplock may harm the EMI fingers on the networking connector. Carefully remove the card from the antistatic bag to avoid damaging the EMI fingers.

  • Page 16: Installation Instructions

     Do not force the bracket onto the adapter card. Screw on the bracket using the screws saved from the bracket removal procedure above.  Use a torque driver to apply up to 2 lbs-in torque on the screws. Installation Instructions This section provides detailed instructions on how to install your adapter card in a system.

  • Page 17: Cables And Modules

    Holding the adapter card from its center, gently pull the ConnectX-6 Lx out of the PCI Express slot.  To uninstall the adapter card, see Uninstalling the Card. Cables and Modules Cable Installation All cables can be inserted or removed with the unit powered on. To insert a cable, press the connector into the port receptacle until the connector is firmly seated.
  • Page 18 “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA Technologies; and DEV is equal to 1018 (for ConnectX-6 Lx) – this is a valid NVIDIA Technologies PCI Device ID.

  • Page 19: Uninstalling The Card

    Uninstalling the Card Safety Precautions The adapter is installed in a system that operates with voltages that can be lethal. Before uninstalling the adapter card, please observe the following precautions to avoid injury and prevent damage to system components. Remove any metallic objects from your hands and wrists. It is strongly recommended to use an ESD strap or other antistatic devices.

  • Page 20: Driver Installation

    NVIDIA ConnectX-6 Lx adapter card installed. Prerequisites Requirements Description Platforms A server platform with an NVIDIA ConnectX-6 Lx EN adapter card installed  (firmware: fw-ConnectX6Lx) Required Disk Space for Installation Operating System Linux operating system. For the list of supported operating system distributions and kernels, please refer to the MLNX_OFED Release Notes.

  • Page 21: Installing Mlnx_Ofed

    Installing MLNX_OFED Installation Script The installation script, mlnxofedinstall, performs the following: • Discovers the currently installed kernel • Uninstalls any software stacks that are part of the standard operating system distribution or another vendor's commercial stack • Installs the MLNX_OFED_LINUX binary RPMs (if they are available for the current kernel) •...
  • Page 22  On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_kernel_support.sh script. The regular installation can be performed and weak-updates mechanism will create symbolic links to the MLNX_OFED kernel modules.  If you regenerate kernel modules for a custom kernel (using --add-kernel- ), the packages installation will not involve automatic regeneration of the support...
  • Page 23 For the list of installation options, run: ./mlnxofedinstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards.  Log in to the installation machine as root. Mount the ISO image on your machine.  host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Run the installation script.
  • Page 24 FW XX.XX.XXXX Status: No matching image found Error message #2: The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor. 4. Case A: If the installation script has performed a firmware update on your network adapter, you need to either restart the driver or reboot your system before the firmware update can take effect.
  • Page 25 Action \ Adapter Driver Restart Standard Reboot (Soft Cold Reboot (Hard Reset) Reset) Standard ConnectX-4/ ConnectX-4 Lx or higher Adapters with Multi-Host Support Socket Direct Cards Case B: If the installations script has not performed a firmware upgrade on your network adapter, restart the driver by running: “/etc/init.d/openibd restart”.

  • Page 26: Driver Load Upon System Boot

    Logs dir: /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0.IBMM2150110033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically.  To prevent the automatic load of the NVIDIA drivers upon system boot: Add the following lines to the "/etc/modprobe.d/mlnx.conf" file.  blacklist mlx5_core blacklist mlx5_ib Set “ONBOOT=no”...
  • Page 27 In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...

  • Page 28: Additional Installation Procedures

    Mount the ISO image on your machine and copy its content to a shared location in your network. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link:  http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox...
  • Page 29 Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mellanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox this ok [y/N]: Check that the key was successfully imported.  # rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n' | grep Mellanox gpg-pubkey-a9e4b643-520791ba gpg(Mellanox Technologies <support@mellanox.com>) Create a yum repository configuration file called "/etc/yum.repos.d/mlnx_ofed.repo" with the following content: ...
  • Page 30 Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log   Checking all needed packages are installed... Building MLNX_OFED_LINUX RPMS . Please wait... Creating metadata-rpms 3.10.0-957.21.3.el7.x86_64 ... WARNING: If you are going to configure this package as a repository, then please note WARNING: that it contains unsigned rpms, therefore, you need to disable the gpgcheck WARNING: by setting 'gpgcheck=0'...
  • Page 31 (User Space packages only where:  mlnx-ofed-all Installs all available packages in MLNX_OFED mlnx-ofed-basic Installs basic packages required for running NVIDIA cards mlnx-ofed-guest Installs packages required by guest OS mlnx-ofed-hpc Installs packages required for HPC mlnx-ofed-hypervisor Installs packages required by hypervisor OS...
  • Page 32 mlnx-ofed-all-3.17.4-301.fc21.x86_64.noarch : MLNX_OFED all installer package for kernel 3.17.4-301.fc21.x8 6_64 (without KMP support) mlnx-ofed-basic-3.17.4-301.fc21.x86_64.noarch : MLNX_OFED basic installer package for kernel 3.17.4-301.fc2 1.x86_64 (without KMP support) mlnx-ofed-guest-3.17.4-301.fc21.x86_64.noarch : MLNX_OFED guest installer package for kernel 3.17.4-301.fc2 1.x86_64 (without KMP support) mlnx-ofed-hpc-3.17.4-301.fc21.x86_64.noarch : MLNX_OFED hpc installer package for kernel 3.17.4-301.fc21.x8 6_64 (without KMP support)
  • Page 33 Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content:  deb file:/<path to extracted MLNX_OFED package>/DEBS ./ Download and install NVIDIA's Technologies GPG-KEY.  # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Verify that the key was successfully imported. ...
  • Page 34 # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Build the packages with kernel support and create the tarball.  # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7.6 under /tmp directory. Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log  ...
  • Page 35 mlnx-ofed-kernel-utils - Userspace tools to restart and tune mlnx-ofed kernel modules mlnx-ofed-vma-vpi - MLNX_OFED vma-vpi installer package (with DKMS support) mlnx-ofed-kernel-only - MLNX_OFED kernel-only installer package (with DKMS support) mlnx-ofed-bluefield - MLNX_OFED bluefield installer package (with DKMS support) mlnx-ofed-hpc-user-only - MLNX_OFED hpc-user-only installer package (User Space packages only) mlnx-ofed-dpdk-user-only - MLNX_OFED dpdk-user-only installer...

  • Page 36: Performance Tuning

    Depending on the application of the user's system, it may be necessary to modify the default configuration of network adapters based on the ConnectX® adapters. In case that tuning is required, please refer to the Performance Tuning Guide for NVIDIA Network Adapters. Windows Driver Installation...

  • Page 37: Installing Winof-2 Driver

    On an x64 (64-bit) machine, the output will be “AMD64”. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > InfiniBand Drivers (Learn More) > Nvidia WinOF-2. Download the .exe image according to the architecture of your machine (see Step 1). ...
  • Page 38 MLNX_WinOF2_<revision_version>_All_Arch.exe /v" MT_DISABLE_RSHIM_INSTALL=1"  The Rshim driver installanion will fail if a prior Rshim driver is already installed. The following fail message will be displayed in the log: "ERROR!!! Installation failed due to following errors: MlxRshim drivers installation disabled and MlxRshim drivers Installed, Please remove the following oem inf files from driver store: <oem inf list>"...
  • Page 39 • If the user has a standard NVIDIA® card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA® card, only the NVIDIA® card will be updated.
  • Page 40 Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes...
  • Page 41 • Management tools - installation tools used for management, such as mlxstat • Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd Click Next to install the desired tools. Click Install to start the installation.
  • Page 42 In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see  ).  Click Finish to complete the installation.
  • Page 43 Unattended Installation  If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD.

  • Page 44: Firmware Upgrade

    Firmware Upgrade If the machine has a standard NVIDIA® card with an older firmware version, the firmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to MFT User Manual. ...

  • Page 45: Removing Earlier Nvidia Drivers

    PartnerSupported 2017-01-31  After the installation process, all kernel modules are loaded automatically upon boot. Removing Earlier NVIDIA Drivers  Please unload the previously installed drivers before removing them. To remove all the drivers: Log into the ESXi server with root permissions.

  • Page 46: Updating Adapter Firmware

    To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility. The utility can query for available NVIDIA adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images.

  • Page 47: Troubleshooting

    Troubleshooting General Troubleshooting • Ensure that the adapter is placed correctly Server unable to find the adapter • Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...

  • Page 48: Linux Troubleshooting

    -d <mst_device> q ibstat Ports Information ibv_devinfo To download the latest firmware version, refer to Firmware Version Upgrade the NVIDIA Update and Query Utility. cat /var/log/messages Collect Log File dmesg >> system.log journalctl (Applicable on new operating systems) cat /var/log/syslog Windows Troubleshooting...

  • Page 49: Specifications

    Specifications MCX631102AC-ADAT / MCX631102AE-ADAT / MCX631102AN-ADAT / MCX631102AS-ADAT Size: 3.79in. x 2.71in (96.30mm x 68.90mm) Physical Connector: Dual SFP28 Ethernet (copper and optical) Data Rate Ethernet 1/10/25 Gb/s Protocol Support Ethernet: 25GBASE-R, 20GBASE-KR2, 10GBASE-LR,10GBASE-ER, 10GBASE-CX4, 10GBASE- CR, 10GBASE-KR, SGMII, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR PCI Express Gen 3.0/4.0: SERDES @ 16.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V Power...

  • Page 50: Mcx631105Ac-Gdat / Mcx631105Ae-Gdat / Mcx631105An-Gdat

    Voltage: 3.3Aux Maximum current: 100mA Maximum power available through QSFP28 port: 2.5W (each port) Hot Aisle - Heatsink to Port Passive Cable 250LFM Airflow Requirements @ 55C Active 1.8W NVIDIA 50G 350LFM Cable Temperature Operational 0°C to 55°C Environmen Non-operational -40°C to 70°C...

  • Page 51: Bracket Mechanical Drawing

     All dimensions are in millimeters. The mechanical tolerances are as follows: • Width: +/- 0.13mm • Height:  +0/-0.13mm Dual-Port SFP28 x8 Adapter Cards Mechanical Single-Port QSFP28 x8 Adapter Cards Mechanical Drawing  Drawing  Bracket Mechanical Drawing  All dimensions are in millimeters. The mechanical tolerances is +/- 0.2mm. Single-Port QSF28 Adapter Card Short Bracket Tall Bracket...

  • Page 52: Dual-Port Sfp28 Adapter Card

    Dual-Port SFP28 Adapter Card Short Bracket Tall Bracket...

  • Page 53: Monitoring

    Monitoring Thermal Sensors The adapter card incorporates the ConnectX IC, which operates in the range of temperatures between 0°C and 105°C. Three thermal threshold definitions impact the overall system operation state: • Warning – 105°C: On managed systems only: When the device crosses the 105°C threshold, a Warning Threshold message is issued by the management SW, indicating to system administration that the card has crossed the warning threshold.

  • Page 54: Finding The Mac On The Adapter Card

    Finding the MAC on the Adapter Card Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol.  The product revisions indicated on the labels in the following figures do not necessarily represent the latest revisions of the cards.

  • Page 55: Document Revision History

    Document Revision History Date Description of Changes May. 2023 • Updated Specifications to include non-operational storage temperature specifications  • Updated airflow specifications Nov. 2022 Updated Specifications Aug. 2022 • Added MCX631102AS-ADAT support across the document. • Updated the memory component. Jun, 2022 Updated the brackets' mechanical tolerance.
  • Page 56 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
  • Page 57 Copyright © 2023 NVIDIA Corporation & affiliates. All Rights Reserved.

This manual is also suitable for:

900-9x662-00 83-st0900-9x662-00 73-st0900-9x662-006 3-st0900-9x601-002 5-st0900-9x601-001 5-sq0900-9x601-004 5-st0

Table of Contents