Nvidia 900-9X4B0-0012-0T1 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Adapter
  5. 900-9X4B0-0012-0T1
  6. User manual
Nvidia 900-9X4B0-0012-0T1 User Manual

Nvidia 900-9X4B0-0012-0T1 User Manual

Connectx-4 lx ethernet adapter cards
Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
 
 
 
 
 
 
 
NVIDIA ConnectX-4 Lx Ethernet Adapter
Cards User Manual
 
 
Exported on Dec/05/2023 11:54 AM

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the 900-9X4B0-0012-0T1 and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia 900-9X4B0-0012-0T1

Summary of Contents for Nvidia 900-9X4B0-0012-0T1

  • Page 1               NVIDIA ConnectX-4 Lx Ethernet Adapter Cards User Manual     Exported on Dec/05/2023 11:54 AM...

  • Page 2: Table Of Contents

    Identifying the Card in Your System............. 18 Uninstalling the Card ................19 Driver Installation................21 Linux Driver Installation ................21 Prerequisites..................21 Downloading NVIDIA OFED ..............21 Installing MLNX_OFED ................. 22 Driver Load Upon System Boot............... 27 Additional Installation Procedures ............28...
  • Page 3 Firmware Upgrade ................43 VMware Driver Installation ............... 43 Hardware and Software Requirements ............. 43 Installing NATIVE ESXi Driver for VMware vSphere ........43 Removing Earlier NVIDIA Drivers ............. 44 Firmware Programming ............... 44 Updating Adapter Firmware ..............45 Troubleshooting ................46 General Troubleshooting ................
  • Page 4 MCX4131A-BCAT Board Label (Example)............58 MCX4131A-GCAT Board Label (Example) ............59 Document Revision History ..............60  ...
  • Page 5 About This Manual This User Manual describes NVIDIA® ConnectX®-4 Lx Ethernet adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. EOL'd (End of Life) Ordering Part Numbers The table below provides the ordering part numbers (OPN) for ConnectX-4 Lx Ethernet adapter cards.
  • Page 6 • URL: https://www.nvidia.com > Support • E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller. Related Documentation...

  • Page 7: Introduction

    Introduction Product Overview This is the User Guide for Ethernet adapter cards based on the ConnectX®-4 Lx integrated circuit device. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0 servers used in Enterprise Data Centers and High- Performance Computing environments.

  • Page 8: Features And Benefits

    Uses PCIe Gen 3.0 (8GT/s) through an x8 edge connector. Gen 1.1 and 2.0 compatible. Up to 50 Gigabit Ethernet NVIDIA adapters comply with the following IEEE 802.3 standards: • 50GbE / 40GbE / 25GbE / 10GbE / 1GbE • IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes •...
  • Page 9 UEFI UEFI is a standard firmware interface designed to replace BIOS. NVIDIA UEFI Network driver allows boot over the network via PXE (Preboot eXecution Environment). This network driver allows remote boot over Ethernet, or Boot over iSCSI (Bo-iSCSI) in UEFI mode, and also supports the Secure Boot standard.

  • Page 10: Operating Systems/Distributions

    Feature Description Secure Firmware A “Secure firmware update” is the ability of a device to verify digital signatures of new firmware binaries, in order to assure that only officially approved versions can be installed from the host, the network or a Board Management Controller (BMC). The firmware of devices with “secure firmware update”...

  • Page 11: Interfaces

    Interfaces Ethernet SFP28 and QSFP28 Interfaces The network ports of the ConnectX®-4 Lx adapter card are compliant with the IEEE 802.3 Ethernet standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28 connectors on the adapter card.  The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables.

  • Page 12: Smbus Interface

    BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.

  • Page 13: Hardware Installation

    Identify the ConnectX-4 Lx adapter card in the Refer to Identifying the Card in Your system. System System Requirements Hardware Requirements  Unless otherwise specified, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination.

  • Page 14: Airflow Requirements

    All cards in the system should be planned with the same airflow direction. Software Requirements • Operating Systems/Distributions section under the Introduction section. • Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. Unpacking the Product ...

  • Page 15: Bracket Replacement Instructions

    Shut down your system if active: Turn off the power to the system, and disconnect the power cord. Refer to the system documentation for instructions. Before you install the ConnectX-4 Lx card, make sure that the system is disconnected from power. Bracket Replacement Instructions The card is usually shipped with an assembled high-profile bracket.

  • Page 16: Installation Instructions

    Place the bracket onto the card until the screw holes line up.   Do not force the bracket onto the adapter card. Screw on the bracket using the screws saved from the bracket removal procedure above.  Use a torque driver to apply up to 2 lbs-in torque on the screws. Installation Instructions This section provides detailed instructions on how to install your adapter card in a system.

  • Page 17: Cables And Modules

    Applying even pressure at both corners of the card, insert the adapter card into the PCI Express slot until firmly seated. When the adapter is properly seated, the port connectors are aligned with the slot opening, and the adapter faceplate is visible against the system chassis. Secure the adapter with the adapter clip or screw.

  • Page 18: Identifying The Card In Your System

    [ConnectX-4 On Windows Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. Expand System Devices and locate your NVIDIA ConnectX-4 Lx adapter card. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window. Click the Details tab and select Hardware Ids (Windows 2012/R2/2016) from...

  • Page 19: Uninstalling The Card

    0x15B3 – this is the Vendor ID; and DEV is equal to 1018 (for ConnectX-4 Lx) – this is a valid PCI Device ID.  If the PCI device does not have a NVIDIA adapter ID, return to Step 2 to check another device. ...
  • Page 20  Please note that the following images are for illustration purposes only. Verify that the system is powered off and unplugged. Wait 30 seconds. To remove the card, disengage the retention mechanisms on the bracket (clips or screws). Holding the adapter card from its center, gently pull the adapter card out of the PCI Express slot.

  • Page 21: Driver Installation

            The image’s name has the format .  MLNX_OFED_LINUX-<ver>-<OS label><CPU arch>.iso         You can download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the NVIDIA web site at nvidia.com/en-us/networking → Products → Software → InfiniBand Drivers → NVIDIA MLNX_OFED Scroll down to the Download wizard, and click the Download tab.

  • Page 22: Installing Mlnx_Ofed

        3. Use the Hash utility to confirm the file integrity of your ISO image. Run the following command and compare the result to the value provided on the download page. SHA256 MLNX_OFED_LINUX-<ver>-<OS label>.iso Installing MLNX_OFED Installation Script The installation script, mlnxofedinstall, performs the following: •...
  • Page 23 • If your kernel version does not match with any of the offered pre-built RPMs, you can add your kernel version by using the “mlnx_add_kernel_support.sh” script located inside the MLNX_OFED package.  On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_kernel_support.sh script.
  • Page 24 For the list of installation options, run: ./mlnxofedinstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards.  Log in to the installation machine as root. Mount the ISO image on your machine.  host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Run the installation script.
  • Page 25 FW XX.XX.XXXX Status: No matching image found Error message #2: The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor. 4. Case A: If the installation script has performed a firmware update on your network adapter, you need to either restart the driver or reboot your system before the firmware update can take effect.
  • Page 26 In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...

  • Page 27: Driver Load Upon System Boot

    Logs dir: /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0.IBMM2150110033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically.  To prevent the automatic load of the NVIDIA drivers upon system boot: Add the following lines to the "/etc/modprobe.d/mlnx.conf" file.  blacklist mlx5_core blacklist mlx5_ib Set “ONBOOT=no”...

  • Page 28: Additional Installation Procedures

    In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. "The firmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain firmware for this device, please contact your HW vendor."...
  • Page 29 Mount the ISO image on your machine and copy its content to a shared location in your network. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link:  http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox...
  • Page 30 # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Build the packages with kernel support and create the tarball.  # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7.6 under /tmp directory. Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log  ...
  • Page 31 (User Space packages only where:  mlnx-ofed-all Installs all available packages in MLNX_OFED mlnx-ofed-basic Installs basic packages required for running NVIDIA cards mlnx-ofed-guest Installs packages required by guest OS mlnx-ofed-hpc Installs packages required for HPC mlnx-ofed-hypervisor Installs packages required by hypervisor OS...
  • Page 32 Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content:  deb file:/<path to extracted MLNX_OFED package>/DEBS ./ Download and install NVIDIA's Technologies GPG-KEY.  # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Verify that the key was successfully imported. ...
  • Page 33 # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Build the packages with kernel support and create the tarball.  # /mnt/mlnx_add_kernel_support.sh --make-tgz <optional --kmp> -k $(uname -r) -m /mnt/ Note: This program will create MLNX_OFED_LINUX TGZ rhel7.6 under /tmp directory. Do you want to continue?[y/N]:y See log file /tmp/mlnx_iso.4120_logs/mlnx_ofed_iso.4120.log  ...

  • Page 34: Performance Tuning

    Depending on the application of the user's system, it may be necessary to modify the default configuration of network adapters based on the ConnectX® adapters. In case that tuning is required, please refer to the Performance Tuning Guide for NVIDIA Network Adapters. Windows Driver Installation...

  • Page 35: Software Requirements

    On an x64 (64-bit) machine, the output will be “AMD64”. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > InfiniBand Drivers (Learn More) > Nvidia WinOF-2. Download the .exe image according to the architecture of your machine (see Step 1). ...
  • Page 36 • Attended Installation An installation procedure that requires frequent user intervention. • Unattended Installation An automated installation procedure that requires no user intervention. Attended Installation The following is an example of an installation session. Double click the .exe and follow the GUI instructions to install MLNX_WinOF2. [Optional] Manually configure your setup to contain the logs option (replace “LogFile”...
  • Page 37 Read and accept the license agreement and click Next. Select the target folder for the installation.
  • Page 38 • If the user has a standard NVIDIA® card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA® card, only the NVIDIA® card will be updated.
  • Page 39 Select a Complete or Custom installation, follow Step a onward. Select the desired feature to install: • Performances tools - install the performance tools that are used to measure performance in user environment • Documentation - contains the User Manual and Release Notes •...
  • Page 40 Click Next to install the desired tools. Click Install to start the installation. In case firmware upgrade option was checked in Step 7, you will be notified if a firmware upgrade is required (see  ). ...
  • Page 41 13. Click Finish to complete the installation.
  • Page 42 Unattended Installation  If no reboot options are specified, the installer restarts the computer whenever necessary without displaying any prompt or warning to the user. To control the reboots, use the /norestart or /forcerestart standard command-line options. The following is an example of an unattended installation session. Open a CMD console-> Click Start-> Task Manager File-> Run new task-> and enter CMD.

  • Page 43: Firmware Upgrade

    Firmware Upgrade If the machine has a standard NVIDIA® card with an older firmware version, the firmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade firmware manually, please refer to MFT User Manual. ...

  • Page 44: Removing Earlier Nvidia Drivers

    PartnerSupported 2017-01-31  After the installation process, all kernel modules are loaded automatically upon boot. Removing Earlier NVIDIA Drivers  Please unload the previously installed drivers before removing them. To remove all the drivers: Log into the ESXi server with root permissions.

  • Page 45: Updating Adapter Firmware

    To check that your card is programmed with the latest available firmware version, download the mlxup firmware update and query utility. The utility can query for available NVIDIA adapters and indicate which adapters require a firmware update. If the user confirms, mlxup upgrades the firmware using embedded images.

  • Page 46: Troubleshooting

    Troubleshooting General Troubleshooting • Ensure that the adapter is placed correctly Server unable to find the adapter • Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot • Use the drivers that came with the adapter or download the latest •...

  • Page 47: Linux Troubleshooting

    -d <mst_device> q ibstat Ports Information ibv_devinfo To download the latest firmware version, refer to Firmware Version Upgrade the NVIDIA Update and Query Utility. cat /var/log/messages Collect Log File dmesg >> system.log journalctl (Applicable on new operating systems) cat /var/log/syslog Windows Troubleshooting...

  • Page 48: Specifications

    Specifications MCX4111A-XCAT Specifications Size: 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Physical Connector: Single SFP28 (copper and optical) Ethernet: 10GBASE-SR, 10GBASE-LR, 10GBASE-ER, 10GBASE-CX4, SGMII, 1000BASE-X Protocol Support Data Rate: 1/10 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V, 3.3V Power and Airflow...

  • Page 49: Mcx4111A-Acat And Mcx4111A-Acut Specifications

    Airflow Airflow Specifications. Temperature Operational 0°C to 55°C Environmenta Non-operational -40°C to 70°C Humidity Operational 10% to 85% relative humidity  Non-operational 10% to 90% relative humidity  Altitude (Operational) 3050m Safety CB / cTUVus / CE Regulatory CE / FCC / VCCI / ICES / RCM RoHS RoHS compliant Notes: a.

  • Page 50: Mcx4121A-Acat/Mcx4121A-Acht/Mcx4121A-Acst/Mcx4121A-Acut Specifications

    MCX4121A-ACAT/MCX4121A-ACHT/MCX4121A-ACST/ MCX4121A-ACUT Specifications Size: 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile Physical Connector: Dual SFP28 (copper and optical) Ethernet: 25GBASE-R, 20GBASE-KR2, 1000BASE-CX, 1000BASE-KX, 10GBASE-SR, 10GBASE- Protocol LR,10GBASE-ER, 10GBASE-CX4, 10GBASE-CR, 10GBASE-KR, SGMII Support Data Rate: 1/10/25 Gb/s Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V, 3.3V Power and...

  • Page 51: Mcx4131A-Gcat Specifications

    Voltage: 12V, 3.3V Power and Airflow Power Cable Type Passive Cables 10.6W Typical Power Maximum Power Passive Cables 12.6W 1.5W Active Cables 14.2W Maximum power available through SFP28 port: 1.5W Airflow Airflow Specifications. Temperature Operational 0°C to 55°C Environmental Non-operational -40°C to 70°C Humidity Operational 10% to 85% relative...

  • Page 52: Airflow Specifications

    Safety CB / cTUVus / CE Regulatory CE / FCC / VCCI / ICES / RCM RoHS RoHS compliant Notes: a. Typical power for ATIS traffic load. b. The non-operational storage temperature specifications apply to the product without its package. Airflow Specifications 10Gb/s Adapter Cards Airflow Airflow (LFM)

  • Page 53: Board Mechanical Drawing And Dimensions

    Board Mechanical Drawing and Dimensions  All PSB dimensions are in millimeters. All the mechanical tolerances are +/- 0.13mm. Single-port 10/25 Gb/s Dual-port 10/25 Gb/s Single-port 40/50 Gb/s Bracket Mechanical Drawing  All bracket dimensions are in millimeters. All the mechanical tolerances are +/- 0.25mm. Single-port Tall Bracket Dual-port Tall Bracket...

  • Page 54: Finding The Mac On The Adapter Card

    Finding the MAC on the Adapter Card Each NVIDIA adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol.  The product revisions indicated on the labels in the following figures do not necessarily...

  • Page 55: Mcx4111A-Xcat Boarding Label (Example)

    MCX4111A-XCAT Boarding Label (Example) MCX4111A-ACAT Board Label (Example)

  • Page 56: Mcx4121A-Acat Board Label (Example)

    MCX4121A-ACAT Board Label (Example) MCX4121A-XCHT  Board Label (Example)

  • Page 57: Mcx4111A-Acut Board Label (Example)

    MCX4111A-ACUT Board Label (Example) MCX4121A-ACUT Board Label (Example)

  • Page 58: Mcx4121A-Acst Board Label (Example)

    MCX4121A-ACST Board Label (Example) MCX4131A-BCAT Board Label (Example)
  • Page 59 MCX4131A-GCAT Board Label (Example)
  • Page 60 Document Revision History Date Description of Changes May. 2023 Added non-operational storage temperature specifications. Nov. 2020 Added MCX4121A-ACHT support across the document. Feb. 2020 Added altitude criteria to "Specifications". Oct. 2019 Updated Ethernet supported protocols in "Specifications". Sep. 2019 Added MCX4121A-XCHT support across the document. May.
  • Page 61 Date Description of Changes Dec. 2015 • Added mlxup to Documents List on page 9 and Updating Adapter Card Firmware • Updated Device ID in Product Overview • Added IEEE 802.3by support. See “Features and Benefits” • Updated PCI Device •...
  • Page 62 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
  • Page 63 Copyright © 2023 NVIDIA Corporation & affiliates. All Rights Reserved.

This manual is also suitable for:

900-9x4b0-0052-0t1900-9x4b0-0052-st0900-9x4b0-0013-0q0900-9x4b0-0053-0t1900-9x4b0-0013-st0900-9x4b0-0053-st0 ... Show all

Table of Contents