Mellanox Technologies 0NHYP5 User Manual
  1. Manuals
  2. Brands
  3. Mellanox Technologies Manuals
  4. Computer Hardware
  5. 0NHYP5
  6. User manual

Mellanox Technologies 0NHYP5 User Manual

Connectx-4 100gb/s ethernet dual qsfp28 adapter card
Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
ConnectX®-4 100Gb/s Ethernet Dual QSFP28
Adapter Card User Manual for Dell PowerEdge
P/N:
0NHYP5, 0XR0K2
Rev 1.2
www.mellanox.com

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the 0NHYP5 and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Mellanox Technologies 0NHYP5

Summary of Contents for Mellanox Technologies 0NHYP5

  • Page 1 ConnectX®-4 100Gb/s Ethernet Dual QSFP28 Adapter Card User Manual for Dell PowerEdge P/N: 0NHYP5, 0XR0K2 Rev 1.2 www.mellanox.com...
  • Page 2 KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.

  • Page 3: Table Of Contents

    4.1.1 Hardware and Software Requirements ....... . . 23 Mellanox Technologies...
  • Page 4 6.1 General ............51 Mellanox Technologies...
  • Page 5 7.1 0NHYP5/0XR0K2 Specifications ........53...

  • Page 6: List Of Tables

    0NHYP5/0XR0K2 Specifications Table ........

  • Page 7: List Of Figures

    Figure 9: 0NHYP5/0XR0K2 Board Labels (Example) ........60...

  • Page 8: Revision History

    Linux Driver on page 23. • Updated Windows Driver on page 36. • Updated VMware Driver on page 47. • Updated 0NHYP5/0XR0K2 Specifications on page 53. • Updated Airflow Specifications on page 54. • Updated Adapter Card LED Operations on page 51. • Updated Bracket Mechanical Drawings on page 56 •...

  • Page 9: About This Manual

    Rev 1.2 About this Manual This User Manual describes Mellanox Technologies ConnectX®-4 100Gb/s Ethernet Dual QSFP28 port PCI Express x16 adapter cards for Dell PowerEdge. The User Manual provides details as to the interfaces of the adapter card, specifications, required software and firmware for operating the adapter card, and relevant documentation.

  • Page 10: Document Conventions

    The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support Or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Mellanox Technologies...

  • Page 11: Chapter 1 Introduction

    Rev 1.2 Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con- nectX®-4 integrated circuit device. These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers, High-Performance Computing, and Embedded environments.

  • Page 12: Product Overview

    Express speed. Each adapter comes with two bracket heights - short and tall. Table 3 - Dual-port Ethernet Adapter Cards 0XR0K2 - short bracket Ordering Part Number (OPN) 0NHYP5 - tall bracket Ethernet: 10/25/40/50/100 Gb/s Data Transmission Rate Dual-port QSFP28 Network Connector Types PCIe 3.0 x16 8GT/s...

  • Page 13: Features And Benefits

    GPU to CPU), and therefore significantly reduces application run time. ConnectX-4 advanced acceleration technology enables higher cluster effi- ciency and scalability to tens of thousands of nodes. Adapter functionality enabling reduced CPU overhead allowing more available CPU offload CPU for computation tasks. Mellanox Technologies...

  • Page 14: Operating Systems/Distributions

    • FreeBSD • VMware • OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Connectivity • Interoperable with 10/25/40/50/100 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Mellanox Technologies...

  • Page 15: Chapter 2 Interfaces

    • Auto-negotiates to x16, x8, x4, x2, or x1 • Support for MSI/MSI-X mechanisms LED Interface There is a one bi-color I/O LED per port located on the adapter card. For LED specifications, please refer to Section 7.3, “Adapter Card LED Operations,” on page Mellanox Technologies...

  • Page 16: Chapter 3 Hardware Installation

    To replace the bracket you will need the following parts: • The new bracket of the proper height • The 2 screws saved from the removal of the bracket Mellanox Technologies...

  • Page 17: Removing The Existing Bracket

    A lesser width adapter can be seated into a greater width slot (x8 in a x16), but a greater width adapter cannot be seated into a lesser width slot (x16 in a x8). Align the adapter connector edge with the PCI Express connector slot. Mellanox Technologies...

  • Page 18: Cables And Modules

    Cables and Modules To obtain the list of supported cables for your adapter, please refer to www.mellanox.com => Prod- ucts => Cables and Transceivers. 3.7.1 Cable Installation 1. All cables can be inserted or removed with the unit powered on. Mellanox Technologies...

  • Page 19: Adapter Card Un-Installation Instructions

    1. Remove any metallic objects from your hands and wrists. 2. It is strongly recommended to use an ESD strap or other antistatic devices. 3. Turn off the system and disconnect the power cord from the server. Mellanox Technologies...

  • Page 20: Card Un-Installation

    5. When the port connectors reach the top of the chassis window, gently pull the adapter card in parallel to the motherboard. Identify the Card in Your System 3.9.1 On Windows 1. Open Device Manager on the server. Click start => Run, and then enter “devmgmt.msc”. Mellanox Technologies...

  • Page 21: On Linux

    5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
  • Page 22 ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 > lspci |grep -i Mellanox Network controller: Mellanox Technologies MT4115 Family [ConnectX-4] Mellanox Technologies...

  • Page 23: Chapter 4 Driver Installation

    Step 1. The following example shows a system with an installed Mellanox HCA: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 Download the ISO image to your host.

  • Page 24: Installing Mellanox Ofed

    Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using is irrelevant on Ubuntu and "mlnx_add_kernel_support.sh" Debian distributions. 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Mellanox Technologies...

  • Page 25: Installation Procedure

    Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed. Uninstalling the previous version of MLNX_OFED_LINUX Starting MLNX_OFED_LINUX-x.x.x installation ....Installation finished successfully. Attempting to perform Firmware update... Querying Mellanox devices firmware ... Mellanox Technologies...
  • Page 26 Most of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files. See the relevant chapters in this manual for details. The list of the modules that will be loaded automatically upon boot can be found in the /etc/ file. infiniband/openib.conf Mellanox Technologies...

  • Page 27: Installation Results

    This can be controlled by setting the variables below in the /etc/infiniband/openibd.conf file. OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh 4.1.3.5 Driver Load Upon System Boot Upon system boot, the Mellanox drivers will be loaded automatically.  To prevent automatic load of the Mellanox drivers upon system boot: Mellanox Technologies...

  • Page 28: Mlnxofedinstall Return Codes

    # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt You can download it from http://www.mellanox.com => Products => Software => Ether- net Drivers. Download and install Mellanox Technologies GPG-KEY: Step 3. The key can be downloaded via the following link: Mellanox Technologies...

  • Page 29: Installing Mlnx_Ofed Using The Yum Tool

    # sudo rpm --import RPM-GPG-KEY-Mellanox warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mel- lanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Is this ok [y/N]: Check that the key was successfully imported.
  • Page 30 MLNX_OFED, you can use the mlnx_add_ker- script to build MLNX_OFED for your kernel. nel_support.sh The script will automatically build the matching group RPM packages for your kernel so that you can still install MLNX_OFED via yum. Mellanox Technologies...

  • Page 31: Uninstalling Mellanox Ofed Using The Yum Tool

    To update the firmware to the version included in MLNX_OFED package, you can either: • Run # yum install mlnx-fw-updater • Update the firmware to the latest version available on Mellanox Technologies’ Web site as described in Section 4.1.7, “Updating Firmware After Installation,” on page 4.1.5.3 Uninstalling Mellanox OFED using the YUM Tool...

  • Page 32: Installing Mlnx_Ofed Using The Apt-Get Tool

    ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 Check that the key was successfully imported. Step 5. # apt-key list 1024D/A9E4B643 2013-08-11 Mellanox Technologies <support@mellanox.com> 1024g/09FCC269 2013-08-11 Update the apt-get cache. Step 6. # sudo apt-get update 4.1.6.2 Installing MLNX_OFED using the apt-get Tool...

  • Page 33: Updating The Device Online

    OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site (http:// www.mellanox.com...

  • Page 34: Uefi Secure Boot

    UEFI Secure Boot is enabled. To see what keys have been added to the system key ring on the current boot, install the 'keyutils' package and run: #keyctl list %:.system_keyring Mellanox Technologies...

  • Page 35: Removing Signature From Kernel Modules

    After the signature has been removed, a massage as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" However, please note that a similar message as the following will still be presented: "my_module: module verification failed: signature and/or required key missing - taint-...

  • Page 36: Windows Driver

    Installing the incorrect .exe file is prohibited. If you do so, an error message will be displayed. For example, if you try to install a 64-bit .exe on a 32-bit machine, the wizard will display the follow- ing (or a similar) error message: Mellanox Technologies...

  • Page 37: Installing Mellanox Winof-2 Driver

    [Optional] Manually configure your setup to contain the logs option. Step 2. > MLNX_WinOF2-1_35_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. > MLNX_WinOF2-1_35_All_x64.exe /v" MT_SKIPFWUPGRD=1" Click Next in the Welcome screen. Step 4. 1. MT_SKIPFWUPGRD default value is False Mellanox Technologies...
  • Page 38 ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 Read then accept the license agreement and click Next. Step 5. Select the target folder for the installation. Step 6. Mellanox Technologies...
  • Page 39 If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a Mellanox card, only the Mellanox card will be updated. Select a Complete or Custom installation, follow Step a and on. Step 8. Mellanox Technologies...
  • Page 40 Analyze tools - install the tools that can be used either to diagnosed or analyzed the InfiniBand envi- ronment. • SDK - contains the libraries and DLLs for developing InfiniBand application over IBAL. • Documentation - contains the User Manual and Release Notes • Management tools - installation tools used for management, such as mlxstat. Mellanox Technologies...
  • Page 41 ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 b. Click Install to start the installation. In case firmware upgrade option was checked in Step 7., you will be notified if a firmware Step 9. upgrade is required. Mellanox Technologies...

  • Page 42: Unattended Installation

    [Optional] if you want to control whether to install ND provider or not Step 4. > MLNX_WinOF2-1_35_All_x64.exe /vMT_NDPROPERTY=1 [Optional] If you do not wish to upgrade your firmware version Step 5. > MLNX_WinOF2-1_35_All_x64.exe /v" MT_SKIPFWUPGRD=1" 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Mellanox Technologies...

  • Page 43: Installation Results

    To extract the files without running installation, perform the following steps. Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. > MLNX_WinOF2-1_35_All_x64 /a Mellanox Technologies...
  • Page 44 ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 • To extract only the driver files. > MLNX_WinOF2-1_35_All_x64 /a /vMT_DRIVERS_ONLY=1 Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Mellanox Technologies...

  • Page 45: Uninstalling Mellanox Winof-2 Driver

    Step 5. To complete the extraction, click Finish. Step 6. 4.2.6 Uninstalling Mellanox WinOF-2 Driver 4.2.6.1 Attended Uninstallation  To uninstall MLNX_WinOF2 on a single node: Click Start -> Control Panel -> Programs and Features -> MLNX_WinOF2 -> Uninstall. Mellanox Technologies...

  • Page 46: Unattended Uninstallation

    For information on how to upgrade firmware manually, please refer to MFT User Manual: www.mellanox.com ->Products -> Ethernet Drivers -> Firmware Tools Performance Tuning For further information on WinOF-2 performance, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Mellanox Technologies...

  • Page 47: Vmware Driver

    4. Verify the driver was installed successfully. # esxcli software vib list | grep mlx ESX 5.5: nmlx5-core 4.5.2.0-1OEM.550.0.0.1391871 PartnerSupported 2016-02-01 ESX 6.0: nmlx5-core 4.15.4.0-1OEM.600.0.0.2768847 PartnerSupported 2016-02-01 After the installation process, all kernel modules are loaded automatically upon boot. Mellanox Technologies...

  • Page 48: Removing Previous Mellanox Driver

    /etc/init.d/sfcbd-watchdog start kill -POLL $(cat /var/run/vmware/vmkdevmgr.pid) 4.3.5 Firmware Programming 1. Download the VMware images v4.3.0 from the site. Mellanox Firmware Tools (MFT) • ESX 5.5 File: mft-4.3.0.25-10EM-550.0.0.1391871.x86_64.vib MD5SUM: 62f953b77f4ea49c420ff4e1ce8b925b • ESX 6.0 File: mft-4.3.0.25-10EM-600.0.0.2768847.x86_64.vib MD5SUM: 6a3bd63882f2ca3f696c1a96d62112df Mellanox Technologies...
  • Page 49 ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 2. Install the image according to the steps described in the MFT User Manual. The following procedure requires custom boot image downloading, mounting and boot- ing from a USB device. Mellanox Technologies...

  • Page 50: Chapter 5 Updating Adapter Card Firmware

    PCI Device Name: 0000:07:00.0 Base MAC: 0000e41d2da206d4 Versions: Current Available 12.12.0780 12.12.1100 Status: Update required Perform FW update? [y/N]: y Device #1: Up to date Device #2: Updating FW ... Done Restart needed for updates to take effect. Log File: /var/log/mlxup/mlxup-yyyymmdd.log Mellanox Technologies...

  • Page 51: Chapter 6 Troubleshooting

    • Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but • Check that both the adapter and its link are set to the same speed with no and duplex settings communication established Mellanox Technologies...

  • Page 52: Linux

    Mellanox Firmware Tool (MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo To download the latest firmware version refer to Firmware Version http://www.mellanox.com/supportdownloader Upgrade /var/log/messages Collect Log File dmesg > system.logF Mellanox Technologies...

  • Page 53: Chapter 7 Specifications

    ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 Specifications 0NHYP5/0XR0K2 Specifications Table 8 - 0NHYP5/0XR0K2 Specifications Table Size: 2.71 in. x5.6 in. (68.90mm x 142.24 mm) – low profile Physical Connector: Dual QSFP28 (Copper and optical)

  • Page 54: Airflow Specifications

    Solid Yellow Indicates an active physical link Blinking Yellow Indicates a problem with the physical link Solid Green Indicates a valid logical (data activity) link with no active traffic Blinking Green Indicates a valid logical link with active traffic Mellanox Technologies...

  • Page 55: Board Mechanical Drawing And Dimensions

    Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 3: Mechanical Drawing of the Single-port x16 Adapter Card 142.24 68.90 Figure 4: Mechanical Drawing of the Dual-port x16 Adapter Card 142.24 68.90 Mellanox Technologies...

  • Page 56: Bracket Mechanical Drawings

    Figure 5: Dual-port Low Profile Bracket Port 1 Link and Activity Bi-color LED Port 2 Link and Activity Bi-color LED Figure 6: Dual-port Full Height Bracket Port 1 Link and Activity Bi-color LED Port 2 Link and Activity Bi-color LED Mellanox Technologies...

  • Page 57: Appendix A Interface Connectors Pinout

    Table 10 - Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name TXN_2 Tx2n TXP_2 Tx2p TXN_4 Tx4n TXP_4 Tx4p ModSelL_Port0 ModSelL ResetL_Port0 ResetL VccRx RXP_3 Rx3p RXN_3 Rx3n Mellanox Technologies...

  • Page 58: A.2 Pci Express X16 Edge Connector Pinout

    TXN_3 Tx3n TXP_1 Tx1p TXN_1 Tx1n PCI Express x16 Edge Connector Pinout The adapter cards use a standard PCI Express x16 edge connector and the PCI Express x16 stan- dard pinout according to the PCI Express 3.0 specification. Mellanox Technologies...

  • Page 59: Figure 8: Pcie X16 Connector Pinout

    ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 Figure 8: PCIe x16 Connector Pinout Mellanox Technologies...

  • Page 60: Appendix B Finding The Mac And Serial Number On The Adapter Card

    Each Mellanox adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. The revisions indicated on the labels in the following figure do not necessarily repre- sent the latest revisions of the card. Figure 9: 0NHYP5/0XR0K2 Board Labels (Example) Mellanox Technologies...
  • Page 61 ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 Mellanox Technologies...

  • Page 62: Appendix C Safety Warnings

    This equipment should be installed in compliance with local and national electrical codes. 8. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. Mellanox Technologies...
  • Page 63 ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Mellanox Technologies...

  • Page 64: Appendix D Avertissements De Sécurité D'installation (Warnings In French)

    L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Mellanox Technologies...
  • Page 65 Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Mellanox Technologies...

  • Page 66: Appendix E Sicherheitshinweise (Warnings In German)

    Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. This equipment should be installed in compliance with local and national elec- trical codes. Mellanox Technologies...
  • Page 67 Rev 1.2 8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1 Mellanox Technologies...

  • Page 68: Appendix F Advertencias De Seguridad Para La Instalación (Warnings In Spanish)

    Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. 8. Exposición a niveles de radiación peligrosos Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. Mellanox Technologies...
  • Page 69 ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1.2 PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Mellanox Technologies...

This manual is also suitable for:

0xr0k2

Table of Contents