Mellanox Technologies ConnectX-3 Pro User Manual
Mellanox Technologies ConnectX-3 Pro User Manual

Mellanox Technologies ConnectX-3 Pro User Manual

Vpi single and dual qsfp port adapter card
Hide thumbs Also See for ConnectX-3 Pro:
Table of Contents

Advertisement

Quick Links

ConnectX®-3 Pro VPI Single and Dual QSFP
Port Adapter Card User Manual for Open
Compute Project
P/N:
MCX345A-FCPN, MCX346A-FCPN
Rev 1.3
www.mellanox.com

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ConnectX-3 Pro and is the answer not in the manual?

Questions and answers

Summary of Contents for Mellanox Technologies ConnectX-3 Pro

  • Page 1 ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project P/N: MCX345A-FCPN, MCX346A-FCPN Rev 1.3 www.mellanox.com...
  • Page 2 KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.
  • Page 3: Table Of Contents

    Installation Script..........21 Mellanox Technologies...
  • Page 4 Appendix E Sicherheitshinweise (Warnings in German) ..... . . 65 Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) . 67 Mellanox Technologies...
  • Page 5: List Of Tables

    I2C-compatible Connector Pinout ......... . 61 Mellanox Technologies...
  • Page 6: List Of Figures

    Figure 10: MCX346A-FCPN Board Label ..........62 Mellanox Technologies...
  • Page 7: Revision History

    • Table 7.2, “MCX346A-FCPN Specifications”. December 2015 • Updated Table 11, “Physical and Logical Link Indications (Ethernet Mode),” on page 59 • Added Table 12, “Physical and Logical Link Indications (Infini- Band Mode),” on page 59 November 2015 First Release Mellanox Technologies...
  • Page 8: About This Manual

    Rev 1.3 About this Manual This User Manual describes Mellanox Technologies ConnectX®-3 Pro VPI Single and Dual QSFP Port PCI Express x8 network interface cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant docu- mentation.
  • Page 9: Online Resources

    ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Mellanox EN for Linux Driver Release Release notes for Mellanox Technologies' MLNX_EN for Linux Notes driver kit for Mellanox adapter cards: http://www.mellanox.com =>...
  • Page 10: Chapter 1 Introduction

    ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Introduction This is the User Guide for Mellanox Technologies adapter cards based on the ConnectX®-3 Pro VPI integrated circuit device. These adapters’ connectivity provide the highest performing and...
  • Page 11: Features And Benefits

    Virtual Protocol management protocol to seamlessly operate over any converged network with the Interconnect (VPI) same software infrastructure. ConnectX-3 Pro delivers low latency, high bandwidth, and computing efficiency InfiniBand Architecture for performance-driven server and storage clustering applications. ConnectX-3 Specification v1.2.1 Pro is InfiniBand Architecture Specification v1.2.1 compliant.
  • Page 12: Operating Systems/Distributions

    Applications utilizing TCP/UDP/IP transport can achieve industry leading throughput over InfiniBand or 10/40/56GbE. The hardware-based stateless off- Sockets Acceleration load engines in ConnectX-3 Pro reduce the CPU overhead of IP packet transport. Sockets acceleration software further increases performance for latency sensitive applications.
  • Page 13: Chapter 2 Interfaces

    • Support for MSI/MSI-X mechanisms C-compatible Interface A three hole footprint for I C harness is provided as the I2C-compatible interface. See Figure 1 and the Mechanical Drawing of the Dual-Port MCX346A-FCPN Adapter Card on page 58 the location on the board. Mellanox Technologies...
  • Page 14: Led Interface

    ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Figure 1: I C Harness LED Interface There are two I/O LEDs per port. For LED specifications please refer to Section 7.4, “Adapter LED Operation,” on page Mellanox Technologies...
  • Page 15: Chapter 3 Hardware Installation

    Please note that if the card is removed hastily from the antistatic bag, the plastic zip lock may harm the EMI fingers on the QSFP connector. Carefully remove the card Figure 2 from the antistatic bag to avoid damaging the EMI fingers. See Figure 3. Mellanox Technologies...
  • Page 16: Card Installation Instructions

    PCI Express Mezzanine connector. If the card is installed in a PCI Mezzanine connector with less lanes than the card requires, then the Mezzanine card will not provide the optimum data transfer Mellanox Technologies...
  • Page 17: Cables And Modules

    6. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
  • Page 18: On Linux

    Rev 1.3 ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi- tory at http://pci-ids.ucw.cz/read/PC/15b3. 3.6.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella-...
  • Page 19: Chapter 4 Driver Installation

    => Products => Soft- ware => InfiniBand/VPI Drivers => Download. This chapter describes how to install and test the Mellanox OFED for Linux package on a single host machine with Mellanox ConnectX-3 Pro adapter hardware installed. Note: The shown versions and/or parameter values in the example below may not reflect the lat- est or actual values for this product, and are included here for illustration purposes only.
  • Page 20: Installing Mellanox Ofed

    Example The following command will create a MLNX_OFED_LINUX ISO image for RedHat 5.6 under the /tmp directory. MLNX_OFED_LINUX-1.5.3-rhel5.6-x86_64/docs/mlnx_add_kernel_support.sh -i /mnt/MLNX- _OFED_LINUX-1.5.3-rhel5.6-x86_64.iso 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Mellanox Technologies...
  • Page 21: 4.1.3.2 Installation Script

    Do you want to continue?[y/N]:y Removing OFED RPMs... Running mkisofs... Created /tmp/MLNX_OFED_LINUX-1.5.3-rhel5.6-x86_64.iso 4.1.3.2 Installation Script The usage of the installation script is described below. You will use it during the installation pro- cedure described in Section 4.1.3.4, “Installation Procedure,” on page 23 Mellanox Technologies...
  • Page 22 Ethernet -v|-vv|-vvv Set verbosity level Set quiet - no messages will be printed --umad-dev-rw Grant non root users read/write permission for umad devices instead of default --hugepages-overcommit Set 80% of MAX_MEMORY as overcommit for a huge page allocation Mellanox Technologies...
  • Page 23: 4.1.3.3 Mlnxofedinstall Return Codes

    Failed to start the mst driver 4.1.3.4 Installation Procedure Login to the installation machine as root. Step 1. Mount the ISO image on your machine Step 2. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Mellanox Technologies...
  • Page 24 ################################################## mlnx-ofa_kernel ################################################## Installing kmod-mlnx-ofa_kernel RPM Preparing... ################################################## kmod-mlnx-ofa_kernel ################################################## Installing mlnx-ofa_kernel-devel RPM Preparing... ################################################## mlnx-ofa_kernel-devel ################################################## Installing kernel-mft RPM Preparing... ################################################## kernel-mft ################################################## Installing knem RPM Preparing... ################################################## knem ################################################## Installing mpi-selector RPM Preparing... ################################################## mpi-selector ################################################## Mellanox Technologies...
  • Page 25 ################################################## Preparing... ################################################## libmlx5-devel ################################################## Preparing... ################################################## libmverbs-devel ################################################## Preparing... ################################################## libmverbs-devel ################################################## Preparing... ################################################## libmqe ################################################## Preparing... ################################################## libmqe ################################################## Preparing... ################################################## libmqe-devel ################################################## Preparing... ################################################## libmqe-devel ################################################## Preparing... ################################################## libibcm ################################################## Preparing... ################################################## libibcm ################################################## Mellanox Technologies...
  • Page 26 ################################################## Preparing... ################################################## ibacm ################################################## Preparing... ################################################## librdmacm ################################################## Preparing... ################################################## librdmacm ################################################## Preparing... ################################################## librdmacm-utils ################################################## Preparing... ################################################## librdmacm-devel ################################################## Preparing... ################################################## librdmacm-devel ################################################## Preparing... ################################################## opensm-libs ################################################## Preparing... ################################################## opensm-libs ################################################## Preparing... ################################################## opensm ################################################## Mellanox Technologies...
  • Page 27 ################################################## dapl-devel-static ################################################## Preparing... ################################################## dapl-utils ################################################## Preparing... ################################################## perftest ################################################## Preparing... ################################################## mstflint ################################################## Preparing... ################################################## ################################################## Preparing... ################################################## srptools ################################################## Preparing... ################################################## rds-tools ################################################## Preparing... ################################################## rds-devel ################################################## Preparing... ################################################## ibutils2 ################################################## Preparing... ################################################## ibutils ################################################## Mellanox Technologies...
  • Page 28 - Check /opt/mellanox/fca/share/doc/fca/README.txt for quick start instructions. Preparing... ################################################## ################################################## Preparing... ################################################## openshmem ################################################## Preparing... ################################################## bupc ################################################## Preparing... ################################################## mvapich2_gcc ################################################## Preparing... ################################################## openmpi_gcc ################################################## Preparing... ################################################## mpitests_mvapich2_gcc ################################################## Preparing... ################################################## mpitests_openmpi_gcc ################################################## Preparing... ################################################## mlnxofed-docs ################################################## Mellanox Technologies...
  • Page 29 ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Device (05:00.0): 05:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) Link Width is not 8x PCI Link Speed: 5Gb/s Device (07:00.0):...
  • Page 30: Installation Results

    4.1.4 Installation Results Software • The OFED and MFT packages are installed under the /usr directory. • The kernel modules are installed under: • InfiniBand subsystem: /lib/modules/`uname -r`/updates/kernel/drivers/infiniband/ • mlx4_core driver: /lib/modules/<kernel_version>/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellanox/ mlx4/mlx4_core.ko • mlx4_ib driver: /lib/modules/<kernel_version>/extra/mlnx-ofa_kernel/drivers/infiniband/hw/mlx4/mlx- 4_ib.ko Mellanox Technologies...
  • Page 31 • The file is installed under 90-ib.rules /etc/udev/rules.d/ • If OpenSM is installed, the daemon is installed under opensmd /etc/init.d/ is installed under opensm.conf /etc • If IPoIB configuration files are included, files will be installed under: ifcfg-ib<n> Mellanox Technologies...
  • Page 32: Post-Installation Notes

    If you need to burn an Expansion ROM image, please refer to OFED User Manual. Table 2, “Documents List,” on page 8. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site ( => Support => Firm- ware Download). Mellanox Technologies...
  • Page 33: Uninstalling Mellanox Ofed

    -dev /dev/mst/mt25418_pci_cr0 -fw /mnt/firmware/fw-25408/fw-25408- rel.mlx Reboot your machine after the firmware burning is completed. 4.1.7 Uninstalling Mellanox OFED Use the script to uninstall the Mellanox OFED package. /usr/sbin/ofed_uninstall.sh The script is part of the RPM. ofed-scripts Mellanox Technologies...
  • Page 34: Port Type Management On Linux

    For example, if the first port is connected to an InfiniBand switch and the second to Ethernet switch, the NIC will automatically load the first switch as InfiniBand and the second as Ethernet. 4.1.9.1 Enabling Auto Sensing Upon driver start up: 1. Sense the adapter card’s port type: Mellanox Technologies...
  • Page 35: Windows Driver

    Enter the following command: > echo %PROCESSOR_ARCHITECTURE% On an x64 (64-bit) machine, the output will be “AMD64”. Go to the MLNX_WinOF for Windows Web page at Step 2. http://www.mellanox.com => Products => Software => Ethernet Drivers => Windows SW/ Drivers. Mellanox Technologies...
  • Page 36: Extracting Files Without Running Installation

    To extract the files without running installation, perform the following steps. Open a CMD console (Click Start-->Run and enter CMD). Step 1. Enter the following command: Step 2. MLNX_VPI_WinOF-<version>_All_<OS>_<arch>.exe /a Click Next to create a server image. Step 3. Mellanox Technologies...
  • Page 37 ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Click Change and specify the location in which the files are extracted to. Step 4. Click Install to extract this folder, or click Change to install to a different folder. Step 5. Mellanox Technologies...
  • Page 38: Installing Mlnx_Winof

    To configure your setup to contain the logs option, please run the following command after opening a CMD console: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v"/l*vx [LogFile]" If you do not want to upgrade your firmware version, run the following command: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" MT_SKIPFWUPGRD=1" For further help, please run: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" /h" Mellanox Technologies...
  • Page 39 ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Click Next in the Welcome screen. Step 2. Read then accept the license agreement and click Next. Step 3. Mellanox Technologies...
  • Page 40 If the user has a standard Mellanox card, and the firmware version is older than the one specified in WinOF Installation Guide 4.40, the firmware will be updated accordingly. However, if the user has both OEM card and Mellanox card, only Mellanox card will be updated. Mellanox Technologies...
  • Page 41 ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Configure your system for maximum performance by checking the maximum performance Step 6. box. This step requires rebooting your machine at the end of the installation. Mellanox Technologies...
  • Page 42 • SDK - contains the libraries and DLLs for developing Ethernet application over IBAL. • Documentation: contains the User Manual and Installation Guide. • ND FLTR DLLs: contains the files for standalone installation of the mlx4nd provider. Mellanox Technologies...
  • Page 43 ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Click Install to start the installation. Mellanox Technologies...
  • Page 44: 4.2.4.2 Unattended Installation

    4.2.4.2 Unattended Installation The following is an example of a MLNX_WinOF_win8 x64 unattended installation session. Open the CMD console (click Start > Run and enter ‘cmd’) Step 1. Install the driver. Run: Step 2. > MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /S /v"/qn" Mellanox Technologies...
  • Page 45: Upgrading Mlnx_Winof

    Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. To see the Mellanox network adapter device, and the Ethernet or IPoIB network device (depending on the used card) for each port, display the Device Manager and expand “System devices” or “Network adapters”. Mellanox Technologies...
  • Page 46: Uninstalling Mlnx_Winof

    1. Click Start-> Control Panel-> Programs and Features-> MLNX_VPI-> Uninstall. (NOTE: This requires elevated administrator privileges.) 2. Double click the .exe and follow the instructions of the install wizard. 3. Click Start-> All Programs-> Mellanox Technologies-> MLNX_WinOF-> Uninstall MLNX- _WinOF. 4.2.7.2 Unattended Uninstall ...
  • Page 47: Assigning Port Ip After Installation

    Open the Network Connections window. Locate Local Area Connections with Mellanox Step 1. devices. Right-click a Mellanox Local Area Connection and left-click Properties. Step 2. Select Internet Protocol Version 4 (TCP/IPv4) from the scroll list and click Properties. Step 3. Mellanox Technologies...
  • Page 48: Performance Tuning

    Default Gateway ..: Performance Tuning For guidelines on improving network adapter performance, please refer to the performance tun- ing guidelines for a Linux/Windows environment in the document http://www.mellanox.com/ related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf. Mellanox Technologies...
  • Page 49: Vmware Driver

    1. Log into the ESXi5.0 server with root permissions. 2. List the existing InfiniBand OFED driver modules. #> esxcli software vib list | grep Mellanox net-ib-core 1.8.1 OEM.500.0.0.472560 Mellanox Partner Supported 2013-02-09 net-ib-ipoib 1.8.1 OEM.500.0.0.472560 Mellanox Partner Supported 2013-02-09 Mellanox Technologies...
  • Page 50: Flexboot

    Preboot Execution Environment (PXE) standard specification, and FlexBoot soft- ware is based on the open source iPXE project (see www.ipxe.org). For more information go to http://www.mellanox.com => Products => Software => InfiniBand/VPI Drivers => Download. Mellanox Technologies...
  • Page 51: Chapter 5 Updating Card Firmware

    Enter your card PSID to display the latest firmware file. The file name of the binary is composed by combin- ing the firmware name, the firmware release version, and the card part number. Note: Please contact Mellanox System Support if you cannot find the firmware binary for your card. Mellanox Technologies...
  • Page 52: Figure 5: Support Download Assistant

    > flint -d /dev/mst/mt4099_pci_cr0 -i <binary image> burn a. Reboot the computer. b. Enter: mst start. c. Verify that the card firmware was updated successfully. > flint -d /dev/mst/mt4099_pci_cr0 q Image type: ConnectX FW Version: 2.9.4100 Device ID: 4099 Mellanox Technologies...
  • Page 53: Chapter 6 Troubleshooting

    • Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but • Check that both the adapter and its link are set to the same speed with no and duplex settings communication established Mellanox Technologies...
  • Page 54: Linux

    Mellanox Firmware Tool (MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo Firmware Version To download the latest firmware version refer to Upgrade http://www.mellanox.com/supportdownloader /var/log/messages Collect Log File dmesg > system.logF Mellanox Technologies...
  • Page 55: Windows

    –d <mst_device> –i <firmware_bin_file> b • Event log viewer • MST device logs: • mst start Collect log file • mst status • C:\Users\Administrator> flint –d <mst_device> dc > dump_configura- tion.log • C:\Users\Administrator> mstdump <mst_device> dc > mstdump.log Mellanox Technologies...
  • Page 56: Chapter 7 Specifications

    => Products => Cables and Transceivers Cable Support a. The ConnectX-3 Pro adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mella- nox InfiniBand product.
  • Page 57: Mcx346A-Fcpn Specifications

    => Products => Cables and Transceivers Cable Support a. The ConnectX-3 Pro adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mella- nox InfiniBand product.
  • Page 58: Board Mechanical Drawing And Dimensions

    ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm Figure 6: Mechanical Drawing of the Dual-Port MCX346A-FCPN Adapter Card I2C connector Mellanox Technologies...
  • Page 59: Adapter Led Operation

    A constant Green indicates a valid logical (data activity) link with- out data transfer. link/activity Port 0: • A blinking Green indicates a valid logical link with data transfer. LED1=D3, LED0=D1 • If LED1 is off, then the logical link has not been established. Mellanox Technologies...
  • Page 60: Appendix A Interface Connectors Pinout

    Appendix A: Interface Connectors Pinout PCI Express x8 Connector Pinout The cards use a standard PCI Express x8 connector and the PCI Express x8 standard pinout according to the PCI Express 3.0 specification. Figure 7: PCIe Connector Pinout Mellanox Technologies...
  • Page 61: Table 13: I2C-Compatible Connector Pinout

    ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 C-compatible Connector Pinout A three hole footprint for I2C Harness is provided as the I2C-compatible interface. Figure 8: Compatible Connector Plug and Pinout Table 13 - I2C-compatible Connector Connector Pin Signal Name Number Pinout Mellanox Technologies...
  • Page 62: Appendix B Finding The Guid/Mac And Serial Number On The Adapter Card

    Each Mellanox adapter card has a different identifier printed on the label: serial numberand the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID). Figure 9: MCX345A-FCPN Board Label Figure 10: MCX346A-FCPN Board Label Mellanox Technologies...
  • Page 63: Appendix C Safety Warnings

    Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...
  • Page 64: Appendix D Avertissements De Sécurité D'installation (Warnings In French)

    PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...
  • Page 65: Appendix E Sicherheitshinweise (Warnings In German)

    Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. 7. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen.. Mellanox Technologies...
  • Page 66 Rev 1.3 Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...
  • Page 67: Appendix F Advertencias De Seguridad Para La Instalación (Warnings In Spanish)

    Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Mellanox Technologies...

Table of Contents