Mellanox Technologies ConnectX-3 User Manual
  1. Manuals
  2. Brands
  3. Mellanox Technologies Manuals
  4. Projector
  5. ConnectX-3 Pro
  6. User manual
Mellanox Technologies ConnectX-3 User Manual

Mellanox Technologies ConnectX-3 User Manual

Hide thumbs Also See for ConnectX-3:
Table of Contents

Advertisement

Quick Links

ConnectX®-3 Ethernet Single and Dual SFP+
Port Mezzanine Card User Manual for Open
Compute Project
P/N:
MCX341A-XCCN, MCX342A-XCCN, MCX341A-XCDN, MCX342A-XCDN, MCX341A-XCEN,
MCX342A-XCEN, MCX341A-XCFN, MCX342A-XCFN, MCX341A-XCGN, MCX342A-XCGN
Rev 1.8
www.mellanox.com

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ConnectX-3 and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Mellanox Technologies ConnectX-3

Summary of Contents for Mellanox Technologies ConnectX-3

  • Page 1 ConnectX®-3 Ethernet Single and Dual SFP+ Port Mezzanine Card User Manual for Open Compute Project P/N: MCX341A-XCCN, MCX342A-XCCN, MCX341A-XCDN, MCX342A-XCDN, MCX341A-XCEN, MCX342A-XCEN, MCX341A-XCFN, MCX342A-XCFN, MCX341A-XCGN, MCX342A-XCGN Rev 1.8 www.mellanox.com...
  • Page 2 KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.

  • Page 3: Table Of Contents

    4.2.8 Uninstalling MLNX_WinOF ..........32 4.2.8.1 Attended Uninstall Mellanox Technologies...
  • Page 4 Appendix F Sicherheitshinweise (Warnings in German) ....... 62 Appendix G Advertencias de seguridad para la instalación (Warnings in Spanish)..64 Mellanox Technologies...

  • Page 5: List Of Figures

    Figure 19: Compatible Connector Plug and Pinout ........57 Mellanox Technologies...

  • Page 6: List Of Tables

    I2C-compatible Connector Pinout ......... . 57 Mellanox Technologies...

  • Page 7: Revision History

    (Example only - see Note),” on page 48 Added note to Section 3.4, “Card Installation Instructions,” on page 16 Added Appendix A, “Card Change History,” on page 47 Added “Cable Support” to Table 8, “MCX341A Specifications Table,” on page 41 September 2012 First Release Mellanox Technologies...

  • Page 8: About This Manual

    Rev 1.8 About this Manual This User Manual describes Mellanox Technologies ConnectX®-3 10 Gigabit Ethernet Single and Dual SFP+ Port PCI Express x8 Mezzanine cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant docu- mentation.
  • Page 9 Rev 1.8 Online Resources • Mellanox Technologies web pages: http://www.mellanox.com • Mellanox Technologies Firmware download web page: http://www.mellanox.com => Support => Download Center Document Conventions When discussing memory sizes, MB and MBytes are used in this document to mean size in mega bytes.

  • Page 10: Chapter 1 Introduction

    Introduction Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con- nectX®-3 EN integrated circuit device. These adapters connectivity provide the highest perform- ing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data...

  • Page 11: Table 4: Dual-Port 10 Gigabit Ethernet Adapter Cards

    MCX342A-XCFN: dual-port SFP+ with IPMI and LACP MCX342A-XCGN: dual-port SFP+ with IPMI and NC-SI support PCIe 3.0 x8 8GT/s PCI Express SERDES Speed RoHS MT27518A1-FCCR-XE Adapter IC Part Number 4099 for Physical Function Device ID (decimal) 4100 for Virtual Function Mellanox Technologies...

  • Page 12: Features And Benefits

    Applications utilizing TCP/UDP/IP transport can achieve industry leading Sockets Acceleration throughput over InfiniBand or 10GbE. The hardware-based stateless offload engines in ConnectX-3 reduce the CPU overhead of IP packet transport. Sockets acceleration software further increases performance for latency sensitive applica- tions.

  • Page 13: Operating Systems/Distributions

    • OpenFabrics Windows Distribution (WinOF) • VMware ESX Server 3.5, vSphere 4.0/4.1 Connectivity • Interoperable with 10GigE switches • SFP+ connectors • Passive copper cable and Optical Modules support • Powered connectors for optical and active cable support Mellanox Technologies...

  • Page 14: Chapter 2 Interfaces

    There are two I/O LEDs per port. For LED specifications please refer to Section 7.4, “Adapter LED Operation,” on page 46. Unified Extensible Firmware Interface (UEFI) The UEFI driver is located on the device option ROM and can be used for network boot (for example remote OS installation). Mellanox Technologies...

  • Page 15: Intelligent Platform Management Interface (Ipmi)

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual for OCP Rev 1.8 Intelligent Platform Management Interface (IPMI) To be updated in future release. Mellanox Technologies...

  • Page 16: Chapter 3 Hardware Installation

    To be updated in future release. Cables and Modules The SFP+ port can connect to the Ethernet through both direct attach cables and stand alone transceivers with optical cables. SFP+ transceiver modules are available for both SR and LR pro- tocols. Mellanox Technologies...

  • Page 17: Identify The Card In Your System

    6. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.

  • Page 18: On Linux

    Rev 1.8 Hardware Installation The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi- tory at http://pci-ids.ucw.cz/read/PC/15b3. 3.6.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella- nox Technologies”:...

  • Page 19: Chapter 4 Driver Installation

    #> tar xzvf mlnx_en-1.5.10.tgz file #> cd mlnx_en-1.5.10 #> ./install.sh The package consists of several source RPMs. The install script rebuilds the source RPMs and then installs the created binary RPMs. The created kernel module binaries are placed under /lib/ Mellanox Technologies...

  • Page 20: Loading The Driver

    Uninstalling the Driver To uninstall the mlnx_en driver run: #> /sbin/mlnx_en_uninstall.sh Windows Driver For Windows, download and install the latest Mellanox WinOF VPI for Windows software pack- age available via the Mellanox web site at: http://www.mellanox.com=> Products => Software Mellanox Technologies...

  • Page 21: Hardware And Software Requirements

    For example, if you try to install a 64-bit .exe on a 32-bit machine, the wizard will dis- play the following (or a similar) error message: 4.2.3 Extracting Files Without Running Installation To extract the files without running installation, perform the following steps. Open a CMD console (Click Start-->Run and enter CMD). Step 1. Mellanox Technologies...
  • Page 22 Rev 1.8 Driver Installation Enter the following command: Step 2. MLNX_VPI_WinOF-<version>_All_<OS>_<arch>.exe /a Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Mellanox Technologies...

  • Page 23: Installing Mlnx_Winof

    Click Install to extract this folder, or click Change to install to a different folder. Step 5. To complete the extraction, click Finish. Step 6. 4.2.4 Installing MLNX_WinOF This section provides instructions for two types of installation procedures: • “Attended Installation” An installation procedure that requires frequent user intervention. • “Unattended Installation” Mellanox Technologies...

  • Page 24: 4.2.4.1 Attended Installation

    CMD console: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v"/l*vx [LogFile]" If you do not want to upgrade your firmware version, run the following command: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" MT_SKIPFWUPGRD=1" For further help, please run: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" /h" Click Next in the Welcome screen. Step 2. Mellanox Technologies...
  • Page 25 Select the target folder for the installation. Step 4. The firmware upgrade screen will be displayed in the following cases: Step 5. • If the user has an OEM card, in this case the firmware will not be updated. Mellanox Technologies...
  • Page 26 WinOF Installation Guide 4.40, the firmware will be updated accordingly. However, if the user has both OEM card and Mellanox card, only Mellanox card will be updated. Configure your system for maximum performance by checking the maximum performance Step 6. box. Mellanox Technologies...
  • Page 27 Band environment. • SDK - contains the libraries and DLLs for developing InfiniBand application over IBAL. • Documentation: contains the User Manual and Installation Guide. • ND FLTR DLLs: contains the files for standalone installation of the mlx4nd provider. Mellanox Technologies...
  • Page 28 Rev 1.8 Driver Installation Click Install to start the installation. Mellanox Technologies...

  • Page 29: 4.2.4.2 Unattended Installation

    4.2.4.2 Unattended Installation The following is an example of a MLNX_WinOF_win8 x64 unattended installation session. Open the CMD console (click Start > Run and enter ‘cmd’) Step 1. Install the driver. Run: Step 2. > MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /S /v"/qn" Mellanox Technologies...

  • Page 30: Upgrading Mlnx_Winof

    Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. To see the Mellanox network adapter device, and the Ethernet or IPoIB network device (depending on the used card) for each port, display the Device Manager and expand “System devices” or “Network adapters”. Mellanox Technologies...

  • Page 31: Opensm Activation

     To run OpenSM as a service, assuming the package was installed in the default path, use: sc create OpenSM1 binPath= "c:\Program Files\Mellanox\MLNX_VPI\IB\Tools\ opensm.exe --service" start=auto"  To start the service, run: sc start opensm For further information, please refer to the “OpenSM - Subnet Manager” chapter in the User Manual. Mellanox Technologies...

  • Page 32: Uninstalling Mlnx_Winof

    1. Click Start-> Control Panel-> Programs and Features-> MLNX_VPI-> Uninstall. (NOTE: This requires elevated administrator privileges.) 2. Double click the .exe and follow the instructions of the install wizard. 3. Click Start-> All Programs-> Mellanox Technologies-> MLNX_WinOF-> Uninstall MLNX_WinOF. 4.2.8.2 Unattended Uninstall ...

  • Page 33: Vmware Driver

    For VMware download and install the latest Mellanox OFED Driver for VMware® ESXi Server- software package available via the Mellanox web site at: http://www.mellanox.com => Products => Software => Ethernet Drivers => VMware Drivers => Download. Follow the installation instructions included in the download package (also available from the download page). Mellanox Technologies...

  • Page 34: Installing And Running The Vbi Driver On Esxi-5.X

    Run: #> esxcli software vib list #> esxcli software vib remove –n net-mlx4-en b. Install the mlx4_en driver offline_bundle package. Run: #> esxcli software vib install –d <path>/mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip c. Reboot ESXi server. (The driver will be loaded automatically). Mellanox Technologies...

  • Page 35: Removing The Vib/Offline_Bundle Driver

    FlexBoot supports remote Boot over Ethernet. This technology is based on the Preboot Execution Environment (PXE) standard specification, and FlexBoot software is based on the open source iPXE project (see www.ipxe.org). For more information go to http://www.mellanox.com => Products => Software => Ethernet Drivers => Download. Mellanox Technologies...

  • Page 36: Chapter 5 Updating Adapter Card Firmware

    Enter your card PSID to display the latest firmware file. The file name of the binary is composed by combin- ing the firmware name, the firmware release version, and the card part number. Note: Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card. Mellanox Technologies...

  • Page 37: Figure 2: Support Download Assistant

    > flint -d /dev/mst/mt4099_pci_cr0 -i <binary image> burn a. Reboot the computer. b. Enter: mst start. c. Verify that the card firmware was updated successfully. > flint -d /dev/mst/mt4099_pci_cr0 Image type: ConnectX FW Version: 2.9.4100 Device ID: 4099 Mellanox Technologies...

  • Page 38: Chapter 6 Troubleshooting

    Rev 1.8 Troubleshooting Troubleshooting General • Please use the GUID value returned by the fabric/driver utilities GUID of ConnectX-3 (not 0xfffff). See Appendix B.1, “Retrieving Adapter Card Ethernet adapter GUID,” on page 53 cards • Ensure that the adapter is placed correctly •...

  • Page 39: Linux

    Mellanox Firmware Tool (MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo Firmware Version To download the latest firmware version refer to Upgrade http://www.mellanox.com/supportdownloader /var/log/messages Collect Log File dmesg > system.logF Mellanox Technologies...

  • Page 40: Windows

    –d <mst_device> –i <firmware_bin_file> b • Event log viewer • MST device logs: • mst start Collect log file • mst status • C:\Users\Administrator> flint –d <mst_device> dc > dump_configuration.log • C:\Users\Administrator> mstdump <mst_device> dc > mstdump.log Mellanox Technologies...

  • Page 41: Chapter 7 Specifications

    RoHS: RoHS-R6 Please refer to www.mellanox.com => Products => Cables and Transceivers Cable Support a. For both operational and non-operational states Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 42: Mcx342A Specifications

    RoHS: RoHS-R6 Please refer to www.mellanox.com => Products => Cables and Transceivers Cable Support a. For both operational and non-operational states Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 43: Board Mechanical Drawing And Dimensions

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual for OCP Rev 1.8 Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm Figure 3: Mechanical Drawing of the Single-port MCX341A Mezzanine Card Mellanox Technologies...

  • Page 44: Figure 4: Mechanical Drawing Of The Dual-Port Mcx342A Mezzanine Card

    Rev 1.8 Specifications Figure 4: Mechanical Drawing of the Dual-port MCX342A Mezzanine Card 116.4 110.5 42.05 Mellanox Technologies...

  • Page 45: Mylar Mechanical Drawing And Dimensions

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual for OCP Rev 1.8 7.3.1 Mylar Mechanical Drawing and Dimensions Figure 5: Mechanical Drawing of Mylar 77.17 67.60 62.20 65.17 Figure 6: Mylar on Board Mylar Mellanox Technologies...

  • Page 46: Adapter Led Operation

    A blinking yellow indicates activity (data transfer) • Stays off when there is no activity activity link) The short bracket has the same port and LED footprints as the tall bracket. Regulatory Statements For regulatory statements for all ConnectX®-3 cards please refer to: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.pdf Mellanox Technologies...

  • Page 47: Appendix A Card Change History

    Section 7.3.1, “Mylar Mechanical Drawing and Dimensions,” on page Changes in MCX341A-XCFN from MCX341A-XCEN FW changed to 2.14.1830 Added LACP support Changes in MCX341A-XCGN from MCX341A-XCFN FW changed to 2.31.1580 HW changed: added NCSI/RMII connectivity Added NC-SI support Mellanox Technologies...

  • Page 48: Appendix B Finding The Mac/Guid And Serial Number On The Mezzanine Card

    Note: The revision indicated on the labels in the following figures do not necessarily represent the latest revision of the card. Card revision changes are communicated via Product Change Notification (PCN) documents that are available via card suppliers. Figure 7: MCX341A-XCCN Board Label (Example only - see Note) Mellanox Technologies...

  • Page 49: Figure 8: Mcx341A-Xcdn Board Label (Example Only - See Note)

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual for OCP Rev 1.8 Figure 8: MCX341A-XCDN Board Label (Example only - see Note) Figure 9: MCX342A-XCCN Board Label (Example only - see Note) Mellanox Technologies...

  • Page 50: Figure 10: Mcx342A-Xcdn Board Label (Example Only - See Note)

    Rev 1.8 Figure 10: MCX342A-XCDN Board Label (Example only - see Note) Figure 11: MCX341A-XCEN Board Label (Example only - see Note) Mellanox Technologies...

  • Page 51: Figure 12: Mcx342A-Xcen Board Label (Example Only - See Note)

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual for OCP Rev 1.8 Figure 12: MCX342A-XCEN Board Label (Example only - see Note) Figure 13: MCX341A-XCFN Board Label (Example only - see Note) Mellanox Technologies...

  • Page 52: Figure 14: Mcx342A-Xcfn Board Label (Example Only - See Note)

    Rev 1.8 Figure 14: MCX342A-XCFN Board Label (Example only - see Note) Figure 15: MCX341A-XCGN Board Label (Example only - see Note) Mellanox Technologies...

  • Page 53: Retrieving Adapter Card Guid

    Figure 16: MCX342A-XCGN Board Label (Example only - see Note) Retrieving Adapter Card GUID On ConnectX-3 Ethernet adapter cards, there is a mismatch between the GUID value returned by firmware management tools and that returned by fabric/driver utilities that read the GUID via device firmware (e.g., using ibstat).
  • Page 54 Capability mask: 0x00010000 Port GUID: 0x0202c9fffe305351 Link layer: Ethernet flint -d /dev/mst/mt4099_pci_cr0 q Image type: ConnectX FW Version: 2.30.4260 Device ID: 4099 Description: Node Port1 Port2 Sys image GUIDs: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff MACs: 0002c9305350 0002c9305351 VSD: PSID: MT_1080120023 Mellanox Technologies...

  • Page 55: Appendix C Interface Connectors Pinout

    Appendix C: Interface Connectors Pinout SFP+ Pinout Figure 17: SFP+ Cage Pinout PCI Express x8 Connector Pinout The Mezzanine cards use a standard PCI Express x8 Mezzanine connector and the PCI Express x8 standard pinout according to the PCI Express 3.0 specification. Mellanox Technologies...

  • Page 56: Figure 18: Pcie Connector Pinout

    Rev 1.8 Figure 18: PCIe Connector Pinout Mellanox Technologies...

  • Page 57: I2C-Compatible Connector Pinout

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual for OCP Rev 1.8 C-compatible Connector Pinout Figure 19: Compatible Connector Plug and Pinout Table 11 - I2C-compatible Connector Pinout Connector Pin Number Signal Name SPSDA SPSCL Mellanox Technologies...

  • Page 58: Appendix D Safety Warnings

    This equipment should be installed in compliance with local and national electrical codes. 6. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. Mellanox Technologies...
  • Page 59 ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual for OCP Rev 1.8 CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 60: Appendix E Avertissements De Sécurité D'installation (Warnings In French)

    L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 6. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Mellanox Technologies...
  • Page 61 PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 62: Appendix F Sicherheitshinweise (Warnings In German)

    5. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 6. Regionale und nationale elektrische Bestimmungen Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. Mellanox Technologies...
  • Page 63 Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen.. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 64: Appendix G Advertencias De Seguridad Para La Instalación (Warnings In Spanish)

    5. Eliminación de equipos La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 6. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Mellanox Technologies...
  • Page 65 Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Mellanox Technologies...

This manual is also suitable for:

Mcx341a-xccnMcx342a-xccnMcx341a-xcdnMcx342a-xcdnMcx341a-xcenMcx342a-xcen ... Show all

Table of Contents