Mellanox Technologies ConnectX-3 MCX314A-BCBT User Manual
  1. Manuals
  2. Brands
  3. Mellanox Technologies Manuals
  4. Adapter
  5. ConnectX-3 MCX314A-BCBT
  6. User manual
Mellanox Technologies ConnectX-3 MCX314A-BCBT User Manual

Mellanox Technologies ConnectX-3 MCX314A-BCBT User Manual

Connectx-3 ethernet single and dual qsfp+ port adapter card
Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
ConnectX®-3 Ethernet Single and Dual QSFP+
Port Adapter Card User Manual
P/N:
MCX313A-BCBT, MCX314A-BCBT
Rev 2.3
www.mellanox.com

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ConnectX-3 MCX314A-BCBT and is the answer not in the manual?

Questions and answers

Related Manuals for Mellanox Technologies ConnectX-3 MCX314A-BCBT

Summary of Contents for Mellanox Technologies ConnectX-3 MCX314A-BCBT

  • Page 1 ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual P/N: MCX313A-BCBT, MCX314A-BCBT Rev 2.3 www.mellanox.com...
  • Page 2 QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;...

  • Page 3: Table Of Contents

    4.1 Linux Driver ..........23 Mellanox Technologies...
  • Page 4 Chapter 7 Specifications ......... . . 46 7.1 MCX313A-BCBT Specifications ....... .46 7.2 MCX314A-BCBT Specifications .

  • Page 5: List Of Figures

    Support Download Assistant ..........42 Figure 8: Mechanical Drawing of the Single-port MCX313A-BCBT Adapter Card ... 48 Figure 9: Mechanical Drawing of the Dual-port MCX314A-BCBT Adapter Card .

  • Page 6: List Of Tables

    MCX313A-BCBT Specifications Table ........

  • Page 7: Revision History

    August 2012 • Added operational and non-operational temperature and humid- ity level to the following tables: - Table 7, “MCX313A-BCBT Specifications Table,” on page 46 - Table 8, “MCX314A-BCBT Specifications Table,” on page 47 May 2012 • Removed “Blinking indicates a problem with the physical link”...
  • Page 8 Rev 2.3 Date Comments/Changes April 2012 • Updated power numbers in Table 7, “MCX313A-BCBT Specifi- cations Table,” on page 46 Table 8, “MCX314A-BCBT Specifications Table,” on page 47 January 2012 • Minor edits • Updated LED functions in Section 2.3, “I2C-compatible Inter- face,”...

  • Page 9: About This Manual

    Rev 2.3 About this Manual This User Manual describes Mellanox Technologies ConnectX®-3 40/56 Gigabit Ethernet Sin- gle and Dual QSFP+ port PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.

  • Page 10: Technical Support

    The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Mellanox Technologies...

  • Page 11: Chapter 1 Introduction

    Rev 2.3 Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con- nectX®-3 EN integrated circuit device. These adapters connectivity provide the highest perform- ing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data...

  • Page 12: Features And Benefits

    Uses PCIe Gen 3.0 (1.1 and 2.0 compatible) through an x8 edge connector up to PCI Express (PCIe) 8GT/s Mellanox adapters comply with the following IEEE 802.3* standards: 10/40/56 Gigabit Ethernet IEEE Std 802.3-2008 Ethernet IEEE Std 802.3ae 10 Gigabit Ethernet IEEE Std 802.3ba 40 Gigabit Ethernet...

  • Page 13: Connectivity

    • VMware ESXi 4.x and 5.x Connectivity • Interoperable with InfiniBand or 10/40 Gb Ethernet switches. • Interoperable with 56GbE Mellanox Switches. • Passive copper cable with ESD protection • Powered connectors for optical and active cable support • QSFP to SFP+ connectivity through QSA module...

  • Page 14: Chapter 2 Interfaces

    The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations. The following lists the PCIe interface features: • PCIe Base 3.0 compliant, 1.1 and 2.0 compatible • 2.5, 5.0, or 8.0GT/s link rate x8 Mellanox Technologies...

  • Page 15: I2C-Compatible Interface

    “Mechanical Drawing of the Dual-port MCX314A-BCBT Adapter Card,” on page 49 the board. LED Interface There are two I/O LEDs per port. For LED specifications please refer to Figure 9, “Mechanical Drawing of the Dual-port MCX314A-BCBT Adapter Card,” on page 49 Mellanox Technologies...

  • Page 16: Chapter 3 Hardware Installation

    Please refer to Section 1.3, “Operating Systems/Distributions,” on page 12 3.1.3 Software Stacks Mellanox OpenFabric software package - MLNX_EN for Linux, WinOF for Windows and ESX 5.1 for VMware. See Chapter 4, “Driver Installation”. Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal.

  • Page 17: Removing The Existing Bracket

    Note: The following instructions relate to the installation of the gasket for adapter cards that are not shipped with a gasket already installed. Mellanox Technologies now provides all ConnectX- 3 adapter cards with the gasket already installed unto the bracket.

  • Page 18: 3.4.2.2 Installing The Bracket

    Do not force the bracket onto the card. You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket. 2. Screw on the bracket using the screws and washers saved from the bracket removal proce- dure above. Mellanox Technologies...

  • Page 19: Card Installation Instructions

    4. Applying even pressure at both corners of the card, insert the adapter card into the slot until it is firmly seated. When the adapter is properly seated, the adapter port connectors are aligned with the slot opening, and the adapter faceplate is visible against the system chassis. Mellanox Technologies...

  • Page 20: Cables And Modules

    5. Secure the adapter with the adapter clip or screw. 6. Close the system case. Cables and Modules To obtain the list of supported cables for your adapter, please refer to “Mellanox Products Approved Cable Lists” at: www.mellanox.com/related-docs/user_manuals/Mellanox_approved_cables.pdf 3.6.1 Cable Installation 1.

  • Page 21: Identify The Card In Your System

    6. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.

  • Page 22: On Linux

    Hardware Installation 3.7.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella- nox Technologies”: > lspci |grep -i Mellanox 27:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] Mellanox Technologies...

  • Page 23: Chapter 4 Driver Installation

    Driver Installation Linux Driver For Linux, download and install the latest MLNX_EN driver software package available via the Mellanox web site at: http://www.mellanox.com => Products => Software => Ethernet Drivers => ConnectX®-3 EN 10/40GigE Linux Driver => Download. Follow the installation instructions included in the download package (also available from the download page).

  • Page 24: Loading The Driver

    Uninstalling the Driver To uninstall the mlnx_en driver run: #> /sbin/mlnx_en_uninstall.sh Windows Driver For Windows, download and install the latest Mellanox WinOF VPI for Windows software pack- age available via the Mellanox web site at: http://www.mellanox.com => Products => Software =>...

  • Page 25: Hardware And Software Requirements

    (or a similar) error message: 4.2.3 Extracting Files Without Running Installation To extract the files without running installation, perform the following steps. Open a CMD console (Click Start-->Run and enter CMD). Step 1. Enter the following command: Step 2. MLNX_VPI_WinOF-<version>_All_<OS>_<arch>.exe /a Mellanox Technologies...
  • Page 26 Rev 2.3 Driver Installation Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Mellanox Technologies...

  • Page 27: Installing Mlnx_Winof

    Click Install to extract this folder, or click Change to install to a different folder. Step 5. To complete the extraction, click Finish. Step 6. 4.2.4 Installing MLNX_WinOF This section provides instructions for two types of installation procedures: • “Attended Installation” An installation procedure that requires frequent user intervention. • “Unattended Installation” Mellanox Technologies...

  • Page 28: 4.2.4.1 Attended Installation

    CMD console: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v"/l*vx [LogFile]" If you do not want to upgrade your firmware version, run the following command: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" MT_SKIPFWUPGRD=1" For further help, please run: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" /h" Click Next in the Welcome screen. Step 2. Mellanox Technologies...
  • Page 29 Select the target folder for the installation. Step 4. The firmware upgrade screen will be displayed in the following cases: Step 5. • If the user has an OEM card, in this case the firmware will not be updated. Mellanox Technologies...
  • Page 30 Driver Installation • If the user has a standard Mellanox card, and the firmware version is older than the one specified in WinOF Installation Guide 4.40, the firmware will be updated accordingly. However, if the user has both OEM card and Mellanox card, only Mellanox card will be updated.
  • Page 31 Band environment. • SDK - contains the libraries and DLLs for developing InfiniBand application over IBAL. • Documentation: contains the User Manual and Installation Guide. • ND FLTR DLLs: contains the files for standalone installation of the mlx4nd provider. Mellanox Technologies...
  • Page 32 Rev 2.3 Driver Installation Click Install to start the installation. Mellanox Technologies...

  • Page 33: 4.2.4.2 Unattended Installation

    4.2.4.2 Unattended Installation The following is an example of a MLNX_WinOF_win8 x64 unattended installation session. Open the CMD console (click Start > Run and enter ‘cmd’) Step 1. Install the driver. Run: Step 2. > MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /S /v"/qn" Mellanox Technologies...

  • Page 34: Upgrading Mlnx_Winof

    • In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade. 4.2.6 Installation Results Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. To see the Mellanox network adapter device, and the Ethernet or Mellanox Technologies...

  • Page 35: Opensm Activation

    Setting OpenSM upon setup results in it working only for the first port and not for the others.  To run OpenSM as a service, assuming the package was installed in the default path, use: sc create OpenSM1 binPath= "c:\Program Files\Mellanox\MLNX_VPI\IB\Tools\ opensm.exe --service" start=auto"  To start the service, run: sc start opensm For further information, please refer to the “OpenSM - Subnet Manager”...

  • Page 36: Uninstalling Mlnx_Winof

    Configuring a static IP is the same for both IPoIB and Ethernet adapters.  To assign a static IP address to a network port after installation: Open the Network Connections window. Locate Local Area Connections with Mellanox Step 1. devices.
  • Page 37 ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 2.3 Right-click a Mellanox Local Area Connection and left-click Properties. Step 2. Select Internet Protocol Version 4 (TCP/IPv4) from the scroll list and click Properties. Step 3. Mellanox Technologies...

  • Page 38: Performance Tuning

    Default Gateway ..: Performance Tuning For guidelines on improving network adapter performance, please refer to the performance tun- ing guidelines for a Linux/Windows environment in the document http://www.mellanox.com/ related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf. Mellanox Technologies...

  • Page 39: Vmware Driver

    ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 2.3 VMware Driver For VMware download and install the latest Mellanox OFED Driver for VMware® ESXi Server- software package available via the Mellanox web site at: http://www.mellanox.com => Products =>...

  • Page 40: Removing The Vib/Offline_Bundle Driver

    FlexBoot supports remote Boot over Ethernet. This technology is based on the Preboot Execution Environment (PXE) standard specification, and FlexBoot software is based on the open source iPXE project (see www.ipxe.org). For more information go to http://www.mellanox.com => Products => Software => Ethernet Drivers => Download. Mellanox Technologies...

  • Page 41: Chapter 5 Updating Adapter Card Firmware

    Firmware Tools. Make sure to download the package corresponding to your computer’s operating system. b. Enter: mst start. c. Get the Mellanox mst device name using the command "mst status". The mst device name will be of the form: /dev/mst/mt4099_pci_cr0.

  • Page 42: Figure 7: Support Download Assistant

    > flint -d /dev/mst/mt4099_pci_cr0 -i <binary image> burn a. Reboot the computer. b. Enter: mst start. c. Verify that the card firmware was updated successfully. > flint -d /dev/mst/mt4099_pci_cr0 Image type: ConnectX FW Version: 2.9.4100 Device ID: 4099 Mellanox Technologies...

  • Page 43: Chapter 6 Troubleshooting

    • Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but • Check that both the adapter and its link are set to the same speed with no and duplex settings communication established Mellanox Technologies...

  • Page 44: Linux

    | head -1 Information ifconfig –a ethtool <interface> ethtool –i <interface_of_Mellanox_port_num> ibdev2netdev Card Detection lspci | grep –i Mellanox Download and install MFT: http://www.mellanox.com/content/ pages.php?pg=management_tools&menu_section=34 Refer to the User Manual for installation instructions. Mellanox Firmware Tool (MFT) Once installed, run:...

  • Page 45: Windows

    Refer to the User Manual for installation instructions. Mellanox Firmware Once installed, open a CMD window and run: Tool (MFT) cd C:\Program Files\Mellanox\WinMFT mst start mst status flint –d <mst_device> q Ports Information vstat Download the latest firmware version using the PSID/board ID: Firmware Version http://www.mellanox.com/supportdownloader/...

  • Page 46: Chapter 7 Specifications

    Rev 2.3 Specifications Specifications MCX313A-BCBT Specifications Table 7 - MCX313A-BCBT Specifications Table Size: 2.12in. x5.6 in. (53.59mm x 142.25 mm) Connector: QSFP Physical Ethernet: 40GBASE-CR4 /-SR4/-LR4; also supports the following when used with a QSA: 10GBASE-CR/-SR/LR; 1000BASE-T/-SR/-LR Protocol Support Data Rate: 1/10/40Gb/s – Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible)

  • Page 47: Mcx314A-Bcbt Specifications

    For both operational and non-operational states b. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Adapter LED Operation There are two I/O LEDs per port. See for different LED functions. Table 9 Mellanox Technologies...

  • Page 48: Board Mechanical Drawing And Dimensions

    The short bracket has the same port and LED footprints as the tall bracket. Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 8: Mechanical Drawing of the Single-port MCX313A-BCBT Adapter Card J1 - I C Connector Flash Jumper 142.24...

  • Page 49: Figure 9: Mechanical Drawing Of The Dual-Port Mcx314A-Bcbt Adapter Card

    ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 2.3 Figure 9: Mechanical Drawing of the Dual-port MCX314A-BCBT Adapter Card Mellanox Technologies...

  • Page 50: Bracket Mechanical Drawing

    Rev 2.3 Specifications Bracket Mechanical Drawing Figure 10: Dual-port Tall Bracket Figure 11: Single-port Tall Bracket Mellanox Technologies...

  • Page 51: Figure 12: Dual-Port Short Bracket

    ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 2.3 Figure 12: Dual-port Short Bracket Port 1 LEDS Port 2 Figure 13: Single-port Short Bracket LEDs Gasket Port 1 Mellanox Technologies...

  • Page 52: Regulatory Statements

    Rev 2.3 Specifications Regulatory Statements For regulatory statements for all ConnectX®-3 cards please refer to: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.pdf Mellanox Technologies...

  • Page 53: Appendix A Interface Connectors Pinout

    Figure 14: Connector and Cage Views Table 10 - Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name TXN_2 Tx2n TXP_2 Tx2p TXN_4 Tx4n TXP_4 Tx4p ModSelL_Port0 ModSelL ResetL_Port0 ResetL VccRx Mellanox Technologies...
  • Page 54 TXP_3 Tx3p TXN_3 Tx3n TXP_1 Tx1p TXN_1 Tx1n PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3.0 specification. Mellanox Technologies...

  • Page 55: Figure 15: Pcie Connector Pinout

    ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 2.3 Figure 15: PCIe Connector Pinout C-compatible Connector Pinout Figure 16: Compatible Connector Plug and Pinout Connector Pin Number Signal Name SPSDA SPSCL Mellanox Technologies...

  • Page 56: Appendix B Finding The Guid/ Mac And Serial Number On The Adapter Card

    Appendix B: Finding the GUID/ MAC and Serial Number on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label: serial number and the card MAC for the Ethernet protocol. Figure 17: Card Product Label...
  • Page 57 Capability mask: 0x00010000 Port GUID: 0x0202c9fffe305351 Link layer: Ethernet flint -d /dev/mst/mt4099_pci_cr0 q Image type: ConnectX FW Version: 2.30.4260 Device ID: 4099 Description: Node Port1 Port2 Sys image GUIDs: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff MACs: 0002c9305350 0002c9305351 VSD: PSID: MT_1080120023 Mellanox Technologies...

  • Page 58: Appendix C Safety Warnings

    This equipment should be installed in compliance with local and national electrical codes. 8. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. Mellanox Technologies...
  • Page 59 ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 2.3 CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Mellanox Technologies...

  • Page 60: Appendix D Avertissements De Sécurité D'installation (Warnings In French)

    L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Mellanox Technologies...
  • Page 61 PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 62: Appendix E Sicherheitshinweise (Warnings In German)

    Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. This equipment should be installed in compliance with local and national elec- trical codes. Mellanox Technologies...
  • Page 63 Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen.. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 64: Appendix F Advertencias De Seguridad Para La Instalación (Warnings In Spanish)

    Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. 8. Exposición a niveles de radiación peligrosos Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. Mellanox Technologies...
  • Page 65 ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 2.3 PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Mellanox Technologies...

Table of Contents