Mellanox Technologies ConnectX-3 MCX312B-XCBT User Manual
  1. Manuals
  2. Brands
  3. Mellanox Technologies Manuals
  4. Computer Hardware
  5. ConnectX-3 MCX312B-XCBT
  6. User manual

Mellanox Technologies ConnectX-3 MCX312B-XCBT User Manual

Connectx-3 ethernet single and dual sfp+ port adapter card user manual
Hide thumbs
Table of Contents

Advertisement

Quick Links

ConnectX®-3 Ethernet Single and Dual SFP+
Port Adapter Card User Manual
P/N:
MCX312A-XCBT, MCX312B-XCBT, MCX311A-XCAT
Rev 2.3
www.mellanox.com

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ConnectX-3 MCX312B-XCBT and is the answer not in the manual?

Questions and answers

Related Manuals for Mellanox Technologies ConnectX-3 MCX312B-XCBT

Summary of Contents for Mellanox Technologies ConnectX-3 MCX312B-XCBT

  • Page 1 ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual P/N: MCX312A-XCBT, MCX312B-XCBT, MCX311A-XCAT Rev 2.3 www.mellanox.com...
  • Page 2 KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.

  • Page 3: Table Of Contents

    4.1 Linux Driver ..........24 4.1.1 Hardware and Software Requirements ......24 Mellanox Technologies...
  • Page 4 Appendix D Avertissements de sécurité d’installation (Warnings in French) 63 Appendix E Sicherheitshinweise (Warnings in German) ....65 Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 67 Mellanox Technologies...

  • Page 5: List Of Figures

    Figure 22: Board Labels for MCX312B-XCBT ........59 Mellanox Technologies...

  • Page 6: List Of Tables

    Software and Hardware Requirements 24 Table 6: Software and Hardware Requirements 26 Table 7: MCX311A-XCAT Specifications Table 47 Table 8: MCX312[AB]-XCBT Specifications Table 48 Table 9: Physical and Logical Link Indication 49 Table 10: SFP+ Connector Pinout 56 Mellanox Technologies...

  • Page 7: Revision History

    October 2012 • Removed all references to 40 Gigabit Ethernet from the User Manual August 2012 • Added operational and non-operational temperature and humid- ity level to the following tables: - Table 8, “MCX312[AB]-XCBT Specifications Table,” on page 48 Mellanox Technologies...
  • Page 8 Section 2.3, “I2C-compatible Inter- face,” on page 16 • Formatted specification tables in Appendix 7, “Specifications,” on page 47 October 2011 Added new OPNs based on ConnectX-3 Step A1 devices July 2011 Minor edits July 2011 0.10 First Release Mellanox Technologies...

  • Page 9: About This Manual

    Rev 2.3 About this Manual This User Manual describes Mellanox Technologies ConnectX®-3 10 Gigabit Ethernet Single and Dual SFP+ port PCI Express x4 or x8 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.

  • Page 10: Technical Support

    The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Mellanox Technologies...

  • Page 11: Chapter 1 Introduction

    Rev 2.3 Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con- nectX®-3 EN integrated circuit device. These adapters connectivity provide the highest perform- ing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data...

  • Page 12: Features And Benefits

    Standard block and Storage Acceleration file access protocols can leverage InfiniBand RDMA for high-performance stor- age access. Operating Systems/Distributions • Citrix XenServer 6.1 • Novell SLES, Red Hat Enterprise Linux(RHEL), and other Linux distributions Mellanox Technologies...

  • Page 13: Connectivity

    • Interoperable with InfiniBand or 10/40 Gb Ethernet switches. • Interoperable with 56GbE Mellanox Switches. • Passive copper cable with ESD protection • Powered connectors for optical and active cable support • QSFP to SFP+ connectivity through QSA module Mellanox Technologies...

  • Page 14: Chapter 2 Interfaces

    Rev 2.3 Interfaces Interfaces Each adapter card includes the following interfaces: • “Ethernet SFP+ Interface” • “PCI Express Interface” • “I2C-compatible Interface” • “LED Interface” Figure 1: MCX312A-XCBT Card Mellanox Technologies...

  • Page 15: Ethernet Sfp+ Interface

    The network ports of the ConnectX®-3 adapter cards are compliant with the IEEE 802.3 Ether- net standards listed in . The SFP+ port has one Tx/Rx pair of Table 4, “Features,” on page 12 SerDes. Ethernet traffic is transmitted through the cards' SFP+ connectors. Mellanox Technologies...

  • Page 16: Pci Express Interface

    Figure 13, “Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card,” on page 51 for the location on the board. LED Interface There are two I/O LEDs per port. For LED specifications please refer to Figure 13, “Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card,” on page 51 Mellanox Technologies...

  • Page 17: Chapter 3 Hardware Installation

    . If you need to replace it with the short bracket that is included in the Instructions,” on page 20 shipping box, please follow the instructions in this section. To replace the bracket you will need the following parts: • The new bracket of the proper height • One new gasket Mellanox Technologies...

  • Page 18: Removing The Existing Bracket

    Do not force the bracket onto the card. You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket. 2. Screw on the bracket using the screws and washers saved from the bracket removal proce- dure above. Mellanox Technologies...

  • Page 19: Figure 5: Placing The Bracket On The Card (Mcx312A-Xcbt)

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual Rev 2.3 Figure 5: Placing the Bracket on the Card (MCX312A-XCBT) Figure 6: Placing the Bracket on the Card (MCX312B-XCBT) Mellanox Technologies...

  • Page 20: Card Installation Instructions

    Do not use excessive force when seating the card, as this may damage the system or the adapter. 5. Secure the adapter with the adapter clip or screw. 6. Close the system case. Mellanox Technologies...

  • Page 21: Cables And Modules

    2. To insert a cable, press the connector into the port receptacle until the connector is firmly seated. When installing cables make sure that the latches engage. Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card. Mellanox Technologies...

  • Page 22: Identify The Card In Your System

    6. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.

  • Page 23: On Linux

    Rev 2.3 If the PCI device does not have a Mellanox adapter ID, return to Step 2 to check another device. The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi- tory at http://pci-ids.ucw.cz/read/PC/15b3. 3.7.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella-...

  • Page 24: Chapter 4 Driver Installation

    #> ./install.sh The package consists of several source RPMs. The install script rebuilds the source RPMs and then installs the created binary RPMs. The created kernel module binaries are placed under /lib/ modules/<kernel-ver>/updates/kernel/drivers/net/mlx4. mlnx_en installer supports 2 modes of Mellanox Technologies...

  • Page 25: Loading The Driver

    Uninstalling the Driver To uninstall the mlnx_en driver run: #> /sbin/mlnx_en_uninstall.sh Windows Driver For Windows, download and install the latest Mellanox WinOF VPI for Windows software pack- age available via the Mellanox web site at: http://www.mellanox.com => Products => Software Mellanox Technologies...

  • Page 26: Hardware And Software Requirements

    (or a similar) error message: 4.2.3 Extracting Files Without Running Installation To extract the files without running installation, perform the following steps. Open a CMD console (Click Start-->Run and enter CMD). Step 1. Enter the following command: Step 2. MLNX_VPI_WinOF-<version>_All_<OS>_<arch>.exe /a Mellanox Technologies...
  • Page 27 ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual Rev 2.3 Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Mellanox Technologies...

  • Page 28: Installing Mlnx_Winof

    Click Install to extract this folder, or click Change to install to a different folder. Step 5. To complete the extraction, click Finish. Step 6. 4.2.4 Installing MLNX_WinOF This section provides instructions for two types of installation procedures: • “Attended Installation” An installation procedure that requires frequent user intervention. • “Unattended Installation” Mellanox Technologies...

  • Page 29: 4.2.4.1 Attended Installation

    CMD console: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v"/l*vx [LogFile]" If you do not want to upgrade your firmware version, run the following command: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" MT_SKIPFWUPGRD=1" For further help, please run: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" /h" Click Next in the Welcome screen. Step 2. Mellanox Technologies...
  • Page 30 Select the target folder for the installation. Step 4. The firmware upgrade screen will be displayed in the following cases: Step 5. • If the user has an OEM card, in this case the firmware will not be updated. Mellanox Technologies...
  • Page 31 WinOF Installation Guide 4.40, the firmware will be updated accordingly. However, if the user has both OEM card and Mellanox card, only Mellanox card will be updated. Configure your system for maximum performance by checking the maximum performance Step 6. box. Mellanox Technologies...
  • Page 32 Band environment. • SDK - contains the libraries and DLLs for developing InfiniBand application over IBAL. • Documentation: contains the User Manual and Installation Guide. • ND FLTR DLLs: contains the files for standalone installation of the mlx4nd provider. Mellanox Technologies...
  • Page 33 ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual Rev 2.3 Click Install to start the installation. Mellanox Technologies...

  • Page 34: 4.2.4.2 Unattended Installation

    4.2.4.2 Unattended Installation The following is an example of a MLNX_WinOF_win8 x64 unattended installation session. Open the CMD console (click Start > Run and enter ‘cmd’) Step 1. Install the driver. Run: Step 2. > MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /S /v"/qn" Mellanox Technologies...

  • Page 35: Upgrading Mlnx_Winof

    • In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade. 4.2.6 Installation Results Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. To see the Mellanox network adapter device, and the Ethernet or Mellanox Technologies...

  • Page 36: Opensm Activation

     To run OpenSM as a service, assuming the package was installed in the default path, use: sc create OpenSM1 binPath= "c:\Program Files\Mellanox\MLNX_VPI\IB\Tools\ opensm.exe --service" start=auto"  To start the service, run: sc start opensm For further information, please refer to the “OpenSM - Subnet Manager” chapter in the User Manual. Mellanox Technologies...

  • Page 37: Uninstalling Mlnx_Winof

    1. Click Start-> Control Panel-> Programs and Features-> MLNX_VPI-> Uninstall. (NOTE: This requires elevated administrator privileges.) 2. Double click the .exe and follow the instructions of the install wizard. 3. Click Start-> All Programs-> Mellanox Technologies-> MLNX_WinOF-> Uninstall MLNX_WinOF. 4.2.8.2 Unattended Uninstall ...
  • Page 38 Rev 2.3 Driver Installation Right-click a Mellanox Local Area Connection and left-click Properties. Step 2. Select Internet Protocol Version 4 (TCP/IPv4) from the scroll list and click Properties. Step 3. Mellanox Technologies...

  • Page 39: Performance Tuning

    Default Gateway ..: Performance Tuning For guidelines on improving network adapter performance, please refer to the performance tun- ing guidelines for a Linux/Windows environment in the document http://www.mellanox.com/ related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf. Mellanox Technologies...

  • Page 40: Vmware Driver

    2. You can install the driver in one of the following ways: a. Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version. Run: #> esxcli software vib list #> esxcli software vib remove –n net-mlx4-en Mellanox Technologies...

  • Page 41: Removing The Vib/Offline_Bundle Driver

    FlexBoot supports remote Boot over Ethernet. This technology is based on the Preboot Execution Environment (PXE) standard specification, and FlexBoot software is based on the open source iPXE project (see www.ipxe.org). For more information go to http://www.mellanox.com => Products => Software => Ethernet Drivers => Download. Mellanox Technologies...

  • Page 42: Chapter 5 Updating Adapter Card Firmware

    Enter your card PSID to display the latest firmware file. The file name of the binary is composed by combin- ing the firmware name, the firmware release version, and the card part number. Note: Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card. Mellanox Technologies...

  • Page 43: Figure 11: Support Download Assistant

    > flint -d /dev/mst/mt4099_pci_cr0 -i <binary image> burn a. Reboot the computer. b. Enter: mst start. c. Verify that the card firmware was updated successfully. > flint -d /dev/mst/mt4099_pci_cr0 Image type: ConnectX FW Version: 2.9.4100 Device ID: 4099 Mellanox Technologies...

  • Page 44: Chapter 6 Troubleshooting

    • Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but • Check that both the adapter and its link are set to the same speed with no and duplex settings communication established Mellanox Technologies...

  • Page 45: Linux

    Mellanox Firmware Tool (MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo Firmware Version To download the latest firmware version refer to http://www.mellanox.com/supportdownloader Upgrade /var/log/messages Collect Log File dmesg > system.logF Mellanox Technologies...

  • Page 46: Windows

    –d <mst_device> –i <firmware_bin_file> b • Event log viewer • MST device logs: • mst start Collect log file • mst status • C:\Users\Administrator> flint –d <mst_device> dc > dump_configuration.log • C:\Users\Administrator> mstdump <mst_device> dc > mstdump.log Mellanox Technologies...

  • Page 47: Chapter 7 Specifications

    Please refer to www.mellanox.com => Products => Cables and Transceivers Cable Support a. For both operational and non-operational states b. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 48: Mcx312[Ab]A-Xcbt Specifications

    Please refer to www.mellanox.com => Products => Cables and Transceivers Cable Support a. For both operational and non-operational states b. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...

  • Page 49: Adapter Led Operation

    If neither LED is lit, then the physical link has not been established Yellow - logical (data • A blinking yellow indicates activity (data transfer) • Stays off when there is no activity activity link) The short bracket has the same port and LED footprints as the tall bracket. Mellanox Technologies...

  • Page 50: Board Mechanical Drawing And Dimensions

    Rev 2.3 Specifications Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 12: Mechanical Drawing of the Single-port MCX311A-XCAT Adapter Card 26.76 49.1 53.59 39.84 106.06 Mellanox Technologies...

  • Page 51: Figure 13: Mechanical Drawing Of The Dual-Port Mcx312A-Xcbt Adapter Card

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual Rev 2.3 Figure 13: Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card 142.24 55.14 64.4 68.9 45.95 Figure 14: Mechanical Drawing of the Dual-port MCX312B-XCBT 142.24 40.84 49.1 53.59 45.95 Mellanox Technologies...

  • Page 52: Bracket Mechanical Drawing

    Rev 2.3 Specifications Bracket Mechanical Drawing Figure 15: Dual-port Bracket for MCX312A-XCBT Mellanox Technologies...

  • Page 53: Figure 16: Dual-Port Bracket For Mcx312B-Xcbt

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual Rev 2.3 Figure 16: Dual-port Bracket for MCX312B-XCBT Figure 17: Single-port Bracket Mellanox Technologies...

  • Page 54: Regulatory Statements

    Rev 2.3 Specifications Regulatory Statements For regulatory statements for all ConnectX®-3 cards please refer to: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.pdf Mellanox Technologies...

  • Page 55: Appendix A Interface Connectors Pinout

    ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual Rev 2.3 Appendix A: Interface Connectors Pinout SFP+ Connector Pinout Figure 18: Rear View of Module With Pin Placement Mellanox Technologies...

  • Page 56: Table 10: Sfp+ Connector Pinout

    LOS is open collector output. Should be pulled up with 4.7kΩ – 10kΩ on host board to a voltage between 2.0V and 3.6V. Logic 0 indicates normal operation; logic 1 indicates loss of signal. Mellanox Technologies...

  • Page 57: Figure 19: Pcie Connector Pinout

    The adapter cards use a standard PCI Express x4 or x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3.0 specification. Figure 19: PCIe Connector Pinout C-compatible Connector Pinout Figure 20: Compatible Connector Plug and Pinout Mellanox Technologies...
  • Page 58 Rev 2.3 Connector Pin Number Signal Name SPSDA SPSCL Mellanox Technologies...

  • Page 59: Appendix B Finding The Guid/ Mac And Serial Number On The Adapter Card

    GUID via device firmware (e.g., using ibstat). Mlxburn/flint returns 0xffff as GUID while the utilities return a value derived from the MAC address. For all driver/firmware/software purposes, the latter value should be used. Please see the below example. Mellanox Technologies...
  • Page 60 Capability mask: 0x00010000 Port GUID: 0x0202c9fffe305351 Link layer: Ethernet flint -d /dev/mst/mt4099_pci_cr0 q Image type: ConnectX FW Version: 2.30.4260 Device ID: 4099 Description: Node Port1 Port2 Sys image GUIDs: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff MACs: 0002c9305350 0002c9305351 VSD: PSID: MT_1080120023 Mellanox Technologies...

  • Page 61: Appendix C Safety Warnings

    This equipment should be installed in compliance with local and national electrical codes. 8. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. Mellanox Technologies...
  • Page 62 Rev 2.3 CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Mellanox Technologies...

  • Page 63: Appendix D Avertissements De Sécurité D'installation (Warnings In French)

    L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Mellanox Technologies...
  • Page 64 PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 65: Appendix E Sicherheitshinweise (Warnings In German)

    Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. This equipment should be installed in compliance with local and national elec- trical codes. Mellanox Technologies...
  • Page 66 Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen.. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 67: Appendix F Advertencias De Seguridad Para La Instalación (Warnings In Spanish)

    Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. 8. Exposición a niveles de radiación peligrosos Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. Mellanox Technologies...
  • Page 68 Rev 2.3 PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Mellanox Technologies...

Table of Contents