Table of Contents
Advertisement
ConnectX®-2 VPI Single and Dual Port QSFP
InfiniBand and Ethernet Adapter Card User
Manual
P/N:
MHRH29B-XTR, MHRH29B-XSR, MHQH29B-XTR, MHQH29B-XSR, MHQH29C-XTR, MHQH29C-
XSR, MHRH19C-XTR, MHRH19C-XSR, MHQH19C-XTR, MHQH19C-XSR, MHQH19B-XSR,
MHQH19B-XTR, MHRH19B-XSR, MHRH19B-XTR
Rev 1.6
www.mellanox.com
Advertisement
Table of Contents
Subscribe to Our Youtube Channel
Related Manuals for Mellanox Technologies MHRH29B-XTR
Summary of Contents for Mellanox Technologies MHRH29B-XTR
- Page 1 ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual P/N: MHRH29B-XTR, MHRH29B-XSR, MHQH29B-XTR, MHQH29B-XSR, MHQH29C-XTR, MHQH29C- XSR, MHRH19C-XTR, MHRH19C-XSR, MHQH19C-XTR, MHQH19C-XSR, MHQH19B-XSR, MHQH19B-XTR, MHRH19B-XSR, MHRH19B-XTR Rev 1.6 www.mellanox.com...
- Page 2 Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniRISC®, InfiniScale®, InfiniPCI®, PhyX® CORE- Direct®, and Virtual Protocol Interconnect® are registered trademarks of Mellanox Technologies, Ltd. FabricIT™ and SwitchX™ are trademarks of Mellanox Technologies, Ltd. All other marks and names mentioned herein may be trademarks of their respective companies. Document Number: 3156...
-
Page 3: Table Of Contents
Inserting a Cable into the Adapter Card 3.4.3.2 Removing a Cable from the Adapter Card Chapter 4 Driver Software and Firmware 4.1 Driver Software 4.1.1 Linux 4.1.2 Windows 4.2 Port Type Management Via Driver SW 4.2.1 Linux 4.2.2 Windows Mellanox Technologies... - Page 4 C.1 Remove the Existing Bracket from the Adapter Card C.2 Installing the New Bracket Appendix D Avertissements de sécurité d’installation (Warnings in French) Appendix E Installation - Sicherheitshinweise (German) Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) Mellanox Technologies...
-
Page 5: List Of Figures
Figure 14: Schematic of the Single Port Adapter Card MH[QR]H19[BC]-X[ST]R Figure 15: Compatible Connector Plug and Pinout Figure 16: Connector and Cage Views Figure 17: Remove the Bracket Figure 18: Gasket Installation Figure 19: Placing the Bracket on the Card Mellanox Technologies... -
Page 6: List Of Tables
Table 17: Hardware Requirements Table 18: Specifications for MHQH19C-X[ST]R Table 19: Specifications for MHRH19B-X[ST]R Table 20: Specifications for MHQH19B-X[ST]R Table 21: Specifications for MHQH29C-X[ST]R Table 22: Specifications for MHRH29B-X[ST]R Table 23: Specifications for MHQH29B-X[ST]R Table 24: I2C-Compatible Connector Pinout Mellanox Technologies... -
Page 7: Revision History
Used graphics from media kit Changed regulatory section on 2 products Minor formatting New overview MAC is derived from GUID New template Apr. 2010 Updated power numbers Added VPI Feb. 2010 Combined singleand dual port cards Jan. 2010 Initial Release Mellanox Technologies... -
Page 8: About This Manual
Rev 1.6 About this Manual This User Manual describes Mellanox Technologies ConnectX®-2 Single and Dual Port VPI InfiniBand and Ethernet PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant docu- mentation. -
Page 9: Firmware And Software Updates
The Mellanox support downloader contains software, firmware and knowledge database informa- tion for Mellanox products. Access the data base from the Mellanox Support Web page, http://www.mellanox.com => Support or use the following link to go directly to the Mellanox Support Download Assistant page: http://www.mellanox.com/supportdownloader/. Mellanox Technologies... -
Page 10: Chapter 1 Overview
Rev 1.6 Overview Overview This document is a User Manual for Mellanox Technologies VPI adapter cards based on the Con- nectX®-2 VPI integrated circuit device. The cards described in this manual have the following main features: • Virtual Protocol Interconnect •... -
Page 11: Adapter Cards Covered In This Manual
20 Gb/s DDR Tall MT25408B0-FCCR-GI 5.0 GT/s 1 port MHQH19B-XSR PCIe Gen2 40 Gb/s QDR Short MT25408B0-FCCR-QI 5.0 GT/s 1 port MHQH19B-XTR PCIe Gen2 40 Gb/s QDR Tall MT25408B0-FCCR-QI 5.0 GT/s 1 port Figure 1: Single Port Card Component Side Mellanox Technologies... -
Page 12: Figure 2: Dual Port Card Component Side
Tall RoHS Part Number (OPN) SERDES Speed Bracket MHRH29B-XSR PCIe Gen2 20Gb/s (DDR) Short MT25408B0-FCCR-GI 5.0 GT/s Dual port MHRH29B-XTR PCIe Gen2 20Gb/s (DDR) Tall MT25408B0-FCCR-GI 5.0 GT/s Dual port MHQH29B-XSR PCIe Gen2 40Gb/s (QDR) Short MT25408B0-FCCR-QI 5.0 GT/s Dual port... -
Page 13: Mellanox Part Numbering Legend
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 Mellanox Part Numbering Legend Table 5 describes the Mellanox Technologies adapter cards part numbering legend. Table 5 - Mellanox Cards Part Numbering Key Adapter Card... -
Page 14: Finding The Guid/ Mac And Serial Number On The Adapter Cards
60. For safety warnings in German see “Installation - Sicherheitshinweise (German)” on page 62. For safety warnings in Spanish see “Advertencias de seguridad para la instalación (Warn- ings in Spanish)” on page 63. 1. Installation Instructions Read all installation instructions before connecting the equipment to the power source. Mellanox Technologies... - Page 15 This equipment should be installed, replaced, or serviced only by trained and qualified personnel. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Mellanox Technologies...
-
Page 16: Chapter 2 Adapter Card Interfaces
Port 2 2.1.1 InfiniBand Interface The ConnectX®-2 adapter card is compliant with the InfiniBand Architecture Specification, Release VPI adapter cards listed above, based on this device, provide access to its network ports by 1.2.1. means of QSFP connectors. Mellanox Technologies... -
Page 17: Ethernet Interface
2.1.4 PCI Express Interface The ConnectX®-2 adapter cards support PCI Express 2.0 (1.1 compatible) through an x8 edge connector. The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations. Mellanox Technologies... -
Page 18: Led Assignment
The short bracket has the same port and LED footprints as the tall bracket. 2.1.6 I C-compatible Interface A three-pin header on the adapter card is provided as the I C-compatible interface. See Appendix A,“Specifications,” on page 44 for the location on the board. Figure 6: I C Connector Mellanox Technologies... -
Page 19: Power
– Flash not present Figure 7: Flash Jumper 2.4.3 EEPROM Each board incorporates an EEPROM that is accessible through the I C-compatible interface. The EEPROM capacity is 4Kb. The EEPROM is used for storing the Vital Product Data (VPD). Mellanox Technologies... -
Page 20: Table 10: Mhqh[12]9B-Xtr
Length [15:8] MSB 0x00 VPD Keyword EFI Driver version Length Data STR_SPC VPD Keyword Asset Tag Length 0x20 Data STR_SPC “N/A” VPD Keyword Remaining read/write area Length 0x6E Data STR_ZERO Reserved (0x00) Small Resource Type END Tag 0x78 (0x11) Mellanox Technologies... -
Page 21: Table 11: Mhrh[12]9B-Xt[Cr]
Length [15:8] MSB 0x00 VPD Keyword EFI Driver version Length Data STR_SPC VPD Keyword Asset Tag Length 0x20 Data STR_SPC “N/A” VPD Keyword Remaining read/write area Length 0x6E Data STR_ZERO Reserved (0x00) Small Resource Type END Tag 0x78 (0x11) Mellanox Technologies... -
Page 22: Table 12: Mh[Qr]H19C-X[Stn]R Vpd
Length [15:8] MSB 0x00 VPD Keyword EFI Driver version Length Data STR_SPC VPD Keyword Asset Tag Length 0x20 Data STR_SPC “N/A” VPD Keyword Remaining read/write area Length 0x69 Data STR_ZERO Reserved (0x00) Small Resource Type END Tag 0x78 (0x11) Mellanox Technologies... -
Page 23: Table 13: Mhrh19B-X[St]R
Length [15:8] MSB 0x00 VPD Keyword EFI Driver version Length Data STR_SPC VPD Keyword Asset Tag Length 0x20 Data STR_SPC “N/A” VPD Keyword Remaining read/write area Length 0x6D Data STR_ZERO Reserved (0x00) Small Resource Type END Tag 0x78 (0x11) Mellanox Technologies... -
Page 24: Table 14: Mhqh19B-X[St]R
Length [15:8] MSB 0x00 VPD Keyword EFI Driver version Length Data STR_SPC VPD Keyword Asset Tag Length 0x20 Data STR_SPC “N/A” VPD Keyword Remaining read/write area Length 0x6D Data STR_ZERO Reserved (0x00) Small Resource Type END Tag 0x78 (0x11) Mellanox Technologies... -
Page 25: Table 15: Mhqh29B-X[St]R Vpd
Length [15:8] MSB 0x00 VPD Keyword EFI Driver version Length Data STR_SPC VPD Keyword Asset Tag Length 0x20 Data STR_SPC “N/A” VPD Keyword Remaining read/write area Length 0x6E Data STR_ZERO Reserved (0x00) Small Resource Type END Tag 0x78 (0x11) Mellanox Technologies... -
Page 26: Table 16: Mhqh29C-X[St]R Vpd
Length [15:8] MSB 0x00 VPD Keyword EFI Driver version Length Data STR_SPC VPD Keyword Asset Tag Length 0x20 Data STR_SPC “N/A” VPD Keyword Remaining read/write area Length 0x6E Data STR_ZERO Reserved (0x00) Small Resource Type END Tag 0x78 (0x11) Mellanox Technologies... -
Page 27: Chapter 3 Vpi Adapter Card Installation
Identify the Card in Your System 3.3.1 On Windows Open Device Manager on the server. Click start => Run, and then enter “devmgmt.msc”. Expand System Devices and locate your Mellanox ConnectX-2 adapter card. Select Properties to display the adapter card properties window. Mellanox Technologies... -
Page 28: Figure 8: Pci Device
In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_6368”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 0x6368 – this is a valid Mellanox Technologies PCI Device ID. -
Page 29: Linux
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi- tory at http://pci-ids.ucw.cz/read/PC/15b3. 3.3.1.1 Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella- nox Technologies”:... -
Page 30: Inserting A Cable Into The Adapter Card
Insert the connector into the adapter card. Be careful to insert the connector straight into the cage. Do not apply any torque, up or down, to the connector cage in the adapter card. Make sure that the connector locks in place. Mellanox Technologies... -
Page 31: Removing A Cable From The Adapter Card
Pull on the latch release mechanism thereby unlatching the connector and pull the connector out of the cage. Do not apply torque to the connector when removing it from the adapter card. Remove any cable supports that were used to support the cable’s weight. Mellanox Technologies... -
Page 32: Chapter 4 Driver Software And Firmware
100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: g Wake-on: d Link detected: yes Mellanox Technologies... - Page 33 Capability mask: 0x02510868 Port GUID: 0x0002c903000e9c5f Link layer: IB Port 2: State: Down Physical state: Polling Rate: 70 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510868 Port GUID: 0x0002c903000e9c60 Link layer: IB Check the OFED version Mellanox Technologies...
- Page 34 The “Usage: openibd {start|stop|restart|status}” command to modify this file and thereby control the drivers. Ethernet Driver Usage and Configuration To assign an IP address to the interface run: where 'n' is the OS assigned interface number. #> ifconfig eth<n> <ip> • To check driver and device information run: Mellanox Technologies...
-
Page 35: Windows
Drivers => InfiniBand & VPI Software/Drivers => Mellanox WinOF VPI => Download. Follow the installation instructions included in the download package (also available from the download page). To ensure that communication has been established follow the instructions below. Mellanox Technologies... -
Page 36: Port Type Management Via Driver Sw
InfiniBand Products => Mellanox OFED. Running "/sbin/connectx_port_config -s" will show the current port configuration for all ConnectX®-2 adapter cards. Port configuration is saved in the file: /etc/infiniband/connectx.conf. This saved configuration is restored at driver restart only if done via "/etc/init.d/openibd restart". Possible port types are: Mellanox Technologies... -
Page 37: Windows
After installing Mellanox WinOF VPI for Win- dows on your machine, you can change a port's protocol configuration. The following steps describe how to configure the port type: Display the Device Manager and expand “Network adapters”. Step 1 Mellanox Technologies... -
Page 38: Flexboot
Boot over Ethernet (BoE), or Boot over iSCSI (Bo-iSCSI). This technology is based on the Pre- boot Execution Environment (PXE) standard specification, and FlexBoot software is based on the open source EtherBoot/gPXE project (see www.etherboot.org). For more information go to Mellanox Technologies... -
Page 39: Nvidia Gpudirect Support
The MLNX driver is called MLNX_OFED_LINUX-Nvidia-1.5.1. - Mount the ISO file: # mount -o ro, loop MLNX_OFED_<version>.iso /mnt - Run the installation script: # /mnt/mlnxofed install - Restart the driver: # /etc/init.d/openibd restart - To make sure that GPUDirect is enabled, run: Mellanox Technologies... -
Page 40: Nvidia Driver Installation
=> Support => VPI SW/Driver. RDMA over Converged Ethernet (RoCE) ConnectX-2 with RoCE utilizes advances in Data Center Bridging (DCB) to enable efficient and low cost implementations of RDMA over Ethernet, supporting the entire breadth of RDMA and Mellanox Technologies... -
Page 41: Hardware And Software Requirements
The following steps describe how to retrieve the PSID (firmware identification) and programmed firmware version of your adapter card. They also describe how to update the card with the latest firmware version available. Retrieve the PSID and firmware version: Mellanox Technologies... -
Page 42: Figure 11: Support Download Assistant
Board ID: (MT_0E90110009) <--- This is the PSID Compare the programmed firmware version with the latest available. a. Go to Mellanox’s web site: http://www.mellanox.com/supportdownloader. See Figure 11 b. Enter your card PSID to display the latest firmware Figure 11: Support Download Assistant Mellanox Technologies... - Page 43 Unzip the firmware image c. Burn the firmware image. Enter: > flint -d <device_name> -i <binary image> burn d. Reboot the computer e. Enter: mst start f. Verify that the card firmware was updated successfully > flint -d /dev/mst/mt23108_pci_cr0 query Mellanox Technologies...
-
Page 44: Appendix A Specifications
RoHS-R6 a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. b. The auto-negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architec- ture Specification, Release 1.2. Mellanox Technologies... -
Page 45: Mhrh19B-X[St]R Specifications
RoHS-R6 a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. b. The auto-negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specification, Release 1.2. Mellanox Technologies... -
Page 46: Mhqh19B-X[St]R Specifications
RoHS-R6 a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. b. The auto-negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specification, Release 1.2. Mellanox Technologies... -
Page 47: Mhqh29C-X[St]R Specifications
RoHS-R6 a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. b. The auto-negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architec- ture Specification, Release 1.2. Mellanox Technologies... -
Page 48: Mhrh29B-X[St]R Specifications
RoHS-R6 a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. b. The auto-negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architec- ture Specification, Release 1.2. Mellanox Technologies... -
Page 49: Mhqh29B-X[St]R Specifications
RoHS-R6 a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. b. The auto-negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architec- ture Specification, Release 1.2. Mellanox Technologies... -
Page 50: Board Mechanical Drawing And Dimensions
Rev 1.6 Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/-0.1mm Figure 12: Schematic of the Dual Port Adapter Card MHQH29C Mellanox Technologies... -
Page 51: Figure 13: Schematic Of The Dual Port Adapter Card Mh[Rq]H29B-X[St]R
Figure 13: Schematic of the Dual Port Adapter Card MH[RQ]H29B-X[ST]R J1- I2C Connector J3 - Flash jumper 167.65 56.15 64.40 56.97 68.40 15.00 33.35 57.15 96.30 Figure 14: Schematic of the Single Port Adapter Card MH[QR]H19[BC]-X[ST]R J1 – I2C Connector J2 – Flash Jumper Mellanox Technologies... -
Page 52: Regulatory Statements
Rev 1.6 Regulatory Statements For regulatory statements for all ConnectX®-3 cards please refer to: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.pdf Mellanox Technologies... -
Page 53: Appendix B Interface Connectors Pinout
Connector Pin Number Signal Name SPSDA 5 1 2 3 4 SPSCL PCI Express x8 Connector Pinout These cards use a standard PCI Express x8edge connector and the PCI Express x8 standard pinout according to the PCI Express 2.0 specification. Mellanox Technologies... -
Page 54: Qsfp Connector Pinout
Table 25 - Connector Pin Name and Number to Signal Name Correspondence Connector Pin Number Connector Pin Name Port A Signal Name TXN_2 Tx2n TXP_2 Tx2p TXN_4 Tx4n TXP_4 Tx4p ModSelL_Port0 ModSelL ResetL_Port0 ResetL VccRx RXP_3 Rx3p RXN_3 Rx3n Mellanox Technologies... - Page 55 Connector Pin Number Connector Pin Name Port A Signal Name RXP_1 Rx1p RXN_1 Rx1n RXN_2 Rx2n RXP_2 Rx2p RXN_4 Rx4n RXP_4 Rx4p ModPrsl_Port0 Mod PrsL IntL IntL VccTx Vcc1 LPMode_Port0 LPMode TXP_3 Tx3p TXN_3 Tx3n TXP_1 Tx1p TXN_1 Tx1n Mellanox Technologies...
-
Page 56: Appendix C Replacing A Tall Bracket With A Short Bracket
To replace the bracket you will need the following parts: • the new bracket of the proper height • one new QSFP EMI gasket • the 2 screws saved from the removal of the bracket • the 2 fiber washers saved from the removal of the bracket Mellanox Technologies... -
Page 57: Remove The Existing Bracket From The Adapter Card
Save the two screws and the two fiber washers. Installing the New Bracket Remove the paper to expose the adhesive on the gasket. Place the gasket for each port onto the new bracket. Make sure to correctly align the gasket with the hole in the bracket. Mellanox Technologies... -
Page 58: Figure 18: Gasket Installation
Do not force the bracket onto the card. You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket. Screw on the bracket using the screws and washers saved from the procedure above. Mellanox Technologies... -
Page 59: Figure 19: Placing The Bracket On The Card
Figure 19: Placing the Bracket on the Card Gaskets in place on the bracket. Make sure that the LEDs are aligned onto the bracket holes. Use a torque driver to apply up to 2 lbs-in torque on the screw Mellanox Technologies... -
Page 60: Appendix D Avertissements De Sécurité D'installation (Warnings In French)
Ce matériel ne doit être installé, remplacé ou entretenu que par du personnel formé et qualifié. 6. Elimination du matériel L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. Mellanox Technologies... - Page 61 PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...
-
Page 62: Appendix E Installation - Sicherheitshinweise (German)
Diese Gerät sollte nur von geschultem und qualifiziertem Personal installiert, ausgetauscht oder gewartet werden. 5. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen. 6. Regionale und nationale elektrische Bestimmungen Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. Mellanox Technologies... -
Page 63: Appendix F Advertencias De Seguridad Para La Instalación (Warnings In Spanish)
5. Eliminación de equipos La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 6. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Mellanox Technologies...
Need help?
Do you have a question about the MHRH29B-XTR and is the answer not in the manual?
Questions and answers