10gb/s ethernet single and dual sfp+ port network interface card (57 pages)
Summary of Contents for Mellanox Technologies MCX353A-FCCT
Page 1
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual P/N: MCX353A-FCCT, MCX354A-FCCT Rev 1.5 www.mellanox.com...
Page 2
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.
Page 5
Appendix E Sicherheitshinweise (Warnings in German) ..... . . 76 Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) . 78 Mellanox Technologies...
Table 1 - Revision History Table Date Comments/Changes December 2015 • Updated non-operational temperature in the following specifica- tions tables: Section 7.1,“MCX353A-FCCT Specifications” on page • • Section 7.2,“MCX354A-FCCT Specifications” on page 61 October 2015 • Updated Device ID. See Product Overview on page 11.
Rev 1.5 About this Manual This User Manual describes Mellanox Technologies ConnectX®-3 Pro VPI Single and Dual QSFP+ port PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documenta- tion.
The Mellanox support downloader contains software, firmware and knowledge database infor- mation for Mellanox products. Access the database from the Mellanox Support web page, http://www.mellanox.com => Support or use the following link to go directly to the Mellanox Support Download Assistant page, http://www.mellanox.com/supportdownloader/. Mellanox Technologies...
Rev 1.5 Introduction This is the User Guide for Mellanox Technologies VPI adapter cards based on the ConnectX®-3 Pro VPI integrated circuit device. These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers, High-Performance Computing, and Embedded environments.
Adapter functionality enabling reduced CPU overhead allowing more available CPU offload Using GPUDirect RDMA, adapters can directly read and write CUDA host and GPUDirect RDMA device memory, eliminating unnecessary system memory copies and CPU over- head, resulting in significant performance improvements. Mellanox Technologies...
• Interoperable with 56GbE Mellanox switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support • QSFP+ to SFP+ connectivity through QSA module • Interoperable with 40/56 Gb/s InfiniBand • Passive copper cable with ESD protection Mellanox Technologies...
Ethernet standards listed in Table 4, “Features,” on page 12. Ethernet traffic is transmitted through the cards' QSFP+ connectors. For connecting to an SFP+ interface, you can use Mella- nox QSA (QSFP to SFP+) adapter module (to be ordered separately). Mellanox Technologies...
“Mechanical Drawing of the Dual-port MCX354A-FCCT Adapter Card,” on page 65 for the location on the board. LED Interface There are two I/O LEDs per port. For LED specifications please refer to Section 7.3, “Adapter Led Operations,” on page Mellanox Technologies...
Note: Due to risk of damaging the EMI gasket, it is not recommended to replace the bracket more than three times. To replace the bracket you will need the following parts: Mellanox Technologies...
Installing the New Bracket 1. Place the bracket onto the card until the screw holes line up. Please make sure to insert the bracket until the end of the EMI fingers. See Figure Figure 3: EMI Fingers (Example) Mellanox Technologies...
A lesser width adapter can be seated into a greater width slot (x4 in a x8), but a greater width adapter cannot be seated into a lesser width slot (x8 in a x4). Align the adapter connector edge with the PCI Express connector slot. Mellanox Technologies...
Section 7.3, “Adapter Led Operations,” on page 4. After plugging in a cable, lock the connector using the latching mechanism particular to the cable vendor. When a logical connection is made the Yellow LED will light. When data is Mellanox Technologies...
1. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
Rev 1.5 If the PCI device does not have a Mellanox adapter ID, return to Step 2 to check another device. The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi- http://pci-ids.ucw.cz/read/PC/15b3 tory at 3.7.2...
ConnectX entries in the display. The following example shows a system with an installed Mellanox HCA: host1# lspci -v | grep Mellanox 02:00.0 InfiniBand: Mellanox Technologies MT27520 [ConnectX-3 Pro] Download the ISO image to your host. Step 2.
-m|--mlnx_ofed <path to MLNX_OFED >directory> [--make-iso|--make-tgz] > >[--make-iso] Create MLNX_OFED ISO image. >[--make-tgz] Create MLNX_OFED tarball. (Default) >[-t|--tmpdir <local work dir>] >[--kmp] >[-v|--verbose] 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Mellanox Technologies...
Removing OFED RPMs... Running mkisofs... Created /tmp/MLNX_OFED_LINUX-1.5.3-rhel5.6-x86_64.iso 4.1.3.2 Installation Script The usage of the installation script is described below. You will use it during the installation pro- cedure described in Section 4.1.3.4, “Installation Procedure,” on page Usage ./mnt/mlnxofedinstall [OPTIONS] Mellanox Technologies...
Set verbosity level Set quiet - no messages will be printed --umad-dev-rw Grant non root users read/write permission for umad devices instead of default --hugepages-overcommit Set 80% of MAX_MEMORY as overcommit for a huge page allocation 4.1.3.3 mlnxofedinstall Return Codes Mellanox Technologies...
Failed to start the mst driver 4.1.3.4 Installation Procedure Login to the installation machine as root. Step 1. Mount the ISO image on your machine Step 2. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Mellanox Technologies...
Page 32
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Device (05:00.0): 05:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) Link Width is not 8x PCI Link Speed: 5Gb/s Device (07:00.0): 07:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]...
/etc/infiniband/info 4.1.4 Installation Results Software • The OFED and MFT packages are installed under the /usr directory. • The kernel modules are installed under: • InfiniBand subsystem: /lib/modules/`uname -r`/updates/kernel/drivers/infiniband/ • mlx4_core driver: /lib/modules/<kernel_version>/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellanox/ mlx4/mlx4_core.ko Mellanox Technologies...
Page 34
The file saves the ConnectX openibd connectx.conf adapter card’s ports configuration to Ethernet and/or InfiniBand. This file is used at driver start/restart ( /etc/init.d/openibd start • The file is installed under 90-ib.rules /etc/udev/rules.d/ Mellanox Technologies...
(manually) update firmware on your adapter card(s), you need to perform the following steps: If you need to burn an Expansion ROM image, please refer to OFED User Manual. Table 2, “Documents List,” on page 9. Mellanox Technologies...
Page 36
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 The following steps are also appropriate in case you wish to burn newer firmware that http://www.mella- you have downloaded from Mellanox Technologies’ Web site ( nox.com => Support => Firmware Download). Start Step 1.
(InfiniBand or Ethernet). For example, if the first port is connected to an InfiniBand switch and the second to Ethernet switch, the NIC will automatically load the first switch as InfiniBand and the second as Ethernet. Mellanox Technologies...
Enter the following command: > echo %PROCESSOR_ARCHITECTURE% On an x64 (64-bit) machine, the output will be “AMD64”. Go to the MLNX_WinOF for Windows Web page at Step 2. http://www.mellanox.com => Products => Software => InfiniBand/VPI Drivers => Windows SW/Drivers. Mellanox Technologies...
To extract the files without running installation, perform the following steps. Open a CMD console (Click Start-->Run and enter CMD). Step 1. Enter the following command: Step 2. MLNX_VPI_WinOF-<version>_All_<OS>_<arch>.exe /a Click Next to create a server image. Step 3. Mellanox Technologies...
Page 40
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Click Change and specify the location in which the files are extracted to. Step 4. Click Install to extract this folder, or click Change to install to a different folder. Step 5. Mellanox Technologies...
To configure your setup to contain the logs option, please run the following command after opening a CMD console: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v"/l*vx [LogFile]" If you do not want to upgrade your firmware version, run the following command: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" MT_SKIPFWUPGRD=1" For further help, please run: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" /h" Mellanox Technologies...
Page 42
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Click Next in the Welcome screen. Read then accept the license agreement and click Next. Step 2. Mellanox Technologies...
Page 43
If the user has a standard Mellanox card, and the firmware version is older than the one specified in WinOF Installation Guide 4.40, the firmware will be updated accordingly. However, if the user has both OEM card and Mellanox card, only Mellanox card will be updated. Mellanox Technologies...
Page 44
Configure your system for maximum performance by checking the maximum performance Step 5. box. This step requires rebooting your machine at the end of the installation. Select a Complete or Custom installation, follow Step a on page 45. Step 6. Mellanox Technologies...
Page 45
Band environment. • SDK - contains the libraries and DLLs for developing InfiniBand application over IBAL. • Documentation: contains the User Manual and Installation Guide. • ND FLTR DLLs: contains the files for standalone installation of the mlx4nd provider. Mellanox Technologies...
Page 46
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Click Install to start the installation. Click Finish to complete the installation. Step 7. Mellanox Technologies...
Windows 2012 driver do not completely uninstall the previous version. • In Windows 2012 (MLNX_WinOF Rev. 4.2 and above), the network configuration is saved upon driver upgrade. • In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade. Mellanox Technologies...
To run OpenSM as a service, assuming the package was installed in the default path, use: sc create OpenSM1 binPath= "c:\Program Files\Mellanox\MLNX_VPI\IB\Tools\ opensm.exe --service" start=auto" To start the service, run: sc start opensm Mellanox Technologies...
1. Click Start => Control Panel => Programs and Features => MLNX_VPI => Uninstall. (NOTE: This requires elevated administrator privileges.) 2. Double click the .exe and follow the instructions of the install wizard. 3. Click Start => All Programs => Mellanox Technologies => MLNX_WinOF => Uninstall MLNX_WinOF. 4.2.7.2 Unattended Uninstall ...
Page 50
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Right-click a Mellanox Local Area Connection and left-click Properties. Step 2. Select Internet Protocol Version 4 (TCP/IPv4) from the scroll list and click Properties. Step 3. Mellanox Technologies...
After installing Mellanox WinOF VPI for Windows on your machine, you can change a port's pro- tocol configuration. The following steps describe how to configure the port type: Step 1: Display the Device Manager and expand “System Devices”. Mellanox Technologies...
Page 52
Select the Port Protocol tab from the Properties sheet. Note: The “Port Protocol” tab is displayed only if the NIC is a VPI (IB and ETH). If the NIC is either only IB or ETH, the tab will not be shown. Mellanox Technologies...
Preboot Execution Environment (PXE) standard specification, and FlexBoot soft- ware is based on the open source iPXE project (see www.ipxe.org). For more information go to http://www.mellanox.com => Products => Software => InfiniBand/VPI Drivers => Download. Mellanox Technologies...
Enter your card PSID to display the latest firmware file. The file name of the binary is composed by combin- ing the firmware name, the firmware release version, and the card part number. Note: Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card. Mellanox Technologies...
• Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but • Check that both the adapter and its link are set to the same speed with no and duplex settings communication established Mellanox Technologies...
Mellanox Firmware Tool (MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo To download the latest firmware version refer to Firmware Version http://www.mellanox.com/supportdownloader Upgrade /var/log/messages Collect Log File dmesg > system.logF Mellanox Technologies...
=> Products => Cables and Transceivers Cable Support a. The ConnectX-3 Pro adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mella- nox InfiniBand product. Mellanox Technologies...
Page 62
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 b. For both operational and non-operational states. c. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet. Mellanox Technologies...
If neither LED is lit, then the physical link has not been established Yellow - logical (data • A blinking yellow indicates activity (data transfer) • Stays off when there is no activity activity link) The short bracket has the same port and LED footprints as the tall bracket. Mellanox Technologies...
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 7: Mechanical Drawing of the Single-port MCX353A-FCCT Adapter Card J1 - I C Connector Flash Jumper 142.24 40.5...
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Figure 8: Mechanical Drawing of the Dual-port MCX354A-FCCT Adapter Card Mellanox Technologies...
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Bracket Mechanical Drawing Figure 9: Dual-port Bracket Mellanox Technologies...
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Figure 10: Single-port Bracket Regulatory Statements For regulatory statements for all ConnectX®-3 Pro cards please refer to: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.pdf Mellanox Technologies...
Figure 11: Connector and Cage Views Table 13 - Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name TXN_2 Tx2n TXP_2 Tx2p TXN_4 Tx4n TXP_4 Tx4p ModSelL_Port0 ModSelL ResetL_Port0 ResetL VccRx Mellanox Technologies...
TXP_3 Tx3p TXN_3 Tx3n TXP_1 Tx1p TXN_1 Tx1n PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3.0 specification. Mellanox Technologies...
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 Figure 12: PCIe Connector Pinout C-compatible Connector Pinout Figure 13: Compatible Connector Plug and Pinout Connector Pin Number Signal Name SPSDA SPSCL Mellanox Technologies...
Figure 14: Card Product Label Port 1 uses the GUID or MAC described on the label. To obtain the GUID or MAC for Port 2 on dual port cards, add 1 to the GUID or MAC of Port 1. Mellanox Technologies...
This equipment should be installed in compliance with local and national electrical codes. 8. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. Mellanox Technologies...
Page 73
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Mellanox Technologies...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Mellanox Technologies...
Page 75
PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...
Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. This equipment should be installed in compliance with local and national elec- trical codes. Mellanox Technologies...
Page 77
Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen.. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...
Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. 8. Exposición a niveles de radiación peligrosos Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. Mellanox Technologies...
Page 79
ConnectX®-3 Pro VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.5 PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Mellanox Technologies...
Need help?
Do you have a question about the MCX353A-FCCT and is the answer not in the manual?
Questions and answers