Connectx-2 vpi dual port qsfp and sfp+ card (37 pages)
Summary of Contents for Mellanox Technologies ConnectX-3 Pro
Page 1
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project P/N: MCX345A-FCPN, MCX346A-FCPN Rev 1.3 www.mellanox.com...
Page 2
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.
Page 4
Appendix E Sicherheitshinweise (Warnings in German) ..... . . 65 Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) . 67 Mellanox Technologies...
• Table 7.2, “MCX346A-FCPN Specifications”. December 2015 • Updated Table 11, “Physical and Logical Link Indications (Ethernet Mode),” on page 59 • Added Table 12, “Physical and Logical Link Indications (Infini- Band Mode),” on page 59 November 2015 First Release Mellanox Technologies...
Rev 1.3 About this Manual This User Manual describes Mellanox Technologies ConnectX®-3 Pro VPI Single and Dual QSFP Port PCI Express x8 network interface cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant docu- mentation.
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Mellanox EN for Linux Driver Release Release notes for Mellanox Technologies' MLNX_EN for Linux Notes driver kit for Mellanox adapter cards: http://www.mellanox.com =>...
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Introduction This is the User Guide for Mellanox Technologies adapter cards based on the ConnectX®-3 Pro VPI integrated circuit device. These adapters’ connectivity provide the highest performing and...
Virtual Protocol management protocol to seamlessly operate over any converged network with the Interconnect (VPI) same software infrastructure. ConnectX-3 Pro delivers low latency, high bandwidth, and computing efficiency InfiniBand Architecture for performance-driven server and storage clustering applications. ConnectX-3 Specification v1.2.1 Pro is InfiniBand Architecture Specification v1.2.1 compliant.
Applications utilizing TCP/UDP/IP transport can achieve industry leading throughput over InfiniBand or 10/40/56GbE. The hardware-based stateless off- Sockets Acceleration load engines in ConnectX-3 Pro reduce the CPU overhead of IP packet transport. Sockets acceleration software further increases performance for latency sensitive applications.
• Support for MSI/MSI-X mechanisms C-compatible Interface A three hole footprint for I C harness is provided as the I2C-compatible interface. See Figure 1 and the Mechanical Drawing of the Dual-Port MCX346A-FCPN Adapter Card on page 58 the location on the board. Mellanox Technologies...
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Figure 1: I C Harness LED Interface There are two I/O LEDs per port. For LED specifications please refer to Section 7.4, “Adapter LED Operation,” on page Mellanox Technologies...
Please note that if the card is removed hastily from the antistatic bag, the plastic zip lock may harm the EMI fingers on the QSFP connector. Carefully remove the card Figure 2 from the antistatic bag to avoid damaging the EMI fingers. See Figure 3. Mellanox Technologies...
PCI Express Mezzanine connector. If the card is installed in a PCI Mezzanine connector with less lanes than the card requires, then the Mezzanine card will not provide the optimum data transfer Mellanox Technologies...
6. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
Rev 1.3 ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi- tory at http://pci-ids.ucw.cz/read/PC/15b3. 3.6.2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella-...
=> Products => Soft- ware => InfiniBand/VPI Drivers => Download. This chapter describes how to install and test the Mellanox OFED for Linux package on a single host machine with Mellanox ConnectX-3 Pro adapter hardware installed. Note: The shown versions and/or parameter values in the example below may not reflect the lat- est or actual values for this product, and are included here for illustration purposes only.
Example The following command will create a MLNX_OFED_LINUX ISO image for RedHat 5.6 under the /tmp directory. MLNX_OFED_LINUX-1.5.3-rhel5.6-x86_64/docs/mlnx_add_kernel_support.sh -i /mnt/MLNX- _OFED_LINUX-1.5.3-rhel5.6-x86_64.iso 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Mellanox Technologies...
Do you want to continue?[y/N]:y Removing OFED RPMs... Running mkisofs... Created /tmp/MLNX_OFED_LINUX-1.5.3-rhel5.6-x86_64.iso 4.1.3.2 Installation Script The usage of the installation script is described below. You will use it during the installation pro- cedure described in Section 4.1.3.4, “Installation Procedure,” on page 23 Mellanox Technologies...
Page 22
Ethernet -v|-vv|-vvv Set verbosity level Set quiet - no messages will be printed --umad-dev-rw Grant non root users read/write permission for umad devices instead of default --hugepages-overcommit Set 80% of MAX_MEMORY as overcommit for a huge page allocation Mellanox Technologies...
Failed to start the mst driver 4.1.3.4 Installation Procedure Login to the installation machine as root. Step 1. Mount the ISO image on your machine Step 2. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Mellanox Technologies...
Page 29
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Device (05:00.0): 05:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) Link Width is not 8x PCI Link Speed: 5Gb/s Device (07:00.0):...
4.1.4 Installation Results Software • The OFED and MFT packages are installed under the /usr directory. • The kernel modules are installed under: • InfiniBand subsystem: /lib/modules/`uname -r`/updates/kernel/drivers/infiniband/ • mlx4_core driver: /lib/modules/<kernel_version>/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellanox/ mlx4/mlx4_core.ko • mlx4_ib driver: /lib/modules/<kernel_version>/extra/mlnx-ofa_kernel/drivers/infiniband/hw/mlx4/mlx- 4_ib.ko Mellanox Technologies...
Page 31
• The file is installed under 90-ib.rules /etc/udev/rules.d/ • If OpenSM is installed, the daemon is installed under opensmd /etc/init.d/ is installed under opensm.conf /etc • If IPoIB configuration files are included, files will be installed under: ifcfg-ib<n> Mellanox Technologies...
If you need to burn an Expansion ROM image, please refer to OFED User Manual. Table 2, “Documents List,” on page 8. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site ( => Support => Firm- ware Download). Mellanox Technologies...
-dev /dev/mst/mt25418_pci_cr0 -fw /mnt/firmware/fw-25408/fw-25408- rel.mlx Reboot your machine after the firmware burning is completed. 4.1.7 Uninstalling Mellanox OFED Use the script to uninstall the Mellanox OFED package. /usr/sbin/ofed_uninstall.sh The script is part of the RPM. ofed-scripts Mellanox Technologies...
For example, if the first port is connected to an InfiniBand switch and the second to Ethernet switch, the NIC will automatically load the first switch as InfiniBand and the second as Ethernet. 4.1.9.1 Enabling Auto Sensing Upon driver start up: 1. Sense the adapter card’s port type: Mellanox Technologies...
Enter the following command: > echo %PROCESSOR_ARCHITECTURE% On an x64 (64-bit) machine, the output will be “AMD64”. Go to the MLNX_WinOF for Windows Web page at Step 2. http://www.mellanox.com => Products => Software => Ethernet Drivers => Windows SW/ Drivers. Mellanox Technologies...
To extract the files without running installation, perform the following steps. Open a CMD console (Click Start-->Run and enter CMD). Step 1. Enter the following command: Step 2. MLNX_VPI_WinOF-<version>_All_<OS>_<arch>.exe /a Click Next to create a server image. Step 3. Mellanox Technologies...
Page 37
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Click Change and specify the location in which the files are extracted to. Step 4. Click Install to extract this folder, or click Change to install to a different folder. Step 5. Mellanox Technologies...
To configure your setup to contain the logs option, please run the following command after opening a CMD console: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v"/l*vx [LogFile]" If you do not want to upgrade your firmware version, run the following command: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" MT_SKIPFWUPGRD=1" For further help, please run: MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /v" /h" Mellanox Technologies...
Page 39
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Click Next in the Welcome screen. Step 2. Read then accept the license agreement and click Next. Step 3. Mellanox Technologies...
Page 40
If the user has a standard Mellanox card, and the firmware version is older than the one specified in WinOF Installation Guide 4.40, the firmware will be updated accordingly. However, if the user has both OEM card and Mellanox card, only Mellanox card will be updated. Mellanox Technologies...
Page 41
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Configure your system for maximum performance by checking the maximum performance Step 6. box. This step requires rebooting your machine at the end of the installation. Mellanox Technologies...
Page 42
• SDK - contains the libraries and DLLs for developing Ethernet application over IBAL. • Documentation: contains the User Manual and Installation Guide. • ND FLTR DLLs: contains the files for standalone installation of the mlx4nd provider. Mellanox Technologies...
Page 43
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 Click Install to start the installation. Mellanox Technologies...
4.2.4.2 Unattended Installation The following is an example of a MLNX_WinOF_win8 x64 unattended installation session. Open the CMD console (click Start > Run and enter ‘cmd’) Step 1. Install the driver. Run: Step 2. > MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe /S /v"/qn" Mellanox Technologies...
Upon installation completion, you can verify the successful addition of the network card(s) through the Device Manager. To see the Mellanox network adapter device, and the Ethernet or IPoIB network device (depending on the used card) for each port, display the Device Manager and expand “System devices” or “Network adapters”. Mellanox Technologies...
1. Click Start-> Control Panel-> Programs and Features-> MLNX_VPI-> Uninstall. (NOTE: This requires elevated administrator privileges.) 2. Double click the .exe and follow the instructions of the install wizard. 3. Click Start-> All Programs-> Mellanox Technologies-> MLNX_WinOF-> Uninstall MLNX- _WinOF. 4.2.7.2 Unattended Uninstall ...
Open the Network Connections window. Locate Local Area Connections with Mellanox Step 1. devices. Right-click a Mellanox Local Area Connection and left-click Properties. Step 2. Select Internet Protocol Version 4 (TCP/IPv4) from the scroll list and click Properties. Step 3. Mellanox Technologies...
Default Gateway ..: Performance Tuning For guidelines on improving network adapter performance, please refer to the performance tun- ing guidelines for a Linux/Windows environment in the document http://www.mellanox.com/ related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf. Mellanox Technologies...
Preboot Execution Environment (PXE) standard specification, and FlexBoot soft- ware is based on the open source iPXE project (see www.ipxe.org). For more information go to http://www.mellanox.com => Products => Software => InfiniBand/VPI Drivers => Download. Mellanox Technologies...
Enter your card PSID to display the latest firmware file. The file name of the binary is composed by combin- ing the firmware name, the firmware release version, and the card part number. Note: Please contact Mellanox System Support if you cannot find the firmware binary for your card. Mellanox Technologies...
• Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but • Check that both the adapter and its link are set to the same speed with no and duplex settings communication established Mellanox Technologies...
Mellanox Firmware Tool (MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo Firmware Version To download the latest firmware version refer to Upgrade http://www.mellanox.com/supportdownloader /var/log/messages Collect Log File dmesg > system.logF Mellanox Technologies...
=> Products => Cables and Transceivers Cable Support a. The ConnectX-3 Pro adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mella- nox InfiniBand product.
=> Products => Cables and Transceivers Cable Support a. The ConnectX-3 Pro adapters supplement the IBTA auto-negotiation specification to get better bit error rates and longer cable reaches. This supplemental feature only initiates when connected to another Mella- nox InfiniBand product.
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm Figure 6: Mechanical Drawing of the Dual-Port MCX346A-FCPN Adapter Card I2C connector Mellanox Technologies...
A constant Green indicates a valid logical (data activity) link with- out data transfer. link/activity Port 0: • A blinking Green indicates a valid logical link with data transfer. LED1=D3, LED0=D1 • If LED1 is off, then the logical link has not been established. Mellanox Technologies...
Appendix A: Interface Connectors Pinout PCI Express x8 Connector Pinout The cards use a standard PCI Express x8 connector and the PCI Express x8 standard pinout according to the PCI Express 3.0 specification. Figure 7: PCIe Connector Pinout Mellanox Technologies...
ConnectX®-3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual for Open Compute Project Rev 1.3 C-compatible Connector Pinout A three hole footprint for I2C Harness is provided as the I2C-compatible interface. Figure 8: Compatible Connector Plug and Pinout Table 13 - I2C-compatible Connector Connector Pin Signal Name Number Pinout Mellanox Technologies...
Each Mellanox adapter card has a different identifier printed on the label: serial numberand the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID). Figure 9: MCX345A-FCPN Board Label Figure 10: MCX346A-FCPN Board Label Mellanox Technologies...
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...
PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...
Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. 7. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen.. Mellanox Technologies...
Page 66
Rev 1.3 Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Mellanox Technologies...
Need help?
Do you have a question about the ConnectX-3 Pro and is the answer not in the manual?
Questions and answers