Mellanox Technologies ConnectX-3 User Manual
  1. Manuals
  2. Brands
  3. Mellanox Technologies Manuals
  4. Adapter
  5. ConnectX-3 Pro
  6. User manual
Mellanox Technologies ConnectX-3 User Manual

Mellanox Technologies ConnectX-3 User Manual

10/40 gigabit ethernet adapters for dell poweredge servers
Hide thumbs Also See for ConnectX-3:
Table of Contents

Advertisement

Quick Links

®
User Manual for Mellanox ConnectX
-3 10/40
Gigabit Ethernet Adapters for Dell PowerEdge
Servers
Rev 1.1
www.mellanox.com

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ConnectX-3 and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Mellanox Technologies ConnectX-3

Summary of Contents for Mellanox Technologies ConnectX-3

  • Page 1 ® User Manual for Mellanox ConnectX -3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 www.mellanox.com...
  • Page 2 KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.

  • Page 3: Restrictions And Disclaimers

    Mellanox Technologies...

  • Page 4: Table Of Contents

    4.1 Linux Driver ..........21 Mellanox Technologies...
  • Page 5 4.3.2 Installing and Running the offline_bundle Driver on ESXi-5.x ..57 4.3.3 Removing the VIB/offline_bundle Driver......57 Mellanox Technologies...
  • Page 6 Chapter 9 Regulatory..........91 Mellanox Technologies...
  • Page 7 Appendix C Avertissements de sécurité d’installation (Warnings in French) 110 Appendix D Sicherheitshinweise (Warnings in German) ....112 Appendix E Advertencias de seguridad para la instalación (Warnings in Spanish) 114 Mellanox Technologies...

  • Page 8: List Of Tables

    Mellanox ConnectX-3 Dual 40GbE QSFP+ Network Adapter Specifications ..88 Table 17: Mellanox ConnectX-3 Dual 10GbE SFP+ Network Adapter Specifications ..89 Table 18: Mellanox ConnectX-3 Dual 10GbE KR Blade Mezzanine Card Specifications .

  • Page 9: List Of Figures

    Mellanox ConnectX-3 Dual Port 40GbE QSFP+ Network Adapter Full Height Bracket 16 Figure 2: Mellanox ConnectX-3 Dual Port 10GbE SFP+ Network Adapter Full Height Bracket . 17 Figure 3: Life Cycle Controller Main Configuration System Setup Menu ....94 Figure 4: Main Configuration Page Options .

  • Page 10: Revision History

    Added Section 4.2, “Linux Driver Features,” on page 25 • Added Section 4.5, “WinOF Features,” on page 59 • Added Section 5, “Remote Boot,” on page 65 • Added Appendix A, “Configuration for Mellanox Adapters through System Setup,” on page 94 November 2013 Initial Release Mellanox Technologies...

  • Page 11: About This Manual

    Rev 1.1 About this Manual ® This User Manual describes Mellanox Technologies ConnectX -3 10/40 Gigabit Ethernet Adapter and Cards for Dell PowerEdge Servers. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant docu- mentation.

  • Page 12: Chapter 1 Introduction

    • End-to-end QoS and congestion control • Ethernet • IEEE 802.3ae 10 Gigabit Ethernet • IEEE 802.3ba 40 Gigabit Ethernet supported on Mellanox ConnectX-3 Dual Port 40GbE QSFP+ Network Adapter only • IEEE 802.3ad Link Aggregation and Failover IEEE 802.1Q, 802.1p VLAN tags and priority •...

  • Page 13: Single Root Io Virtualization (Sr-Iov)

    User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 • 128 MAC/VLAN addresses per port • Wake on LAN (WoL) supported on Mellanox ConnectX-3 Dual Port 10GbE KR Blade Mezzanine Card only • PCI Express Interface • PCIe Base 3.0 compliant, 1.1 and 2.0 compatible •...

  • Page 14: Remote Direct Memory Access (Rdma)

    • Microsoft Windows Server Family of Operating Systems • VMware ESX For the list of the specific supported operating systems and distributions, please refer to http:// the release notes for the applicable software downloads on the Dell support site: www.dell.com/support Mellanox Technologies...

  • Page 15: Chapter 2 Adapter Card Interfaces

    • Backplane connection to the M1000e chassis for the 10GbE KR Blade Mezzanine Card • PCI Express (PCIe) x8 edge connector • I/O panel LEDs (does not apply with Mellanox ConnectX-3 Dual Port 10GbE KR Blade Mezzanine Card) 2.1.1 Ethernet QSFP+/ SFP+ Interface Note: This section does not apply to Mellanox ConnectX-3 Dual Port 10GbE KR Blade Mezza- nine Card.

  • Page 16: Table 4: Led Assignment For 40Gbe Qsfp+ Network Adapters

    10 Gb/s Link Speed is only supported with the Mellanox Quad to Serial Small Form Factor Pluggable Adapter (QSFP+ to SFP+ adapter or QSA). The QSA is not currently supported. Figure 1: Mellanox ConnectX-3 Dual Port 40GbE QSFP+ Network Adapter Full Height Bracket Port 2 Link...

  • Page 17: Figure 2: Mellanox Connectx-3 Dual Port 10Gbe Sfp+ Network Adapter Full Height Bracket

    User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Figure 2: Mellanox ConnectX-3 Dual Port 10GbE SFP+ Network Adapter Full Height Bracket Port 1 Link Port 2 Link Port 2 Activity Port 1 Activity...

  • Page 18: Chapter 3 Installing The Hardware

    1. Remove any metallic objects from your hands and wrists. 2. Make sure to use only insulated tools. 3. Verify that the system is powered off and is unplugged. 4. It is required to use an ESD strap or other antistatic devices. Mellanox Technologies...

  • Page 19: Pre-Installation Checklist

    Refer to the owner's manuals that were supplied with your Dell PowerEdge Blade Server for instructions on installing blade mezzanine cards. 1. Before installing the card, take the blade server out of the chassis 2. Open the system case. 3. The card can be placed in an available slot. Mellanox Technologies...

  • Page 20: Connecting The Network Cables

    Identifying the Card in A System 3.6.1 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mella- nox Technologies”: > lspci |grep -i Mellanox 27:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] Mellanox Technologies...

  • Page 21: Chapter 4 Driver Installation And Configuration

    Rev 1.1 Driver Installation and Configuration Linux Driver For Linux, download and install the latest Linux Drivers for Mellanox ConnectX-3 Ethernet adapters software package available on the Dell support site http://www.dell.com/support driver installation instructions, please refer to Dell documentation via http://www.dell.com/sup-...

  • Page 22: Pre-Installation Notes

    Not applicable to this system configuration. This can occur when the required hardware is not present on the system. Prerequisites are not met. For example, missing the required software installed or the hardware is not configured correctly. Failed to start the driver Mellanox Technologies...

  • Page 23: Installation Procedure

    Preparing... ################################################## ofed-scripts ################################################## Preparing... ################################################## libibverbs ################################################## Preparing... ################################################## libibverbs-devel ################################################## Preparing... ################################################## libibverbs-devel-static ################################################## Preparing... ################################################## libibverbs-utils ################################################## Preparing... ################################################## libmlx4 ################################################## Preparing... ################################################## libmlx4-devel ################################################## Preparing... ################################################## libibumad ################################################## Preparing... ################################################## libibumad-devel ################################################## Preparing... ################################################## Mellanox Technologies...

  • Page 24: Installation Results

    ################################################## Preparing... ################################################## perftest ################################################## Device (02:00.0): 02:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX- Link Width: 8x PCI Link Speed: Unknown Installation finished successfully. The script adds the following lines to /etc/security/limits.conf for the userspace Step 6. components such as MPI:...

  • Page 25: Post-Installation Notes

    4.2.3 Quality of Service (QoS) Ethernet 4.2.3.1 Mapping Traffic to Traffic Classes Mapping traffic to TCs consists of several actions which are user controllable, some controlled by the application itself and others by the system/network administrators. Mellanox Technologies...

  • Page 26: Plain Ethernet Quality Of Service Mapping

    4 values possible via ToS. In case of VLAN interface, the UP obtained according to the above mapping is also used in the VLAN tag of the traffic Mellanox Technologies...

  • Page 27: Roce Quality Of Service Mapping

    4.2.3.5 Map Priorities with tc_wrap.py/mlnx_qos Network flow that can be managed by QoS attributes is described by a User Priority (UP). A user's is mapped to UP which in turn is mapped into TC. sk_prio • Indicating the UP Mellanox Technologies...

  • Page 28: Quality Of Service Properties

    TC1 did not use its share of 20%, the reminder will be used by TC0. Rate Limit Rate limit defines a maximum bandwidth allowed for a TC. Please note that 10% deviation from the requested values is considered acceptable. Mellanox Technologies...

  • Page 29: Quality Of Service Tools

    Rate limit for TCs (in Gbps). LIST is a comma seperated Gbps limit for each TC. Example: 1,8,8 will limit TC0 to 1Gbps, and TC1,TC2 to 8 Gbps each. -i INTF, --interface=INTF Interface name Show all interface's TCs Mellanox Technologies...
  • Page 30 5 skprio: 6 (tos: 16) skprio: 7 skprio: 8 skprio: 9 skprio: 10 skprio: 11 skprio: 12 skprio: 13 skprio: 14 skprio: 15 up: 1 up: 2 up: 3 up: 4 up: 5 up: 6 up: 7 Mellanox Technologies...
  • Page 31 -u SKPRIO_UP, --skprio_up=SKPRIO_UP maps sk_prio to UP. LIST is <=16 comma separated UP. index of element is sk_prio. -i INTF, --interface=INTF Interface name Example: set skprio 0-2 to UP0, and skprio 3-7 to UP1 on eth4 Mellanox Technologies...

  • Page 32: Ethernet Time-Stamping

    PCI depending on the congestion in the PCI buffers. Outgoing packets are time-stamped very close to placing them on the wire. 4.2.4.1 Enabling Time Stamping Time-stamping is off by default and should be enabled before use.  To enable time stamping for a socket: Mellanox Technologies...
  • Page 33 SOF_TIMESTAMPING_RAW_HARDWARE: return original raw hardware time stamp SOF_TIMESTAMPING_SYS_HARDWARE: return hardware time stamp transformed to the system time base SOF_TIMESTAMPING_SOFTWARE: return system time stamp generated in software SOF_TIMESTAMPING_TX/RX determine how time stamps are generated. SOF_TIMESTAMPING_RAW/SYS determine how they are reported Mellanox Technologies...
  • Page 34 * directly into Sync packets. In this case, transmitted Sync * packets will not received a time stamp via the socket error * queue. HWTSTAMP_TX_ONESTEP_SYNC, Note: for send side time stamping currently only HWTSTAMP_TX_OFF and HWTSTAMP_TX_ON are supported. Mellanox Technologies...

  • Page 35: Getting Time Stamping

    The call returns the original outgoing packet data including all headers preprended down to and including the link layer, the scm_timestamping control message and a sock_extended_err control message with ee_errno==ENOMSG and ee_origin==SO_EE_ORIGIN_TIMESTAMPING. A socket with such Mellanox Technologies...

  • Page 36: Querying Time Stamping Capabilities Via Ethtool

    4.2.5.1 Query Capabilities Time stamping is available if and only the hardware reports it is capable of reporting it. To verify whether RoCE Time Stamping is available, run i bv_ex_query_device Mellanox Technologies...

  • Page 37: Creating Time Stamping Completion Queue

    (wc_ex.wc_flags & IBV_WC_WITH_TIMESTAMP) { /* This wc contains a timestamp */ timestamp = wc_ex.timestamp; /* Timestamp is given in raw hardware time */ CQs that are opened with the versb should be always be polled with ibv_create_cq_ex verb. ibv_poll_cq_ex Mellanox Technologies...

  • Page 38: Querying The Hardware Time

    Step 2. log_num_mgm_entry_size mlx4_core log_num_mgm_entry_size=-1 Restart the driver Step 3.  To disable Flow Steering: Open the file. Step 1. /etc/modprobe.d/mlnx.conf Remove the Step 2. options mlx4_core log_num_mgm_entry_size= -1 Restart the driver Step 3. Mellanox Technologies...

  • Page 39: Flow Domains And Priorities

    InfiniBand link type packets always include QP number. For further information, please refer to the relevant man pages. • ibv_destroy_flow int ibv_destroy_flow(struct ibv_flow *flow_id) Input parameters: requires which is the return value of ibv_destroy_flow struct ibv_flow ibv_create_flow case of success. Mellanox Technologies...

  • Page 40: Table 6: Flow Specific Parameters

    , which, in ndo_rx_flow_steer turn, calls the underlying flow steering mechanism with the RFS domain. Enabling the RFS requires enabling the flag via the ethtool, ‘ntuple’ For example, to enable ntuple for eth0, run: ethtool -K eth0 ntuple on Mellanox Technologies...

  • Page 41: Single Root Io Virtualization (Sr-Iov)

    Appendix A.5, “SR-IOV Configuration,” on page 104. Enable “SR-IOV Global Enable” in system BIOS - integrated Devices section Install a hypervisor that supports SR-IOV. Step 5. Depending on your system, update the /boot/grub/grub.conf file to include a similar com- Step 6. mand line load parameter for the Linux kernel. Mellanox Technologies...
  • Page 42 ConnectX® HCAs on the host • - The driver will enable 5 num_vfs=00:04.0-5,00:07.0-8 VFs on the HCA positioned in BDF 00:04.0 and 8 on the one in 00:07.0) Note: PFs not included in the above list will not have SR- IOV enabled. Mellanox Technologies...
  • Page 43 5. Enable SR-IOV in the System BIOS Load the driver and verify the SR-IOV is supported. Run: Step 11. lspci | grep Mellanox Where: • “03:00" represents the Physical Function • “03:00.X" represents the Virtual Function connected to the Physical Function Mellanox Technologies...

  • Page 44: Enabling Sr-Iov And Para Virtualization On The Same Setup

    This section will describe a mechanism for adding a SR-IOV VF to a Virtual Machine. Assigning the SR-IOV Virtual Function to the Red Hat KVM VM Server Run the virt-manager. Step 1. Double click on the virtual machine and open its Properties. Step 2. Mellanox Technologies...

  • Page 45: Uninstalling Sr-Iov Driver

    /etc/infiniband/openib.conf saved as /etc/infiniband/openib.conf.rpmsave Running /tmp/2818-ofed_vendor_post_uninstall.sh Restart the server. Step 3. Mellanox Technologies...

  • Page 46: Configuring Pkeys And Guids Under Sr-Iov

    The value returned will present which guid index to modify on Dom0. Modify the physical GUID table via the sysfs interface. Step 2. admin_guids To configure the GUID at index on port <n> <port_num> cd /sys/class/infiniband/mlx4_0/iov/ports/<port_num>/admin_guids echo <your desired guid> > n Mellanox Technologies...

  • Page 47: Ethernet Virtual Function Configuration When Running Sr-Iov

    [ spoofchk { on | off} ] ] use: ip link set dev <PF device> vf <NUM> vlan <vlan_id> [qos <qos>] • where NUM = 0..max-vf-num • vlan_id = (4095 means 0..4095 "set VGT" • qos = 0..7 For example: Mellanox Technologies...

  • Page 48: Ethtool

    • Get identification and diagnostic information • Get extended device statistics • Control speed, duplex, autonegotiation and flow control for Ethernet devices • Control checksum offload and other hardware offload features • Control DMA ring sizes and interrupt moderation Mellanox Technologies...

  • Page 49: Table 7: Ethtool Supported Options

    [rx-usecs-high N] • Above an upper limit of packet rate, adaptive moderation will set the moderation time to its highest value. • Below a lower limit of packet rate, the moderation time will be set to its lowest value. Mellanox Technologies...

  • Page 50: Ethernet Performance Counters

    • 2 counters reserved for VF - 1 counter for each port • All other counters if exist are allocated by demand • RoCE counters are available only through sysfs located under: • # /sys/class/infiniband/mlx4_*/ports/*/counters/ • # /sys/class/infiniband/mlx4_*/ports/*/counters_ext/ Mellanox Technologies...

  • Page 51: Table 8: Port In Counters

    (decimal) range [1535:1501] rx_lt_64_bytes_packets Number of received 64-or-less-octet frames rx_127_bytes_packets Number of received 65-to-127-octet frames rx_255_bytes_packets Number of received 128-to-255-octet frames rx_511_bytes_packets Number of received 256-to-511-octet frames rx_1023_bytes_packets Number of received 512-to-1023-octet frames rx_1518_bytes_packets Number of received 1024-to-1518-octet frames Mellanox Technologies...

  • Page 52: Table 9: Port Out Counters

    Table 10 - Port VLAN Priority Tagging (where <i> is in the range 0…7) Counter Description rx_prio_<i>_packets Total packets successfully received with priority i. rx_prio_<i>_bytes Total bytes in successfully received packets with priority i. rx_novlan_packets Total packets successfully received with no VLAN priority. Mellanox Technologies...

  • Page 53: Table 11: Port Pause (Where Is In The Range 0

    Table 12 - VPort Statistics (where <i>=<empty_string> is the PF, and ranges 1…NumOfVf per VF) Counter Description vport<i>_rx_unicast_packet Unicast packets received successfully vport<i>_rx_unicast_bytes Unicast packet bytes received successfully vport<i>_rx_multicast_pack Multicast packets received successfully vport<i>_rx_multicast_byte Multicast packet bytes received successfully Mellanox Technologies...

  • Page 54: Table 13 - Sw Statistics

    Number of packets received with good checksum rx_csum_none Number of packets received with no checksum indication tx_chksum_offload Number of packets transmitted with checksum offload tx_queue_stopped Number of times transmit queue suspended tx_wake_queue Number of times transmit queue resumed tx_timeout Number of times transmitter timeout Mellanox Technologies...

  • Page 55: Vmware Driver

    Reboot ESXi server (The driver will be loaded automatically). a. Update the driver. Run: #> esxcli software vib update –v <vib_url> b. Reboot ESXi server (The driver will be loaded automatically). » To verify that the driver is loaded, run: #> vmkload_mod -l | grep mlx4_en Mellanox Technologies...
  • Page 56 #> esxcli network nic list The number of uplinks claimed by MLX4_EN driver should be displayed. In Non Multifunction Mode, port 2 is identified as a pseudo device. Therefore devices are not seen by vSphere when added as uplink. Mellanox Technologies...

  • Page 57: Installing And Running The Offline_Bundle Driver On Esxi-5.X

    » To remove the VIB/offline_bundle driver package from the ESXi server machine, run: #> esxcli software vib remove –n net-mlx4-en Windows For Windows, download and install the latest Mellanox OFED for Windows (WinOF) software package available at the Dell support site http://www.dell.com/support. Mellanox Technologies...

  • Page 58: Installation Requirements

    1. Attended Installation - An installation procedure that requires frequent user intervention. 2. Unattended Installation - An automated installation procedure that requires no user interven- tion. 4.4.4.1 Attended Installation Double click the Dell Update Package and follow the GUI instructions to install Mellanox_WinOF. Mellanox Technologies...

  • Page 59: Unattended Installation

    Option 1. Click Start-> Control Panel-> Programs and Features. (NOTE: This requires elevated administrator privileges) Option 2. Double click the Dell Update Package and follow the instructions of the install wizard. Option 3. Click Start-> All Programs-> Mellanox Technologies-> MLNX_WinOF-> Uninstall MLNX_WinOF. 4.4.6 Windows Performance Tuning The user can configure the Ethernet adapter by setting some registry keys.
  • Page 60 PS $ Set-DnsClientServerAddress -InterfaceAlias “Ethernet 4” -ServerAddresses 192.168.1.2 After establishing the priorities of ND/NDK traffic, the priorities must have PFC enabled on them. Disable Priority Flow Control (PFC) for all other priorities except for 3. Step 8. PS $ Disable-NetQosFlowControl 0,1,2,4,5,6,7 Mellanox Technologies...

  • Page 61: Rdma Over Converged Ethernet

    Step 1. PS $ Remove-NetQosTrafficClass PS $ Remove-NetQosPolicy -Confirm:$False Enable PowerShell to configure DCB. Step 2. PS $ Set-ExecutionPolicy Unrestricted Enable the Data Center Bridging (DCB) feature which is mandatory for PFC. Step 3. PS $ Install-WindowsFeature Data-Center-Bridging Mellanox Technologies...
  • Page 62 After establishing the priorities of ND/NDK traffic, the priorities must have PFC enabled on them. Enable Priority Flow Control (PFC) on priority 3, run. Step 9. PS $ Enable-NetQosFlowControl -Priority 3 Disable PFC for all other priorities except for 3. Step 10. PS $ Disable-NetQosFlowControl 0,1,2,4,5,6,7 Mellanox Technologies...

  • Page 63: Configuring Router (Pfc Only)

    SMB and that their RDMA capability is properly identified. • On the SMB client, run the following PowerShell cmdlets: Get-SmbClientConfiguration | Select EnableMultichannel Get-SmbClientNetworkInterface • On the SMB server, run the following PowerShell cmdlets: Get-SmbServerConfiguration | Select EnableMultichannel Mellanox Technologies...

  • Page 64: Verifying Smb Connection

     To confirm RDMA connection, verify the SMB events: Open a PowerShell window on the SMB client. Step 1. Run the following cmdlets. Step 2. NOTE: Any RDMA-related connection errors will be displayed as well. Get-WinEvent -LogName Microsoft-Windows-SMBClient/Operational | ? Message -match "RDMA" Mellanox Technologies...

  • Page 65: Chapter 5 Remote Boot

    NOPTimeout # Wait that many seconds for a response on a NOP-In ping # If 0 or > NOPInterval, NOPInterval is used! # Various target parameters Wthreads # Number of IO threads QueuedCommands # Number of queued commands Mellanox Technologies...

  • Page 66: Installing Rhel6.4/Rhel6.5 On A Remote Storage Over Iscsi

    In this example, NFS is used, which also requires us to enter the NFS server name be entered, and the directory path to the installation media on the NFS. Select Specialized Storage Devices. Step 4. Click on the + Add Advanced Target button. Step 5. Mellanox Technologies...
  • Page 67 No authentication. Check the relevant Node Name to log in. Step 8. If as a result of the discovery, multiple Node Names are found, select the one that is rele- vant. Mellanox Technologies...
  • Page 68 A successful LUN discovery is mandatory to proceed. A failure at this stage is probably a result of a target or network configuration error and recovery/troubleshooting that is out of the scope of this document. Click Next. Step 11. Select Fresh Installation and proceed with the Installation Step 12. Mellanox Technologies...
  • Page 69 Select the Basic Server option. Step 15. This is only one of the options that can be chosen, not the mandatory one. Check the Customize Now checkbox. Step 16. Click Next. Step 17. Select Infiniband Support and iSCSI Storage Client. Step 18. Mellanox Technologies...

  • Page 70: San-Booting The Diskless Client With Flexboot

    For boot over Ethernet, when using adapter cards with older firmware version than 2.30.8000, onewill need to extract the PXE package, otherwise use Mellanox WinOF VPI package. Example: Mellanox.msi.exe -a 3. Add the Mellanox driver to boot.wim dism /Mount-Wim /WimFile:boot.wim /index:2 /MountDir:mnt dism /Image:mnt /Add-Driver /Driver:drivers /recurse dism /Unmount-Wim /MountDir:mnt /commit Mellanox Technologies...

  • Page 71: Configuring The Client Machine

    6. Set the “Mellanox Adapter Card” as the first boot device in the BIOS settings boot order. 1. Use ‘index:2’ for Windows setup and ‘index:1’for WinPE. 1. When adding the Mellanox driver to install.wim, verify the appropriate index for the OS distribution is used. To check the OS run ‘imagex /info install.win’. Mellanox Technologies...

  • Page 72: Installing Iscsi

    2. Press F12 when asked to proceed to iSCSI boot. 3. Choose the relevant boot image from the list of all available boot images presented. The above figure is an example only. Microsoft Windows 4.60 is the supported operat- ing system. Mellanox Technologies...

  • Page 73: Sles11 Sp3

    Installation process will start once completing all the required steps in the Wizard, the Client will reboot and will boot from the iSCSI target. 5.1.3 SLES11 SP3 5.1.3.1 Configuring the iSCSI Target Machine  To configure the iSCSI target: Download the IET target software from. Step 1. http://sourceforge.net/projects/iscsitarget/files/iscsitarget/1.4.20.2/ Mellanox Technologies...
  • Page 74 Edit the /etc/sysconfig/iscsi-target file as follow. Step 4. OPTIONS="-c /etc/iet/ietd.conf --address=12.7.6.30" Start the iSCSI target service. Step 5. [root@sqa030 ~]# /etc/init.d/iscsi-target start Perform a sanity check by connecting to the iSCSI target from a remote PC on the 10GE Step 6. network link. Mellanox Technologies...

  • Page 75: Configuring The Dhcp Server

    The clients’ HDD was removed beforehand; therefore the installer will ask to locate a HDD. The built-in iSCSI discovery will be used to connect to the iSCSI target LUN parti- tion. Mellanox Technologies...
  • Page 76 Select the "Install SLES11.3" boot option from the menu (see pxelinux.cfg example above). Step 3. After about 30 seconds, the SLES installer will issue the notification below due to the PXELINUX boot label we used above. Click OK. Step 4. Mellanox Technologies...
  • Page 77 User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Click on the Configure iSCSI Disks button. Step 5. Choose Connected Targets tab. Step 6. Mellanox Technologies...
  • Page 78 Step 7. Enter the IP address of the iSCSI storage target. Step 8. Click Next. Step 9. Select the relevant target from the table (In our example, only one target exist so only one Step 10. was discovered). Mellanox Technologies...
  • Page 79 User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Click Connect. Step 11. Select onboot from drop-list. Step 12. Click Next to exit the discovery screen. Step 13. Mellanox Technologies...
  • Page 80 Go to the Connected Targets tab again to confirm iSCSI connection with target. Step 14. Click OK. Step 15. Click Next back at the Disk Activation screen. Step 16. Select New Installation. Step 17. Click Next. Step 18. Complete Clock and Time Zone configuration. Step 19. Mellanox Technologies...
  • Page 81 RPM is selected for the installation under "open-iscsi" "Software" After the installation is completed, the system will reboot. Choose label from the boot menu (See Section 5.1.3.3, "SLES11.3x64_iscsi_boot" “Installing SLES11 SP3 on a Remote Storage over iSCSI,” on page 75). Mellanox Technologies...

  • Page 82: Using Pxe Boot Services For Booting The Sles11 Sp3 From The Iscsi Target

    • Boot the client into the below SLES11 SP3 iso and proceed with the installation until the client fully boots up the installer program. • Discover and connect to a remote iSCSI storage. During the installation process you will be asked to insert the original installation medium to continue with the installation. SLES11SP3-kISO-VPI/sles11-sp3-x86_64-mlnx_ofed-2.1-1.0.6.iso Mellanox Technologies...
  • Page 83 SLES unattented installation with autoyast, please refer to: https://www.suse.com/documentation/sles11/book_autoyast/?page=/documentation/sles11/ book_autoyast/data/book_autoyast.html The following is known to work with Mellanox NIC: append initrd=SLES-11-SP3-DVD-x86_64-GM-DVD1/boot/x86_64/loader/initrd install=nfs:// <NFS IP Address>/<path the the repository directory>/ autoyast=nfs://<NFS IP Address>/ <path to autoyast xml directory>/autoyast-unattended.xml biosdevname=0 IPAPPEND 2 Mellanox Technologies...

  • Page 84: Chapter 6 Firmware

    Firmware and update instructions for these cards can be obtained from the Dell support web page: http://www.dell.com/support. Note: The firmware version on the adapter can be checked using the following methods: 1. System Setup > Device Settings 2. Dell iDRAC Mellanox Technologies...

  • Page 85: Chapter 7 Troubleshooting

    • Check that both the adapter and its link are set to the same speed with no and duplex settings communication established • Check to make sure flow-control is enabled on the switch ports. Low Performance with RDMA over Converged Ethernet (RoCE) Mellanox Technologies...

  • Page 86: Linux

    If a yellow sign appears near the Mellanox ConnectX Ethernet Adapter instance in the Device Manager display, this can happen due to a hardware error. Try to disable and re-enable “Mellanox Con- nectX Adapter” from the Device Manager display. Mellanox Technologies...
  • Page 87 Mellanox ConnectX EN 10Gbit Ethernet Adapter <X> device failed to create enough MSI-X vectors. The Network interface will not use MSI-X interrupts. This may affects the performance. To fix the prob- lem, configure the number of MSI-X vectors in the registry to be at least <Y>. Mellanox Technologies...

  • Page 88: Chapter 8 Specifications

    Rev 1.1 Specifications Specifications Table 16 - Mellanox ConnectX-3 Dual 40GbE QSFP+ Network Adapter Specifications Board Size: 2.71in. x 5.6in. (68.90 mm x 142.25 mm) Full height Bracket Size: 4.5 in. (116 mm) Low profile Bracket Size: 3.16 in. (80.3 mm)

  • Page 89: Table 17: Mellanox Connectx-3 Dual 10Gbe Sfp+ Network Adapter Specifications

    User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Table 17 - Mellanox ConnectX-3 Dual 10GbE SFP+ Network Adapter Specifications Size: 2.71in. x 5.6in. (68.90 mm x 142.25 mm) Full height Bracket Size: 3.8in (96.52 mm) Low profile Bracket Size: 3.16in (80.3 mm)

  • Page 90: Table 18: Mellanox Connectx-3 Dual 10Gbe Kr Blade Mezzanine Card Specifications

    Rev 1.1 Specifications Table 18 - Mellanox ConnectX-3 Dual 10GbE KR Blade Mezzanine Card Specifications Size: 3.4in. x 3.3in. (88 mm x 84mm) Physical Ethernet: 10GBASE-KR, 10GBASE-KX4, 1000BASE-KX Protocol Support Data Rate: 1/10Gb/s – Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V, 3.3Vaux...

  • Page 91: Chapter 9 Regulatory

    § 15.21 Statement Warning! Changes or modifications to this equipment not expressly approved by the party responsible for compliance (Mellanox Technologies) could void the user's authority to operate the equip- ment. §15.105(a) Statement NOTE: This equipment has been tested and found to comply with the limits for a Class A digi- tal device, pursuant to Part 15 of the FCC Rules.

  • Page 92: En Statements (Europe)

    (Translation - “This is a Class A product based on the standard of the Voluntary Control Coun- cil for Interference by Information Technology Equipment (VCCI). If this equipment is used in a domestic environment, radio interference may occur, in which case the user may be required to take corrective actions.”) Mellanox Technologies...

  • Page 93: Kcc Certification (Korea)

    User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 9.1.5 KCC Certification (Korea) Mellanox Technologies...

  • Page 94: Appendix A Configuration For Mellanox Adapters Through System Setup

    This section covers the main configuration options in Dell PowerEdge System Setup which can be accessed through BIOS or through Lifecycle Controller. Figure 3: System Setup Menu Setup menu options: 1. System BIOS 2. iDRAC Settings 3. Device Settings Mellanox Technologies...

  • Page 95: Figure 4: Main Configuration Page Options

    1. Shows general info regarding the adapter. 2. Allows configuration of SR-IOV on the adapter - see Appendix A.5, “SR-IOV Configura- tion,” on page 104. 3. Allows setting the Blink LEDs to allow physical identification of the card Mellanox Technologies...
  • Page 96 Rev 1.1 a. To trigger Blink LEDs configure the number of seconds for it to blink (Max is 15 seconds). Mellanox Technologies...
  • Page 97 1. Allows configuration of Legacy Boot Protocol: None, PXE, iSCSI. 2. Allows configuration of Virtual LAN Mode and Virtual LAN ID. 3. Allows to Enable or Disable the Option ROM. 1. These version numbers are just an example. Mellanox Technologies...
  • Page 98 Rev 1.1 4. Allows setting Boot Retry Count. Mellanox Technologies...
  • Page 99 User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 5. Allows setting Boot Retry Count. Mellanox Technologies...
  • Page 100 Rev 1.1 Main Configuration Page - iSCSI Configuration This section allows to override the default configurations of iSCSI and replaces DHCP configu- ration of iSCSI. Mellanox Technologies...

  • Page 101: Figure 5: Main Configuratiop Page - Iscsi Configuration - Iscsi General Parameters

    User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Figure 5: Main ConfiguratioP page - iSCSI Configuration - iSCSI General Parameters Mellanox Technologies...

  • Page 102: Figure 6: Main Configuration Page - Iscsi Configuration - Iscsi Initiator Parameters

    Rev 1.1 Figure 6: Main Configuration Page - iSCSI Configuration - iSCSI Initiator Parameters Mellanox Technologies...

  • Page 103: Figure 7: Main Configuration Page - Iscsi Configuration - Iscsi Target Parameters

    User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Figure 7: Main Configuration Page - iSCSI Configuration - iSCSI Target Parameters Main Configuration Page - Device Level Configuration Allows setting Global Flow control settings for the adapter's port. Mellanox Technologies...
  • Page 104 Rev 1.1 SR-IOV Configuration Enabling SRIOV requires configuration both for the system and the specific Mellanox adapter. To enable SR-IOV - follow the below steps 1-4. To disable SR-IOV - set the configuration in steps 1-3 to disabled. Mellanox Technologies...
  • Page 105 User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Step 1. Enable “Virtualization Technology” in System BIOS => Processor setting : Step 2. Enable “SR-IOV Global Enable” Go to: System BIOS => integrated Devices section Mellanox Technologies...
  • Page 106 By default, Mellanox adapter is set to SRI-OV enabled with 8 virtual functions. Note the maximum number of virtual functions supported by the adapter PCIe. Refer to the relevant driver user manual for support for SR-IOV and number of supported func- tions. Mellanox Technologies...
  • Page 107 User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Step 4. Reboot the server for the SR-IOV configuration to take effect. Mellanox Technologies...

  • Page 108: Appendix B Safety Warnings

    This equipment should be installed in compliance with local and national electrical codes. 8. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. Mellanox Technologies...
  • Page 109 User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 110: Appendix C Avertissements De Sécurité D'installation (Warnings In French)

    Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Exposition au rayonnement grave Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. Mellanox Technologies...
  • Page 111 User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825- 1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 112: Appendix D Sicherheitshinweise (Warnings In German)

    Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungent Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. This equipment should be installed in compliance with local and national elec- trical codes. Mellanox Technologies...
  • Page 113 Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen.. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001 Mellanox Technologies...

  • Page 114: Appendix E Advertencias De Seguridad Para La Instalación (Warnings In Spanish)

    Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. 8. Exposición a niveles de radiación peligrosos Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. Mellanox Technologies...
  • Page 115 User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Mellanox Technologies...

Table of Contents