Qlogic QL45212 User Manual
  1. Manuals
  2. Brands
  3. Qlogic Manuals
  4. Adapter
  5. QL45212
  6. User manual

Qlogic QL45212 User Manual

25gb intelligent ethernet adapter
Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
User's Guide
25Gb Intelligent Ethernet Adapter
QL45212
Third party information brought to
BC0154503-00 C
you courtesy of Dell EMC.

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the QL45212 and is the answer not in the manual?

Questions and answers

Related Manuals for Qlogic QL45212

Summary of Contents for Qlogic QL45212

  • Page 1 User’s Guide 25Gb Intelligent Ethernet Adapter QL45212 Third party information brought to BC0154503-00 C you courtesy of Dell EMC.

  • Page 2: Document Revision History

    This document is provided for informational purposes only and may contain errors. QLogic reserves the right, without notice, to make changes to this document or in product design or specifications. QLogic disclaims any warranty of any kind, expressed or implied, and does not guarantee that any results or performance described in the document will be achieved by you.
  • Page 3 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 Removed “Running the Dell Update Package from “Installing Drivers” on page 9 the Command Line” topic. Updated the list of RHEL source and kmod RPM “Linux Driver Software” on page 10 package names. Added SLES source and kmod RPM packages.
  • Page 4 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 Added reference on CentOS on Step 2 and Step 3, “Installing Linux Drivers Using the src RPM Pack- under “For RHEL” age” on page 14 Updated Step 1 command “Installing Linux Drivers Using the kmp/kmod RPM Package”...
  • Page 5 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 Added new bullet “RoCE is enabled by default for “Planning” on page 49 both V1 and V2.” Deleted the following: “Planning” on page 49 qedr cannot operate with RDMA connection man- ager (CM) when I/O memory management unit (IOMMU) is enabled.
  • Page 6 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 Added new topic and subtopics on “RoCE v2 Con- “RoCE v2 Configuring for Linux” on page 59 figuring for Linux.” Added reference to CentOS. “iSCSI Extensions for RDMA” on page 69 Added reference to SLES 12.

  • Page 7: Table Of Contents

    Table of Contents Preface Overview ........... xiii Intended Audience .
  • Page 8 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 Installing the Linux Drivers without RoCE ..... . Removing the Linux Drivers ......
  • Page 9 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 Preparing the Ethernet Switch ........
  • Page 10 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 Adapter LEDS Dell Z9100 Switch Configuration Glossary Index BC0154503-00 C...

  • Page 11: List Of Figures

    Dell Update Package Window ......... QLogic InstallShield Wizard: Welcome Window ......

  • Page 12: List Of Tables

    QLogic QL45212 Linux Drivers ........

  • Page 13: Preface

    Human Infrastructure Interface (HII) application.  Chapter 6, Configuring RoCE describes how to configure the QL45212 adapter, the Ethernet switch, and the host to use RDMA over converged Ethernet (RoCE).  Chapter 7, iSCSI Extensions for RDMA describes how to configure iSCSI Extensions for RDMA (iSER) for RHEL and SLES.

  • Page 14: Documentation Conventions

     “Installation Checklist” on page  For more information, visit www.qlogic.com.  Text in bold font indicates user interface elements such as a menu items, buttons, check boxes, or column headings. For example: ...

  • Page 15: License Agreements

    License Agreements Refer to the QLogic Software End User License Agreement for a complete listing of all license agreements affecting this product. BC0154503-00 C...

  • Page 16: Legal Notices

    Preface Legal Notices Legal Notices Warranty For warranty details, please check the QLogic website: www.qlogic.com/Support/Pages/Warranty.aspx Laser Safety FDA Notice This product complies with DHHS Rules 21CFR Chapter I, Subchapter J. This product has been designed and manufactured according to IEC60825-1 on the safety label of laser product.

  • Page 17: Agency Certification

    Legal Notices Agency Certification The following sections summarize the EMC and EMI test specifications performed on the QL45212 Intelligent Ethernet Adapter to comply with emission, immunity, and product safety standards: EMI and EMC Requirements FCC Part 15 compliance: Class A FCC compliance information statement: This device complies with Part 15 of the FCC Rules.

  • Page 18: Kcc: Class A

    Legal Notices KCC: Class A Korea RRA Class A Certified Product Name/Model: Converged Network Adapters and Intelligent Ethernet Adapters Certification holder: QLogic Corporation Manufactured date: Refer to date code listed on product Manufacturer/Country of origin: QLogic Corporation/USA A class equipment...

  • Page 19: Product Overview

    Product Overview This chapter provides information about the following topics:  Functional Description  Features  Adapter Specifications BC0154503-00 C...

  • Page 20: Functional Description

    1–Product Overview Functional Description Functional Description The QL45212 Adapter is a 25Gb Intelligent Ethernet Adapter that is designed to perform accelerated data networking for Dell PowerEdge systems. The QL45212 Adapter includes a 25Gb Ethernet MAC with full-duplex capability. Features The QL45212 Adapter provides the following features: ...

  • Page 21: Adaptive Interrupt Frequency

    CPU cycles. ASIC with Embedded RISC Processor The core control for the QL45212 Adapter resides in a tightly integrated, high-performance ASIC. The ASIC includes a RISC processor, which provides the flexibility to add new features to the card and adapts it to future network requirements through software downloads.

  • Page 22: Adapter Specifications

    1–Product Overview Adapter Specifications Adapter Specifications Physical Characteristics ® The QL45212 Adapter is a standard PCI Express card and ships with either a ® full-height or a low-profile bracket for use in a standard PCIe slot. Standards Specifications  PCI Express Base Specification, rev. 3.0 ...

  • Page 23: Installing The Hardware

    Installing the Hardware This chapter provides information about the following topics:  System Requirements  Safety Precautions  Preinstallation Checklist Installing the Adapter  BC0154503-00 C...

  • Page 24: System Requirements

    2–Installing the Hardware System Requirements System Requirements Before you install a QLogic QL45212 Adapter, verify that your system meets the following hardware (Table 2-1) and operating system (Table 2-2) requirements. For a complete list of supported operating systems, see driverdownloads.qlogic.com.

  • Page 25: Safety Precautions

    2–Installing the Hardware Safety Precautions Safety Precautions WARNING The adapter is being installed in a system that operates with voltages that can be lethal. Before you open the case of your system, observe the following precautions to protect yourself and to prevent damage to the system components.

  • Page 26: Installing The Adapter

    2–Installing the Hardware Installing the Adapter Installing the Adapter The following instructions apply to installing the QL45212 Adapter in most systems. For details about performing these tasks, refer to the documents that were supplied with your system. Review “Safety Precautions” on page 7 “Preinstallation Checklist”...

  • Page 27: Installing Drivers

    Installing Drivers This chapter provides information about the following topics:  Linux Driver Software  Windows Driver Software  VMware Driver Software BC0154503-00 C...

  • Page 28: Linux Driver Software

    Linux Ethernet driver for the QL45212 Adapter. This driver directly controls the hardware and is responsible for sending and receiving Ethernet pack- ets on behalf of the Linux host networking stack. This driver also receives and processes device interrupts, on behalf of itself (for L2 networking).

  • Page 29: Installing The Linux Drivers Without Roce

    When using the following procedures in a CentOS environment, follow the instructions for RHEL. To install the Linux drivers without RoCE, do the following: Download the QL45212 Adapter Linux drivers from dell.support.com. Remove the existing Linux drivers, as described in “Removing the Linux Drivers”...

  • Page 30: Removing The Linux Drivers

    3–Installing Drivers Linux Driver Software Removing the Linux Drivers There are two procedures for removing Linux drivers: one for a non-RoCE environment and another for a RoCE environment. Choose the procedure that matches your environment.  Removing Linux Drivers in a non-RoCE Environment Removing Linux Drivers in a RoCE Environment ...
  • Page 31 3–Installing Drivers Linux Driver Software Removing Linux Drivers in a RoCE Environment To remove Linux drivers in a RoCE environment: Type the following command to get the path to the currently installed drivers: modinfo <drivername> Unload and remove the Linux drivers. modprobe -r qedr Remove driver module files.

  • Page 32: Installing Linux Drivers Using The Src Rpm Package

    3–Installing Drivers Linux Driver Software Installing Linux Drivers Using the src RPM Package To install Linux drivers using the src RPM package: Type the following at a command prompt: rpm -ivh RPMS/<arch>/qlgc-fastlinq-<version>.src.rpm Change the directory to the RPM path and build the binary RPM for the kernel.

  • Page 33: Installing Linux Drivers Using The Kmp/Kmod Rpm Package

    3–Installing Drivers Linux Driver Software Installing Linux Drivers Using the kmp/kmod RPM Package To install kmod RPM package: Type the following command at a command prompt: rpm -ivh qlgc-fastlinq-<version>.<arch>.rpm Reload the driver: modprobe -r qede modprobe qede Installing Linux Drivers Using the TAR File To install Linux drivers using the TAR file: Create a directory and extract the TAR files to the directory: tar xjvf fastlinq-minor-<version>.tar.bz2...

  • Page 34: Installing The Linux Drivers With Roce

    When using the following procedures in a CentOS environment, follow the instructions for RHEL. To install Linux drivers in an in-box OFED environment: Download the QL45212 Adapter Linux drivers from dell.support.com. Configure RoCE on the adapter, as described in “Configuring RoCE on the Adapter for Linux”...

  • Page 35: Linux Driver Optional Parameters

    3–Installing Drivers Linux Driver Software Linux Driver Optional Parameters Table 3-2 describes the qede driver optional parameters. Table 3-2. qede Driver Optional Parameters Parameter Description Controls driver verbosity level. Similar to ethtool -s debug <dev> msglvl. Controls interrupt mode other than MSI-X. int_mode Enables the HW GRO feature (under development).

  • Page 36: Linux Driver Messages

    3–Installing Drivers Windows Driver Software Linux Driver Messages To set the Linux driver message detail level, use one of the following commands:  ethtool -s <interface> msglvl <value>  modprobe qede debug=<value> The <value> represents bits 0–15, which are standard Linux networking values, and bits 16 and greater are driver specific.

  • Page 37: Running The Dell Update Package In The Gui

    The actual file name of the Dell update package varies. In the Dell Update Package window (Figure 3-1), click Install. Figure 3-1. Dell Update Package Window ® In the QLogic Super Installer—InstallShield Wizard’s Welcome window (Figure 3-2), click Next. Figure 3-2. QLogic InstallShield Wizard: Welcome Window BC0154503-00 C...
  • Page 38 Complete the following in the wizard’s License Agreement window (Step 3-3): Read the QLogic End User Software License Agreement. To continue, select I accept the terms in the license agreement. Click Next. Figure 3-3. QLogic InstallShield Wizard: License Agreement Window...
  • Page 39 3–Installing Drivers Windows Driver Software Complete the wizard’s Setup Type window (Figure 3-4) as follows: Select one of the following setup types:  Click Complete to install all program features.  Click Custom to manually select the features to be installed. To continue, click Next.
  • Page 40 3–Installing Drivers Windows Driver Software If you selected Custom in Step 5, complete the Custom Setup window (Figure 3-5) as follows: Select the features to install. By default, all features are selected. To change a feature’s install setting, click the icon next to it, and then select one of the following options: ...
  • Page 41 In the InstallShield Wizard’s Ready To Install window (Figure 3-6), click Install. The InstallShield Wizard installs the QLogic Adapter drivers and Management Software Installer. Figure 3-6. InstallShield Wizard: Ready to Install the Program Window When the installation is complete, the InstallShield Wizard Completed...

  • Page 42: Installation Options

    3–Installing Drivers Windows Driver Software In the Dell Update Package window (Figure 3-8), “Complete” indicates successful installation.  (Optional) To open the log file, click View Installation Log. The log file shows the progress of the DUP installation, any previous installed versions, any error messages, and other information about the installation.

  • Page 43: Examples

    <DUP_file_name>.exe /l="C:\my path with spaces\log.txt" Removing the Windows Drivers To remove the Windows drivers: In Control Panel, click Programs, Programs and Features. In the list of programs, select QLogic Driver and Management Super Installer (x64), and then click Uninstall. Follow the instructions. BC0154503-00 C...

  • Page 44: Managing Adapter Properties

    3–Installing Drivers Windows Driver Software Managing Adapter Properties To view or change the QL45212 Adapter properties In Control Panel, click Device Manager. Click the Advanced section of the chosen port. Setting Power Management Options You can set power management options to allow the operating system to turn off the controller to save power or to allow the controller to wake up the computer.

  • Page 45: Vmware Driver Software

    3–Installing Drivers VMware Driver Software VMware Driver Software This section describes the qedentv VMware ESXi driver for the QL45212 Adapter.  Installing VMware Drivers  VMware Driver Optional Parameters  VMware Driver Parameter Defaults  Removing the VMware Driver Installing VMware Drivers You can use the driver zip file to install a new driver or update an existing driver.

  • Page 46: Vmware Driver Optional Parameters

    3–Installing Drivers VMware Driver Software To upgrade an existing driver, follow the steps for a new installation, except replace the command in Step 6 with the following: # esxcli software vib update -v /tmp/qedentv-1.0.3.11-1OEM.550.0.0.1331820.x86_64.vib VMware Driver Optional Parameters Table 3-4 describes the optional parameters that can be supplied as a command line arguments to the esxcfg-module command.
  • Page 47 3–Installing Drivers VMware Driver Software Table 3-4. VMware Driver Optional Parameters (Continued) Parameter Description The RSS parameter specifies the number of receive side scal- ing queues used by the host or virtual extensible LAN (VxLAN) tunneled traffic for a PF. RSS can be 2, 3, or 4, or one of the following values: ...

  • Page 48: Vmware Driver Parameter Defaults

    3–Installing Drivers VMware Driver Software VMware Driver Parameter Defaults Table 3-5 lists the VMware driver parameter defaults. Table 3-5. VMware Driver Parameter Defaults Parameter Default Speed Auto-negotiation with speed advertised Flow Control Auto-negotiation with Rx and Tx advertised 1,500 (range 46–9,600) Rx Ring Size 8,192 (range 128–8,192) Tx Ring Size...

  • Page 49: Upgrading Firmware

    Upgrading Firmware This chapter provides information about the following topics:  Run by Double-Clicking  Run from the Command Line  Updating Dell DUP Using .BIN File The firmware Dell update package is a Flash update utility only (it is not used for adapter configuration).

  • Page 50: Run By Double-Clicking

    4–Upgrading Firmware Run by Double-Clicking Run by Double-Clicking To run the firmware Dell update package by double-clicking the executable file, follow these instructions: Double-click the icon representing the firmware Dell update package file. The Dell update package splash screen appears, as shown in Figure 4-1.
  • Page 51 4–Upgrading Firmware Run by Double-Clicking Follow the on-screen instructions. Click Yes to continue the installation, as shown in Figure 4-2. Figure 4-2. Continue Dell Update Package Installation BC0154503-00 C...
  • Page 52 4–Upgrading Firmware Run by Double-Clicking The installer indicates that it is loading the new firmware, as shown in Figure 4-3. Figure 4-3. Loading New Firmware BC0154503-00 C...
  • Page 53 4–Upgrading Firmware Run by Double-Clicking When complete, the installer indicates the result of the installation, as shown Figure 4-4. Click Yes to reboot. Figure 4-4. Result of Installation Click Finish to complete the installation, as shown in Figure 4-5. Figure 4-5. Finish Installation BC0154503-00 C...

  • Page 54: Run From The Command Line

    4–Upgrading Firmware Run from the Command Line Run from the Command Line Running the firmware Dell update package from the command line, with no options specified, results in the same behavior as double-clicking the Dell update package icon. Note that the actual file name of the Dell update package will vary. C:\>...

  • Page 55: Updating Dell Dup Using .Bin File

    4–Upgrading Firmware Updating Dell DUP Using .BIN File Updating Dell DUP Using .BIN File Use the following procedure to update the Dell DUP using the .bin file: Copy the Network_Firmware_NJCX1_LN_X.Y.Z.BIN file to SUT. Change the file type into an executable file: chmod 777 Network_Firmware_NJCX1_LN_X.Y.Z.BIN Run ./Network_Firmware_NJCX1_LN_X.Y.Z.BIN to start the update process.

  • Page 56: Preboot Adapter Configuration

    Preboot Adapter Configuration This chapter provides information about the following topics: Getting Started  Displaying Firmware Image Properties   Device Level Configuration  NIC Configuration  Data Center Bridging Configuration  NIC Partitioning Configuration BC0154503-00 C...

  • Page 57: Getting Started

    5–Preboot Adapter Configuration Getting Started Getting Started During the host boot process, you have the opportunity to pause and perform adapter management tasks using the Human Infrastructure Interface (HII) application. These tasks include the following:  Displaying Firmware Image Properties ...
  • Page 58 5–Preboot Adapter Configuration Getting Started In the Device Settings page, select the QL45212 adapter port that you want to configure. The Main Configuration Page presents the adapter management options, as shown in Figure 5-2. Figure 5-1. Main Configuration Page BC0154503-00 C...

  • Page 59: Displaying Firmware Image Properties

    5–Preboot Adapter Configuration Displaying Firmware Image Properties In addition to the management options, the Main Configuration Page presents the adapter properties shown in Table 5-1. Table 5-1. Adapter Properties Adapter Property Description Device Name Factory-assigned device name Chip Type ASIC version PCI Device Id Unique vendor specific PCI device ID PCI Address...

  • Page 60: Device Level Configuration

    5–Preboot Adapter Configuration Device Level Configuration Device Level Configuration Device level configuration comprises enabling single root-I/O virtualization (SR-IOV) and/or NIC partitioning, and enabling or disabling NPAReP. To perform device level configuration, select Device Level Configuration in the Main Configuration Page, and then click Finish. The Device Level Configuration page is shown in Figure 5-2.
  • Page 61 Click Enabled to specify eight partitions per port. NOTE The QL45212 Adapter supports 8 virtual functions per port, with a total of 16 virtual functions per adapter. If the NPAReP is enabled, the QL45212 Adapter supports 16 virtual functions per partition. When...

  • Page 62: Nic Configuration

    5–Preboot Adapter Configuration NIC Configuration NIC Configuration To configure the port link speed: Select NIC Configuration in the Main Configuration Page (Figure 5-1), and then click Finish. In the NIC Configuration page, select from the link speed options listed in Table 5-2.

  • Page 63: Data Center Bridging Configuration

    5–Preboot Adapter Configuration Data Center Bridging Configuration Data Center Bridging Configuration The data center bridging settings comprise the DCBX protocol and the remote direct memory access (RDMA) over converged Ethernet (RoCE) priority. To configure the data center bridging (DCB) settings: Select Data Center Bridging (DCB) Settings in the Main Configuration Page (Figure...

  • Page 64: Nic Partitioning Configuration

    5–Preboot Adapter Configuration NIC Partitioning Configuration NIC Partitioning Configuration To configure the maximum and minimum bandwidth allocations for each partition: Select NIC Partitioning Configuration in the Main Configuration Page (Figure 5-1), and then click Finish. In the NIC Partitioning Configuration page, select Global Bandwidth Allocation.
  • Page 65 5–Preboot Adapter Configuration NIC Partitioning Configuration  Partition N Maximum TX Bandwidth is the maximum transmit bandwidth of the selected partition expressed as a percentage of the maximum physical port link speed. Values can be 1–100. The per partition maximum TX bandwidth value applies regardless of the DCBX ETS mode setting.

  • Page 66: Configuring Roce

    Configuring RoCE This chapter describes RoCE (v1 and v2) configuration on the QL45212 Adapter, the Ethernet switch, and the Windows or Linux host. This chapter describes the following topics:  Supported Operating Systems and OFED  Planning Preparing the Adapter ...

  • Page 67: Supported Operating Systems And Ofed

     qedr and libqedr RPM packages are required for in-box OFED.  OFED/RDMA applications that depend on libibverbs also require the QLogic RDMA user space library, libqedr. libqedr can be installed using libqedr RPM or source packages Mounting NFS over RDMA may take more time than normal on RHEL 7.0, ...

  • Page 68: Preparing The Adapter

    Settings, and then click Finish. In the Data Center Bridging (DCB) Settings window, click the DCBX Protocol drop-down. The QL45212 Adapter supports both CEE and IEEE protocols. This value should match the corresponding value on the DCB switch. In this example, select CEE.

  • Page 69: Preparing The Ethernet Switch

    6–Configuring RoCE Preparing the Ethernet Switch Preparing the Ethernet Switch ® ® This section describes how to configure a Cisco Nexus 6000 Ethernet Switch and a Dell Z9100 Ethernet switch for RoCE.  Cisco Nexus 6000 Ethernet Switch  Dell Z9100 Ethernet Switch Cisco Nexus 6000 Ethernet Switch Configuring the Cisco Nexus 6000 Ethernet Switch for RoCE comprises configuring class maps, configuring policy maps, applying the policy, and...

  • Page 70: Dell Z9100 Ethernet Switch

    6–Configuring RoCE Preparing the Ethernet Switch Configure network QoS policy maps to set priority flow control for no drop traffic class. switch(config)# policy-map type network-qos roce switch(config)# class type network-qos class-roce switch(config)# pause no-drop Apply the new policy at the system level. switch(config)# system qos switch(config)# service-policy type qos input roce switch(config)# service-policy type queuing output roce...
  • Page 71 6–Configuring RoCE Preparing the Ethernet Switch Verify the DCB map configuration. Priority group Dell(conf-dcbmap-roce)# do show qos dcb-map roce ----------------------- State :Complete PfcMode :ON -------------------- PG:0 TSA:ETS BW:40 PFC:OFF Priorities:0 1 2 3 4 6 7 PG:1 TSA:ETS BW:60 PFC:ON Priorities:5 Apply the DCB map to the port.

  • Page 72: Configuring Roce On The Adapter For Windows Server

    Network Direct MTU size. To configure RoCE on a Windows Server host: Enable RoCE on the adapter. Open the Windows Device Manager, and then open the QL45212 NDIS Miniport Properties. Select Network Direct Functionality from the Property menu, and then select Enabled in the Value field.
  • Page 73 6–Configuring RoCE Configuring RoCE on the Adapter for Windows Server Select Network Direct Mtu Size from the Property menu. The network direct MTU size must be less than the Ethernet jumbo packet size. In this example, select 1024. Click OK. Select RoCE Mode from the Property menu.
  • Page 74 InterfaceDescription Enabled ----- -------------------- ------- SLOT 4 3 Port 1 QLogic FastLinQ QL45212... True SLOT 4 3 Port 2 QLogic FastLinQ QL45212... True Verify that RoCE is enabled on the host operating system using PowerShell. The Get-NetOffloadGlobalSetting command shows NetworkDirect is enabled.
  • Page 75 6–Configuring RoCE Configuring RoCE on the Adapter for Windows Server Select <Local computer> under Select counters from computer:. Select Add RDMA Activity, and then select <All instances> under Instances of selected object;. The following figure shows the presence of RDMA traffic. BC0154503-00 C...
  • Page 76 6–Configuring RoCE Configuring RoCE on the Adapter for Windows Server Verify the SMB connection using Windows PowerShell. Type the net use and netstat -xan commands to show Share1 is mapped as SMB share. PS C:\Users\Administrator> net use Status Local Remote Network ------------------------------------------------------------- \\192.168.10.10\Share1 Microsoft Window...

  • Page 77: Configuring Roce On The Adapter For Linux

    6–Configuring RoCE Configuring RoCE on the Adapter for Linux Configuring RoCE on the Adapter for Linux This section describes the RoCE configuration procedure for RHEL and SLES. It also describes how to verify the RoCE configuration and provides some guidance about using global IDs (GIDs) with VLAN interfaces.

  • Page 78: Sys And Class Parameters

    6–Configuring RoCE Configuring RoCE on the Adapter for Linux Verifying RoCE v1 or v2 GID Index, and Address from sys and class Parameters Use one of the following options to verify RoCE v1 or v2 GID Index and address from the sys and class parameters below: ...

  • Page 79: Verifying Roce V1 Or V2 Functionality Through Perftest Applications

    6–Configuring RoCE Configuring RoCE on the Adapter for Linux Verifying RoCE v1 or v2 Functionality Through perftest Applications Use this section to verify RoCE v1 or v2 functionality through perftest applications. In this example, the following Server IP and Client IP will be used: ...
  • Page 80 6–Configuring RoCE Configuring RoCE on the Adapter for Linux  Client Configuration #/sbin/ip link add link p4p1 name p4p1.101 type vlan id 101 #ifconfig p4p1.101 192.168.101.3/24 up #ip route add 192.168.100.0/24 via 192.168.101.1 dev p4p1.101 Set the switch settings using the following procedure. Use any flow control method (Pause,DCBX-CEE/DCBX-IEEE), and enable IP routing for RoCE v2.
  • Page 81 6–Configuring RoCE Configuring RoCE on the Adapter for Linux Client: Configuring RoCE v1 or v2 Settings for RDMA_CM Applications Use the following scripts from fastlinq-minor source package: #./show_rdma_cm_roce_ver.sh qedr0 is configured to IB/RoCE v1 qedr1 is configured to IB/RoCE v1 # ./config_rdma_cm_roce_ver.sh v2 configured rdma_cm for qedr0 to RoCE v2 configured rdma_cm for qedr1 to RoCE v2...

  • Page 82: Roce Configuration For Rhel

    6–Configuring RoCE Configuring RoCE on the Adapter for Linux Client: RoCE Configuration for RHEL To configure RoCE on the adapter, the Open Fabrics Enterprise Distribution (OFED) must be installed and configured on the RHEL host. To prepare in-box OFED for RHEL: While installing or upgrading the operating system, select the Infiniband and OFED support packages.

  • Page 83: Roce Configuration For Sles

    6–Configuring RoCE Configuring RoCE on the Adapter for Linux RoCE Configuration for SLES To configure RoCE on the adapter for an SLES host, OFED must be installed and configured on the SLES host. To install in-box OFED for SLES Linux: While installing or upgrading the operating system, select the Infiniband support packages.
  • Page 84 6–Configuring RoCE Configuring RoCE on the Adapter for Linux ib_ucm,ib_iser,ib_srpt,ib_umad, ib_uverbs,rdma_ucm,ib_ipoib,ib_isert Configure the IP address and enable the port using a configuration method, such as ifconfig: # ifconfig ethX 192.168.10.10/24 up Type the ibv_devinfo command. For each PCI function, you should see a separate hca_id, as shown in the following example: root@captain:~# ibv_devinfo hca_id: qedr0...
  • Page 85 6–Configuring RoCE Configuring RoCE on the Adapter for Linux The following are examples of successful ping pong tests on the server and the client: Server: root@captain:~# ibv_rc_pingpong -d qedr0 -g 0 local address: LID 0x0000, QPN 0xff0000, PSN 0xb3e07e, GID fe80::20e:1eff:fe50:c7c0 remote address: LID 0x0000, QPN 0xff0000, PSN 0x934d28, GID fe80::20e:1eff:fe50:c570...

  • Page 86: Vlan Interfaces And Gid Index Values

    6–Configuring RoCE Configuring RoCE on the Adapter for Linux VLAN Interfaces and GID Index Values If you are using VLAN interfaces on both server and client, then configure the same VLAN ID on the switch. If you are running traffic through a switch, the Infiniband applications must use the correct GID value, which is based on the VLAN ID and VLAN IP address.

  • Page 87: Iscsi Extensions For Rdma

    iSCSI Extensions for RDMA This chapter describes the procedure for configuring iSCSI Extensions for RDMA (iSER) for RHEL and SLES.  Configuring iSER for RHEL  Configuring iSER for SLES12 Linux  Optimizing Linux Performance As you prepare to configure iSER, consider the following: ...

  • Page 88: Configuring Iser For Rhel

    7–iSCSI Extensions for RDMA Configuring iSER for RHEL Configuring iSER for RHEL To configure iSER for RHEL: Configure in-box OFED, as described “RoCE Configuration for RHEL” on page 64. Out-of-box OFEDs are not supported for iSER because the ib_isert module is not available in the out-of-box OFED-3.18/3.81-1 GA versions. The inbox ib_isert module does not work with any out-of-box OFED versions.
  • Page 89 7–iSCSI Extensions for RDMA Configuring iSER for RHEL The following figure shows an example of a successful RDMA ping. You can use a Linux TCM-LIO target with which to test iSER. The setup is the same for any iSCSI target, except you type command enable_iser Boolean=true on the applicable portals.
  • Page 90 7–iSCSI Extensions for RDMA Configuring iSER for RHEL Connect to or login to the iSER target by typing a command such as the following: iscsiadm -m node -l -p 192.168.100.99:3260 -T iqn.2015-06.test.target1 Confirm that the Iface Transport is iser in the target connection, as shown in the following figure.

  • Page 91: Configuring Iser For Sles12 Linux

    7–iSCSI Extensions for RDMA Configuring iSER for SLES12 Linux Configuring iSER for SLES12 Linux The TargetCLI is not in-box on SLES12, therefore you must do the following: To install TargetCLI, copy and install the following RPMs from ISO image (x86_64 and noarch location). lio-utils-4.1-14.6.x86_64.rpm python-configobj-4.7.2-18.10.noarch.rpm python-PrettyTable-0.7.2-8.5.noarch.rpm...

  • Page 92: Optimizing Linux Performance

    7–iSCSI Extensions for RDMA Optimizing Linux Performance Optimizing Linux Performance Consider the Linux performance enhancements. Set CPUs to Maximum Performance Mode Set the CPU scaling governor to performance by using the following script to set all CPUs to maximum Performance mode for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor;...

  • Page 93: Irq Affinity Settings

    7–iSCSI Extensions for RDMA Optimizing Linux Performance IRQ Affinity Settings This example sets CPU core 0, 1, 2 and 3 to IRQ XX, YY, ZZ and XYZ respectively. This will need to be done for each IRQ assigned to a port (default is eight queues per port) systemctl disable irqbalance systemctl stop irqbalance...

  • Page 94: Troubleshooting

    Troubleshooting This chapter provides information about the following topics:  Troubleshooting Checklist  Verifying that Current Drivers are Loaded  Testing Network Connectivity Microsoft Virtualization with Hyper-V   Linux  Miscellaneous  Collecting Debug Data BC0154503-00 C...

  • Page 95: Troubleshooting Checklist

    “Safety Precautions” on page The following checklist provides recommended actions to resolve problems that may arise while installing the QL45212 Adapter or running it in your system.  Inspect all cables and connections. Verify that the cable connections at the network adapter and the switch are attached properly.

  • Page 96: Verifying That Current Drivers Are Loaded

    In this example, the last entry identifies the driver that will be active upon reboot. # dmesg | grep -i "QLogic" | grep -i "qede" 10.097526] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x 23.093526] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x 34.975396] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x...

  • Page 97: Testing Network Connectivity

    8–Troubleshooting Testing Network Connectivity Testing Network Connectivity NOTE When using forced link speeds, verify that both the adapter and the switch are forced to the same speed. Network Connectivity Testing for Windows Network connectivity can be tested using the ping command to determine if the network connection is working.

  • Page 98: Network Connectivity Testing For Linux

    2012 R2. For more information on Hyper-V, see https://technet.microsoft.com/en-us/library/Dn282278.aspx. Linux Problem: QL45212 devices with SFP+ Flow Control default to Off rather than Rx/Tx Enable. Solution: The Flow Control default setting for revision 1.6.x and newer has been changed to Rx Off and Tx Off because SFP+ devices do not support auto-negotiation for Flow Control.

  • Page 99: Collecting Debug Data

    8–Troubleshooting Collecting Debug Data Collecting Debug Data Use the information in table x.x to collect debug data. Table 8-1. Collecting Debug Data Commands Debug Data Description Kernel logs demesg-T Register dump. ethtool-d System information. This is available in the driver bundle. sys_info.sh BC0154503-00 C...

  • Page 100: Adapter Leds

    Adapter LEDS Network link and activity status are indicated by the Link and Activity LEDs located adjacent to the port connector, as described in Table A-1. Table A-1. Port LED Indications Port LED LED Appearance Network State Link LED No link (cable discon- nected) Continuously illuminated Link...
  • Page 101 However, until the auto-negotiation process is standardized, the switch must be explicitly configured to connect to the adapter at 25Gbps. To configure a Dell Z9100 switch port to connect to the QL45212 Adapter at 25Gbps: Establish a serial port connection between your management workstation and the switch.
  • Page 102 B–Dell Z9100 Switch Configuration Identify the module and port to be configured. This example uses module 1, port 5. Dell(conf)#stack-unit 1 port 5 ? portmode Set portmode for a module Dell(conf)#stack-unit 1 port 5 portmode ? dual Enable dual mode quad Enable quad mode single...
  • Page 103 B–Dell Z9100 Switch Configuration Verify that auto-negotiation is disabled. Dell(conf-if-tf-1/5/1)#do show run interface tw 1/5/1 interface twentyFiveGigE 1/5/1 no ip address mtu 9416 switchport flowcontrol rx on tx on no shutdown no intf-type cr4 autoneg For more information about configuring the DZ9100 switch, refer to the Dell Z9100 Switch Configuration Guide at support.dell.com.
  • Page 104 Glossary adapter data center bridging (DCB) The board that interfaces between the Provides enhancements to existing 802.1 host system and the target devices. bridge specifications to satisfy the require- Adapter is synonymous with Host Bus ments of protocols and applications in the Adapter, host adapter, and board.
  • Page 105 A specification that defines a software interface between an operating system NIC partitioning (NPAR) and platform firmware. EFI is a replace- On QLogic Converged Network Adapters ment for the older BIOS firmware interface and Intelligent Ethernet adapters, refers to present in all IBM PC-compatible personal the partitioning of a single NIC port into up computers.
  • Page 106 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 quality of service (QoS) unified extensible firmware interface (UEFI) Refers to the methods used to prevent A specification detailing an interface that bottlenecks and ensure business conti- helps hand off control of the system for the...
  • Page 107 Index Activity LED data center bridging exchange, See DCBX. adapter DCB map 52, enable RoCE DCBX installation enable LEDs protocol management debug parameter 17, properties (Windows) Dell update package adaptive interrupt frequency command line address, MAC address, PCI device architecture auto_fw_reset parameter name auto-negotiation...
  • Page 108 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 driver removal Linux, non-RoCE IEEE protocol Linux, RoCE in-box OFED VMware information level Windows int_mode parameter interrupt frequency enable_vxlan_offld parameter jumbo packet enhanced transmission selection, See ETS. Extensible Firmware Interface, See EFI. kmod RPM package...
  • Page 109 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 messages priority MSI-X RoCE MTU size 17, 30, priority-based flow control, See PFC. multi_rx_filters parameter processor, RISC network qed driver 10, connectivity 79, qede driver 10, installations qedr driver Nexus 6000 Ethernet switch NIC partitioning, See NPAR.
  • Page 110 User’s Guide—25Gb Intelligent Ethernet Adapter QL45212 SFP+ Flow Control VxLAN single root input/output virtualization, See filtering SR-IOV. tunneled traffic SMB drive vxlan_filter_en parameter specifications speed src RPM package SR-IOV wake on LAN, See WoL. standards Windows Server operating system statistics, driver...
  • Page 112 International Offices UK | Ireland | Germany | France | India | Japan | China | Hong Kong | Singapore | Taiwan | Israel © 2016-2017 QLogic Corporation. QLogic Corporation is a wholly owned subsidiary of Cavium, Inc. All rights reserved worldwide. QLogic and the QLogic logo are registered trademarks of QLogic Corporation.

Table of Contents