Broadcom NetXtreme-C User Manual
  1. Manuals
  2. Brands
  3. Broadcom Manuals
  4. Recording Equipment
  5. NetXtreme-C
  6. User manual

Broadcom NetXtreme-C User Manual

Interface card
Hide thumbs
Table of Contents

Advertisement

Quick Links

®
NetXtreme
-C/NetXtreme-E
User Guide
NetXtreme-UG100
August 23, 2018

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the NetXtreme-C and is the answer not in the manual?

Questions and answers

Related Manuals for Broadcom NetXtreme-C

Summary of Contents for Broadcom NetXtreme-C

  • Page 1 ® NetXtreme -C/NetXtreme-E User Guide NetXtreme-UG100 August 23, 2018...
  • Page 2 The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. For more information, please visit www.broadcom.com. Broadcom reserves the right to make changes without further notice to any products or data herein to improve reliability, function, or design. Information furnished by Broadcom is believed to be accurate and reliable. However, Broadcom does not assume any liability arising out of the application or use of this information, nor the application or use of any product or circuit described herein, neither does it convey any license under its patent rights nor the rights of others.

  • Page 3: Table Of Contents

    NetXtreme-C/NetXtreme-E User Guide Table of Contents 1 Regulatory and Safety Approvals..........................7 1.1 Regulatory................................7 1.2 Safety...................................7 1.3 Electromagnetic Compatibility (EMC) ........................7 1.4 Electrostatic Discharge (ESD) Compliance ......................8 1.5 FCC Statement ..............................8 2 Functional Description ..............................8 3 Network Link and Activity Indication .........................8 4 Features ..................................9...
  • Page 4 NetXtreme-C/NetXtreme-E User Guide 5.4.1 Supported Cables and Modules ......................16 5.4.1.1 Copper................................16 5.4.1.2 SFP+ ................................16 5.4.1.3 SFP28 ................................16 5.4.1.4 QSFP................................16 6 Software Packages and Installation .........................17 6.1 Supported Operating Systems ...........................17 6.2 Installing the Linux Driver...........................17 6.2.1 Linux Ethtool Commands.........................17 6.3 Installing the VMware Driver ..........................18...
  • Page 5 NetXtreme-C/NetXtreme-E User Guide 10.2.2 Configuring iSCSI Boot Parameters ......................29 10.2.3 MBA Boot Protocol Configuration ......................30 10.2.4 iSCSI Boot Configuration ........................30 10.2.4.1 Static iSCSI Boot Configuration ........................30 10.2.4.2 Dynamic iSCSI Boot Configuration ....................... 31 10.2.5 Enabling CHAP Authentication ......................32 10.3 Configuring the DHCP Server to Support iSCSI Boot..................33 10.3.1 DHCP iSCSI Boot Configurations for IPv4 ....................33...
  • Page 6 NetXtreme-C/NetXtreme-E User Guide 14.3.4.2 Tagging vmknic for PVRDMA on ESX Hosts ....................48 14.3.4.3 Setting the Firewall Rule for PVRDMA ......................48 14.3.4.4 Adding a PVRDMA Device to the VM ......................48 14.3.4.5 Configuring the VM on Linux Guest OS ......................48 15 DCBX –...

  • Page 7: Regulatory And Safety Approvals

    NetXtreme-C/NetXtreme-E User Guide 1 Regulatory and Safety Approvals The following sections detail the Regulatory, Safety, Electromagnetic Compatibility (EMC), and Electrostatic Discharge ® (ESD) standard compliance for the NetXtreme -C/NetXtreme-E Network Interface Card. 1.1 Regulatory Table 1: Regulatory Approvals Item Applicable Standard...

  • Page 8: Electrostatic Discharge (Esd) Compliance

    2 Functional Description The Broadcom NetXtreme-C (BCM573XX) and NetXtreme-E (BCM574XX) family of Ethernet Controllers are highly- integrated, full-featured Ethernet LAN controllers optimized for data center and cloud infrastructures. Adapters support 100G/50G/40G/25G/10G/1G in both single and dual-port configurations. On the host side, these devices support sixteen lanes of a PCIe Generation 3 interface.

  • Page 9: Features

    NetXtreme-C/NetXtreme-E User Guide 4 Features Refer to the following sections for device features. 4.1 Software and Hardware Features Table 6 provides a list of host interface features. Table 6: Host Interface Features Feature Details Host Interface PCIe 3.0 (Gen 3: 8 GT/s; Gen 2: 5 GT/s; Gen 1: 2.5 GT/s).

  • Page 10: Virtualization Features

    IEEE 802.1q VLAN Supported. Interrupt Moderation Supported. MAC/VLAN filters Supported. 4.2 Virtualization Features Table 7 lists the virtualization features of the NetXtreme-C/NetXtreme-E. Table 7: Virtualization Features Feature Details Linux KVM Multiqueue Supported. VMware NetQueue Supported. NDIS Virtual Machine Queue (VMQ) Supported.

  • Page 11: Vxlan

    NetXtreme-C/NetXtreme-E User Guide Table 7: Virtualization Features (Continued) Feature Details MSI-X vector port 74 per port default value (two port configuration). 16 per VF and is configurable in HII and CCM. 4.3 VXLAN A Virtual eXtensible Local Area Network (VXLAN), defined in IETF RFC 7348, is used to address the need for overlay networks within virtualized data centers accommodating multiple tenants.

  • Page 12: Stateless Transport Tunnel Offload

    NetXtreme-C/NetXtreme-E User Guide The UFO enables the NIC to handle fragmentation of a UDP datagram into UDP/IP packets. This can result in the  reduction of CPU overhead for transmit side UDP/IP processing. 4.7 Stateless Transport Tunnel Offload Stateless Transport Tunnel Offload (STT) is a tunnel encapsulation that enables overlay networks in virtualized data centers.

  • Page 13: Network Partitioning (Npar)

    RoCE Physical Functions (PF) and SRIOV Virtual Functions (VF) are available in single function mode and in mutli-function mode (NIC Partitioning mode). Broadcom supports RoCE in Windows, Linux, and VMware. Refer to the following links for RDMA support for each operating system: Windows https://technet.microsoft.com/en-us/library/jj134210(v=ws.11).aspx...

  • Page 14: Npar, Sr-Iov, And Dpdk

    NetXtreme-C/NetXtreme-E User Guide Table 8: NPAR, SR-IOV, and RoCE (Continued) SW Feature Notes RoCE on PFs Up to 4 PFs RoCE on VFs Valid for VFs attached to RoCE-enabled PFs Host OS Linux, Windows, ESXi (no vRDMA support) Guest OS...

  • Page 15: Installing The Hardware

     personal antistatic devices and an antistatic mat is strongly recommended. 5.2 System Requirements Before installing the Broadcom NetXtreme-E Ethernet adapter, verify that the system meets the requirements listed for the operating system. 5.2.1 Hardware Requirements Refer to the following list of hardware requirements: One open PCIe Gen 3 x8 or x 16 slot.

  • Page 16: Installing The Adapter

    5.4 Connecting the Network Cables Broadcom Ethernet switches are productized with SFP+/SFP28/QSFP28 ports that support up to 100 Gb/s. These 100 Gb/ s ports can be divided into 4 x 25 Gb/s SFP28 ports. QSFP ports can be connected to SFP28 ports using 4 x 25G SFP28 breakout cables.

  • Page 17: Software Packages And Installation

    SLES 11 SP 4, SLES 12 SP 2 or above VMware ESXi 6.0 U3 or above 6.2 Installing the Linux Driver Linux drivers can be downloaded from the Broadcom public website: https://www.broadcom.com/support/download-search/ ?pg=Ethernet+Connectivity,+Switching,+and+PHYs&pf=Ethernet+Network+Adapters+-+NetXtreme&pa=Driver. See the package readme.txt files for specific instructions and optional parameters.

  • Page 18: Installing The Vmware Driver

    NetXtreme-C/NetXtreme-E User Guide Table 11: Linux Ethtool Commands (Continued) Command Description ethtool --set-eee ethX eee on tx-lpi off Enable EEE, but disable LPI. ethtool -L ethX combined 1 rx 0 tx 0 Disable RSS. Set the combined channels to 1.

  • Page 19: Installing The Windows Driver

    2. Unzip the Wiin20xx_2xx.xx.x.zip file. 3. Launch the Device Manager. 4. Right-click on the Broadcom devices under Network Adapters. 5. Select Update Driver. 6. Select Browse My Computer For Driver Software and navigate to the folder where the driver files located. The driver l updates automatically.

  • Page 20: Event Log Messages

    NetXtreme-C/NetXtreme-E User Guide Table 13: Windows Driver Advanced Properties (Continued) Driver Key Parameters Description Max Number of RSS Queues 2, 4, or 8. Default is 8. Allows user to configure Receive Side Scaling queues. Priority and VLAN Priority and VLAN Disable, Priority Default Enabled.

  • Page 21: Updating The Firmware

    NetXtreme-C/NetXtreme-E User Guide Table 14: Windows Event Log Messages (Continued) Message ID Comment 0x0014 Incompatible firmware version. 0x0015 Single interrupt. Table 15: Event Log Messages 0x0016 Firmware failed to respond within allocated time. 0x0017 Firmware returned failure status. 0x0018 Firmware is in unknown state.

  • Page 22: Windows/Esx

    8 Teaming 8.1 Windows The Broadcom NetXtreme-C/NetXtreme-E devices can participate in NIC teaming functionality using the Microsoft teaming solution. Refer to Microsoft public documentation described in the following link: https://docs.microsoft.com/en-us/windows-server/networking/technologies/nic-teaming/create-a-new-nic-team Microsoft LBFO is a native teaming driver that can be used in the Windows OS. The teaming driver also provides VLAN tagging capabilities.

  • Page 23: System-Level Configuration

    The following sections provide information on system-level NIC configuration. 9.1 UEFI HII Menu Broadcom NetXtreme-E series controllers can be configured for pre-boot, iSCSI and advanced configuration such as SR- IOV using HII (Human Interface) menu. To configure the settings, during system boot, enter BIOS setup and navigate to the network interface control menus.

  • Page 24: Comprehensive Configuration Management

    AN. Ensure that the link partner port has been set to the matching auto-negotiation protocol. For example, if the local Broadcom port is set to IEEE 802.3by AN protocol, the link partner must support AN and must be set to IEEE 802.3by AN protocol.
  • Page 25 NetXtreme-C/NetXtreme-E User Guide The supported combination of link speed settings for two ports NetXtreme-E network controller are shown in Table Table 16: Supported Combination of Link Speed Settings Port 2 Link Setting Port1 Link Speed AN Enabled AN Enabled AN Enabled {1/...
  • Page 26 NetXtreme-C/NetXtreme-E User Guide The expected link speeds based on the local and link partner settings are shown in Table Table 17: Expected Link Speeds Link Partner Speed Settings Local Speed Forced Forced AN Enabled Enabled Enabled AN Enabled Enabled {1/...

  • Page 27: Operational Link Speed

    This is the supported auto-negotiation protocol used to negotiate the link speed with the link partner. This option must match the AN protocol setting in the link partner port. The Broadcom NetXtreme-E NIC supports the following auto-negotiation protocols: IEEE 802.3by, 25G/50G consortiums and 25G/50G BAM. By default, this option is set to IEEE 802.3by.

  • Page 28: Linux Driver Settings

    10 iSCSI Boot Broadcom NetXtreme-E Ethernet adapters support iSCSI boot to enable the network boot of operating systems to diskless systems. iSCSI boot allows a Windows, Linux, or VMware operating system to boot from an iSCSI target machine located remotely over a standard IP network.

  • Page 29: Setting Up Iscsi Boot

    – Target LUN – Initiator IQN – CHAP ID and secret 10.2.2 Configuring iSCSI Boot Parameters Configure the Broadcom iSCSI boot software for either static or dynamic configuration. Refer to Table 18 for configuration options available from the General Parameters menu.

  • Page 30: Mba Boot Protocol Configuration

    NetXtreme-C/NetXtreme-E User Guide Table 18: Configuration Options (Continued) Option Description Link Up Delay Time Controls how long the iSCSI boot host software waits, in seconds, after an Ethernet link is established before sending any data over the network. The valid values are 0 to 255.

  • Page 31: Dynamic Iscsi Boot Configuration

    NetXtreme-C/NetXtreme-E User Guide 2. Select ESC to return to the Main menu. 3. From the Main menu, select Initiator Parameters. 4. From the Initiator Parameters screen, enter values for the following: – IP Address (unspecified IPv4 and IPv6 addresses should be "0.0.0.0" and "::", respectively) –...

  • Page 32: Enabling Chap Authentication

    NetXtreme-C/NetXtreme-E User Guide iqn.1995-05.com.broadcom.<11.22.33.44.55.66>.iscsiboot where the string 11.22.33.44.55.66 corresponds to the controller's MAC address. If DHCP option 43 (IPv4 only) is used, then any settings on the Initiator Parameters, 1st Target Parameters, or 2nd  Target Parameters screens are ignored and do not need to be cleared.

  • Page 33: Configuring The Dhcp Server To Support Iscsi Boot

    10.3.1 DHCP iSCSI Boot Configurations for IPv4 The DHCP protocol includes a number of options that provide configuration information to the DHCP client. For iSCSI boot, Broadcom adapters support the following DHCP configurations: 10.3.1.1 DHCP Option 17, Root Path Option 17 is used to pass the iSCSI target information to the iSCSI client. The format of the root path as defined in IETC RFC 4173 is: iscsi:"<servername>":"<protocol>":"<port>":"<LUN>":"<targetname>...

  • Page 34: Dhcp Option 43, Vendor-Specific Information

    DHCPv6 client. For iSCSI boot, Broadcom adapters support the following DHCP configurations: NOTE: The DHCPv6 standard Root Path option is not yet available. Broadcom suggests using Option 16 or Option 17 for dynamic iSCSI Boot IPv6 support. 10.3.2.1 DHCPv6 Option 16, Vendor Class Option DHCPv6 Option 16 (vendor class option) must be present and must contain a string that matches the configured DHCP Vendor ID parameter.

  • Page 35: Configuring The Dhcp Server

    NetXtreme-C/NetXtreme-E User Guide Table 21: DHCP Option 17 Suboption Definition Suboption Definition First iSCSI target information in the standard root path format "iscsi:"[<servername>]":"<protocol>":"<port>":"<LUN>":"<targetname>" iSCSI initiator IQN NOTE: Table 21, the brackets [ ] are required for the IPv6 addresses. The content of option 17 should be <2-byte Option Number 201|202|203> <2-byte length> <data>.

  • Page 36: Sr-Iov: Configuration And Use Case Examples

    NetXtreme-C/NetXtreme-E User Guide 12 SR-IOV: Configuration and Use Case Examples SR-IOV can be configured, enabled, and used on 10-Gb and 25-Gb Broadcom NetExtreme-E NICs. 12.1 Linux Use Case Example 1. Enable SR-IOV in the NIC cards: a. SR-IOV in the NIC card can be enabled using the HII menu. During system boot, access the system BIOS ->...

  • Page 37: Windows Use Case Example

    NetXtreme-C/NetXtreme-E User Guide 8. Use the Virtual Manager to install a Virtualized Client system (VMs). Refer to the Linux documentation for Virtual Manager installation. Ensure that the hypervisor’s built in driver is removed. An example would be . Remove this driver.

  • Page 38: Vmware Sriov Use Case Example

    $ esxcli software vib list | grep bnxtnet 4. Install the Broadcom provided BNXTNETCLI (esxcli bnxtnet) utility to set/view the miscellaneous driver parameters that are not natively supported in esxcli, such as: link speed to 25G, show driver/firmware chip information, show NIC configuration (NPAR, SRIOV).

  • Page 39: Npar - Configuration And Use Case Example

    NetXtreme-C/NetXtreme-E User Guide $ esxcli bnxtnet link set -S <speed> -D <full> -n <iface> This returns an OK message if the speed is correctly set. Example: $ esxcli bnxtnet link set -S 25000 -D full -n vmnic5 e. Show the link stats $ esxcli bnxtnet link get -n vmnic6 f.

  • Page 40: Limitations

    In NPAR + SRIOV mode, only two VFs from each parent physical port can enable RDMA support, or total of four VFs + RDMA per physical device. 13.3 Configuration NPAR can be configured using BIOS configuration HII menus or by using the Broadcom CCM utility on legacy boot systems. Some vendors also expose the configuration via additional proprietary interfaces. To enable NPAR: 1.
  • Page 41 NetXtreme-C/NetXtreme-E User Guide 2. Once NPAR is enabled, the NIC Partitioning Main Configuration Menu option is available from the main NIC Configuration Menu associated with each physical port. 3. The NIC Partition Configuration Menu (shown below) allows the user to choose the number of partitions that should be allocated from the selected physical port.

  • Page 42: Notes On Reducing Nic Memory Consumption

    NetXtreme-C/NetXtreme-E User Guide 4. Set the NIC Partition Configuration parameters (see Table 24 on page 42). Table 24: NPAR Parameters Parameter Description Valid Options BW Reservation Percentage of total available bandwidth that should be reserved for this partition. 0 Value 0-100 indicates equal division of bandwidth between all partitions.

  • Page 43: Roce - Configuration And Use Case Examples

    NetXtreme-C/NetXtreme-E User Guide The parameters “Maximum Number of RSS Queues” and “Receive Buffers (0=Auto)” can be modified using the Advanced properties tab for each NIC in the Device Manager. If you want to modify multiple NICs at the same time, it is faster to use the Set-NetAdapterAdvancedProperty PowerShell cmdlet.

  • Page 44: Installation

    NetXtreme-C/NetXtreme-E User Guide NOTE: It is necessary to load the correct bnxt_en version that is included in the same netxtreme-bnxt_en-1.7.x.tar.gz package. Bnxt_re and Bnxt_en function as a pair to enable RoCE traffic. Using mismatching versions of these two drivers produces unreliable or unpredictable results.

  • Page 45: Limitations

    Windows Server 2012 and beyond invokes the RDMA capability in the NIC for SMB file traffic if both ends are enabled for RDMA. Broadcom NDIS miniport bnxtnd.sys v20.6.2 and beyond support RoCEv1 and RoCEv2 via the NDKPI interface. The default setting is RoCEv1.

  • Page 46: User Mode

    NetXtreme-C/NetXtreme-E User Guide If both Client and Server show True, then any file transfers over this SMB connection use SMB. 3. The following commands can be used to enable/disable SMB Multichannel: Server Side: – Enable: Set-SmbServerConfiguration -EnableMultiChannel $true – Disable: Set-SmbServerConfiguration -EnableMultiChannel $false Client Side: –...

  • Page 47: Vmware Esx Configuration And Use Case Examples

    NetXtreme-C/NetXtreme-E User Guide 14.3 VMware ESX Configuration and Use Case Examples 14.3.1 Limitations The current version of the RoCE supported driver requires ESXi-6.5.0 GA build 4564106 or above. 14.3.2 BNXT RoCE Driver Requirements The BNXTNET L2 driver must be installed with the module parameter before installing the driver.

  • Page 48: Tagging Vmknic For Pvrdma On Esx Hosts

    NetXtreme-C/NetXtreme-E User Guide 14.3.4.2 Tagging vmknic for PVRDMA on ESX Hosts To tag a vmknic for PVRDMA to use on ESX hosts: 1. Select the host and right-click on Settings to switch to the settings page of the Manage tabs.
  • Page 49 NetXtreme-C/NetXtreme-E User Guide NOTE: The installation path of the library needs to be in the shared library cache. Follow the instructions in the INSTALL file in the library's directory. NOTE: The firewall settings may need to be modified to allow RDMA traffic. Ensure the proper firewall settings are in place.

  • Page 50: Dcbx - Data Center Bridging

    15 DCBX – Data Center Bridging Broadcom NetXtreme-E controllers support IEEE 802.1Qaz DCBX as well as the older CEE DCBX specification. DCB configuration is obtained by exchanging the locally configured settings with the link peer. Since the two ends of a link may be configured differently, DCBX uses a concept of 'willing' to indicate which end of the link is ready to accept parameters from the other end.

  • Page 51: Dcbx Mode - Enable (Ieee Only)

    After enabling the indicated options in the UEFI HII menu to set firmware level settings, perform the follow selection in the Windows driver advanced properties. Open Windows Manager → Broadcom NetXtreme E Series adapter → Advanced Properties → Advanced tab Quality of Service = Enabled Priority &...
  • Page 52 NetXtreme-C/NetXtreme-E User Guide Use the following in modifying the Traffic Class: PS C:\> Set-NetQoSTrafficClass -Name "SMB class" -BandwidthPercentage 40 PS C:\> get-NetQosTrafficClass Name Algorithm Bandwidth(%) Priority ------------------------------------------------------------------ [Default] ETS 60 0-3,5-7 SMB class ETS Use the following to remove the Traffic Class: PS C:\>...
  • Page 53 NetXtreme-C/NetXtreme-E User Guide Use the following to create QoS Policy based on the Source/Destination Address: PS C:\> New-NetQosPolicy "Network Management" -IPDstPrefixMatchCondition 10.240.1.0/24 - IPProtocolMatchCondition both -NetworkProfile all -PriorityValue8021Action 7 Name : Network Management Owner : Group Policy (Machine) Network Profile : All...

  • Page 54: Dpdk - Configuration And Use Case Examples

    4. The PCI device information is displayed by lspci with the following commands: [root@localhost /]# lspci |grep Eth 3b:00.0 Ethernet controller: Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01) 3b:00.1 Ethernet controller: Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01) 16.2 Running the Application...

  • Page 55: Testpmd Runtime Functions

    NetXtreme-C/NetXtreme-E User Guide 16.3 Testpmd Runtime Functions When the testpmd application is started in interactive mode, (-i|--interactive), it displays a prompt that can be used to start and stop forwarding, configure the application, display statistics (including the extended NIC statistics aka xstats), set the Flow Director, and other tasks.

  • Page 56: Configuration Functions

    NetXtreme-C/NetXtreme-E User Guide 16.6 Configuration Functions The testpmd application can be configured from the runtime as well as from the command line. This section describes the available configuration functions that are available. csum set – Selects the hardware or software calculation of the checksum when transmitting a packet using the csum forwarding engine: testpmd>...

  • Page 57: Revision History

    NetXtreme-C/NetXtreme-E User Guide Revision History NetXtreme-UG100; August 23, 2018 Initial Release. NetXtreme-UG100...

This manual is also suitable for:

Netxtreme-e

Table of Contents