Solarflare Ultra SFN7142Q User Manual

page of 375

/ 375
Contents
Table of Contents
Bookmarks

Table of Contents

Quick Links

Solarflare® Server Adapter User Guide

• Introduction...Page 1

• Installation...Page 18

• Solarflare Adapters on Linux...Page 39

• Solarflare Adapters on Windows...Page 109

• Solarflare Adapters on VMware...Page 236

• Solarflare Adapters on Solaris...Page 263

• SR-IOV Virtualization Using KVM...Page 307

• SR-IOV Virtualization for XenServer...Page 324

• Solarflare Adapters on Mac 0S X...Page 334

• Solarflare Boot ROM Agent...Page 344

Information in this document is subject to change without notice.

Trademarks used in this text are registered trademarks of Solarflare® Communications Inc; Adobe is

a trademark of Adobe Systems. Microsoft® and Windows® are registered trademarks of Microsoft

Corporation.

Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.

Other trademarks and trade names may be used in this document to refer to either the entities

claiming the marks and names or their products. Solarflare Communications Inc. disclaims any

proprietary interest in trademarks and trade names other than its own.

SF-103837-CD

Last revised: July 2014

Issue 11

Solarflare Server Adapter

User Guide

Table of Contents

Need help?

Do you have a question about the Ultra SFN7142Q and is the answer not in the manual?

Questions and answers

Summary of Contents for Solarflare Ultra SFN7142Q

Page 1 Information in this document is subject to change without notice. © 2008-2014 Solarflare Communications Inc. All rights reserved. Trademarks used in this text are registered trademarks of Solarflare® Communications Inc; Adobe is a trademark of Adobe Systems. Microsoft® and Windows® are registered trademarks of Microsoft Corporation.
Page 2: Table Of Contents
2.1 Solarflare Network Adapter Products ........
Page 3 Chapter 4: Solarflare Adapters on Windows ........
Page 4 Chapter 5: Solarflare Adapters on VMware ........
Page 5 Chapter 9: Solarflare Adapters on Mac 0S X ....... . .
Page 6: Chapter 1: Introduction
Onload to achieve minimum latency and maximum throughput. 1.1 Virtual NIC Interface Solarflare’s VNIC architecture provides the key to efficient server I/O and is flexible enough to be applied to multiple server deployment scenarios. These deployment scenarios include: •...
Page 7 Advanced Features and Benefits Virtual NIC support The core of Solarflare technology. Protected VNIC interfaces can be instantiated for each running guest operating system or application, giving it a direct pipeline to the Ethernet network. This architecture provides the most efficient way to maximize network and CPU efficiency.
Page 8 Flexible deployment of 1024 channels between Virtual and Physical Functions. Support Alternate Routing ID (ARI). SR-IOV is not supported for Solarflare adapters on IBM System p servers. 10-gigabit Ethernet Supports the ability to design a cost effective, high performance 10 Gigabit Ethernet solution.
Page 9: Product Specifications
Solarflare Server Adapter User Guide 1.2 Product Specifications Solarflare Flareon™ Network Adapters Solarflare Flareon™ Ultra SFN7142Q Dual-Port 40GbE QSFP+ PCIe 3.0 Server I/O Adapter Part number SFN7142Q Controller silicon SFC9140 Power 13W typical PCI Express 8 lanes Gen 3 (8.0GT/s), 127 SR-IOV virtual functions per port...
Page 10 Solarflare Server Adapter User Guide Solarflare Flareon™ Ultra SFN7122F Dual-Port 10GbE PCIe 3.0 Server I/O Adapter Part number SFN7122F Controller silicon SFC9120 Power 5.9W typical PCI Express 8 lanes Gen 3 (8.0GT/s), 127 SR-IOV virtual functions per port Virtual NIC support...
Page 11 8 lanes Gen2 (5.0GT/s), 127 SR-IOV virtual functions per port Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload SR-IOV Network ports 2 x 10GBASE-T (10G/1G/100M) Solarflare SFN5122F Dual-Port 10G SFP+ Server Adapter Part number SFN5122F Controller silicon SFC9020 Power 4.9W typical PCI Express 8 lanes Gen2 (5.0GT/s), 127 SR-IOV virtual functions per port...
Page 12 Solarflare Server Adapter User Guide Network ports 2 x SFP+ (10G/1G) Regulatory Product Code S6102 1. SR-IOV is not supported for Solarflare adapters on IBM System p servers. Issue 11 © Solarflare Communications 2014...
Page 13 Solarflare Server Adapter User Guide Solarflare SFN6322F Dual-Port 10GbE SFP+ Server Adapter Part number SFN6122F Controller silicon SFC9020 Power 5.9W typical PCI Express 8 lanes Gen2 (5.0GT/s), 127 SR-IOV virtual functions per port Virtual NIC support 1024 vNIC interfaces per port...
Page 14 8 lanes Gen2 (5.0GT/s) Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload SR-IOV Network ports 2 x SFP+ (10G/1G) 1. SR-IOV is not supported for Solarflare adapters on IBM System p servers. Issue 11 © Solarflare Communications 2014...
Page 15 Solarflare Server Adapter User Guide Solarflare Mezzanine Adapters Solarflare SFN5812H Dual-Port 10G Ethernet Mezzanine Adapter Part number SFN5812H Controller silicon SFC9020 Power 3.9W typical PCI Express 8 lanes Gen2 (5.0GT/s), 127 SR-IOV virtual functions per port Virtual NIC support 1024 vNIC interfaces per port...
Page 16 Solarflare Server Adapter User Guide SR-IOV Ports 2 x SFP+ (10G/1G) Regulatory Product Code S6930 Solarflare SFN6822F Dual-Port 10GbE SFP+ FlexibleLOM Onload Server Adapter Part number SFN6822F Controller silicon SFC9020 Power 5.9W typical PCI Express 8 lanes Gen2 (5.0GT/s), 127 SR-IOV virtual functions per port...
Page 17: Software Driver Support
RHEL 6.4 on IBM System p servers. The Solarflare accelerated network middleware, OpenOnload and EnterpriseOnload, is supported on all Linux variants listed above, and is available for all Solarflare Onload network adapters. Solarflare are not aware of any issues preventing OpenOnload installation on other Linux variants such as Ubuntu, Gentoo, Fedora and Debian variants.
Page 18: Open Source Licenses
1.5 Open Source Licenses 1.4.1 Solarflare Boot Manager The Solarflare Boot Manager is installed in the adapter's flash memory. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation;...
Page 19: Support And Download
Do not expose the Solarflare network adapter to rain or moisture. The Solarflare network adapter is a Class III SELV product intended only to be powered by a certified limited power source.
Page 20: Regulatory Approval
9210176-094 R72080250 When installed in a 10Gb ETHERNET NETWORK INTERFACE CARD FROM THE Solarflare SFN5000, SFN6000 or SFN7000 SERIES, the laser emission levels remain under Class I limits as specified in the FDA regulations for lasers, 21 CFR Part 1040.
Page 21 IEC 60950-1:2005 2nd Ed. RoHS Europe Complies with EU directive 2011/65/EU 1. The safety assessment has been concluded on this product as a component/sub-assem- bly only. Additional Regulatory Information for SFN5812H, SFN5814H and SFN6832F adapters. Issue 11 © Solarflare Communications 2014...
Page 22 UL 60950-1 2nd Ed. Safety Canada CSA C22.2 60950-1-07 2nd Ed. IEC 60950-1:2005 2nd Ed. RoHS Europe Complies with EU directive 2002/95/EC 1. The safety assessment has been concluded on this product as a component/sub-assem- bly only. Issue 11 © Solarflare Communications 2014...
Page 23: Chapter 2: Installation
• Supported Speed and Mode...Page 30 • Configure QSFP+ Adapter...Page 31 • LED States...Page 33 • Solarflare Mezzanine Adapters: SFN5812H and SFN5814H...Page 34 • Solarflare Mezzanine Adapter SFN6832F-C61...Page 35 • Solarflare Mezzanine Adapter SFN6832F-C62...Page 37 • Solarflare Precision Time Synchronization Adapters...Page 38 •...
Page 24: Solarflare Network Adapter Products
- Solarflare SFN6822F Dual-Port 10GbE SFP+ FlexibleLOM Onload Server Adapter Solarflare network adapters can be installed on Intel/AMD x86 based 32 bit or 64 bit servers. The network adapter must be inserted into a PCIe x8 OR PCIe x 16 slot for maximum performance. Refer PCI Express Lane Configurations on page 226 for details.
Page 25: Fitting A Full Height Bracket (Optional)
User Guide 2.2 Fitting a Full Height Bracket (optional) Solarflare adapters are supplied with a low-profile bracket fitted to the adapter. A full height bracket has also been supplied for PCIe slots that require this type of bracket. To fit a full height bracket to the Solarflare adapter: From the back of the adapter, remove the screws securing the bracket.
Page 26: Inserting The Adapter In A Pci Express (Pcie) Slot
After restarting the server, the host operating system may prompt you to install drivers for the new hardware. Click Cancel or abort the installation and refer to the relevant chapter in this manual for how to install the Solarflare adapter drivers for your operating system. Issue 11...
Page 27: Attaching A Cable (Rj-45)
Solarflare Server Adapter User Guide 2.4 Attaching a Cable (RJ-45) Solarflare 10GBASE-T Server Adapters connect to the Ethernet network using a copper cable fitted with an RJ-45 connector (shown below). RJ-45 Cable Specifications Table 1 below lists the recommended cable specifications for various Ethernet port types.
Page 28: Attaching A Cable (Sfp+)
Solarflare Server Adapter User Guide 2.5 Attaching a Cable (SFP+) Solarflare SFP+ Server Adapters can be connected to the network using either an SFP+ Direct Attach cable or a fiber optic cable. Attaching the SFP+ Direct Attach Cable: Turn the cable so that the connector retention tab and gold fingers are on the same side as the network adapter retention clip.
Page 29 Remove the cable from the adapter bracket and replace the fiber optic connector cover. Pull the plastic or wire tab to release the adapter bracket. Hold the main body of the adapter bracket and remove it from the adapter. Issue 11 © Solarflare Communications 2014...
Page 30: Supported Sfp+ Cables
SFP+ cables that have been tested by Solarflare. Solarflare is not aware of any issues preventing the use of other brands of SFP+ cables (of up to 5m in length) with Solarflare network adapters. However, only cables in the table below have been fully verified and are therefore supported.
Page 31: Supported Sfp+ 10G Sr Optical Transceivers
SFP+10G SR optical transceivers that have been tested by Solarflare. Solarflare is not aware of any issues preventing the use of other brands of 10G SR transceivers with Solarflare network adapters. However, only transceivers in the table below have been fully verified and are therefore supported.
Page 32: Supported Sfp+ 10G Lr Optical Transceivers
SFP+10G LR optical transceivers that have been tested by Solarflare. Solarflare is not aware of any issues preventing the use of other brands of 10G LR transceivers with Solarflare network adapters. However, only transceivers in the table below have been fully verified and are therefore supported.
Page 33 Solarflare Server Adapter User Guide Supported QSFP+ 40G Active Optical Cables (AOC) The Solarflare Flareon Ultra SFN7142Q adapter has been tested with the following QSFP+ Active Optical Cables (AOC). Table 6: Supported QSFP+ Active Optical Cables Manufacturer Product Code Notes...
Page 34: Supported Sfp 1000Base-T Transceivers
SFP 1000BASE-T transceivers that have been tested by Solarflare. Solarflare is not aware of any issues preventing the use of other brands of 1000BASE-T transceivers with the Solarflare network adapters. However, only transceivers in the table below have been fully verified and are therefore supported.
Page 35: Supported 1G Optical Transceivers
2.11 Supported 1G Optical Transceivers Table 10 is a list of supported 1G transceivers that have been tested by Solarflare. Solarflare is not aware of any issues preventing the use of other brands of 1G transceivers with Solarflare network adapters. However, only transceivers in the table below have been fully verified and are therefore supported.
Page 36: Configure Qsfp+ Adapter
SFP+ Breakout Cables on page 28. Breakout cables from other suppliers may have 4 physical cables. When connecting a third party breakout cable into the Solarflare 40G QSFP+ cage (in 10G mode), only cables 1 and 3 will be active.
Page 37 100Mbps speed. If needed any of the three speeds can be explicitly configured 100Base-T in a Solarflare adapter back-to-back (no intervening switch) configuration will not work and is not supported. Issue 11 © Solarflare Communications 2014...
Page 38: Led States
Solarflare Server Adapter User Guide 2.14 LED States There are two LEDs on the Solarflare network adapter transceiver module. LED states are as follows: Table 13: LED States Adapter Type LED Description State QSFP+, SFP/SFP+ Link Green (solid) at all speeds...
Page 39: Solarflare Mezzanine Adapters: Sfn5812H And Sfn5814H
The Solarflare SFN5812H Dual-Port and SFN5814H Quad-Port are 10G Ethernet Mezzanine Adapters for the IBM BladeCenter. Solarflare mezzanine adapters are supported on the IBM BladeCenter E, H and S chassis, HS22, HS22V and HX5 servers. The IBM BladeCenter blade supports a single Solarflare mezzanine adapter.
Page 40: Solarflare Mezzanine Adapter Sfn6832F-C61
2.16 Solarflare Mezzanine Adapter SFN6832F-C61 The Solarflare SFN6832F-C61 is a Dual-Port SFP+ are 10GbE Mezzanine Adapters for the DELL PowerEdge C6100 series rack server. Each DELL PowerEdge node supports a single Solarflare mezzanine adapter.
Page 41 Lower the adapter into position making sure to connect the adapter slot with the to of the PCB riser card. Secure the adapter using the supplied screws at the positions shown in the diagram. Issue 11 © Solarflare Communications 2014...
Page 42: Solarflare Mezzanine Adapter Sfn6832F-C62
Solarflare Server Adapter User Guide 2.17 Solarflare Mezzanine Adapter SFN6832F-C62 The Solarflare SFN6832F-C61 is a Dual-Port SFP+ are 10GbE Mezzanine Adapters for the DELL PowerEdge C6200 series rack server. Each DELL PowerEdge node supports a single Solarflare mezzanine adapter. The node should be extracted from the rack server in order to install the mezzanine adapter.
Page 43: Solarflare Precision Time Synchronization Adapters
’bump-in-the-wire’ FPGA. For details of installation and configuring applications that run on the AOE refer to the Solarflare AOE User’s Guide (SF-108389-CD). For details on developing custom applications to run on the FPGA refer to the AOE Firmware Development Kit User Guide (SF-108390-CD).
Page 44: Chapter 3: Solarflare Adapters On Linux
Software Driver Support on page 12 for supported Linux Distributions. NOTE: SUSE Linux Enterprise Server 11 includes a version of the Solarflare network adapter Driver. This driver does not support the SFN512x family of adapters. To update the supplied driver, see...
Page 45: Linux Platform Feature Set
NOTE: Red Hat Enterprise Linux versions 5.5 and 6.0 include a version of the Solarflare adapter driver. This driver does not support the SFN512x family of adapters. Red Hat Enterprise Linux 5.6 and 6.1 includes a version of the Solarflare network driver for the SFN512x family of adapters. To update the supplied driver, see Installing Solarflare Drivers and Utilities on Linux on page 43 3.2 Linux Platform Feature Set...
Page 46: Solarflare Rpms
SR-IOV Virtualization for XenServer on page 324 • See SR-IOV Virtualization Using KVM on page 307 SR-IOV is not supported for Solarflare adapters on IBM System p servers. Standby and Power Solarflare adapters support Wake On LAN on Linux. These...
Page 47: Building The Source Rpm
/usr/src/<dir>/RPMS/<arch>/ <dir> distribution specific. Possible folders include Red Hat, packages or extra. The RPM file will be named using the same convention as the Solarflare provided pre-built binary RPMs. The command: will list the locations of all Solarflare find /usr/src -name "*sfc*.rpm”...
Page 48: Installing Solarflare Drivers And Utilities On Linux
• SUSE Linux Enterprise Server Distributions...Page 44 • Building the Source RPM...Page 42 Linux drivers for Solarflare are available in DKMS and source RPM packages. The source RPM can be used to build binary RPMs for a wide selection of distributions and kernel variants. This section details how to install the resultant binary RPM.
Page 49: Suse Linux Enterprise Server Distributions
Solarflare Server Adapter User Guide 3.6 SUSE Linux Enterprise Server Distributions These instructions cover installation and configuration of the Solarflare Network Adapter drivers on SUSE Linux Enterprise Server. Refer to Software Driver Support...Page 12 for details of supported distributions. Refer to Building the Source RPM on page 42 for directions on creating the binary RPM.
Page 50: Unattended Installations
Figure 6 shows how the unattended installation process works. Build a minimal Solarflare driver needed for use in the installation kernel (Kernel A in the diagram above). This is achieved by defining “sfc_minimal” to rpmbuild. This macro disables hardware monitoring, MTD support (used for access to the adapters flash), I2C and...
Page 51 Solarflare Server Adapter User Guide Driver Disks for Unattended Installations Solarflare are preparing binary driver disks to help avoid the need to build the minimal drivers required in unattended installations. Please contact Solarflare support to obtain these driver disks Table 15...
Page 52: Unattended Installation - Red Hat Enterprise Linux
To install Red Hat Enterprise you need the following: A modified file with amended which initrd.img modules.alias modules.dep incorporates the Solarflare minimal driver for the installation kernel. , add the following entries: modules.alias alias: pci:v00001924d00000813sv*sd*bc*sc*i* alias: pci:v00001924d00000803sv*sd*bc*sc*i* alias: pci:v00001924d00000710sv*sd*bc*sc*i*...
Page 53: Unattended Installation - Suse Linux Enterprise Server
*For Red Hat Enterprise Linux from version 5.5 add to this line. mdio A configured kickstart file with the Solarflare Driver RPM manually added to the %Post section. For example: %post /bin/mount -o ro <IP Address of Installation server>:/<path to location directory containing Solarflare RPM>...
Page 54: Hardware Timestamps
Solarflare Server Adapter User Guide 3.10 Hardware Timestamps The Solarflare Flareon SFN7000 series adapters can support hardware timestamping for all received network packets. The Linux kernel must support the SO_TIMESTAMPING socket option (2.6.30+) therefore hardware packet timestamping is not supported on RHEL 5.
Page 55: Configuring Receive/Transmit Ring Buffer Size
/etc/modprobe.d options sfc rx_ring=4096 Using the modprobe method sets the value for all Solarflare interfaces. Then reload the driver for the option to become effective: modprobe -r sfc modprobe sfc Issue 11 © Solarflare Communications 2014...
Page 56: Setting Up Vlans
Solarflare Server Adapter User Guide Configuring Jumbo Frames Solarflare adapters support frame sizes from 1500 bytes to 9216 bytes. For example, to set a new frame size (MTU) of 9000 bytes, enter the following command: ifconfig <ethX> mtu 9000 To make the changes permanent, edit the network configuration file for <...
Page 57: Setting Up Teams
• 802.3ad Dynamic link aggregation • Static link aggregation • Fault Tolerant To set up an adapter team, consult the following documentation: General: http://www.kernel.org/doc/Documentation/networking/bonding.txt RHEL 5: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Deployment_Guide/s2- modules-bonding.html RHEL6: http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s2- networkscripts-interfaces-chan.html SLES: http://www.novell.com/documentation/sles11/book_sle_admin/data/ sec_basicnet_yast.html#sec_basicnet_yast_netcard_man Issue 11 © Solarflare Communications 2014...
Page 58: Running Adapter Diagnostics
The driver utilizes multiple event queues to spread the load over multiple CPU cores (RSS). phy.bist Examines the PHY by initializing it and causing any available built-in self tests to run. Issue 11 © Solarflare Communications 2014...
Page 59: Running Cable Diagnostics
Solarflare Server Adapter User Guide 3.15 Running Cable Diagnostics Cable diagnostic data can be gathered from the Solarflare 10GBASE-T adapters physical interface using the command which runs a comprehensive set of diagnostic tests on the ethtool -t controller, PHY, and attached cables. To run the cable tests enter the following command: ethtool -t ethX [online | offline] Online tests are non-intrusive and will not disturb live traffic.
Page 60: Linux Utilities Rpm
The RPM package, is supplied as 64bit and 32bit binaries compiled to be compatible with GLIBC versions for all supported distributions. The Solarflare utilities RPM file can be downloaded from the following location: https://support.solarflare.com/ • SF-104451-LS is a 32bit binary RPM package.
Page 61: Configuring The Boot Rom With Sfboot
• Sfboot: Examples...Page 66 Sfboot is a command line utility for configuring the Solarflare adapter Boot ROM for PXE and iSCSI booting. Using sfboot is an alternative to using Ctrl + B to access the Boot Rom agent during server startup.
Page 62 (see Performance Tuning on Windows on page 221). Lists all available Solarflare adapters. This option -l --list shows the ifname and MAC address. Note: this option may not be used in conjunction with the any other option. If this option is used with configuration parameters, those parameters will be silently ignored.
Page 63 = 0-256 seconds Sets the adapter boot type. boot- type=<pxe|iscsi|disabled> – PXE (Preboot eXecution Environment) booting – iSCSI (Internet Small Computer System iscsi Interface) booting – Disable adapter booting disabled Issue 11 © Solarflare Communications 2014...
Page 64 Note that this option initiator-dhcp is only valid if iSCSI booting is enabled ( boot- type=iscsi Example: sfboot boot-type=iscsi initiator- dhcp=disabled gateway=192.168.0.10 Issue 11 © Solarflare Communications 2014...
Page 65 The IQN is a symbolic name in the “.” notation form; for example: iqn.2009.01.com.solarflare, and is a maximum of 223 characters long. Note that this option is only valid if iSCSI booting is enabled (...
Page 66 Specifies the Logical Unit Number (LUN) to be used target-lun=<LUN> by the iSCSI target when is disabled. target-dhcp The default LUN is 0. Note that this option is only valid if iSCSI booting is enabled ( boot-type=iscsi Issue 11 © Solarflare Communications 2014...
Page 67 Note that this option is only valid if iSCSI booting is enabled (boot-type=iscsi To be valid, this option also requires the following sub-options to be specified: username=<initiator username> secret=<initiator password> Example: sfboot boot-type=iscsi chap=enabled username=initiatorusername secret=initiatorsecret Issue 11 © Solarflare Communications 2014...
Page 68 Enables/disables Mutual CHAP authentication when mutual- chap=<enabled|disabled> iSCSI booting is enabled. This option also requires the following sub-options to be specified: target-username=<username> target-secret=<password> username=<username> secret=<password> Example: sfboot boot-type=iscsi mutual- chap=enabled username=username secret=veryverysecret target- username=targetusername target- secret=anothersecret Issue 11 © Solarflare Communications 2014...
Page 69 The default is 32. <8|16|32|64|128|256|512|1024> Note: Using the incorrect setting can impact the performance of the adapter. Contact Solarflare technical support before changing this setting. Enable PF-IOV support. This allows switching of pfiov=<enabled|disabled> traffic between any PFs. When disabled traffic is switched only between a PF and related VFs.
Page 70 Description The number of virtual functions (VF) advertised to vf-count=<vf count> the operating system. The Solarflare SFC9000 family of controllers support a total limit of 127 virtual functions per port and a total 1024 interrupts. Depending on the values of msix-limit and vf-msix- limit, some of these virtual functions may not be configured.
Page 71 Boot skip delay time 5 seconds Boot type MSI-X interrupt limit • List all Solarflare adapters installed on the localhost: sfboot --list Solarflare boot configuration utility [v3.0.2] Copyright Solarflare Communications 2006-2010, Level 5 Networks 2002-2005 eth4 - 00-0F-53-01-3C-AC, eth5 - 00-0F-53-01-3C-AD, eth6 - 00-0F-53-01-...
Page 72 Solarflare Server Adapter User Guide • Enable iSCSI booting on adapter eth4. Implement default iSCSI settings: sfboot --adapter=eth4 boot-type=iscsi Solarflare boot configuration utility [v3.0.2] Copyright Solarflare Communications 2006-2010, Level 5 Networks 2002-2005 eth4: Boot image Option ROM and UEFI Link speed...
Page 73 • iSCSI enable adapter eth2. Disable DHCP. Specify adapter IP address and netmask: sfboot boot-type=iscsi --adapter=eth2 initiator-dhcp=disabled initiator- ip=192.168.0.1 netmask=255.255.255.0 Solarflare boot configuration utility [v3.0.2] Copyright Solarflare Communications 2006-2010, Level 5 Networks 2002-2005 eth2: Boot image Option ROM and UEFI...
Page 74 Solarflare Server Adapter User Guide • SFN7000 Series - Firmware Variant sfboot firmware-variant=full-feature Solarflare boot configuration utility [v4.0.0] Copyright Solarflare Communications 2006-2013, Level 5 Networks 2002-2005 eth4: Boot image Option ROM only Link speed Negotiated automatically Link-up delay time 7 seconds...
Page 75: Upgrading Adapter Firmware With Sfupdate
• Sfupdate: Command Line Options...Page 73 • Sfupdate: Examples...Page 74 Sfupdate is a command line utility to manage and upgrade the Solarflare adapter Boot ROM, Phy and adapter firmware. Embedded within the sfupdate executable are firmware images for various Solarflare adapters - the exact updates available via sfupdate depend on the adapter.
Page 76 User Guide If issues are encountered during sfupdate, the user should consider one of the following options when upgrading firmware on systems equipped with more than two Solarflare adapters: • Upgrade two adapters at a time with the other adapters removed.
Page 77 Sfupdate: SLES 11 Limitation Due to limitations in SLES 11 using kernel versions prior to 2.6.27.54 it is necessary to reboot the server after running the sfupdate utility to upgrade server firmware. Issue 11 © Solarflare Communications 2014...
Page 78 -y, --yes firmware to the adapter. Verbose mode. -v, --verbose Suppress output while the utility is running; useful when -s, --silent the utility is used in a script. Display version information and exit. -V --version Issue 11 © Solarflare Communications 2014...
Page 79 Solarflare Server Adapter User Guide Sfupdate: Examples • Display firmware versions for all adapters: sfupdate Solarstorm firmware update utility [v3.0.3] Copyright Solarflare Communications 2006-2010, Level 5 Networks 2002-2005 eth1 - MAC: 00-0F-53-01-39-70 Firmware version: v3.0.3 PHY type: QT2025C PHY version: v2.0.2.5...
Page 80: License Install With Sfkey
3.19 License Install with sfkey The sfkey utility is distributed with the Linux Utilities RM package. This utility is used to install Solarflare AppFlex™ licenses and enable selected on-board services for Solarflare adapters. For more information about license requirements see Solarflare AppFlex™...
Page 81 Solarflare Server Adapter User Guide Product name: Solarflare SFN7122F SFP+ Adapter Part number: SFN7122F Serial number: 712200205071133867100591 MAC address: 00-0F-53-21-9B-B0 Installed keys: OpenOnload, PTP, SolarCapture Pro Active keys: OpenOnload, PTP, SolarCapture Pro Blacklisted keys: 0 Invalid keys: 0 License Inventory Use the combined --inventory and --keys options to identify the licenses installed on an adapter.
Page 82 -a --all licensing. Delete all existing license keys from an adapter - except -c --clear factory installed keys. Display all sfkey options. -h, --help identify specific adapter to apply sfkey operation to. -i --adapter Issue 11 © Solarflare Communications 2014...
Page 83 - use an updated sfkey version. Silent mode, output errors only. -s --silent Verbose mode. -v --verbose Display sfkey version and exit. -V --version Report format as XML. -x --xml Issue 11 © Solarflare Communications 2014...
Page 84: Performance Tuning On Linux
• Other Considerations...Page 94 Introduction The Solarflare family of network adapters are designed for high-performance network applications. The adapter driver is pre-configured with default performance settings that have been chosen to give good performance across a broad class of applications. In many cases, application performance can be improved by tuning these settings to best suit the application.
Page 85 Ethernet endpoints. However if a larger MTU is used, adapter throughput and CPU utilization can be improved. CPU utilization is improved because it takes fewer packets to send and receive the same amount of data. Solarflare adapters support frame sizes up to 9216 bytes (this does not include the Ethernet preamble or frame-CRC).
Page 86 # /sbin/ethtool –K <ethX> rx <on|off> Transmit Checksum: # /sbin/ethtool –K <ethX> tx <on|off> Verification of the checksum settings may be performed by running ethtool with the option. –k Solarflare recommend you do not disable checksum offload. Issue 11 © Solarflare Communications 2014...
Page 87 –k Solarflare recommend you do not disable TSO. TCP Large Receive Offload (LRO) TCP Large Receive Offload (LRO) is a feature whereby the adapter coalesces multiple packets received on a TCP connection into a single call to the operating system TCP Stack. This reduces CPU utilization, and so improves peak throughput when the CPU is fully utilized.
Page 88 Or, as root before loading the Solarflare driver: sysctl -w net.ipv4.conf.default.forwarding=0 (This command can be loaded into /etc/rc.local Or, after loading the Solarflare driver, turn off forwarding for only the Solarflare interfaces and re-enable LRO: sysctl -w net.ipv4.conf.ethX.forwarding=0 ethtool -K ethX lro on (where X is the id of the Solarflare interface).
Page 89 Modifying this file instantly enables or disables LRO, no reboot or driver reload is required. This setting takes precedence over the module parameter Current LRO settings can be identified with Linux ethtool .e.g. ethtool -k ethX Issue 11 © Solarflare Communications 2014...
Page 90 CPU core. RSS can be restricted to only process receive queues on the NUMA node local to the Solarflare adapter. To configure this the driver module option should be set to 1.
Page 91 NOTE: The association of RSS receive queues to a CPU is governed by the receive queue's MSI-X interrupt affinity. See Interrupt and Irqbalance Service on page 91 for more details. The driver must be reloaded to enable option changes: rmmod sfc modprobe sfc Issue 11 © Solarflare Communications 2014...
Page 92 To workaround this issue, you must reboot the host after modifying rss_cpus NOTE: RSS also works for UDP packets. For UDP traffic the Solarflare adapter will select the Receive CPU based on IP source and destination addresses. Solarflare adapters support IPv4 and IPv6 RSS except the SFN4xxx adapters which only supports IPv4 RSS.
Page 93 The tables support the steering of incoming packets from the network adapter to a receive queue affinitized to a CPU where the application is waiting to receive them. The Solarflare accelerated RFS implementation requires configuration through the two tables and the ethtool -K command.
Page 94: Transmit Packet Steering
# echo c000c > /sys/class/net/eth3/queues/tx-0/xps_cpus XPS - Example Configuration System Configuration: • Single Solarflare adapter • 2 x 8 core processors with hyperthreading enabled to give a total of 32 cores • rss_cpus=8 • Only 1 interface on the adapter is configured •...
Page 95 > echo 4096 > /sys/class/net/eth3/queues/rx-1/rps_flow_cnt > echo 4096 > /sys/class/net/eth3/queues/rx-2/rps_flow_cnt > echo 4096 > /sys/class/net/eth3/queues/rx-3/rps_flow_cnt > echo 4096 > /sys/class/net/eth3/queues/rx-4/rps_flow_cnt > echo 4096 > /sys/class/net/eth3/queues/rx-5/rps_flow_cnt > echo 4096 > /sys/class/net/eth3/queues/rx-6/rps_flow_cnt > echo 4096 > /sys/class/net/eth3/queues/rx-7/rps_flow_cnt Issue 11 © Solarflare Communications 2014...
Page 96 To disable irqbalance temporarily, run: /sbin/service irqbalance stop Once the irqbalance service has been stopped, the Interrupt affinities can be configured manually. NOTE: The Solarflare driver will evenly distribute interrupts across the available host CPUs (based on the module parameter).
Page 97 /proc/irq/N/smp_affinity interrupt arrives. Set the application to run on the same CPU (in this case ) as the network interface’s CPU[0] interrupts: # taskset 1 netperf # taskset 1 netperf -H <host> Issue 11 © Solarflare Communications 2014...
Page 98 Having determined that cpu0 and cpu10 are on package 1, we can assign each ethX interface’s MSI- X interrupt to its own CPU on the same package. In this case we choose package 1: Issue 11 © Solarflare Communications 2014...
Page 99 CPU involvement. Solarflare 7000 series adapters support TX PIO, where packet s on the transmit path can be “pushed” to the adapter directly by the CPU. This improves the latency of transmitted packets but can cause a very small increase in CPU utilisation.
Page 100 Solarflare recommend disabling the cpuspeed service if minimum latency is the main consideration. The service can be disabled temporarily:...
Page 101 Buffer Allocation Method Leave at default. Some applications may benefit from specific setting. The Solarflare driver now supports a single optimized buffer allocation strategy and any value set by the rx_alloc_method parameter is ignored. PCI Express Lane Configuration Ensure current speed (not the supported speed) reads back as “x8 and 5Gb/s”...
Page 102 /sbin/service irqbalance stop Interrupt affinity settings are application dependent Buffer Allocation Method The Solarflare driver now supports a single optimized buffer allocation strategy and any value set by the rx_alloc_method parameter is ignored. PCI Express Lane Ensure current speed (not the supported speed) reads back Configuration as “x8 and 5Gb/s”...
Page 103 (if possible select CPU's on the same Package). Refer to Interrupt and Irqbalance Service...Page 91 Buffer Allocation Method The Solarflare driver now supports a single optimized buffer allocation strategy and any value set by the rx_alloc_method parameter is ignored. Issue 11...
Page 104: Module Parameters
Solarflare Server Adapter User Guide 3.21 Module Parameters Table 25 lists the available parameters in the Solarflare Linux driver module ( modinfo sfc Table 25: Driver Module Parameters Possible Default Parameter Description Value Value piobuf_size Identify the largest packet size that can...
Page 105 Force internal SRAM to be used rss_numa_local Constrain RSS to use CPU cores on the numa node local the Solarflare adapter. Set to 1 to restrict, 0 otherwise. 1. Check OS documentation for availability on SUSE and RHEL versions. Issue 11...
Page 106: Linux Ethtool Statistics
3.22 Linux ethtool Statistics The Linux command ethtool will display an extensive range of statistics originated from the MAC on the Solarflare network adapter. To display statistics use the following command: ethtool -S ethX (where X is the ID of the Solarflare interface) Table 26 lists the complete output from the ethtool -S command.
Page 107 Number of frames transmitted where the length is greater tx_gtjumbo than 9000 bytes. Number of collisions incurred during transmission attempts. tx_collision This should always be zero as Solarflare adapters operate in full duplex mode. Number of occurrences when a single collision delayed tx_single_collision immediate transmission of a packet.
Page 108 Number of packets received with correct CRC value and no rx_good error codes. Number of packets received with incorrect CRC value. rx_bad Number of pause frames received with valid pause rx_pause op_code. Number of control frames received. Does not include pause rx_control frames. Issue 11 © Solarflare Communications 2014...
Page 109 1518 bytes (1522 with VLAN tag) and 9000 bytes. Number of frames received with incorrect CRC value and rx_bad_gtjumbo where the length is greater than 9000 bytes. Number of packets dropped by receiver because of FIFO rx_overflow overrun. Issue 11 © Solarflare Communications 2014...
Page 110 2. Distribute the traffic load across the available CPU/cores by setting rss_cpus=cores. Refer to Receive Side Scaling section 3. Increase receive queue size using ethtool. Overflow of the packet memory burst buffer - should not rx_pm_trunc_bb_overflow occur. Issue 11 © Solarflare Communications 2014...
Page 111 MCPU. Not currently used. Count the number of times the adapter descriptor cache is rx_dp_emerg_fetch empty when a new packet arrives, for which the adapter must do an emergency fetch to replenish the cache with more descriptors. Issue 11 © Solarflare Communications 2014...
Page 112 IGMP snooping - a process from which it learns which endpoints are interested in which multicast streams. Issue 11 © Solarflare Communications 2014...
Page 113 MAC will truncate the frame - marking it as such and discard the remainder. The driver on seeing a 'partial' packet which has been truncated will discard it. rx_char_error_lane0 rx_char_error_lane1 rx_char_error_lane2 rx_char_error_lane3 rx_disp_error_lane0 rx_disp_error_lane1 rx_disp_error_lane2 rx_disp_error_lane3 rx_match_fault Issue 11 © Solarflare Communications 2014...
Page 114: Chapter 4: Solarflare Adapters On Windows
• Windows Event Log Error Messages...Page 233 4.1 System Requirements • Refer to Software Driver Support on page 12 for details of supported Windows versions. • Microsoft.NET Framework 3.5 is required if installing Solarflare Adapter Manager on any platform. Issue 11 © Solarflare Communications 2014...
Page 115: Windows Feature Set
Teaming and or Link Improve server reliability and bandwidth by bonding physical Aggregation ports, from one or more Solarflare adapters, into a team, having a single MAC address and which function as a single port providing redundancy against a single point of failure.
Page 116 Sfteam: Adapter Teaming and VLAN Tool for teaming statistics. • See Sfnet for per interface statistics. Solarflare drivers support static VMQ for Windows Server 2008 R2 and Dynamic VMQ on Windows Server 2012 Virtual Machine Queue on page 144. Issue 11 © Solarflare Communications 2014...
Page 117: Installing The Solarflare Driver Package On Windows
• Full Solarflare Package Installation...Page 115 • Repair, Remove and Change Drivers and Utilities...Page 118 NOTE: The Solarflare adapter should be physically inserted before installing the drivers. See Installation on page The user must have administrative rights to install adapter drivers and may be prompted to enter an administrator user name and password.
Page 118: Adapter Drivers Only Installation
Solarflare Server Adapter User Guide 4.4 Adapter Drivers Only Installation The steps below describe how to install only the Solarflare adapter drivers in Windows. To install the drivers from the command line, see Install Drivers and Options From a Windows Command Prompt on page 119.
Page 119 Solarflare Server Adapter User Guide From the Custom Setup screen, select the Install Solarflare® device drivers option only. Figure 8: Solarflare Custom Setup Click Finish to close the wizard. Restart Windows if prompted to do so. Issue 11 © Solarflare Communications 2014...
Page 120: Full Solarflare Package Installation
Repair, Remove and Change Drivers and Utilities...Page 118 Prerequisites • The Solarflare Adapter Manager Utility (SAM) requires Microsoft .NET Framework 3.5 assemblies. These are available by installing .NET version 3.5 and may also be available in version 4.x with backward compatibility for 3.5.
Page 121 • Start Solarflare Adapter Manager (Start > All Programs > Solarflare Drivers > Solarflare Adapter Manager). If the Solarflare adapter is installed and working correctly, it will be shown in the SAM main screen, along with any other adapters, as in Table 12 on page 129.
Page 122 Solarflare Server Adapter User Guide Figure 10: Solarflare Driver Package Custom Setup Table 28: Solarflare Custom Setup Option Description Install Solarflare device drivers Installs Solarflare NDIS drivers for Windows. The Solarflare drivers are installed by default. Install Solarflare command line tools...
Page 123 The icon is not installed by default. Repair, Remove and Change Drivers and Utilities From the Control Panel > Programs > Programs and Features, select the Solarflare Driver Package then select Uninstall, Change or Repair from the menu bar above the program list.
Page 124: Install Drivers And Options From A Windows Command Prompt
Setup Window will be displayed. Figure 11: Command Line Install. Installing from the Windows command line allows scripted, silent and unattended installation of the core Solarflare drivers and package utilities. The drivers install package is named after the Solarflare document part number e.g. SF-107785-LS-2_Solarflare_Windows_x86_64-bit_Driver_Package.exe This can be renamed e.g setup.exe before invoking from the command line.
Page 125 Identify the package file to use for the setup.exe /Package operation. <PackageFilename> Shows version information for the drivers. setup.exe /Version Silently installs the drivers and Solarflare setup.exe /Quiet /Install Adapter Manager only (other utilities will ADDLOCAL=NetworkAdapterManager not be installed). See, Using ADDLOCAL on page 121.
Page 126 • CommandLineTools – Installs Solarflare command line tools: sfboot.exe, sfupdate.exe, sfcable.exe, sfteam.exe, sfnet.exe. • Launcher – Installs the Solarflare system tray icon, providing easy access to the Solarflare Adapter Manager (SAM). Multiple features may be installed by separating each feature with a comma (spaces are not allowed).
Page 127 User Guide Extract Solarflare Drivers If it is necessary to extract the Solarflare Windows drivers, e.g. before WDS installs, this can be done from the Windows command line. From the Command prompt, navigate to the directory where the installation package is located.
Page 128: Unattended Installation
• Create the WDSClientUnattend.xml File...Page 126 • Create the AutoUnattend.xml File...Page 127 • Further Reading...Page 127 Windows Driver Locations The following steps use drivers extracted from the Solarflare installation package. Refer to Table 30 for driver folder locations. Unattended Installation using WDS Windows Deployment Services (WDS) enables the deployment of Windows over a network (from a WDS server), avoiding the need to install each operating system directly from a CD or DVD.
Page 129 Adding Solarflare Drivers to the WDS Boot Image These steps describe how to add the Solarflare drivers into the Boot Image. Modifying the Boot Image You next need to modify the boot image to include the Solarflare Drivers extracted from the setup package. Table 30 identifies drivers required for the target operating system.
Page 130 From the right-click menu, select Enable. Create Custom Install Image These steps describe how to add the Solarflare drivers into the Custom Install Image. These are the same Solarflare drivers added to the boot image. Preparing the Custom Install Image From WDS, locate the install image from the Install Images folder on your server.
Page 131 Solarflare Server Adapter User Guide Add the Solarflare NDIS driver to the image by entering the following command from your PE customization working directory: peimg /inf=c:\windowspe-x86\drivers\netSFN6*.inf mount\windows Unmount the image, using the following command from your PE customization working directory:...
Page 132 • Getting Started with the Windows AIK: http://technet.microsoft.com/en-us/library/cc749082%28WS.10%29.aspx • Performing Unattended Installations: http://technet.microsoft.com/en-us/library/cc771830%28WS.10%29.aspx • How to add network driver to WDS boot image: http://support.microsoft.com/kb/923834 • Windows Deployment Services Getting Started Guide for Windows Server 2012 http://technet.microsoft.com/en-us/library/jj648426.aspx Issue 11 © Solarflare Communications 2014...
Page 133: Managing Adapters With Sam
Windows OS versions. Introduction The Solarflare Adapter Manager (SAM) is a Microsoft Management Console (MMC) plug-in for managing Solarflare adapters, teams and VLANs. SAM identifies information for all adapters installed on the server, as well as the standard MMC plug-in Actions pane.
Page 134: Managing Adapters Remotely With Sam
SAM is included with the Solarflare drivers installation package. 4.9 Managing Adapters Remotely with SAM SAM can be used to administer Solarflare adapters on your server from a remote computer. SAM can be used remotely to administer adapters on any supported Windows platform, including a Windows Server Core Installation.
Page 135: Using Sam
• Click Start > All Programs > Solarflare Network Adapters > Manage network adapters on a remote computer. • If the Solarflare notification area icon is installed, you can right-click the icon and select Manage network adapters on a remote computer.
Page 136 • MAC address • Transmit load • Receive load For Solarflare adapters only, SAM also lists any teams or VLANs that have been configured, along with details that allow you to quickly check performance and status. Issue 11 © Solarflare Communications 2014...
Page 137 User Guide Viewing Performance Graphs To view Solarflare performance graphs, Right-click on an adapter and select Show graphs from the menu. By default, SAM shows the load, transmitted packets and received packets graphs only. To view other available graphs, Select Graphs from the right-click menu, or from the Actions Pane/ Action menu.
Page 138 To configure SAM options: Start SAM. From the Actions pane, click Options, or choose Action > Options. Figure 15: SAM - Actions > Options In the Configuration window, select required options (seeTable 31). Issue 11 © Solarflare Communications 2014...
Page 139 Third-party adapters installed in the server are also listed in the SAM’s Network Adapters list, along with the Solarflare adapters and any teams and VLANs which have been set up on the server. SAM provides some options for working with third-party adapters. The available actions for third party adapters are shown in the Action pane.
Page 140: Using Sam To Configure Adapter Features
NOTE: Changing the value of an Adapter feature can negatively impact the performance of the adapter. You are strongly advised to leave them at their default values. NOTE: Before making any changes to your Solarflare adapter features, read the Performance...
Page 141 From SAM, right-click on an adapter and select Configuration > Configure Offload tasks, Ethernet and other features From SAM, select an adapter and from the Action menu, select Configure Offload tasks, Ethernet and other features. Issue 11 © Solarflare Communications 2014...
Page 142 Solarflare Server Adapter User Guide The Adapter Features dialog box will be displayed: Figure 16: Solarflare Adapter Manager Adapter Features Click Apply or OK when changes to Adapter Features are modified. Issue 11 © Solarflare Communications 2014...
Page 143: Checksum Offload
By offloading the checksum process to the network adapter, the load is decreased on the server CPU. By default, Solarflare adapters are set up to offload both the calculation and verification of TCP, IP and UDP checksums. The following Checksum Offload options are supported:...
Page 144 Solarflare Server Adapter User Guide RSS and Interrupts Solarflare network adapters support RSS (Receive Side Scaling) and interrupt moderation. Both are enabled by default and can significantly improve the performance of the host CPU when handling large amounts of network data.
Page 145 The maximum number of processors to be used by RSS. The value is in the range 0-256. Max. RSS queues The maximum number of receive queues created per interface. The value is in the range 0-64. Issue 11 © Solarflare Communications 2014...
Page 146 NUMA architectures. This constrains the set of CPU cores used for RSS to the specified NUMA node. Solarflare recommend you leave this at the default setting of All. The adapter will attempt to use only processors from the specified NUMA node for RSS.
Page 147: Segmentation Offload
Solarflare adapters offload the tasks of packet segmentation and reassembly to the adapter hardware, reducing the CPU processing burden and improving performance. LRO is a Solarflare proprietary mechanism similar to the Windows Receive Side Coalescing feature. When enabled the adapter will coalesce multiple received TCP packets on a TCP connection into a single call to the TCP/IP stack.
Page 148 NOTE: For optimum performance set the Ethernet frame length to your network MTU. If the network uses Jumbo frames, SAM supports frames up to a maximum of 9216 bytes. Issue 11 © Solarflare Communications 2014...
Page 149 Solarflare Server Adapter User Guide Virtual Machine Queue Solarflare adapters support VMQ to offload the classification and delivery of network traffic destined for Hyper-V virtual machines to the network adapter thereby reducing the CPU load on Hyper-V hosts. Windows Server 2008 R2 allows the administrator user to statically configure the number of CPUs available to process interrupts for VMQ.
Page 150: Using Sam To Configure Teams And Vlans
Microsoft MultiPath I/O (MPIO), which is supported on all Solarflare adapters. This section is only relevant to teams of Solarflare adapters. Solarflare adapters can be used in multi- vendor teams when teamed using the other vendor’s teaming driver.
Page 151 Start SAM and select a Solarflare adapter in the Network Adapter list. From the Action menu, select Create a Team. The Solarflare Create a team Wizard starts. Figure 17: Team Create Wizard...
Page 152 To change team settings: NOTE: Changing team settings can disrupt network traffic flow to and from services running on the server. Solarflare recommend only changing network settings when disruption to the services can be tolerated. Start SAM and, from the Network Adapter list, select the team you want to reconfigure.
Page 153 The highest prioritized adapter becomes the active adapter in the team after you apply your changes. To change adapter priority, use the up and down buttons. Issue 11 © Solarflare Communications 2014...
Page 154 After making your changes, click Set and then click Close. Adding Adapters to a Team If additional Solarflare adapters are installed in your server, you can add them to an existing team to increase the overall resilience or performance (aggregation) of the server connection.
Page 155 Any VLANs set up for the team will be deleted when the team is deleted. To delete a team: NOTE: Changing team settings can disrupt current services running on the server. Solarflare recommend only changing network settings when disruption to network services can be tolerated.
Page 156 VLAN 0. To create VLANs: NOTE: Creating VLANs can disrupt current services running on the server. Solarflare recommend only changing network settings when disruption to network services can be tolerated. Start SAM and select the adapter or adapter team from the Network Adapter list.
Page 157 VLANs can be removed from a team or single adapter when no longer required. To delete VLANs: NOTE: Deleting VLANs can disrupt current processes and applications running on the server. Solarflare recommend only changing network settings when disruption to network services can be tolerated. Start SAM.
Page 158: Using Sam To View Statistics And State Information
Figure 23: Solarflare Adapter Statistics and State To view Solarflare statistics and state information: Start SAM and select a Solarflare adapter from the Network Adapter list. From the Actions list, click Statistics and State. The Details from <adapter name> dialog box is displayed.
Page 159: Using Sam To Run Adapter And Cable Diagnostics
Click Close. 4.15 Using SAM to Run Adapter and Cable Diagnostics You can verify the Solarflare adapter, driver and cable by running SAM’s built-in diagnostic tools (Solarflare 10GBASE-T adapter only). The tools provide a simple way to verify that the adapter and driver are working correctly, and that the cable has the correct characteristics for high-speed data transfer.
Page 160 Completion Message column describing the reason any particular test has failed. CAUTION: The adapter will stop functioning while the tests are being run. Solarflare recommend only running diagnostics tests when disruption to network services can be tolerated.
Page 161 PHY BIST Examines the PHY by initializing it and starting any available built-in self tests to run. Bootrom Verifies the Boot ROM configuration and image checksum. Will warn if no Boot ROM is present. Issue 11 © Solarflare Communications 2014...
Page 162 NOTE: Cable diagnostics are only available on Solarflare 10GBASE-T Adapters. For these adapters, Solarflare recommend using good quality Category 6, 6a or 7 cable up to the maximum length as determined by the cable category.
Page 163 Pair short at … The cable or the connector is faulty and must be replaced. Error An open circuit has been detected. Pair is open circuit The cable or the connector is faulty and must be replaced. Issue 11 © Solarflare Communications 2014...
Page 164: Using Sam For Boot Rom Configuration
Configuring the Boot ROM for PXE or iSCSI Booting For more information on configuring the iSCSI target and DHCP settings from the Solarflare Boot Configuration Utility, and how to install an operating system that is enabled for remote iSCSI booting...
Page 165 NOTE: iSCSI booting will not be available if the adapter is a member of a team or has VLANs. NOTE: Solarflare recommend not changing the MSI-X Interrupts setting. If necessary, from the Link tab, change the Link Speed option depending on your link requirement.
Page 166 DHCP vendor Id specifies the device vendor ID to be advertised to the DHCP server. This setting is always enabled and not affected by any of the other DHCP options. See DHCP Server Setup on page 356 for more details on this and other DHCP options. Issue 11 © Solarflare Communications 2014...
Page 167 TCP port ISCSI port number that has been configured on the target. Default is 3260. Target device name The iSCSI Qualified Name (IQN) of the target server, which will look something like: iqn:2009-01.com.solarflare. Issue 11 © Solarflare Communications 2014...
Page 168 Specifies the number of times the boot device will attempt to connect to the target LUN (logical unit number) before failing. The default setting is 2 retries, but can be set from 0–255. This setting is enabled, even if using DHCP is being used. Issue 11 © Solarflare Communications 2014...
Page 169 Name of this initiator (as set on the target). A minimum of 9 characters. Used for Mutual authentication only Initiator secret Password of this initiator (as set on the target). A minimum of 12 characters. Used for Mutual authentication only. Issue 11 © Solarflare Communications 2014...
Page 170 You can stop the adapter from attempting to initiate either a PXE or iSCSI boot after a restart. Start SAM and select the Solarflare adapter from the Network Adapter list. From the Action menu, click the Configure Boot ROM option. The Configure Boot ROM dialog box displays with the BIOS tab selected.
Page 171: Managing Firmware With Sam
User Guide 4.17 Managing Firmware with SAM SAM allows you to monitor the firmware (PHY, Boot ROM and Adapter) for your Solarflare adapters. Either select Manage firmware from the Actions pane, or from the Action menu. The firmware update window is displayed: Figure 32: Solarflare firmware update window If the firmware is up to date, the window will contain the OK button.
Page 172: Configuring Network Adapter Properties In Windows
From the Control Panel, select System. Select Device Manager from the left hand menu. Expand the Network adapters. Right-click the on the Solarflare adapter, and then click Properties to display the properties dialog box. Figure 33: Adapter Properties Dialog Issue 11...
Page 173 NOTE: Changing these properties may impact the performance of your Solarflare adapter. You are strongly advised to leave them at their default values. NOTE: Before making any changes to your Solarflare adapter features, read the...
Page 174 Solarflare Server Adapter User Guide Table 43: Solarflare Network Adapter Properties Property Name Values Description Interrupt Enabled Interrupt moderation is a technique used to reduce Moderation the number of interrupts sent to the CPU. With Disabled interrupt moderation, the adapter will not generate interrupts closer together than the interrupt moderation time.
Page 175 Solarflare Server Adapter User Guide Table 43: Solarflare Network Adapter Properties Property Name Values Description Locally Value: (MAC Assigns the specified MAC address to the adapter, Administered address) overriding the permanent MAC address assigned by Address the adapter's manufacturer. Not Present...
Page 176 Solarflare Server Adapter User Guide Table 43: Solarflare Network Adapter Properties Property Name Values Description Receive Side Scaling Enabled Receive Side Scaling (RSS) is a technology that (RSS) enables packet receive processing to scale with the Disabled number of available processors (CPUs), distributing the processing workload across the available resources.
Page 177: Windows Command Line Tools
A tool for managing fault-tolerant adapter teams and VLANs. See Sfteam: sfteam.exe Adapter Teaming and VLAN Tool on page 193. A tool for that runs cable diagnostics for Solarflare 10GBASE-T server sfcable.exe adapters. See Sfcable: Cable Diagnostics Tool on page 200.
Page 178 To start a command line tool, open a Command Line Interface windows and enter the command tool.exe: Figure 34: Windows console to run Solarflare command line tools. NOTE: For all the utilities, the options are documented with the forward slash (/) prefix. You can also use a single dash ( ) or a double dash (--) as a prefix.
Page 179: Sfboot: Boot Rom Configuration Tool
ID of the adapter that you want to manage. Specifying the Identifier adapter is optional - if it is not included the command is applied to all Solarflare adapters in the machine. is the option you want to apply. See...
Page 180 Performs the operation on a specified remote /Computer <ComputerName> computer. Administrator rights on the remote computer is required. Lists all available Solarflare adapters. This option /List shows the adapter’s ID number, ifname and MAC address. Note: this option may not be used in conjunction with the any other option.
Page 181 This can be used to wait for spanning tree protocol on a connected switch to unblock the switch port after the physical network link is established. The default is 5 seconds. Issue 11 © Solarflare Communications 2014...
Page 182 Note that this option is only valid if iSCSI booting is enabled ( boot-type=iscsi Example: sfboot boot-type=iscsi initiator- dhcp=disabled initiator- ip=<192.168.1.3> Issue 11 © Solarflare Communications 2014...
Page 183 This option is only valid if iSCSI booting is enabled boot-type=iscsi Example: sfboot boot-type=iscsi initiator- dhcp=disabled primary-dns=192.168.0.3 Enables or disables use of DHCP for the initiator IQN initiator-iqn- dhcp=<enabled|disabled> only. Issue 11 © Solarflare Communications 2014...
Page 184 The IQN is a symbolic name in the “.” notation form; for example: iqn.2009.01.com.solarflare, and is a maximum of 223 characters long. Note that this option is only valid if iSCSI booting is enabled (...
Page 185 Specifies the device vendor ID to be advertised to the vendor-id=<dhcp_id> DHCP server. This must match the vendor id configured at the DHCP server when using DHCP option 43 to obtain the iSCSI target . Issue 11 © Solarflare Communications 2014...
Page 186 Note that this option is valid if either CHAP or Mutual CHAP is enabled (chap=enabled, mutual- chap=enabled Note that if there are spaces contained in < >, secret then it must be wrapped in double quotes (“”). Example: sfboot boot-type=iscsi chap=enabled username=username secret=veryverysecret Issue 11 © Solarflare Communications 2014...
Page 187 The range is 1- 255, with 1 being the highest priority. Specifies the number of times MPIO will try and use mpio-attempts=<attempt each port in turn to login to the iSCSI target before count> failing. Issue 11 © Solarflare Communications 2014...
Page 188 Enable SR-IOV support for the operating systems that sriov=<enabled|disabled> support it. SR-IOV is not supported by Solarflare adapters on Windows Hyper-V servers. The number of virtual functions advertised to the vf-count=<vf count> operating system. The Solarflare SFC9000 series device has a limit of 1024 interrupts.
Page 189 Number of Virtual Functions VF MSI-X interrupt limit Firmware variant full feature / virtualization Insecure filters Disabled Solarflare SFN7122F SFP+ Server Adapter #2 - MAC: 00:0F:53:21:9B:B0 Boot image Option ROM only Link speed Negotiated automatically Link-up delay time 5 seconds...
Page 190 Solarflare Server Adapter User Guide • List all Solarflare adapters installed on the localhost: sfboot /List Sample console output: Solarflare boot ROM configuration utility [v4.1.4] Copyright Solarflare Communications 2006-2014, Level 5 Networks 2002-2005 Network adapters in this computer: 1 : Solarflare SFN7122F SFP+ Server Adapter...
Page 191 Solarflare Server Adapter User Guide Sample console output: Solarflare boot ROM configuration utility [v4.1.4] Copyright Solarflare Communications 2006-2014, Level 5 Networks 2002-2005 Solarflare SFN7122F SFP+ Server Adapter - MAC: 00:0F:53:21:9B:B1 Boot image Option ROM only Link speed Negotiated automatically Link-up delay time...
Page 192 /Adapter 1 boot-type=iscsi initiator-dhcp=disabled initiator- ip=192.168.0.1 netmask=255.255.255.0 Sample console output: Solarflare boot ROM configuration utility [v4.1.4] Copyright Solarflare Communications 2006-2014, Level 5 Networks 2002-2005 Solarflare SFN7122F SFP+ Server Adapter - MAC: 00:0F:53:21:9B:B1 Boot image Option ROM only Link speed...
Page 193 - Secret “password12345” sfboot /Adapter 1 boot-type=iscsi chap=enabled username=user1 secret=password12345 Sample output: Solarflare boot ROM configuration utility [v4.1.4] Copyright Solarflare Communications 2006-2014, Level 5 Networks 2002-2005 Solarflare SFN7122F SFP+ Server Adapter - MAC: 00:0F:53:21:9B:B1 Boot image Option ROM only Link speed...
Page 194: Sfupdate: Firmware Update Tool
ID of the adapter that you want to manage. Specifying the Identifier adapter is optional - if it is not included the command is applied to all Solarflare adapters in the machine. is the option to apply. See...
Page 195 Performs the operation on the identified remote /Computer <ComputerName> computer. Administrator rights on the remote host computer is required. Performs the action on the identified Solarflare network /Adapter <Identifier> adapter. The identifier can be the adapter ID number, name or MAC address.
Page 196 • Display firmware versions for all adapters: sfupdate Sample output from a host with a single SFN5122F adapter installed: Solarflare firmware update utility [v4.1.4] Copyright Solarflare Communications 2006-2014, Level 5 Networks 2002-2005 1: Solarflare SFN7122F SFP+ Server Adapter MAC address: 00:0F:53:21:9B:B1 Firmware: v4.1.0...
Page 197 • Update all adapters to latest version of PHY and Boot ROM firmware: sfupdate /Write Sample output: Solarflare firmware update utility [v4.1.4] Copyright Solarflare Communications 2006-2014, Level 5 Networks 2002-2005 1: Solarflare SFN7122F SFP+ Server Adapter MAC address: 00:0F:53:21:9B:B1 Firmware: v4.1.0...
Page 198: Sfteam: Adapter Teaming And Vlan Tool
Sfteam is a Windows command line utility used to configure and manage the teaming and VLAN features of the Solarflare adapters. You may find it easier to create and manage teams and VLANs with SAM, Solarflare’s graphical adapter manager. As an alternative, or where SAM is not available, sfteam provides a method of creating teams and VLANs from the command line or configuration script.
Page 199 Performs the operation on the identified remote /Computer <ComputerName> computer. Administrator rights on the remote host computer is required. Lists all available Solarflare adapters and any teams and /List VLANs. This options shows the adapter’s ID number, name and MAC address.
Page 200 This option must be used when a team is first created. It cannot be applied to a team once it has been setup. Issue 11 © Solarflare Communications 2014...
Page 201 Priority: either P (priority supported) or N (no priority) DHCP: may be omitted, and will be assumed, if it's the last field IP Addresses: IPv4, dotted-quad format note that <mask> must be present if <addr> is present Issue 11 © Solarflare Communications 2014...
Page 202 Specify how the driver distributes conversations across /Distribution <type> dynamic or static link aggregation team members. The available types are: • auto (default) • activeadapter • layer2hash • layer3hash • layer4hash Issue 11 © Solarflare Communications 2014...
Page 203 • Create TeamA with adapter ID 1 and adapter ID 2: sfteam /Create /Adapter 1 /Adapter 2 /Name Team_A Sample output: Solarflare teaming configuration utility [v4.1.4] Copyright Solarflare Communications 2006-2014 Level 5 Networks 2002-2005 Creating team done (new id=2F) Setting team name "Team_A" ... done Adding adapter 1 ...
Page 204 Solarflare Server Adapter User Guide Sample output: Solarflare teaming configuration utility [v4.1.4] Copyright Solarflare Communications 2006-2014 Level 5 Networks 2002-2005 Creating VLAN group done (new id=4V) Setting VLAN group name (using default name "Group 4V") ... done Adding adapter 2 ...
Page 205: Sfcable: Cable Diagnostics Tool
ID of the adapter that you want to manage. Specifying the Identifier adapter is optional - if it is not included the command is applied to all Solarflare adapters in the machine. is the option you to apply. See Table 49 for a list of available options.
Page 206 Performs the operation on the identified remote /Computer <ComputerName> computer. Administrator rights on the remote host computer is required. Performs the action on the identified Solarflare network /Adapter <Identifier> adapter. The identifier can be the adapter ID number, name or MAC address, as given by the option.
Page 207 User Guide Sfcable: Sample Commands • Run tests offline sfcable /Offline Sample output from a computer with two Solarflare adapters installed: C: sfcable /Offline Solarflare cable diagnostics utility [v4.1.4] Copyright Solarflare Communications 2006-2014, Level 5 Networks 2002-2005 1 : Solarflare SFN5121T 10GBASE-T Server Adapter...
Page 208: Sfnet
ID of the adapter that you want to manage. Specifying the Identifier adapter is optional - if it is not included the command is applied to all Solarflare adapters in the machine. is the option to apply. See Sfnet: Command Line Options for a list of available options.
Page 209 Performs the operation on the identified remote host. /Computer <ComputerName> Administrator rights on the remote host computer is required. Perform the action on the identified Solarflare physical /Adapter <Identifier> or virtual network adapter. Lists all available Solarflare adapters, options and current /List parameter settings.
Page 210 The maximum number of processors available for maxrssprocessors=<count> RSS. If count is specified it must be in the range 1- 256. Support for this option is independent of the version of the operating system and networking stack. Issue 11 © Solarflare Communications 2014...
Page 211 = VMQ enabled. vmq=<enabled|nosplit|novlan|ba sic|disabled> nosplit = VMQ enabled without lookahead split. novlan = VMQ enabled without VLAN filtering. basic = VMQ enabled MAC address filtering only. disabled = VMQ disabled. Issue 11 © Solarflare Communications 2014...
Page 212: Completion Codes (%Errorlevel%)
Solarflare adapters are present, or adapters have been disabled. Failed to obtain driver lock. The application has tried to take the Solarflare driver lock because it wants to do something that must not be interrupted by another utility (or SAM) and failed to do so.
Page 213 Could not open log file. A general WMI error occurred. Can occur when the connection is lost. Missing prerequisite. The application needs something that is not present in the system. Not supported. Platform/System not supported. General exit failure. Issue 11 © Solarflare Communications 2014...
Page 214: Teaming And Vlans
A and the second port a member of team B or both ports members of the same team. This section is only relevant to teams of Solarflare adapters. Solarflare adapters can be used in multi- vendor teams when teamed using another vendor’s teaming driver.
Page 215 • Short recovery time in case of failover. • Solarflare supports up to 64 link aggregation port groups per system. • Solarflare supports up to 64 ports and VLANs in a link aggregation port group. There are two methods of link aggregation, dynamic and static.
Page 216 Solarflare Server Adapter User Guide Figure 35: 802.1AX Team Figure 36 shows a 802.1AX team with a failed link. All traffic is re-routed and shared between the other team links. Figure 36: 802.1AX with Failed Link Issue 11 © Solarflare Communications 2014...
Page 217 In the event of failover, the load on the failed link is redistributed over the remaining links. Figure 37: Static Link Aggregation Team Figure 38: Static Link Team with Failed Link Issue 11 © Solarflare Communications 2014...
Page 218 (down) or that it fails in service, swaps to one of the standby adapters. In Figure 40 previously active adapter has entered a failed state and will not be available in the standby list while the failed state persists. Issue 11 © Solarflare Communications 2014...
Page 219 Note that, in this example, Adapter 3 is now active. The order in which the adapters are used is determined by a number of factors, including user-definable rank. VLANs VLANs offer a method of dividing one physical network into multiple broadcast domains. Figure 41: VLANs routing through Solarflare adapter Issue 11 © Solarflare Communications 2014...
Page 220 Solarflare Server Adapter User Guide VLANs and Teaming VLANs are supported on all Solarflare adapter teaming configurations. VLANs with Fault Tolerant Teams Figure 42 shows a fault tolerant team with two VLANs. Figure 42: Fault Tolerant VLANs Failover works in the same way regardless of the number of VLANs, as show in...
Page 221 Solarflare Server Adapter User Guide Figure 43: Failover in Fault Tolerant Team VLAN Issue 11 © Solarflare Communications 2014...
Page 222 Figure 44: VLAN with Dynamic or Static Link Team In case of link failure, all traffic is distributed over the remaining links, as in Figure Figure 45: VLAN with Failed Dynamic or Static Team Link Issue 11 © Solarflare Communications 2014...
Page 223 When a link failure occurs in the active adapter (for example the physical link is lost) the driver will select another adapter to become active but it will not re-assign the Key Adapter. In Figure Adapter 1 has failed and the team is now using Adapter 2 for all traffic. Issue 11 © Solarflare Communications 2014...
Page 224 Note that although the Key Adapter (Adapter 1) has a link failure, the integrity of the team is not affected by this failure. Dynamic and Static Link Aggregation Teams The assignment of key adapters is supported in both dynamic and static link aggregated teams, and works in the same way for both. Issue 11 © Solarflare Communications 2014...
Page 225 Solarflare Server Adapter User Guide Any link failure on the key adapter does not affect the redistribution of traffic to the other links in the team. Issue 11 © Solarflare Communications 2014...
Page 226: Performance Tuning On Windows
KX4. This is because the physical media interface chip (PHY) used on the adapter can introduce additional latency. This section is designed for performance tuning Solarflare adapters on Microsoft Windows. This should be read in conjunction with the reference design board errata documents and the following Microsoft performance tuning guides: •...
Page 227 Solarflare Server Adapter User Guide Tuning Settings Tuning settings for the Solarflare adapter are available through the Solarflare Adapter Manager (SAM) utility, or via the Advanced tab in the Windows Device Manager (right-click the adapter and select Properties). See Using SAM to Configure Adapter Features on page 135...
Page 228 Solarflare Server Adapter User Guide receive the same amount of data. Solarflare adapters supports maximum frame sizes up to 9216 bytes (this does not include CRC). NOTE: The maximum frame size setting should include the Ethernet frame header. The Solarflare drivers support 802.1p.
Page 229 LSO is changed by changing the Large Send Offload setting in the Network Adapter’s Advanced Properties Page. TCP and IP checksum offloads must be enabled for LSO to work. NOTE: Solarflare recommend that you do not disable this setting. Issue 11...
Page 230 Receive Side Scaling (RSS) was first supported as part of the scalable networking pack for Windows Server 2003 and has been improved with each subsequent operating system release. RSS is enabled by default and will be used on network adapters that support it. Solarflare recommend that RSS is enabled for best networking performance.
Page 231 (2.5Gbps for PCIe Gen 1, 5.0 Gbps for PCIe Gen 2 and 8.0Gbps for PCIe Gen 3) in each direction. Solarflare adapters are designed for x8 or x16 lane operation.
Page 232 Intel® QuickData / NetDMA On systems that support Intel I/OAT (I/O Acceleration Technology) features such as QuickData (a.k.a NetDMA), Solarflare recommend that these are enabled as they are rarely detrimental to performance. Using Intel® QuickData Technology allows data copies to be performed by the system and not the operating system.
Page 233 CPU core is signaled to return to the fully on power state. These changes in CPU core power states create additional network latency and jitter. Solarflare recommend to achieve the lowest latency and lowest jitter that the "C1E power state" or "CPU power saving mode"...
Page 234 NOTE: The Windows (or any third party) Firewall should be disabled with caution. The network administrator should be consulted before making any changes. Issue 11 © Solarflare Communications 2014...
Page 235 ================ =========== ================== 52420.411392 44.767 1459.846 9367.688 Total Buffers Throughput(Buffers/s) Pkts(sent/intr) Intr(count/s) Cycles/ Byte ============= ===================== 99984.000 2233.431 27 29187.48 0.8 Packets Sent Packets Received Total Retransmits Total Errors Avg. CPU % ============ ================ ================= Issue 11 © Solarflare Communications 2014...
Page 236 Ensure Memory utilizes all memory channels on system motherboard. Intel QuickData (Intel chipsets Enable in BIOS and configure as described in guide. only) Table 55: Latency Tuning Settings Tuning Parameter How? Adaptive Interrupt Leave at default (Enabled). Moderation Issue 11 © Solarflare Communications 2014...
Page 237 Disable C1E and other CPU sleep modes to prevent OS from putting CPUs into lowering power modes when idle. Memory bandwidth Ensure Memory utilizes all memory channels on system motherboard. Intel QuickData (Intel chipsets Enable in BIOS and configure as described in guide. only) Issue 11 © Solarflare Communications 2014...
Page 238: Windows Event Log Error Messages
(SAFEMODE_MINIMA 0xA0000005L BUS_STATUS_DRIVER_NUMA_ALLO Warning The driver could not For maximum CATION_FAILED allocate memory on a performance all specific NUMA node. NUMA nodes should be populated. Install additional memory. Issue 11 © Solarflare Communications 2014...
Page 239 BUS_STATUS_DEVICE_TX_WATCHD Warning The transmit watchdog fired 0xA001000D BUS_STATUS_DEVICE_UNEXPECTED Warning An unexpected No action required. _EVENT event was received from the device. 0xA0010010L BUS_STATUS_DEVICE_WRONG_RX_ Warning A non-contiguous EVENT RX event was received from the device. Issue 11 © Solarflare Communications 2014...
Page 240 Hardware MCDI None required. communication timed out. 0xA0010019L BUS_STATUS_DEVICE_MCDI_BOOT Warning Hardware MCDI Run sfupdate or _ERROR boot from non- update via SAM. primary flash. Possible flash corruption. 0x6001001BL BUS_STATUS_DEVICE_MCDI_VERSI Informational Hardware MCDI None required. version Issue 11 © Solarflare Communications 2014...
Page 241: Chapter 5: Solarflare Adapters On Vmware
This chapter covers the following topics on the VMware® platform: • System Requirements...Page 236 • VMware Feature Set...Page 237 • Installing Solarflare Drivers and Utilities on VMware...Page 238 • Configuring Teams...Page 239 • Configuring VLANs...Page 239 • Running Adapter Diagnostics...Page 241 •...
Page 242: Vmware Feature Set
Support for diskless booting to a target operating system via PXE boot. • See Sfboot: Command Line Options on page 242 • See Solarflare Boot ROM Agent on page 344 Fault diagnostics Support for comprehensive adapter and cable fault diagnostics and system reports. • See...
Page 243: Installing Solarflare Drivers And Utilities On Vmware
Installing on VMware vSphere 4.0 and 4.1 The Solarflare adapter drivers are provided as an iso image file. Copy the .iso image to a CD-ROM and refer to the VMware install instructions in the VMware NIC Device Driver Configuration Guide available from http://www.vmware.com/support/pubs/.
Page 244: Configuring Teams
You can configure the following settings: • Load Balancing • Network Failover Detection • Notify Switches • Failover • Failover Order 5.5 Configuring VLANs There are three methods for creating VLANs on VMware ESX: Virtual Switch Tagging (VST) Issue 11 © Solarflare Communications 2014...
Page 245 • NIC teaming in VMware ESX Server: http://kb.vmware.com/selfservice/microsites/ search.do?cmd=displayKC&docType=kc&externalId=1004088&sliceId=1&docTypeID=DT_KB_1_1& dialogID=40304190&stateId=0%200%2037866989 • VMware ESX Server host requirements for link aggregation: http://kb.vmware.com/selfservice/microsites/ search.do?language=en_US&cmd=displayKC&externalId=1001938 • VLAN Configuration on Virtual Switch, Physical Switch, and virtual machines: http://kb.vmware.com/selfservice/microsites/ search.do?language=en_US&cmd=displayKC&externalId=1003806 Issue 11 © Solarflare Communications 2014...
Page 246: Running Adapter Diagnostics
The driver utilizes multiple event queues to spread the load over multiple CPU cores (RSS). phy.bist Examines the PHY by initializing it and causing any available built-in self tests to run. Issue 11 © Solarflare Communications 2014...
Page 247: Configuring The Boot Rom With Sfboot
• Sfboot: Command Line Options...Page 242 • Sfboot: Examples...Page 250 Sfboot is a command line utility for configuring the Solarflare adapter Boot ROM for PXE and iSCSI booting. Using sfboot is an alternative to using Ctrl+B to access the Boot Rom agent during server startup.
Page 248 MAC address, as output by the -- option. If is not included, the list --adapter action will apply to all installed Solarflare adapters. Resets all adapter options except --clear boot-image their default values. Note that can also be...
Page 249 = 0-256 seconds Sets the adapter boot type. boot- type=<pxe|iscsi|disabled> – PXE (Preboot eXecution Environment) booting – iSCSI (Internet Small Computer System iscsi Interface) booting – Disable adapter booting disabled Issue 11 © Solarflare Communications 2014...
Page 250 Note that this option initiator-dhcp is only valid if iSCSI booting is enabled ( boot- type=iscsi Example: sfboot boot-type=iscsi initiator- dhcp=disabled gateway=192.168.0.10 Issue 11 © Solarflare Communications 2014...
Page 251 The IQN is a symbolic name in the “.” notation form; for example: iqn.2009.01.com.solarflare, and is a maximum of 223 characters long. Note that this option is only valid if iSCSI booting is enabled (...
Page 252 Specifies the Logical Unit Number (LUN) to be used target-lun=<LUN> by the iSCSI target when is disabled. target-dhcp The default LUN is 0. Note that this option is only valid if iSCSI booting is enabled ( boot-type=iscsi Issue 11 © Solarflare Communications 2014...
Page 253 Note that this option is only valid if iSCSI booting is enabled (boot-type=iscsi To be valid, this option also requires the following sub-options to be specified: username=<initiator username> secret=<initiator password> Example: sfboot boot-type=iscsi chap=enabled username=initiatorusername secret=initiatorsecret Issue 11 © Solarflare Communications 2014...
Page 254 Enables/disables Mutual CHAP authentication when mutual- chap=<enabled|disabled> iSCSI booting is enabled. This option also requires the following sub-options to be specified: target-username=<username> target-secret=<password> username=<username> secret=<password> Example: sfboot boot-type=iscsi mutual- chap=enabled username=username secret=veryverysecret target- username=targetusername target- secret=anothersecret Issue 11 © Solarflare Communications 2014...
Page 255 The default is 32. Note: Using the incorrect setting can impact the performance of the adapter. Contact Solarflare technical support before changing this setting. Sfboot: Examples • Show the current boot configuration for all adapters: Issue 11 ©...
Page 256 Solarflare Server Adapter User Guide Sfboot Solarflare boot configuration utility [v3.0.3] Copyright Solarflare Communications 2006-2010, Level 5 Networks 2002-2005 eth1: Boot image Option ROM and UEFI Link speed Negotiated automatically Link-up delay time 5 seconds Banner delay time 2 seconds...
Page 257: Upgrading Adapter Firmware With Sfupdate
Solarflare adapter you want to upgrade. Specifing the adapter vmnicX is optional - if it is not included the command is applied to all Solarflare adapters in the machine. is one of the command options listed in...
Page 258 Specifies a specific firmware image. --image=<filename> This option is not normally required and is only necessary if you need to provide writing the sfupdate embedded image file. Prompts for user confirmation before writing the -y, --yes firmware. Issue 11 © Solarflare Communications 2014...
Page 259 Solarflare Server Adapter User Guide Sfupdate: Examples • List all Solarflare adapters installed on the host with the installed firmware: sfupdate Solarflare firmware update utility [v3.0.3] Copyright Solarflare Communications 2006-2010, Level 5 Networks 2002-2005 eth1 - MAC: 00-0F-53-01-39-70 Firmware version: v3.0.3...
Page 260: Performance Tuning On Vmware
The purpose of this section is to highlight adapter driver settings that affect the performance metrics described. This guide covers the tuning of all members of the Solarflare family of adapters. Performance between adapters should be identical, with the exception of latency measurements.
Page 261 # rpm –i /mnt/VMwareTools*.rpm # vmware-tools-config.pl VMware ESX NetQueue Solarflare adapters supports VMware’s NetQueue technology, which accelerates network performance in 10 Gigabit Ethernet virtualized environments. NetQueue is enabled by default in VMware versions. There is usually no reason not to enable NetQueue.
Page 262 $IRQVEC # cat /proc/vmware/interrupts Locate the interrupts associated with the Solarflare adapter (e.g. vmnic2). Interrupts are listed in order: the first interrupt will be for the default queue, the second interrupt for the queue dedicated to the first virtual machine to have been started, the third interrupt for the queue dedicated to the second virtual machine to have been started, and so on.
Page 263 Ethernet endpoints. However if a larger MTU is used, adapter throughput and CPU utilization can be improved. CPU utilization is improved because it takes fewer packets to send and receive the same amount of data. Solarflare adapters support frame sizes up to 9216 bytes (this does not include the Ethernet preamble or frame-CRC).
Page 264 For example, an interrupt moderation setting of 30μs will not delay the reception of the first packet received, but the interrupt for any following packets will be delayed until 30μs after the reception of that first packet. Issue 11 © Solarflare Communications 2014...
Page 265 –k Solarflare recommend you do not disable checksum offload. For advice on configuring checksum offload in the guest, consult the relevant Solarflare section for that guest, or the documentation for the guest operating system. TCP Segmentation Offload (TSO) TCP Segmentation offload (TSO) offloads the splitting of outgoing TCP data into packets to the adapter.
Page 266 In x4 lane slots, Solarflare PCIe adapters will continue to operate, but not at full speed. The Solarflare driver will warn you if it detects the adapter is plugged into a PCIe slot which electrically has fewer than x8 lanes.
Page 267 I/O AT can be enabled on the host and on guest operating systems. For advice on enabling I/OAT in the guest, consult the relevant Solarflare section for that guest, or the documentation for the guest operating system. I/OAT must be enabled on the host if it is to be used in the guests.
Page 268: Chapter 6: Solarflare Adapters On Solaris
• Kernel and Network Adapter Statistics...Page 294 6.1 System Requirements Refer to Software Driver Support on page 12 for details of supported Solaris distributions. Solarflare drivers for Solaris support the GLDv3 API, but do not support the Crossbow API framework. Issue 11 © Solarflare Communications 2014...
Page 269: Solaris Platform Feature Set
Solarflare Server Adapter User Guide 6.2 Solaris Platform Feature Set Table 63 lists the features supported by Solarflare adapters on Solaris. Table 63: Solaris Feature Set Jumbo frames Support for MTUs (Maximum Transmission Units) to 9000 bytes. • See Configuring Jumbo Frames on page 269...
Page 270: Installing Solarflare Drivers
User Guide 6.3 Installing Solarflare Drivers The Solaris drivers for Solarflare are available in a binary package for both 32 and 64 bit platforms. • A driver package (pkg format) is available for Solaris 10.8, 10.9 and 10.10. • A driver package (pkg format) is available for Solaris 11.0.
Page 271: Unattended Installation Solaris 10
These processes are documented here: http://docs.oracle.com/cd/E19253-01/819-6397/819-6397.pdf NOTE: The Solarflare server adapter can be used to PXE boot the installer, but as there is no driver, the adapter can not be used during installation. To install the Solarflare Solaris package as part of an unattended installation, it must be added using the package command to the JumpStart machine profile.
Page 272: Unattended Installation Solaris 11
6.5 Unattended Installation Solaris 11 Please refer to the Oracle Solaris 11 documentation for details of transitioning from Solaris 10 to 11 or for details of the Automated Installer feature for Solaris 11. https://blogs.oracle.com/unixman/entry/how_to_get_started_with https://blogs.oracle.com/unixman/entry/migrating_from_jumpstart_to_automated Issue 11 © Solarflare Communications 2014...
Page 273: Configuring The Solarflare Adapter
<x> There will be one interface per physical port on the Solarflare adapter. To plumb an interface enter the following: ifconfig sfxge<x> plumb You then need to configure the interface and bring it up to allow data to pass. Enter the following: ifconfig sfxge<x>...
Page 274 The default setting in is 1500. sfxge.conf Solarflare adapters support frame sizes from 1500 bytes to 9000 bytes. For example, to set a new frame size (MTU) of 9000 bytes, enter the following command: $ ifconfig sfxge<x> mtu 9000 To view the current MTU, enter: $ ifconfig sfxge<x>...
Page 275: Setting Up Vlans
“trunked” mode) see the “How to Configure a VLAN” section in the following documentation: http://docs.oracle.com/cd/E19253-01/816-4554/fpdga/index.html 6.8 Solaris Utilities Package The Solarflare Solaris Utilities package, supplied as a 32 bit SVR package and available from https:// support.solarflare.com/ contains the following utilities:...
Page 276 Solarflare Server Adapter User Guide Sfboot is a command line utility for configuring the Solarflare adapter Boot ROM for PXE and iSCSI booting. Using sfboot is an alternative to using Ctrl + B to access the Boot Rom agent during server startup.
Page 277 Performs the operation on a specified remote --computer <computer_name> computer. Administrator rights on the remote computer is required. Lists all available Solarflare adapters. This option --list shows the adapter’s ID number, ifname and MAC address. Note: this option may not be used in conjunction with the any other option.
Page 278 Specifies the wait period for Ctrl-B to be pressed to banner-delay=<seconds> enter adapter configuration tool. = 0-256 seconds Specifies the time allowed for Esc to be pressed to bootskip-delay=<seconds> skip adapter booting. = 0-256 seconds Issue 11 © Solarflare Communications 2014...
Page 279 Note that this option initiator-dhcp is only valid if iSCSI booting is enabled ( boot- type=iscsi Example: sfboot boot-type=iscsi initiator- dhcp=disabled netmask=255.255.255.0 Issue 11 © Solarflare Communications 2014...
Page 280 The IQN is a symbolic name in the “.” notation form; for example: iqn.2009.01.com.solarflare, and is a maximum of 223 characters long. Note that this option is only valid if iSCSI booting is enabled (...
Page 281 Specifies the Logical Unit Number (LUN) to be used target-lun=<LUN> by the iSCSI target when is disabled. target-dhcp The default LUN is 0. Note that this option is only valid if iSCSI booting is enabled ( boot-type=iscsi Issue 11 © Solarflare Communications 2014...
Page 282 Note that this option is only valid if iSCSI booting is enabled (boot-type=iscsi To be valid, this option also requires the following sub-options to be specified: username=<initiator username> secret=<initiator password> Example: sfboot boot-type=iscsi chap=enabled username=initiatorusername secret=initiatorsecret Issue 11 © Solarflare Communications 2014...
Page 283 Enables/disables Mutual CHAP authentication when mutual- chap=<enabled|disabled> iSCSI booting is enabled. This option also requires the following sub-options to be specified: target-username=<username> target-secret=<password> username=<username> secret=<password> Example: sfboot boot-type=iscsi mutual- chap=enabled username=username secret=veryverysecret target- username=targetusername target- secret=anothersecret Issue 11 © Solarflare Communications 2014...
Page 284 The default is 32. 2|1024> Note: Using the incorrect setting can impact the performance of the adapter. Contact Solarflare technical support before changing this setting. Enable SR-IOV support for OS system that support it. sriov=<enabled|disabled>...
Page 285 MSI-X interrupt limit sfxge1: Boot image Disabled MSI-X interrupt limit • List all Solarflare adapters installed on the localhost: sfboot --list Solarflare boot configuration utility [v3.0.5] Copyright Solarflare Communications 2006-2010, Level 5 Networks 2002-2005 sfxge0 - 00-0F-53-01-38-40 sfxge1 - 00-0F-53-01-38-41 Issue 11 ©...
Page 286: Upgrading Adapter Firmware With Sfupdate
Running the command with no additional parameters will show the current firmware sfupdate version for all Solarflare adapters and whether the firmware within sfupdate is more up to date. To update the firmware for all Solarflare adapters run the command sfupdate --write Solarflare recommend that you use sfupdate in the following way: to check that the firmware on all your adapters are up to date.
Page 287 Update the firmware using the image contained in the --image=(filename) specified file, rather than the image embedded in the utility. Use with the and, if needed, --write --force options. Prompts for user confirmation before re-writing the -y, --yes firmware. Issue 11 © Solarflare Communications 2014...
Page 288 User Guide Sfupdate: Examples • Display firmware versions for all adapters: sfupdate sfupdate: Solarflare Firmware Update Utility [v3.0.5.2164] Copyright Solarflare Communications 2006-2010, Level 5 Networks 2002-2005 Network adapter driver version: v3.0.5.2163 sfxge0 - MAC: 00:0F:53:01:38:90 Firmware version: v3.0.5 Boot ROM version: v3.0.5.2163...
Page 289: Performance Tuning On Solaris
The purpose of this guide is to highlight adapter driver settings that affect the performance metrics described. This guide covers the tuning of all of the Solarflare family of adapters. In addition to this guide, you may need to consider other issues influencing performance such as application settings, server motherboard chipset, additional software installed on the system, such as a firewall, and the specification and configuration of the LAN.
Page 290 Ethernet endpoints. However if a larger MTU is used, adapter throughput and CPU utilization can be improved. CPU utilization is improved because it takes fewer packets to send and receive the same amount of data. Solarflare adapters support frame sizes up to 9000 bytes (this does not include the Ethernet preamble or frame-CRC).
Page 291 -set /dev/tcp tcp_rst_sent_rate_enabled 0 # disable rst rate limiting See the Internet Protocol Suite Tunable Parameters chapter of the Tunable Parameters Reference Guide for more details: http://docs.oracle.com/cd/E19082-01/819-2724/6n50b07lr/index.html NOTE: You can also add these settings to /etc/system Issue 11 © Solarflare Communications 2014...
Page 292 NOTE: RSS also works for UDP packets. For UDP traffic the Solarflare adapter will select the Receive CPU based on IP source and destination addresses. SFN5xxx adapters support IPv4 and IPv6 RSS, while SFN4xxx adapters support just IPv4 RSS.
Page 293 In x4 lane slots, Solarflare PCIe adapters will continue to operate, but not at full speed. The Solarflare driver will warn you if it detects the adapter is plugged into a PCIe slot which electrically has fewer than x8 lanes.
Page 294 PCI Express Lane Configuration Ensure current speed (not the supported speed) reads back as “x8 and 5Gb/s” Or “x8 and Unknown” CPU Power Management Leave enabled Memory bandwidth Ensure Memory utilizes all memory channels on system motherboard Issue 11 © Solarflare Communications 2014...
Page 295 Leave at default TCP Segmentation Offload Leave at default TCP Large Receive Offload Leave at default TCP Protocol Tuning Can leave at default Receive Side Scaling (RSS) sfxge.conf Buffer Allocation Method Leave at default Issue 11 © Solarflare Communications 2014...
Page 296 Ensure current speed (not the supported speed) reads back as Configuration “x8 and 5Gb/s” Or “x8 and Unknown” CPU Speed service (cpuspeed) Leave enabled Memory bandwidth Ensure Memory utilizes all memory channels on system motherboard Issue 11 © Solarflare Communications 2014...
Page 297: Module Parameters
Solarflare Server Adapter User Guide 6.12 Module Parameters The normal syntax when using parameters is param=<value>. Table 72 lists the available parameters in the Solarflare Solaris driver module ( sfxge.conf Table 72: Driver Module Parameters Parameter Description Values Default Maximum number of RSS channels to rx_scale_count use per port.
Page 298 Number of pkt buffers to allocate at start limited by rx_prealloc_pkt of a receive queue and maintain a free available system _buffers pkt pool of at least this many buffers. memory. Issue 11 © Solarflare Communications 2014...
Page 299: Kernel And Network Adapter Statistics
Solarflare Server Adapter User Guide 6.13 Kernel and Network Adapter Statistics Statisitical data originating from the MAC on Solarflare network adapters can be gathered using the Solaris Kernel Statistics command ( ). The following tables identify kernel and adapter kstat...
Page 300 1 = send pause frames when there is receive congestion and pause transmission when a pause frame is received. collisions Number of collisions detected when attempting to send. crtime Timestamp when samples were taken. Issue 11 © Solarflare Communications 2014...
Page 301 0 = in the event of receive congestion, the local host will transmit pause frames to the peer. link_state link status 0 = link down, 1 = link up. link_up 1 =link is up, 0 = link is down. Issue 11 © Solarflare Communications 2014...
Page 302 Count the number of frames for which transmission fails due to multiple collisions. multircv Number of multicast packets received. multixmt Number of multicast packets transmitted. norcvbuf noxmtbuf obytes Number of bytes output (32 bit counter). Issue 11 © Solarflare Communications 2014...
Page 303 Late collisions can occur if the length of the network seg- ment is greater than the standard allowed length. Collision occurred beyond the collision window (512 bit times). This should always be zero as Solarflare adapters operate in full duplex mode. unknowns xcvr_addr MII address in the 0-31 range of the physical layer device in use for a given Ethernet device.
Page 304 Table 74: Name sfxge_cfg Parameter Description crtime Timestamp when samples were taken. Adapter hardware address. version Solarflare sfxge driver version Table 75: Name sfxge_mac Parameter Description crtime Timestamp when samples were taken. link_duplex 0 = down, 1 = half duplex, 2 = full duplex.
Page 305 Number of packets received with FCS errors - these are dropped by the Solarflare driver. rx_ge_15xx_pkts Number of packets received with payload size greater than 1518 bytes (1522 bytes VLAN).
Page 306 Number of packets transmitted where the length is between 128 and 255 bytes. tx_256_to_511_pkts Number of packets transmitted where the length is between 256 and 511 bytes. tx_512_to_1023_pkts Number of packets transmitted where the length is between 512 and 1023 bytes. Issue 11 © Solarflare Communications 2014...
Page 307 Collision occurred beyond the collision window (512 bit times). This should always be zero as Solarflare adapters operate in full duplex mode. tx_le_64_pkts Number of frames transmitted where the length is less than 64 bytes.
Page 308 Ethernet device. The parameters are also a subset of the cap_1000hdx cap_* statistics. If the cap_* value is 0, the corresponding cap_100fdx adv_cap_* must also be 0, except for adv_cap_asmpause and adv_cap_pause parameters. cap_100hdx cap_10fdx cap_10gfdx cap_10hdx cap_asm_pause cap_autoneg cap_pause Issue 11 © Solarflare Communications 2014...
Page 309 Number of RSS channels to use per port. Default is 128, Minimum is 1. Table 77: Name sfxge_rss Parameter Description crtime Timestamp when samples were taken. evq0000_count Number of RSS table entries for this event queue. scale Actual number of MSI-X interrupts. Issue 11 © Solarflare Communications 2014...
Page 310 Number of times the Deferred Packet List limit was reached dpl_put_pkt_limit Deferred Packet max packet limit. unaligned_split Always 0. Table 80: Name sfxge_vpd Parameter Description crtime Timestamp when samples were taken. Engineering change data. Solarflare adapter type. Solarflare adapter part number. Issue 11 © Solarflare Communications 2014...
Page 311 Solarflare Server Adapter User Guide Table 80: Name sfxge_vpd Parameter Description Solarflare adapter serial number. Adapter firmware version. Issue 11 © Solarflare Communications 2014...
Page 312: Chapter 7: Sr-Iov Virtualization Using Kvm
• Red Hat Enterprise Linux 6.0, 6.1 and 6.2 Adapters • All Solarflare SFN5xxx and SFN6xxx series adapters. Solarflare adapters support 127 VFs per port and 1024 vNICs each with a dedicated MSI-X interrupt. Acceleration of guest VMs running other (non-Linux) operating systems is not currently supported although Solarflare are considering accelerating Windows guests in the longer term.
Page 313: Linux Kvm Sr-Iov
Figure 48: Guest to Guest Connectivity The bridge establishes guest to guest connectivity. Unfortunately performance is poor, requiring data to be copied between the guest and KVM host, and between user level and kernel within the KVM host. Issue 11 © Solarflare Communications 2014...
Page 314 It is also worth noting that the networking interface in this model is now a vendor supplied driver and creation/configuration of networking interfaces must be done via this vendor driver and not the standard KVM network drivers. Issue 11 © Solarflare Communications 2014...
Page 315 User Guide Solarflare Plug-in Model The Solarflare model for SR-IOV support is a hybrid of the traditional and typical approaches. The Solarflare "plugin" maintains the traditional (software) data path through virtio frontend to the KVM host (and then through the Linux bridge to the PF of the network driver). However, there is also an alternative (accelerated) data path through the VF driver directly to the network adapter from the guest.
Page 316: Installation
SR-IOV solution or to a KVM machine with a non- Solarflare adapter. • Large number of VFs. Solarflare adapters support 127 VFs per port - far more than other SR-IOV capable adapters. This allows up to 127 VMs per 10G port to be accelerated per host.
Page 317 NOTE: Ensure that VFs are NOT in use when you unload the sfc driver. This is particularly relevant if you are upgrading from a previous alpha version of the Solarflare KVM SR-IOV package. Step 2: Update adapter firmware and enable SR-IOV Download the Solarflare Linux Utilities SF-104451-LS from support.solarflare.com:...
Page 318 Guest Drivers The Solarflare accelerated driver package is currently supplied as a tarball. It contains two drivers: an upgraded virtio-net driver (backported from 2.6.37), and xnap, the PCI driver for Solarflare VFs. To install drivers on Red Hat 6.1: # tar xvfz sfc-xnap-virtio-v1_0_0_0012.tgz...
Page 319: Configuration Red Hat 6.1
This section presents an overview of how to configure virtio based networking. • Configure Bridged networking on the hosts, creating a bridge for each Solarflare network device and attach the physical network device to the bridge. The instructions to configure bridging can be found at Redhat: Bridged networking with libvirt.
Page 320 <vf-hotplug source='eth4' hybrid='yes' vlan='123' /> </interface> Virt-manager configuration for SR-IOV: Adding a new interface A virt-manager GUI tool is supplied with the Solarflare driver distribution package. To run the GUI tool as root use the following command: # /usr/bin/virt-manager Alternatively, virt-manager can be on another host and connect to the KVM host via “File”- >”Add Connection”.
Page 321 VFs available when the VM is started then one (or more) VFs will be attached into the running guest. If the guest OS has the Solarflare VF drivers, and the interface is a virtio interface, then the virtio interface will be accelerated.
Page 322: Configuration Red Hat 6.2
7.5 Configuration Red Hat 6.2 Host Network Configuration As previously described, Solarflare SR-IOV support uses a plugin model, whereby virtual functions accelerate virtio network adapters. These virtio network adapters are attached to the physical network device via a macvtap interface.
Page 323 </interface> Virt-manager configuration for SR-IOV: Adding a new interface This Solarflare driver distribution includes an updated version of virt-manager - a GUI tool for managing VMs (refer to Virt-manager configuration for SR-IOV: Adding a new interface on page 315 for details).
Page 324 If there are spare VFs available when the VM is started then one (or more) VFs will be attached into the running guest. If the guest OS has the Solarflare VF drivers, and the interface is a virtio interface, then the virtio interface will be accelerated.
Page 325: Performance Tuning
- Host OS: RHEL 6.2 64 bit. - The irqbalance service is stopped. • Guest - The guest VM was running RHEL6.1 64bit. - The irqbalance service was stopped in the guest VM. - Benchmarking tool: Netperf. Issue 11 © Solarflare Communications 2014...
Page 326 Table 83: Linux KVM Performance Number of Number Baseline (no Accelerated Traffic 10G ports of VMs SR-IOV) (SR-IOV plug-in) Unidirectional 6.64 Gbps 9.37 Gbps Bidirectional 6.74 Gbps 11.5 Gbps Bidirectional 7.37 Gbps 11.2 Gbps Issue 11 © Solarflare Communications 2014...
Page 327: Migration
• RHEL 6.2 driver - Libvirt will not configure VFs manually added to a VM configuration (via virt- manager or virsh edit). Only VFs attached via the pci_passthrough-hybrid forward mode XML element will be configured. and pf dev Issue 11 © Solarflare Communications 2014...
Page 328 Solarflare are following up this issue with RedHat. A workaround for this issue is disabling tx-checksumming on the PXE server VM.
Page 329: Chapter 8: Sr-Iov Virtualization For Xenserver
Currently the default configuration is 127 VFs each allocated 1 vNIC. NOTE: Solarflare’s hybrid SR-IOV solution is no longer supported by Citrix and is limited to XenServer 6.0. Further, this solution is limited to supporting Windows guests and Linux guests running RHEL 6.1 (or earlier).
Page 330: Xenserver6 Sr-Iov
(virtualization, data copies, etc.) are eliminated, providing significantly improved performance, see Figure However, a significant downside of this approach is that XenServer blocks migration of a VM when a VF is present, which significantly reduces the functionality of server virtualization. Issue 11 © Solarflare Communications 2014...
Page 331 Unlike other adapters, each Solarflare VF can use multiple DMA channels that enable the VF to scale over vCPUs using RSS and/or RFS. Solarflare's 2048 DMA channels provide the most scalable virtualized network I/O solution available in the market.
Page 332 SR-IOV solution or to a XenServer system with a non- Solarflare adapter. • Large number of VFs. Solarflare adapters support 127 VFs per port - far more than other SR-IOV capable adapters. This allows up to 127 VMs per 10G port to be accelerated per host.
Page 333: Installation
(RSS) and is typically used in non-virtualized environments (Linux/ Windows) to improve network performance. With the Solarflare plugin model when VMs are deployed to run on multiple (virtual) CPUs, each CPU can be given a dedicated transmit and receive queue so that network performance scales over the available CPU cores.
Page 334: Configuration
For Linux Guests: # tar xvfz sfc-xnap-netfront-v1_0_0_0012.tgz # make -C sfc-xnap-netfront-v1_0_0_0012/ modules_install For Windows Guests: Install the Solarflare acceleration driver into the Windows guest using xnapsetup.exe. 8.4 Configuration Acceleration The SR-IOV supplemental pack manages all VF configuration. SR-IOV acceleration is configured on a per Virtual Interface (VIF) basis allowing the administrator to control which VMs are accelerated on a per network basis.
Page 335: Performance Tuning
Name Application Parameter Windows Linux Streaming Latency Receive ring size rx_ring rx_ring Default Default Interrupt Moderation irq_usecs rx_irq_mod_use Default Windows Guests Install the Solarflare SR-IOV guest drivers for Windows. Create the following registry key: Issue 11 © Solarflare Communications 2014...
Page 336 Accelerated Traffic 10G ports of VMs SR-IOV) (SR-IOV plug-in) Unidirectional 1.66 Gbps 3.49 Gbps Bidirectional 2.52 Gbps 3.88 Gbps Bidirectional 2.71 Gbps 4.20 Gbps Unidirectional 4.85 Gbps 9.39 Gbps Bidirectional 5.99 Gbps 17.9 Gbps Issue 11 © Solarflare Communications 2014...
Page 337 VMs SR-IOV) (SR-IOV plug-in) Unidirectional 2.23 Gbps 9.25 Gbps Bidirectional 1.91Gbps 15.3 Gbps Bidirectional 3.65 Gbps 30.7 Gbps Unidirectional 4.80 Gbps 9.31 Gbps Bidirectional 5.80 Gbps 14.9 Gbps Bidirectional 6.24 Gbps 32.8 Gbps Issue 11 © Solarflare Communications 2014...
Page 338 Solarflare Server Adapter User Guide Limitations • The Citrix XenTools bundled with XenServer6 do not support RSS. Therefore the Solarflare SR- IOV accelerated drivers currently do not support RSS on Windows. Known Issues • Some hardware platforms are unable to support 127 VFs on both ports simultaneously. The...
Page 339: Chapter 9: Solarflare Adapters On Mac 0S X
• Refer to Software Driver Support on page 12 for supported Mac OS X Distributions. • Solarflare Mac OS X drivers are supported for all Solarflare SFN5xxx AND SFN6xxx series adapters. • Driver package SF-107120-LS supports OS X 10.8 and earlier versions.
Page 340: Mac 0S X Platform Feature Set
Solarflare Server Adapter User Guide 9.3 Mac 0S X Platform Feature Set The following table lists the features supported by Solarflare adapters on Mac OS X distributions. Table 88: Mac OS X Feature Set Large Receive Offload TCP receive frame coalescing to reduce CPU utilization and...
Page 341 Figure 57: Disable Driver Services Repeat above steps for all ports of the driver. Double-click SF-107120-LS.dmg in Finder to mount the disk image. Invoke the Solarflare driver uninstall script in Terminal as root (replacing <version> with the version number of the install package that is being used) /Volumes/Solarflare10GbE-<version>/uninstall.sh...
Page 342 Download and Install the Mac OS X Driver Download SF-107120-LS.dmg into a convenient working directory. Double click SF-107120-LS.dmg in Finder to mount the disk image. Run the Solarflare10GbE.pkg install package and follow the install instructions. Figure 58: Install Solarflare Driver Window Issue 11 © Solarflare Communications 2014...
Page 343: Interface Configuration
Solarflare Server Adapter User Guide 9.6 Interface Configuration With the adapter driver installed, the network interface can be configured using the network interface settings menu: Figure 59: Solarflare Adapter Interface Configuration Issue 11 © Solarflare Communications 2014...
Page 344: Tuning
Table 89: Mac OS X sysctl driver values sysctl name Value Description Driver version string net.sfxge.version MAC address net.sfxge.<enX>.mac Disable interrupt moderation net.sfxge.<enX>.moderation > 0 interrupt moderation (microsecs) Issue 11 © Solarflare Communications 2014...
Page 345: Firmware Update
IPv6 TSO disabled net.sfxge.<enX>.ipv6tso IPv6 TSO enabled 9.9 Firmware Update The Solarflare driver package for Apple Mac OS X also includes the firmware update utility program sfupdate. When the driver package is installed the sfupdate binary is installed into /Library/Application...
Page 346 Running the command with no additional parameters will display the current firmware sfupdate version for all Solarflare adapters and identifies whether the firmware within sfupdate is more up to date. sfupdate: All Solarflare adapters to check that the firmware on all adapters is up to date.
Page 347: Performance
-V --version 9.10 Performance The following section is an overview of benchmark tests results measured by Solarflare to provide an indication of expected performance with current drivers. Performance tests were conducted on Mac OS X 10.7.2 on a pair of Mac Pro servers configured back- to-back.
Page 348 720 X 468, 8 bit 439.5 547.6 433.0 574.4 1920 x 1080, 10 bit 509.3 728.3 502.3 770.1 4096 x 2160, 10 bit-RGB 521.3 807.0 516.0 849.5 1920 X 1080, 10 bit 312.7 255.7 370.0 291.3 Issue 11 © Solarflare Communications 2014...
Page 349: Chapter 10: Solarflare Boot Rom Agent
PXE support when this is required. The Boot ROM Agent can also be invoked using the Solarflare supplied sfboot utility - For instructions on the sfboot method refer to the sfboot commands in the relevant OS section of this user guide. PXE boot is supported on all Solarflare adapters.
Page 350: Pxe Support
242. 10.2 PXE Support Solarflare Boot ROM agent supports the PXE 2.1 specification. PXE requires DHCP and TFTP Servers, the configuration of these servers depends on the deployment service used. Linux For Red Hat Enterprise and SUSE Linux Enterprise Server, please consult your Linux documentation.
Page 351 Solarflare Server Adapter User Guide On starting or re-starting the server, press Ctrl+B when prompted. The Solarflare Boot Configuration Utility is displayed. Issue 11 © Solarflare Communications 2014...
Page 352 Solarflare Server Adapter User Guide Use the arrow keys to highlight the adapter you want to boot via PXE and press Enter. The Adapter Menu is displayed. Issue 11 © Solarflare Communications 2014...
Page 353 Type. From the Boot Type, press Space until PXE is selected. Solarflare recommend leaving the Adapter Options and BIOS Options at their default values. For details on the default values for the various adapter settings, see Table 94 on page 367.
Page 354: Iscsi Boot
Computer System Interface (iSCSI). iSCSI is a fast, efficient method of implementing storage area network solutions. The Boot ROM in the Solarflare adapter contains an iSCSI initiator allowing the booting of an operating system directly from an iSCSI target. NOTE: Adapter teaming and VLANs are not supported in Windows for iSCSi remote boot enabled Solarflare adapters.
Page 355 Solarflare Server Adapter User Guide Start or re-start the iSCSI initiator server and when prompted, press Ctrl+B. The Solarflare Boot Configuration Utility will display. NOTE: If the BIOS supports console redirection, and you enable it, then Solarflare recommends that you enable ANSI terminal emulation on both the BIOS and your terminal. Some BIOSs are known to not render the Solarflare Boot Manager properly when using vt100 terminal emulation.
Page 357 Solarflare Server Adapter User Guide From the BootROM Mode option, press the space bar to change the Boot Image and or the Boot Type. Issue 11 © Solarflare Communications 2014...
Page 358 Solarflare Server Adapter User Guide From the Boot Type, press Space until iSCSI is selected. Press Enter. The iSCSI Initiator options are displayed. Issue 11 © Solarflare Communications 2014...
Page 359 DHCP Server Setup on page 356 for more details Initiator IQN: The iSCSI initiator IQN of the Solarflare adapter if you are not using DHCP to obtain the iSCSI initiator IQN. DHCP Vendor Class ID: If you are using DHCP to obtain the iSCSI initiator IQN, the adapter will use DHCP option 43 to try and obtain this information from the DHCP server.
Page 360 You can use the MPIO option to configure the MPIO rank for all adapters. Ensure all adapters to be used for MPIO are correctly configured for iSCSI boot. Highlight iSCSI MPIO and press Enter. Issue 11 © Solarflare Communications 2014...
Page 361: Dhcp Server Setup
Solarflare Server Adapter User Guide Note that you can set the MPIO rank for all Solarflare adapters from the configuration menus of any of the available adapters. Press Esc to return to the Adapter menu. When you have finished, select Save and exit.
Page 362 “vendor specific information” and requires that the vendor id (DHCP option 60) configured at the DHCP server matches the vendor id configured in the Boot ROM. By default the Boot ROM uses the vendor id SFCgPXE Issue 11 © Solarflare Communications 2014...
Page 363: Installing An Operating System To An Iscsi Target
Configuring the Boot ROM on page 349. • Copy the correct Solarflare driver files to a floppy disk or USB flash drive. Refer to Steps to Install Insert the Windows Server 2008 R2 DVD and restart the server. The Windows Server setup program will start.
Page 364 • Ensure you have all your iSCSI configuration information for the iSCSI target and iSCSI initiator. You will need to enter these details during the installation process. • Ensure that the Solarflare Boot ROM is configured for iSCSI boot and can login to the selected iSCSI target.
Page 365 Solarflare Server Adapter User Guide If you see a Driver Updates added screen for a Solarflare driver disk, click OK. When prompted for further driver updates, click Back to return to the installer. Select the network device. To check which is the Solarflare network adapter, press Ctrl+Alt+F4.
Page 366 For SUSE Enterprise Linux Server 11, it should be onboot. Click Next to continue. 13 From the Installation Settings screen, select the Expert tab, then click the Booting hyperlink. Select the Boot Loader installation tab. Issue 11 © Solarflare Communications 2014...
Page 367 Following the server reboot, check that the iSCSI disk is in an appropriate place in the BIOS boot order. It may be displayed as 'Solarflare Boot Manager' or 'Hard drive C:', as there is no physical hard disk in the system.
Page 368 • Ensure you have all your iSCSI configuration information for the iSCSI target and iSCSI initiator. You will need to enter these details during the installation process. • Ensure that Solarflare Boot ROM is configured for iSCSI boot and can login to the selected iSCSI target.
Page 369 You will be prompted to load more driver disks. Select No. A CD Found screen will prompt you to test the CD before installation. Select Skip. When an Enable network interface screen displays, select the Solarflare adapter interface. Ensure that Use dynamic IP configuration (DHCP) is selected.
Page 370 11 In the Advanced Storage Options window, select iSCSI and click Add iSCSI target. 12 In the Configure iSCSI Parameters dialog box, enter your Target IP Address. 13 Click Add target to continue. Issue 11 © Solarflare Communications 2014...
Page 371 Following the server reboot, check that the iSCSI disk is in an appropriate place in the BIOS boot order. It may be displayed as 'Solarflare Boot Manager' or as 'Hard drive C:', as there is no physical hard disk in the system.
Page 372: Default Adapter Settings
2 seconds Boot skip delay 5 seconds Boot Type Initiator DHCP Enabled Initiator-IQN-DHCP Enabled LUN busy retry count 2 seconds Target-DHCP Enabled TCP port 3260 Boot LUN DHCP Vendor SFCgPXE MPIO attempts MSIX Limit Issue 11 © Solarflare Communications 2014...
Page 373: Index
Configure with SAM Large Receive Offload (LRO) Ethtool Configure on Linux Configure Interrupt moderation on Linux Configure on Solaris Configure on VMware Configure Interrupt moderation on VMware Configure on Windows Server 2008 Large Send Offload (LSO) Issue 11 © Solarflare Communications 2014...
Page 374 On Solaris On Linux On VMware On VMware On Windows On Windows Solarflare AppFlex™ Technology Licensing Configure with the Boot ROM agent Standby and power management Configure with SAM Static link aggregation Receive see also Teaming Receive Side Scaling (RSS)
Page 375 Setting up on Linux Setting up with SAM VMware Access to NIC from virtual machine Configure MTU ESX Service Console NetQueue VMware Tools Windows Installing from the Command Prompt Installing on Repairing and modifying installation Issue 11 © Solarflare Communications 2014...

This manual is also suitable for:

Ultra sfn7322f Sfn5122f Ultra sfn7122f Sfn6122f Sfn6322f Sfn5161t ... Show all

Solarflare Ultra SFN7142Q User Manual

Chapter 1: Introduction

Chapter 2: Installation

Chapter 3: Solarflare Adapters on Linux

Chapter 4: Solarflare Adapters on Windows

Chapter 5: Solarflare Adapters on Vmware

Chapter 6: Solarflare Adapters on Solaris

Chapter 7: SR-IOV Virtualization Using KVM

Chapter 8: SR-IOV Virtualization for Xenserver

Chapter 9: Solarflare Adapters on Mac 0S X

Chapter 10: Solarflare Boot ROM Agent

Index

Quick Links

Need help?

Questions and answers

Related Manuals for Solarflare Ultra SFN7142Q

Summary of Contents for Solarflare Ultra SFN7142Q

This manual is also suitable for:

Table of Contents