Sun storage 10 gbe fcoe pcie converged network adapter from qlogic (68 pages)
Summary of Contents for Oracle Dual Port 25 Gb
Page 1
Oracle Dual Port 25 Gb Ethernet Adapter ® User's Guide Part No: E87989-01 September 2017...
Page 3
Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.
Page 4
Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou services émanant de tiers, sauf mention contraire stipulée dans un contrat entre vous et Oracle. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des dommages causés par l'accès à...
Electrical Specifications .................. 20 Environmental Specifications ................ 20 Hardware and Software Requirements ............... 21 Updating Software and Firmware .............. 23 ▼ Update the OS (Oracle Solaris x86) ............. 23 Firmware Update Tool Overview .............. 24 Related Information ................ 24 Installing the Driver ................... 25 ▼ Verify the bnxt Driver (Oracle Solaris x86) .......... 25...
Page 6
Configuring Driver Parameters ................. 51 ▼ Set Driver Parameters (Oracle Solaris x86) ........... 51 Driver Parameters (Oracle Solaris x86) ............. 53 ▼ Set Driver Parameters (Linux) .............. 53 Driver Parameters (Linux) ................ 54 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Page 7
▼ Recover From a Port Hang (Oracle Solaris x86) .......... 76 ▼ Analyze Slow Network Performance (Oracle Solaris x86) ........ 77 ▼ Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86) .................... 78 ▼ Analyze Why Changing the MTU Does Not Correctly Set the Link Property...
Page 8
Contents Glossary ...................... 81 Index ........................ 85 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Required knowledge – Advanced experience troubleshooting and replacing hardware. ■ In this document, the term “adapter” refers to the Oracle Dual Port 25 Gb Ethernet Adapter, the term “x86” refers to 64-bit and 32-bit systems manufactured using processors compatible with the AMD64, Intel Xeon, or Intel Pentium product families , and the term “Oracle Solaris”...
Page 10
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
If your OS is out of date, update the entire OS image, “Update the OS (Oracle Solaris x86)” on page 23 or download and apply the latest OS patch. Verify the driver installation. “Verify the bnxt Driver (Oracle Solaris x86)” on page 25 Understanding the Installation Process...
“Download and Install the bnxt_en Driver (Linux)” on page 26 “Verify the bnxt_en Driver (Linux)” on page 27 Install and verify the adapter. “Install the Adapter” on page 34 “Verify the Adapter Installation (Linux)” on page 37 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Installation Task Overview (Windows) Step Description Links Boot over the network. “Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)” on page 42 Configure the driver parameters. “Set Driver Parameters (Linux)” on page 53 “Driver Parameters (Linux)” on page 54 (Optional) Configure jumbo frames.
Page 14
Installation Task Overview (Windows) Related Information “Installation Task Overview (Oracle Solaris x86)” on page 11 ■ “Installation Task Overview (Linux)” on page 12 ■ “Understanding the Adapter” on page 15 ■ Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
■ “Configuring Driver Parameters” on page 51 ■ “Configuring Jumbo Frames” on page 55 ■ “Configuring a Link Aggregation (Oracle Solaris x86)” on page 59 ■ “Configuring VLANs and VXLANs” on page 63 ■ “Removing the Driver” on page 69 ■...
Conforms to Ethernet standard 802.3 Boot ROM Mb SPI Flash FCC Class A Related Information “Shipping Kit Contents” on page 15 ■ “Front Panel Connectors and LEDs” on page 17 ■ Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Front Panel Connectors and LEDs Front Panel Connectors and LEDs On the front panel, two LEDs display the port speed and activity for each port. Both ports operate at the same speed. This figure and the table explain the meaning of the LEDs for ports 1 and 2.
Page 18
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
■ “Configuring Driver Parameters” on page 51 ■ “Configuring Jumbo Frames” on page 55 ■ “Configuring a Link Aggregation (Oracle Solaris x86)” on page 59 ■ “Configuring VLANs and VXLANs” on page 63 ■ “Removing the Driver” on page 69 ■...
-40°C to 65°C (-40°F to 149°F), noncondensing Humidity 10% to 90% noncondensing relative humidity at 93% noncondensing relative humidity at 27°C (80.6°F) maximum wet bulb 38°C (100.4°F) maximum wet bulb Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Hardware and software support changes over time. For the latest information concerning I/O options that are supported by your server, refer to the Product Page. For Oracle Solaris OS systems, the minimum supported version is Oracle Solaris 11.3 SRU23. Related Information “Physical Specifications” on page 19 ■...
Page 22
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
For the latest list of supported platforms and operating systems, see “Hardware and Software Requirements” on page For Solaris OS systems, the minimum supported version is Oracle Solaris 11.3 SRU23. Update the entire OS image on the client server. Ensure that you are using at least Oracle Solaris 11.3 SRU23.
Use the firmware update tool to update the firmware. You can get the tool in these ways for the Oracle Solaris x86, Linux, and Windows environment: Oracle System Assistant (OSA), a built-in tool on x86 servers or a USB thumb drive ■...
The bnxt software package comes bundled in the Oracle Solaris software. The bnxt driver does not support SRIOV. The Oracle Solaris 11.3 SRU23 OS is the first release to support this adapter. You can Note - install or upgrade to this release, but the version of the driver must be the same on both the client and the server.
■ Download and Install the bnxt_en Driver (Linux) If your server uses the Oracle Linux OS, you must download the bnxt_en device driver to install it. The bnxt_en driver is supported on Oracle Linux 6.8, 6.9, and 7.3. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
In a browser, log in to My Oracle Support. Go to https://myoraclesupport.com. Search for the Oracle Dual Port 25 Gb Ethernet Adapter. Open the readme.txt file and follow the download and installation instructions. Review and accept the software license agreement.
The ORACLE_SSM flash drive contains the bnxt_en driver. Right-click the bnxt_en network driver and choose Update Driver. Choose the Broadcom NIC driver for your OS. Reboot the server. Verify that the driver is installed. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Page 29
Download and Install the bnxt_en Driver (Windows) “Verify the Adapter Installation (Windows)” on page Related Information “Download and Install the bnxt_en Driver (Linux)” on page 26 ■ “Installing the Adapter” on page 31 ■ Installing the Driver...
Page 30
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
“Install the Adapter” on page 34 “Install the SFP28 Transceivers” on page 35 Verify the adapter installation. “Verify the Adapter Installation (Oracle Solaris x86)” on page 36 “Verify the Adapter Installation (Linux)” on page 37 “Verify the Adapter Installation (Windows)” on page 37 Related Information “Understanding the Adapter”...
Adapter Description Adapter Description The Oracle Dual Port 25 Gb Ethernet Adapter comes with the factory default mode of 25Gb, and supports direct attach cables. 25GbE Mode Part Number Oracle Dual Port 25 Gb Ethernet ■ 7118015, factory installed Adapter ■...
Cable Cautions A direct-attach SFP28 passive copper cable is available in 1, 2, 3, or 5 meters. The end ■ of the SFP28 direct-attached copper splitter cable connects to the adapter's SFP28 port. The two pigtails connect to the SFP28 ports 1 and 2 on the adapter. When using SFP28 to SFP28 cables to connect the adapter to a 25 GbE switch, Note - ensure that the SFP28 ports on the switch are configured in the 25GbE mode.
Power off all of the peripherals connected to the server. Open the server chassis. Attach an antistatic wrist strap to the server chassis. Remove the slot cover from the chassis. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
When using the adapter in the 2x25 GbE mode, there are two cabling options. See “Cable and Transceiver Overview” on page 32. The optical transceivers are available from Oracle. For the latest information on available transceivers and cables, refer to Oracle's Product Page. Installing the Adapter...
You must have at least Oracle Solaris 11.3 SRU23 installed. For example: $ pkg info entire Name: entire Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.23.1.0). <output omitted> Version: 0.5.23 (Oracle Solaris 11.3.23.1.0) Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Related Information “Adapter Description” on page 32 ■ “Install the Adapter” on page 34 ■ “Verify the Adapter Installation (Oracle Solaris x86)” on page 36 ■ “Verify the Adapter Installation (Windows)” on page 37 ■ Verify the Adapter Installation (Windows) Click Control Panel.
Page 38
Verify the Adapter Installation (Windows) If the driver is installed correctly, the adapter name Oracle Dual Port 25 Gb Ethernet Adapter is displayed in the Network Adapters folder. In the Administration tool, click Computer Management → Device Manager → Network Adapter.
“Configuring Driver Parameters” on page 51 ■ “Configuring Jumbo Frames” on page 55 ■ “Configuring a Link Aggregation (Oracle Solaris x86)” on page 59 ■ “Configuring VLANs and VXLANs” on page 63 ■ “Troubleshooting the Adapter (Oracle Solaris x86)” on page 75 ■...
Configure the Network Interface (Windows) Use the Network and Sharing Center to configure the network settings for the bnxt_en adapter. These steps are for Microsoft Windows7. Ensure that the adapter is installed. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Boot Options Click Start > Control Panel > Network and Sharing Center > Change Adapter Settings. Select Oracle Dual Port 25Gb Ethernet Adapter and click Change Settings of This Connection. Related Information “Configure the Network Interface (Oracle Solaris x86)” on page 40 ■...
Boot Over a 10GbE Network (Oracle Solaris x86 and Linux) Related Information “Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)” on page 42 ■ ■ “Installing RoCE” on page 44 ■ Boot Over a 10GbE Network (Oracle Solaris x86 and...
Page 43
Choose a boot option. Press F10 to save and exit. The host will perform a PXE boot from the Oracle Dual Port 25 Gb Ethernet adapter port, DHCP, and the UEFI PXE boot server response. This screen appears. Choose the UEFI PXE boot server using TFTP protocol.
Installing RoCE Ensure that the OS installation begins on the Oracle Dual Port 25 Gb Ethernet adapter. Ensure that the OS installer completes and begins the post-installation steps. After the OS installation completes, use the BIOS to change the boot device priority to boot from hard disk to boot up the newly installed OS.
VMware – ■ https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere. networking.doc/GUID-4A5EBD44-FB1E-4A83-BB47-BBC65181E1C2.html Install RoCE (Linux) Locate these files before you install and configure RoCE in Oracle Linux: Before You Begin The bnxt_en-roce file – RoCE-supported bnxt_en driver, which is part of released gzip ■ compressed tar archive.
20.6.2 supports RoCEv1 and RoCEv2 with the NDKPI interface. The default setting is RoCEv1. Upgrade the NIC NVRAM using the appropriate board packages. In CCM or in UEFI HII, enable support for RDMA. Enable each miniport. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Install RoCE in Windows User Mode You can go to the adapter's Advanced Properties page and set Network Direct Functionality to Enabled for each BCM5741X miniport. You can also run this command in the PowerShell window: Set-NetAdapterAdvancedProperty -RegistryKeyword *NetworkDirect -RegistryValue 1 Verify that RDMA is enabled.
Enable or disable SMB Multichannel support. On the Server side: Enable: Set-SmbServerConfiguration -EnableMultiChannel $true Disable: Set-SmbServerConfiguration -EnableMultiChannel $false On the Client side: Enable: Set-SmbClientConfiguration -EnableMultiChannel $true Disable: Set-SmbClientConfiguration -EnableMultiChannel $false Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Page 49
Verify RoCE in Windows By default, the driver sets up two RDMA connections for each network share per IP Note - address (on a unique subnet). You can increase the number of RDMA connections by adding multiple IP addresses, each with different a subnet, for the same physical port under test. Multiple network shares can be created and mapped to each link partner using the unique IP addresses that you created.
Page 50
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75 ■ Set Driver Parameters (Oracle Solaris x86) Locate the path names and the associated instance numbers in the /etc/ path_to_inst file. For example, on an Oracle Solaris x86 server, you should see output similar to this: Configuring Driver Parameters...
Page 52
For more information, refer to the Oracle Solaris man page. dladm(5) Related Information “Driver Parameters (Oracle Solaris x86)” on page 53 ■ “Set Driver Parameters (Linux)” on page 53 ■ Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
8. Allowed values: 64 to 4096 (default = 1024) Related Information “Set Driver Parameters (Oracle Solaris x86)” on page 51 ■ “Set Driver Parameters (Linux)” on page 53 ■ Set Driver Parameters (Linux) Use the ethtool utility or the configtool utility to set parameters on a Linux platform.
TCP) to the Ethernet adapter hardware. Related Information “Set Driver Parameters (Linux)” on page 53 ■ “Set Driver Parameters (Oracle Solaris x86)” on page 51 ■ “Configuring Jumbo Frames” on page 55 ■ Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Jumbo frames can support up to 9500 MTU. The default value is 1500 MTU. These topics describe how to configure jumbo frames: “Change the MTU Permanently (Oracle Solaris x86)” on page 55 ■ “Change the MTU Temporarily (Oracle Solaris x86)” on page 56 ■...
For example, where the IP address for eth7 is 192.1.1.200, this command increases MTUs to the maximum: # ifconfig eth7 192.1.1.200 mtu 9500 up For more information, refer to the Oracle Solaris man page. ifconfig(1M) Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Configure Jumbo Frames (Windows) Click Control Panel. Click Device Manager and open the Network Adapters folder. Right-click the Oracle Dual 25G Ethernet Adapter and choose Properties. Click Configure. Highlight the jumbo frames item and select the desired setting. Click Apply for each port.
Page 58
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Oracle Solaris x86 OS: “Configure a Link Aggregation (Oracle Solaris x86)” on page 59 ■ “Display Information About a Link Aggregation (Oracle Solaris x86)” on page 60 ■ “Delete a Link Aggregation (Oracle Solaris x86)” on page 61 ■...
0:15:17:75:ff:81 # ifconfig aggr2aggr2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 33 inet 193.2.2.84 netmask ffffff00 broadcast 193.2.2.255 ether 0:15:17:75:ff:83 ■ Use the dladm show-aggr command to show link aggregation status. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
For example: # dladm delete-aggr aggr2 # dladm delete-aggr aggr2 Related Information “Display Information About a Link Aggregation (Oracle Solaris x86)” on page 60 ■ “Configuring VLANs and VXLANs” on page 63 ■ Configuring a Link Aggregation (Oracle Solaris x86)
Page 62
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
“Configuring Driver Parameters” on page 51 ■ “Removing the Driver” on page 69 ■ “Troubleshooting the Adapter (Oracle Solaris x86)” on page 75 ■ VLANs Overview Virtual LANs enable you to divide the network into subnetworks without having to add to the physical network environment.
VLAN name. Related Information “Configure VLANs (Linux)” on page 65 ■ “Configure VLANs (Windows)” on page 65 ■ “Configure VXLANs (Oracle Solaris x86)” on page 66 ■ Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
For example: # ifconfig eth6.5 ipv6addressup where ipv6address is the IP address of the interface. Related Information “Configure VLANs (Oracle Solaris x86)” on page 64 ■ “Configure VLANs (Windows)” on page 65 ■ “Configure VXLANs (Oracle Solaris x86)” on page 66 ■...
Configure VXLANs (Oracle Solaris x86) Right-click the Oracle Dual Port 25 Gb Ethernet Adapter port, then select Properties. Click Configure. Click VLAN, then click New. Type VLAN and the ID (for example, type VLAN10). Click OK. Open the Local Connections for VLAN window from the Network Connections window (Control Panel →...
Page 67
MGROUP vxlan1 10.10.10.1 224.0.0.1 vxlan2 10.10.10.1 224.0.0.1 Related Information “Configure VLANs (Oracle Solaris x86)” on page 64 ■ “Configure VLANs (Linux)” on page 65 ■ “Configure VLANs (Windows)” on page 65 ■ “Removing the Driver” on page 69 ■ Configuring VLANs and VXLANs...
Page 68
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
These topics explain how to remove the bnxt and bnxt_en device drivers: Description Links Remove the driver on an Oracle “Remove the bnxt Driver (Oracle Solaris x86)” on page 69 Solaris x86 server. Remove the driver on a Linux server. “Remove the bnxt_en Driver (Linux)” on page 70 Remove the driver on a Windows “Remove the bnxt_en Driver (Windows)”...
“Remove the bnxt Driver (Oracle Solaris x86)” on page 69 ■ “Remove the bnxt_en Driver (Linux)” on page 70 ■ “Upgrading the Adapter (Linux)” on page 71 ■ Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Upgrading the Adapter (Linux) These topics explain how to upgrade the firmware on the Oracle Dual Port 25 Gb Ethernet Adapter and verify the upgrade. The upgrade command compares the package version against the currently installed package version to determine if an upgrade should be performed. The command does not check the versions of the individual components in the NVM against those within the specified package file.
$ bnxtnvm pkgver Device: ens2f1d1 Package version: 20.06.04.01 To display more details about the device and the currently installed package, use the -v option to enable maximum verbosity. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Page 73
Verify the Upgrade Related Information “Installing the Adapter” on page 31 ■ “Troubleshooting the Adapter (Oracle Solaris x86)” on page 75 ■ Upgrading the Adapter (Linux)
Page 74
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
These topics describe how to troubleshoot the installation and operation of the Oracle Dual Port 25 Gb Ethernet Adapter on an Oracle x86 server running the Oracle Solaris 11.3 OS. These topics cover basic installation issues and are not intended to be comprehensive.
■ “Analyze Slow Network Performance (Oracle Solaris x86)” on page 77 ■ “Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris ■ x86)” on page 78 “Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle ■...
Packet size to determine bcopy or not during transmit0 ~ 9216, 128 by default. tx_copy_threshold Related Information “Analyze Why the Device Link Is Missing (Oracle Solaris x86)” on page 75 ■ “Analyze Slow Network Performance (Oracle Solaris x86)” on page 77 ■...
Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86) # truss -p PID Look for NIS, DNS, and network routing outages. If you find any issues, fix them before proceeding. View the I/O statistics to ensure that there are no bottlenecks on the disk.
Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle Solaris x86) Check that the switch ports have been configured to operate in the mode in which the adapter is running. Related Information “Analyze Why the Device Link Is Missing (Oracle Solaris x86)” on page 75 ■...
Page 80
Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Page 81
Glossary adapter The Oracle Dual Port 25 Gb Ethernet Adapter. Direct attached cables. DHCP Dynamic Host Configuration Protocol. Part of the application layer in the Internet protocol suite. Domain name system. Translates human-readable domain names into numerical identifiers. EEPROM Electronically erasable programmable read-only memory.
Page 82
Maximum transmission unit. The MTU (payload without the Ethernet header) affects how jumbo frames function. Network Information Service. Originally known as Yellow Pages, NIS is a protocol for distributed system configuration data. PCIe Peripheral Component Interconnect Express. Physical function. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...
Page 83
Preboot execution environment. Enables clients to boot over a network interface, independent of the OS or other devices. RDMA Remote Direct Memory Access. Allows computers in a network to exchange data in main memory without involving the processor, cache, or OS of either computer. Response.
Page 84
IP subnets. VXLAN Virtual eXtensive LAN. A tunneling mechanism for providing isolated virtual Layer 2 (L2) segments that can span multiple physical L2 segments. Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017...