1. Manuals
  2. Brands
  3. Chelsio Communications Manuals
  4. Adapter
  5. Terminator Series
  6. Installation and user manual

Chelsio Communications Terminator Series Installation And User Manual

Unified wire for linux
Hide thumbs Also See for Terminator Series:
1
2
3
4
Table Of Contents
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
Table of Contents

Advertisement

Quick Links

Download this manual
Chelsio Unified Wire for Linux
i

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Terminator Series and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Chelsio Communications Terminator Series

Summary of Contents for Chelsio Communications Terminator Series

  • Page 1 Chelsio Unified Wire for Linux...
  • Page 2 IMPLIED WARRANTIES MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE “SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO COMMUNICATIONS, INC. Chelsio (India) Private Limited Chelsio Communications (Headquarters) Subramanya Arcade, Floor 3, Tower B 209 North Fair Oaks Avenue, No.
  • Page 3 Document History Version Revision Date 1.0.0 12/08/2011 1.0.1 01/09/2013 1.0.2 01/27/2013 1.0.3 03/26/2013 1.0.4 04/12/2013 1.0.5 06/20/2013 1.0.6 08/17/2013 1.0.7 10/22/2013 1.0.8 03/08/2013 1.0.9 05/15/2013 1.1.0 07/26/2013 1.1.1 08/14/2013 1.1.2 12/06/2013 1.1.3 12/19/2013 1.1.4 03/13/2014 1.1.5 05/02/2014 1.1.6 06/30/2014 1.1.7 10/22/2014 1.1.8 11/04/2014...
  • Page 4 1.3.6 09/29/2017 1.3.7 12/29/2017 1.3.8 02/28/2018 1.3.9 03/30/2018 1.4.0 04/18/2018 1.4.1 07/05/2018 1.4.2 07/18/2018 1.4.3 10/01/2018 Chelsio Unified Wire for Linux...

  • Page 5: Table Of Contents

    TABLE OF CONTENTS CHELSIO UNIFIED WIRE Introduction 1.1. Features 1.2. Hardware Requirements 1.3. Software Requirements 1.4. Package Contents Hardware Installation Software/Driver Installation 3.1. Pre-requisites 3.2. Enabling RDMA on ARM Platforms 3.3. Mounting debugfs 3.4. Allowing unsupported modules on SLES 3.5. Installing Chelsio Unified Wire from source 3.6.
  • Page 6 4.2. Enabling Busy waiting 4.3. Precision Time Protocol (PTP) 4.4. VXLAN Offload 4.5. Performance Tuning Software/Driver Unloading 5.1. Unloading the NIC Driver 5.2. Unloading the TOE Driver III. VIRTUAL FUNCTION NETWORK (VNIC) Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Installation 2.1.
  • Page 7 Introduction 1.1. Hardware Requirements 1.2. Software Requirements Kernel Installation Software/Driver Installation 3.1. Pre-requisites 3.2. Installation Software/Driver Loading Software/Driver Configuration and Fine-tuning 5.1. 5.2. Performance Tuning Software/Driver Unloading WD-UDP Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Installation Software/Driver Loading Software/Driver Configuration and Fine-tuning 4.1.
  • Page 8 3.2. Installation Software/Driver Loading Software/Driver Configuration and Fine-tuning 5.1. Target 5.2. Initiator 5.3. 5.4. Performance Tuning Software/Driver Unloading LIO ISCSI TARGET OFFLOAD Introduction 1.1. Hardware Requirements 1.2. Software Requirements Kernel Configuration Software/Driver Installation 3.1. Pre-requisites 3.2. Installation Software/Driver Loading Software/Driver Configuration and Fine-tuning 5.1.
  • Page 9 4.7. Target Storage Device Configuration 4.8. Target Redirection Support 4.9. The command line interface tools “iscsictl” & “chisns” 4.10. Rules of Target Reload (i.e. “on the fly” changes) 4.11. System Wide Parameters 4.12. Performance Tuning Software/Driver Unloading ISCSI PDU OFFLOAD INITIATOR Introduction 1.1.
  • Page 10 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Installation Software/Driver Loading Software/Driver Configuration and Fine-tuning 4.1. Configuring Cisco Nexus 5010 switch 4.2. Configuring the Brocade 8000 switch Running NIC & iSCSI Traffic together with DCBx XIV. FCOE FULL OFFLOAD INITIATOR Introduction 1.1.
  • Page 11 XVII. UDP SEGMENTATION OFFLOAD AND PACING Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Installation Software/Driver Loading Software/Driver Configuration and Fine-tuning 4.1. Modifying the Application 4.2. Configuring UDP Pacing Software/Driver Unloading XVIII.OFFLOAD IPV6 Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Installation 2.1.
  • Page 12 2.4. Removing Filter Rules 2.5. Layer 3 Example 2.6. Layer 2 Example 2.7. Filtering VF traffic Hash/DDR Filters 3.1. Configuration 3.2. Creating Filter Rules 3.3. Listing Filter Rules 3.4. Removing Filter Rules 3.5. Filter Priority 3.6. Swap MAC Feature 3.7. Traffic Mirroring 3.8.
  • Page 13 Software/Driver Configuration and Fine-tuning 3.1. Traffic Management Rules 3.2. Configuring Traffic Management Usage 4.1. Non-Offloaded Connections 4.2. Offloaded Connections 4.3. Offloaded Connections with Modified Application Software/Driver Unloading XXIV. DPDK DRIVER Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Installation 2.1. Pre-requisites 2.2.
  • Page 14 Configuring PXE Server PXE Boot Process 5.1. Legacy PXE Boot 5.2. uEFI PXE Boot FCoE Boot Process 6.1. Legacy FCoE Boot 6.2. uEFI FCoE Boot iSCSI Boot Process 7.1. Legacy iSCSI Boot 7.2. uEFI iSCSI Boot Creating Driver Update Disk (DUD) 8.1.

  • Page 15: Chelsio Unified Wire

    Chapter I. Chelsio Unified Wire I. Chelsio Unified Wire Chelsio Unified Wire for Linux...

  • Page 16: Introduction

    Network and Storage related requirements. The Terminator series is Chelsio’s next generation of highly integrated, hyper-virtualized 1/10/25/40/50/100GbE controllers. The adapters are built around a programmable protocol- processing engine, with full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, iSCSI, FCoE and NAT support.

  • Page 17: Hardware Requirements

    For detailed instructions on loading, unloading and configuring the drivers/tools please refer to their respective sections. 1.2. Hardware Requirements The Chelsio Unified Wire software supports Chelsio Terminator series of Unified Wire adapters. To know more about the list of adapters supported by each driver, please refer to their respective sections. 1.3.
  • Page 18 Chapter I. Chelsio Unified Wire docs: This directory contains support documents - README, Release Notes and User’s Guide (this document) for the software. kernels: This directory contains kernel.org-4.9.105 installation files. libs: This directory is for libraries required to install the WD-UDP, WD-TOE and iWARP drivers.
  • Page 19 Chapter I. Chelsio Unified Wire • chdebug: This script collects operating system environment details and debug information which can be sent to the support team, to troubleshoot Chelsio hardware/software related issues. • chiscsi_set_affinity.sh: This shell script is used for mapping iSCSI Worker threads to different CPUs.
  • Page 20 Chapter I. Chelsio Unified Wire uninstall.py: Python script that uninstalls the RPM package. See Chelsio Unified Wire’s Software/Driver Uninstallation section for more information. Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD. The OptionROM directory contains Unified Boot Option ROM image (cubt4.bin), uEFI driver (ChelsioUD.efi), default boot configuration file (bootcfg) and a legacy flash utility (cfut4.exe), which can be used to flash the option ROM onto Chelsio adapters (CNAs).

  • Page 21: Hardware Installation

    Chapter I. Chelsio Unified Wire Hardware Installation Follow these steps to install Chelsio adapter in your system: Shutdown/power off your system. ii. Power off all remaining peripherals attached to your system. iii. Unpack the Chelsio adapter and place it on an anti-static surface. iv.
  • Page 22 Chapter I. Chelsio Unified Wire Physical function 6: for FCoE Physical function 7: Currently not assigned Once Unified Wire package is installed and loaded, examine the output of dmesg to see if the card is discovered. You should see a similar output: The above outputs indicate the hardware configuration of the adapter as well as serial number.

  • Page 23: Software/Driver Installation

    Chapter I. Chelsio Unified Wire Software/Driver Installation There are two main methods to install the Chelsio Unified Wire package: from source and RPM. If you decide to use source, you can install the package using CLI or GUI mode. If you decide to use RPM, you can install the package using Menu or CLI mode. RPM packages support only distro base kernels.

  • Page 24: Enabling Rdma On Arm Platforms

    Chapter I. Chelsio Unified Wire 3.2. Enabling RDMA on ARM Platforms RDMA is disabled by default in RHEL 7.3 build of ARM architecture. To enable this feature, follow the steps mentioned below: Download the kernel source package and extract it. ii.

  • Page 25: Allowing Unsupported Modules On Sles

    Chapter I. Chelsio Unified Wire 3.4. Allowing unsupported modules on SLES On SLES11 SPx platforms, edit the /etc/modprobe.d/unsupported-modules file and change allow_unsupported_modules to 1. On SLES12 SPx platforms, edit the /etc/modprobe.d/10-unsupported-modules.conf file and change allow_unsupported_modules to 1. 3.5. Installing Chelsio Unified Wire from source GUI mode (with Dialog utility) Download the Unified Wire driver package (tarball) from Chelsio Download...
  • Page 26 Chapter I. Chelsio Unified Wire vii. Select Enable IPv6-Offload to install drivers with IPv6 Offload support or Disable IPv6-offload to continue installation without IPv6 offload support. viii. Select the required configuration tuning option: The tuning options may vary depending on the Linux distribution. Note ix.
  • Page 27 Chapter I. Chelsio Unified Wire x. Select the required performance tuning option. a. Enable Binding IRQs to CPUs: Bind MSI-X interrupts to different CPUs and disable IRQ balance daemon. b. Retain IRQ balance daemon: Do not disable IRQ balance daemon. c.
  • Page 28 Chapter I. Chelsio Unified Wire xiii. After successful installation, summary of installed components will be displayed. xiv. Select “View log” to view the installation log or “Exit” to continue. xv. Select “Yes” to exit the installer or “No” to go back. xvi.
  • Page 29 Chapter I. Chelsio Unified Wire CLI mode (without Dialog utility) If your system does not have Dialog or you choose not to install it, follow the steps mentioned below to install the Unified Wire package: Download the Unified Wire driver package from Chelsio Download Center.
  • Page 30 Chapter I. Chelsio Unified Wire Chelsio’s Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a single command. Follow the procedure mentioned below: Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii.
  • Page 31 Chapter I. Chelsio Unified Wire iv. Install the drivers, tools and libraries using the following command: [root@host~]# make install v. The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF=<configuration_tuning> [root@host~]# make CONF=<configuration_tuning>...

  • Page 32: Installing Chelsio Unified Wire From Rpm

    Chapter I. Chelsio Unified Wire ▪ The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF=<configuration_tuning> <Build Target> [root@host~]# make CONF=<configuration_tuning> <Install Target> ▪ To build and install drivers along with benchmarks: [root@host~]# make BENCHMARKS=1 [root@host~]# make BENCHMARKS=1 install ▪...
  • Page 33 Chapter I. Chelsio Unified Wire Menu Mode Download the tarball specific to your operating system and architecture from Chelsio Download Center. ii. Untar the tarball: E.g., for RHEL 6.9, untar using the following command: [root@host~]# tar zxvf <driver_package>-RHEL6.9_x86_64.tar.gz iii. Change your current working directory to Chelsio Unified Wire package directory [root@host~]# cd ChelsioUwire-x.x.x.x-<OS>-<arch>...
  • Page 34 Chapter I. Chelsio Unified Wire CLI mode Download the tarball specific to your operating system and architecture from Chelsio Download Center. ii. Untar the tarball: E.g., for RHEL 6.9, untar using the following command: [root@host~]# tar zxvf ChelsioUwire-x.x.x.x-RHEL6.9_x86_64.tar.gz iii. Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x-<OS>-<arch>...
  • Page 35 Chapter I. Chelsio Unified Wire vi. To install OFED and Chelsio drivers built against OFED, run the above command with -o option. [root@host~]# ./install.py –i <Installation mode> -c <Configuration> -o vii. Reboot your machine for changes to take effect. iWARP driver installation on cluster nodes Please make sure that you have enabled password less authentication with ssh Important on the peer nodes for this feature to work.

  • Page 36: Firmware Update

    Chapter I. Chelsio Unified Wire 3.7. Firmware Update /lib/firmware/cxgb4 The firmware is installed on the system, typically in , and the driver will auto-load the firmware if an update is required. The kernel must be configured to enable userspace firmware loading support: Device Drivers ->...

  • Page 37: Configuring Chelsio Network Interfaces

    Chapter I. Chelsio Unified Wire Configuring Chelsio Network Interfaces To test Chelsio adapters’ features it is required to use two machines both with Chelsio’s network adapters installed. These two machines can be connected directly without a switch (back-to- back), or both connected to a switch. The interfaces have to be declared and configured. The configuration files for network interfaces on Red Hat Enterprise Linux (RHEL) distributions are /etc/sysconfig/network-scripts kept under...
  • Page 38 Chapter I. Chelsio Unified Wire If auto-negotiation is disabled, ensure that the same FEC is set on both sides of the link, for the link to come up. Setting Speed T6 100G ports support multiple speeds viz. 100G, 50G, 40G, 25G, 10G and 1G. T6 25G ports support 25G, 10G and 1G speeds.
  • Page 39 Chapter I. Chelsio Unified Wire Example: To set 25G speed on 100G port: [root@host~]# ethtool -s <ethX> speed 25000 autoneg off • Twinax Twinax cables support auto-negotiation. The following speeds can be set in the advertise field. o Advertise only 100G [root@host~]# ethtool --change <ethX>...
  • Page 40 Chapter I. Chelsio Unified Wire Spider and QSA Modes • T5 Adapters Chelsio T5 40G adapters can be configured in the following 3 modes: 2X40Gbps: This is the default mode of operation where each port functions as 40Gbps link. The port nearest to the motherboard will appear as the first network interface (Port 0). ii.

  • Page 41: Configuring Network-Scripts

    Chapter I. Chelsio Unified Wire iv. Reload the network driver for changes to take effect. [root@host~]# rmmod cxgb4 [root@host~]# modprobe cxgb4 If default option is selected in step ii, reboot the machine for changes to take effect. Note 4.2. Configuring network-scripts A typical interface network-script (e.g., eth0) on RHEL 6.X looks like the following: # file: /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE="eth0"...

  • Page 42: Checking Link

    Chapter I. Chelsio Unified Wire modprobe cxgb4 Then load the driver using the command (for the moment it does not make any difference whether we are using NIC-only or the TOE-enabling driver). The output of ifconfig should display the adapter interfaces as: [root@host~]# ifconfig -a | grep HWaddr eth0 Link encap:Ethernet HWaddr 00:30:48:32:6A:AA eth1 Link encap:Ethernet HWaddr 00:07:43:04:6B:E9...

  • Page 43: Performance Tuning

    Chapter I. Chelsio Unified Wire Performance Tuning The following section lists the steps to tune the system for optimal performance. 5.1. Generic • Install the adapter into a PCIe Gen3 x8/x16 slot. Ensure that T6 100G adapters are placed in x16 slots and not in x8_in_x16 slots.
  • Page 44 Chapter I. Chelsio Unified Wire • Disable few services. [root@host~]# t4_latencytune.sh <interface> • Set sysctl param net.ipv4.tcp_low_latency to 1 [root@host~]# sysctl -w net.ipv4.tcp_low_latency=1 To optimize your system for different protocols, please refer to their respective chapters. Chelsio Unified Wire for Linux...

  • Page 45: Software/Driver Update

    Chapter I. Chelsio Unified Wire Software/Driver Update For any distribution-specific problems, please check README and Release Notes included in the release for possible workaround. Please visit Chelsio Download Center for regular updates on various software/drivers. You can also subscribe to our newsletter for the latest software updates. Chelsio Unified Wire for Linux...

  • Page 46: Software/Driver Uninstallation

    Chapter I. Chelsio Unified Wire Software/Driver Uninstallation Similar to installation, the Chelsio Unified Wire package can be uninstalled using two main methods: from the source and RPM, based on the method used for installation. If you decide to use source, you can uninstall the package using CLI or GUI mode. 7.1.
  • Page 47 Chapter I. Chelsio Unified Wire iv. The selected components will now be uninstalled. v. After successful uninstalltion, summary of the uninstalled components will be displayed. vi. Select “View log” to view uninstallation log or “Exit” to continue. Chelsio Unified Wire for Linux...
  • Page 48 Chapter I. Chelsio Unified Wire vii. Select “Yes” to exit the installer or “No” to go back. Press Esc or Ctrl+C to exit the installer at any point of time. Note CLI mode (without Dialog utility) Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x –u ii.
  • Page 49 Chapter I. Chelsio Unified Wire CLI mode Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Uninstall using the following command: [root@host~]# make uninstall CLI mode (individual drivers/software) You can also choose to uninstall drivers/software individually. Provided here are steps to uninstall few of them.

  • Page 50: Uninstalling Chelsio Unified Wire From Rpm

    Chapter I. Chelsio Unified Wire 7.2. Uninstalling Chelsio Unified Wire from RPM Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x-<OS>-<arch> ii. Uninstall Unified Wire: [root@host~]# ./uninstall.py <inbox/ofed> inbox : for removing all Chelsio drivers. ofed : for removing OFED and Chelsio drivers.

  • Page 51: Network (Nic/Toe)

    Chapter II. Network (NIC/TOE) II. Network (NIC/TOE) Chelsio Unified Wire for Linux...

  • Page 52: Introduction

    Chapter II. Network (NIC/TOE) Introduction Chelsio’s Unified Wire adapters provide extensive support for NIC operation, including all stateless offload mechanisms for both IPv4 and IPv6 (IP, TCP and UDP checksum offload, LSO - Large Send Offload aka TSO - TCP Segmentation Offload, and assist mechanisms for accelerating LRO - Large Receive Offload).

  • Page 53: Software Requirements

    Chapter II. Network (NIC/TOE) T520-OCP-SO* T520-BT T420-CR T440-CR T422-CR T420-SO-CR* T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX Only NIC driver supported ^ Memory-free; 256 IPv4/128 IPv6 offload connections supported. 1.2. Software Requirements Linux Requirements Currently the Network driver is available for the following versions: RHEL 7.5, 3.10.0-862.el7 RHEL 7.5, 3.10.0-862.el7.ppc64le (POWER8 LE) RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64)

  • Page 54: Software/Driver Installation

    Chapter II. Network (NIC/TOE) Software/Driver Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x • To build and install NIC only driver (without offload support): [root@host~]# make nic_install • To build and install drivers with offload support: [root@host~]# make toe_install make help install.py -h...

  • Page 55: Software/Driver Loading

    Chapter II. Network (NIC/TOE) Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 56: Software/Driver Configuration

    Chapter II. Network (NIC/TOE) Software/Driver Configuration 4.1. Enabling TCP Offload Load the offload drivers and bring up the Chelsio interface. [root@host~]# modprobe t4_tom [root@host~]# ifconfig ethX <IP> up All TCP traffic will be offloaded over the Chelsio interface now. To see the number of connections offloaded, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4/<bus-id>/tids Where,...

  • Page 57: Precision Time Protocol (Ptp)

    Chapter II. Network (NIC/TOE) the ability for the socket layer code to poll directly on an Ethernet device's Rx queue. This eliminates the cost of interrupts and context switching, and with proper tuning allows to achieve latency performance similar to that of hardware. Chelsio's NIC and TOE drivers support this feature and can be enabled on Chelsio supported devices to attain improved latency.
  • Page 58 Chapter II. Network (NIC/TOE) Load the network driver on all master and slave nodes; [root@host~]# modprobe cxgb4 ii. Assign IP addresses and ensure that master and slave nodes are connected. iii. Start the ptp4l tool on master using the Chelsio interface: [root@host~]# ptp4l -i <interface>...

  • Page 59: Vxlan Offload

    Chapter II. Network (NIC/TOE) v. Synchronize the system clock to a PTP hardware clock (PHC) on slave nodes. [root@host~]# phc2sys -s <interface> -c CLOCK_REALTIME -w -m 4.4. VXLAN Offload Virtual Extensible LAN (VXLAN) is a network virtualization technique that uses overlay encapsulation protocol to provide Ethernet Layer 2 network services with extended scalability and flexibility.
  • Page 60 Chapter II. Network (NIC/TOE) iii. Create the VXLAN interface, assign the VNI, multicast group and the port number and bing it up. [root@host~]# modprobe vxlan [root@host~]# ip link add <vxlan_interface> type vxlan id <vni> group 239.1.1.1 dev <interface> dstport 4789 [root@host~]# ifconfig <vxlan_interface>...
  • Page 61 Chapter II. Network (NIC/TOE) iii. Bring up the Virtual Network interface with the required IP address. [root@host~]# ifconfig <virtual-interface> <IP address> up For better performance, the following settings are recommended: Increase the number of queues for the Virtual network interface to 8. [root@host~]# virsh edit <VM>...

  • Page 62: Performance Tuning

    Chapter II. Network (NIC/TOE) 4.5. Performance Tuning Apply the performance settings mentioned in the Performance Tuning section in the Unified Wire chapter before proceeding. Run the performance tuning script to map TOE queues to different CPUs: [root@host~]# t4_perftune.sh -n -Q ofld ii.
  • Page 63 Chapter II. Network (NIC/TOE) Run the performance tuning script to map NIC queues to different CPUs: [root@host~]# t4_perftune.sh -n -Q nic ii. Enable adaptive-rx [root@host~]# ethtool -C enp2s0f4 adaptive-rx on iii. Set the following sysctl parameters [root@host~]# sysctl -w net.ipv4.tcp_timestamps=0 [root@host~]# sysctl -w net.core.netdev_max_backlog=250000 [root@host~]# sysctl -w net.core.rmem_max=4194304 [root@host~]# sysctl -w net.core.wmem_max=4194304...
  • Page 64 Chapter II. Network (NIC/TOE) [root@host~]# t4_perftune.sh Discovering Chelsio T4/T5 devices ... Configuring Chelsio T4/T5 devices ... Tuning eth7 IRQ table length 4 Writing 1 in /proc/irq/62/smp_affinity Writing 2 in /proc/irq/63/smp_affinity Writing 4 in /proc/irq/64/smp_affinity Writing 8 in /proc/irq/65/smp_affinity eth7 now up and tuned Because there are 4 CPUs on the system, 4 entries of interrupts are assigned.
  • Page 65 Chapter II. Network (NIC/TOE) /proc/interrupts gives the following output in [root@receiver_host~]# cat /proc/interrupts | grep eth6 CPU0 CPU1 CPU2 CPU3 type interface 17418 PCI-MSI-edge eth6 (queue 0) 21718 2063 PCI-MSI-edge eth6 (queue 1) 391519 PCI-MSI-edge eth6 (queue 2) 17798 PCI-MSI-edge eth6 (queue 3) Here there are 4 receiving queues from the eth6 interface, but they are not bound to a specific CPU or interrupt entry.
  • Page 66 Chapter II. Network (NIC/TOE) doing this, rather than processing every small packet, the receiver CPU works on fewer packet headers but with same amount of data. This helps reduce the receive host CPU load and improve throughput in a 40/10Gb network environment where CPU can be the bottleneck. LRO and GRO are different names to refer to the same receiver packets aggregating feature.
  • Page 67 Chapter II. Network (NIC/TOE) generic-receive-offload Now the option is on. This means GRO is enabled. Please note that generic-receive-offload large-receive-offload there are two offload options here: This is because on this Linux system (RHEL6.0), the kernel supports both GRO and LRO. As mentioned earlier, GRO is always the preferred option when both of them are present.

  • Page 68: Software/Driver Unloading

    Chapter II. Network (NIC/TOE) Software/Driver Unloading 5.1. Unloading the NIC Driver To unload the NIC driver, run the following command: [root@host~]# rmmod cxgb4 5.2. Unloading the TOE Driver A reboot is required to unload the TOE driver. To avoid rebooting, follow the steps mentioned below: Load t4_tom driver with unsupported_allow_unload parameter.
  • Page 69 Chapter II. Network (NIC/TOE) iii. Unload the driver: [root@host~]# rmmod t4_tom [root@host~]# rmmod toecore [root@host~]# rmmod cxgb4 Chelsio Unified Wire for Linux...

  • Page 70: Virtual Function Network (Vnic)

    Chapter III. Virtual Function Network (vNIC) III. Virtual Function Network (vNIC) Chelsio Unified Wire for Linux...

  • Page 71: Introduction

    Chapter III. Virtual Function Network (vNIC) Introduction The ever-increasing network infrastructure of IT enterprises has lead to a phenomenal increase in maintenance and operational costs. IT managers are forced to acquire more physical servers and other data center resources to satisfy storage and network demands. To solve the Network and I/O overhead, users are opting for server virtualization which consolidates I/O workloads onto lesser physical servers thus resulting in efficient, dynamic and economical data center environments.

  • Page 72: Software Requirements

    Chapter III. Virtual Function Network (vNIC) T420-BT T420-LL-CR T420-CX 1.2. Software Requirements Linux Requirements Currently the vNIC driver is available for the following versions: RHEL 7.5, 3.10.0-862.el7 RHEL 7.4, 3.10.0-693.el7 RHEL 6.9, 2.6.32-696.el6 SLES 15, 4.12.14-23-default SLES 12 SP3, 4.4.73-5-default SLES 12 SP2, 4.4.21-69-default Ubuntu 18.04.1, 4.15.0-29-generic Ubuntu 16.04.4, 4.4.0-116-generic...

  • Page 73: Software/Driver Installation

    Chapter III. Virtual Function Network (vNIC) Software/Driver Installation The Virtual Function implementation for Chelsio adapters comprises of two modules: • Standard NIC driver module, cxgb4, which runs on base Hypervisor and is responsible for instantiation and management of the PCIe Virtual Functions (VFs) on the adapter. •...

  • Page 74: Software/Driver Loading

    Chapter III. Virtual Function Network (vNIC) Software/Driver Loading 3.1. Instantiate Virtual Functions (SR-IOV) To instantiate Virtual Functions (VFs) on the host, run the following commands: [root@host~]# modprobe cxgb4 [root@host~]# echo n > /sys/class/net/ethX/device/driver/<bus_id>/sriov_numvfs Here, ethX is the interface and n specifies the number of VFs to be instantiated per physical function (bus_id).
  • Page 75 Chapter III. Virtual Function Network (vNIC) The vNIC driver must be loaded on the Guest OS by the root user. Any attempt to load the driver as a regular user will fail. To load the driver, run the following command: [root@host~]# modprobe cxgb4vf Chelsio Unified Wire for Linux...

  • Page 76: Software/Driver Configuration And Fine-Tuning

    Chapter III. Virtual Function Network (vNIC) Software/Driver Configuration and Fine-tuning 4.1. VF Rate Limiting This section describes the method to rate-limit traffic passing through virtual functions (VFs). The VF rate limit needs to be set on the Host (hypervisor). Apply rate-limiting using: [root@host~]# ip link set dev mgmtpfXX vf <vf_number>...

  • Page 77: Bonding

    Chapter III. Virtual Function Network (vNIC) 4.2. Bonding The VF network interfaces (assigned to a VM) can be aggregated into a single logical bonded interface effectively combining the bandwidth into a single connection. It also provides redundancy in case one of the link fails. Execute the following steps in the VM (attached with more than 1 VF interface): Load the Virtual Function network driver using force_link_up module parameter.

  • Page 78: High Capacity Vf Configuration

    Chapter III. Virtual Function Network (vNIC) ii. 1 VM was configured with VF0 of PF0 and VF1 of PF1. [root@host~]# modprobe cxgb4vf force_link_up=0 [root@host~]# ifconfig enp8s1 enp8s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 06:44:3c:a8:40:00 txqueuelen 1000 (Ethernet) [root@host~]# ifconfig enp8s1f5d1 enp8s1f5d1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 06:44:3c:a8:40:11 txqueuelen 1000...
  • Page 79 Chapter III. Virtual Function Network (vNIC) [root@host~]# chelsio_adapter_config.py Chelsio adapter detected |------------------------------------| | Choose Chelsio card: | 1. T580_SO_CR 3:0.0 | 2. T6225_SO 4:0.0 |------------------------------------| Select card: 2 Card T6225_SO(4:0.0) selected |------------------------------------| | Choose option | 1. Change to Default settings | 2.
  • Page 80 Chapter III. Virtual Function Network (vNIC) [root@host ~]# ip link show 14: mgmtpf1,0: <NOARP> mtu 0 qdisc noop state DOWN mode DEFAULT qlen 1 link/none vf 0 MAC 06:44:3c:b1:00:00, link-state auto 15: mgmtpf1,1: <NOARP> mtu 0 qdisc noop state DOWN mode DEFAULT qlen 1 link/none vf 0 MAC 06:44:3c:b1:80:10, link-state auto 16: mgmtpf1,2: <NOARP>...

  • Page 81: Software/Driver Unloading

    Chapter III. Virtual Function Network (vNIC) Software/Driver Unloading 5.1. Unloading the Driver The vNIC driver must be unloaded on the Guest OS by the root user. Any attempt to unload the driver as a regular user will fail. To unload the driver, execute the following command: [root@host~]# rmmod cxgb4vf Chelsio Unified Wire for Linux...

  • Page 82: Iwarp (Rdma)

    Chapter IV. iWARP (RDMA) IV. iWARP (RDMA) Chelsio Unified Wire for Linux...

  • Page 83: Introduction

    Chapter IV. iWARP (RDMA) Introduction Chelsio’s Terminator engine implements a feature rich RDMA implementation which adheres to the IETF standards with optional markers and MPA CRC-32C. The iWARP RDMA operation benefits from the virtualization, traffic management and QoS mechanisms provided by Terminator engine. It is possible to ACL process iWARP RDMA packets. It is also possible to rate control the iWARP traffic on a per-connection or per-class basis, and to give higher priority to QPs that implement distributed locking mechanisms.

  • Page 84: Software Requirements

    Chapter IV. iWARP (RDMA) 1.2. Software Requirements Linux Requirements Currently the iWARP driver is available for the following versions: RHEL 7.5, 3.10.0-862.el7 RHEL 7.5, 3.10.0-862.el7.ppc64le (POWER8 LE) RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64) RHEL 7.4, 3.10.0-693.el7 RHEL 7.4, 3.10.0-693.el7.ppc64le (POWER8 LE) RHEL 7.3, 4.5.0-15.el7.aarch64 (ARM64) RHEL 6.9, 2.6.32-696.el6 SLES 15, 4.12.14-23-default SLES 12 SP3, 4.4.73-5-default...

  • Page 85: Software/Driver Installation

    Chapter IV. iWARP (RDMA) Software/Driver Installation 2.1. Pre-requisites • libnl-devel, libnl3-devel and valgrind-devel packages should be installed for libiwpm and OFED installation. • If you are planning to upgrade OFED on one member of the cluster, the upgrade needs to be installed on all the members.

  • Page 86: Software/Driver Loading

    Chapter IV. iWARP (RDMA) Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb 3.1. Loading iWARP Driver The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 87: Software/Driver Configuration And Fine-Tuning

    Chapter IV. iWARP (RDMA) Software/Driver Configuration and Fine-tuning 4.1. Testing connectivity with ping and rping Load the NIC, iWARP & core RDMA modules as mentioned in Software/Driver Loading section. After which, you will see two or four ethernet interfaces for the Terminator device. Configure them with an appropriate ip address, netmask, etc.

  • Page 88: Enabling Various Mpis

    Chapter IV. iWARP (RDMA) 4.2. Enabling various MPIs Setting shell for Remote Login User needs to set up authentication on the user account on all systems in the cluster to allow user to remotely logon or executing commands without password. Quick steps to set up user authentication: Change to user home directory [root@host~]# cd...
  • Page 89 Chapter IV. iWARP (RDMA) vi. Set IntelMPI with mpi-selector (do this on all nodes). [root@host~]# mpi-selector --register intelmpi --source-dir /opt/intel/impi/3.1/bin/ [root@host~]# mpi-selector --set intelmpi .bashrc vii. Edit and add these lines: export RSH=ssh export DAPL_MAX_INLINE=64 export I_MPI_DEVICE=rdssm:chelsio export MPIEXEC_TIMEOUT=180 export MPI_BIT_MODE=64 viii.
  • Page 90 Chapter IV. iWARP (RDMA) iii. Untar and change your current working directory to openMPI package directory. iv. Configure and install as: [root@host~]#./configure --with-openib=/usr CC=gcc CXX=g++ F77=gfortran FC=gfortran --enable-mpirun-prefix-by-default --prefix=/usr/mpi/gcc/openmpi- x.y.z/ --with-openib-libdir=/usr/lib64/ --libdir=/usr/mpi/gcc/openmpi- x.y.z/lib64/ --with-contrib-vt-flags=--disable-iotrace [root@host~]# make [root@host~]# make install The above step will install openMPI in /usr/mpi/gcc/openmpi-x.y.z/ To enable multithreading, add “--enable-mpi-thread-multiple”...
  • Page 91 Chapter IV. iWARP (RDMA) vi. Simlarly, create another shell script, mpivars.sh, with the following entry: # PATH if test -z "`echo $PATH | grep /usr/mpi/gcc/openmpi-x.y.z/bin`"; then PATH=/usr/mpi/gcc/openmpi-x.y.z/bin:${PATH} export PATH # LD_LIBRARY_PATH if test -z "`echo $LD_LIBRARY_PATH | grep /usr/mpi/gcc/openmpi- x.y.z/lib64`"; then LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi- x.y.z/lib64${LD_LIBRARY_PATH:+:}$ {LD_LIBRARY_PATH}...
  • Page 92 Chapter IV. iWARP (RDMA) MVAPICH2 (Installation and Setup) Download the latest MVAPICH2 software package from http://mvapich.cse.ohio-state.edu/ ii. Untar and change your current working directory to MVAPICH2 package directory. iii. Configure and install as: [root@host~]# ./configure --prefix=/usr/mpi/gcc/mvapich2-x.y/ --with- device=ch3:mrail --with-rdma=gen2 --enable-shared --with-ib- libpath=/usr/lib64/ -enable-rdma-cm --libdir=/usr/mpi/gcc/mvapich2-x.y/lib64 [root@host~]# make [root@host~]# make install...
  • Page 93 Chapter IV. iWARP (RDMA) v. Simlarly, create another shell script, mpivars.sh, with the following entry: # PATH if test -z "`echo $PATH | grep /usr/mpi/gcc/ mvapich2-x.y/bin`"; then PATH=/usr/mpi/gcc/mvapich2-x.y/bin:${PATH} export PATH # LD_LIBRARY_PATH if test -z "`echo $LD_LIBRARY_PATH | grep /usr/mpi/gcc/mvapich2- x.y/lib64`";...
  • Page 94 Chapter IV. iWARP (RDMA) x. Set MVAPICH2: [root@host~]# mpi-selector --set mvapich2 –yes xi. Logut and log back in. mpd.hosts xii. Populate with node names. xiii. On each node, create /etc/mv2.conf with a single line containing the IP address of the local adapter interface.
  • Page 95 Chapter IV. iWARP (RDMA) The performance is best with NIC MTU set to 9000 bytes. • Run Open MPI application as: mpirun --host node1,node2 -mca btl openib,sm,self /usr/mpi/gcc/openmpi- x.y.z/tests/imb/IMB-MPI1 For OpenMPI/RDMA clusters with node counts greater than or equal to 8 nodes, and Note process counts greater than or equal to 64, you may experience the following RDMA address resolution error when running MPI jobs with the default OpenMPI settings:...

  • Page 96: Setting Up Nfs-Rdma

    Chapter IV. iWARP (RDMA) 4.3. Setting up NFS-RDMA On RHEL 7 systems, uninstall OFED 4.8-2 if present in the machine. Important Starting NFS-RDMA Server-side settings Follow the steps mentioned below to set up an NFS-RDMA server. /etc/exports Make entry in file for the directories you need to export using NFS-RDMA on server as: /share/rdma...

  • Page 97: Performance Tuning

    Chapter IV. iWARP (RDMA) exportfs vi. Run to make local directories available for Network File System (NFS) clients to mount. [root@host~]# exportfs Now the NFS-RDMA server is ready. Client-side settings Follow the steps mentioned below at the client side. Load the iwarp modules and make sure peer2peer is set to 1. Make sure you are able to ping and ssh to the server Chelsio interface through which directories will be exported.

  • Page 98: Software/Driver Unloading

    Chapter IV. iWARP (RDMA) Software/Driver Unloading To unload the iWARP driver, run the following command: [root@host~]# rmmod iw_cxgb4 Chelsio Unified Wire for Linux...

  • Page 99: Iser

    Chapter V. iSER V. iSER Chelsio Unified Wire for Linux...

  • Page 100: Introduction

    Chapter V. iSER Introduction The iSCSI Extensions for RDMA (iSER) protocol is a translation layer for operating iSCSI over RDMA transports, such as iWARP/Ethernet or InfiniBand. 1.1. Hardware Requirements Supported adapters The following are the currently shipping Chelsio adapters that are compatible with iSER driver: T62100-CR T62100-LP-CR T6425-CR...

  • Page 101: Kernel Installation

    Chapter V. iSER Kernel Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. To install 4.14.67 kernel with iSER components enabled, use the following command: [root@host~]# make kernel_install If you wish to use custom 4.9.X/4.14.X kernel, enable the following iSER Note parameters in the kernel configuration file and then proceed with kernel installation...

  • Page 102: Software/Driver Installation

    Chapter V. iSER Software/Driver Installation 3.1. Pre-requisites • libnl-devel and libnl3-devel packages should be installed for libiwpm installation. • Python v2.7 or above is required for targetcli installation. If Python v2.7 is not already present in the system, or if an older version exists, v2.7.10 provided in the package will be installed. •...

  • Page 103: Software/Driver Loading

    Chapter V. iSER Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Follow the steps mentioned below on both target and initiator machines: Unload Chelsio iWARP driver if previously loaded: [root@host~]# rmmod iw_cxgb4 ii.

  • Page 104: Software/Driver Configuration And Fine-Tuning

    Chapter V. iSER Software/Driver Configuration and Fine-tuning Configure LIO target with iSER support, using ramdisk as LUN: [root@host~]# targetcli /backstores/ramdisk create name=ram0 size=1GB [root@host~]# targetcli /iscsi create wwn=iqn.2003-01.org.lun0.target [root@host~]# targetcli /iscsi/iqn.2003-01.org.lun0.target/tpg1/luns create /backstores/ramdisk/ram0 [root@host~]# targetcli /iscsi/iqn.2003-01.org.lun0.target/tpg1 set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 [root@host~]# targetcli saveconfig ii.

  • Page 105: Hma

    Chapter V. iSER 5.1. To use HMA, please ensure that Unified Wire is installed using the Unified Wire (Default) configuration tuning option. Currently 256 IPv4/128 IPv6 iSER Offload connections are supported on T6 25G SO adapters. The following image shows the HMA reserved memory. The following image shows the number of offloaded iSER connections.

  • Page 106: Software/Driver Unloading

    Chapter V. iSER Software/Driver Unloading To unload iSER driver: On target, run the following commands: [root@host~]# rmmod ib_isert [root@host~]# rmmod iw_cxgb4 On initiator, run the following commands: [root@host~]# rmmod ib_iser [root@host~]# rmmod iw_cxgb4 Chelsio Unified Wire for Linux...

  • Page 107: Vi. Wd-Udp

    Chapter VI. WD-UDP VI. WD-UDP Chelsio Unified Wire for Linux...

  • Page 108: Introduction

    Chapter VI. WD-UDP Introduction Chelsio WD-UDP (Wire Direct-User Datagram Protocol) with Multicast is a user-space UDP stack with Multicast address reception and socket acceleration that enables users to run their existing UDP socket applications unmodified. It features software modules that enable direct wire access from user space to the Chelsio network adapter with complete bypass of the kernel, which results in an ultra-low latency Ethernet solution for high frequency trading and other delay-sensitive applications.
  • Page 109 Chapter VI. WD-UDP RHEL 7.4, 3.10.0-693.el7 RHEL 6.9, 2.6.32-696.el6 SLES 15, 4.12.14-23-default SLES 12 SP3, 4.4.73-5-default SLES 12 SP2, 4.4.21-69-default Ubuntu 18.04.1, 4.15.0-29-generic Ubuntu 16.04.4, 4.4.0-116-generic Kernel.org linux-4.14.67 Kernel.org linux-4.9 (Minimum 4.9 kernel version supported is 4.9.13) Other kernel versions have not been tested and are not guaranteed to work. Chelsio Unified Wire for Linux...

  • Page 110: Software/Driver Installation

    Chapter VI. WD-UDP Software/Driver Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install WD-UDP driver and libraries: [root@host~]# make wdudp_install For more installation options, please run make help install.py -h Note Chelsio Unified Wire for Linux...

  • Page 111: Software/Driver Loading

    Chapter VI. WD-UDP Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 112: Software/Driver Configuration And Fine-Tuning

    Chapter VI. WD-UDP Software/Driver Configuration and Fine-tuning 4.1. Accelerating UDP Socket Communications The libcxgb4_sock library is a LD_PRELOAD-able library that accelerates UDP Socket communications transparently and without recompilation of the user application. This section describes how to use libcxgb4_sock. By preloading libcxgb4_sock, all sockets created by the application are intercepted and possibly accelerated based on the user’s configuration.
  • Page 113 Chapter VI. WD-UDP Routed through the Terminator acceleration device. If the ingress datagram arrives via a device other than the Terminator acceleration device, then it will not utilize the acceleration path. On egress, if the destination IP address will not route out via the Terminator device, then it too will not be accelerated.
  • Page 114 Chapter VI. WD-UDP # Syntax: # endpoint {attributes} ... # where attributes include: interface = interface-name port = udp-port-number vlan = vlan-id priority = vlan-priority # e.g. # endpoint { interface=eth2.5 port = 8000 vlan = 5 prio rity=1 # endpoint { interface=eth2 port=9999} # endpoints that bind to port 0 (requesting the host allocate a port) # can be accelerated with port=0: # endpoint {interface=eth1 port=0}...
  • Page 115 Chapter VI. WD-UDP Now, preload libcxgb4_sock: [root@host~]# CXGB4_SOCK_CFG=<path to config file> LD_PRELOAD=libcxgb4_sock.so <pathto>/your_application In order to offload IPv6 UDP sockets, please select “low latency networking” as Note configuration tuning option during installation. Running WD-UDP in debug mode To use libcxgb4_sock’s debug capabilities, use the libcxgb4_sock_debug library provided in the package.
  • Page 116 Chapter VI. WD-UDP Example with hpcbench/udp The udp benchmark from the hpcbench suite can be used to show the benefits of libcxgb4_sock. The hpcbench suite can be found at: Source: http://hpcbench.sourceforge.net/index.html Sample: http://hpcbench.sourceforge.net/udp.html The nodes in this example, r9 and r10, have Terminator eth1 configured and the ports are connected point-to-point.
  • Page 117 Chapter VI. WD-UDP Then on r9, we run udptest to port 9000 to see the host stack UDP latency: [root@r9 ~]# /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9000 Running the same test with libcxgb4_sock: [root@r9 ~]# LD_PRELOAD=libcxgb4_sock.so /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9001 Determining if the application is being offloaded To see if the application is being offloaded, open a window on one of the machines, and run tcpdump...

  • Page 118: Software/Driver Unloading

    Chapter VI. WD-UDP Software/Driver Unloading To unload the WD-UDP driver, run the following command: [root@host~]# rmmod iw_cxgb4 Chelsio Unified Wire for Linux...

  • Page 119: Vii. Wd-Toe

    Chapter VII. WD-TOE VII. WD-TOE Chelsio Unified Wire for Linux...

  • Page 120: Introduction

    Chapter VII. WD-TOE Introduction Chelsio WD-TOE (Wire Direct-Transmission Control Protocol) with a user-space TCP stack enables users to run their existing TCP socket applications unmodified. It features software modules that enable direct wire access from user space to the Chelsio network adapter with complete bypass of the kernel, which results in a low latency Ethernet solution for high frequency trading and other delay-sensitive applications.

  • Page 121: Software/Driver Installation

    Chapter VII. WD-TOE Software/Driver Installation 2.1. Pre-requisites Before proceeding, please ensure that the offload drivers are installed with WD-TOE support. They are installed by default with Low Latency Networking, Wire Direct Latency or High Capacity WD configuration tuning Options. With any other configuration tuning option, the installation needs to be customized.

  • Page 122: Software/Driver Loading

    Chapter VII. WD-TOE Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 123: Software/Driver Configuration And Fine-Tuning

    Chapter VII. WD-TOE Software/Driver Configuration and Fine-tuning 4.1. Running the Application Bring up the Chelsio interface to enable offload: [root@host~]# ifconfig ethX <IP> up ii. Run the application with WD-TOE using: [root@host~]# PROT=TCP wdload <path to application> • Netperf, sockperf and IPerf only supported. Note •...

  • Page 124: Software/Driver Unloading

    Chapter VII. WD-TOE Software/Driver Unloading Unload NIC/TOE drivers as mentioned in Software/Driver Unloading section of Network (NIC/TOE) chapter. Chelsio Unified Wire for Linux...

  • Page 125: Viii. Nvme-Of

    Chapter VIII. NVMe-oF VIII. NVMe-oF Chelsio Unified Wire for Linux...

  • Page 126: Introduction

    Chapter VIII. NVMe-oF Introduction NVMe over Fabrics specification extends the benefits of NVMe to large fabrics, beyond the reach and scalability of PCIe. NVMe enables deployments with hundreds or thousands of SSDs using a network interconnect, such as RDMA over Ethernet. Thanks to an optimized protocol stack, an end-to-end NVMe solution is expected to reduce access latency and improve performance, particularly when paired with a low latency, high efficiency transport such as RDMA.
  • Page 127 Chapter VIII. NVMe-oF SLES 12 SP3, 4.4.73-5-default Ubuntu 18.04.1, 4.15.0-29-generic Kernel.org linux-4.14.67 (kernel compiled on RHEL 7.3 and above) Kernel.org linux-4.9 (Minimum 4.9 kernel version supported is 4.9.13; kernel compiled on RHEL 7.3 and above) Other kernel versions have not been tested and are not guaranteed to work. Chelsio Unified Wire for Linux...

  • Page 128: Kernel Installation

    Chapter VIII. NVMe-oF Kernel Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install 4.14.67 kernel with NVMe-oF components enabled, use the following command: [root@host~]# make kernel_install If you wish to use custom 4.9.X/4.14.X kernel, enable the following parameters in Note the kernel configuration file and then proceed with kernel installation: CONFIG_BLK_DEV_NVME=m...

  • Page 129: Software/Driver Installation

    Chapter VIII. NVMe-oF Software/Driver Installation 3.1. Pre-requisites • libnl-devel, libnl3-devel and valgrind-devel packages should be installed for libiwpm and OFED installation. • bzip2-devel, zlib-devel, ncurses-devel, sqlite-devel, libudev-devel packages for nvmetcli utility should be installed. • Python v2.7 or above should be installed. If this version is not already present in the system, or if an older version exists, v2.7.10 provided in the package will be installed.

  • Page 130: Software/Driver Loading

    Chapter VIII. NVMe-oF Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Follow the steps mentioned below on both target and initiator machines: Load the following modules: [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm...

  • Page 131: Software/Driver Configuration And Fine-Tuning

    Chapter VIII. NVMe-oF Software/Driver Configuration and Fine-tuning The following sections describe the method to configure target and initiator: 5.1. Target The following commands will configure target using nvmetcli with a LUN: [root@host~]# nvmetcli /> cd subsystems /subsystems> create nvme-ram0 /subsystems> cd nvme-ram0/namespaces /subsystems/n...m0/namespaces>...

  • Page 132: Initiator

    Chapter VIII. NVMe-oF 5.2. Initiator Discover the target [root@host~]# nvme discover -t rdma -a <target_ip> -s 4420 ii. Connect to target • Connecting to a specific target: [root@host~]# nvme connect -t rdma -a <target_ip> -s 4420 -n <target_name> • Connecting to all targets configured on a portal [root@host~]# nvme connect-all -t rdma -a <target_ip>...

  • Page 133: Performance Tuning

    Chapter VIII. NVMe-oF The following image shows the HMA reserved memory. The following image shows the number of offloaded NVMe-oF connections: 5.4. Performance Tuning Apply the performance settings mentioned in the Performance Tuning section in the Unified Wire chapter before proceeding. Run the performance tuning script to map iWARP queues to different CPUs.

  • Page 134: Software/Driver Unloading

    Chapter VIII. NVMe-oF Software/Driver Unloading Follow the steps mentioned below to unload the drivers: On target, run the following commands: [root@host~]# rmmod nvmet-rdma [root@host~]# rmmod nvmet [root@host~]# rmmod iw_cxgb4 On initiator, run the following commands: [root@host~]# rmmod nvme-rdma [root@host~]# rmmod nvme [root@host~]# rmmod iw_cxgb4 Chelsio Unified Wire for Linux...

  • Page 135: Ix. Lio Iscsi Target Offload

    Chapter IX. LIO iSCSI Target Offload IX. LIO iSCSI Target Offload Chelsio Unified Wire for Linux...

  • Page 136: Introduction

    Chapter IX. LIO iSCSI Target Offload Introduction Linux-IO Target (LIO) is the in-kernel SCSI target implementation in Linux. This open-source standard supports common storage fabrics, including Fibre Channel, FCoE, iEEE 1394, iSCSI, NVMe-oF, iSER, SRP, USB, vHost, etc. The LIO iSCSI fabric module implements many advanced iSCSI features that increase performance and resiliency.
  • Page 137 Chapter IX. LIO iSCSI Target Offload RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64) RHEL 7.4, 3.10.0-693.el7 RHEL 7.4, 3.10.0-693.el7.ppc64le (POWER8 LE) SLES 15, 4.12.14-23-default SLES 12 SP3, 4.4.73-5-default Ubuntu 18.04.1, 4.15.0-29-generic Kernel.org linux-4.14.67 Kernel.org linux-4.9 (Minimum 4.9 kernel version supported is 4.9.13) Other versions have not been tested and are not guaranteed to work. Chelsio Unified Wire for Linux...

  • Page 138: Kernel Configuration

    Chapter IX. LIO iSCSI Target Offload Kernel Configuration RHEL 7.4/ RHEL 7.5 Download the kernel source RPM kernel-3.10.0-xxx.el7.src.rpm. ii. Install the kernel source. [root@host~]# rpm -ivh kernel-3.10.0-xxx.el7.src.rpm iii. Prepare the kernel source. [root@host~]# cd /root/rpmbuild/SPECS/ [root@host~]# rpmbuild -bp kernel.spec --nodeps [root@host~]# cd /root/rpmbuild/BUILD/kernel-3.10.0-xxx.el7/linux-3.10.0- xxx.el7.x86_64/ [root@host~]# make prepare...
  • Page 139 Chapter IX. LIO iSCSI Target Offload → d. Select Device Drivers Generic Target Core Mod (TCM) and ConfigFS Infrastructure. e. Enable Linux-iSCSI.org iSCSI Target Mode Stack as a Module (if not already enabled). Select Save. g. Exit from the installation menu. h.
  • Page 140 Chapter IX. LIO iSCSI Target Offload iv. Check out to the changeset. [root@host~]# git checkout Ubuntu-4.15.0-29.31 v. Proceed with driver installation as directed in the Software/Driver Installation section. 3.14.57 Download the kernel from kernel.org. ii. Untar the tar-ball. iii. Change your working directory to kernel directory and run the following command to invoke the installation menu.

  • Page 141: Software/Driver Installation

    Chapter IX. LIO iSCSI Target Offload Software/Driver Installation 3.1. Pre-requisites The LIO iSCSI Target Offload driver requires the following components to be installed to function: • Python (v2.7.10 provided in the package) • TargetCLI (v2.1 provided in the package) • OpenSSL (Download from https://www.openssl.org/source/) If not already present in the system, the component provided in the package will be installed along with the kernel.
  • Page 142 Chapter IX. LIO iSCSI Target Offload • CLI mode (without Dialog utility) [root@host~]# ./install.py --ksrc=<kernel_source_dir> -- kobj=<kernel_object_dir> Example: For RHEL 7.5, [root@host~]# ./install.py --ksrc=/usr/src/linux-3.10.0-862.el7 - kobj=/lib/modules/3.10.0-862.el7.x86_64/build/ • GUI mode [root@host~]# ./install.py --set-kpath Provide the paths for kernel source and kernel object on the last screen of the installer. Select “OK”.

  • Page 143: Software/Driver Loading

    Chapter IX. LIO iSCSI Target Offload Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 144: Software/Driver Configuration And Fine-Tuning

    Chapter IX. LIO iSCSI Target Offload Software/Driver Configuration and Fine-tuning 5.1. Configuring LIO iSCSI Target iSCSI Target needs configured before become useful. Please refer the user manual at http://www.linux-iscsi.org/Doc/LIO Admin Manual.pdf to do so. Sample Configuration Here is a sample iSCSI configuration listing a target configured with 1 RAM disk LUN and ACL not configured: 5.2.

  • Page 145: Running Lio Iscsi And Network Traffic Concurrently

    Chapter IX. LIO iSCSI Target Offload 5.3. Running LIO iSCSI and Network Traffic Concurrently If you wish to run network traffic with offload support (TOE) and LIO iSCSI traffic together, follow the steps mentioned below: If not done already, load network driver with offload support (TOE): [root@host~]# modprobe t4_tom ii.

  • Page 146: Performance Tuning

    Chapter IX. LIO iSCSI Target Offload 5.4. Performance Tuning Apply the performance settings mentioned in the Performance Tuning section in the Unified Wire chapter before proceeding. ii. Run the performance tuning script to map LIO Target queues to different CPUs. [root@host~]# t4_perftune.sh -Q iSCSIT -n iii.

  • Page 147: Software/Driver Unloading

    Chapter IX. LIO iSCSI Target Offload Software/Driver Unloading 6.1. Unloading the LIO iSCSI Target Offload Driver To unload the LIO iSCSI Target Offload kernel module, follow the steps mentioned below: Log out from the initiator. ii. Run the following command: [root@host~]# echo 0 >...

  • Page 148: Iscsi Pdu Offload Target

    Chapter X. iSCSI PDU Offload Target X. iSCSI PDU Offload Target Chelsio Unified Wire for Linux...

  • Page 149: Introduction

    Chapter X. iSCSI PDU Offload Target Introduction This section describes how to install and configure iSCSI PDU Offload Target software for use as a key element in your iSCSI SAN. The software runs on Linux-based systems that use Chelsio or non-Chelsio based Ethernet adapters.

  • Page 150: Hardware Requirements

    Chapter X. iSCSI PDU Offload Target • Error Recovery Level 0 (ERL 0) • CHAP support for both discovery and login including mutual authentication • Internet Storage Name Service (iSNS) Client • Target Access Control List (ACL) • Multiple Connections per Session •...

  • Page 151: Software Requirements

    Chapter X. iSCSI PDU Offload Target T420-CR T440-CR T422-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX Adapter Requirements The Chelsio iSCSI PDU Offload Target software can be used with or without hardware protocol offload technology. There are two modes of operation using the iSCSI PDU Offload Target software on Ethernet-based adapters: Regular NIC –...
  • Page 152 Chapter X. iSCSI PDU Offload Target Linux Requirements Currently the iSCSI PDU Offload Target software is available for the following versions: RHEL 7.3, 4.5.0-15.el7.aarch64 (ARM64) RHEL 6.9, 2.6.32-696.el6 SLES 12 SP2, 4.4.21-69-default Ubuntu 16.04.4, 4.4.0-116-generic Other kernel versions have not been tested and are not guaranteed to work. Requirements for Installing the iSCSI Software When installing the iSCSI software, it is required that the system have Linux kernel source or its headers installed in order to compile the iSCSI software as a kernel module.

  • Page 153: Software/Driver Installation

    Chapter X. iSCSI PDU Offload Target Software/Driver Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install iSCSI-target driver,firmware and utilities: [root@host~]# make iscsi_pdu_target_install make help install.py -h For more installation options, please run Note Chelsio Unified Wire for Linux...

  • Page 154: Software/Driver Loading

    Chapter X. iSCSI PDU Offload Target Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb There are two main steps to installing the Chelsio iSCSI PDU Offload Target software. They are: 1.
  • Page 155 Chapter X. iSCSI PDU Offload Target While using rpm-tar-ball for installation Note a. Uninstallation will result into chiscsi.conf file renamed into chiscsi.conf.rpmsave, but if again uninstallation is done then it will lead to overwriting of the old chiscsi.rpmsave file. b. Its advised to take a backup of chiscsi.conf file before you do an uninstallation and installation of new/same unified wire package.

  • Page 156: Software/Driver Configuration And Fine-Tuning

    Chapter X. iSCSI PDU Offload Target Software/Driver Configuration and Fine-tuning The Chelsio iSCSI software needs configuration before it can become useful. The following sections describe how this is done. There are two main components used in configuring the Chelsio iSCSI software: the configuration file and the iSCSI control tool.

  • Page 157: A Quick Start Guide For Target

    Chapter X. iSCSI PDU Offload Target configurable item is “ACL” (for Access Control List). “ACL” is one of the few items that have no default. Before starting any iSCSI target, an iSCSI configuration file must be created. An easy way to create this file is to use the provided sample configuration file and modify it.
  • Page 158 Chapter X. iSCSI PDU Offload Target /etc/chelsio- Here is a sample of a minimum iSCSI target configuration located at iscsi/chiscsi.conf target: TargetName=iqn.2006-02.com.chelsio.diskarray.san1 TargetDevice=/dev/sda PortalGroup=1@192.0.2.178:3260 TargetDevice PortalGroup value must match with the storage device in the system. The value must have a matching IP address of the Ethernet adapter card in the system. For more information about TargetDevice configuration see Target Storage Device Configuration.

  • Page 159: The Iscsi Configuration File

    Chapter X. iSCSI PDU Offload Target iscsictl View Configuration: To see the configuration of all the active iSCSI targets, execute with “ ” option. [root@host~]# iscsictl –c with “ ” To see the more detailed configuration settings of a specific target, execute iscsictl option followed by the target name.
  • Page 160 Chapter X. iSCSI PDU Offload Target The parameter list in an entity block contains both: iSCSI parameters that override the default values Parameters that facilitate passing of control information to the iSCSI module All lines in the configuration file that begin with “ ”...
  • Page 161 Chapter X. iSCSI PDU Offload Target Table of Chelsio Global Entity Settings Default Multiple Valid Values Description Value Values Authorization order for login verification on the target. Valid only when a target’s “ACL” ACL_Enable=Yes iscsi_auth_order “CHAP” “CHAP” ACL: ACL first then CHAP CHAP: CHAP first then ACL Applies to Target(s) Only “CHAP”...
  • Page 162 Chapter X. iSCSI PDU Offload Target using Chelsio iscsi initiator driver. iscsi_target_vendor_id “CHISCSI” a string of The target vendor ID part of the device identification sent by an maximum of iSCSI target in response of 8 characters SCSI Inquiry command. Time allowed (in seconds) for the initiator to complete the iscsi_login_complete_tim...
  • Page 163 Chapter X. iSCSI PDU Offload Target DataPDUInOrder “Yes” “Yes” To indicate the data PDUs with “No” sequence must be at continuously increasing order or can be in any order. Chelsio only supports “Yes”. DataSequenceInOrder “Yes” “Yes” To indicate the Data PDU sequences “No”...
  • Page 164 Chapter X. iSCSI PDU Offload Target 3. Access Control List (ACL) Table of Chelsio Entity Settings Default Multiple Valid Values Description Value Values Chelsio CHAP Parameter (Target) Auth_CHAP_Target “<user id>” CHAP user id and secret for the :“<secret>” target. <user id> must be less than 256 characters.
  • Page 165 Chapter X. iSCSI PDU Offload Target <timeout value group tag. milliseconds>] <port number> is the port number [,[portalgroupta associated with the portal group tag. It is optional and if not specified the portalgrouptag2, well-known iSCSI port number of … 3260 is used. portalgrouptagn] <timeout>...
  • Page 166 Chapter X. iSCSI PDU Offload Target SYNC specifies that the device will function in the write-through mode (i.e., the data will be flushed to the device before the response is returned to the initiator). NOTE: SYNC is only applicable with FILE mode.
  • Page 167 Chapter X. iSCSI PDU Offload Target NOTE: FILE mode is the most versatile mode, if in doubt use FILE mode. ACL_Enable “Yes” “No” Defines if Chelsio’sAccess Control “No” List (ACL) method will be enforced on the target: Yes: ACL is enforced on the target No: ACL is not enforced on the target NOTE: ACL flag is not allowed to be...
  • Page 168 Chapter X. iSCSI PDU Offload Target 0..N: only one value from 0 through 0~N: a range of values between 0 through N ALL: all currently supported LUNs. Multiple lists of LUN numbers are allowed. When specifying the list separate the LUN ranges by a comma.
  • Page 169 Chapter X. iSCSI PDU Offload Target # iSCSI configuration TargetName=iqn.2005-8.com.chelsio:diskarrays.san.328 TargetAlias=iTarget1 MaxOutstandingR2T=1 MaxRecvDataSegmentLength=8192 HeaderDigest=None,CRC32C DataDigest=None,CRC32C ImmediateData=Yes InitialR2T=No FirstBurstLength=65535 MaxBurstLength=262144 # Local block devices being served up # lun 0 is pointed to /dev/sda # lun 1 is pointed to /dev/sdb TargetDevice=/dev/sda,ID=aabbccddeeffgghh,WWN=aaabbbcccdddeeef TargetDevice=/dev/sdb # Portal groups served this target...

  • Page 170: Challenge-Handshake Authenticate Protocol (Chap)

    Chapter X. iSCSI PDU Offload Target # ACL configuration # initiator “iqn.2006-02.com.chelsio.san1” is allowed full access # to this target ACL=iname=iqn.2006-02.com.chelsio.san1 # any initiator from IP address 102.50.50.101 is allowed full access # of this target ACL=sip=102.50.50.101 # any initiator connected via the target portal 102.60.60.25 # allowed full access to this target ACL=dip=102.60.60.25 # initiator “iqn.2005-09.com.chelsio.san2”...
  • Page 171 Chapter X. iSCSI PDU Offload Target For mutual CHAP, in addition to the initiator CHAP id and secret, the target CHAP id and secret are required. They are configured and stored on a per target basis with Chelsio Entity parameter “Auth_CHAP_Target”.

  • Page 172: Target Access Control List (Acl) Configuration

    Chapter X. iSCSI PDU Offload Target Auth_CHAP_Policy=Oneway , the iSCSI target will accept a relevant initiator only if it does a) CHAP Oneway or b) CHAP Mutual Auth_CHAP_Policy=Mutual , the iSCSI node will accept a relevant initiator only if it does a) CHAP Mutual Discovery Session CHAP CHAP authentication is also supported for the discovery sessions where an initiator queries all of...
  • Page 173 Chapter X. iSCSI PDU Offload Target The format for ACL rule is as follows: ACL=[iname=<initiator name>][;<sip=<source ip addresses>] [;dip=<destination ip addresses>][;lun=<lun_list>:<permissions>] target: TargetName=iqn.2006-02.com.chelsio.diskarray.san1 TargetDevice=/dev/sda PortalGroup=1@102.50.50.25:3260 PortalGroup=2@102.60.60.25:3260 # initiator “iqn.2006-02.com.chelsio.san1” is allowed # full read-write access to this target ACL=iname=iqn.2006-02.com.chelsio.san1 # any initiator from IP address 102.50.50.101 is allowed full # read-write access of this target ACL=sip=102.50.50.101 # any initiator connected via the target portal 102.60.60.25...

  • Page 174: Target Storage Device Configuration

    Chapter X. iSCSI PDU Offload Target When a target device is marked as read-only (RO), it takes precedence over ACL’s write permission (i.e., all of ACL write permission of an initiator is ignored). 4.7. Target Storage Device Configuration An iSCSI Target can support one or more storage devices. The storage device can either be the built-in RAM disk or actual backend storage.
  • Page 175 Chapter X. iSCSI PDU Offload Target FILE Mode Storage Device Details The FILE mode storage device is the most common and versatile mode to access the actual storage attached to the target system: The FILE mode can accommodate both block devices and virtual block devices. The device is accessed in the exclusive mode.

  • Page 176: Target Redirection Support

    Chapter X. iSCSI PDU Offload Target TargetDevice=<path to the storage device>,BLK <path> Where: Is the path to the actual storage device, such as /dev/sdb. The path must exist in the system. target: #<snip> # /dev/sdb is accessed in BLK mode TargetDevice=/dev/sdb,BLK #<snip>...
  • Page 177 Chapter X. iSCSI PDU Offload Target target: #<snip> # any login requests received on 10.193.184.81:3260 will be # redirected to 10.193.184.85:3261. PortalGroup=1@10.193.184.81:3260,[2] PortalGroup=2@10.193.184.85:3261 # the PortalGroup “2” is NOT presented on the same system. ShadowMode=Yes #<snip> Below is an example with ShadowMode disabled: target: #<snip>...

  • Page 178: The Command Line Interface Tools "Iscsictl" & "Chisns

    Chapter X. iSCSI PDU Offload Target The command line interface tools “iscsictl” & “chisns” 4.9. iscsictl iscsictl is the tool Chelsio provides for controlling the iSCSI target. It is a Command Line Interface (CLI) that is invoked from the console. Its usage is as follows: iscsictl <options>...
  • Page 179 Chapter X. iSCSI PDU Offload Target target=<name> Display the Chelsio iSCSI target configuration. [,name2 target=<name> parameter: Where name is the name of the node whose information will be returned. name ,<nameN>] can be one or more string of names, separated by a comma, <name1[,name2,…,nameN] | ALL>...
  • Page 180 Chapter X. iSCSI PDU Offload Target To flush all the targets in the system: iscsictl -F To flush a particular target: iscsictl -F target=iqn.com.cc.it1 To flush only the lun 0 of a particular target: iscsictl -F target=iqn.com.cc.it1 -k lun=0 Display the Chelsio iSCSI Global Entity Settings. <var=const>...
  • Page 181 Chapter X. iSCSI PDU Offload Target A name of ALL starts or reloads all targets specified in the configuration file. ALL is a reserved string that must be uppercase. The target=<name> parameter is optional. If this command line option is specified without the -f option, the default configuration file /etc/chelsio-iscsi/chiscsi.conf will be used.
  • Page 182 Chapter X. iSCSI PDU Offload Target specified iSCSI configuration file. Will delete any non-active targets’ configuration from the specified file. The -f option MUST be specified along with this option. Display the help messages. server=<IP id=<isns entity id> Start the Chelsio iSNS client. address>...

  • Page 183: Rules Of Target Reload (I.e. "On The Fly" Changes)

    Chapter X. iSCSI PDU Offload Target chisns server=192.0.2.10 chisns server=192.0.2.10:3205 id=isnscln2 query=30 In the first example the minimum command set is given where the IP address of the iSNS server is specified. In the second example a fully qualified command is specified by also setting three optional parameters.

  • Page 184: System Wide Parameters

    Chapter X. iSCSI PDU Offload Target Auth_CHAP_ChallengeLength Auth_CHAP_Policy The following parameters SHOULD NOT be changed because only one valid value is supported: DataPDUInOrder (support only "Yes") DataSequenceInOrder (support only "Yes") ErrorRecoveryLevel (support only "0") OFMarker (support only "No") IFMarker (support only "No") The following parameters can be changed but would not have any effect because they are either not supported or they are irrelevant: DefaultTime2Wait...

  • Page 185: Performance Tuning

    Chapter X. iSCSI PDU Offload Target iscsi_auth_order=CHAP CHAP: When setting , initiator authorization will be performed at the end of the login phase for an iSCSI normal session: before going to the full feature phase, the target will check its ACL. If this iSCSI connection does not match any ACL provisioned, the login attempt will be terminated.

  • Page 186: Software/Driver Unloading

    Chapter X. iSCSI PDU Offload Target Software/Driver Unloading Use the following command to unload the module: [root@host~]# rmmod chiscsi_t4 Chelsio Unified Wire for Linux...

  • Page 187: Xi. Iscsi Pdu Offload Initiator

    Chapter XI. iSCSI PDU Offload Initiator XI. iSCSI PDU Offload Initiator Chelsio Unified Wire for Linux...

  • Page 188: Introduction

    Chapter XI. iSCSI PDU Offload Initiator Introduction The Chelsio Unified Wire series of adapters support iSCSI acceleration and iSCSI Direct Data Placement (DDP) where the hardware handles the expensive byte touching operations, such as CRC computation and verification, and direct DMA to the final host memory destination: •...

  • Page 189: Software Requirements

    Chapter XI. iSCSI PDU Offload Initiator T580-LP-CR T540-CR T540-LP-CR T540-BT T520-CR T520-LL-CR T520-BT T420-CR T440-CR T422-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX ^ Memory-free 256 IPv4/128 IPv6 offload connections supported. 1.2. Software Requirements Linux Requirements Currently the iSCSI PDU Offload Initiator software is available for the following versions: RHEL 7.5, 3.10.0-862.el7 RHEL 7.5, 3.10.0-862.el7.ppc64le (POWER8 LE) RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64)

  • Page 190: Software/Driver Installation

    Chapter XI. iSCSI PDU Offload Initiator Software/Driver Installation 2.1. Pre-requisites Please make sure that the following requirements are met before installation: • The iSCSI PDU Offload Initiator driver (cxgb4i) runs on top of NIC module (cxgb4) and open- iscsi-2.0-872/873/874 only, on a Chelsio card. •...

  • Page 191: Software/Driver Loading

    Chapter XI. iSCSI PDU Offload Initiator Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 192: Software/Driver Configuration And Fine-Tuning

    Chapter XI. iSCSI PDU Offload Initiator Software/Driver Configuration and Fine-tuning 4.1. Accelerating open-iSCSI Initiator The following steps need to be taken to accelerate the open-iSCSI initiator: Configuring interface (iface) file Create the file automatically by loading cxgb4i driver and then executing the following command: [root@host~]# iscsiadm -m iface Alternatively, you can create an interface file located under iface directory for the new transport...
  • Page 193 Chapter XI. iSCSI PDU Offload Initiator /etc/iscsi/ifaces/ i. The interface file needs to be created in directory. Note iface.ipaddress ii. If is specified, it needs to be either the same as the ethX's IP address or an address on the same subnet. Make sure the IP address is unique in the network.

  • Page 194: Hma

    Chapter XI. iSCSI PDU Offload Initiator Example: [root@host~]# iscsiadm -m node -T iqn.2004-05.com.chelsio.target1 -p 102.2.2.155:3260,1 -I cxgb4i.00:07:43:04:5b:da -l If the login fails with an error message in the format of ERR! MaxRecvSegmentLength <X> too big. Need to be <= <Y> iscsi/iscsid.conf .

  • Page 195: Auto Login From Cxgb4I Initiator At Os Bootup

    Chapter XI. iSCSI PDU Offload Initiator Load the driver using the following parameters. [root@host~]# modprobe cxgb4i cxgb4i_snd_win=131072 cxgb4i_rcv_win=262144 The following image shows the number of offloaded connections. 4.3. Auto login from cxgb4i initiator at OS bootup For iSCSI auto login (via cxgb4i) to work on OS startup, please add the following line to start() /etc/rc.d/init.d/iscsid file on RHEL:...

  • Page 196: Performance Tuning

    Chapter XI. iSCSI PDU Offload Initiator 4.4. Performance Tuning Apply the performance settings mentioned in the Performance Tuning section in the Unified Wire chapter before proceeding. In case iSCSI Initiator IRQs pose a bottleneck for multiple connections, you can improve IOPS performance using the steps mentioned below.

  • Page 197: Software/Driver Unloading

    Chapter XI. iSCSI PDU Offload Initiator Software/Driver Unloading To unload the driver, execute the following commands: [root@host~]# rmmod cxgb4i [root@host~]# rmmod libcxgbi Chelsio Unified Wire for Linux...

  • Page 198: Xii. Crypto Offload

    Chapter XII. Crypto Offload XII. Crypto Offload Chelsio Unified Wire for Linux...

  • Page 199: Introduction

    Chapter XII. Crypto Offload Introduction Chelsio’s Terminator 6 (T6) Unified Wire ASIC enables concurrent secure communication and secure storage with support for integrated TLS/SSL/DTLS and inline cryptographic functions, leveraging the proprietary TCP/IP offload engine. Chelsio’s full offload TLS/SSL/DTLS is uniquely capable of 100Gb line-rate performance.

  • Page 200: Kernel Configuration

    Chapter XII. Crypto Offload Kernel Configuration Kernel.org linux-4.14.67/4.9 Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install the 4.14.67 kernel with Crypto components enabled by default: [root@host~]# make kernel_install iii. Boot into the new kernel and proceed with driver installation as directed in the Software/Driver Installation section.
  • Page 201 Chapter XII. Crypto Offload CONFIG_CRYPTO_PCOMP=y CONFIG_CRYPTO_PCOMP2=y CONFIG_CRYPTO_MANAGER=y CONFIG_CRYPTO_MANAGER2=y CONFIG_CRYPTO_NULL=y CONFIG_CRYPTO_WORKQUEUE=y CONFIG_CRYPTO_CRYPTD=y CONFIG_CRYPTO_AUTHENC=y CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CCM=y CONFIG_CRYPTO_GCM=y CONFIG_CRYPTO_SEQIV=y CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_CTR=y CONFIG_CRYPTO_CTS=y CONFIG_CRYPTO_ECB=y CONFIG_CRYPTO_XTS=y CONFIG_CRYPTO_HMAC=y CONFIG_CRYPTO_GHASH=y CONFIG_CRYPTO_MD4=m CONFIG_CRYPTO_MD5=y CONFIG_CRYPTO_SHA1=y CONFIG_CRYPTO_SHA256=y CONFIG_CRYPTO_SHA512=y CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_AES_X86_64=y CONFIG_CRYPTO_DEFLATE=y CONFIG_CRYPTO_ZLIB=y CONFIG_CRYPTO_LZO=y CONFIG_CRYPTO_ANSI_CPRNG=y CONFIG_CRYPTO_USER_API=y CONFIG_CRYPTO_USER_API_HASH=y CONFIG_CRYPTO_USER_API_SKCIPHER=y CONFIG_CRYPTO_USER_API_RNG=y CONFIG_CRYPTO_USER_API_AEAD=m CONFIG_CRYPTO_HW=y RHEL 7.3 To support digests on RHEL 7.3 x86_64 (base 3.10.0-514.el7 kernel), the kernel needs to be patched.
  • Page 202 Chapter XII. Crypto Offload iii. Prepare the kernel source: [root@host~]# cd /root/rpmbuild/SPECS/ [root@host~]# rpmbuild -bp kernel.spec iv. Download the patch from Kernel.org git repository and apply: [root@host~]# cd /root/rpmbuild/BUILD/kernel-3.10.0-514.el7/linux-3.10.0- 514.el7 [root@host~]# patch -p1 < kernel.patch v. Copy the patched kernel source to /usr/src/kernels [root@host~]# cp -r /root/rpmbuild/BUILD/kernel-3.10.0-514.el7/linux-3.10.0- 514.el7 /usr/src/kernels/.

  • Page 203: Software/Driver Installation

    Chapter XII. Crypto Offload Software/Driver Installation 3.1. Pre-requisites Please make sure that SELinux and firewall are disabled. 3.2. Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install Crypto driver and Chelsio openSSL modules: [root@host~]# make crypto_install For more installation options, please run make help...

  • Page 204: Software/Driver Loading

    Chapter XII. Crypto Offload Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb 4.1. Co-processor To load the Crypto Offload driver in Co-processor mode (chcr), run the following command: [root@host~]# modprobe cxgb4 [root@host~]# modprobe chcr ii.

  • Page 205: Software/Driver Configuration And Fine-Tuning

    Chapter XII. Crypto Offload Software/Driver Configuration and Fine-tuning Limitations • AEAD is not supported on RHEL7.3 and RHEL7.4 x86_64 (Kernel limitation). • httpd service supports only AEAD in SLES12 SP3. • Applications using AF_ALG are not supported on RHEL 7.4 (due to a kernel bug in AF_ALG framework).

  • Page 206: Inline

    Chapter XII. Crypto Offload 5.2. Inline Configure TLS Offload and TOE Ports To configure TLS Offload ports and TOE ports, COP rule for each TCP port must be defined. Follow the steps mentioned below: Create a new policy file and add the following line for each TCP port (to be TLS offloaded): src or dst port <tcp_port>...
  • Page 207 Chapter XII. Crypto Offload iii. Apply the policy: [root@host~]# cxgbtool <iface> policy <policy_out> Example: Upon applying the above policy, traffic on all the mentioned TCP ports are TLS Note offloaded, while traffic on other TCP ports are TOE offloaded. Configuring Chelsio OpenSSL and Applications OpenSSL which supports Chelsio inline offload is installed as part of Unified Wire package.
  • Page 208 Chapter XII. Crypto Offload • Custom Applications To compile custom applications using Chelsio OpenSSL library: [root@host~]# export LD_LIBRARY_PATH=/usr/chssl/lib/ [root@host~]# gcc -g -o <server/client output file> <server/client file> - lcrypto -lssl -L/usr/chssl/lib/ Example: Client: Server: • nginx server Download the latest stable version from nginx website.

  • Page 209: Performance Tuning

    Chapter XII. Crypto Offload Inline TLS Counters To verify if Chelsio Inline is used, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4/<PF4_id>/tls Chelsio Inline TLS Stats TLS PDU Tx: 32661534 TLS PDU Rx: 231039210 TLS Keys (DDR) Count: 48 5.3. Performance Tuning Apply the performance settings mentioned in the Performance Tuning section in the Unified...

  • Page 210: Software/Driver Unloading

    Chapter XII. Crypto Offload Software/Driver Unloading To unload Crypto Offload driver in Co-processor mode, run the following command: [root@host~]# rmmod chcr To unload Crypto Offload driver in Inline mode, unload the network driver in TOE mode. See Software/Driver Unloading section in Network (NIC/TOE) chapter for more information. Chelsio Unified Wire for Linux...

  • Page 211: Data Center Bridging (Dcb)

    Chapter XIII. Data Center Bridging (DCB) XIII. Data Center Bridging (DCB) Chelsio Unified Wire for Linux...

  • Page 212: Introduction

    Chapter XIII. Data Center Bridging (DCB) Introduction Data Center Bridging (DCB) refers to a set of bridge specification standards, aimed to create a converged Ethernet network infrastructure shared by all storage, data networking and traffic management services. An improvement to the existing specification, DCB uses priority-based flow control to provide hardware-based bandwidth allocation and enhances transport reliability.
  • Page 213 Chapter XIII. Data Center Bridging (DCB) Ubuntu 18.04.1, 4.15.0-29-generic Ubuntu 16.04.4, 4.4.0-116-generic Kernel.org linux-4.14.67 Kernel.org linux-4.9 (Minimum 4.9 kernel version supported is 4.9.13) Other kernel versions have not been tested and are not guaranteed to work. Chelsio Unified Wire for Linux...

  • Page 214: Software/Driver Installation

    Chapter XIII. Data Center Bridging (DCB) Software/Driver Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Build and install all drivers with DCB support: [root@host~]# make dcbx=1 install For more installation options, please run make help Note Chelsio Unified Wire for Linux...

  • Page 215: Software/Driver Loading

    Chapter XIII. Data Center Bridging (DCB) Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Before proceeding, please ensure that Unified Wire is installed with DCB support as mentioned in previous section.
  • Page 216 Chapter XIII. Data Center Bridging (DCB) The storage driver (FCoE Full Offload Initiator) uses the DCBX negotiated parameters (ETS, PFC etc.) without any further configuration. The network drivers (cxgb4, t4_tom) and iSCSI drivers (cxgb4i, chiscsi) need further VLAN configuration to be setup, which is explained in the Running NIC &...

  • Page 217: Software/Driver Configuration And Fine-Tuning

    Chapter XIII. Data Center Bridging (DCB) Software/Driver Configuration and Fine-tuning 4.1. Configuring Cisco Nexus 5010 switch Configuring the DCB parameters By default, the Cisco Nexus switch enables DCB functionality and configures PFC Note for FCoE traffic making it no drop with bandwidth of 50% assigned to FCoE class of traffic and another 50% for the rest (like NIC).
  • Page 218 Chapter XIII. Data Center Bridging (DCB) v. Configure qos policy-maps. switch(config)#policy-map type qos policy-test switch(config-pmap-qos)#class type qos class-nic switch(config-pmap-c-qos)#set qos-group 2 vi. Configure queuing policy-maps and assign network bandwidth. Divide the network bandwidth between FcoE and NIC traffic. switch(config)#policy-map type queuing policy-test switch(config-pmap-que)#class type queuing class-nic switch(config-pmap-c-que)#bandwidth percent 50 switch(config-pmap-c-que)#class type queuing class-fcoe...
  • Page 219 Chapter XIII. Data Center Bridging (DCB) Following steps will enable FCoE services on a particular VLAN and does a VSAN-VLAN mapping. Need not do these steps every time, unless a new mapping has to be created. switch(config)# vlan 2 switch(config-vlan)# fcoe vsan 2 switch(config-vlan)#exit ii.

  • Page 220: Configuring The Brocade 8000 Switch

    Chapter XIII. Data Center Bridging (DCB) v. On the FC Ports, if a FC target is connected then perform the following steps - switch(config)#vsan database switch(config-vsan-db)#vsan 2 switch(config-vsan-db)# vsan 2 interface fc 2/2 switch(config-vsan-db)#exit switch(config)interface fc 2/2 switch(config-if)# switchport mode auto switch(config-if)# switchport speed auto switch(config-if)# no shutdown.
  • Page 221 Chapter XIII. Data Center Bridging (DCB) iii. Configure the CEE interface as a Layer 2 switch port. Example of configuring the switch port as a 10-Gigabit Ethernet interface. switch(config)#interface tengigabitethernet 0/16 switch(config-if-te-0/16)#switchport switch(config-if-te-0/16)#no shutdown switch(config-if)#exit iv. Create an FCoE VLAN and add an interface to it. Example of creating a FCoE VLAN and adding a single interface.

  • Page 222: Running Nic & Iscsi Traffic Together With Dcbx

    Chapter XIII. Data Center Bridging (DCB) Running NIC & iSCSI Traffic together with DCBx Please refer iSCSI PDU Offload Initiator chapter to configure iSCSI Initiator. Note Use the following procedure to run NIC and iSCSI traffic together with DCBx enabled. Identify the VLAN priority configured for NIC and iSCSI class of traffic on the switch.

  • Page 223: Xiv. Fcoe Full Offload Initiator

    Chapter XIV. FCoE Full Offload Initiator XIV. FCoE Full Offload Initiator Chelsio Unified Wire for Linux...

  • Page 224: Introduction

    Chapter XIV. FCoE Full Offload Initiator Introduction Fibre Channel over Ethernet (FCoE) is a mapping of Fibre Channel over selected full duplex IEEE 802.3 networks. The goal is to provide I/O consolidation over Ethernet, reducing network complexity in the Datacenter. Chelsio FCoE initiator maps Fibre Channel directly over Ethernet while being independent of the Ethernet forwarding scheme.

  • Page 225: Software/Driver Installation

    Chapter XIV. FCoE Full Offload Initiator Software/Driver Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install FCoE full offload initiator driver: [root@host~]# make fcoe_full_offload_initiator_install For more installation options, please run make help install.py -h Note Chelsio Unified Wire for Linux...

  • Page 226: Software/Driver Loading

    Chapter XIV. FCoE Full Offload Initiator Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 227: Software/Driver Configuration And Fine-Tuning

    Chapter XIV. FCoE Full Offload Initiator Software/Driver Configuration and Fine-tuning 4.1. Configuring Cisco Nexus 5010 and Brocade switch To configure various Cisco and Brocade switch settings, please refer Software/Driver Configuration and Fine-tuning section of Data Center Bridging (DCB) chapter. 4.2. FCoE fabric discovery verification Verifying Local Ports Once connected to the switch, use the following command to see if the FIP has gone through and...
  • Page 228 Chapter XIV. FCoE Full Offload Initiator Verifying the target discovery To view the list of targets discovered on a particular FCoE port, follow the below mentioned steps: Determine the WWPN of the initiator local port under sysfs. The hosts under fc_host depends on the number of ports on the adapter used.
  • Page 229 Chapter XIV. FCoE Full Offload Initiator R can correspond to NameServer, Management Server and other initiator ports Note logged in to the switch and targets. Alternatively, the local ports can also be found using: [root@host~]# cat /sys/kernel/debug/csiostor/<pci_id>/lnodes After finding out the WWPN of the local node, to verify the list of discovered targets, use the following command.
  • Page 230 Chapter XIV. FCoE Full Offload Initiator Chelsio Unified Wire for Linux...

  • Page 231: Formatting The Luns And Mounting The Filesystem

    Chapter XIV. FCoE Full Offload Initiator 4.3. Formatting the LUNs and Mounting the Filesystem Use lsscsi -g to list the LUNs discovered by the initiator [root@host~]# lsscsi –g Chelsio Unified Wire for Linux...

  • Page 232: Creating Filesystem

    Chapter XIV. FCoE Full Offload Initiator Alternatively, the LUNs discovered by the Chelsio FCoE initiators can be accessed via easily- identifiable ‘udev’ path device files like: [root@host~]# ls /dev/disk/by-path/pci-0000:04:00.0-csio-fcoe <local_wwpn>:<remote_wwpn>:<lun_wwn> 4.4. Creating Filesystem Create an ext3 filesystem using the following command: [root@host~]# mkfs.ext3 /dev/sdx Chelsio Unified Wire for Linux...

  • Page 233: Mounting The Formatted Lun

    Chapter XIV. FCoE Full Offload Initiator 4.5. Mounting the formatted LUN The formatted LUN can be mounted on the specified mountpoint using the following command: [root@host~]# mount /dev/sdx /mnt Chelsio Unified Wire for Linux...

  • Page 234: Software/Driver Unloading

    Chapter XIV. FCoE Full Offload Initiator Software/Driver Unloading To unload the driver: [root@host~]# modprobe -r csiostor If multipath services are running, unload of FCoE driver is not possible. Stop the Note multipath service and then unload the driver. Chelsio Unified Wire for Linux...

  • Page 235: Xv. Offload Bonding

    Chapter XV. Offload Bonding XV. Offload Bonding Chelsio Unified Wire for Linux...

  • Page 236: Introduction

    Chapter XV. Offload Bonding Introduction The Chelsio Offload bonding driver provides a method to aggregate multiple network interfaces into a single logical bonded interface effectively combining the bandwidth into a single connection. It also provides redundancy in case one of link fails. The traffic running over the bonded interface can be fully offloaded to the adapter, thus freeing the CPU from TCP/IP overhead.
  • Page 237 Chapter XV. Offload Bonding RHEL 7.5, 3.10.0-862.el7 RHEL 7.5, 3.10.0-862.el7.ppc64le (POWER8 LE) RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64) RHEL 7.4, 3.10.0-693.el7 RHEL 7.4, 3.10.0-693.el7.ppc64le (POWER8 LE) RHEL 7.3, 4.5.0-15.el7.aarch64 (ARM64) RHEL 6.9, 2.6.32-696.el6 SLES 15, 4.12.14-23-default SLES 12 SP3, 4.4.73-5-default SLES 12 SP2, 4.4.21-69-default Ubuntu 18.04.1, 4.15.0-29-generic Ubuntu 16.04.4, 4.4.0-116-generic Kernel.org linux-4.14.67...

  • Page 238: Software/Driver Installation

    Chapter XV. Offload Bonding Software/Driver Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install Chelsio Offload bonding driver: [root@host~]# make bonding_install make help install.py -h For more installation options, please run Note Chelsio Unified Wire for Linux...

  • Page 239: Software/Driver Loading

    Chapter XV. Offload Bonding Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 240: Software/Driver Configuration And Fine-Tuning

    Chapter XV. Offload Bonding Software/Driver Configuration and Fine-tuning 4.1. Offloading TCP traffic over a bonded interface The Chelsio Offload Bonding driver supports all the bonding modes in NIC Mode. In offload mode (t4_tom loaded) however, only the balance-rr (mode=0), active-backup (mode=1), balance- xor (mode=2) and 802.3ad (mode=4) modes are supported.

  • Page 241: Software/Driver Unloading

    Chapter XV. Offload Bonding Software/Driver Unloading To unload the driver, run the following command: [root@host~]# rmmod bonding Chelsio Unified Wire for Linux...

  • Page 242: Offload Multi-Adapter Failover (Mafo)

    Chapter XVI. Offload Multi-Adapter Failover (MAFO) XVI. Offload Multi-Adapter Failover (MAFO) Chelsio Unified Wire for Linux...

  • Page 243: Introduction

    Chapter XVI. Offload Multi-Adapter Failover (MAFO) Introduction Chelsio’s adapters offer a complete suite of high reliability features, including adapter-to-adapter failover. The patented offload Multi-Adapter Failover (MAFO) feature ensures all offloaded traffic continue operating seamless in the face of port failure. MAFO allows aggregating network interfaces across multiple adapters into a single logical bonded interface, providing effective fault tolerance.

  • Page 244: Software Requirements

    Chapter XVI. Offload Multi-Adapter Failover (MAFO) T420-LL-CR T420-CX 1.2. Software Requirements Linux Requirements Currently the Offload Multi-Adapter Failover driver is available for the following versions: RHEL 7.5, 3.10.0-862.el7 RHEL 7.5, 3.10.0-862.el7.ppc64le (POWER8 LE) RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64) RHEL 7.4, 3.10.0-693.el7 RHEL 7.4, 3.10.0-693.el7.ppc64le (POWER8 LE) RHEL 7.3, 4.5.0-15.el7.aarch64 (ARM64) RHEL 6.9, 2.6.32-696.el6...

  • Page 245: Software/Driver Installation

    Chapter XVI. Offload Multi-Adapter Failover (MAFO) Software/Driver Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install MAFO feature: [root@host~]# make bonding_install For more installation options, please run make help install.py -h Note Chelsio Unified Wire for Linux...

  • Page 246: Software/Driver Loading

    Chapter XVI. Offload Multi-Adapter Failover (MAFO) Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 247: Software/Driver Configuration And Fine-Tuning

    Chapter XVI. Offload Multi-Adapter Failover (MAFO) Software/Driver Configuration and Fine-tuning 4.1. Offloading TCP traffic over a bonded interface The Chelsio MAFO driver supports only the active-backup (mode=1) mode. To offload TCP traffic over a bonded interface, use the following method: Load the network driver with TOE support: [root@host~]# modprobe t4_tom ii.

  • Page 248: Software/Driver Unloading

    Chapter XVI. Offload Multi-Adapter Failover (MAFO) Software/Driver Unloading To unload the driver, run the following command: [root@host~]# rmmod bonding Chelsio Unified Wire for Linux...

  • Page 249: Xvii. Udp Segmentation Offload And Pacing

    Chapter XVII. UDP Segmentation Offload and Pacing XVII. UDP Segmentation Offload and Pacing Chelsio Unified Wire for Linux...

  • Page 250: Introduction

    Chapter XVII. UDP Segmentation Offload and Pacing Introduction Chelsio’s Terminator series of adapters provide UDP segmentation offload and per-stream rate shaping to drastically lower server CPU utilization, increase content delivery capacity, and improve service quality. Tailored for UDP content, UDP Segmentation Offload (USO) technology moves the processing required to packetize UDP data and rate control its transmission from software running on the host to the network adapter.

  • Page 251: Hardware Requirements

    Chapter XVII. UDP Segmentation Offload and Pacing 1.1. Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatible with the UDP Segmentation Offload and Pacing driver: T62100-CR T62100-LP-CR T6425-CR T6225-CR T6225-LL-CR T580-CR T580-LP-CR T540-CR T540-BT T520-CR T520-LL-CR T520-BT...

  • Page 252: Software/Driver Installation

    Chapter XVII. UDP Segmentation Offload and Pacing Software/Driver Installation The offload drivers support UDP Segmentation Offload with limited number of connections (1024 connections).To build and install UDP Offload drivers which support large number of offload connections (approx 10K): 10K UDP Segmentation offload connections currently not supported on T6. Note Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x...

  • Page 253: Software/Driver Loading

    Chapter XVII. UDP Segmentation Offload and Pacing Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.

  • Page 254: Software/Driver Configuration And Fine-Tuning

    Chapter XVII. UDP Segmentation Offload and Pacing Software/Driver Configuration and Fine-tuning 4.1. Modifying the Application To use the UDP offload functionality, the application needs to be modified. Follow the steps mentioned below: Determine the UDP socket file descriptor in the application through which data is sent Declare and initialize two variables in the application: int fs=1316;...

  • Page 255: Configuring Udp Pacing

    Chapter XVII. UDP Segmentation Offload and Pacing Here: • sockfd : The file descriptor of the UDP socket • &fs / &cl : Pointer to the framesize and class variables • sizeof(fs) / sizeof(cl) : The size of the variables v.
  • Page 256 Chapter XVII. UDP Segmentation Offload and Pacing Bring up the network interface: [root@host~]# ifconfig <ethX> up Run the following command: [root@host~]# cxgbtool <ethX> sched-class params type packet level cl-rl mode flow rate-unit bits rate-mode absolute channel <Channel No.> class <scheduler-class-index> max-rate <maximum-rate> pkt-size <Packet size>...

  • Page 257: Software/Driver Unloading

    Chapter XVII. UDP Segmentation Offload and Pacing Software/Driver Unloading Reboot the system to unload the driver. To unload without rebooting, refer Unloading the TOE driver section of Network (NIC/TOE) chapter. Chelsio Unified Wire for Linux...

  • Page 258: Xviii.offload Ipv6

    Chapter XVIII. Offload IPv6 XVIII. Offload IPv6 Chelsio Unified Wire for Linux...

  • Page 259: Introduction

    Chapter XVIII. Offload IPv6 Introduction The growth of the Internet has created a need for more addresses than are possible with IPv4. Internet Protocol version 6 (IPv6) is a version of the Internet Protocol (IP) designed to succeed the Internet Protocol version 4 (IPv4). Chelsio’s Offload IPv6 feature provides support to fully offload IPv6 traffic to the Unified Wire adapter.
  • Page 260 Chapter XVIII. Offload IPv6 RHEL 7.5, 3.10.0-862.el7 RHEL 7.5, 3.10.0-862.el7.ppc64le (POWER8 LE) RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64) RHEL 7.4, 3.10.0-693.el7 RHEL 7.4, 3.10.0-693.el7.ppc64le (POWER8 LE) RHEL 7.3, 4.5.0-15.el7.aarch64 (ARM64) RHEL 6.9, 2.6.32-696.el6 SLES 15, 4.12.14-23-default SLES 12 SP3, 4.4.73-5-default SLES 12 SP2, 4.4.21-69-default Ubuntu 18.04.1, 4.15.0-29-generic Ubuntu 16.04.4, 4.4.0-116-generic Kernel.org linux-4.14.67...

  • Page 261: Software/Driver Installation

    Chapter XVIII. Offload IPv6 Software/Driver Installation 2.1. Pre-requisites Please make sure that the following requirements are met before installation: • IPv6 must be enabled in your system (enabled by default). • Unified Wire must be installed with IPv6 support as explained in the Unified Wire chapter.

  • Page 262: Software/Driver Loading

    Chapter XVIII. Offload IPv6 Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb After installing Unified Wire package and rebooting the host, load the NIC (cxgb4) and TOE (t4_tom) drivers.

  • Page 263: Software/Driver Configuration And Fine-Tuning

    Chapter XVIII. Offload IPv6 Software/Driver Configuration and Fine-tuning Load the Offload capable drivers. [root@host~]# modprobe t4_tom ii. Assign IPv6 address and bring up the interface. [root@host~]# ifconfig ethX <IPv6 address> up iii. All the IPv6 traffic over the Chelsio interface will be offloaded now. To see the number of connections offloaded, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4/<bus-id>/tids Chelsio Unified Wire for Linux...

  • Page 264: Software/Driver Unloading

    Chapter XVIII. Offload IPv6 Software/Driver Unloading 5.1. Unloading the NIC Driver To unload the NIC driver, run the following command: [root@host~]# rmmod cxgb4 5.2. Unloading the TOE Driver Please reboot the system to unload the TOE driver. To unload without rebooting, refer Unloading the TOE driver section of Network (NIC/TOE) chapter.

  • Page 265: Xix. Wd Sniffing And Tracing

    Chapter XIX. WD Sniffing and Tracing XIX. WD Sniffing and Tracing Chelsio Unified Wire for Linux...

  • Page 266: Theory Of Operation

    Chapter XIX. WD Sniffing and Tracing Theory of Operation The objective of these utilities (wd_sniffer and wd_tcpdump_trace) is to provide sniffing and tracing capabilities by making use of Chelsio adapter’s hardware features. Sniffer is a tool to measure bandwidth and involves targeting specific multicast traffic and sending it directly to user space.

  • Page 267: Hardware Requirements

    Chapter XIX. WD Sniffing and Tracing Schematic diagram of sniffer and tracer 1.1. Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatible with the tools: T62100-CR T62100-LP-CR T6225-CR T6225-LL-CR T580-CR T580-LP-CR T540-CR Chelsio Unified Wire for Linux...

  • Page 268: Software Requirements

    Chapter XIX. WD Sniffing and Tracing T520-CR T520-LL-CR T520-BT T420-CR T440-CR T422-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX 1.2. Software Requirements Linux Requirements Currently the WD Sniffing and Tracing utility is available for the following version: RHEL 7.5, 3.10.0-862.el7 RHEL 7.4, 3.10.0-693.el7 RHEL 6.9, 2.6.32-696.el6 SLES 15, 4.12.14-23-default SLES 12 SP3, 4.4.73-5-default...

  • Page 269: Software/Driver Installation

    Chapter XIX. WD Sniffing and Tracing Software/Driver Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install Sniffer tracing & filtering tcpdump and iWARP driver: [root@host~]# make sniffer_install For more installation options, please run make help install.py -h Note...

  • Page 270: Usage

    Chapter XIX. WD Sniffing and Tracing Usage 3.1. Installing Basic Support iw_cxgb4 (Chelsio iWARP driver) and cxgb4 (Chelsio NIC driver) drivers should be compiled and loaded before running the utilities. Refer to the Software/Driver Loading section for each driver and follow the instructions mentioned before proceeding. 3.2.
  • Page 271 Chapter XIX. WD Sniffing and Tracing DUT: Machine B PEER: Machine A <-----> (port 0) (port 1) <-----> PEER: Machine C 192.168.1.100 IP-dont-care IP-dont-care 192.168.1.200 2. Procedure: wd_tcpdump_trace -i iface on the command prompt where iface is one of the interfaces whose traffic you want to trace.

  • Page 272: Xx. Classification And Filtering

    Chapter XX. Classification and Filtering XX. Classification and Filtering Chelsio Unified Wire for Linux...

  • Page 273: Introduction

    Chapter XX. Classification and Filtering Introduction Classification and Filtering feature enhances network security by controlling incoming traffic as they pass through network interface based on source and destination addresses, protocol, source and receiving ports, or the value of some status bits in the packet. This feature can be used in the ingress path to: •...

  • Page 274: Software Requirements

    Chapter XX. Classification and Filtering T404-BT* T420-BCH* T440-LP-CR* T420-BT* T420-LL-CR* T420-CX* Hash filter not supported. 1.2. Software Requirements Linux Requirements Currently the Classification and Filtering feature is available for the following versions: RHEL 7.5, 3.10.0-862.el7 RHEL 7.5, 3.10.0-862.el7.ppc64le (POWER8 LE) RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64) RHEL 7.4, 3.10.0-693.el7 RHEL 7.4, 3.10.0-693.el7.ppc64le (POWER8 LE)

  • Page 275: Le-Tcam Filters

    Chapter XX. Classification and Filtering LE-TCAM Filters The default (Unified Wire) configuration tuning option allows you to create LE-TCAM filters, which has a limit of 496 for T5, and 560 for T6 adapters. For T5 adapters, all available filter indices can be optionally configured as high priority.
  • Page 276 Chapter XX. Classification and Filtering : Type of Service protocol : IP protocol number (ICMP=1, TCP=6, UDP=17, etc) Ethertype : Layer 2 EtherType Macmatch :MAC index in MPS TCAM mpshittype :MAC address "match type" (none, unicast, multicast, promiscuous, broadcast) fragmentation :Fragmented IP packets Supported Filter Combinations The following combination is set by default and packets will be matched accordingly:...
  • Page 277 Chapter XX. Classification and Filtering fragmentation, mpshittype, tos, vnic_id, port, fcoe fragmentation, mpshittype, vlan, vnic_id, fcoe fragmentation, macmatch, ethertype, protocol, port, fcoe fragmentation, macmatch, ethertype, tos, port, fcoe fragmentation, macmatch, protocol, vlan, port, fcoe fragmentation, macmatch, protocol, vnic_id, port, fcoe fragmentation, macmatch, tos, vlan, port, fcoe fragmentation, macmatch, tos, vnic_id, port, fcoe fragmentation, ethertype, vlan, port, fcoe...

  • Page 278: Creating Filter Rules

    Chapter XX. Classification and Filtering network driver needs reloaded next using following command: [root@host~]# rmmod cxgb4 [root@host~]# modprobe cxgb4 2.2. Creating Filter Rules Network driver (cxgb4) must be installed and loaded before setting the filter rule. If you haven’t done already, run the Unified Wire Installer with the appropriate configuration tuning option to install the network driver.

  • Page 279: Listing Filter Rules

    Chapter XX. Classification and Filtering The above filter rule will drop all ingress packets from source IP 192.168.1.5. Remaining packets will be sent to the host. • pass action [root@host~]# cxgbtool ethX filter 100 action pass lport 10001 fport 355 queue 2 The above filter rule will pass all ingress packets that match destination port 10001 and source port 355 to ingress queue 2 for load balancing.

  • Page 280: Removing Filter Rules

    Chapter XX. Classification and Filtering 2.4. Removing Filter Rules To remove a filter, run the following command with the corresponding filter rule index [root@host~]# cxgbtool ethX filter <index> <delete|clear> For more information on additional parameters, refer cxgbtool manual by running Note man cxgbtool command...
  • Page 281 Chapter XX. Classification and Filtering ii. Setup a static OR default route towards T6/T5 router to reach 102.1.2.0/24 network [root@host~]# route add -net 102.1.2.0/24 gw 102.1.1.250 • Follow these steps on Node 2 Configure IP address and enable the 3 interfaces: [root@host~]# ifconfig eth0 102.1.2.1/24 up [root@host~]# ifconfig eth0:2 102.1.2.2/24 up [root@host~]# ifconfig eth0:3 102.1.2.3/24 up...

  • Page 282: Layer 2 Example

    Chapter XX. Classification and Filtering ii. Create filter rule to send packets for 102.1.2.0/24 network out via eth1 interface: [root@host~]# cxgbtool eth0 filter 100 lip 102.1.2.0/24 hitcnts 1 action switch eport 1 smac 00:07:43:04:96:48 dmac 00:07:43:12:D4:88 smac dmac Where, is the MAC address of eth1 interface on T6/T5 adapter machine and is the MAC address of eth0 interface on Node 2.
  • Page 283 Chapter XX. Classification and Filtering ii. Setup ARP entry to reach 102.1.1.2 [root@host~]# arp -s 102.1.1.2 00:07:43:12:D4:88 • Follow these steps on Node 2 Configure IP address and enable the interface: [root@host~]# ifconfig eth0 102.1.1.2/24 up [root@host~]# ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:07:43:12:D4:88 inet addr:102.1.1.2...

  • Page 284: Filtering Vf Traffic

    Chapter XX. Classification and Filtering [root@host~]# bridge fdb add 00:07:43:04:7D:50 dev eth1 self vii. Both MAC entries should show up in MPS table. Run the following command to view the table and note the index (idx field) of the entries: [root@host~]# cat /sys/kernel/debug/cxgb4/0000\:01\:00.4/mps_tcam | more viii.
  • Page 285 Chapter XX. Classification and Filtering On the host, check the MPS TCAM entry for the VF MAC and note the corresponding VF ID. [root@host~]# cat /sys/kernel/debug/cxgb4/<pci_bus_id>/mps_tcam | less Apply filter rules on the Host using cxgbtool: [root@host~]# cxgbtool ethX filter <index> vf <vf_id> action [pass/drop/switch] Example: 4 VFs (VF0, VF1, VF2, Vf3) are instantiated on PF0.

  • Page 286: Hash/Ddr Filters

    Chapter XX. Classification and Filtering Hash/DDR Filters If you wish to create more filters, select T5/T6 Hash Filter configuration tuning option during installation which allows you to create ~0.5 million filter rules. You can create both LE-TCAM and Hash/DDR filters in this configuration. T5/T6 SO adapters do not support Hash Filters as they are memory free.
  • Page 287 Chapter XX. Classification and Filtering Serial # Filter Combination fragmentation, mpshittype, macmatch, ethertype, protocol, port fragmentation, mpshittype, macmatch, ethertype, protocol, fcoe fragmentation, mpshittype, macmatch, protocol, tos, port, fcoe fragmentation, mpshittype, macmatch, protocol, vlan, fcoe fragmentation, mpshittype, macmatch, protocol, vnic_id, fcoe fragmentation, mpshittype, ethertype, protocol, tos, port, fcoe fragmentation, mpshittype, protocol, tos, vlan, port fragmentation, mpshittype, protocol, tos, vlan, fcoe...

  • Page 288: Creating Filter Rules

    Chapter XX. Classification and Filtering You can change the default filter mode to any one of the following combinations, based on your requirement. The network driver needs to be reloaded next using the following command: [root@host~]# rmmod cxgb4 [root@host~]# modprobe cxgb4 use_ddr_filters=1 3.2.
  • Page 289 Chapter XX. Classification and Filtering In case of Hash/DDR filters, source_ip, destination_ip, source_port and Note destination_port are mandatory, since the filters don't support masks and hence, 4-tuple must always be supplied. Proto is also a mandatory parameter. Choosing filterMode and filterMask As mentioned earlier filterMask tuples can be subset of or equal to filterMode tuples.
  • Page 290 Chapter XX. Classification and Filtering ii. Select a filterMask so that it is a subset of or equal to filterMode based on application without VLAN tuple. E.g.; filterMask = fragmentation, mpshittype, protocol, port, fcoe Here, we have selected fragmentation, mpshittype, protocol, port, fcoe in filterMask so it is mandatory to create a filter based on only those tuples mentioned in filterMask.

  • Page 291: Listing Filter Rules

    Chapter XX. Classification and Filtering The above filter rule will switch all the packets matching above 4 tuple from Chelsio port 0 to Chelsio port 1. Remaining packets will be sent to the host. For more information on additional parameters, refer cxgbtool manual by running Note man cxgbtool command.

  • Page 292: Swap Mac Feature

    Chapter XX. Classification and Filtering 3.6. Swap MAC Feature Chelsio’s T6/T5 Swap MAC feature swaps packet source MAC and destination MAC addresses. This is applicable only for switch filter rules. Here’s an example: [root@host~]# cxgbtool eth2 filter 100 action switch lip 102.2.2.1 fip 102.2.2.2 lport 5001 fport 14000 hitcnts 1 iport 1 eport 0 swapmac 1 proto 17 cap maskless Hash-Filter Index = 21936...
  • Page 293 Chapter XX. Classification and Filtering Switch Filter with Mirroring The following example explains the method to switch and mirror traffic simultaneously: Obtain the PF and VF values of the incoming port from /sys/kernel/debug/cxgb4/<bus- id>/mps_tcam ii. Create the desired switch filter rule: [root@host~]# cxgbtool ethX filter 100 fip 102.8.8.2 lip 102.8.8.1 fport 20000 lport 12865 pf 4 vf 64 action switch iport 0 eport 1 cap maskless The above hash filter rule is created to switch traffic from 102.8.8.2,20000 to 102.8.8.1,12865...

  • Page 294: Packet Tracing And Hit Counters

    Chapter XX. Classification and Filtering 3.8. Packet Tracing and Hit Counters For T5/T6 LE-TCAM and T6 Hash/DDR filters, hit counters will work simply by adding hitcnts 1 parameter to the filter rule. However, for T5 Hash/DDR filters, you will have to make use of tracing feature and RSS queues.
  • Page 295 Chapter XX. Classification and Filtering Enable Tracing as well as MultiRSSFilter [root@host~]# cxgbtool ethX reg 0x09800=0x33 ii. Setup a trace filter [root@host~]# echo tx0 snaplen=40 > /sys/kernel/debug/cxgb4/<bus_id>/trace0 iii. Configure the RSS Queue corresponding to trace0 filter configured above. Determine the RspQ ID of the queues by looking at Trace QType in /sys/kernel/debug/cxgb4/<bus- id>/sge_qinfo file.

  • Page 296: Nat Filtering

    Chapter XX. Classification and Filtering NAT Filtering T5/T6 adapters support offloading of stateless/static NAT functionality i.e. translating source/ destination L3 IP addresses, and source/destination L4 port numbers. This feature is supported with both LE-TCAM and Hash filters. This feature is only supported with filter action switch. Note Syntax: [root@host~]# cxgbtool ethX filter <index>...

  • Page 297: Xxi. Ovs Kernel Datapath Offload

    Chapter XXI. OVS Kernel Datapath Offload XXI. OVS Kernel Datapath Offload Chelsio Unified Wire for Linux...

  • Page 298: Introduction

    Chapter XXI. OVS Kernel Datapath Offload Introduction Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols. Chelsio’s T6/T5 Unified Wire solution can offload OVS datapath flow match entries and action processing onto Chelsio adapter for hardware acceleration of OVS datapath flow processing.

  • Page 299: Software Requirements

    Chapter XXI. OVS Kernel Datapath Offload 1.2. Software Requirements The Chelsio OVS driver has been developed to run on the following 64-bit Linux platforms: RHEL 7.5, 3.10.0-862.el7 RHEL 7.4, 3.10.0-693.el7 Kernel.org linux-4.9 (Minimum 4.9 kernel version supported is 4.9.13) Other kernel versions have not been tested and are not guaranteed to work. Chelsio Unified Wire for Linux...

  • Page 300: Software/Driver Installation

    Chapter XXI. OVS Kernel Datapath Offload Software/Driver Installation 2.1. Pre-requisites GCC 4.6+, Python 2.7+, Python-six, Autoconf 2.63+, Automake 1.10+, libtool 2.4+ packages should be installed. For the complete list of software required visit http://docs.openvswitch.org/en/latest/intro/install/general/ 2.2. Installation Change your current working directory to Chelsio Unified Wire package directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii.

  • Page 301: Software/Driver Configuration And Fine Tuning

    Chapter XXI. OVS Kernel Datapath Offload Software/Driver Configuration and Fine Tuning Supported Fields The following match fields are supported for offload: • Input Port • L2 Ethernet Type • L3 IP Protocol Type • L3 IPv4 address • L3 IPv6 address •...

  • Page 302: Configuring Ovs Machine

    Chapter XXI. OVS Kernel Datapath Offload 3.1. Configuring OVS Machine The following example explains the method to configure an OVS machine: OVS Switch Host A Host B eth2 eth3 *eth2 and eth3 are Chelsio interfaces. Please install Unified Wire with Hash Filter (T5/T6) configuration tuning option. ii.
  • Page 303 Chapter XXI. OVS Kernel Datapath Offload vi. Configure OVS [root@host~]# ovs-appctl exit [root@host~]# pkill -9 ovs [root@host~]# rm -rf /usr/local/etc/ovs-vswitchd.conf [root@host~]# rm -rf /usr/local/var/run/openvswitch/db.sock [root@host~]# rm -rf /usr/local/etc/openvswitch/conf.db [root@host~]# touch /usr/local/etc/ovs-vswitchd.conf [root@host~]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db <uwire_package>/src/openvswitch-x.x.x/vswitchd/vswitch.ovsschema [root@host~]# ovsdb-server /usr/local/etc/openvswitch/conf.db -- remote=punix:/usr/local/var/run/openvswitch/db.sock -- remote=db:Open_vSwitch,Open_vSwitch,manager_options --bootstrap-ca- cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --log-file...

  • Page 304: Creating Ovs Flows

    Chapter XXI. OVS Kernel Datapath Offload 3.2. Creating OVS flows It is mandatory to specify L2 Ethernet Type (dl_type) to offload OVS flows. There are two types of flows: • exact-match: Protocol and 4-tuple are mandatory to create an exact-match flow. ~0.5 million exact-match flows can be offloaded.
  • Page 305 Chapter XXI. OVS Kernel Datapath Offload • Exact-match flow to match 4-tuple IPv4 traffic and do NAT rewrite: [root@host~]# ovs-ofctl add-flow br0 dl_type=0x800,nw_proto=6,nw_src=10.1.1.66,tp_src=11000,nw_dst=10.1.1.58,tp_d st=21000,action=mod_nw_src=10.2.2.66,mod_tp_src=11005,mod_nw_dst:10.2.2.62,m od_tp_dst:12345,output:2 • Exact-match flow to match 4-tuple IPv6 traffic and do NAT rewrite: [root@host~]# ovs-ofctl add-flow br0 in_port=1,dl_type=0x86dd,nw_proto=6,ipv6_src=2000::66,tp_src=11000,ipv6_dst= 2000::58,tp_dst=11000,action=set_field:2001::66- \>ipv6_src,mod_tp_src=15000,output:2...
  • Page 306 Chapter XXI. OVS Kernel Datapath Offload • Strip VLAN tag and switch traffic: [root@host~]# ovs-ofctl add-flow br0 in_port=1,dl_type=0x800,action=strip_vlan,output:2 • Insert VLAN tag 100 and switch traffic: [root@host~]# ovs-ofctl -O OpenFlow11 add-flow br0 in_port=1,dl_type=0x800,action=push_vlan:0x8100,set_field:100- \>vlan_vid,output:2 • Modify VLAN tag 30 to 50 and switch traffic: [root@host~]# ovs-ofctl -O OpenFlow11 add-flow br0 in_port=1,dl_type=0x800,vlan_vid=30,action=mod_vlan_vid=50,output:2 If vlan_vid is not specieifed, the mod_vlan_vid tag will be added with a priority of 0.
  • Page 307 Chapter XXI. OVS Kernel Datapath Offload iii. Bring up Chelsio interfaces in promiscous mode: [root@host~]# ifconfig ethX <ip_address> promisc up [root@host~]# ifconfig ethY <ip_address> promisc up iv. Load Open vSwitch module: [root@host~]# modprobe openvswitch v. Configure OVS: [root@host~]# ovs-appctl exit [root@host~]# pkill -9 ovs [root@host~]# rm -rf /usr/local/etc/ovs-vswitchd.conf [root@host~]# rm -rf /usr/local/var/run/openvswitch/db.sock...

  • Page 308: Verifying Ovs Flow Dump

    Chapter XXI. OVS Kernel Datapath Offload Client Follow steps (i)-(vi) described in the previous section Server. ii. Set a rule to set VNI to 42 and send traffic: [root@host~]# ovs-ofctl add-flow br0 in_port=LOCAL,action=set_tunnel:0x2a,output:2 3.3. Verifying OVS Flow Dump OVS flow dump can be verified using: [root@host~]# ovs-ofctl dump-flows br0 Run traffic between hosts which matches the flow and verify if the n_packet counter is incrementing.
  • Page 309 Chapter XXI. OVS Kernel Datapath Offload iii. Create an entry in .bashrc which points to the extracted folder: export JAVA_HOME=<path>/jdk1.8.0_92 export PATH=$PATH:$JAVA_HOME iv. Logout & log back in. v. Download ODL controller pre-built zip package. vi. Unzip the package and change your working directory to opendaylight. vii.

  • Page 310: Software/Driver Uninstallation

    Chapter XXI. OVS Kernel Datapath Offload Software/Driver Uninstallation Change your working directory to Chelsio Unified Wire directory: [root@host~]# cd ChelsioUwire-x.x.x.x ii. Uninstall OVS driver: [root@host~]# make ovs_uninstall Chelsio Unified Wire for Linux...

  • Page 311: Xxii. Ring Backbone

    Chapter XXII. Ring Backbone XXII. Ring Backbone Chelsio Unified Wire for Linux...

  • Page 312: Introduction

    Chapter XXII. Ring Backbone Introduction Chelsio Ring Backbone utility is an easy to use tool developed to configure machines (with Chelsio adapters) in a ring backbone. This replaces the need for a ToR switch to connect a set of servers. 1.1.

  • Page 313: Ring Connectivity

    Chapter XXII. Ring Backbone Ubuntu 18.04.1, 4.15.0-29-generic Ubuntu 16.04.4, 4.4.0-116-generic Kernel.org linux-4.14.67 Kernel.org linux-4.9 (Minimum 4.9 kernel version supported is 4.9.13) Other versions have not been tested and are not guaranteed to work. 1.3. Ring Connectivity Connect the machines in ring backbone. Connect port 1 of one machine to the port 0 of next machine such that a closed ring is formed.

  • Page 314: Software/Driver Installation

    Chapter XXII. Ring Backbone Software/Driver Installation Install Unified Wire on all the machines in the ring backbone: Change your current working directory to Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x ii. Install the drivers, tools and libraries using the following command: [root@host~]# make install CONF=RING Chelsio Unified Wire for Linux...

  • Page 315: Software/Driver Configuration And Fine-Tuning

    Chapter XXII. Ring Backbone Software/Driver Configuration and Fine-tuning Configure ring backbone on all the machines using the below steps: Please ensure that all inbox drivers are unloaded before proceeding with unified Important wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Load network driver (cxgb4) with enable_ringbb module parameter.
  • Page 316 Chapter XXII. Ring Backbone If you want to use tunnel in ring configuration, apply filter as: [root@host~]# cxgbtool <port0_interface> configure_ringbb pf 4 vi <vf_id> mvi <mirror_vf_id> tunnel <tunnel_interface_name> man cxgbtool For more ring back bone configuration options, please run Note refer to the configure_ringbb section.

  • Page 317: Xxiii. Traffic Management

    Chapter XXIII. Traffic Management XXIII. Traffic Management Chelsio Unified Wire for Linux...

  • Page 318: Introduction

    Chapter XXIII. Traffic Management Introduction Traffic Management capabilities built-in to Chelsio adapters can shape transmit data traffic through the use of sophisticated queuing and scheduling algorithms built-in to the ASIC hardware which provides fine-grained software control over latency and bandwidth parameters such as packet rate and byte rate.

  • Page 319: Software Requirements

    Chapter XXIII. Traffic Management T520-BT T420-CR T440-CR T422-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX Only NIC driver supported 1.2. Software Requirements Linux Requirements Currently the Traffic Management feature is available for the following versions: RHEL 7.5, 3.10.0-862.el7 RHEL 7.5, 3.10.0-862.el7.ppc64le (POWER8 LE) RHEL 7.5, 4.14.0-49.el7a.aarch64 (ARM64) RHEL 7.4, 3.10.0-693.el7 RHEL 7.4, 3.10.0-693.el7.ppc64le (POWER8 LE)

  • Page 320: Software/Driver Loading

    Chapter XXIII. Traffic Management Software/Driver Loading Please ensure that all inbox drivers are unloaded before proceeding with Important unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Traffic Management can be performed on non-offloaded connections as well as on offloaded connections.

  • Page 321: Software/Driver Configuration And Fine-Tuning

    Chapter XXIII. Traffic Management Software/Driver Configuration and Fine-tuning 3.1. Traffic Management Rules Traffic Management supports the following types of scheduler hierarchy levels which can be configured using the cxgbtool utility: Class Rate Limiting ii. Class Weighted Round Robin iii. Channel Rate Limiting Class Rate Limiting This scheduler hierarchy level can be used to rate limit individual traffic classes or individual connections (flow) in a traffic class.
  • Page 322 Chapter XXIII. Traffic Management Class Weighted Round Robin Incoming traffic flows from various applications can be prioritized and provisioned using a weighted round-robin scheduling algorithm. Class weighted round robin can be configured using the following command: [root@host~]# cxgbtool <ethX> sched-class params type packet level cl-wrr channel <Channel No.>...

  • Page 323: Configuring Traffic Management

    Chapter XXIII. Traffic Management 3.2. Configuring Traffic Management For Non-offloaded connections Traffic Management of non-offloaded connections is a 2-step process. In the first step bind connections to indicated NIC TX queue using tc utility from iproute2-3.9.0 package. In the second step bind the indicated NIC TX queue to the specified TC Scheduler class using the cxgbtool utility.
  • Page 324 Chapter XXIII. Traffic Management For Offloaded connections Traffic Management of offloaded connections can be configured either by applying COP policies that associate offloaded connections to classes or by modifying the application. Both the methods have been described below: Applying COP policy Load the TOE driver and bring up the interface: [root@host~]# modprobe t4_tom [root@host~]# ifconfig ethX up...
  • Page 325 Chapter XXIII. Traffic Management Modifying the application The application can also be modified to associate connections to scheduling classes. Follow the steps mentioned below: Determine the TCP socket file descriptor in the application through which data is sent. ii. Declare and initialize a variable in the application: int cl=1;...

  • Page 326: Usage

    Chapter XXIII. Traffic Management Usage 4.1. Non-Offloaded Connections The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for Non-offload connections: Load the network driver and bring up the interface: [root@host~]# modprobe cxgb4 [root@host~]# ifconfig eth0 up Bind connections with destination IP address 192.168.5.3 to NIC TX queue 3...

  • Page 327: Offloaded Connections With Modified Application

    Chapter XXIII. Traffic Management ii. Create a new policy file (say new_policy_file) and add the following line to associate connections with the given scheduling class.: src host 102.1.1.1 => offload class 0 iii. Compile the policy file using COP [root@host~]# cop -d –o <output_policy_file> <new_policy_file> iv.

  • Page 328: Software/Driver Unloading

    Chapter XXIII. Traffic Management Software/Driver Unloading Reboot the system to unload the driver. To unload without rebooting, refer Unloading the TOE driver section of Network (NIC/TOE) chapter. Chelsio Unified Wire for Linux...

  • Page 329: Xxiv. Dpdk Driver

    Chapter XXIV. DPDK Driver XXIV. DPDK Driver Chelsio Unified Wire for Linux...

  • Page 330: Introduction

    Chapter XXIV. DPDK Driver Introduction Chelsio Data Plane Development Kit (DPDK) driver package is a collection of data plane libraries and NIC drivers optimized for running in the Linux user space to boost packet processing. The driver has support for: Multiple queues for Tx and Rx Receive Side Scaling (RSS) VLAN filtering...

  • Page 331: Software Requirements

    Chapter XXIV. DPDK Driver 1.2. Software Requirements The Chelsio DPDK driver has been developed to run on 64-bit Linux platforms. Following is the list of supported distributions: Linux Distribution Driver/Software RHEL 7.5, 3.10.0-862.el7 UIO, VFIO RHEL 7.4, 3.10.0-693.el7 Other kernel versions have not been tested and are not guaranteed to work. Chelsio Unified Wire for Linux...

  • Page 332: Software/Driver Installation

    Chapter XXIV. DPDK Driver Software/Driver Installation 2.1. Pre-requisites flex, bison, byacc, patch, patchutils, autoconf, automake and rpm-build packages must be present in the machine (required for libpcap installation). 2.2. Installation Please ensure that any existing version of DPDK driver is uninstalled before Important proceeding.

  • Page 333: Flashing Firmware Configuration File

    Chapter XXIV. DPDK Driver Flashing Firmware Configuration File If not done already, load the kernel mode NIC driver (cxgb4): [root@host~]# modprobe cxgb4 ii. Flash the firmware configuration file (for T6 adapters): [root@host~]# cxgbtool <iface> loadcfg /lib/firmware/cxgb4/t6-config.txt For T5 adapters, use t5-config.txt. Note Chelsio Unified Wire for Linux...

  • Page 334: Software/Driver Loading

    Chapter XXIV. DPDK Driver Software/Driver Loading UIO Support Follow the steps mentioned below to load DPDK driver with UIO support: Disable Intel iommu by adding below entry to kernel command line in grub/grub2 menu: intel_iommu=off ii. Reboot the system for changes to take effect. iii.

  • Page 335: Software/Driver Configuration And Fine Tuning

    Chapter XXIV. DPDK Driver Software/Driver Configuration and Fine Tuning 5.1. Huge Pages Using script Run the res_hugepages.sh shell script (copied to /sbin during installation): [root@host~]# res_hugepages.sh Manual Mount hugetlbfs: [root@host~]# mkdir -p /mnt/huge [root@host~]# mount -t hugetlbfs nodev /mnt/huge ii. Reserve Huge Page memory manually: •...

  • Page 336: Binding Network Ports

    Chapter XXIV. DPDK Driver 5.2. Binding Network Ports PF4 Interfaces Before proceeding, unload all Chelsio inbox and outbox kernel mode drivers Important using the following commands. [root@host~]# rmmod cxgb4 [root@host~]# rmmod csiostor Execute the following commands to bind PF4 network ports to DPDK environment. •...
  • Page 337 Chapter XXIV. DPDK Driver VF Interfaces Refer Instantiate Virtual Functions section and instantiate the required number of VFs. ii. Unload the kernel mode cxgb4vf driver (if loaded) [root@host~]# rmmod cxgb4vf iii. The VF can be assigned to a Virtual Machine (VM) or can be used in the Hypervisor (Host). Execute the following commands on the VM/Host to bind the VF to DPDK environment: •...

  • Page 338: Unbinding Network Ports

    Chapter XXIV. DPDK Driver 5.3. Unbinding Network Ports Run the following command to unbind network ports from DPDK environment: [root@host~]# dpdk-devbind.py --unbind <PCI-ID-PF4/VF-PCI-ID> ii. Run the following command to verify: [root@host~]# dpdk-devbind.py --status Example: 5.4. Performance Tuning In order to auto tune the system for best performance, Chelsio recommends: •...

  • Page 339: Running Dpdk Test Applications

    Chapter XXIV. DPDK Driver Running DPDK Test Applications 6.1. Testpmd application The Testpmd application is provided as a part of the Chelsio DPDK driver package, and can be used to test the DPDK in a packet forwarding mode and also to access NIC hardware features such as Flow Director.
  • Page 340 Chapter XXIV. DPDK Driver Flow Control Flow control pause Tx/Rx is disabled by default and can be enabled via Testpmd as follows: testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1 The PEER should be advertising the PAUSE capabilities for the Tx and Rx pause Note...

  • Page 341: Pktgen Application

    Chapter XXIV. DPDK Driver • Testpmd This method involves using the mtu command, which changes the MTU of an individual port without having to stop the selected port. To configure each port individually, run the mtu command as follows: testpmd> port config mtu 0 9000 testpmd>...
  • Page 342 Chapter XXIV. DPDK Driver Examples • Running single Tx and Rx queue traffic Configure Pktgen tool to use single Tx and Rx queue, mapping core 1 to Rx and core 2 to Tx queue on port 0: [root@host~]# pktgen -c f -n 4 -- -T -P -m "[1:2].0" –N Pktgen created by Keith Wiles -- >>>...
  • Page 343 Chapter XXIV. DPDK Driver ii. Set packet size to 64B on all ports Pktgen> set all size 64 iii. Start Tx traffic on port 0 Pktgen> start 0 iv. Stop Tx traffic on port 0 Pktgen> stop 0 • Running multiple Tx and Rx queue traffic Configure Pktgen tool to use 4 Tx and Rx queues, mapping cores 1-4 to Rx and 5-8 to Tx queues, on port 0: [root@host~]# pktgen -c 3ff -n 4 -- -T -P -m "[1-4:5-8].0"...

  • Page 344: Runtime Options

    Chapter XXIV. DPDK Driver ii. Set packet size to 9018B on all ports Pktgen> set all size 9018 iii. Start Tx traffic on port 0 Pktgen> start 0 iv. Stop Tx traffic on port 0 Pktgen> stop 0 6.3. Runtime Options The following devargs options can be used while running the applications.

  • Page 345: Software/Driver Unloading

    Chapter XXIV. DPDK Driver Software/Driver Unloading UIO Support Run the following commands to unload DPDK driver with UIO support: [root@host~]# rmmod igb_uio [root@host~]# rmmod uio If unloading uio module reports an error, unload the following dependent modules and try again: [root@host~]# rmmod bnx2fc [root@host~]# rmmod bnx2i [root@host~]# rmmod cnic...

  • Page 346: Software/Driver Uninstallation

    Chapter XXIV. DPDK Driver Software/Driver Uninstallation Change your working directory to Chelsio Unified Wire directory: [root@host~]# cd ChelsioUwire-x.x.x.x Uninstall the DPDK driver using the following command: [root@host~]# make uninstall Chelsio Unified Wire for Linux...

  • Page 347: Xxv. Unified Boot

    Chapter XXV. Unified Boot XXV. Unified Boot Chelsio Unified Wire for Linux...

  • Page 348: Introduction

    Chapter XXV. Unified Boot Introduction PXE is short for Preboot eXecution Environment and is used for booting computers over an Ethernet network using a Network Interface Card (NIC). FCoE SAN boot process involves installation of an operating system to an FC/FCoE disk and then booting from it. iSCSI SAN boot process involves installation of an operating system to an iSCSI disk and then booting from it.

  • Page 349: Software Requirements

    Chapter XXV. Unified Boot * If system BIOS version is lower than 1.5 and both Legacy and uEFI are enabled, system will hang during POST. Please upgrade the BIOS version to 1.5 or higher to avoid this issue. Supported Switches Following is the list of network switches supported by Chelsio Unified Boot software: Cisco Nexus 5010 with 5.1(3) N1 (1a) firmware.

  • Page 350: Pre-Requisites

    Chapter XXV. Unified Boot The Chelsio Driver Update Disk driver has been developed to run on 64-bit Linux platforms. Following is the list of Drivers/Software and supported Linux distributions: Linux Distribution Driver/Software (DUDs) RHEL 7.5, 3.10.0-862.el7 PXE, FCoE, iSCSI RHEL 7.4, 3.10.0-693.el7 RHEL 6.9, 2.6.32-696.el6 PXE, iSCSI SLES 15, 4.12.14-23-default...

  • Page 351: Secure Boot

    Chapter XXV. Unified Boot Secure Boot Secure Boot, a high-performance computing software solution is a method to restrict which binaries can be executed to boot the system. With Secure Boot, the system BIOS will only allow the execution of boot loaders that carry the cryptographic signature of trusted entities. In other words, anything run in the BIOS must be “signed”...

  • Page 352: Flashing Firmware And Option Rom

    Chapter XXV. Unified Boot Flashing firmware and option ROM Depending on the boot mode selected, Chelsio Unified Boot provides two methods to flash firmware and option ROM onto Chelsio adapters: Flash utility cfut4 for Legacy mode and HII for uEFI mode. On HP machines, you can also use Firmware Manager Protocol (FMP), in addition to HII.

  • Page 353: Legacy

    Chapter XXV. Unified Boot 3.2. Legacy In BIOS, configure the system having Chelsio CNA to boot in Legacy mode. Once the system boots from the USB flash drive, change your working directory to CHELSIO directory: C:\>cd CHELSIO iii. Run the following command to list all Chelsio CNAs present on the system. The list displays a unique index for each CNA found.
  • Page 354 Chapter XXV. Unified Boot Here, is the CNA index found in step iii (0 in this case) Delete any previous firmware using the following command: C:\CHELSIO>cfut4 -d <idx> -xh –xf vi. Delete any previous Option ROM settings: C:\CHELSIO>cfut4 -d <idx> -xc vii.
  • Page 355 Chapter XXV. Unified Boot viii. Flash the unified option ROM onto the Chelsio CNA using the following command: C:\CHELSIO>cfut4 -d <idx> -ub cubt4.bin cubt4.bin Here, is the unified option ROM image file present in the CHELSIO directory. ix. Reboot the system for changes to take effect. Chelsio Unified Wire for Linux...

  • Page 356: Uefi

    Chapter XXV. Unified Boot To configure the base MAC address (optional), use the below command: C:\CHELSIO>cfut4 -d <idx> -um <Hex MAC Address> Example: C:\CHELSIO>cfut4 -d 0 -um 000743000123 3.3. uEFI Reboot the system and go into BIOS setup. Disable Secure Boot. iii.
  • Page 357 Chapter XXV. Unified Boot iv. Boot to EFI Shell. Chelsio Unified Wire for Linux...
  • Page 358 Chapter XXV. Unified Boot drivers Issue command to determine if Chelsio uEFI driver is loaded. If the driver is loaded (as shown in the image below), continue to step (vi) If the driver is not loaded, load the uEFI driver (ChelsioUD.efi) found in the CHELSIO directory, and try again.
  • Page 359 Chapter XXV. Unified Boot vii. Chelsio HII should be listed as Chelsio T5/T6. Highlight it and press [Enter]. If Chelsio T5/T6 is not listed, please ensure that Chelsio uEFI driver is loaded Note correctly as mentioned here in the Flashing Firmware and Option ROM section. Chelsio Unified Wire for Linux...
  • Page 360 Chapter XXV. Unified Boot viii. Highlight the Chelsio adapter to be configured and press [Enter]. Chelsio Unified Wire for Linux...
  • Page 361 Chapter XXV. Unified Boot ix. Highlight Flash Utility and press [Enter]. Erase or update firmware using the methods explained below: • Erase existing firmware a. Select [Erase] as Flash Operation b. Select [FW File] as Flash File Type c. Select Update/Erase d.

  • Page 362: Hp Firmware Management Protocol (Fmp)

    Chapter XXV. Unified Boot 3.4. HP Firmware Management Protocol (FMP) Enabling FMP For HP machines with Unified Boot v1.0.0.99 and below installed, FMP must be manually enabled. For v2.0.0.1 and above, you can skip to the next sub-section for instructions on how to upgrade firmware using FMP.
  • Page 363 Chapter XXV. Unified Boot Upgrading Firmware ▪ Using CLI Use the adapter’s device name to update the firmware: FS1:\CHELSIO\> fwupdate -d <device_name> -f cubt4.bin Example: ii. Reboot machine for changes to take effect. ▪ Using FMP Reboot system and press F9 to access System Utilities Chelsio Unified Wire for Linux...
  • Page 364 Chapter XXV. Unified Boot → → ii. Go to Embedded Applications Firmware Update Chelsio T6 Controller iii. Highlight Select a firmware file option and hit [Enter]. iv. Select the USB flash drive which contains the latest Option ROM and hit [Enter]. v.
  • Page 365 Chapter XXV. Unified Boot vi. Select Start firmware update and hit [Enter]. vii. After Firmware update completed successfully prompt appears, reboot the machine for changes to take effect. Chelsio Unified Wire for Linux...

  • Page 366: Default Option Rom Settings

    Chapter XXV. Unified Boot 3.5. Default Option ROM Settings If you wish to restore option ROM settings to their default values, i.e., PXE enabled, iSCSI and FCoE disabled, use any of the methods mentioned below: Using Option ROM (boot level) •...
  • Page 367 Chapter XXV. Unified Boot Using cxgbtool (OS level) Change your working directory to OptionROM directory and use cxgbtool to flash the default boot configuration onto the adapter: [root@host~]# cd <driver_pakage>/Uboot/OptionROM/ [root@host~]# cxgbtool <ethX> loadboot-cfg bootcfg Chelsio Unified Wire for Linux...

  • Page 368: Configuring Pxe Server

    • TFTP Server PXE server configuration steps for different operating systems can be found on following links: Chelsio Communications does not take any responsibility regarding contents given Note in below mentioned links. They are provided for example purposes only. Linux •...

  • Page 369: Pxe Boot Process

    Chapter XXV. Unified Boot PXE Boot Process Before proceeding, please ensure that the Chelsio CNA has been flashed with the provided firmware and option ROM (See Flashing Firmware and option ROM). 5.1. Legacy PXE Boot After configuring the PXE server, make sure the PXE server works. Then reboot the client machine.
  • Page 370 Chapter XXV. Unified Boot vi. Choose PXE from the list to configure. Hit [Enter]. vii. Use the arrow keys to highlight the appropriate function among the supported NIC functions and hit [Enter] to select. viii. Enable NIC function bios if not already enabled. ix.
  • Page 371 Chapter XXV. Unified Boot Hit [F10] or [Esc] and then [Y] to save configuration changes. xi. Reboot the system. xii. Allow the Chelsio option ROM to initialize and setup PXE devices. DO NOT PRESS ALT-S to skip Chelsio option ROM. xiii.

  • Page 372: Uefi Pxe Boot

    Chapter XXV. Unified Boot 5.2. uEFI PXE Boot • Only uEFI v2.3.1, v2.4 and v2.5 supported. Important • Any other uEFI version is NOT SUPPORTED and may render your system unusable. This section describes the method to configure and use Chelsio uEFI PXE interfaces. Reboot the system and go into the BIOS setup.
  • Page 373 Chapter XXV. Unified Boot vi. Enable adapter BIOS if not already enabled. It is highly recommended that you use the Save Changes option every time a Note parameter/option is changed. vii. Select Chelsio Protocol Selection and press [Enter]. Chelsio Unified Wire for Linux...
  • Page 374 Chapter XXV. Unified Boot viii. Select PXE and press [Enter]. ix. Choose the boot port to try PXE boot. It is recommended to enable only those functions and ports which are going to be used. Please note that enabling PXE Function 0 will enable port 0 for PXE, enabling PXE Function 1 will enable port 1 and so on, for NIC function.
  • Page 375 Chapter XXV. Unified Boot xi. Reboot the system and in BIOS, choose any of the available Chelsio PXE devices for PXE boot. xii. Reboot and hit [F12] key when prompted to start PXE boot. Chelsio Unified Wire for Linux...

  • Page 376: Fcoe Boot Process

    Chapter XXV. Unified Boot FCoE Boot Process Before proceeding, please ensure that the Chelsio CNA has been flashed with the provided firmware and option ROM (See Flashing firmware and option ROM). 6.1. Legacy FCoE Boot Reboot the system. Press [Alt+C] when the message to configure Chelsio adapters appears on the screen. iii.
  • Page 377 Chapter XXV. Unified Boot vi. Choose FCoE from the list to configure and hit [Enter]. vii. Choose the first option, Configure function parameters, from the list of parameter type and hit [Enter]. viii. Enable FCoE BIOS if not already enabled. Chelsio Unified Wire for Linux...
  • Page 378 Chapter XXV. Unified Boot ix. Choose the order of the ports to discover FCoE targets. Set discovery timeout to a suitable value. Recommended value is >= 30. xi. Hit [F10] or [Esc] and then [Y] to save the configuration. Chelsio Unified Wire for Linux...
  • Page 379 Chapter XXV. Unified Boot xii. Choose Configure boot parameters. xiii. Select the first boot device and hit [Enter] to discover FC/FCoE targets connected to the switch. Wait till all reachable targets are discovered. xiv. List of discovered targets will be displayed. Highlight a target using the arrow keys and hit [Enter] to select.
  • Page 380 Chapter XXV. Unified Boot xv. From the list of LUNs displayed for the selected target, choose one on which operating system has to be installed. Hit [Enter]. xvi. Hit [F10] or [Esc] and then [Y] to save the configuration. xvii. Reboot the machine. Chelsio Unified Wire for Linux...

  • Page 381: Uefi Fcoe Boot

    Chapter XXV. Unified Boot xviii. During POST, allow the Chelsio Option ROM to discover FCoE targets. xix. Enter BIOS setup and choose FCoE disk discovered via Chelsio adapter as the first boot device. xx. Reboot and boot from the FCoE disk or install the required OS using PXE. 6.2.
  • Page 382 Chapter XXV. Unified Boot If Chelsio T5/T6 is not listed, please ensure that Chelsio uEFI driver is loaded Note correctly as mentioned here in the Flashing Firmware and Option ROM section. iv. Select the Chelsio adapter to be configured and press [Enter]. Select Configuration Utility and press [Enter].
  • Page 383 Chapter XXV. Unified Boot vi. Enable adapter BIOS if not already enabled. It is highly recommended that you use the Save Changes option every time a Note parameter/option is changed. vii. Select Chelsio Protocol Selection and press [Enter]. viii. Select FCoE and press [Enter]. ix.
  • Page 384 Chapter XXV. Unified Boot Set discovery timeout to a suitable value. Recommended value is >= 30 xi. Choose the order of the ports to discover FCoE targets. Chelsio Unified Wire for Linux...
  • Page 385 Chapter XXV. Unified Boot xii. Under the first boot device, select Discover Target and press [Enter] to discover FC/FCoE targets connected to the switch. Wait till all reachable targets are discovered. xiii. List of discovered targets will be displayed. Highlight a target to select it and hit [Enter]. xiv.
  • Page 386 Chapter XXV. Unified Boot xv. Select Save Changes and press [Enter]. xvi. Reboot the system for changes to take effect. xvii. The discovered LUN should appear in the Boot Configuration section and system BIOS section. xviii. Select the LUN as the first boot device and exit from BIOS. xix.

  • Page 387: Iscsi Boot Process

    Chapter XXV. Unified Boot iSCSI Boot Process Before proceeding, please ensure that the Chelsio CNA has been flashed with the provided firmware and option ROM (See Flashing firmware and option ROM). 7.1. Legacy iSCSI Boot Reboot the system. Press [Alt+C] when the message to configure Chelsio adapters appears on the screen. iii.
  • Page 388 Chapter XXV. Unified Boot vi. Choose iSCSI from the list to configure and hit [Enter]. vii. Choose the first option, Configure Function Parameters, from the list of parameter type and hit [Enter]. viii. Enable iSCSI BIOS if not already enabled. iBFT (iSCSI Boot Firmware Table) will be selected by default.
  • Page 389 Chapter XXV. Unified Boot ix. Choose the order of the ports to discover iSCSI targets. Set discovery timeout to a suitable value. Recommended value is >= 30. xi. Hit [Esc] and then [Y] to save the configuration. Chelsio Unified Wire for Linux...
  • Page 390 Chapter XXV. Unified Boot xii. Go back and choose Configure Initiator Parameters to configure initiator related properties. xiii. Initiator properties like IQN, Header Digest, Data Digest, etc. will be displayed. Change the values appropriately or continue with the default values. Hit [F10] to save. xiv.
  • Page 391 Chapter XXV. Unified Boot xv. Enable CHAP authentication by selecting ONE-WAY or MUTUAL in the CHAP Policy field. Next, choose the CHAP method. Finally, provide Initiator and Target CHAP credentials as per the authentication method selected. Hit [F10] to save. xvi.
  • Page 392 Chapter XXV. Unified Boot xviii. Select Yes in the Enable DHCP field to configure port using DHCP or No to manually configure the port. Hit [F10] to save. xix. Go back and choose Configure Target Parameters to configure iSCSI target related properties.
  • Page 393 Chapter XXV. Unified Boot xxi. Go back and choose Discover iSCSI Target (s) to connect to a target. xxii. Select the portal group on which iSCSI service is provided by the target. xxiii. A list of available targets will be displayed. Select the target you wish to connect to and hit [Enter].
  • Page 394 Chapter XXV. Unified Boot xxiv. A list of LUNs configured on the selected target will be displayed. Select the LUN you wish to connect to and hit [Enter]. xxv. Hit [Esc] and then [Y] to save the configuration. xxvi. Reboot the machine. xxvii.

  • Page 395: Uefi Iscsi Boot

    Chapter XXV. Unified Boot xxviii. Enter BIOS setup and choose iSCSI target LUN discovered via Chelsio adapter as the first boot device. xxix. Reboot and boot from the iSCSI Target LUN or install the required OS using PXE. 7.2. uEFI iSCSI Boot •...
  • Page 396 Chapter XXV. Unified Boot If Chelsio T5/T6 is not listed, please ensure that Chelsio uEFI driver is loaded Note correctly as mentioned here in the Flashing Firmware and Option ROM section. iv. Select the Chelsio adapter to be configured and press [Enter]. Select Configuration Utility and press [Enter].
  • Page 397 Chapter XXV. Unified Boot vi. Enable adapter BIOS if not already enabled. It is highly recommended that you use the Save Changes option every time a Note parameter/option is changed. vii. Select Chelsio Protocol Selection and press [Enter]. viii. Select iSCSI and press [Enter]. ix.
  • Page 398 Chapter XXV. Unified Boot Set discovery timeout to a suitable value. Recommended value is >= 30 xi. Choose the order of the ports to discover iSCSI targets. Chelsio Unified Wire for Linux...
  • Page 399 Chapter XXV. Unified Boot xii. Under Initiator Parameters, iSCSI Initiaitor properties like IQN, Header Digest, Data Digest, etc will be displayed. Change the values appropriately or continue with the default values. xiii. Under the first port, select Enable DHCP field, hit [Enter] and select Enabled. This will configure port using DHCP.
  • Page 400 Chapter XXV. Unified Boot xiv. Under Target Parameters, select Enabled for the Boot Target via DHCP parameter to discover target using DHCP. To discover target via static IP, select Disabled and provide the target IP. Chelsio Unified Wire for Linux...
  • Page 401 Chapter XXV. Unified Boot xv. CHAP authentication is disabled by default. To enable and configure, highlight CHAP Policy and hit [Enter]. Select the policy type from the corresponding pop-up and hit [Enter] again. xvi. Provide Initiator and Target CHAP credentials as per the CHAP policy selected. Chelsio Unified Wire for Linux...
  • Page 402 Chapter XXV. Unified Boot xvii. Select Discover Target and press [Enter] to discover iSCSI targets connected to the switch. Wait till all reachable targets are discovered. xviii. A list of available targets will be displayed. Select the target you wish to connect to and hit [Enter].
  • Page 403 Chapter XXV. Unified Boot xx. Select Save Changes and press [Enter] xxi. Reboot the system for changes to take effect. xxii. The discovered LUN should appear in the Boot Configuration/ Boot Information section and system BIOS. xxiii. Select the LUN as the first boot device and exit from BIOS. xxiv.

  • Page 404: Creating Driver Update Disk (Dud)

    Chapter XXV. Unified Boot Creating Driver Update Disk (DUD) The following section describes the procedure to create Driver Update Disks for RHEL and SLES distributions for Chelsio adapters. 8.1. Creating DUD for RedHat Enterprise Linux If you haven’t done already, download the Chelsio Unified Wire driver package from Chelsio Download Center.
  • Page 405 Chapter XXV. Unified Boot iii. Insert a blank USB flash drive. iv. Format the USB drive [root@host~]# mkfs.vfat /dev/sda1 Depending on the distribution to be installed, copy the corresponding image file to the USB stick. For example, execute the following command for SLES 12 sp3. [root@host~]# dd if=/root/<driver_package>/Uboot/LinuxDUD/Chelsio- DriverUpdateDisk-SLES12sp3-x86_64-x.xx.x.x.img of=/dev/sda1 Chelsio Unified Wire for Linux...

  • Page 406: Os Installation

    Chapter XXV. Unified Boot OS Installation 9.1. Installation using Chelsio DUD This is the recommended method for installing Linux OS using Chelsio PXE boot. The Chelsio Driver Update Disk (DUD) has support for all the new adapters. Use Network Boot (PXE Boot) media to install the OS, and provide the Driver Update Disk as per the detailed instructions for each OS.
  • Page 407 Chapter XXV. Unified Boot RHEL 7.X Installation Please make sure that the USB drive with DUD image is inserted. Type e and then dd at the boot prompt for the installation media. The dd option specifies that you will be providing a Driver Update Disk during the installation.
  • Page 408 Chapter XXV. Unified Boot iv. Drivers provided in the DUD will be listed. Enter 1 to select Network driver (cxgb4), 2 to selct FCoE driver (csiostor) or 3 to select iSCSI Initiator (cxgb4i). Hit [Enter] v. To install more drivers, enter the index of the next driver and hit [Enter]. To start the loading process, enter c and hit [Enter].
  • Page 409 Chapter XXV. Unified Boot RHEL 6.X Installation Please make sure that the USB drive with DUD image is inserted. Press Tab and then type dd at the boot prompt for the installation media. The dd option specifies that you will be providing a Driver Update Disk during the installation.
  • Page 410 Chapter XXV. Unified Boot iv. Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. v.
  • Page 411 Chapter XXV. Unified Boot vi. Select the required language from the list. vii. Select the type of keyboard you have from the list. Chelsio Unified Wire for Linux...
  • Page 412 Chapter XXV. Unified Boot viii. In this step, you can choose the source which contains the OS installation ISO image. In this case, select “NFS directory”. ix. The Chelsio detwork devices will be displayed. Select the appropriate Chelsio NIC interface to proceed with installation.
  • Page 413 Chapter XXV. Unified Boot x. Here you can specify if you want to configure your network interfaces using DHCP or manually using IPv4. IPv6 is currently not supported. Hence disable IPv6 before proceeding. xi. Proceeding with the installation will get NFS/FTP/HTTP setup page. Here, provide NFS server details to proceed with the installation.
  • Page 414 Chapter XXV. Unified Boot SLES 15/SLES 12 SPx Installation Please make sure that the USB drive with DUD image is inserted. ii. Select the appropriate entry from the PXE menu and press [Enter]. Chelsio Unified Wire for Linux...
  • Page 415 Chapter XXV. Unified Boot iii. Select “Start Installation” and then “Start Installation or Update”. iv. Select “Network” as the source of medium to install the SLES Operating System. Chelsio Unified Wire for Linux...
  • Page 416 Chapter XXV. Unified Boot v. Select the desired Network protocol from the list presented. vi. Select the appropriate Chelsio interface from the list to proceed with installation. You can view the messages from the Linux kernel and drivers to determine the name of NIC interface by pressing [Alt] + [F3] or [Alt] + [F4].

  • Page 417: Installation On Fcoe Lun

    Chapter XXV. Unified Boot viii. Provide a valid NFS/FTP/HTTP/TFTP Server IP address to proceed. ix. Provide a valid directory path to the operating system to be installed. x. Proceed with the installation as usual. 9.2. Installation on FCoE LUN If you are installing using CD/DVD, please make sure that the USB drive with DUD image is inserted.
  • Page 418 Chapter XXV. Unified Boot RHEL 7.X Choose your installation language and click Continue. Click INSTALLATION DESTINATION under SYSTEM. Chelsio Unified Wire for Linux...
  • Page 419 Chapter XXV. Unified Boot iii. The discovered FC/FCoE LUNs will appear as local storage in the Local Standard Disks section. Select the LUN which was saved as boot device in system BIOS. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation.

  • Page 420: Installation On Iscsi Lun

    Chapter XXV. Unified Boot 9.3. Installation on iSCSI LUN If you are installing using CD/DVD, please make sure that the USB drive with DUD image is inserted. Also, change the boot priority to boot from CD/DVD in the BIOS setup. Insert the OS installation disc into your CD/DVD ROM.
  • Page 421 Chapter XXV. Unified Boot RHEL 7.X On the installer, welcome screen, choose your installation language and click Continue ii. Click Installation Destination under SYSTEM. Chelsio Unified Wire for Linux...
  • Page 422 Chapter XXV. Unified Boot iii. Click Add a disk iv. The discovered iSCSI LUNs will appear in the Search tab. Select it and click Done. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation.
  • Page 423 Chapter XXV. Unified Boot v. Under Other Storage Options, you can either chose to configure partition automatically or manually. Select the appropriate option and click Done. Then proceed with the installation as usual. RHEL 6.X Click Next when the graphical installer screen appears. ii.
  • Page 424 Chapter XXV. Unified Boot iii. The discovered LUNs will appear in the Basic Devices tab. Select the LUN which was saved as boot device in system BIOS and click Next. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation.
  • Page 425 Chapter XXV. Unified Boot SLES 15/SLES 12 SPx Installation Choose installation language and Keyboard layout type. Select the checkbox I Agree to the License terms and click Next. ii. Click Configure iSCSI Disks in the Disk Activation screen. Chelsio Unified Wire for Linux...
  • Page 426 Chapter XXV. Unified Boot iii. The discovered LUNs will appear in the Connected Targets tab. Select the LUN which was saved as boot device in system BIOS and click OK. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation.

  • Page 427: Xxvi. Appendix A

    Chapter XXVI. Appendix A XXVI. Appendix A Chelsio Unified Wire for Linux...

  • Page 428: Troubleshooting

    Chapter XXVI. Appendix A Troubleshooting Cannot bring up Chelsio interface Make sure you have created the corresponding network-script configuration file as stated in Cheslsio Unified Wire chapter (See Creating network-scripts). If the file does exist, make sure the structure and contents are correct. A sample is given in the Chelsio Unified Wire chapter (See Configuring network-scripts).
  • Page 429 Chapter XXVI. Appendix A Configuring Ethernet interfaces on Cisco switch Always configure Ethernet interfaces on Cisco switch in trunk mode. Binding VFC to MAC If you are binding the VFC to MAC address in case of Cisco Nexus switch, then make sure you make the Ethernet interface part of both Ethernet VLAN and FCoE VLAN.

  • Page 430: Chelsio End-User License Agreement (Eula)

    IF YOU DO NOT AGREE TO THE TERMS OF THIS AGREEMENT, CLICK THE "DO NOT ACCEPT" BUTTON TO TERMINATE THE INSTALLATION PROCESS. 1. License. Chelsio Communications, Inc. ("Chelsio") hereby grants you, the Licensee, and you hereby accept, a limited, non-exclusive, non-transferable license to install and use the Software with one or more Chelsio network adapters on a single server computer for use in communicating with one or more other computers over a network.
  • Page 431 Chapter XXVI. Appendix A material or workmanship under normal use and service for a period of thirty (30) days from the date of delivery to you. CHELSIO DOES NOT AND CANNOT WARRANT THE PERFORMANCE OR RESULTS YOU MAY OBTAIN BY USING THE SOFTWARE OR ANY PART THEREOF.
  • Page 432 Agreement. Should you have any questions concerning this Agreement, you may contact Chelsio by writing to: Chelsio Communications, Inc. 209 North Fair Oaks Avenue, Sunnyvale, CA 94085 U.S.A...

This manual is also suitable for:

Terminator 6

Table of Contents