1. Manuals
  2. Brands
  3. Chelsio Communications Manuals
  4. Computer Hardware
  5. Chelsio T5
  6. User manual

Chelsio Communications Chelsio T5 User Manual

Unified wire for linux
Hide thumbs Also See for Chelsio T5:
1
2
3
Table Of Contents
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
Table of Contents

Advertisement

Quick Links

See also: Installer and User Manual

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Chelsio T5 and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Chelsio Communications Chelsio T5

Summary of Contents for Chelsio Communications Chelsio T5

  • Page 2 For all support related questions please send email to support@chelsio.com Copyright © 2014.Chelsio Communications. All Rights Reserved. Chelsio ® is a registered trademark of Chelsio Communications. All other marks and names mentioned herein may be trademarks of their respective companies. Chelsio T5/T4 Unified Wire For Linux...
  • Page 3 1.0.0 12/08/2011 1.0.1 01/09/2013 1.0.2 01/27/2013 1.0.3 03/26/2013 1.0.4 04/12/2013 1.0.5 06/20/2013 1.0.6 08/17/2013 1.0.7 10/22/2013 1.0.8 03/08/2013 1.0.9 05/15/2013 1.1.0 07/26/2013 1.1.1 08/14/2013 1.1.2 12/06/2013 1.1.3 12/19/2013 1.1.4 03/13/2014 1.1.5 05/02/2014 1.1.6 06/30/2014 Chelsio T5/T4 Unified Wire For Linux...

  • Page 4: Table Of Contents

    Loading in TOE mode (with full offload support) Software/Driver Unloading 3.1. Unloading the NIC driver 3.2. Unloading the TOE driver Software/Driver Configuration and Fine-tuning 4.1. Instantiate Virtual Functions (SR-IOV) 4.2. Performance tuning III. VIRTUAL FUNCTION NETWORK (VNIC) Introduction 1.1. Hardware Requirements Chelsio T5/T4 Unified Wire For Linux...
  • Page 5 Software/Driver Configuration and Fine-tuning 4.1. Accelerating UDP Socket communications WD-TOE Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Running the application VII. ISCSI PDU OFFLOAD TARGET Introduction Chelsio T5/T4 Unified Wire For Linux...
  • Page 6 Auto login from cxgb4i initiator at OS bootup DATA CENTER BRIDGING (DCB) Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Configuring Cisco Nexus 5010 switch 4.2. Configuring the Brocade 8000 switch Chelsio T5/T4 Unified Wire For Linux...
  • Page 7 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Offloading TCP traffic over a bonded interface XIII. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) Introduction 1.1. Hardware Requirements 1.2. Software Requirements Chelsio T5/T4 Unified Wire For Linux...
  • Page 8 Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Starting ba server 4.2. Bypass API (CLI) XVII. WD SNIFFING AND TRACING Theory of Operation 1.1. Hardware Requirements 1.2. Software Requirements Installation and Usage Chelsio T5/T4 Unified Wire For Linux viii...
  • Page 9 Traffic Management Rules 4.2. Configuring Traffic Management Usage 5.1. Non-Offloaded Connections 5.2. Offloaded Connections 5.3. Offloaded Connections with Modified Application UNIFIED WIRE MANAGER (UM) Introduction 1.1. Features 1.2. Reference Architecture 1.3. Unified Wire Manager Components Chelsio T5/T4 Unified Wire For Linux...
  • Page 10 Uninstalling Management Client 8.3. Uninstalling Management Station XXI. UNIFIED BOOT Introduction 1.1. Hardware Requirements 1.2. Software Requirements Flashing firmware and option ROM 2.1. Using Flash Utility 2.2. Using cxgbtool Configuring PXE Server PXE boot process Chelsio T5/T4 Unified Wire For Linux...
  • Page 11 8.2. Installation on FCoE LUN 8.3. Installation on iSCSI LUN XXII. LUSTRE FILE SYSTEM Introduction 1.1. Hardware Requirements 1.2. Software Requirements Creating/Configuring Lustre File System XXIII. APPENDIX A Troubleshooting Chelsio End-User License Agreement (EULA) Chelsio T5/T4 Unified Wire For Linux...

  • Page 12: Chelsio Unified Wire

    Chapter I. Chelsio Unified Wire I. Chelsio Unified Wire Chelsio T5/T4 Unified Wire For Linux Page 12...

  • Page 13: Introduction

    Chapter I. Chelsio Unified Wire 1. Introduction Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip, single firmware cards provide enterprises and data centers with high performance solutions for various Network and Storage related requirements.

  • Page 14: Hardware Requirements

    1.2. Hardware Requirements The Chelsio Unified Wire software supports Chelsio T5 and T4 Series of Unified Wire Adapters. To know more about the list of adapters supported by each driver, please refer to their respective sections. 1.3.
  • Page 15 Management Station configuration files.  chsetup: The chsetup tool loads NIC,TOE and iWARP drivers, and creates WD- UDP configuration file.  chstatus: This utility provides status information on any Chelsio NIC in the system. Chelsio T5/T4 Unified Wire For Linux Page 15...
  • Page 16 Legacy Flash Utility (cfut4.exe), which can be used to flash Unified Boot Option ROM onto Chelsio's T5 and T4 Converged Network Adapters (CNAs). The LinuxDUD directory contains image (.img) files required to update drivers for Linux (RHEL and SLES) distributions. Chelsio T5/T4 Unified Wire For Linux Page 16...

  • Page 17: Hardware Installation

    07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR Unified Wire Storage Controller 07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire Storage Controller Chelsio T5/T4 Unified Wire For Linux Page 17...
  • Page 18 Storage Controller 03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000 For Chelsio T5/T4 adapters, the physical functions are currently assigned as: Physical functions 0 - 3: for the SR-IOV functions of the adapter Physical function 4: for all NIC functions of the adapter...
  • Page 19 However, for T5 40G and T420-BT adapters, the association of physical Ethernet ports and their corresponding network device names is opposite. For these adapters, the port nearest to the motherboard will appear as the first network interface. Chelsio T5/T4 Unified Wire For Linux Page 19...

  • Page 20: Software/Driver Installation

    T5 High Capacity Configures T5 adapters to establish a large NIC/TOE,WD-UDP,WD-TOE, number of WD-UDP connections. UM(Agent,Client,WebGUI) Configures T5 adapters to create more T5 Hash Filter* NIC,Filtering, UM(Agent,Client,WebGUI) filters. * Supported only on T5 adapters. Chelsio T5/T4 Unified Wire For Linux Page 20...

  • Page 21: Pre-Requisites

    GUI installer: [root@host]# ./install.py iv. If Dialog utility is present, you can skip to step (v). If not, press ‘y’ to install it when the installer prompts for input. Chelsio T5/T4 Unified Wire For Linux Page 21...
  • Page 22 Select Enable IPv6-Offload to install drivers with IPv6 Offload support or Disable IPv6- offload to continue installation without IPv6 offload support. vii. Select the required T5/T4 configuration tuning option: The tuning options may vary depending on the Linux distribution. Note Chelsio T5/T4 Unified Wire For Linux Page 22...
  • Page 23 Retain IRQ balance daemon: Do not disable IRQ balance daemon. c. TX-Coalasce: Write tx_coal=2 to modprobe.d/conf. For more information on the Performance tuning options, please refer to Note Software/Driver Configuration and Fine-tuning section of the Network (NIC/TOE) chapter. Chelsio T5/T4 Unified Wire For Linux Page 23...
  • Page 24 To install OFED-3.5-2 choose the Install-OFED option. To install a different version, select Choose-OFED-Version and then select the appropriate version. To skip this step, select Skip-OFED. OFED is currently not supported on RHEL 6.5 Note xi. The selected components will now be installed: Chelsio T5/T4 Unified Wire For Linux Page 24...
  • Page 25 Select “Yes” to exit the installer or “No” to go back. xv. Reboot your machine for changes to take effect. Press Esc or Ctrl+C to exit the installer at any point of time. Note Chelsio T5/T4 Unified Wire For Linux Page 25...
  • Page 26 Follow steps (xv) to (xvii) mentioned in the previous section. 3.2.2. CLI mode (without Dialog utility) If your system does not have Dialog or you choose not to install it, follow the steps mentioned below to install the Unified Wire package: Chelsio T5/T4 Unified Wire For Linux Page 26...
  • Page 27 Please make sure that you have enabled password less authentication with ssh Important on the peer nodes for this feature to work. Chelsio’s Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a single command. Follow the procedure mentioned below: Chelsio T5/T4 Unified Wire For Linux Page 27...
  • Page 28 Change your current working directory to Chelsio Unified Wire package directory and build the source using : [root@host]# make iv. Install the drivers, tools and libraries using the following command: [root@host]# make install Chelsio T5/T4 Unified Wire For Linux Page 28...
  • Page 29 279.2.1.el6 kernel, run the following commands: [root@host]# make UNAME_R=2.6.32-279.el6 [root@host]# make UNAME_R=2.6.32-279.el6 install To view the list of the supported kernel versions, run the following command: [root@host]# make list_kernels Reboot your machine for changes to take effect. Chelsio T5/T4 Unified Wire For Linux Page 29...
  • Page 30 To build and install all drivers without IPv6 support: [root@host]# make ipv6_disable=1 [root@host]# make ipv6_disable=1 install The above step will not install Bypass and WD-TOE drivers.  To build and install Bypass driver: [root@host]# make bypass [root@host]# make bypass_install Chelsio T5/T4 Unified Wire For Linux Page 30...
  • Page 31  The default T5/T4 configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host]# make CONF=<T5/T4 configuration> <Build Target> [root@host]# make CONF=<T5/T4 configuration> <Install Target> Chelsio T5/T4 Unified Wire For Linux Page 31...

  • Page 32: Installing Chelsio Unified Wire From Rpm

    2. Install NIC and TOE drivers: Install only the Network and TOE drivers. 3. Install bypass drivers and tools: Install only the bypass drivers and tools. 4. Install WD drivers and library: Install only Wire Direct drivers and libraries. 5. EXIT: Exit the installer. Chelsio T5/T4 Unified Wire For Linux Page 32...
  • Page 33 : bypass drivers and tools udpso : UDP segmentation offload capable NIC and TOE drivers only : Wire Direct drivers and libraries only The Installation options may vary depending on the Linux distribution. Note Chelsio T5/T4 Unified Wire For Linux Page 33...
  • Page 34 [root@host]# ./install.py -C -m <machinefilename> -i <nic_toe/all/bypass/udpso/wd> -c <T5/T4 configuration> -o Here, -o parameter will install OFED and Chelsio drivers built against OFED. iii. Reboot your machine for changes to take effect. Chelsio T5/T4 Unified Wire For Linux Page 34...

  • Page 35: Firmware Update

    The kernel must be configured to enable userspace firmware loading support: Device Drivers -> Generic Driver Options -> Userspace firmware loading support The firmware version can be verified using ethtool: [root@host]# ethtool -i <iface> Chelsio T5/T4 Unified Wire For Linux Page 35...

  • Page 36: Software/Driver Uninstallation

    GUI installer: [root@host]# ./install.py ii. Select “uninstall” , Under “Choose an action” iii. Select “all” to uninstall all the installed drivers, libraries and tools or select “custom” to remove specific components. Chelsio T5/T4 Unified Wire For Linux Page 36...
  • Page 37 The selected components will now be uninstalled. v. After successful uninstalltion, summary of the uninstalled components will be displayed. vi. Select “View log” to view uninstallation log or “Exit” to continue. Chelsio T5/T4 Unified Wire For Linux Page 37...
  • Page 38 4.1.3.1. iWARP driver uninstallation on Cluster nodes To uninstal iWARP drivers on multiple Cluster nodes with a single command, run the following command: [root@host]# ./install.py -C -m <machinefilename> -u all Chelsio T5/T4 Unified Wire For Linux Page 38...
  • Page 39 To uninstall iWARP driver: [root@host]# make iwarp_uninstall  To uninstall Bypass driver: [root@host]# make bypass_uninstall  To uninstall UDP Segmentation Offload driver: [root@host]# make udp_offload_uninstall  To uninstall WD-TOE driver: [root@host]# make wdtoe_uninstall Chelsio T5/T4 Unified Wire For Linux Page 39...

  • Page 40: Uninstalling Chelsio Unified Wire From Rpm

    [root@host]# ./uninstall.py –h by typing for more information. Uninstalling Unified Wire package will not uninstall Unified Wire Manager. Refer Note to the Unified Wire Manager (UM) chapter to remove the software manually (Click here). Chelsio T5/T4 Unified Wire For Linux Page 40...
  • Page 41 To uninstal iWARP drivers on multiple Cluster nodes with a single command, run the following: [root@host] # ./install.py -C -m <machinefilename> -u The above command will remove Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. Chelsio T5/T4 Unified Wire For Linux Page 41...

  • Page 42: Configuring Chelsio Network Interfaces

    5.1. Configuring 40G adapters Chelsio T5 40G adapters can be configured in the following two modes: 2X40Gbps: This is the default mode of operation where each port functions as 40Gbps link. ii. 4X10Gbps: In this mode, port 0 functions as 4 10Gbps links and port 1 is disabled.

  • Page 43: Configuring Network-Scripts

    On earlier versions of RHEL the NETMASK attribute is named IPMASK. Make sure Note you are using the right attribute name. In the case of DHCP addressing the last two lines should be removed and BOOTPROTO="static" BOOTPROTO="dhcp" should be changed to Chelsio T5/T4 Unified Wire For Linux Page 43...

  • Page 44: Creating Network-Scripts

    Note that it is of course not compulsory to create a configuration file for every interface if you are not planning to use them all. Chelsio T5/T4 Unified Wire For Linux Page 44...

  • Page 45: Checking Link

    First, bring up the interface you want to test using ifup eth1 You should now be able to ping any other machine from your network provided it has ping response enabled. Chelsio T5/T4 Unified Wire For Linux Page 45...

  • Page 46: Software/Driver Update

    For any distribution specific problems, please check README and Release Notes included in the release for possible workaround. Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various software/drivers. You can also subscribe to our newsletter for the latest software updates. Chelsio T5/T4 Unified Wire For Linux Page 46...

  • Page 47: Network (Nic/Toe)

    Chapter II. Network (NIC/TOE) II. Network (NIC/TOE) Chelsio T5/T4 Unified Wire For Linux Page 47...

  • Page 48: Introduction

    The following are the currently shipping Chelsio Adapters that are compatible with Chelsio Network driver: T502-BT T580-CR T580-SO-CR* T580-LP-CR T520-LL-CR T520-SO-CR* T520-CR T522-CR T540-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX *Only NIC driver supported Chelsio T5/T4 Unified Wire For Linux Page 48...

  • Page 49: Software Requirements

    Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 49...

  • Page 50: Software/Driver Loading

    To enable full offload support, run the following command: [root@host]# modprobe t4_tom Offload support needs to be enabled upon each reboot of the system. This can Note be done manually as shown above. Chelsio T5/T4 Unified Wire For Linux Page 50...

  • Page 51: Software/Driver Unloading

    3. Software/Driver Unloading 3.1. Unloading the NIC driver To unload the NIC driver, run the following command: [root@host]# rmmod cxgb4 3.2. Unloading the TOE driver Please reboot the system to unload the TOE driver. Chelsio T5/T4 Unified Wire For Linux Page 51...

  • Page 52: Software/Driver Configuration And Fine-Tuning

    In order to auto tune the system and TOE devices for best performance, Chelsio recommends t4_perftune.sh installing the tools which will copy script to /sbin directory. Run the script by using the following command: [root@host]# t4_perftune.sh This script will configure RSS and enable Interrupt Coalescing. Chelsio T5/T4 Unified Wire For Linux Page 52...
  • Page 53 Then on the sender host, send data to the server using the iperf client mode. To emulate a moderate traffic workload, use -P option to request 20 TCP streams from the server: [root@sender_host]# iperf -c receiver_host_name_or_IP -P 20 Chelsio T5/T4 Unified Wire For Linux Page 53...
  • Page 54 40/10Gbps, the network interrupt demands are significantly high. Under such circumstances, it is necessary to enable RSS to balance the network load across multiple processors and achieve the best performance. Chelsio T5/T4 Unified Wire For Linux Page 54...
  • Page 55 Chelsio’s T5/T4 card supports both hardware assisted GRO/LRO and Linux-based GRO/LRO. t4_tom is the kernel module that enables the hardware assisted GRO/LRO. If it is not already in the kernel module list, use the following command to insert it: Chelsio T5/T4 Unified Wire For Linux Page 55...
  • Page 56 LRO on and off as well. When Linux’s GRO is enabled, Chelsio’s T5/T4 driver provides two GRO-related statistics. They are displayed using the following command: [root@host]# ethtool -S eth6 GROPackets : 0 GROMerged : 897723 Chelsio T5/T4 Unified Wire For Linux Page 56...
  • Page 57 [root@host]# ethtool -K eth6 gro off [root@host]# ethtool -k eth6 Offload parameters for eth6: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off The output above shows a disabled GRO. Chelsio T5/T4 Unified Wire For Linux Page 57...

  • Page 58: Virtual Function Network (Vnic)

    Chapter III. Virtual Function Network (vNIC) III. Virtual Function Network (vNIC) Chelsio T5/T4 Unified Wire For Linux Page 58...

  • Page 59: Introduction

    The following are the currently shipping Chelsio Adapters that are compatible with the Chelsio vNIC driver: T502-BT T580-CR T520-LL-CR T520-CR T522-CR T580-LP-CR T540-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T440-LP-CR T420-BT T420-LL-CR T420-CX Chelsio T5/T4 Unified Wire For Linux Page 59...

  • Page 60: Software Requirements

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 60...

  • Page 61: Software/Driver Loading

    The vNIC driver must be loaded or unloaded on the Guest OS by the root user. Any attempt to load the driver as a regular user will fail. 2.1. Loading the driver To load the driver, run the following command: [root@host]# modprobe cxgb4vf Chelsio T5/T4 Unified Wire For Linux Page 61...

  • Page 62: Software/Driver Unloading

    Chapter III. Virtual Function Network (vNIC) 3. Software/Driver Unloading 3.1. Unloading the driver To unload the driver, execute the following command: [root@host]# rmmod cxgb4vf Chelsio T5/T4 Unified Wire For Linux Page 62...

  • Page 63: Software/Driver Configuration And Fine-Tuning

    Chapter III. Virtual Function Network (vNIC) 4. Software/Driver Configuration and Fine-tuning 4.1. Instantiate Virtual Functions To instantiate Chelsio Virtual Functions, please refer to the Network (NIC/TOE) section (click here) Chelsio T5/T4 Unified Wire For Linux Page 63...

  • Page 64: Iwarp (Rdma)

    Chapter IV. iWARP (RDMA) IV. iWARP (RDMA) Chelsio T5/T4 Unified Wire For Linux Page 64...

  • Page 65: Introduction

    Currently the iWARP driver is available for the following versions: Redhat Enterprise Linux 5 update 9 kernel (RHEL5.9), 2.6.18-348.el5* Redhat Enterprise Linux 5 update 10 kernel (RHEL5.10), 2.6.18-371.el5* Redhat Enterprise Linux 6 update 4 kernel (RHEL6.4), 2.6.32-358.el6 Chelsio T5/T4 Unified Wire For Linux Page 65...
  • Page 66 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 66...

  • Page 67: Software/Driver Loading

    [root@host]# make install  To load the iWARP driver we need to load the NIC driver and core RDMA drivers first. Run the following commands: [root@host]# modprobe cxgb4 [root@host]# modprobe iw_cxgb4 [root@host]# modprobe rdma_ucm Chelsio T5/T4 Unified Wire For Linux Page 67...

  • Page 68: Software/Driver Unloading

    Chapter IV. iWARP (RDMA) 3. Software/Driver Unloading To unload the iWARP driver, run the following command: [root@host]# rmmod iw_cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 68...

  • Page 69: Software/Driver Configuration And Fine-Tuning

    CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA client DISCONNECT EVENT... Chelsio T5/T4 Unified Wire For Linux Page 69...

  • Page 70: Enabling Various Mpis

    Create authorization file [root@host]# cd .ssh [root@host]# cat *.pub > authorized_keys [root@host]# chmod 600 authorized_keys v. Copy directory .ssh to all systems in the cluster [root@host]# cd [root@host]# scp -r /root/.ssh remotehostname-or-ipaddress: Chelsio T5/T4 Unified Wire For Linux Page 70...
  • Page 71 Contact Intel for obtaining their MPI with DAPL support. xi. To run Intel MPI applications: mpdboot -n <no_of_nodes_in_cluster> -r ssh mpdtrace mpiexec -ppn -n 2 /opt/intel/impi/3.1/tests/IMB-3.1/IMB-MPI1 Chelsio T5/T4 Unified Wire For Linux Page 71...
  • Page 72 --with-openib-libdir=/usr/lib64/ --libdir=/usr/mpi/gcc/openmpi- x.y.z/lib64/ --with-contrib-vt-flags=--disable-iotrace [root@host]# make [root@host]# make install The above step will install openMPI in /usr/mpi/gcc/openmpi-x.y.z/ To enable multithreading, add “--enable-mpi-thread-multiple” and Note “--with-threads=posix” parameters to the above configure command. Chelsio T5/T4 Unified Wire For Linux Page 72...
  • Page 73 -z "`echo $LD_LIBRARY_PATH | grep /usr/mpi/gcc/openmpi- x.y.z/lib64`"; then LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi- x.y.z/lib64${LD_LIBRARY_PATH:+:}${LD_LIBRARY_PATH} export LD_LIBRARY_PATH # MPI_ROOT MPI_ROOT=/usr/mpi/gcc/openmpi-x.y.z export MPI_ROOT vii. Next, copy the two files created in steps (v) and (vi) to /usr/mpi/gcc/openmpi-x.y.z/bin and /usr/mpi/gcc/openmpi-x.y.z/etc Chelsio T5/T4 Unified Wire For Linux Page 73...
  • Page 74 Untar and change your current working directory to MVAPICH2 package directory. iii. Configure and install as: [root@host]# ./configure --prefix=/usr/mpi/gcc/mvapich2-x.y/ --with- device=ch3:mrail --with-rdma=gen2 --enable-shared --with-ib- libpath=/usr/lib64/ -enable-rdma-cm --libdir=/usr/mpi/gcc/mvapich2-x.y/lib64 [root@host]# make [root@host]# make install The above step will install MVAPICH2 in /usr/mpi/gcc/mvapich2-x.y/ Chelsio T5/T4 Unified Wire For Linux Page 74...
  • Page 75 -z "`echo $PATH | grep /usr/mpi/gcc/ mvapich2-x.y/bin`"; then PATH=/usr/mpi/gcc/mvapich2-x.y/bin:${PATH} export PATH # LD_LIBRARY_PATH if test -z "`echo $LD_LIBRARY_PATH | grep /usr/mpi/gcc/mvapich2- x.y/lib64`"; then LD_LIBRARY_PATH=/usr/mpi/gcc/mvapich2- x.y/lib64${LD_LIBRARY_PATH:+:}${LD_LIBRARY_PATH} export LD_LIBRARY_PATH # MPI_ROOT MPI_ROOT=/usr/mpi/gcc/mvapich2-x.y export MPI_ROOT Chelsio T5/T4 Unified Wire For Linux Page 75...
  • Page 76 Untar and change your current working directory to src directory. iii. Edit make_mpich file and set MPI_HOME variable to the MPI which you want to build the benchmarks tool against. For example, in case of openMPI-1.6.4 set the variable as: MPI_HOME=/usr/mpi/gcc/openmpi-1.6.4/ Chelsio T5/T4 Unified Wire For Linux Page 76...
  • Page 77 Local device: Unknown Error name: RDMA_CM_EVENT_ADDR_ERROR Peer: core96n8 Workaround: Increase the OpenMPI rdma route resolution timeout. The default is 1000, or 1000ms. Increase it to 30000 with this parameter: --mca btl_openib_connect_rdmacm_resolve_timeout 30000 Chelsio T5/T4 Unified Wire For Linux Page 77...

  • Page 78: Setting Up Nfs-Rdma

    All services in NFS should start without errors. /proc/fs/nfsd/ v. Now we need to edit the file portlist in the path vi. Include the rdma port 2050 into this file as: [root@host]# echo rdma 2050 > /proc/fs/nfsd/portlist Chelsio T5/T4 Unified Wire For Linux Page 78...
  • Page 79 Run the showmount command to show all directories from server as: [root@host]# showmount –e <server-chelsio-ip> iv. Once the exported directories are listed, mount them as: [root@host]# mount.nfs <serverip>:<directory> <mountpoint-on-client> -o vers=3,rdma,port=2050,wsize=65536,rsize=65536 Chelsio T5/T4 Unified Wire For Linux Page 79...

  • Page 80: Wd-Udp

    Chapter V. WD-UDP V. WD-UDP Chelsio T5/T4 Unified Wire For Linux Page 80...

  • Page 81: Introduction

    Redhat Enterprise Linux 5 update 10 kernel (RHEL5.10), 2.6.18-371.el5* Redhat Enterprise Linux 6 update 4 kernel (RHEL6.4), 2.6.32-358.el6 Redhat Enterprise Linux 6 update 5 kernel (RHEL6.5), 2.6.32-431.el6* Suse Linux Enterprise Server 11 SP1 kernel (SLES11SP1),2.6.32.12-0.7 Chelsio T5/T4 Unified Wire For Linux Page 81...
  • Page 82 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 82...

  • Page 83: Software/Driver Loading

    RDMA core modules from the OFED package should be loaded before proceeding. To load the WD-UDP driver, use the following commands which will automatically load RDMA core modules: [root@host]# modprobe cxgb4 [root@host]# modprobe iw_cxgb4 [root@host]# modprobe rdma_ucm Chelsio T5/T4 Unified Wire For Linux Page 83...

  • Page 84: Software/Driver Unloading

    Chapter V. WD-UDP 3. Software/Driver Unloading To unload the WD-UDP driver, run the following command: [root@host]# rmmod iw_cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 84...

  • Page 85: Software/Driver Configuration And Fine-Tuning

    Due to HW resource limitations, only a small number of queues can be allocated for UDP acceleration. Therefore only performance critical UDP applications should use libcxgb4_sock. Only 64 IPv4 UDP sockets / 28 IPv6 UDP Sockets can be accelerated per Chelsio T5/T4 device used.
  • Page 86 Create this file on all systems using libcxgb4_sock. Here is the syntax: # Syntax: # endpoint {attributes} ... # where attributes include: interface = interface-name port = udp-port-number vlan = vlan-id priority = vlan-priority Chelsio T5/T4 Unified Wire For Linux Page 86...
  • Page 87 {interface = eth2.7 priority=3 port=9000} Now, preload libcxgb4_sock: [root@host]# CXGB4_SOCK_CFG=<path to config file> LD_PRELOAD=libcxgb4_sock.so <pathto>/your_application In order to offload IPv6 UDP sockets, please select “low latency networking” as Note T5/T4 configuration tuning option during installation. Chelsio T5/T4 Unified Wire For Linux Page 87...
  • Page 88 The udp benchmark from the hpcbench suite can be used to show the benefits of libcxgb4_sock. The hpcbench suite can be found at: Source: http://hpcbench.sourceforge.net/index.html Sample: http://hpcbench.sourceforge.net/udp.html The nodes in this example, r9 and r10, have T5/T4 eth1 configured and the ports are connected point-to-point. Chelsio T5/T4 Unified Wire For Linux Page 88...
  • Page 89 Then on r9, we run udptest to port 9000 to see the host stack UDP latency: [root@r9 ~]# /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9000 Running the same test with libcxgb4_sock: [root@r9 ~]# LD_PRELOAD=libcxgb4_sock.so /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9001 Chelsio T5/T4 Unified Wire For Linux Page 89...
  • Page 90 To see if the application is being offloaded, open a window on one of the machines, and run tcpdump against the Chelsio interface. If you see minimal UDP output on the interface, then the UDP traffic is being properly offloaded. Chelsio T5/T4 Unified Wire For Linux Page 90...

  • Page 91: Wd-Toe

    Chapter VI. WD-TOE VI. WD-TOE Chelsio T5/T4 Unified Wire For Linux Page 91...

  • Page 92: Introduction

    Redhat Enterprise Linux 5 update 10 kernel (RHEL5.10), 2.6.18-371.el5* Redhat Enterprise Linux 6 update 4 kernel (RHEL6.4), 2.6.32-358.el6 Redhat Enterprise Linux 6 update 5 kernel (RHEL6.5), 2.6.32-431.el6* Suse Linux Enterprise Server 11 SP1 kernel (SLES11SP1), 2.6.32.12-0.7* Chelsio T5/T4 Unified Wire For Linux Page 92...
  • Page 93 Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6 Kernel.org linux-3.7 Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 93...

  • Page 94: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. Run the following commands to load the driver: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Chelsio T5/T4 Unified Wire For Linux Page 94...

  • Page 95: Software/Driver Unloading

    Chapter VI. WD-TOE 3. Software/Driver Unloading Reboot the system to unload the driver. Chelsio T5/T4 Unified Wire For Linux Page 95...

  • Page 96: Software/Driver Configuration And Fine-Tuning

    Start netserver at the PEER, using the following command: [root@host]# PROT=TCP wdload netserver –D -4 ii. On the Test Machine, run the following command to run netperf application. [root@host]# PROT=TCP wdload netperf -H <PEER_IP> -t TCP_RR -l 10 Chelsio T5/T4 Unified Wire For Linux Page 96...

  • Page 97: Iscsi Pdu Offload Target

    Chapter VII. iSCSI PDU Offload Target VII. iSCSI PDU Offload Target Chelsio T5/T4 Unified Wire For Linux Page 97...

  • Page 98: Introduction

    Offloads iSCSI Fast Data Path with Direct Data Placement (DDP)  Offloads iSCSI Header and Data Digest Calculations  Offload Speeds at 1 Gb and 10 Gb  Offloads TCP/IP for NAS simultaneously with iSCSI Target Specific features Chelsio T5/T4 Unified Wire For Linux Page 98...

  • Page 99: Hardware Requirements

    1.2.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with iSCSI PDU Offload Target software: T502-BT T580-CR T520-LL-CR T520-CR T522-CR T580-LP-CR T540-CR T420-CR T440-CR T422-CR T404-BT T420-BCH Chelsio T5/T4 Unified Wire For Linux Page 99...
  • Page 100 For deployment in a production environment a more sophisticated system would be needed. That typically consists of a system with one or more storage controllers with multiple disk drives attached running software or hardware based RAID. Chelsio T5/T4 Unified Wire For Linux Page 100...

  • Page 101: Software Requirements

    A production/evaluation license file,is required for each copy of the Chelsio iSCSI PDU Offload Target software installed. The license is tied to the selected Chelsio NIC present in the system. The license file will be generated depending on your requirement for a Chelsio iSCSI Target. Chelsio T5/T4 Unified Wire For Linux Page 101...
  • Page 102 The key file will be sent back and must be installed on the system in order to unlock the software. Details on this mechanism can be found in the next section on software installation. Chelsio T5/T4 Unified Wire For Linux Page 102...

  • Page 103: Software/Driver Loading

    It also provides control for iSCSI global settings. 1. Loading the Kernel module For RHEL distributions, run modprobe as follows: [root@host]# modprobe chiscsi_t4 For SLES11 spX, run modprobe as follows: [root@host]# modprobe chiscsi_t4 --allow-unsupported Chelsio T5/T4 Unified Wire For Linux Page 103...
  • Page 104 It can also be used to stop the targets, and restart/reload configuration. For the script to execute properly, make sure the following flag is set on all Note kernel.org kernels. # CONFIG_MODULE_FORCE_LOAD=y Chelsio T5/T4 Unified Wire For Linux Page 104...

  • Page 105: Generating Single Rpm For T3 And T4 Adapters

    7. Next, build the iSCSI PDU offload target driver (chiscsi_t3) for T3 adapters using the Chelsio Unified Wire source package: [root@host]# cd ChelsioUwire-x.xx.x.x/build/src/chiscsi [root@host]# make CXGB3TOE_SRC=<path to T3 NIC package>/src 8. Now, generate the RPM: [root@host]# make iscsi_pdu_target_rpm Chelsio T5/T4 Unified Wire For Linux Page 105...

  • Page 106: Obtaining The Iscsi Software License

    PCI vendor ID MAC address Link speed Privacy notice: The above information is used by Chelsio Communications strictly for iSCSI license management. Chelsio Communications, Inc. does not share, sell, rent, or lease the information to any third party. Chelsio T5/T4 Unified Wire For Linux...
  • Page 107 ******************************************************* SUMMARY: Interface with MAC Address 00:07:43:00:00:10 (10000 Mbps) is selected Please send info file "/etc/xyz_chelsio_infofile" to Chelsio Communications at support@chelsio.com Chelsio Support will send back the keyfile. The keyfile unlocks the software as long as the above interface is present...

  • Page 108: Software/Driver Unloading

    Chapter VII. iSCSI PDU Offload Target 3. Software/Driver Unloading Use the following command to unload the module: [root@host]# rmmod chiscsi_t4 Chelsio T5/T4 Unified Wire For Linux Page 108...

  • Page 109: Software/Driver Configuration And Fine-Tuning

    There are only a few global configuration items that can be changed. An example is “iscsi_offload_mode” which specifies the hardware protocol offload method that should be used (e.g. “TOE”, “ULP”, or “AUTO” modes) for all iSCSI targets. Chelsio T5/T4 Unified Wire For Linux Page 109...

  • Page 110: A Quick Start Guide For Target

    TargetDevice – A device served up by the associated target. A device can be:  A block device (for example, /dev/sda)  A virtual block device (for example, /dev/md0)  A RAM disk  A regular file Chelsio T5/T4 Unified Wire For Linux Page 110...
  • Page 111 To start a specific target execute [root@host]# iscsictl –f /etc/chelsio-iscsi/chiscsi.conf –S target=iqn.2006-02.com.chelsio.diskarray.san1 with “ ” option followed by iscsictl Stop Target: To stop the all the iSCSI target(s), execute “ ”. target=ALL [root@host]# iscsictl –s target=ALL Chelsio T5/T4 Unified Wire For Linux Page 111...
  • Page 112 ” iscsictl Change Global Settings: To change Chelsio global settings, execute option. [root@host]# iscsictl –G iscsi_offload_mode=AUTO with “ ” option. View Help: To print help to stdout, execute iscsictl [root@host]# iscsictl –h Chelsio T5/T4 Unified Wire For Linux Page 112...

  • Page 113: The Iscsi Configuration File

    Chelsio System Wide Global Entity Parameters pass system control information to the iSCSI software which affects all targets in the same way. More detail of these parameters below can be found in a later section entitled “System Wide Parameters”. Chelsio T5/T4 Unified Wire For Linux Page 113...
  • Page 114 Entity Parameters pass iSCSI protocol control information to the Chelsio iSCSI module. This information is unique for each entity block. The parameters follow the IETF iSCSI standard RFC 3720 in both definition and syntax. The descriptions below are mostly from this RFC. Chelsio T5/T4 Unified Wire For Linux Page 114...
  • Page 115 - 1) iSCSI PDU. OFMarker “Yes” “No” To turn on or off the initiator to target “No” markers on the connection. Chelsio only supports “No”. Chelsio T5/T4 Unified Wire For Linux Page 115...
  • Page 116 NOTE: The double quotes are required as part of the format. Auth_CHAP_Initiator “<user id>” CHAP user id and secret for the :“<secret>” initiator. <user id> must be less than 256 characters. Commas “,” Chelsio T5/T4 Unified Wire For Linux Page 116...
  • Page 117 ShadowMode “Yes” “No” To turn ShadowMode on or off for “No” iSCSI Target Redirection Chelsio T5/T4 Unified Wire For Linux Page 117...
  • Page 118 16MB (16 Megabytes). The minimum value of x is 1 (1MB) and the maximum value is limited by system memory. ScsiID=xxxxxx is a 16 character unique value set for multipath aware iSCSI initiator host. Chelsio T5/T4 Unified Wire For Linux Page 118...
  • Page 119 IP addresses that the iSCSI target is listening on (i.e., the target portal IP addresses). NOTE: when configuring an ACL at least one of the above three must be provided:  iname, and/or Chelsio T5/T4 Unified Wire For Linux Page 119...
  • Page 120 # Chelsio iSCSI Global Settings global: iscsi_offload_mode=AUTO iscsi_auth_order=ACL # an iSCSI Target “iqn.2006-02.com.chelsio.diskarray.san1” # being served by the portal group "5". Setup as a RAM Disk. target: TargetName=iqn.2006-02.com.chelsio.diskarray.san1 Chelsio T5/T4 Unified Wire For Linux Page 120...
  • Page 121 # Local block devices being served up # lun 0 is pointed to /dev/sda # lun 1 is pointed to /dev/sdb TargetDevice=/dev/sda,ScsiID=aabbccddeeffgghh,WWN=aaabbbcccdddeeef TargetDevice=/dev/sdb # Portal groups served this target PortalGroup=1@102.50.50.25:3260 PortalGroup=2@102.60.60.25:3260 # CHAP configuration Chelsio T5/T4 Unified Wire For Linux Page 121...

  • Page 122: Challenge-Handshake Authenticate Protocol (Chap)

    4.5.1. Oneway CHAP authentication With one-way CHAP (also called unidirectional CHAP) the target uses CHAP to authenticate the initiator. The initiator does not authenticate the target. This method is the default method. Chelsio T5/T4 Unified Wire For Linux Page 122...
  • Page 123 CHAP is used. The default setting of AuthMethod=None,CHAP On an iSCSI node, with Auth_CHAP_Policy=Oneway , Chelsio iSCSI target will accept a relevant initiator if it does a) no CHAP b) CHAP Oneway or c) CHAP Mutual Chelsio T5/T4 Unified Wire For Linux Page 123...

  • Page 124: Target Access Control List (Acl) Configuration

    The format for ACL rule is as follows: ACL=[iname=<initiator name>][;<sip=<source ip addresses>] [;dip=<destination ip addresses>][;lun=<lun_list>:<permissions>] target: TargetName=iqn.2006-02.com.chelsio.diskarray.san1 TargetDevice=/dev/sda PortalGroup=1@102.50.50.25:3260 PortalGroup=2@102.60.60.25:3260 # initiator “iqn.2006-02.com.chelsio.san1” is allowed # full read-write access to this target Chelsio T5/T4 Unified Wire For Linux Page 124...

  • Page 125: Target Storage Device Configuration

    This option is usefuly for measuring network performance. Chelsio T5/T4 Unified Wire For Linux Page 125...
  • Page 126 “SYNC” can be used with FILE mode to make sure the data is flushed to the storage device before the Target responds back to the Initiator. FILE To configure a FILE storage device specify as the device mode: TargetDevice=<path to the storage device>[,FILE][,SYNC] Chelsio T5/T4 Unified Wire For Linux Page 126...
  • Page 127 TargetDevice=<path to the storage device>,BLK <path> Where: Is the path to the actual storage device, such as /dev/sdb. The path must exist in the system. target: #<snip> # /dev/sdb is accessed in BLK mode TargetDevice=/dev/sdb,BLK #<snip> Chelsio T5/T4 Unified Wire For Linux Page 127...

  • Page 128: Target Redirection Support

    Below is an example with ShadowMode disabled: target: #<snip> # any login requests received on 10.193.184.81:3260 will be # redirected to 10.193.184.85:3261 PortalGroup=1@10.193.184.81:3260,[2] PortalGroup=2@10.193.184.85:3261 # the PortalGroup “2” IS present on the same system ShadowMode=No #<snip> Chelsio T5/T4 Unified Wire For Linux Page 128...

  • Page 129: The Command Line Interface Tools "Iscsictl" & "Chisns

    4.9.2. chisns chisns is the command line tool for controlling the iSNS client. This is a simple tool that starts the iSNS client with a client and server parameter. Chelsio T5/T4 Unified Wire For Linux Page 129...
  • Page 130 Chelsio iSCSI targets(s) configuration(s) will be displayed. If target=ALL is specified or no parameters are specified the output will be abbreviated. Specify specific targets to get detailed configuration data. Chelsio T5/T4 Unified Wire For Linux Page 130...
  • Page 131 | TOE | ULP> iscsi_auth_order=<ACL | CHAP> iscsi_target_vendor_id=<string of max. 8 characters> Example: iscsictl -G iscsi_auth_order=ACL The var=const parameter(s) are mandatory. If the var=const parameter is not specified, the command will be denied. Chelsio T5/T4 Unified Wire For Linux Page 131...
  • Page 132 All non-active targets specified will not be loaded / started. For Rules 1-3, if the specified targets are currently active (running), they will get reloaded. For Rules 1 & 2, if the specified targets are not Chelsio T5/T4 Unified Wire For Linux Page 132...
  • Page 133 It defaults to <hostname>. query=<query interval> where query is the initiator query interval (in seconds). It defaults to 60 seconds. Examples: chisns server=192.0.2.10 chisns server=192.0.2.10:3205 id=isnscln2 query=30 Chelsio T5/T4 Unified Wire For Linux Page 133...

  • Page 134: Rules Of Target Reload (I.e. "On The Fly" Changes)

    The following parameter CAN be changed by reloading of the configuration file. The new value will NOT affect any connections and sessions that already completed login phase: TargetAlias MaxConnections InitialR2T ImmediateData FirstBurstLength MaxBurstLength MaxOutstandingR2T HeaderDigest DataDigest MaxRecvDataSegmentLength AuthMethod Auth_CHAP_Initiator Auth_CHAP_Target Chelsio T5/T4 Unified Wire For Linux Page 134...

  • Page 135: System Wide Parameters

    It will select ULP mode when the connection is going through a Chelsio iSCSI acceleration enabled HBA. The purpose of this mode is to optimize the performance of the Chelsio acceleration hardware. Chelsio T5/T4 Unified Wire For Linux Page 135...
  • Page 136 4.11.3. iscsi_target_vendor_id CHISCSI Options: A string of maximum of 8 characters, defaults to iscsi_target_vendor_id is part of the device identification sent by an iSCSI target in response of a SCSI Inquiry request. Chelsio T5/T4 Unified Wire For Linux Page 136...

  • Page 137: Iscsi Pdu Offload Initiator

    Chapter VIII. iSCSI PDU Offload Initiator VIII. iSCSI PDU Offload Initiator Chelsio T5/T4 Unified Wire For Linux Page 137...

  • Page 138: Introduction

    Chapter VIII. iSCSI PDU Offload Initiator 1. Introduction The Chelsio T5/T4 series Adapters support iSCSI acceleration and iSCSI Direct Data Placement (DDP) where the hardware handles the expensive byte touching operations, such as CRC computation and verification, and direct DMA to the final host memory destination:...

  • Page 139: Software Requirements

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Kernel.org linux-3.4* Kernel.org linux-3.6 Kernel.org linux-3.7* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 139...

  • Page 140: Software/Driver Loading

    If loading the driver fails, then follow the steps mentioned below and try again. Run the following command to view all the loaded iSCSI modules [root@host]# lsmod | grep iscsi Chelsio T5/T4 Unified Wire For Linux Page 140...
  • Page 141 Chapter VIII. iSCSI PDU Offload Initiator ii. Now, unload them using the following command: [root@host]# rmmod <modulename> Chelsio T5/T4 Unified Wire For Linux Page 141...

  • Page 142: Software/Driver Unloading

    Chapter VIII. iSCSI PDU Offload Initiator 3. Software/Driver Unloading To unload the driver, execute the following commands: [root@host]# rmmod cxgb4i [root@host]# rmmod libcxgbi Chelsio T5/T4 Unified Wire For Linux Page 142...

  • Page 143: Software/Driver Configuration And Fine-Tuning

    Create an interface file located under iface directory for the new transport class cxgb4i in the following format: iface.iscsi_ifacename = <iface file name> iface.hwaddress = <MAC address> iface.transport_name = cxgb4i iface.net_ifacename = <ethX> iface.ipaddress = <iscsi ip address> Chelsio T5/T4 Unified Wire For Linux Page 143...
  • Page 144 Start Daemon from /sbin by using the following command: [root@host]# iscsid -f -d 3 iscsid is already running, then kill the service and start it as shown above Note after installing the Chelsio Unified Wire package. Chelsio T5/T4 Unified Wire For Linux Page 144...
  • Page 145 -m node -T <iqn name of target> -p <target ip address>:<target port no> -I <cxgb4i iface file name> -u E.g.:- [root@host]# iscsiadm -m node -T iqn.2004-05.com.chelsio.target1 -p 102.2.2.155:3260,1 -I cxgb4i.00:07:43:04:5b:da -u Other options can be found by typing iscsiadm --help Note Chelsio T5/T4 Unified Wire For Linux Page 145...

  • Page 146: Auto Login From Cxgb4I Initiator At Os Bootup

    -q iscsi_tcpmodprobe -q ib_iser modprobe -q cxgb4i modprobe -q cxgb3i modprobe -q bnx2i modprobe -q be2iscsi daemon brcm_iscsiuio daemon $prog retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval Chelsio T5/T4 Unified Wire For Linux Page 146...

  • Page 147: Data Center Bridging (Dcb)

    Chapter IX. Data Center Bridging (DCB) IX. Data Center Bridging (DCB) Chelsio T5/T4 Unified Wire For Linux Page 147...

  • Page 148: Introduction

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Kernel.org linux-3.4* Kernel.org linux-3.6 Kernel.org linux-3.7* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 148...

  • Page 149: Software/Driver Loading

    [root@host]# modprobe t4_tom [root@host]# modprobe csiostor Once the storage and networking traffic are started simultaneously, they will honor DCB settings defined on the switch. Limited QA performed for IEEE version of DCBX. Note Chelsio T5/T4 Unified Wire For Linux Page 149...

  • Page 150: Software/Driver Unloading

    3. Software/Driver Unloading To disable DCB feature, unload FCoE Initiator and Network drivers: [root@host]# rmmod csiostor [root@host]# rmmod cxgb4 If t4_tom is loaded, please reboot machine after unloading FCoE Initiator and Note Network drivers. Chelsio T5/T4 Unified Wire For Linux Page 150...

  • Page 151: Software/Driver Configuration And Fine-Tuning

    0 switch(config-cmap-qos)# class-map type qos class-fcoe switch(config-cmap-qos)# match cos 3 iii. Configure queuing class-maps. switch(config)#class-map type queuing class-nic switch(config-cmap-que)#match qos-group 2 iv. Configure network-qos class-maps. switch(config)#class-map type network-qos class-nic switch(config-cmap-nq)#match qos-group Chelsio T5/T4 Unified Wire For Linux Page 151...
  • Page 152 4.1.2. Configuring the FCoE/FC ports In this procedure, you may need to adjust some of the parameters to suit your environment, such as VLAN IDs, Ethernet interfaces, and virtual Fibre Channel interfaces Chelsio T5/T4 Unified Wire For Linux Page 152...
  • Page 153 VLAN and FCoE VLAN. iii. Assign VLAN ID to the Ethernet port on which FCoE service was enabled in step1. switch(config)# interface ethernet 1/13 switch(config-if)#switchport mode trunk switch(config-if)#switchport trunk allowed vlan 2 switch(config-if)#no shutdown switch(config)#exit Chelsio T5/T4 Unified Wire For Linux Page 153...

  • Page 154: Configuring The Brocade 8000 Switch

    To enable it, execute the below command. switch(config)# zone default-zone permit vsan 2 4.2. Configuring the Brocade 8000 switch Configure LLDP for FCoE.Example of configuring LLDP for 10-Gigabit Ethernet interface. switch(config)#protocol lldp switch(conf-lldp)#advertise dcbx-fcoe-app-tlv switch(conf-lldp)#advertise dcbx-fcoe-logical-link-tlv Chelsio T5/T4 Unified Wire For Linux Page 154...
  • Page 155 1 vlan 1002 switch(config-if-te-0/16)#cee default switch(config-if-te-0/16)#no shutdown switch(config-if-te-0/16)#exit Unlike cisco, only one VLAN ID can carry FCoE traffic for now on Brocade 8000. Note It is their limitation. Chelsio T5/T4 Unified Wire For Linux Page 155...
  • Page 156 Chapter IX. Data Center Bridging (DCB) v. Save the Configuration switch#copy running-config startup-config Chelsio T5/T4 Unified Wire For Linux Page 156...

  • Page 157: Fcoe Pdu Offload Target

    Chapter X. FCoE PDU Offload Target X. FCoE PDU Offload Target Chelsio T5/T4 Unified Wire For Linux Page 157...

  • Page 158: Introduction

    The license is tied to the selected Chelsio NIC present in the system. The license file will be generated depending on your requirement for a Chelsio FCoE Target. To obtain a license key file, which could be either a production or an evaluation version, please follow the steps below. Chelsio T5/T4 Unified Wire For Linux Page 158...
  • Page 159 For the selected NIC, the following information is gathered: PCI vendor ID MAC address Link speed Privacy notice: The above information is used by Chelsio Communications strictly for license management. Chelsio Communications, Inc. does not share, sell, rent, or lease the information to any third party. [root@host]# chinfotool Scanning System for network devices..
  • Page 160 Chelsio, the user will receive a license file tailored to the system. Copy the file /etc/chelsio-fcoe/ [root@host]# cp -f chfcoe.key /etc/chelsio-fcoe/ The FCoE target software is now unlocked, ready to be configured and used. Chelsio T5/T4 Unified Wire For Linux Page 160...

  • Page 161: Software/Driver Loading

    [root@host]# modprobe chfcoe chfcoe_fip_mode=1  To load the driver in VN2VN mode with VLAN support, run the following command: [root@host]# modprobe chfcoe chfcoe_fip_mode=1 chfcoe_vlanid=<vlan_id> Alternatively, you can edit /etc/modprobe.d/chfcoe.conf for specifying VN2VN and VLAN options. Chelsio T5/T4 Unified Wire For Linux Page 161...

  • Page 162: Software/Driver Configuration And Fine-Tuning

    Configure qos class-maps and set the traffic priorities: NIC uses priority 0 and FcoE uses priority 3. switch(config)#class-map type qos class-nic switch(config-cmap-qos)# match cos 0 switch(config-cmap-qos)# class-map type qos class-fcoe switch(config-cmap-qos)# match cos 3 iii. Configure queuing class-maps. switch(config)#class-map type queuing class-nic switch(config-cmap-que)#match qos-group 2 Chelsio T5/T4 Unified Wire For Linux Page 162...
  • Page 163 Apply the new policy (PFC on NIC and FcoE traffic) to the entire system. switch(config)#system qos switch(config-sys-qos)#service-policy type qos input policy-test switch(config-sys-qos)#service-policy type queuing output policy-test switch(config-sys-qos)#service-policy type queuing input policy-test switch(config-sys-qos)#service-policy type network-qos policy-test Chelsio T5/T4 Unified Wire For Linux Page 163...
  • Page 164 VLAN and FCoE VLAN. iii. Assign VLAN ID to the Ethernet port on which FCoE service was enabled in step1. switch(config)# interface ethernet 1/13 switch(config-if)#switchport mode trunk switch(config-if)#switchport trunk allowed vlan 2 switch(config-if)#no shutdown switch(config)#exit Chelsio T5/T4 Unified Wire For Linux Page 164...

  • Page 165: Collecting Port Information

    You can use cxgbtool to view both target and initiator WWPNs. The command requires an adapter device file to be specified. After loading chfcoe driver, device files will be created for each adapter in /dev/chfcoeX, where X is index of the adapter. [root@host]# ls /dev/chfcoe* /dev/chfcoe0 Chelsio T5/T4 Unified Wire For Linux Page 165...
  • Page 166 If FCoE PDU Offload Target is operating in FCF mode, then the local node information will be available only after the target completes FLOGI to the swtich. In order to identify Chelsio target’s wwpn from other vendors’, the WWPN always Note begins with 0x5000743. Chelsio T5/T4 Unified Wire For Linux Page 166...
  • Page 167 Chapter X. FCoE PDU Offload Target 3.2.2. Verifying remote ports To verify remote ports (fabric, name server, initiator ports etc.), use the following command: [root@host] # cxgbtool stor -a <adapter device file> --show-rnode wwn=<target wwpn> Chelsio T5/T4 Unified Wire For Linux Page 167...
  • Page 168 Chapter X. FCoE PDU Offload Target Chelsio T5/T4 Unified Wire For Linux Page 168...

  • Page 169: Configuring Luns On Target

    Ensure that SCST handler modules, which are used in configuration (eg: scst_vdisk), are loaded before proceeding. iv. Configure LUNs on target by running the following command. [root@host]# scstadmin -config <LUN Config file> Chelsio T5/T4 Unified Wire For Linux Page 169...
  • Page 170 Chapter X. FCoE PDU Offload Target The following configuration file adds three LUNs(ram disk, physical disk & nullio disk) for the target specified. Only initiators present in the group will be able to access the LUNs. Chelsio T5/T4 Unified Wire For Linux Page 170...

  • Page 171: Configuring Persistent Target

    Place the configuration file created in the previous step, scst.conf, under /etc/chelsio-fcoe/ directory. iii. If any handler other than scst_vdisk is used in SCST configuration, please add module load entry for the same under chfcoe_start() section of /etc/init.d/chfcoe file. Chelsio T5/T4 Unified Wire For Linux Page 171...

  • Page 172: Verifying Initiators Connected To The Target

    Once connected to the target, initiators will be able to access the LUNs configured on the target. Execute the following command to list the initiators which are connected to the target: [root@host] # scstadmin –list_session Chelsio T5/T4 Unified Wire For Linux Page 172...

  • Page 173: Removing Luns

    If hyperthreading is enabled, use both thread siblings of a core for binding either workers or irqs. [root@host]# cat /sys/devices/system/cpu/cpu0/topology/threads_siblings_list 0,16 Use both 0 and 16 CPUs for either workers or irqs. Chelsio T5/T4 Unified Wire For Linux Page 173...
  • Page 174 19161's new affinity list: 19 chfcoeworker_8: pid 19162's new affinity list: 20 chfcoeworker_9: pid 19163's new affinity list: 21 chfcoeworker_10: pid 19164's new affinity list: 22 chfcoeworker_11: pid 19165's new affinity list: 23 Chelsio T5/T4 Unified Wire For Linux Page 174...
  • Page 175 Writing 1 in /proc/irq/165/smp_affinity Writing 2 in /proc/irq/166/smp_affinity Writing 10000 in /proc/irq/167/smp_affinity Writing 20000 in /proc/irq/168/smp_affinity Writing 1 in /proc/irq/169/smp_affinity Writing 2 in /proc/irq/170/smp_affinity Writing 10000 in /proc/irq/171/smp_affinity Writing 20000 in /proc/irq/172/smp_affinity Chelsio T5/T4 Unified Wire For Linux Page 175...

  • Page 176: Software/Driver Unloading

    Chapter X. FCoE PDU Offload Target 4. Software/Driver Unloading To unload the driver, run the following command: [root@host]# modprobe –r chfcoe Chelsio T5/T4 Unified Wire For Linux Page 176...

  • Page 177: Fcoe Full Offload Initiator

    Chapter XI. FCoE Full Offload Initiator XI. FCoE Full Offload Initiator Chelsio T5/T4 Unified Wire For Linux Page 177...

  • Page 178: Introduction

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Kernel.org linux-3.4* Kernel.org linux-3.6 Kernel.org linux-3.7* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 178...

  • Page 179: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver, execute the following: [root@host]# modprobe csiostor To load the driver on any SLES distribution, execute the following command: Note [root@host]# modprobe csiostor --allow Chelsio T5/T4 Unified Wire For Linux Page 179...

  • Page 180: Software/Driver Unloading

    3. Software/Driver Unloading To unload the driver: [root@host]# modprobe -r csiostor If multipath services are running, unload of FCoE driver is not possible. Stop the Note multipath service and then unload the driver. Chelsio T5/T4 Unified Wire For Linux Page 180...

  • Page 181: Software/Driver Configuration And Fine-Tuning

     Inorder to identify chelsio fc_host from other vendor fc_host, the WWPN always begins with 0x5000743 Alternatively, the local port information can also be found using: [root@host]# cxgbtool stor –a <adapter_no> --show-lnode Chelsio T5/T4 Unified Wire For Linux Page 181...
  • Page 182 Chapter XI. FCoE Full Offload Initiator Chelsio T5/T4 Unified Wire For Linux Page 182...
  • Page 183 R can correspond to NameServer, Management Server and other initiator ports Note logged in to the switch and targets. Alternatively, the local ports can also be found using cxgbtool [root@host]# cxgbtool stor –a <adapter no> --show-lnode Chelsio T5/T4 Unified Wire For Linux Page 183...
  • Page 184 Chapter XI. FCoE Full Offload Initiator After finding out the wwpn of the local node, to verify the list of discovered targets, use the following command. [root@host]# cxgbtool stor –a <adapter_no> --show-rnode --wwn=<wwpn of lnode> Chelsio T5/T4 Unified Wire For Linux Page 184...

  • Page 185: Formatting The Luns And Mounting The Filesystem

    Chapter XI. FCoE Full Offload Initiator 4.3. Formatting the LUNs and Mounting the Filesystem Use lsscsi –g to list the LUNs discovered by the initiator [root@host]# lsscsi –g Chelsio T5/T4 Unified Wire For Linux Page 185...
  • Page 186 Chapter XI. FCoE Full Offload Initiator Alternatively, the LUNs discovered by the Chelsio FCoE initiators can be accessed via easily- identifiable ‘udev’ path device files like: [root@host]# ls /dev/disk/by-path/pci-0000:04:00.0-csio-fcoe <local_wwpn>:<remote_wwpn>:<lun_wwn> Chelsio T5/T4 Unified Wire For Linux Page 186...

  • Page 187: Creating Filesystem

    Chapter XI. FCoE Full Offload Initiator 4.4. Creating Filesystem Create an ext3 filesystem using the following command: [root@host]# mkfs.ext3 /dev/sdx Chelsio T5/T4 Unified Wire For Linux Page 187...

  • Page 188: Mounting The Formatted Lun

    Chapter XI. FCoE Full Offload Initiator 4.5. Mounting the formatted LUN The formatted LUN can be mounted on the specified mountpoint using the following command: [root@host]# mount /dev/sdx /mnt Chelsio T5/T4 Unified Wire For Linux Page 188...

  • Page 189: Offload Bonding Driver

    Chapter XII. Offload Bonding driver XII. Offload Bonding driver Chelsio T5/T4 Unified Wire For Linux Page 189...

  • Page 190: Introduction

    Redhat Enterprise Linux 5 update 10 kernel (RHEL5.10), 2.6.18-371.el5* Redhat Enterprise Linux 5 update 9 kernel (RHEL5.9), 2.6.18-348.el5* Redhat Enterprise Linux 6 update 4 kernel (RHEL6.4), 2.6.32-358.el6 Redhat Enterprise Linux 6 update 5 kernel (RHEL6.5), 2.6.32-431.el6* Chelsio T5/T4 Unified Wire For Linux Page 190...
  • Page 191 Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 191...

  • Page 192: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host]# modprobe bonding Chelsio T5/T4 Unified Wire For Linux Page 192...

  • Page 193: Software/Driver Unloading

    Chapter XII. Offload Bonding driver 3. Software/Driver Unloading To unload the driver, run the following command: [root@host]# rmmod bonding Chelsio T5/T4 Unified Wire For Linux Page 193...

  • Page 194: Software/Driver Configuration And Fine-Tuning

    Assign IPv4/IPv6 address to the bonded interface: [root@host]# ifconfig bond0 X.X.X.X/Y [root@host]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up All TCP traffic will be offloaded over the bonded interface now. Chelsio T5/T4 Unified Wire For Linux Page 194...

  • Page 195: Offload Multi-Adapter Failover (Mafo)

    Chapter XIII. Offload Multi-Adapter Failover (MAFO) XIII. Offload Multi-Adapter Failover (MAFO) Chelsio T5/T4 Unified Wire For Linux Page 195...

  • Page 196: Introduction

    The following are the currently shipping Chelsio Adapters that are compatible with the Offload Multi-Adapter Failover driver. T502-BT T580-CR T520-LL-CR T520-CR T522-CR T580-LP-CR T540-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX Chelsio T5/T4 Unified Wire For Linux Page 196...

  • Page 197: Software Requirements

    Multi-Adapter failover feature will work for Link Down events caused by:  Cable unplug on bonded interface.  Bringing corresponding switch port down. The feature will not work if the bonded interfaces are administratively taken Note down. Chelsio T5/T4 Unified Wire For Linux Page 197...

  • Page 198: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host]# modprobe bonding Chelsio T5/T4 Unified Wire For Linux Page 198...

  • Page 199: Software/Driver Unloading

    Chapter XIII. Offload Multi-Adapter Failover (MAFO) 3. Software/Driver Unloading To unload the driver, run the following command: [root@host]# rmmod bonding Chelsio T5/T4 Unified Wire For Linux Page 199...

  • Page 200: Software/Driver Configuration And Fine-Tuning

    [root@host]# ifconfig bond0 X.X.X.X/Y [root@host]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up All TCP traffic will be offloaded over the bonded interface now and fail-over will happen in case of link-down event. Chelsio T5/T4 Unified Wire For Linux Page 200...

  • Page 201: Network Device Configuration

    Some operating systems may attempt to auto-configure the detected hardware Note and some may not detect all ports on a multi-port adapter. If this happens, please refer to the operating system documentation for manually configuring the network device. Chelsio T5/T4 Unified Wire For Linux Page 201...

  • Page 202: Udp Segmentation Offload And Pacing

    Chapter XIV. UDP Segmentation Offload and Pacing XIV. UDP Segmentation Offload and Pacing Chelsio T5/T4 Unified Wire For Linux Page 202...

  • Page 203: Introduction

    Service Level Agreements (SLAs). Without dedicated hardware based pacing support, the video server must perform this in software, which is a CPU intensive task and can be prohibitive at 10Gb and higher rates. Chelsio T5/T4 Unified Wire For Linux Page 203...

  • Page 204: Hardware Requirements

    Suse Linux Enterprise Server 11 SP2 kernel (SLES11SP2), 3.0.13-0.27 Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6 Kernel.org linux-3.7 Kernel.org linux-3.8* Chelsio T5/T4 Unified Wire For Linux Page 204...
  • Page 205 Chapter XIV. UDP Segmentation Offload and Pacing Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Performed. Chelsio T5/T4 Unified Wire For Linux Page 205...

  • Page 206: Software/Driver Loading

    [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Though normally associated with the Chelsio TCP Offload engine, the t4_tom module is required in order to allow for the proper redirection of UDP socket calls. Chelsio T5/T4 Unified Wire For Linux Page 206...

  • Page 207: Software/Driver Unloading

    Chapter XIV. UDP Segmentation Offload and Pacing 3. Software/Driver Unloading Reboot the system to unload the driver. Chelsio T5/T4 Unified Wire For Linux Page 207...

  • Page 208: Software/Driver Configuration And Fine-Tuning

    # define IPPROTO_UDP 17 Use the setsockopt() function to set socket options: //Get the UDP socket descriptor variable setsockopt (sockfd , IPPROTO_UDP, SO_FRAMESIZE, &fs, sizeof(fs)); setsockopt (sockfd , IPPROTO_UDP, SOL_SCHEDCLASS, &cl, sizeof(cl)); Chelsio T5/T4 Unified Wire For Linux Page 208...
  • Page 209 3. In step (iv), define a new socket option: setsockopt (sockfd,17,UDP_RTPHEADERLEN,&rtp_header_size, sizeof(rtp_header_size)); Here,  &rtp_header_size : pointer to the RTP header length variable  sizeof(rtp_header_size) : the size of the RTP header length variable Chelsio T5/T4 Unified Wire For Linux Page 209...

  • Page 210: Configuring Udp Pacing

    10240 UDP connections, it can check the return code of ENOSPC from a send() or sendto() call and close this socket and open a new one that uses the kernel UDP stack. Chelsio T5/T4 Unified Wire For Linux Page 210...

  • Page 211: Offload Ipv6 Driver

    Chapter XV. Offload IPv6 driver XV. Offload IPv6 driver Chelsio T5/T4 Unified Wire For Linux Page 211...

  • Page 212: Introduction

    Redhat Enterprise Linux 5 update 9 kernel (RHEL5.9), 2.6.18-348.el5* Redhat Enterprise Linux 6 update 4 kernel (RHEL6.4), 2.6.32-358.el6 Redhat Enterprise Linux 6 update 5 kernel (RHEL6.5), 2.6.32-431.el6* Suse Linux Enterprise Server 11 SP1 kernel (SLES11SP1), 2.6.32.12-0.7 Chelsio T5/T4 Unified Wire For Linux Page 212...
  • Page 213 Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 213...

  • Page 214: Software/Driver Loading

    (t4_tom) drivers. The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom All the IPv6 traffic will be offloaded now. Chelsio T5/T4 Unified Wire For Linux Page 214...

  • Page 215: Software/Driver Unloading

    To disable Offload IPv6 feature, unload NIC and TOE drivers: 3.1. Unloading the NIC driver To unload the NIC driver, run the following command: [root@host]# rmmod cxgb4 3.2. Unloading the TOE driver Please reboot the system to unload the TOE driver. Chelsio T5/T4 Unified Wire For Linux Page 215...

  • Page 216: Bypass Driver

    Chapter XVI. Bypass Driver XVI. Bypass Driver Chelsio T5/T4 Unified Wire For Linux Page 216...

  • Page 217: Introduction

    B420, port 0 to port 1 and vice versa; and in B404, port 0 to port 1 and port 2 to port 3 and vice versa. Normal Mode The Bypass Adapters can be programmed to function as a NIC. In this mode all the packets are redirected to the host. Chelsio T5/T4 Unified Wire For Linux Page 217...

  • Page 218: Hardware Requirements

    Finally, the input action directs the packet to the host where it can be processed by the application. 1.2. Hardware Requirements 1.2.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with Chelsio Bypass driver: B404-BT B420-SR Chelsio T5/T4 Unified Wire For Linux Page 218...

  • Page 219: Software Requirements

    Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 219...

  • Page 220: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. Run the following command to load the Bypass driver: [root@host]# modprobe cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 220...

  • Page 221: Software/Driver Unloading

    Chapter XVI. Bypass Driver 3. Software/Driver Unloading Run the following command to unload the Bypass driver: [root@host]# rmmod cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 221...

  • Page 222: Software/Driver Configuration And Fine-Tuning

    The redirect management element will interface with the SDK server to take the appropriate actions to manage tables and rules. 4.2.1. Bypass Management The bypass CLI will have the following syntax: [root@host]# bypass ethX get|set --key [value] Chelsio T5/T4 Unified Wire For Linux Page 222...
  • Page 223 4.2.3. Bypass State To obtain the current state of the switch: [root@host]# bypass ethX get --current_state To set the state of the switch: [root@host]# bypass ethX set --current_state [bypass|disconnect|normal] Chelsio T5/T4 Unified Wire For Linux Page 223...
  • Page 224 When a rule matches the action associated with it, that rule is performed and higher number rules are not evaluated. Rules are not evaluated until a table is activated. Chelsio T5/T4 Unified Wire For Linux Page 224...
  • Page 225 Redirect rule srcaddr6 IPv6 Source IP address dstaddr6 IPv6 Destination IP address srcmask6 IPv6 Source IP address mask (Default: FFFF:FFFF:FFFF:FFFF:FFF F:FFFF:FFFF:FFFF) dstmask6 IPv6 Destination IP address mask (Default:FFFF:FFFF:FFFF:F FFF:FFFF:FFFF:FFFF:FFFF) Chelsio T5/T4 Unified Wire For Linux Page 225...
  • Page 226 [root@host]# redirect ethX create_table --table 1 Activate the table: [root@host]# redirect ethX activate_table --table 1 Add a rule to the newly created table: [root@host]# redirect ethX add --table 1 --proto icmp –-action drop –-port 0 Chelsio T5/T4 Unified Wire For Linux Page 226...
  • Page 227 IPv4 mode. For IPv6, the ba_server needs to be started manually. [root@host]# service bad start To stop the service: [root@host]# service bad stop To restart the service: [root@host]# service bad restart Chelsio T5/T4 Unified Wire For Linux Page 227...

  • Page 228: Wd Sniffing And Tracing

    Chapter XVII. WD Sniffing and Tracing XVII. WD Sniffing and Tracing Chelsio T5/T4 Unified Wire For Linux Page 228...

  • Page 229: Theory Of Operation

    Connect Client A and B to ports 0 and 1 or ports 2 and 3. d) Enable tracing. In either mode the targeted traffic bypasses the kernel TCP/IP stack and is delivered directly to user space by means of an RX queue. Chelsio T5/T4 Unified Wire For Linux Page 229...

  • Page 230: Hardware Requirements

    Schematic diagram of T4 sniffer and tracer 1.1. Hardware Requirements 1.1.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with the tools: T420-CR T440-CR T422-CR T404-BT T420-BCH Chelsio T5/T4 Unified Wire For Linux Page 230...

  • Page 231: Software Requirements

    Suse Linux Enterprise Server 11 SP2 kernel (SLES11SP2), 3.0.13-0.27 * Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 231...

  • Page 232: Installation And Usage

    2.3. Using Tracer (wd_tcpdump_trace) 1. Setup: Wire tapping requires 3 systems with one machine having a T4 card with two or more ports. The machines should be setup in the following manner: Chelsio T5/T4 Unified Wire For Linux Page 232...
  • Page 233 [root@host]# wd_tcpdump_trace -i <iface> Try ping or ssh between machines A and B. The traffic should successfully make it from end to end and wd_tcpdump_trace on the DUT should show the tapped traffic. Chelsio T5/T4 Unified Wire For Linux Page 233...

  • Page 234: Xviii.classification And Filtering

    Chapter XVIII. Classification and Filtering XVIII. Classification and Filtering Chelsio T5/T4 Unified Wire For Linux Page 234...

  • Page 235: Introduction

    The following are the currently shipping Chelsio Adapters that are compatible with the Classification and Filtering feature: T502-BT T580-CR T520-LL-CR T520-SO-CR T520-CR T522-CR T540-CR T580-LP-CR T580-SO-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX Chelsio T5/T4 Unified Wire For Linux Page 235...

  • Page 236: Software Requirements

    Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8 Kernel.org linux-3.9 Kernel.org linux-3.10 Kernel.org linux-3.11 Kernel.org linux-3.12 Kernel.org linux-3.13 Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Peformed. Chelsio T5/T4 Unified Wire For Linux Page 236...

  • Page 237: Usage

    If you haven’t done already, run the Unified Wire Installer with the appropriate T5/T4 configuration tuning option to install the Network Driver. Next, run the following command to load the network driver: [root@host]# modprobe cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 237...
  • Page 238 [root@host]# cxgbtool ethX filter 0 action switch iport 0 eport 1 vlan 3 The above filter rule will route all ingress packets that match VLAN id 3 from port 0 of Chelsio adapter to port 1.Remaining packets will be sent to the host. Chelsio T5/T4 Unified Wire For Linux Page 238...

  • Page 239: Listing Filter Rules

    To remove a filter, run the following command with the corresponding filter rule index [root@host]# cxgbtool ethX filter index <delete|clear> For more information on additional parameters, refer cxgbtool manual by running Note man cxgbtool command Chelsio T5/T4 Unified Wire For Linux Page 239...

  • Page 240: Layer 3 Example

    Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:14372 errors:0 dropped:0 overruns:0 frame:0 TX packets:62203 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1585952 (1.5 MiB) TX bytes:4798122 (4.5 MiB) Interrupt:16 Chelsio T5/T4 Unified Wire For Linux Page 240...
  • Page 241 TX packets:141 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:218606 (213.4 KiB) TX bytes:17483 (17.0 KiB) Interrupt:17 Setup a static OR default route towards T5 router to reach 102.1.1.0/24 network [root@host]# route add -net 102.1.1.0/24 gw 102.1.2.250 Chelsio T5/T4 Unified Wire For Linux Page 241...
  • Page 242 [root@host]# cxgbtool eth0 filter 0 lip 102.1.2.0/24 hitcnts 1 action switch eport 1 smac 00:07:43:04:96:48 dmac 00:07:43:12:D4:88 smac dmac Where, is the MAC address of eth1 interface on T5 adapter machine and is the MAC address of eth0 interface on Node 2. Chelsio T5/T4 Unified Wire For Linux Page 242...

  • Page 243: Layer 2 Example

    Here’s an example on how to achieve L2 switching functionality. The following will only work on kernel 3.10 and above. eth0 eth1 promisc promisc 102.1.1.1/24 102.1.1.2/24 eth0 eth0 Node 1 Node 2 Chelsio T5/T4 Unified Wire For Linux Page 243...
  • Page 244 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:1961 errors:0 dropped:2 overruns:0 frame:0 TX packets:141 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:218606 (213.4 KiB) TX bytes:17483 (17.0 KiB) Interrupt:17 Chelsio T5/T4 Unified Wire For Linux Page 244...
  • Page 245 Create a filter to match incoming packet's dst-mac 00:07:43:12:d4:88 with particular mac-idx and switch it out via eport 1: [root@host]# cxgbtool eth0 filter 0 macidx 5 action switch eport 1 hitcnts 1 Chelsio T5/T4 Unified Wire For Linux Page 245...
  • Page 246 Create a filter to match incoming packet's dst-mac 00:07:43:04:7d:50 with particular mac-idx and switch it out via eport 0: [root@host]# cxgbtool eth0 filter 1 macidx 7 action switch eport 0 hitcnts 1 Chelsio T5/T4 Unified Wire For Linux Page 246...

  • Page 247: Hash/Ddr Filters

    : Ingress packets will be passed through set ingress queues switch : Ingress packets will be routed to an output port with optional header rewrite. drop : Ingress packets will be dropped. Chelsio T5/T4 Unified Wire For Linux Page 247...

  • Page 248: Listing Filter Rules

    For more information on additional parameters, refer cxgbtool manual by running Note man cxgbtool command. 3.2. Listing Filter Rules  To list the Hash/DDR filters set, run the following command: [root@host]# cat /proc/drivers/cxgb4/<bus-id>/hash_filters Chelsio T5/T4 Unified Wire For Linux Page 248...

  • Page 249: Removing Filter Rules

    3.4. Swap MAC feature Chelsio T5’s Swap MAC feature swaps packet source MAC and destination MAC addresses. This is applicable only for switch filter rules. Here’s an example: [root@host]# cxgbtool eth2 filter 1 action switch lip 102.2.2.1 fip 102.2.2.2 lport 5001 fport 14000...

  • Page 250: Hit Counters

    4 different RSS Queues separately corresponding to 4 trace-filters. Enable Tracing as well as MultiRSSFilter [root@host]# cxgbtool ethX reg 0x09800=0x33 Setup a trace filter [root@host]# echo tx0 snaplen=40 > /proc/driver/cxgb4/<bus_id>/trace0 Chelsio T5/T4 Unified Wire For Linux Page 250...
  • Page 251 [root@host]# echo rx1 snaplen=40 > /proc/driver/cxgb4/<bus_id>/trace3 [root@host]# cxgbtool ethX reg 0x09ff4=<Trace-Queue1- RspQ ID > [root@host]# cxgbtool ethX reg 0x09ffc=<Trace-Queue2- RspQ ID > [root@host]# cxgbtool ethX reg 0x0a004=<Trace-Queue3- RspQ ID > Use “snaplen=60” in case of IPV6. Note Chelsio T5/T4 Unified Wire For Linux Page 251...

  • Page 252: Traffic Management

    XIX. Traffic Management XIX. Traffic Management Chelsio T5/T4 Unified Wire For Linux Page 252...

  • Page 253: Introduction

    XIX. Traffic Management 1. Introduction Traffic Management capabilities built-in to Chelsio T5/T4 CNAs can shape transmit data traffic through the use of sophisticated queuing and scheduling algorithms built-in to the ASIC hardware which provides fine-grained software control over latency and bandwidth parameters such as packet rate and byte rate.

  • Page 254: Software Requirements

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Peformed. Chelsio T5/T4 Unified Wire For Linux Page 254...

  • Page 255: Software/Driver Loading

    The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. Run the following commands to load the drivers: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Chelsio T5/T4 Unified Wire For Linux Page 255...

  • Page 256: Software/Driver Unloading

    XIX. Traffic Management 3. Software/Driver Unloading Reboot the system to unload the driver. Chelsio T5/T4 Unified Wire For Linux Page 256...

  • Page 257: Software/Driver Configuration And Fine-Tuning

    (Kbps) for this TCP stream.  Packet size is the TCP mss size in bytes; for example – for an MTU of 1500, use a packet size of 1460. Chelsio T5/T4 Unified Wire For Linux Page 257...
  • Page 258 Channel No. is the port on which data is flowing (0-3).  maximum-rate is the bit rate (Kbps) for this TCP stream. The lower limit is 300 Mbps and 600 Mbps for T4 and T5 adapters respectively. Chelsio T5/T4 Unified Wire For Linux Page 258...

  • Page 259: Configuring Traffic Management

    TX queue(s) will be unbound from any current TX Scheduler Class binding. 4.2.2. For Offloaded connections Traffic Management of offloaded connections can be configured either by applying COP policies that associate offloaded connections to classes or by modifying the application. Chelsio T5/T4 Unified Wire For Linux Page 259...
  • Page 260 For more information on additional parameters, refer cop manual by running the Note man cop command. Modifying the application The application can also be modified in order to associate connections to scheduling classes. Follow the steps mentioned below: Chelsio T5/T4 Unified Wire For Linux Page 260...
  • Page 261 Here:  sockfd : The file descriptor of the TCP socket  &cl : Pointer to the class variables  sizeof(cl) : The size of the variable 5. Now, compile the application. Chelsio T5/T4 Unified Wire For Linux Page 261...

  • Page 262: Usage

    The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for offloaded connections: 1. Load the Network driver in TOE mode [root@host]# modprobe t4_tom Chelsio T5/T4 Unified Wire For Linux Page 262...

  • Page 263: Offloaded Connections With Modified Application

    2. Modify the application as mentioned in the Configuring Traffic Management section. 3. Set the appropriate rule for class 0 [root@host]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 Chelsio T5/T4 Unified Wire For Linux Page 263...

  • Page 264: Unified Wire Manager (Um)

    XX. Unified Wire Manager (UM) XX. Unified Wire Manager (UM) Chelsio T5/T4 Unified Wire For Linux Page 264...

  • Page 265: Introduction

    Single tool with CLI and Web interface that works across Storage, Networking and Hardware. Manage all Chelsio adapters installed on the system. Tool for FAE to debug issues on the Customer front. Freedom to choose various modes of management i.e. CLI or Web GUI. Chelsio T5/T4 Unified Wire For Linux Page 265...

  • Page 266: Reference Architecture

    1.3.2. Management Client The Management Client can be used to connect to agents and manage them. Once connected you can view and configure Chelsio HBAs and related networking, storage and hardware Chelsio T5/T4 Unified Wire For Linux Page 266...

  • Page 267: Authentication And Encryption

    CLI client or the Web GUI and the agent, are encrypted using SSL. This ensures that the data cannot be accessed when it is being transmitted over the network. Chelsio T5/T4 Unified Wire For Linux Page 267...

  • Page 268: Hardware And Software

    T520-CR T522-CR T580-LP-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX S302E S302E-C S310E-CR S310E-CR-C S310E-CXA S310E-SR+ S310E-SR S310E-BT S320E-CR S320E-LP-CR S320E-CXA S320EM-BS S320EM-BCH N320E-G2-CR N320E N320E-CXA N320E-BT N310E N310E-CXA Chelsio T5/T4 Unified Wire For Linux Page 268...

  • Page 269: Platform/Component Matrix

    T3 : 2.0.0.1 (RHEL 5.8,6.3) T4,T5: 2.10.0.0 Bypass T4: 2.10.0.0 T3 : 2.0.0.1 Bonding T4: 3.7.1 T3: 5.2.0-0560 iSCSI Target T4,T5: 2.10.0.0-783 Open iSCSI Initiator T3,T4,T5: 2.0-872 iWARP T3,T4,T5: 2.10.0.0 FCoE Initiator T4: 2.10.0.0 Chelsio T5/T4 Unified Wire For Linux Page 269...

  • Page 270: Installing Unified Wire Manager

    Chelsio Unified Wire has been designed to install Unified Wire Manager (UM) by default. All the three UM components, i.e. Management Agent, Client and Station, will be installed on selecting any of the Terminator 4/Terminator 5 configuration tuning options during installation. Hence, no separate installation is required. Chelsio T5/T4 Unified Wire For Linux Page 270...

  • Page 271: Verifying Um Components Status

    2. You can also execute the following comand to determine if Management Agent is running: [root@chelsio]# /etc/init.d/chelsio-uwire_mgmtd status The above command should display the following result: Unified Wire Manager Agent : Running Chelsio T5/T4 Unified Wire For Linux Page 271...

  • Page 272: Verifying Management Client

    While accessing the Web Management Interface, if an error “Management Note Station is not running” is displayed at the log-in page, follow the above steps to restart Management Station and try again. Chelsio T5/T4 Unified Wire For Linux Page 272...

  • Page 273: Management Agent

    Execute the following command to list the service configuration. [root@host]# chkconfig --list chelsio-uwire_mgmtd Execute the following command to enable/disable the service to start at system runlevel 5. [root@host]# chkconfig --level 5 chelsio-uwire_mgmtd on/off Chelsio T5/T4 Unified Wire For Linux Page 273...

  • Page 274: Firewall

    If there is a firewall appliance / software protecting the network that the system is on, and you wish to connect to the system from a different network, using the client, the firewall appliance also needs to be configured appropriately. Chelsio T5/T4 Unified Wire For Linux Page 274...

  • Page 275: Cli Client

    Also, two different Web GUI or CLI clients cannot connect to a management agent at the same time. There is no mechanism to allow this scenario. Chelsio T5/T4 Unified Wire For Linux Page 275...

  • Page 276: Web Gui Client

    [root@host]# openssl rsa -in server.key.org -out server.key [root@host]# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt [root@host]# cp server.crt ./ssl.crt [root@host]# cp server.key ./ssl.key 2. Start/Restart Apache services [root@host]# rcapache2 [start|restart] Chelsio T5/T4 Unified Wire For Linux Page 276...

  • Page 277: Accessing Web Management Interface

    2. The security certificate used by the web server is a generic one. It may cause the following types of prompts in different browsers. You will need to select the correct option to continue. Security Certificate prompt in Internet Explorer Figure 7.2 (a) - Chelsio T5/T4 Unified Wire For Linux Page 277...
  • Page 278 XX. Unified Wire Manager (UM) Security Certificate prompt in Mozilla Firefox Figure 7.2 (b) - Security Certificate prompt in Apple Safari Figure 7.2 (c) - Security Certificate prompt in Google Chrome Figure 7.2 (d) - Chelsio T5/T4 Unified Wire For Linux Page 278...
  • Page 279 Figure 7.2 (e) - Web GUI Login page Not performing any operation/action for 5 minutes will result in session timeout. Note You will have to re-login and connect to the Agents again. Chelsio T5/T4 Unified Wire For Linux Page 279...

  • Page 280: Layout And Navigation

    Navigation Pane. The panels can be expanded and collapsed by clicking on the panel heading.  The Bottom bar has the About link on the right and copyright details on the left. Figure 7.3 – Web Management Interface Chelsio T5/T4 Unified Wire For Linux Page 280...

  • Page 281: Home Page

    The passwords are encrypted, but it is still advisable to store passwords only if the system you are running the GUI client on, is secure. Figure 7.4.1 (a) - Bookmarks and history module Chelsio T5/T4 Unified Wire For Linux Page 281...
  • Page 282 Once removed, the system will no longer appear in the Bookmarks and History Note module. If you wish to manage that system again, you will have to use the Add system option. Chelsio T5/T4 Unified Wire For Linux Page 282...
  • Page 283 The Add Agents button adds the selected system to the list of discovered agents in the Bookmarks and History module. The Clear Agents button resets the list of discovered agents. Figure 7.4.1 (b) - Service Discovery module Chelsio T5/T4 Unified Wire For Linux Page 283...
  • Page 284 Installation section. 4. Download the appropriate driver from Chelsio’s Download Center, service.chelsio.com. 5. Locate the driver package. 6. Click Install button to install the driver. Figure 7.4.1 (c) – Bulk Driver Installation module Chelsio T5/T4 Unified Wire For Linux Page 284...
  • Page 285 If you deselect the ‘Remember Password’ option, you will be asked to enter password every time you try to connect to the system. Figure 7.4.2 - Adding a UM Agent Chelsio T5/T4 Unified Wire For Linux Page 285...
  • Page 286 Recipients field separated a comma. Enter Mail server details and ensure that the “Enable email Alerts” field is enabled. Select the Agent(s), for which you want to receive alerts and Click on Save. Chelsio T5/T4 Unified Wire For Linux Page 286...
  • Page 287 Various configurable parameters have been categorized into several modules like Boot Configuration module to install and erase option ROM, Network Configuration module to set MTU and VLAN ID, Card Configuration module to change driver parameters, etc. Chelsio T5/T4 Unified Wire For Linux Page 287...
  • Page 288 SNO field and finally click Delete Member(s). Figure 7.4.6 (a) – Creating a group Chelsio T5/T4 Unified Wire For Linux Page 288...
  • Page 289 Using this module, you can install option ROM or erase option ROM on Chelsio network devices. The Set Default Boot Settings button will reset the adapter to factory boot settings. Figure 7.4.6 (c) – Boot Configuration module Chelsio T5/T4 Unified Wire For Linux Page 289...
  • Page 290 MTU can be set between 1500-9000 bytes. VLAN id can be set for an adapter within the range 0-4094 (enter 0 to disable it). Figure 7.4.6 (d) – Network Configuration module Chelsio T5/T4 Unified Wire For Linux Page 290...
  • Page 291 Offload. Offload settings are only available when using the TOE capable drivers (t4_tom and toecore for T5 and T4 adapters; t3_tom and toecore for T3 adapters). Figure 7.4.6 (e) – Card Configuration module Chelsio T5/T4 Unified Wire For Linux Page 291...
  • Page 292 In the Port Configuration module, you can set various port settings like enabling Tx checksum and TCP segmentation offload, setting Link speed and link duplex mode, etc. The settings depend on the device driver installed. Figure 7.4.6 (f) - Port Configuration module Chelsio T5/T4 Unified Wire For Linux Page 292...
  • Page 293 Use the Bypass Configuration module to configure Chelsio’s bypass adapters like B420-SR and B404-BT. For more information on different bypass modes and configurational parameters, Bypass Driver chapter. Figure 7.4.6 (g) - Bypass Configuration module Chelsio T5/T4 Unified Wire For Linux Page 293...

  • Page 294: System Page

    The system page contains generic system and support modules which are discussed below: 7.5.1. System Summary This module lists the system Hostname, Operating System, platform and also gives the count of the Chelsio cards found. Figure 7.5.1 - System Summary module Chelsio T5/T4 Unified Wire For Linux Page 294...
  • Page 295 Using this module, one can install various Chelsio drivers for different operating systems. You can choose the configuration file type (Linux Agents only). Figure 7.5.2 (a) - Drivers Installation module connected to Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 295...
  • Page 296 XX. Unified Wire Manager (UM) Figure 7.5.2 (b) - Drivers Installation module connected to FreeBSD Agent Chelsio T5/T4 Unified Wire For Linux Page 296...
  • Page 297 To reload a driver select Reload. Finally click Load/Unload Driver button. Click Refresh if changes are not reflected immediately. To reject the load/unload option selected, click Discard Changes. Figure 7.5.3 - Driver Details module Chelsio T5/T4 Unified Wire For Linux Page 297...
  • Page 298 (type of test; you can run more than one test at a time) and click Run Test. After the tests are completed, the results will be displayed in a tabular format. Figure 7.5.4 - System Diagnostics module for a T4 CNA Chelsio T5/T4 Unified Wire For Linux Page 298...
  • Page 299 KVM modules (kvm and kvm_intel/kvm_amd) are loaded. If not done already, access Agent CLI and follow these steps to do so: If any or all the aforementioned drivers are already loaded, unload them before proceeding: [root@host]# rmmod <kvm_intel/kvm_amd> [root@host]# rmmod kvm [root@host]# rmmod cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 299...
  • Page 300 The VM Configurations module allows you to view UUID and domain state of Virtual Machines. You can perform various system power options like start or resume (if VM is paused), turn off, restart or suspend (pause) a VM. Chelsio T5/T4 Unified Wire For Linux Page 300...
  • Page 301 Now, click on any of the system power actions provided at the bottom. Figure 7.5.7 (a) – VM Configurations module Figure 7.5.7 (b) – VM Configurations module: viewing properties for multiple VMs Chelsio T5/T4 Unified Wire For Linux Page 301...
  • Page 302 The VF Configurations module lists all the VMs, Virtual Functions mapped to each Virtual Machine and all the available VFs. You can also add and remove VFs for a particular VM. Figure 7.5.8 - VF Configurations module Chelsio T5/T4 Unified Wire For Linux Page 302...
  • Page 303 Now, click on any of the system power actions provided at the bottom. Figure 7.5.9 (a) – Xen Configurations module Figure 7.5.9 (b) – Xen Configurations module: viewing properties for multiple VMs Chelsio T5/T4 Unified Wire For Linux Page 303...
  • Page 304 Virtual Functions (VFs) you want to create for the physical ports. If VFs are not listed, unload the driver, reload it and create VFs again. Chelsio T5/T4 Unified Wire For Linux Page 304...
  • Page 305 Use the Get Logs button to retrieve, and Hide Logs button to clear the log entries. The Delete Logs button will remove the logs permanently from the agent. Figure 7.5.11 (a) - Managed System Application Logs module for Windows Agent Chelsio T5/T4 Unified Wire For Linux Page 305...

  • Page 306: Network Page

    The modules available on this page are as below: Card summary This module provides PCI, firmware and other details of the card. The card's serial number and factory MAC address are also provided for inventory purposes. Chelsio T5/T4 Unified Wire For Linux Page 306...
  • Page 307 XX. Unified Wire Manager (UM) Figure 7.6.2 (a) - Card Summary module Chelsio T5/T4 Unified Wire For Linux Page 307...
  • Page 308 Highlight the system item in the tree menu on the left, and click Refresh, to refresh data from the system, in case the updated settings are not being shown. Figure 7.6.2 (b) – TCP Offload Settings module for a Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 308...
  • Page 309 Figure 7.6.2 (c) - TCP Offload Settings module for a FreeBSD Agent Device Driver settings (Windows) The device driver settings applicable to the card are shown here. For Chelsio T5 and T4 adapters, only the MaxVMQueues field will be displayed. On changing the settings, the changed settings may not reflect immediately on refreshing the data.
  • Page 310 TCP/IP offload capability. The statistics are for TCP and IP protocol processing done in the card's hardware. These statistics may only be applicable if the card is TOE enabled. Figure 7.6.2 (e) - Card Statistics module for a T3 HBA Chelsio T5/T4 Unified Wire For Linux Page 310...
  • Page 311 The modules available on this page are as below: Port summary Port details such as the Ethernet adapter nam, link status, etc are shown in this module. Figure 7.6.2.1 (a) - Port Summary of T5/T4 CNA on Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 311...
  • Page 312 Port settings such as MTU, Link speed and others can be set in this module. The settings depend on the device driver installed. Figure 7.6.2.1 (b) - Port Settings of T4/T5 CNA on Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 312...
  • Page 313 Network teams are also indicated with an appropriate icon. The primary IP address (IPv4) can be set for the adapter, when it is selected. There is an option to add/modify/delete additional IP Chelsio T5/T4 Unified Wire For Linux Page 313...
  • Page 314 0-4094 (enter 0 to disable it). You can use the View/Set IP addresses option to add, modify or delete IP aliases. Figure 7.6.3 (a) - System network configuration module Chelsio T5/T4 Unified Wire For Linux Page 314...
  • Page 315 XX. Unified Wire Manager (UM) Figure 7.6.3 (b) – Managing IP aliases Chelsio T5/T4 Unified Wire For Linux Page 315...
  • Page 316 Table view in the Advanced NIC characteristics. The polling time field sets the average time (in seconds) based on which the table/graph updates the report. Figure 7.6.3 (c) - System network statistics module Chelsio T5/T4 Unified Wire For Linux Page 316...
  • Page 317 XX. Unified Wire Manager (UM) Figure 7.6.3 (d) - Network Throughput Vs Time instant Graph Figure 7.6.3 (e) - Network Throughput Vs Time instant Table Chelsio T5/T4 Unified Wire For Linux Page 317...
  • Page 318 Windows, you may set multiple default gateways. Use the option to set multiple default gateways with caution, since it may cause the system to stop communicating with external networks. Figure 7.6.3 (f) - Default gateway and DNS configuration module for a FreeBSD Agent Chelsio T5/T4 Unified Wire For Linux Page 318...
  • Page 319 Chelsio cards. Do not mix third party cards and offload-enabled Chelsio cards in a single team. Figure 7.6.3 (g) - Create a network team/bond device module Chelsio T5/T4 Unified Wire For Linux Page 319...
  • Page 320 Use the GetConnections utility to view currently active TCP/UDP connections. Offload status for each connection is also displayed if protocol offload hardware is available. This is useful for troubleshooting any connectivity issues for clients to various services. Figure 7.6.3 (h) - Ping Utility Chelsio T5/T4 Unified Wire For Linux Page 320...
  • Page 321 XX. Unified Wire Manager (UM) Figure 7.6.3 (i) - TraceRoute Utility Chelsio T5/T4 Unified Wire For Linux Page 321...
  • Page 322 To delete a bridge, click on the bridge name and then click “Delete Bridge”. This will also delete the virtual interface attached to that bridge and the VM(s) to which the interface was assigned. Chelsio T5/T4 Unified Wire For Linux Page 322...
  • Page 323 To delete a bridge, click on the bridge name and then click “Delete Bridge”. If there are virtual network interfaces attached, you will have to detach them first. Chelsio T5/T4 Unified Wire For Linux Page 323...
  • Page 324 To assign interfaces on the host to a bridge, specify the bridge name and the host interface in the “Add Interface to Bridge” section. Click “Add”. You can add multiple interfaces to the same bridge using the aforementioned method. The Cost and Priority parameters are optional. Chelsio T5/T4 Unified Wire For Linux Page 324...
  • Page 325 XX. Unified Wire Manager (UM) Figure 7.6.3.1 (c) – Creating Bridge Figure 7.6.3.1 (d) – Adding Bridge to VM Figure 7.6.3.1 (e) – Adding interfafe to Bridge Chelsio T5/T4 Unified Wire For Linux Page 325...
  • Page 326 You can attach upto 64 virtual interfaces to a particular bridge. Device Number, Bridge Name and VM Name are the mandatory fields while creating a virtual interface. Figure 7.6.3.1 (f) – Creating Bridge Figure 7.6.3.1 (g) – Creating and attaching virtual interface Chelsio T5/T4 Unified Wire For Linux Page 326...
  • Page 327 This type of virtual network does is not bound to any physical network adapter and no access to external networks is provided. Chelsio T5/T4 Unified Wire For Linux Page 327...
  • Page 328 Once created, you can manage the virtual networks in the Virtual Network Manager module. Figure 7.6.3.1 (i) – Creating external virtual network Figure 7.6.3.1 (j) – Creating internal virtual network Figure 7.6.3.1 (k) – Creating private virtual network Chelsio T5/T4 Unified Wire For Linux Page 328...
  • Page 329 On Windows Agents, only T3 HBAs are supported currently. Parameters can be set per port. On FreeBSD Agents, only T4 CNAs are supported. iWARP parameter settings can be set only when iw_cxgbe driver is loaded. Chelsio T5/T4 Unified Wire For Linux Page 329...
  • Page 330 XX. Unified Wire Manager (UM) Figure 7.6.4 (a) – iWARP settings for T4 CNA for FreeBSD Agent Chelsio T5/T4 Unified Wire For Linux Page 330...
  • Page 331 XX. Unified Wire Manager (UM) Figure 7.6.4 (b) – iWARP settings for T4 CNA for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 331...
  • Page 332 The WD-UDP module lists the process ids (pid) of UDP traffic running on the agent and displays the corresponding statistics and attributes. Please ensure that WD-UDP traffic is running on the agent before accessing this Note module. Figure 7.6.5 (a) – WD-UDP Process Statistics Chelsio T5/T4 Unified Wire For Linux Page 332...
  • Page 333 XX. Unified Wire Manager (UM) Figure 7.6.5 (b) – WD-UDP Process Attributes Chelsio T5/T4 Unified Wire For Linux Page 333...
  • Page 334 Please ensure that WD-TOE traffic is running on the agent before accessing this Note module. Figure 7.6.5 (c) – WD-TOE Process Statistics Figure 7.6.5 (d) – WD-TOE Process Attributes Chelsio T5/T4 Unified Wire For Linux Page 334...

  • Page 335: Storage Page

    This module provides details about the driver installed; such as driver name and its version. The module also gives information about the number of FCoE enabled cards that are present on the machine. Figure 7.7.1 (a) – FCoE Initiator Summary module for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 335...
  • Page 336 Figure 7.7.1.1 (a) – FCoE Card Summary module FCoE Attributes Information such as Interrupt modes (MSI/MSI-X/INTx), SCSI mode and the card state are provided in this module. Figure 7.7.1.1 (b) – FCoE Attributes module Chelsio T5/T4 Unified Wire For Linux Page 336...
  • Page 337 FCoE Port Summary The SCSI adapter name and the underlying ENODE MAC address of the physical port can be found here. Figure 7.7.1.2 (a) – FCoE Port Summary module for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 337...
  • Page 338 (10G/1G) being used. Note that only class 3 service is supported by the initiator for now and the frame size is fixed to 2128 bytes as per spec. Figure 7.7.1.2 (b) – FCoE Port Attributes module for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 338...
  • Page 339 To delete a virtual port, select the option Delete and select the virtual port WWPN which you want to delete and click on delete. Figure 7.7.1.2 (c) – FCoE NPIV management module Chelsio T5/T4 Unified Wire For Linux Page 339...
  • Page 340 This module provides details about the discovered target such as target’s FC ID, WWPN and WWNN so that the user can identify the discovered target accordingly. Figure 7.7.1.3 (a) – FCoE Remote Port Attributes module Chelsio T5/T4 Unified Wire For Linux Page 340...
  • Page 341 For Linux, the SCSI address is displayed in H:C:T:L (Host:Channel:Target:Lun) format and for Windows, it is displayed in P:B:T:L(SCSI Port:Bus:Target:Lun) format. Figure 7.7.1.3 (b) – FCoE Remote Port Lun Details module Chelsio T5/T4 Unified Wire For Linux Page 341...
  • Page 342 FCoE Virtual Port Summary The SCSI adapter name and the underlying ENODE MAC address of the physical port can be found here. Figure 7.7.1.4 (a) – FCoE Virtual Port Summary module for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 342...
  • Page 343 (10G/1G) being used. Note that only class 3 service is supported by the initiator for now and the frame size is fixed to 2128 bytes as per spec. Figure 7.7.1.4 (b) – FCoE Virtual Port Attributes module Chelsio T5/T4 Unified Wire For Linux Page 343...
  • Page 344 LUN, SCSI address, and LUN address are provided. For Linux, the SCSI address is displayed in H:C:T:L (Host:Channel:Target:Lun) format and for Windows, it is displayed in P:B:T:L(SCSI Port:Bus:Target:Lun) format. Figure 7.7.1.4 (d) - FCoE Remote Port Lun Details module Chelsio T5/T4 Unified Wire For Linux Page 344...
  • Page 345 The CHAP authentication secret should be between 12 and 16 characters in length, and the initiator's IQN name should start with "iqn." Figure 7.7.2 (a) - Open iSCSI initiator Chelsio T5/T4 Unified Wire For Linux Page 345...
  • Page 346 XX. Unified Wire Manager (UM) Figure 7.7.2 (b) - Microsoft iSCSI initiator Chelsio T5/T4 Unified Wire For Linux Page 346...
  • Page 347 XX. Unified Wire Manager (UM) Figure 7.7.2 (c) - Chelsio iSCSI initiator Chelsio T5/T4 Unified Wire For Linux Page 347...
  • Page 348 (the IP address and TCP Port specified when discovering the target). If there are any pre-existing iSCSI sessions established to the target, deletion of Note the target Portal from the discovered targets list will fail. Figure 7.7.2 (d) - Discover targets module Chelsio T5/T4 Unified Wire For Linux Page 348...
  • Page 349 XX. Unified Wire Manager (UM) Figure 7.7.2 (e) - Discover targets module Chelsio T5/T4 Unified Wire For Linux Page 349...
  • Page 350 In Windows, you can specify the initiator HBA to use and its IP address while logging in. Figure 7.7.2 (f) - Targets module after logging in Chelsio T5/T4 Unified Wire For Linux Page 350...
  • Page 351 You can also view details like link id, status, enode mac, etc of all the ports of the selected adapter. Figure 7.7.3 (a) - Full Offload iSCSI Hardware Information module Chelsio T5/T4 Unified Wire For Linux Page 351...
  • Page 352 The Port Up and Port Down buttons will enable and disable the selected port respectively. The Clear IP button deletes values set for the IP Type, IP, Subnet Mask and Gateway properties and resets them. Figure 7.7.3 (b) - FO iSCSI Manage Ports module Chelsio T5/T4 Unified Wire For Linux Page 352...
  • Page 353 In the FO iSCSI Initiator Properties module, you can configure FO iSCSI Initiator by setting different properties like enabling/disabling CHAP authentication, setting Header and Data digest, etc. Figure 7.7.3 (c) - FO iSCSI Initiator Properties Chelsio T5/T4 Unified Wire For Linux Page 353...
  • Page 354 In this module, you can set upto 8 instances. Configurable parameters include initiator node name (IQN), alias (friendly) name, Initiator (CHAP) Username and password. Figure 7.7.3 (d) - FO iSCSI Manage Instances module Chelsio T5/T4 Unified Wire For Linux Page 354...
  • Page 355 (target) IP. Finally, click Discover. After successful discovery, all the discovered targets will appear in the Discovered Targets section. To view more details, click on the Target name. Figure 7.7.3 (e) - FO iSCSI Discover Details module Chelsio T5/T4 Unified Wire For Linux Page 355...
  • Page 356 CHAP for a secure iSCSI connection. One-way (target authenticates the initiator) and Mutual (target and initiator authenticate each other) authentication methods are supported. Figure 7.7.3 (f) - FO iSCSI Session Details module: Login Chelsio T5/T4 Unified Wire For Linux Page 356...
  • Page 357 Established sessions section. Select the Adapter and session id. Details of the selected session will be displayed. To end the session, click Logout. Figure 7.7.3 (g) - FO iSCSI Session Details module: Established Sessions Chelsio T5/T4 Unified Wire For Linux Page 357...
  • Page 358 Target Stack Globals This module displays various global properties of a currently connected iSCSI target. Authentication priority between CHAP and ACL can be set here. Figure 7.7.4 (a) - Target Stack Globals module Chelsio T5/T4 Unified Wire For Linux Page 358...
  • Page 359 Properties such as Target name and Alias, Max Data Receive Length, Authentication mode related to a specific iSCSI target can be viewed and modified here. iSCSI targets can be started/stopped or deleted. Chelsio T5/T4 Unified Wire For Linux Page 359...
  • Page 360 XX. Unified Wire Manager (UM) Figure 7.7.4 (b) - Target properties module Chelsio T5/T4 Unified Wire For Linux Page 360...
  • Page 361 XX. Unified Wire Manager (UM) Session details Details including Session ID, Initiator IQN and Connections List of all discovered and currently connected iSCSI targets are listed here. Figure 7.7.4 (c) - Session Details module Chelsio T5/T4 Unified Wire For Linux Page 361...
  • Page 362 XX. Unified Wire Manager (UM) New Target Creation New iSCSI target can be created here by specifying the Target IQN and Target Alias name. Figure 7.7.4 (d) - New Target Creation module Chelsio T5/T4 Unified Wire For Linux Page 362...
  • Page 363 View/Edit iSCSI Target LUNs This module displays various Logical Units created in an iSCSI Target. Selected LUNs can be deleted. Figure 7.7.5 (a) - View/Edit iSCSI Target LUNs module Chelsio T5/T4 Unified Wire For Linux Page 363...
  • Page 364 Device and RAM Disk Size etc. RW (Read-Write) and RO (Read Only) are the two kinds of permissions that can be set. If Ram Disk is selected, then a minimum of 16 MB should be provided. Figure 7.7.5 (b) - Adding a new LUN Chelsio T5/T4 Unified Wire For Linux Page 364...
  • Page 365 Portal List on the left displays details of the portal group on which an iSCSI target is listening and the related info is displayed on the right under Portal Details. Selected portals can be deleted. Figure 7.7.6 (a) - View/Edit iSCSI Target Portals module Chelsio T5/T4 Unified Wire For Linux Page 365...
  • Page 366 XX. Unified Wire Manager (UM) Add Portal New Portals can be added here by choosing the specific target and Portal IP address. The Port number should be 3260. Figure 7.7.6 (b) - Adding a new Portal Chelsio T5/T4 Unified Wire For Linux Page 366...
  • Page 367 View/Edit iSCSI Target ACLs This module displays details for all the ACLs configured for an iSCSI Target. Selected ACLs can be deleted. Figure 7.7.7 (a) - Target ACL operations module Chelsio T5/T4 Unified Wire For Linux Page 367...
  • Page 368 XX. Unified Wire Manager (UM) Add ACL New ACLs can be configured by specifying Target name, initiator IQN name, IP address and permission type. Figure 7.7.7 (b) - Adding new ACL Chelsio T5/T4 Unified Wire For Linux Page 368...

  • Page 369: Hardware Features

    Protocol field are tcp, udp and frag. The Priority field can be used to determine the priority of a filter when Vlan ids are same. Insert at position features allows user to add a filter at a specified position. Chelsio T5/T4 Unified Wire For Linux Page 369...
  • Page 370 T5/T4 CARD fields are mandatory. After providing appropriate values for the parameters click “Save Changes”. For a detailed explanation regarding different fields, please refer cxgbtool Note manual by running man cxgbtool command on Management Agent CLI. Chelsio T5/T4 Unified Wire For Linux Page 370...
  • Page 371 Policies can be added and deleted. Policy Details displays the primitives (maximum 8) and actions which can be modified. For more details on creating policies, please refer to COP man pages. Chelsio T5/T4 Unified Wire For Linux Page 371...
  • Page 372 /ChelsioUwire- x.xx.x.x/src/network/firmware. For instance, to select Low latency Networking for T4 adapter, locate the file, t4-config.txt, in /ChelsioUwire-x.xx.x.x/src/network/firmware/low_latency_config directory. Chelsio T5/T4 Unified Wire For Linux Page 372...
  • Page 373 Figure 7.8.3 (a) – T4/T5 Save Config File module T5/T4 Boot Option ROM management This module allows managing the PXE and FCoE boot capability for Chelsio T5 and T4 adapters. The Option ROM (PXE and FCoE) may be installed to or erased from the card. The version of Option ROM flashed can be viewed here.
  • Page 374 Using the Boot Devices option you can set various iSCSI target properties. Enable Option ROM only if you are planning to boot the system via PXE or Note install the operating system on discovered iSCSI or FCoE LUN. Chelsio T5/T4 Unified Wire For Linux Page 374...
  • Page 375 XX. Unified Wire Manager (UM) Figure 7.8.3 (c) - PXE Boot configuration for T4 CNAs Chelsio T5/T4 Unified Wire For Linux Page 375...
  • Page 376 XX. Unified Wire Manager (UM) Figure 7.8.3 (d) - FCoE Boot configuration for T4 CNAs: Function parameter Figure 7.8.3 (e) - FCoE Boot configuration for T4 CNAs: Boot parameter Chelsio T5/T4 Unified Wire For Linux Page 376...
  • Page 377 XX. Unified Wire Manager (UM) Figure 7.8.3 (f) - FCoE Boot configuration for T4 CNAs: Show WWPN parameter Figure 7.8.3 (g) - iSCSI Boot configuration for T4 CNAs: Function parameter Chelsio T5/T4 Unified Wire For Linux Page 377...
  • Page 378 XX. Unified Wire Manager (UM) Figure 7.8.3 (h) - iSCSI Boot configuration for T4 CNAs: Initiator parameter Chelsio T5/T4 Unified Wire For Linux Page 378...
  • Page 379 XX. Unified Wire Manager (UM) Figure 7.8.3 (i) - iSCSI Boot configuration for T4 CNAs: Network parameter Chelsio T5/T4 Unified Wire For Linux Page 379...
  • Page 380 Current bypass mode. The Watchdog timer is used to ensure that if there is a software failure, the switch will enter the default state. The Watchdog timeout value should be provided in milliseconds. Chelsio T5/T4 Unified Wire For Linux Page 380...
  • Page 381 Purge table: Delete all the rules present in the selected table. This action will not delete the table.  Activate table: Enable the selected table.  Deactivate table: Disable the selected table. Chelsio T5/T4 Unified Wire For Linux Page 381...
  • Page 382 To delete a rule, select the corresponding checkbox and click Delete a Filter. Finally, click Save Changes. You can delete multiple rules using this method. For more information on Redirection, see Bypass Driver chapter. Figure 7.8.4 (b) - Redirect Configuration module: Table configuration tab Chelsio T5/T4 Unified Wire For Linux Page 382...
  • Page 383 XX. Unified Wire Manager (UM) Figure 7.8.4 (c) - Redirect Configuration module: Rules configuration tab Chelsio T5/T4 Unified Wire For Linux Page 383...
  • Page 384 Limiting, Class Weighted Round Robin and Channel Rate Limiting). Based on the parameters specified, different scheduler levels can be configured. To know more about the levels and related parameters, please refer Traffic Management chapter (Click here). 7.8.5 (b) – Egress Packet Scheduler module Chelsio T5/T4 Unified Wire For Linux Page 384...

  • Page 385: Uninstalling Unified Wire Manager

    1. Use the following query command to determine the name of the client RPM: [root@host]# rpm –qa | grep chelsio-uwire_mgmt-client 2. Now, execute the following command with the result from the above query to uninstall Management Client: E.g. for RHEL 6.3: [root@host]# rpm –e chelsio-uwire_mgmt-client-rhel6u3-2.2-xyz.x86_64 Chelsio T5/T4 Unified Wire For Linux Page 385...

  • Page 386: Uninstalling Management Station

    1. Use the following query command to determine the name of the Management Station RPM: [root@host]# rpm –qa | grep chelsio-uwire_mgmt-station 2. Now, execute the following command with the result from the above query to uninstall Management Station: E.g. for RHEL 6.3: [root@host]# rpm –e chelsio-uwire_mgmt-station-rhel6u3-2.2-xyz.x86_64 Chelsio T5/T4 Unified Wire For Linux Page 386...

  • Page 387: Unified Boot

    Chapter XXI. Unified Boot XXI. Unified Boot Chelsio T5/T4 Unified Wire For Linux Page 387...

  • Page 388: Introduction

    1.5 or higher. Otherwise the system will hang during POST. 1.1.2. Supported Switches Following is the list of network switches supported by Chelsio Unified Boot software: Cisco Nexus 5010 with 5.1(3)N1(1a) firmware. Arista DCS-7124S-F Chelsio T5/T4 Unified Wire For Linux Page 388...

  • Page 389: Software Requirements

    The Chelsio Driver Update Disk driver has been developed to run on 64-bit Linux platforms. Following is the list of Drivers/Software and supported Linux distributions: Linux Distribution Driver/Software (DUDs) RHEL6.5, 2.6.32-431.el6 PXE, iSCSI, FCoE SLES11SP3, 3.0.76-0.11 PXE,iSCSI, FCoE Chelsio T5/T4 Unified Wire For Linux Page 389...

  • Page 390: Flashing Firmware And Option Rom

    Make the USB flash drive as the primary boot device. Save the changes and exit. ix. Once the system boots from the USB flash drive, change your working directory to CHELSIO directory: C:\>cd CHELSIO Chelsio T5/T4 Unified Wire For Linux Page 390...
  • Page 391 CNA index found in step x (0 in this case) xii. Delete any previous firmware using the following command: C:\CHELSIO>cfut4 -d <idx> -xh –xf Here, is the CNA index found in step x (0 in this case) Chelsio T5/T4 Unified Wire For Linux Page 391...
  • Page 392 T4 adapters). C:\CHELSIO>cfut4 -d <idx> -uf <firmware_file>.bin firmware_file Here, is the CNA index found in step x (0 in this case) and is the firmware image file present in the CHELSIO folder. Chelsio T5/T4 Unified Wire For Linux Page 392...
  • Page 393 Delete any previous Option ROM settings: C:\CHELSIO>cfut4 -d <idx> -xc Here, is the CNA index found in step x (0 in this case) xvi. Reboot the system for changes to take effect. Chelsio T5/T4 Unified Wire For Linux Page 393...

  • Page 394: Using Cxgbtool

    Flash appropriate firmware using the following command (t4fw-x.xx.xx.x.bin for T4 adapters;t5fw-x.xx.xx.x.bin for T5 adapters): [root@host]# cxgbtool ethX loadfw <firmware_file>.bin Where, ethX is the Chelsio interface. vii. Reboot for changes to take effect. Chelsio T5/T4 Unified Wire For Linux Page 394...

  • Page 395: Configuring Pxe Server

    (Use port # 67, set PXE option and provide bootable file name in settings)  http://unattended.sourceforge.net/pxe-win2k.html Chelsio Communications does not take any responsibility regarding contents Note given in above mentioned links. Those are given for example purposes only. Chelsio T5/T4 Unified Wire For Linux...

  • Page 396: Pxe Boot Process

    Press [Alt+C] when the message Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2014 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 397 Enable the Adapter BIOS using arrow keys if not already enabled. Hit [ENTER]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose PXE from the list to configure. Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 397...
  • Page 398 Please note that enabling NIC Func 00 will enable port 0 for PXE, enabling NIC Func 01 will enable port 1 and so on for NIC function. Please refer the table below: Chelsio T5/T4 Unified Wire For Linux Page 398...
  • Page 399 Hit [F2] or [DEL] or any other key as mentioned during system startup to enter the system setup. xiii. Allow the Chelsio option ROM to initialize and setup PXE devices. DO NOT PRESS ALT-S to skip Chelsio option ROM. Chelsio T5/T4 Unified Wire For Linux Page 399...
  • Page 400 In the system setup, choose any of the Chelsio PXE devices as the first boot device. xv. Reboot. DO NOT PRESS ALT-S to skip Chelsio option ROM, during POST. xvi. Hit [F12] key when prompted to start PXE boot. Chelsio T5/T4 Unified Wire For Linux Page 400...

  • Page 401: Fcoe Boot Process

    Reboot the system. Press [Alt+C] when the message Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2014 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 402 Enable the Adapter BIOS if not already enabled. Hit [ENTER]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose FCoE from the list to configure and hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 402...
  • Page 403 Chapter XXI. Unified Boot vii. Choose the first option (function parameters) from the list of parameter type and hit [Enter]. viii. Enable FCoE BIOS if not already enabled. Chelsio T5/T4 Unified Wire For Linux Page 403...
  • Page 404 Chapter XXI. Unified Boot ix. Choose the order of the ports to discover FCoE targets. Set discovery timeout to a suitable value. Recommended value is >= 30. Chelsio T5/T4 Unified Wire For Linux Page 404...
  • Page 405 Chapter XXI. Unified Boot xi. Hit [F10] or [Esc] and then [Y] to save the configuration. xii. Choose boot parameters to configure. Chelsio T5/T4 Unified Wire For Linux Page 405...
  • Page 406 Select the first boot device and hit [Enter] to discover FC/FCoE targets connected to the switch. Wait till all reachable targets are discovered. xiv. List of discovered targets will be displayed. Highlight a target using the arrow keys and hit [Enter] to select. Chelsio T5/T4 Unified Wire For Linux Page 406...
  • Page 407 Chapter XXI. Unified Boot xv. From the list of LUNs displayed for the selected target, choose one on which operating system has to be installed. Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 407...
  • Page 408 Chapter XXI. Unified Boot xvi. Hit [F10] or [Esc] and then [Y] to save the configuration. xvii. Reboot the machine. xviii. During POST, allow the Chelsio option ROM to discover FCoE targets. Chelsio T5/T4 Unified Wire For Linux Page 408...
  • Page 409 Enter BIOS setup and choose FCoE disk discovered via Chelsio adapter as the first boot device. xx. Reboot and boot from the FCoE disk or install the required OS using PXE. Chelsio T5/T4 Unified Wire For Linux Page 409...

  • Page 410: Iscsi Boot Process

    Reboot the system. Press [Alt+C] when the message Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2014 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 411 Enable the Adapter BIOS if not already enabled. Hit [Enter]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose iSCSI from the list to configure and hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 411...
  • Page 412 For Windows, select IBFT, since Microsoft iSCSI Initiator will be used. You can also configure the number of iSCSI login attempts (retries) incase the network is unreachable or slow. Chelsio T5/T4 Unified Wire For Linux Page 412...
  • Page 413 Chapter XXI. Unified Boot ix. Choose the order of the ports to discover iSCSI targets. Set discovery timeout to a suitable value. Recommended value is >= 30. Chelsio T5/T4 Unified Wire For Linux Page 413...
  • Page 414 Chapter XXI. Unified Boot xi. Hit [Esc] and then [Y] to save the configuration. xii. Go back and choose Configure Initiator Parameters to configure initiator related properties. Chelsio T5/T4 Unified Wire For Linux Page 414...
  • Page 415 Inititiar properties like IQN, Header Digest, Data Digest, etc will be displayed. Change the values appropriately or continue with the default values. Hit [F10] to save. xiv. CHAP authentication is disabled by default. To enable and configure, go back and choose Configure CHAP Parameters Chelsio T5/T4 Unified Wire For Linux Page 415...
  • Page 416 Finally, provide Initiator and Target CHAP credentials according to the authentication method selected. Hit [F10] to save. xvi. Go back and choose Configure Network Parameters to configure iSCSI Network related properties. Chelsio T5/T4 Unified Wire For Linux Page 416...
  • Page 417 Select the port using which you want to connect to the target. Hit [Enter]. xviii. Select Yes in the Enable DHCP field to configure port using DHCP or No to manually configure the port. Hit [F10] to save. Chelsio T5/T4 Unified Wire For Linux Page 417...
  • Page 418 If you want to discover target using DHCP, select Yes in the Discover Boot Target via DHCP field. To discover target via static IP, select No and provide the target IP and Hit [F10] to save. The default TCP port selected is 3260. Chelsio T5/T4 Unified Wire For Linux Page 418...
  • Page 419 Chapter XXI. Unified Boot xxi. Go back and choose Discover iSCSI Target (s) to connect to a target. xxii. Select the portal group on which iSCSI service is provided by the target Chelsio T5/T4 Unified Wire For Linux Page 419...
  • Page 420 A list of available targets will be displayed. Select the target you wish to connect to and hit [Enter]. xxiv. A list of LUNs configured on the selected target will be displayed. Select the LUN you wish to connect to and hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 420...
  • Page 421 Chapter XXI. Unified Boot xxv. Hit [Esc] and then [Y] to save the configuration. xxvi. Reboot the machine. xxvii. During POST, allow the Chelsio option ROM to discover iSCSI targets. Chelsio T5/T4 Unified Wire For Linux Page 421...
  • Page 422 Enter BIOS setup and choose iSCSI target LUN discovered via Chelsio adapter as the first boot device. xxi. Reboot and boot from the iSCSI Target LUN or install the required OS using PXE. Chelsio T5/T4 Unified Wire For Linux Page 422...

  • Page 423: Creating Driver Update Disk (Dud)

    Creating DUD for Suse Enterprise Linux If you haven’t done already, download ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center, service.chelsio.com ii. Untar the package, [root@host]# tar -zxvfm ChelsioUwire-x.xx.x.x.tar.gz iii. Insert a blank USB flash drive. Chelsio T5/T4 Unified Wire For Linux Page 423...
  • Page 424 [root@host]# mkfs.vfat /dev/sda Depending on the distribution to be installed, copy the corresponding image file to the USB stick. For example, execute the following command for SLES11sp3. [root@host]# dd if=/root/ ChelsioUwire-x.xx.x.x/Uboot/LinuxDUD/Chelsio- DriverUpdateDisk-SLES11sp3-x86_64-x.x.x.x-y.img of=/dev/sda Chelsio T5/T4 Unified Wire For Linux Page 424...

  • Page 425: Os Installation

    Network. There may be built-in Chelsio driver in these distributions. The driver may or may not work with Chelsio adapters, depending on the adapter in use, and the version of the driver that shipped in that particular distribution. Please flash the firmware provided in the package. Chelsio T5/T4 Unified Wire For Linux Page 425...
  • Page 426 Driver Update Disk during the installation. ii. Driver disk prompt: The installer will load and prompt you for the driver update disk. Now select “Yes” and hit [Enter] to proceed. Chelsio T5/T4 Unified Wire For Linux Page 426...
  • Page 427 Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. Chelsio T5/T4 Unified Wire For Linux Page 427...
  • Page 428 The installer will ask if you wish to load more drivers. Choose “Yes” to load if you have any other drivers to load. Otherwise choose “No”. vi. Choose language and Keyboard type: Select the required language from the list. Chelsio T5/T4 Unified Wire For Linux Page 428...
  • Page 429 Select the type of keyboard you have from the list. viii. Select Installation method: In this step, you can choose the source which contains the OS installation ISO image. In this case, select “NFS directory”. Chelsio T5/T4 Unified Wire For Linux Page 429...
  • Page 430 Configure TCP/IP settings: Here you can specify if you want to configure your network interfaces using DHCP or manually using IPv4. IPv6 is currently not supported. Hence disable IPv6 before proceeding. Chelsio T5/T4 Unified Wire For Linux Page 430...
  • Page 431 Type dd at the boot prompt for the installation media. The dd option specifies that you will be providing a Driver Update Disk during the installation. The DUD gets detected and the driver is automatically loaded by installer. Chelsio T5/T4 Unified Wire For Linux Page 431...
  • Page 432 Chapter XXI. Unified Boot ii. Start Installation Select “Start Installation” and then “Start Installation or Update”. Chelsio T5/T4 Unified Wire For Linux Page 432...
  • Page 433 Select method of install: Select “Network” as the source of medium to install the SLES Operating System. iv. Select the Network protocol: Select the desired Network protocol from the list presented. Chelsio T5/T4 Unified Wire For Linux Page 433...
  • Page 434 [Alt] + [F3] or [Alt] + [F4]. Press [Alt] + [F1] to get back to the list. vi. Configure DHCP IP Select “Yes” to configure the network interface selected in the previous step using DHCP. Chelsio T5/T4 Unified Wire For Linux Page 434...

  • Page 435: Installation On Fcoe Lun

    Insert the OS installation disc into your CD/DVD ROM. ii. On the Grub menu, choose Install or upgrade an existing system option if not already selected. iii. Press Tab and type at the boot prompt. Chelsio T5/T4 Unified Wire For Linux Page 435...
  • Page 436 Chapter XXI. Unified Boot iv. Driver disk prompt: The installer will load and prompt you for the driver update disk. Now select “Yes” and hit [Enter] to proceed. Chelsio T5/T4 Unified Wire For Linux Page 436...
  • Page 437 Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. Chelsio T5/T4 Unified Wire For Linux Page 437...
  • Page 438 The installer will ask if you wish to load more drivers. Choose “Yes” to load if you have any other drivers to load. Otherwise choose “No” and hit [Enter]. viii. Testing Installation media If you want to test the installation media, choose OK or else Skip. Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 438...
  • Page 439 Chapter XXI. Unified Boot ix. Graphical Installer Red Hat graphical installer will now start. Click Next. Select Specialized Storage Devices radio button and click Next. Chelsio T5/T4 Unified Wire For Linux Page 439...

  • Page 440: Installation On Iscsi Lun

    Insert the OS installation disc into your CD/DVD ROM. ii. On the Grub menu, choose Install or upgrade an existing system option if not already selected. iii. Press Tab and type at the boot prompt. Chelsio T5/T4 Unified Wire For Linux Page 440...
  • Page 441 Chapter XXI. Unified Boot iv. Driver disk prompt: The installer will load and prompt you for the driver update disk. Now select “Yes” and hit [Enter] to proceed. Chelsio T5/T4 Unified Wire For Linux Page 441...
  • Page 442 Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. Chelsio T5/T4 Unified Wire For Linux Page 442...
  • Page 443 The installer will ask if you wish to load more drivers. Choose “Yes” to load if you have any other drivers to load. Otherwise choose “No” and hit [Enter]. xii. Testing Installation media If you want to test the installation media, choose OK or else Skip. Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 443...
  • Page 444 Chapter XXI. Unified Boot xiii. Graphical Installer Red Hat graphical installer will now start. Click Next. Chelsio T5/T4 Unified Wire For Linux Page 444...
  • Page 445 Select Specialized Storage Devices radio button and click Next. xv. The discovered LUNs will appear in the Basic Devices tab and you can proceed with installation as usual. If not, follow the steps mentioned below to add target. Chelsio T5/T4 Unified Wire For Linux Page 445...
  • Page 446 Click +Add Advanced Target button and select the Add iSCSI target radio button. Click +Add drive button. xvii. Enter the target IP. Initiator IQN will be autogenerated by the installer. Click Start Discovery. Chelsio T5/T4 Unified Wire For Linux Page 446...
  • Page 447 Select the checkbox corresponding to the LUN on which you wish to install operating system and click Login. Make sure the same LUN discovered at the Option ROM stage is selected for Note OS installation. Chelsio T5/T4 Unified Wire For Linux Page 447...
  • Page 448 Chapter XXI. Unified Boot xx. Next, choose the CHAP authentication method and click Login. xxi. On successful login, a dialog box will display the results along with the connected LUN name. Click OK. Chelsio T5/T4 Unified Wire For Linux Page 448...
  • Page 449 Chapter XXI. Unified Boot xxii. Select the LUN under the Other SAN Devices tab, and click Next. xxiii. Proceed with the installation as usual. Chelsio T5/T4 Unified Wire For Linux Page 449...

  • Page 450: Lustre File System

    Chapter XXII. Lustre File System XXII. Lustre File System Chelsio T5/T4 Unified Wire For Linux Page 450...

  • Page 451: Introduction

    Metadata servers (MDS), which manage the names and directories in the file system. Systems in a Lustre cluster 1.1. Hardware Requirements 1.1.1. Supported Adapters Currently Lustre File System is supported on following Chelsio Adapters: T502-BT T580-CR Chelsio T5/T4 Unified Wire For Linux Page 451...

  • Page 452: Software Requirements

    1.2.1. Linux Requirements Currently Lustre File System is supported on following distributions: Suse Linux Enterprise Server 11 SP1 kernel (SLES11SP1), 2.6.32.12-0.7 Other kernel versions have not been tested and are not guaranteed to work. Chelsio T5/T4 Unified Wire For Linux Page 452...

  • Page 453: Creating/Configuring Lustre File System

    Edit modprobe.conf file with appropriate interface name options lnet networks=tcp0(ethX),tcp1(ethY) //For NIC/TOE options lnet networks=o2ib0(ethX),o2ib1(ethY) //For iWARP where, ethX and ethY represent Chelsio interfaces. Load the following Lustre modules: [root@host]# modprobe lnet [root@host]# modprobe lustre Chelsio T5/T4 Unified Wire For Linux Page 453...
  • Page 454 [root@host]# mkfs.lustre --ost --fsname=<fsname> --mgsnode=<NID> <block device name> On Client node, follow steps (i)-(vi). Mount the Lustre file system on the client node by running the following command: [root@host]# mount -t lustre <MGS node>:/<fsname> <mount_point> Chelsio T5/T4 Unified Wire For Linux Page 454...

  • Page 455: Xxiii. Appendix A

    Chapter XXIII. Appendix XXIII. Appendix A Chelsio T5/T4 Unified Wire For Linux Page 455...

  • Page 456: Troubleshooting

    Please refer to the appropriate documentation for the Linux distribution on how to configure or disable the firewall. FCoE link not up Always enable LLDP on the interfaces as FCoE link won’t come up until and unless a successful LLDP negotiation happens. Chelsio T5/T4 Unified Wire For Linux Page 456...
  • Page 457 Ethernet interface part of both Ethernet VLAN and FCoE VLAN. Cisco nexus switch reporting “pauseRateLimitErrDisable” If in any case the switch-port on the Cisco nexus switch is reporting “pauseRateLimitErrDisable”, then perform an Ethernet port shut/no shut. Chelsio T5/T4 Unified Wire For Linux Page 457...

  • Page 458: Chelsio End-User License Agreement (Eula)

    IF YOU DO NOT AGREE TO THE TERMS OF THIS AGREEMENT, CLICK THE "DO NOT ACCEPT" BUTTON TO TERMINATE THE INSTALLATION PROCESS. 1. License. Chelsio Communications, Inc. ("Chelsio") hereby grants you, the Licensee, and you hereby accept, a limited, non-exclusive,...
  • Page 459 If the Software is acquired under the terms of a DoD or civilian agency contract, use, duplication or disclosure by the Government is subject to the restrictions of this Agreement in accordance with 48 C.F.R. 12.212 of the Chelsio T5/T4 Unified Wire For Linux Page 459...
  • Page 460 Should you have any questions concerning this Agreement, you may contact Chelsio by writing to: Chelsio Communications, Inc. 370 San Aleso Ave. Sunnyvale, CA 94085 Chelsio T5/T4 Unified Wire For Linux Page 460...

This manual is also suitable for:

Chelsio t4T580-crT520-ll-crT520-crT522-crT580-lp-cr ... Show all

Table of Contents