1. Manuals
  2. Brands
  3. Chelsio Communications Manuals
  4. Adapter
  5. Chelsio T5
  6. Installation and user manual

Chelsio Communications Chelsio T5 Installation And User Manual

Unified wire adapters for linux
Hide thumbs Also See for Chelsio T5:
1
2
3
Table Of Contents
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
Table of Contents

Advertisement

Quick Links

Download this manual See also: User Manual

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Chelsio T5 and is the answer not in the manual?

Questions and answers

Related Manuals for Chelsio Communications Chelsio T5

Summary of Contents for Chelsio Communications Chelsio T5

  • Page 2 For all support related questions please send email to support@chelsio.com Copyright © 2016.Chelsio Communications. All Rights Reserved. Chelsio ® is a registered trademark of Chelsio Communications. All other marks and names mentioned herein may be trademarks of their respective companies. Chelsio T5/T4 Unified Wire for Linux...
  • Page 3 1.0.8 03/08/2013 1.0.9 05/15/2013 1.1.0 07/26/2013 1.1.1 08/14/2013 1.1.2 12/06/2013 1.1.3 12/19/2013 1.1.4 03/13/2014 1.1.5 05/02/2014 1.1.6 06/30/2014 1.1.7 10/22/2014 1.1.8 11/04/2014 1.1.9 02/05/2015 1.2.0 03/04/2015 1.2.1 03/25/2015 1.2.2 06/03/2015 1.2.3 08/05/2015 1.2.4 02/29/2016 Chelsio T5/T4 Unified Wire for Linux...

  • Page 4: Table Of Contents

    Loading in TOE mode (with full offload support) Software/Driver Unloading 3.1. Unloading the NIC driver 3.2. Unloading the TOE driver Software/Driver Configuration and Fine-tuning 4.1. Instantiate Virtual Functions (SR-IOV) 4.2. Enabling Busy waiting 4.3. Performance Tuning III. VIRTUAL FUNCTION NETWORK (VNIC) Chelsio T5/T4 Unified Wire for Linux...
  • Page 5 Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning WD-UDP Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Accelerating UDP Socket communications VII. ISCSI PDU OFFLOAD TARGET Introduction Chelsio T5/T4 Unified Wire for Linux...
  • Page 6 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Configuring Cisco Nexus 5010 switch 4.2. Configuring the Brocade 8000 switch Running NIC & iSCSI Traffic together with DCBx Chelsio T5/T4 Unified Wire for Linux...
  • Page 7 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Offloading TCP traffic over a bonded interface XIII. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) Introduction 1.1. Hardware Requirements 1.2. Software Requirements Chelsio T5/T4 Unified Wire for Linux...
  • Page 8 Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Starting ba server 4.2. Bypass API (CLI) XVII. WD SNIFFING AND TRACING Theory of Operation 1.1. Hardware Requirements 1.2. Software Requirements Installation and Usage Chelsio T5/T4 Unified Wire for Linux viii...
  • Page 9 Traffic Management Rules 4.2. Configuring Traffic Management Usage 5.1. Non-Offloaded Connections 5.2. Offloaded Connections 5.3. Offloaded Connections with Modified Application UNIFIED WIRE MANAGER (UM) Introduction 1.1. Features 1.2. Reference Architecture 1.3. Unified Wire Manager Components Chelsio T5/T4 Unified Wire for Linux...
  • Page 10 Uninstalling Management Client 8.3. Uninstalling Management Station XXI. UNIFIED BOOT Introduction 1.1. Hardware Requirements 1.2. Software Requirements Flashing firmware and option ROM 2.1. Preparing USB flash drive 2.2. Legacy 2.3. uEFI Configuring PXE Server Chelsio T5/T4 Unified Wire for Linux...
  • Page 11 8.2. Installation on FCoE LUN 8.3. Installation on iSCSI LUN XXII. LUSTRE FILE SYSTEM Introduction 1.1. Hardware Requirements 1.2. Software Requirements Creating/Configuring Lustre File System XXIII. APPENDIX A Troubleshooting Chelsio End-User License Agreement (EULA) Chelsio T5/T4 Unified Wire for Linux...

  • Page 12: Chelsio Unified Wire

    Chapter I. Chelsio Unified Wire I. Chelsio Unified Wire Chelsio T5/T4 Unified Wire for Linux Page 12...

  • Page 13: Introduction

    Chapter I. Chelsio Unified Wire 1. Introduction Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip, single firmware cards provide enterprises and data centers with high performance solutions for various Network and Storage related requirements.

  • Page 14: Hardware Requirements

    1.2. Hardware Requirements The Chelsio Unified Wire software supports Chelsio T5 and T4 Series of Unified Wire Adapters. To know more about the list of adapters supported by each driver, please refer to their respective sections. 1.3.

  • Page 15: Package Contents

    Please find more details on man cxgbtool this tool in its manual page (run command). To use cxbtool for FCoE Initiator driver, use [root@host~]# cxgbtool stor -h Note Chelsio T5/T4 Unified Wire for Linux Page 15...
  • Page 16 The LinuxDUD directory contains image (.img) files required to update drivers for Linux distributions. 1.4.2. RPM package The Chelsio Unified Wire RPM package consists of the following: config: This directory contains T5/T4 firmware configuration files. Chelsio T5/T4 Unified Wire for Linux Page 16...
  • Page 17 ROM onto Chelsio's T5 and T4 based Converged Network Adapters (CNAs). The LinuxDUD directory contains image (.img) files required to update drivers for Linux distributions. Chelsio T5/T4 Unified Wire for Linux Page 17...

  • Page 18: Hardware Installation

    13. Verify if the adapter was installed successfully by using the lspci command a. For T5 adapters : [root@host~]# lspci |grep -i Chelsio 07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified...
  • Page 19 Storage Controller 03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000 For Chelsio T5/T4 adapters, the physical functions are currently assigned as: Physical functions 0 - 3: for the SR-IOV functions of the adapter Physical function 4: for all NIC functions of the adapter...
  • Page 20 However, for T5 40G and T420-BT adapters, the association of physical Ethernet ports and their corresponding network device names is opposite. For these adapters, the port nearest to the motherboard will appear as the first network interface. Chelsio T5/T4 Unified Wire for Linux Page 20...

  • Page 21: Software/Driver Installation

    NIC/TOE, WD-UDP, UM (Agent, Client, High Capacity WD number of WD-UDP connections. WebGUI) Configures T5 adapters to create more T5 Hash Filter* NIC, Filtering, UM (Agent, Client, WebGUI) filters. * Supported only on T5 adapters. Chelsio T5/T4 Unified Wire for Linux Page 21...

  • Page 22: Pre-Requisites

    GUI installer: [root@host~]# ./install.py iv. If Dialog utility is present, you can skip to step (v). If not, press ‘y’ to install it when the installer prompts for input. Chelsio T5/T4 Unified Wire for Linux Page 22...
  • Page 23 Select Enable IPv6-Offload to install drivers with IPv6 Offload support or Disable IPv6- offload to continue installation without IPv6 offload support. vii. Select the required T5/T4 configuration tuning option: The tuning options may vary depending on the Linux distribution. Note Chelsio T5/T4 Unified Wire for Linux Page 23...
  • Page 24 Retain IRQ balance daemon: Do not disable IRQ balance daemon. c. TX-Coalasce: Write tx_coal=2 to modprobe.d/conf. For more information on the Performance tuning options, please refer to Note Performance Tuning section of the Network (NIC/TOE) chapter. Chelsio T5/T4 Unified Wire for Linux Page 24...
  • Page 25 To install OFED-3.18-1 choose the Install-OFED option. To install OFED-3.12-1, select Choose-OFED-Version and then OFED-3.12-1. This step will be prompted only for OFED supported platforms. Note xi. The selected components will now be installed: Chelsio T5/T4 Unified Wire for Linux Page 25...
  • Page 26 Select “Yes” to exit the installer or “No” to go back. xv. Reboot your machine for changes to take effect. Press Esc or Ctrl+C to exit the installer at any point of time. Note Chelsio T5/T4 Unified Wire for Linux Page 26...
  • Page 27 If your system does not have Dialog or you choose not to install it, follow the steps mentioned below to install the Unified Wire package: Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center, http://service.chelsio.com/ Chelsio T5/T4 Unified Wire for Linux Page 27...
  • Page 28 Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed. Chelsio T5/T4 Unified Wire for Linux Page 28...
  • Page 29 Install the drivers, tools and libraries using the following command: [root@host~]# make install v. The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF=<T5/T4 configuration> [root@host~]# make CONF=<T5/T4 configuration> install Chelsio T5/T4 Unified Wire for Linux Page 29...
  • Page 30 You can also choose to install drivers individually. Provided here are steps to build and install NIC, TOE, iWARP, RDM Block Device, Bypass, WD-UDP, UDP Segmentation Offload, FCoE PDU Offload target, DCB drivers and benchmarking tools. To know about other drivers, view help by running make help. Chelsio T5/T4 Unified Wire for Linux Page 30...
  • Page 31 To build and install all drivers without IPv6 support: [root@host~]# make ipv6_disable=1 [root@host~]# make ipv6_disable=1 install  The above step will not install Bypass driver. Run the following commands to install it: [root@host~]# make bypass [root@host~]# make bypass_install Chelsio T5/T4 Unified Wire for Linux Page 31...
  • Page 32 To build and install drivers along with benchmarks: [root@host~]# make BENCHMARKS=1 [root@host~]# make BENCHMARKS=1 install  Unified Wire Manager will be installed by default. To skip the installation: [root@host~]# make INSTALL_UM=0 install Chelsio T5/T4 Unified Wire for Linux Page 32...

  • Page 33: Installing Chelsio Unified Wire From Rpm

    2. T5 Wire Direct Latency: Install Wire Direct Latency drivers needed for Low latency applications. 3. Custom: Customize the installation. Use this option to install drivers/software and related components (like OFED-3.18-1) according to the tuning option selected. Chelsio T5/T4 Unified Wire for Linux Page 33...
  • Page 34 : bypass drivers and tools udpso : UDP segmentation offload capable NIC and TOE drivers only : Wire Direct drivers and libraries only The Installation options may vary depending on the Linux distribution. Note Chelsio T5/T4 Unified Wire for Linux Page 34...

  • Page 35: Firmware Update

    The kernel must be configured to enable userspace firmware loading support: Device Drivers -> Generic Driver Options -> Userspace firmware loading support Chelsio T5/T4 Unified Wire for Linux Page 35...
  • Page 36 Chapter I. Chelsio Unified Wire The firmware version can be verified using ethtool: [root@host~]# ethtool -i <iface> Chelsio T5/T4 Unified Wire for Linux Page 36...

  • Page 37: Software/Driver Uninstallation

    GUI installer: [root@host~]# ./install.py ii. Select “uninstall” , Under “Choose an action” iii. Select “all” to uninstall all the installed drivers, libraries and tools or select “custom” to remove specific components. Chelsio T5/T4 Unified Wire for Linux Page 37...
  • Page 38 The selected components will now be uninstalled. v. After successful uninstalltion, summary of the uninstalled components will be displayed. vi. Select “View log” to view uninstallation log or “Exit” to continue. Chelsio T5/T4 Unified Wire for Linux Page 38...
  • Page 39 4.1.3.1. iWARP driver uninstallation on Cluster nodes To uninstal iWARP drivers on multiple Cluster nodes with a single command, run the following command: [root@host~]# ./install.py -C -m <machinefilename> -u all Chelsio T5/T4 Unified Wire for Linux Page 39...
  • Page 40 To uninstall iWARP driver: [root@host~]# make iwarp_uninstall  To uninstall RDMA Block Device driver: [root@host~]# make rdma_block_device_uninstall  To uninstall Bypass driver: [root@host~]# make bypass_uninstall  To uninstall UDP Segmentation Offload driver: [root@host~]# make udp_offload_uninstall Chelsio T5/T4 Unified Wire for Linux Page 40...

  • Page 41: Uninstalling Chelsio Unified Wire From Rpm

    [root@host~]# ./uninstall.py –h by typing for more information. Uninstalling Unified Wire package will not uninstall Unified Wire Manager. Refer Note to the Unified Wire Manager (UM) chapter to remove the software manually (Click here). Chelsio T5/T4 Unified Wire for Linux Page 41...
  • Page 42 To uninstal iWARP drivers on multiple Cluster nodes with a single command, run the following: [root@host~]# ./install.py -C -m <machinefilename> -u The above command will remove Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. Chelsio T5/T4 Unified Wire for Linux Page 42...

  • Page 43: Configuring Chelsio Network Interfaces

    5.1. Configuring 40G adapters Chelsio T5 40G adapters can be configured in the following three modes: 2X40Gbps: This is the default mode of operation where each port functions as 40Gbps link. The port nearest to the motherboard will appear as the first network interface (Port 0).
  • Page 44 Select mode for adapter (1,2,3): v. Reload the network driver for changes to take effect. [root@host~]# rmmod cxgb4 [root@host~]# modprobe cxgb4 In case of T580-SO-CR adapters, reboot the machine for changes to take effect. Note Chelsio T5/T4 Unified Wire for Linux Page 44...

  • Page 45: Configuring Network-Scripts

    NIC-only or the TOE-enabling driver). The output of ifconfig should display the T5/T4 interfaces as: [root@host~]# ifconfig -a | grep HWaddr eth0 Link encap:Ethernet HWaddr 00:30:48:32:6A:AA eth1 Link encap:Ethernet HWaddr 00:07:43:04:6B:E9 eth2 Link encap:Ethernet HWaddr 00:07:43:04:6B:F1 Chelsio T5/T4 Unified Wire for Linux Page 45...

  • Page 46: Checking Link

    First, bring up the interface you want to test using ifup eth1. You should now be able to ping any other machine from your network provided it has ping response enabled. Chelsio T5/T4 Unified Wire for Linux Page 46...

  • Page 47: Performance Tuning

    Run the following script to disable few services. [root@host~]# t4_latencytune.sh <interface> iii. Set sysctl param net.ipv4.tcp_low_latency to 1 [root@host~]# sysctl -w net.ipv4.tcp_low_latency=1 To optimize your system for different protocols, please refer to their respective chapters. Chelsio T5/T4 Unified Wire for Linux Page 47...

  • Page 48: Software/Driver Update

    For any distribution specific problems, please check README and Release Notes included in the release for possible workaround. Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various software/drivers. You can also subscribe to our newsletter for the latest software updates. Chelsio T5/T4 Unified Wire for Linux Page 48...

  • Page 49: Network (Nic/Toe)

    Chapter II. Network (NIC/TOE) II. Network (NIC/TOE) Chelsio T5/T4 Unified Wire for Linux Page 49...

  • Page 50: Introduction

    The following are the currently shipping Chelsio Adapters that are compatible with Chelsio Network driver: T580-OCP-SO* T520-OCP-SO* T520-BT T580-CR T580-SO-CR* T580-LP-CR T520-LL-CR T520-SO-CR* T520-CR T540-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX *Only NIC driver supported Chelsio T5/T4 Unified Wire for Linux Page 50...

  • Page 51: Software Requirements

    Ubuntu 14.04.3, 3.19.0-25-generic * Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire for Linux Page 51...

  • Page 52: Software/Driver Loading

    Offload support needs to be enabled upon each reboot of the system. This can Note be done manually as shown above. In VMDirect Path environment, it is recommended to load the offload driver using the following command: [root@host~]# modprobe t4_tom vmdirectio=1 Chelsio T5/T4 Unified Wire for Linux Page 52...

  • Page 53: Software/Driver Unloading

    Create a policy file which will disable offload [root@host~]# cat policy_file all => !offload b. Compile and apply the output policy file [root@host~]# cop –o no-offload.cop policy_file [root@host~]# cxgbtool ethX policy no-offload.cop Chelsio T5/T4 Unified Wire for Linux Page 53...
  • Page 54 Chapter II. Network (NIC/TOE) iii. Unload the driver: [root@host~]# rmmod t4_tom [root@host~]# rmmod toecore [root@host~]# rmmod cxgb4 Chelsio T5/T4 Unified Wire for Linux Page 54...

  • Page 55: Software/Driver Configuration And Fine-Tuning

    To make use of BUSY_POLL feature, follow the steps mentioned below: Enable BUSY_POLL support in kernel config file by setting CONFIG_NET_RX_BUSY_POLL=y ii. Enable BUSY_POLL globally in the system by setting the values of following sysctl parameters depending on the number of connections: Chelsio T5/T4 Unified Wire for Linux Page 55...

  • Page 56: Performance Tuning

    Now the receiving traffic is dynamically assigned to one of the system’s CPUs through a T5/T4 queue. This achieves a balanced usage among all the processors. This can be verified, for example, by using the iperf tool. First set up a server on the receiver host: Chelsio T5/T4 Unified Wire for Linux Page 56...
  • Page 57 CPU or interrupt entry. Queue 2 has caused a very large number of interrupts on CPU2 while CPU0 and CPU1 are barely used by any of the four queues. Enabling RSS is thus essential for best performance. Chelsio T5/T4 Unified Wire for Linux Page 57...
  • Page 58 -w net.ipv4.tcp_timestamps=0 sysctl -w net.ipv4.tcp_low_latency=1 sysctl -w net.core.netdev_max_backlog=250000 sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216 sysctl -w net.core.rmem_default=16777216 sysctl -w net.core.wmem_default=16777216 sysctl -w net.core.optmem_max=16777216 sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216' sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216' Chelsio T5/T4 Unified Wire for Linux Page 58...
  • Page 59 Please note that if your Linux system has IP forwarding enabled, i.e. acting as a bridge or router, the LRO needs to be disabled. This is due to a known kernel issue. Chelsio T5/T4 Unified Wire for Linux Page 59...
  • Page 60 LRO. As mentioned earlier, GRO is always the preferred option when both of them are present. On other systems LRO might be the only available option. Then ethtool could be used to switch LRO on and off as well. Chelsio T5/T4 Unified Wire for Linux Page 60...
  • Page 61 [root@host~]# ethtool -K eth6 gro off [root@host~]# ethtool -k eth6 Offload parameters for eth6: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off The output above shows a disabled GRO. Chelsio T5/T4 Unified Wire for Linux Page 61...

  • Page 62: Virtual Function Network (Vnic)

    Chapter III. Virtual Function Network (vNIC) III. Virtual Function Network (vNIC) Chelsio T5/T4 Unified Wire for Linux Page 62...

  • Page 63: Introduction

    1.1.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with the Chelsio vNIC driver: T520-BT T580-CR T520-LL-CR T520-CR T580-LP-CR T540-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T440-LP-CR T420-BT T420-LL-CR T420-CX Chelsio T5/T4 Unified Wire for Linux Page 63...

  • Page 64: Software Requirements

    Ubuntu 14.04.3, 3.19.0-25-generic * Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire for Linux Page 64...

  • Page 65: Software/Driver Loading

    The vNIC driver must be loaded on the Guest OS by the root user. Any attempt to load the driver as a regular user will fail. To load the driver, run the following command: [root@host~]# modprobe cxgb4vf Chelsio T5/T4 Unified Wire for Linux Page 65...

  • Page 66: Software/Driver Unloading

    The vNIC driver must be unloaded on the Guest OS by the root user. Any attempt to unload the driver as a regular user will fail. To unload the driver, execute the following command: [root@host~]# rmmod cxgb4vf Chelsio T5/T4 Unified Wire for Linux Page 66...

  • Page 67: Iwarp (Rdma)

    Chapter IV. iWARP (RDMA) IV. iWARP (RDMA) Chelsio T5/T4 Unified Wire for Linux Page 67...

  • Page 68: Introduction

    Software Requirements 1.2.1. Linux Requirements Currently the iWARP driver is available for the following versions: RHEL 7.2, 3.10.0-327.el7 * RHEL 7.1, 3.10.0-229.el7 RHEL 7.1 BE, 3.10.0-229.el7.ppc64 (POWER7) RHEL 7.1 LE, 3.10.0-229.ael7b.ppc64le (POWER8) Chelsio T5/T4 Unified Wire for Linux Page 68...
  • Page 69 Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work *Limited QA performed Chelsio T5/T4 Unified Wire for Linux Page 69...

  • Page 70: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the iWARP driver we need to load the NIC driver and core RDMA drivers first. Run the following commands: [root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm Chelsio T5/T4 Unified Wire for Linux Page 70...

  • Page 71: Software/Driver Unloading

    IMB benchmark stalls due to a shared memory Important BTL issue. This issue is fixed in openmpi-1.4.5 and later releases. Hence, it is recommended that you download and install the latest stable release from Open MPI's official website, http://www.open-mpi.org Chelsio T5/T4 Unified Wire for Linux Page 71...

  • Page 72: Software/Driver Configuration And Fine-Tuning

    CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA client DISCONNECT EVENT... Chelsio T5/T4 Unified Wire for Linux Page 72...

  • Page 73: Enabling Various Mpis

    Copy the license file (.lic file) into directory l_mpi_p_x.y.z iii. Create machines.LINUX (list of node names) in iv. Select advanced options during installation and register the MPI. v. Install software on every node. [root@host~]# ./install.py Chelsio T5/T4 Unified Wire for Linux Page 73...
  • Page 74 Those mca-btl-openib-device-params.ini can be found in the "Chelsio T4" section of . Follow the steps mentioned below to install and configure Open MPI. If not alreay done, install mpi-selector tool. Chelsio T5/T4 Unified Wire for Linux Page 74...
  • Page 75 ("" == "`echo $path | grep /usr/mpi/gcc/openmpi-x.y.z/bin`") then set path=(/usr/mpi/gcc/openmpi-x.y.z/bin $path) endif # LD_LIBRARY_PATH if ("1" == "$?LD_LIBRARY_PATH") then if ("$LD_LIBRARY_PATH" !~ */usr/mpi/gcc/openmpi-x.y.z/lib64*) then setenv LD_LIBRARY_PATH /usr/mpi/gcc/openmpi- x.y.z/lib64:${LD_LIBRARY_PATH} endif else setenv LD_LIBRARY_PATH /usr/mpi/gcc/openmpi-x.y.z/lib64 endif # MPI_ROOT setenv MPI_ROOT /usr/mpi/gcc/openmpi-x.y.z Chelsio T5/T4 Unified Wire for Linux Page 75...
  • Page 76 Register OpenMPI with MPI-selector: [root@host~]# mpi-selector --register openmpi --source-dir /usr/mpi/gcc/openmpi-x.y.z/bin ix. Verify if it is listed in mpi-selector: [root@host~]# mpi-selector --l x. Set OpenMPI: [root@host~]# mpi-selector --set openmpi –yes xi. Logut and log back in. Chelsio T5/T4 Unified Wire for Linux Page 76...
  • Page 77 ("" == "`echo $path | grep /usr/mpi/gcc/mvapich2-x.y/bin`") then set path=(/usr/mpi/gcc/mvapich2-x.y/bin $path) endif # LD_LIBRARY_PATH if ("1" == "$?LD_LIBRARY_PATH") then if ("$LD_LIBRARY_PATH" !~ */usr/mpi/gcc/mvapich2-x.y/lib64*) then setenv LD_LIBRARY_PATH /usr/mpi/gcc/mvapich2- x.y/lib64:${LD_LIBRARY_PATH} endif else setenv LD_LIBRARY_PATH /usr/mpi/gcc/mvapich2-x.y/lib64 endif # MPI_ROOT setenv MPI_ROOT /usr/mpi/gcc/mvapich2-x.y Chelsio T5/T4 Unified Wire for Linux Page 77...
  • Page 78 Add the following entries in .bashrc file: export MVAPICH2_HOME=/usr/mpi/gcc/mvapich2-x.y/ export MV2_USE_IWARP_MODE=1 export MV2_USE_RDMA_CM=1 viii. Register MPI: [root@host~]# mpi-selector --register mvapich2 --source-dir /usr/mpi/gcc/mvapich2-x.y/bin/ ix. Verify if it is listed in mpi-selector: [root@host~]# mpi-selector --l Chelsio T5/T4 Unified Wire for Linux Page 78...
  • Page 79 Follow steps (v), (vi) and (vii) for all the nodes. 4.2.4. Running MPI applications  Run Intel MPI applications as: mpdboot -n <no_of_nodes_in_cluster> -r ssh mpdtrace mpiexec -ppn -n 2 /opt/intel/impi/3.1/tests/IMB-3.1/IMB-MPI1 Chelsio T5/T4 Unified Wire for Linux Page 79...

  • Page 80: Setting Up Nfs-Rdma

     Run MVAPICH2 application as : mpirun_rsh -ssh -np 8 -hostfile mpd.hosts $MVAPICH2_HOME/tests/imb/IMB-MPI1 4.3. Setting up NFS-RDMA 4.3.1. Starting NFS-RDMA Server-side settings Follow the steps mentioned below to set up an NFS-RDMA server. Chelsio T5/T4 Unified Wire for Linux Page 80...
  • Page 81 Load the iwarp modules and make sure peer2peer is set to 1. Make sure you are able to ping and ssh to the server Chelsio interface through which directories will be exported. Chelsio T5/T4 Unified Wire for Linux Page 81...

  • Page 82: Performance Tuning

    Once the exported directories are listed, mount them as: [root@host~]# mount.nfs <serverip>:<directory> <mountpoint-on-client> -o vers=3,rdma,port=2050,wsize=65536,rsize=65536 4.4. Performance Tuning See performance tuning section in the Unified Wire chapter for generic performance settings. Chelsio T5/T4 Unified Wire for Linux Page 82...

  • Page 83: Rdma Block Device Driver (Rbd)

    Chapter V. RDMA Block Device Driver (RBD) V. RDMA Block Device Driver (RBD) Chelsio T5/T4 Unified Wire for Linux Page 83...

  • Page 84: Introduction

    SLES 12 SP1, 3.12.49-11-default * SLES 12, 3.12.28-4-default * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed Chelsio T5/T4 Unified Wire for Linux Page 84...

  • Page 85: Software/Driver Loading

    [root@host~]# modprobe rdma_ucm [root@host~]# modprobe rbdt On Initiator Run the following commands to load the RDMA block device driver on the initiator machine: [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm [root@host~]# modprobe rbdi Chelsio T5/T4 Unified Wire for Linux Page 85...

  • Page 86: Software/Driver Unloading

    [root@host~]# rmmod rdma_ucm [root@host~]# rmmod iw_cxgb4 On Initiator Run the following commands to unload the RDMA block device driver on the initiator machine: [root@host~]# rmmod rbdi [root@host~]# rmmod rdma_ucm [root@host~]# rmmod iw_cxgb4 Chelsio T5/T4 Unified Wire for Linux Page 86...

  • Page 87: Software/Driver Configuration And Fine-Tuning

    Run the following command on the initiator, to list all the targets available: [root@host~]# rbdctl -l E.g.: [root@host~]# rbdctl -l local /dev/rbdi0 102.1.1.105:49093 remote /dev/ram0 102.1.1.106:65000 state CONNECTED local /dev/rbdi1 102.1.1.105:49685 remote /dev/ram1 102.1.1.106:65000 state CONNECTED Chelsio T5/T4 Unified Wire for Linux Page 87...

  • Page 88: Wd-Udp

    Chapter VI. WD-UDP VI. WD-UDP Chelsio T5/T4 Unified Wire for Linux Page 88...

  • Page 89: Introduction

    1.2. Software Requirements 1.2.1. Linux Requirements Currently the WD-UDP driver is available for the following versions: RHEL 7.2, 3.10.0-327.el7 * RHEL 7.1, 3.10.0-229.el7 RHEL 6.7, 2.6.32-573.el6* RHEL 6.6, 2.6.32-504.el6* RHEL 6.5, 2.6.32-431.el6* Chelsio T5/T4 Unified Wire for Linux Page 89...
  • Page 90 Ubuntu 14.04.3, 3.19.0-25-generic * Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire for Linux Page 90...

  • Page 91: Software/Driver Loading

    RDMA core modules from the OFED package should be loaded before proceeding. To load the WD-UDP driver, use the following commands which will automatically load RDMA core modules: [root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm Chelsio T5/T4 Unified Wire for Linux Page 91...

  • Page 92: Software/Driver Unloading

    Chapter VI. WD-UDP 3. Software/Driver Unloading To unload the WD-UDP driver, run the following command: [root@host~]# rmmod iw_cxgb4 Chelsio T5/T4 Unified Wire for Linux Page 92...

  • Page 93: Software/Driver Configuration And Fine-Tuning

    Due to HW resource limitations, only a small number of queues can be allocated for UDP acceleration. Therefore only performance critical UDP applications should use libcxgb4_sock. Only 64 IPv4 UDP / 28 IPv6 UDP sockets can be accelerated per Chelsio T5/T4 device, with Unified Wire Configuration tuning option. If you want more sockets to be accelerated, please use Low Latency or High Capacity WD tuning option.
  • Page 94 Create this file on all systems using libcxgb4_sock. Here is the syntax: # Syntax: # endpoint {attributes} ... # where attributes include: interface = interface-name port = udp-port-number vlan = vlan-id priority = vlan-priority Chelsio T5/T4 Unified Wire for Linux Page 94...
  • Page 95 {interface = eth2.7 priority=3 port=9000} Now, preload libcxgb4_sock: [root@host~]# CXGB4_SOCK_CFG=<path to config file> LD_PRELOAD=libcxgb4_sock.so <pathto>/your_application In order to offload IPv6 UDP sockets, please select “low latency networking” as Note T5/T4 configuration tuning option during installation. Chelsio T5/T4 Unified Wire for Linux Page 95...
  • Page 96 The udp benchmark from the hpcbench suite can be used to show the benefits of libcxgb4_sock. The hpcbench suite can be found at: Source: http://hpcbench.sourceforge.net/index.html Sample: http://hpcbench.sourceforge.net/udp.html The nodes in this example, r9 and r10, have T5/T4 eth1 configured and the ports are connected point-to-point. Chelsio T5/T4 Unified Wire for Linux Page 96...
  • Page 97 Then on r9, we run udptest to port 9000 to see the host stack UDP latency: [root@r9 ~]# /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9000 Running the same test with libcxgb4_sock: [root@r9 ~]# LD_PRELOAD=libcxgb4_sock.so /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9001 Chelsio T5/T4 Unified Wire for Linux Page 97...
  • Page 98 To see if the application is being offloaded, open a window on one of the machines, and run tcpdump against the Chelsio interface. If you see minimal UDP output on the interface, then the UDP traffic is being properly offloaded. Chelsio T5/T4 Unified Wire for Linux Page 98...

  • Page 99: Iscsi Pdu Offload Target

    Chapter VII. iSCSI PDU Offload Target VII. iSCSI PDU Offload Target Chelsio T5/T4 Unified Wire for Linux Page 99...

  • Page 100: Introduction

    Offloads iSCSI Fast Data Path with Direct Data Placement (DDP)  Offloads iSCSI Header and Data Digest Calculations  Offload Speeds at 1 Gb, 10 Gb and 40Gb  Offloads TCP/IP for NAS simultaneously with iSCSI Target Specific features Chelsio T5/T4 Unified Wire for Linux Page 100...

  • Page 101: Hardware Requirements

    1.2.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with iSCSI PDU Offload Target software: T520-BT T580-CR T520-LL-CR T520-CR T580-LP-CR T540-CR T420-CR T440-CR T422-CR T404-BT T420-BCH T440-LP-CR Chelsio T5/T4 Unified Wire for Linux Page 101...

  • Page 102: Software Requirements

    Whereas in iscsi non-offloaded target (NIC) mode, only cxgb4 chiscsi_base.ko needed by module. 1.3.1. Linux Requirements Currently the iSCSI PDU Offload Target software is available for the following versions: RHEL 7.2, 3.10.0-327.el7 * Chelsio T5/T4 Unified Wire for Linux Page 102...
  • Page 103 RHEL6 as an example, or a complete tree. The source tree needs to be configured and the header files need to be compiled. Additionally, the Linux kernel must be configured to use modules. Chelsio T5/T4 Unified Wire for Linux Page 103...

  • Page 104: Software/Driver Loading

    It also provides control for iSCSI global settings. 1. Loading the Kernel module For RHEL distributions, run modprobe as follows: [root@host~]# modprobe chiscsi_t4 For SLES distributions, run modprobe as follows: [root@host~]# modprobe chiscsi_t4 --allow-unsupported Chelsio T5/T4 Unified Wire for Linux Page 104...
  • Page 105 It can also be used to stop the targets, and restart/reload configuration. For the script to execute properly, make sure the following flag is set on all Note kernel.org kernels. # CONFIG_MODULE_FORCE_LOAD=y Chelsio T5/T4 Unified Wire for Linux Page 105...

  • Page 106: Software/Driver Unloading

    Chapter VII. iSCSI PDU Offload Target 3. Software/Driver Unloading Use the following command to unload the module: [root@host~]# rmmod chiscsi_t4 Chelsio T5/T4 Unified Wire for Linux Page 106...

  • Page 107: Software/Driver Configuration And Fine-Tuning

    There are many specific parameters that can be configured, some of which are iSCSI specific and the rest being Chelsio specific. An example of an iSCSI specific item is “HeaderDigest” which is defaulted to “None” but can be overridden to “CRC32C”. An example of a Chelsio Chelsio T5/T4 Unified Wire for Linux Page 107...

  • Page 108: A Quick Start Guide For Target

    (i.e., the first device specified is assigned LUN 0, the second one LUN 1, …, and so on and so forth). Multiple TargetDevice key=value pairs are needed to indicate multiple devices. Chelsio T5/T4 Unified Wire for Linux Page 108...
  • Page 109 Stop Target: To stop the all the iSCSI target(s), execute “ ”. target=ALL [root@host~]# iscsictl –s target=ALL with “ ” followed by the target name. To stop a specific target execute iscsictl [root@host~]# iscsictl –s target=iqn.2006-02.com.chelsio.diskarray.san1 Chelsio T5/T4 Unified Wire for Linux Page 109...

  • Page 110: The Iscsi Configuration File

    2. target There can be only one global entity block whereas multiple target entity blocks are allowed. The global entity block is optional but there must be at least one target entity block. Chelsio T5/T4 Unified Wire for Linux Page 110...
  • Page 111 Chelsio System Wide Global Entity Parameters pass system control information to the iSCSI software which affects all targets in the same way. More detail of these parameters below can be found in a later section entitled “System Wide Parameters”. Chelsio T5/T4 Unified Wire for Linux Page 111...
  • Page 112 “<user id>” DISC_Auth_CHAP_Initiator are not allowed. :“<secret>” The initiator user id and secret are used by the target to authenticate the initiator. NOTE: The double quotes are required as part of the format. Chelsio T5/T4 Unified Wire for Linux Page 112...
  • Page 113 Out iSCSI sequence between the initiator and target. DefaultTime2Wait 0 to 3600 The minimum time, in seconds, to wait before attempting an explicit / implicit logout or connection reset between initiator and target. Chelsio T5/T4 Unified Wire for Linux Page 113...
  • Page 114 1. Challenge Handshake Authentication Protocol (CHAP). 2. Target specific settings. All of the following parameters can have multiple instances in one target entity block (i.e., they can be declared multiple times for one particular target). Chelsio T5/T4 Unified Wire for Linux Page 114...
  • Page 115 [:<port It‟s required to have at least one per number>] target. <portal group tag> is a unique tag identifying the portal group. It must be [,<target IP a positive integer. Chelsio T5/T4 Unified Wire for Linux Page 115...
  • Page 116 NULLRW specifies that random data is returned for reads, and for writes data is dropped. Useful for testing network performance. SYNC specifies that the device will function in the write-through mode (i.e., Chelsio T5/T4 Unified Wire for Linux Page 116...
  • Page 117 1, …, and so on and so forth). NOTE: FILE mode is the most versatile mode, if in doubt use FILE mode. ACL_Enable “Yes” “No” Defines if Chelsio’sAccess Control List “No” (ACL) method will be enforced on the target: Chelsio T5/T4 Unified Wire for Linux Page 117...
  • Page 118 Multiple lists of LUN numbers are allowed. When specifying the list separate the LUN ranges by a comma. RegisteriSNS “Yes” “Yes” To turn on or off exporting of target “No” information via iSNS Chelsio T5/T4 Unified Wire for Linux Page 118...
  • Page 119 # lun 0: a ramdisk with default size of 16MB TargetDevice=ramdisk,MEM PortalGroup=5@192.0.2.178:3260 # an iSCSI Target “iqn.2005-8.com.chelsio:diskarrays.san.328” # being served by the portal group "1" and "2" target: # iSCSI configuration TargetName=iqn.2005-8.com.chelsio:diskarrays.san.328 TargetAlias=iTarget1 MaxOutstandingR2T=1 MaxRecvDataSegmentLength=8192 HeaderDigest=None,CRC32C DataDigest=None,CRC32C Chelsio T5/T4 Unified Wire for Linux Page 119...
  • Page 120 Auth_CHAP_ChallengeLength=16 # ACL configuration # initiator “iqn.2006-02.com.chelsio.san1” is allowed full access # to this target ACL=iname=iqn.2006-02.com.chelsio.san1 # any initiator from IP address 102.50.50.101 is allowed full access # of this target ACL=sip=102.50.50.101 Chelsio T5/T4 Unified Wire for Linux Page 120...

  • Page 121: Challenge-Handshake Authenticate Protocol (Chap)

    They are configured and stored on a per target basis with Chelsio Entity parameter “Auth_CHAP_Target”. 4.5.4. Adding CHAP User ID and Secret A single Auth_CHAP_Target key and multiple Auth_CHAP_Initiator keys could be configured per target: Chelsio T5/T4 Unified Wire for Linux Page 121...
  • Page 122 , the iSCSI target will accept a relevant initiator only if it does a) CHAP Oneway or b) CHAP Mutual Auth_CHAP_Policy=Mutual , the iSCSI node will accept a relevant initiator only if it does a) CHAP Mutual Chelsio T5/T4 Unified Wire for Linux Page 122...

  • Page 123: Target Access Control List (Acl) Configuration

    Possible SCSI layer associations of LUNs with the initiator More than one initiator can be allowed to access a target and each initiator‟s access rights can be independently configured. The format for ACL rule is as follows: Chelsio T5/T4 Unified Wire for Linux Page 123...
  • Page 124 When a target device is marked as read-only (RO), it takes precedence over ACL’s write permission (i.e., all of ACL write permission of an initiator is ignored). Chelsio T5/T4 Unified Wire for Linux Page 124...

  • Page 125: Target Storage Device Configuration

    # 16 Megabytes RAM Disk named ramdisk1 TargetDevice=ramdisk1,MEM,size=16MB #<snip> 4.7.2. FILE Mode Storage Device Details The FILE mode storage device is the most common and versatile mode to access the actual storage attached to the target system: Chelsio T5/T4 Unified Wire for Linux Page 125...
  • Page 126 To configure a block storage device specify BLK as the device mode: TargetDevice=<path to the storage device>,BLK <path> Where: Is the path to the actual storage device, such as /dev/sdb. The path must exist in the system. Chelsio T5/T4 Unified Wire for Linux Page 126...

  • Page 127: Target Redirection Support

    Below is an example with ShadowMode enabled: target: #<snip> # any login requests received on 10.193.184.81:3260 will be # redirected to 10.193.184.85:3261. PortalGroup=1@10.193.184.81:3260,[2] PortalGroup=2@10.193.184.85:3261 # the PortalGroup “2” is NOT presented on the same system. Chelsio T5/T4 Unified Wire for Linux Page 127...

  • Page 128: The Command Line Interface Tools "Iscsictl" & "Chisns

    Chelsio provides for controlling the iSCSI target. It is a Command Line Interface (CLI) that is invoked from the console. Its usage is as follows: iscsictl <options> <mandatory parameters> [optional parameters] Chelsio T5/T4 Unified Wire for Linux Page 128...
  • Page 129 <name1[,name2,…,nameN] | ALL> A name of ALL returns information on all targets. ALL is a reserved string that must be uppercase. Example: iscsictl -c target=iqn.com.cc.it1 Iscsictl -c target=iqn.com.cc.target1 -k Chelsio T5/T4 Unified Wire for Linux Page 129...
  • Page 130 Display the Chelsio iSCSI Global Entity Settings. <var=const> Set the Chelsio iSCSI Global Entity Settings. var=const parameter: Where var=const can be any key listed under Chelsio Global Entity Settings. Example: iscsictl -G iscsi_auth_order=ACL Chelsio T5/T4 Unified Wire for Linux Page 130...
  • Page 131 3. If the target=<name> parameter is not specified, all active targets configurations will be reloaded from the configuration file while those targets are running. All non-active targets specified will not be loaded / started. Chelsio T5/T4 Unified Wire for Linux Page 131...
  • Page 132 ID used to register with the server. It defaults to <hostname>. query=<query interval> where query is the initiator query interval (in seconds). It defaults to 60 seconds. Examples: chisns server=192.0.2.10 Chelsio T5/T4 Unified Wire for Linux Page 132...

  • Page 133: Rules Of Target Reload (I.e. "On The Fly" Changes)

    30 seconds. Rules of Target Reload (i.e. “on the fly” changes) 4.10. After a target has been started its settings can be modified via reloading of the configuration file (i.e., iscsictl -S Chelsio T5/T4 Unified Wire for Linux Page 133...
  • Page 134 IFMarker (support only "No") The following parameters can be changed but would not have any effect because they are either not supported or they are irrelevant: DefaultTime2Wait (not supported) DefaultTime2Retain (not supported) Chelsio T5/T4 Unified Wire for Linux Page 134...

  • Page 135: System Wide Parameters

    4.11.3. iscsi_target_vendor_id CHISCSI Options: A string of maximum of 8 characters, defaults to iscsi_target_vendor_id is part of the device identification sent by an iSCSI target in response of a SCSI Inquiry request. Chelsio T5/T4 Unified Wire for Linux Page 135...

  • Page 136: Performance Tuning

    Unified Wire chapter for generic performance settings. ii. Next, load the iSCSI PDU offload target driver (chiscsi_t4) and run the chiscsi_set_affinity.sh script to map iSCSI worker threads to different CPUs. [root@host~]# chiscsi_set_affinity.sh Chelsio T5/T4 Unified Wire for Linux Page 136...

  • Page 137: Iscsi Pdu Offload Initiator

    Chapter VIII. iSCSI PDU Offload Initiator VIII. iSCSI PDU Offload Initiator Chelsio T5/T4 Unified Wire for Linux Page 137...

  • Page 138: Introduction

    Chapter VIII. iSCSI PDU Offload Initiator 1. Introduction The Chelsio T5/T4 series Adapters support iSCSI acceleration and iSCSI Direct Data Placement (DDP) where the hardware handles the expensive byte touching operations, such as CRC computation and verification, and direct DMA to the final host memory destination:...

  • Page 139: Software Requirements

    Ubuntu 14.04.3, 3.19.0-25-generic * Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire for Linux Page 139...

  • Page 140: Software/Driver Loading

    Kill iSCSI daemon iscsid ii. View all the loaded iSCSI modules [root@host~]# lsmod | grep iscsi iii. Now, unload them using the following command: [root@host~]# rmmod <modulename> iv. Finally reload the cxgb4i driver. Chelsio T5/T4 Unified Wire for Linux Page 140...

  • Page 141: Software/Driver Unloading

    Chapter VIII. iSCSI PDU Offload Initiator 3. Software/Driver Unloading To unload the driver, execute the following commands: [root@host~]# rmmod cxgb4i [root@host~]# rmmod libcxgbi Chelsio T5/T4 Unified Wire for Linux Page 141...

  • Page 142: Software/Driver Configuration And Fine-Tuning

     iface.net_ifacename denotes the Chelsio interface via which iSCSI traffic will be running.  iface.ipaddress denotes the IP address which is assigned to the interface. Chelsio T5/T4 Unified Wire for Linux Page 142...
  • Page 143 -m node -T <iqn name of target> -p <target ip address>:<target port no> -I <cxgb4i iface file name> -l E.g.:- [root@host~]# iscsiadm -m node -T iqn.2004-05.com.chelsio.target1 -p 102.2.2.155:3260,1 -I cxgb4i.00:07:43:04:5b:da -l Chelsio T5/T4 Unified Wire for Linux Page 143...

  • Page 144: Auto Login From Cxgb4I Initiator At Os Bootup

    Auto login from cxgb4i initiator at OS bootup For iSCSI auto login (via cxgb4i) to work on OS startup, please add the following line to start() /etc/rc.d/init.d/iscsid file on RHEL: modprobe -q cxgb4i Chelsio T5/T4 Unified Wire for Linux Page 144...
  • Page 145 -q iscsi_tcpmodprobe -q ib_iser modprobe -q cxgb4i modprobe -q cxgb3i modprobe -q bnx2i modprobe -q be2iscsi daemon brcm_iscsiuio daemon $prog retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval Chelsio T5/T4 Unified Wire for Linux Page 145...

  • Page 146: Data Center Bridging (Dcb)

    Chapter IX. Data Center Bridging (DCB) IX. Data Center Bridging (DCB) Chelsio T5/T4 Unified Wire for Linux Page 146...

  • Page 147: Introduction

    RHEL 7.2, 3.10.0-327.el7 * RHEL 7.1, 3.10.0-229.el7 RHEL 6.7, 2.6.32-573.el6* RHEL 6.6, 2.6.32-504.el6* RHEL 6.5, 2.6.32-431.el6* SLES 12 SP1, 3.12.49-11-default * SLES 12, 3.12.28-4-default* SLES 11 SP4, 3.0.101-63-default * SLES 11 SP3, 3.0.76-0.11-default * Chelsio T5/T4 Unified Wire for Linux Page 147...
  • Page 148 Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire for Linux Page 148...

  • Page 149: Software/Driver Loading

    Run the following commands: [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom [root@host~]# modprobe csiostor Once the storage and networking traffic are started simultaneously, they will honor DCB settings defined on the switch. Chelsio T5/T4 Unified Wire for Linux Page 149...

  • Page 150: Software/Driver Unloading

    3. Software/Driver Unloading To disable DCB feature, unload FCoE Initiator and Network drivers: [root@host~]# rmmod csiostor [root@host~]# rmmod cxgb4 If t4_tom is loaded, please reboot machine after unloading FCoE Initiator and Note Network drivers. Chelsio T5/T4 Unified Wire for Linux Page 150...

  • Page 151: Software/Driver Configuration And Fine-Tuning

    0 switch(config-cmap-qos)# class-map type qos class-fcoe switch(config-cmap-qos)# match cos 3 iii. Configure queuing class-maps. switch(config)#class-map type queuing class-nic switch(config-cmap-que)#match qos-group 2 iv. Configure network-qos class-maps. switch(config)#class-map type network-qos class-nic switch(config-cmap-nq)#match qos-group 2 Chelsio T5/T4 Unified Wire for Linux Page 151...
  • Page 152 4.1.2. Configuring the FCoE/FC ports In this procedure, you may need to adjust some of the parameters to suit your environment, such as VLAN IDs, Ethernet interfaces, and virtual Fibre Channel interfaces Chelsio T5/T4 Unified Wire for Linux Page 152...
  • Page 153 2 switch(config-if)#no shutdown switch(config)#exit iv. Enabling DCB: switch(config)# interface ethernet 1/13 switch(config-if)# priority-flow-control mode auto switch(config-if)# flowcontrol send off switch(config-if)# flowcontrol receive off switch(config-if)# lldp transmit switch(config-if)# lldp receive switch(config-if)# no shutdown Chelsio T5/T4 Unified Wire for Linux Page 153...

  • Page 154: Configuring The Brocade 8000 Switch

    Create a CEE Map to carry LAN and SAN traffic if it does not exist. Example of creating a CEE map. switch(config)# cee-map default switch(conf-cee-map)#priority-group-table 1 weight 40 pfc switch(conf-cee-map)#priority-group-table 2 weight 60 switch(conf-cee-map)#priority-table 2 2 2 1 2 2 2 2 Chelsio T5/T4 Unified Wire for Linux Page 154...
  • Page 155 Unlike cisco, only one VLAN ID can carry FCoE traffic for now on Brocade 8000. Note It is their limitation. v. Save the Configuration switch#copy running-config startup-config Chelsio T5/T4 Unified Wire for Linux Page 155...

  • Page 156: Running Nic & Iscsi Traffic Together With Dcbx

    NIC traffic is run on VLAN10 and iSCSI traffic is run on VLAN20. Assign proper VLAN priorities on the interface (here eth5), using the following commands on the host machine: [root@host~]# vconfig set_egress_map eth5.10 0 2 [root@host~]# vconfig set_egress_map eth5.20 5 5 Chelsio T5/T4 Unified Wire for Linux Page 156...

  • Page 157: Fcoe Pdu Offload Target

    Chapter X. FCoE PDU Offload Target X. FCoE PDU Offload Target Chelsio T5/T4 Unified Wire for Linux Page 157...

  • Page 158: Introduction

    The Chelsio FCoE PDU offload target driver runs on Linux-based platforms and therefore it is a base requirement for running the driver. Currently the driver is available for the following version: Kernel.org linux-3.6.11 on RHEL 6.x distribution (Limited QA performed) Other kernel versions are not supported. Chelsio T5/T4 Unified Wire for Linux Page 158...

  • Page 159: Software/Driver Loading

    [root@host~]# modprobe chfcoe chfcoe_fip_mode=1  To load the driver in VN2VN mode with VLAN support, run the following command: [root@host~]# modprobe chfcoe chfcoe_fip_mode=1 chfcoe_vlanid=<vlan_id> Alternatively, you can edit /etc/modprobe.d/chfcoe.conf for specifying VN2VN and VLAN options. Chelsio T5/T4 Unified Wire for Linux Page 159...

  • Page 160: Software/Driver Unloading

    Chapter X. FCoE PDU Offload Target 3. Software/Driver Unloading To unload the driver, run the following command: [root@host~]# modprobe –r chfcoe Chelsio T5/T4 Unified Wire for Linux Page 160...

  • Page 161: Software/Driver Configuration And Fine-Tuning

    Configure qos class-maps and set the traffic priorities: NIC uses priority 0 and FcoE uses priority 3. switch(config)#class-map type qos class-nic switch(config-cmap-qos)# match cos 0 switch(config-cmap-qos)# class-map type qos class-fcoe switch(config-cmap-qos)# match cos 3 iii. Configure queuing class-maps. switch(config)#class-map type queuing class-nic switch(config-cmap-que)#match qos-group 2 Chelsio T5/T4 Unified Wire for Linux Page 161...
  • Page 162 Apply the new policy (PFC on NIC and FcoE traffic) to the entire system. switch(config)#system qos switch(config-sys-qos)#service-policy type qos input policy-test switch(config-sys-qos)#service-policy type queuing output policy-test switch(config-sys-qos)#service-policy type queuing input policy-test switch(config-sys-qos)#service-policy type network-qos policy-test Chelsio T5/T4 Unified Wire for Linux Page 162...
  • Page 163 Assign VLAN ID to the Ethernet port on which FCoE service was enabled in step (i). switch(config)# interface ethernet 1/13 switch(config-if)#switchport mode trunk switch(config-if)#switchport trunk allowed vlan 2 switch(config-if)#no shutdown switch(config)#exit Chelsio T5/T4 Unified Wire for Linux Page 163...

  • Page 164: Collecting Port Information

    You can use cxgbtool to view both target and initiator WWPNs. The command requires an adapter device file to be specified. After loading chfcoe driver, device files will be created for each adapter /dev/chfcoeX, where index adapter. [root@host~]# ls /dev/chfcoe* /dev/chfcoe0 Chelsio T5/T4 Unified Wire for Linux Page 164...
  • Page 165 If FCoE PDU Offload Target is operating in FCF mode, then the local node information will be available only after the target completes FLOGI to the swtich. To identify Chelsio target’s WWPN from other vendors’, remember that the WWPN Note always begins with 0x5000743. Chelsio T5/T4 Unified Wire for Linux Page 165...
  • Page 166 Chapter X. FCoE PDU Offload Target 4.2.2. Verifying remote ports To verify remote ports (fabric, name server, initiator ports etc.), use the following command: [root@host~]# cxgbtool stor -a <adapter_device_file> --show-rnode -- wwn=<target_wwpn> Chelsio T5/T4 Unified Wire for Linux Page 166...

  • Page 167: Configuring Luns On Target

    /etc/chelsio-fcoe/ after Unified Wire installation. iii. Ensure that SCST handler modules, which are used in configuration (eg: scst_vdisk), are loaded before proceeding. iv. Configure LUNs on target by running the following command. [root@host~]# scstadmin -config <LUN_Config_file> Chelsio T5/T4 Unified Wire for Linux Page 167...
  • Page 168 Chapter X. FCoE PDU Offload Target The following configuration file adds three LUNs(RAMDisk, physical disk & NULLIO disk) for the target specified. Only initiators present in the group will be able to access the LUNs. Chelsio T5/T4 Unified Wire for Linux Page 168...

  • Page 169: Configuring Persistent Target

    Place the file in /etc/chelsio-fcoe/ directory. iii. If any handler other than scst_vdisk is used in SCST configuration, please add module load entry for the same under chfcoe_start() section of /etc/init.d/chfcoe file. Chelsio T5/T4 Unified Wire for Linux Page 169...

  • Page 170: Verifying Initiators Connected To The Target

    Once connected to the target, initiators will be able to access the LUNs configured on the target. Execute the following command to list the initiators which are connected to the target: [root@host~]# scstadmin –list_session Chelsio T5/T4 Unified Wire for Linux Page 170...

  • Page 171: Removing Luns

    If hyperthreading is enabled, use both thread siblings of a core for binding either workers or irqs. [root@host~]# cat /sys/devices/system/cpu/cpu0/topology/threads_siblings_list 0,16 Use both 0 and 16 CPUs for either workers or IRQs. Chelsio T5/T4 Unified Wire for Linux Page 171...
  • Page 172 19161's new affinity list: 19 chfcoe_0_8: pid 19162's new affinity list: 20 chfcoe_0_9: pid 19163's new affinity list: 21 chfcoe_0_10: pid 19164's new affinity list: 22 chfcoe_0_11: pid 19165's new affinity list: 23 Chelsio T5/T4 Unified Wire for Linux Page 172...
  • Page 173 Writing 1 in /proc/irq/165/smp_affinity Writing 2 in /proc/irq/166/smp_affinity Writing 10000 in /proc/irq/167/smp_affinity Writing 20000 in /proc/irq/168/smp_affinity Writing 1 in /proc/irq/169/smp_affinity Writing 2 in /proc/irq/170/smp_affinity Writing 10000 in /proc/irq/171/smp_affinity Writing 20000 in /proc/irq/172/smp_affinity Chelsio T5/T4 Unified Wire for Linux Page 173...

  • Page 174: Fcoe Full Offload Initiator

    Chapter XI. FCoE Full Offload Initiator XI. FCoE Full Offload Initiator Chelsio T5/T4 Unified Wire for Linux Page 174...

  • Page 175: Introduction

    RHEL 6.5, 2.6.32-431.el6* SLES 12 SP1, 3.12.49-11-default * SLES 12, 3.12.28-4-default* SLES 11 SP4, 3.0.101-63-default * SLES 11 SP3, 3.0.76-0.11-default Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Chelsio T5/T4 Unified Wire for Linux Page 175...
  • Page 176 Chapter XI. FCoE Full Offload Initiator Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire for Linux Page 176...

  • Page 177: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver, execute the following: [root@host~]# modprobe csiostor To load the driver on any SLES distribution, execute the following command: [root@host~]# modprobe csiostor --allow Chelsio T5/T4 Unified Wire for Linux Page 177...

  • Page 178: Software/Driver Unloading

    3. Software/Driver Unloading To unload the driver: [root@host~]# modprobe -r csiostor If multipath services are running, unload of FCoE driver is not possible. Stop the Note multipath service and then unload the driver. Chelsio T5/T4 Unified Wire for Linux Page 178...

  • Page 179: Software/Driver Configuration And Fine-Tuning

     Inorder to identify chelsio fc_host from other vendor fc_host, the WWPN always begins with 0x5000743 Alternatively, the local port information can also be found using: [root@host~]# cxgbtool stor –a <adapter_no> --show-lnode Chelsio T5/T4 Unified Wire for Linux Page 179...
  • Page 180 Chapter XI. FCoE Full Offload Initiator Chelsio T5/T4 Unified Wire for Linux Page 180...
  • Page 181 R can correspond to NameServer, Management Server and other initiator ports Note logged in to the switch and targets. Alternatively, the local ports can also be found using cxgbtool: [root@host~]# cxgbtool stor –a <adapter no> --show-lnode Chelsio T5/T4 Unified Wire for Linux Page 181...
  • Page 182 Chapter XI. FCoE Full Offload Initiator After finding out the WWPN of the local node, to verify the list of discovered targets, use the following command. [root@host~]# cxgbtool stor –a <adapter_no> --show-rnode --wwn=<wwpn of lnode> Chelsio T5/T4 Unified Wire for Linux Page 182...

  • Page 183: Formatting The Luns And Mounting The Filesystem

    Chapter XI. FCoE Full Offload Initiator 4.3. Formatting the LUNs and Mounting the Filesystem Use lsscsi –g to list the LUNs discovered by the initiator [root@host~]# lsscsi –g Chelsio T5/T4 Unified Wire for Linux Page 183...

  • Page 184: Creating Filesystem

    Alternatively, the LUNs discovered by the Chelsio FCoE initiators can be accessed via easily- identifiable ‘udev’ path device files like: [root@host~]# ls /dev/disk/by-path/pci-0000:04:00.0-csio-fcoe <local_wwpn>:<remote_wwpn>:<lun_wwn> 4.4. Creating Filesystem Create an ext3 filesystem using the following command: [root@host~]# mkfs.ext3 /dev/sdx Chelsio T5/T4 Unified Wire for Linux Page 184...

  • Page 185: Mounting The Formatted Lun

    Chapter XI. FCoE Full Offload Initiator 4.5. Mounting the formatted LUN The formatted LUN can be mounted on the specified mountpoint using the following command: [root@host~]# mount /dev/sdx /mnt Chelsio T5/T4 Unified Wire for Linux Page 185...

  • Page 186: Offload Bonding Driver

    Chapter XII. Offload Bonding driver XII. Offload Bonding driver Chelsio T5/T4 Unified Wire for Linux Page 186...

  • Page 187: Introduction

    Software Requirements 1.2.1. Linux Requirements Currently the Offload Bonding driver is available for the following versions: RHEL 7.2, 3.10.0-327.el7 * RHEL 7.1, 3.10.0-229.el7 RHEL 6.7, 2.6.32-573.el6* RHEL 6.6, 2.6.32-504.el6* RHEL 6.5, 2.6.32-431.el6* Chelsio T5/T4 Unified Wire for Linux Page 187...
  • Page 188 Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire for Linux Page 188...

  • Page 189: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host~]# modprobe bonding Chelsio T5/T4 Unified Wire for Linux Page 189...

  • Page 190: Software/Driver Unloading

    Chapter XII. Offload Bonding driver 3. Software/Driver Unloading To unload the driver, run the following command: [root@host~]# rmmod bonding Chelsio T5/T4 Unified Wire for Linux Page 190...

  • Page 191: Software/Driver Configuration And Fine-Tuning

    [root@host~]# ifconfig bond0 X.X.X.X/Y [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up v. Disable FRTO on the PEER: [root@host~]# sysctl -w net.ipv4.tcp_frto=0 All TCP traffic will be offloaded over the bonded interface now. Chelsio T5/T4 Unified Wire for Linux Page 191...

  • Page 192: Offload Multi-Adapter Failover (Mafo)

    Chapter XIII. Offload Multi-Adapter Failover (MAFO) XIII. Offload Multi-Adapter Failover (MAFO) Chelsio T5/T4 Unified Wire for Linux Page 192...

  • Page 193: Introduction

    The following are the currently shipping Chelsio Adapters that are compatible with the Offload Multi-Adapter Failover driver. T520-BT T580-CR T520-LL-CR T520-CR T580-LP-CR T540-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX Chelsio T5/T4 Unified Wire for Linux Page 193...

  • Page 194: Software Requirements

    Multi-Adapter failover feature will work for Link Down events caused by:  Cable unplug on bonded interface.  Bringing corresponding switch port down. The feature will not work if the bonded interfaces are administratively taken Note down. Chelsio T5/T4 Unified Wire for Linux Page 194...

  • Page 195: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host~]# modprobe bonding Chelsio T5/T4 Unified Wire for Linux Page 195...

  • Page 196: Software/Driver Unloading

    Chapter XIII. Offload Multi-Adapter Failover (MAFO) 3. Software/Driver Unloading To unload the driver, run the following command: [root@host~]# rmmod bonding Chelsio T5/T4 Unified Wire for Linux Page 196...

  • Page 197: Software/Driver Configuration And Fine-Tuning

    [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up vi. Disable FRTO on the PEER: [root@host~]# sysctl -w net.ipv4.tcp_frto=0 All TCP traffic will be offloaded over the bonded interface now and fail-over will happen in case of link-down event. Chelsio T5/T4 Unified Wire for Linux Page 197...

  • Page 198: Udp Segmentation Offload And Pacing

    Chapter XIV. UDP Segmentation Offload and Pacing XIV. UDP Segmentation Offload and Pacing Chelsio T5/T4 Unified Wire for Linux Page 198...

  • Page 199: Introduction

    Service Level Agreements (SLAs). 1.1. Hardware Requirements 1.1.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with the UDP Segmentation Offload and Pacing driver. Chelsio T5/T4 Unified Wire for Linux Page 199...

  • Page 200: Software Requirements

    Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Performed. Chelsio T5/T4 Unified Wire for Linux Page 200...

  • Page 201: Software/Driver Loading

    [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom Though normally associated with the Chelsio TCP Offload engine, the t4_tom module is required in order to allow for the proper redirection of UDP socket calls. Chelsio T5/T4 Unified Wire for Linux Page 201...

  • Page 202: Software/Driver Unloading

    Chapter XIV. UDP Segmentation Offload and Pacing 3. Software/Driver Unloading Reboot the system to unload the driver. To unload without rebooting, refer Unloading the TOE driver section of Network (NIC/TOE) chapter. Chelsio T5/T4 Unified Wire for Linux Page 202...

  • Page 203: Software/Driver Configuration And Fine-Tuning

    # define IPPROTO_UDP 17 Use the setsockopt() function to set socket options: //Get the UDP socket descriptor variable setsockopt (sockfd , IPPROTO_UDP, SO_FRAMESIZE, &fs, sizeof(fs)); setsockopt (sockfd , IPPROTO_UDP, SOL_SCHEDCLASS, &cl, sizeof(cl)); Chelsio T5/T4 Unified Wire for Linux Page 203...
  • Page 204 3. In step (iv), define a new socket option: setsockopt (sockfd,17,UDP_RTPHEADERLEN,&rtp_header_size, sizeof(rtp_header_size)); Here,  &rtp_header_size : pointer to the RTP header length variable  sizeof(rtp_header_size) : the size of the RTP header length variable Chelsio T5/T4 Unified Wire for Linux Page 204...

  • Page 205: Configuring Udp Pacing

    100 kbps to 1 Gbps.  Packet size is the UDP packet payload size in bytes; it should be equal to the value set in the SO_FRAMESIZE socket option in the application in section 4.1. Chelsio T5/T4 Unified Wire for Linux Page 205...
  • Page 206 10240 UDP connections, it can check the return code of ENOSPC from a send() or sendto() call and close this socket and open a new one that uses the kernel UDP stack. Chelsio T5/T4 Unified Wire for Linux Page 206...

  • Page 207: Offload Ipv6 Driver

    Chapter XV. Offload IPv6 driver XV. Offload IPv6 driver Chelsio T5/T4 Unified Wire for Linux Page 207...

  • Page 208: Introduction

    Currently the Offload IPv6 feature is available for the following versions: RHEL 7.2, 3.10.0-327.el7 * RHEL 7.1, 3.10.0-229.el7 RHEL 7.1 BE, 3.10.0-229.el7.ppc64 (POWER7) RHEL 7.1 LE, 3.10.0-229.ael7b.ppc64le (POWER8) RHEL 6.7, 2.6.32-573.el6* RHEL 6.6, 2.6.32-504.el6* Chelsio T5/T4 Unified Wire for Linux Page 208...
  • Page 209 Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire for Linux Page 209...

  • Page 210: Software/Driver Loading

    (t4_tom) drivers. The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom All the IPv6 traffic will be offloaded now. Chelsio T5/T4 Unified Wire for Linux Page 210...

  • Page 211: Software/Driver Unloading

    To unload the NIC driver, run the following command: [root@host~]# rmmod cxgb4 3.2. Unloading the TOE driver Please reboot the system to unload the TOE driver. To unload without rebooting, refer Unloading the TOE driver section of Network (NIC/TOE) chapter. Chelsio T5/T4 Unified Wire for Linux Page 211...

  • Page 212: Bypass Driver

    Chapter XVI. Bypass Driver XVI. Bypass Driver Chelsio T5/T4 Unified Wire for Linux Page 212...

  • Page 213: Introduction

    B420, port 0 to port 1 and vice versa; and in B404, port 0 to port 1 and port 2 to port 3 and vice versa. Normal Mode The Bypass Adapters can be programmed to function as a NIC. In this mode all the packets are redirected to the host. Chelsio T5/T4 Unified Wire for Linux Page 213...

  • Page 214: Hardware Requirements

    Finally, the input action directs the packet to the host where it can be processed by the application. 1.2. Hardware Requirements 1.2.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with Chelsio Bypass driver: B404-BT B420-SR Chelsio T5/T4 Unified Wire for Linux Page 214...

  • Page 215: Software Requirements

    Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire for Linux Page 215...

  • Page 216: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. Run the following command to load the Bypass driver: [root@host~]# modprobe cxgb4 Chelsio T5/T4 Unified Wire for Linux Page 216...

  • Page 217: Software/Driver Unloading

    Chapter XVI. Bypass Driver 3. Software/Driver Unloading Run the following command to unload the Bypass driver: [root@host~]# rmmod cxgb4 Chelsio T5/T4 Unified Wire for Linux Page 217...

  • Page 218: Software/Driver Configuration And Fine-Tuning

    The redirect management element will interface with the SDK server to take the appropriate actions to manage tables and rules. 4.2.1. Bypass Management The bypass CLI will have the following syntax: [root@host~]# bypass ethX get|set --key [value] Chelsio T5/T4 Unified Wire for Linux Page 218...
  • Page 219 4.2.3. Bypass State To obtain the current state of the switch: [root@host~]# bypass ethX get --current_state To set the state of the switch: [root@host~]# bypass ethX set --current_state [bypass|disconnect|normal] Chelsio T5/T4 Unified Wire for Linux Page 219...
  • Page 220 When a rule matches the action associated with it, that rule is performed and higher number rules are not evaluated. Rules are not evaluated until a table is activated. Chelsio T5/T4 Unified Wire for Linux Page 220...
  • Page 221 Redirect rule srcaddr6 IPv6 Source IP address dstaddr6 IPv6 Destination IP address srcmask6 IPv6 Source IP address mask (Default: FFFF:FFFF:FFFF:FFFF:FFF F:FFFF:FFFF:FFFF) dstmask6 IPv6 Destination IP address mask (Default:FFFF:FFFF:FFFF:F FFF:FFFF:FFFF:FFFF:FFFF) Chelsio T5/T4 Unified Wire for Linux Page 221...
  • Page 222 0. The rules in a table are not active until the table is activated. Create a table: [root@host~]# redirect ethX create_table --table 1 Activate the table: [root@host~]# redirect ethX activate_table --table 1 Chelsio T5/T4 Unified Wire for Linux Page 222...
  • Page 223 IPv4 mode. For IPv6, the ba_server needs to be started manually. [root@host~]# service bad start To stop the service: [root@host~]# service bad stop To restart the service: [root@host~]# service bad restart Chelsio T5/T4 Unified Wire for Linux Page 223...

  • Page 224: Wd Sniffing And Tracing

    Chapter XVII. WD Sniffing and Tracing XVII. WD Sniffing and Tracing Chelsio T5/T4 Unified Wire for Linux Page 224...

  • Page 225: Theory Of Operation

    Connect Client A and B to ports 0 and 1 or ports 2 and 3. d) Enable tracing. In either mode the targeted traffic bypasses the kernel TCP/IP stack and is delivered directly to user space by means of an RX queue. Chelsio T5/T4 Unified Wire for Linux Page 225...

  • Page 226: Hardware Requirements

    Chapter XVII. WD Sniffing and Tracing Schematic diagram of sniffer and tracer 1.1. Hardware Requirements 1.1.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with the tools: T520-BT T580-CR T580-LP-CR T520-LL-CR T520-CR Chelsio T5/T4 Unified Wire for Linux Page 226...

  • Page 227: Software Requirements

    Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire for Linux Page 227...

  • Page 228: Installation And Usage

    2.3. Using Tracer (wd_tcpdump_trace) 1. Setup: Wire tapping requires 3 systems with one machine having a T5/T4 card with two or more ports. The machines should be setup in the following manner: Chelsio T5/T4 Unified Wire for Linux Page 228...
  • Page 229 Use any tool (like ping or ssh) to run traffic between machines A and B. The traffic should successfully make it from end to end and wd_tcpdump_trace on the DUT should show the tapped traffic. Chelsio T5/T4 Unified Wire for Linux Page 229...

  • Page 230: Xviii.classification And Filtering

    Chapter XVIII. Classification and Filtering XVIII. Classification and Filtering Chelsio T5/T4 Unified Wire for Linux Page 230...

  • Page 231: Introduction

    The following are the currently shipping Chelsio Adapters that are compatible with the Classification and Filtering feature: T520-BT T580-CR T520-LL-CR T520-SO-CR T520-CR T540-CR T580-LP-CR T580-SO-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX Chelsio T5/T4 Unified Wire for Linux Page 231...

  • Page 232: Software Requirements

    Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Peformed. Chelsio T5/T4 Unified Wire for Linux Page 232...

  • Page 233: Usage

    4=PROM, 5=hyper PROM, 6=broadcast, 7=none) protocol : IP protocol number (ICMP=1, TCP=6, UDP=17, etc) vlan : Inner VLAN Tag port : Packet ingress port number fcoe : Fibre Channel over Ethernet frames Chelsio T5/T4 Unified Wire for Linux Page 233...

  • Page 234: Creating Filter Rules

    [root@host~]# cxgbtool ethX filter 0 action drop fip 192.168.1.5 The above filter rule will drop all ingress packets from IP 192.168.1.5  Pass action [root@host~]# cxgbtool ethX filter 0 action pass lport 10001 fport 355 queue 2 Chelsio T5/T4 Unified Wire for Linux Page 234...

  • Page 235: Listing Filter Rules

    To remove a filter, run the following command with the corresponding filter rule index [root@host~]# cxgbtool ethX filter index <delete|clear> For more information on additional parameters, refer cxgbtool manual by running Note man cxgbtool command Chelsio T5/T4 Unified Wire for Linux Page 235...

  • Page 236: Layer 3 Example

    Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:14372 errors:0 dropped:0 overruns:0 frame:0 TX packets:62203 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1585952 (1.5 MiB) TX bytes:4798122 (4.5 MiB) Interrupt:16 Chelsio T5/T4 Unified Wire for Linux Page 236...
  • Page 237 RX bytes:218606 (213.4 KiB) TX bytes:17483 (17.0 KiB) Interrupt:17 ii. Setup a static OR default route towards T5 router to reach 102.1.1.0/24 network [root@host~]# route add -net 102.1.1.0/24 gw 102.1.2.250 Chelsio T5/T4 Unified Wire for Linux Page 237...
  • Page 238 1 smac 00:07:43:04:96:48 dmac 00:07:43:12:D4:88 smac dmac Where, is the MAC address of eth1 interface on T5 adapter machine and is the MAC address of eth0 interface on Node 2. Chelsio T5/T4 Unified Wire for Linux Page 238...

  • Page 239: Layer 2 Example

    Here’s an example on how to achieve L2 switching functionality. The following will only work on kernel 3.10 and above. eth0 eth1 promisc promisc 102.1.1.1/24 102.1.1.2/24 eth0 eth0 Node 1 Node 2 Chelsio T5/T4 Unified Wire for Linux Page 239...
  • Page 240 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:1961 errors:0 dropped:2 overruns:0 frame:0 TX packets:141 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:218606 (213.4 KiB) TX bytes:17483 (17.0 KiB) Interrupt:17 Chelsio T5/T4 Unified Wire for Linux Page 240...
  • Page 241 Create a filter to match incoming packet's dst-mac 00:07:43:12:d4:88 with particular mac-idx and switch it out via eport 1: [root@host~]# cxgbtool eth0 filter 0 macidx 5 action switch eport 1 hitcnts Chelsio T5/T4 Unified Wire for Linux Page 241...
  • Page 242 Create a filter to match incoming packet's dst-mac 00:07:43:04:7d:50 with particular mac-idx and switch it out via eport 0: [root@host~]# cxgbtool eth0 filter 1 macidx 7 action switch eport 0 hitcnts Chelsio T5/T4 Unified Wire for Linux Page 242...

  • Page 243: Hash/Ddr Filters

    Hash filters don't support masks and hence, 4-tuple must always be supplied for Hash filter. “cap maskless” parameter should be appended in order to create Hash/DDR filter rules. Otherwise the above command will create LE- TCAM filter rules. Chelsio T5/T4 Unified Wire for Linux Page 243...

  • Page 244: Listing Filter Rules

    For more information on additional parameters, refer cxgbtool manual by running Note man cxgbtool command. 3.2. Listing Filter Rules  To list the Hash/DDR filters set, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4/<bus-id>/hash_filters Chelsio T5/T4 Unified Wire for Linux Page 244...

  • Page 245: Removing Filter Rules

    3.4. Swap MAC feature Chelsio T5’s Swap MAC feature swaps packet source MAC and destination MAC addresses. This is applicable only for switch filter rules. Here’s an example: [root@host~]# cxgbtool eth2 filter 1 action switch lip 102.2.2.1 fip 102.2.2.2 lport 5001 fport 14000 hitcnts 1 iport 1 eport 0 swapmac 1 proto...

  • Page 246: Hit Counters

    4 different RSS Queues separately corresponding to 4 trace-filters. Enable Tracing as well as MultiRSSFilter [root@host~]# cxgbtool ethX reg 0x09800=0x33 ii. Setup a trace filter [root@host~]# echo tx0 snaplen=40 > /sys/kernel/debug/cxgb4/<bus_id>/trace0 Chelsio T5/T4 Unified Wire for Linux Page 246...
  • Page 247 [root@host~]# echo tx1 snaplen=40 > /sys/kernel/debug/cxgb4/<bus_id>/trace2 [root@host~]# echo rx1 snaplen=40 > /sys/kernel/debug/cxgb4/<bus_id>/trace3 [root@host~]# cxgbtool ethX reg 0x09ff4=<Trace-Queue1-RspQ ID> [root@host~]# cxgbtool ethX reg 0x09ffc=<Trace-Queue2-RspQ ID> [root@host~]# cxgbtool ethX reg 0x0a004=<Trace-Queue3-RspQ ID> Use “snaplen=60” in case of IPV6. Note Chelsio T5/T4 Unified Wire for Linux Page 247...

  • Page 248: Traffic Management

    XIX. Traffic Management XIX. Traffic Management Chelsio T5/T4 Unified Wire for Linux Page 248...

  • Page 249: Introduction

    XIX. Traffic Management 1. Introduction Traffic Management capabilities built-in to Chelsio T5/T4 CNAs can shape transmit data traffic through the use of sophisticated queuing and scheduling algorithms built-in to the ASIC hardware which provides fine-grained software control over latency and bandwidth parameters such as packet rate and byte rate.

  • Page 250: Software Requirements

    Ubuntu 14.04.3, 3.19.0-25-generic* Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Peformed. Chelsio T5/T4 Unified Wire for Linux Page 250...

  • Page 251: Software/Driver Loading

    The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. Run the following commands to load the TOE driver: [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom Chelsio T5/T4 Unified Wire for Linux Page 251...

  • Page 252: Software/Driver Unloading

    XIX. Traffic Management 3. Software/Driver Unloading Reboot the system to unload the driver. To unload without rebooting, refer Unloading the TOE driver section of Network (NIC/TOE) chapter. Chelsio T5/T4 Unified Wire for Linux Page 252...

  • Page 253: Software/Driver Configuration And Fine-Tuning

    (Kbps) for this TCP stream.  Packet size is the TCP mss size in bytes; for example – for an MTU of 1500, use a packet size of 1460. Chelsio T5/T4 Unified Wire for Linux Page 253...
  • Page 254 Possible values include relative or absolute.  Channel No. is the port on which data is flowing (0-3).  maximum-rate is the bit rate (Kbps) for this TCP stream. The lower limit is 1 Gbps. Chelsio T5/T4 Unified Wire for Linux Page 254...

  • Page 255: Configuring Traffic Management

    If the TX queue is all, * or any negative value, the binding will apply to all of the TX Note queues associated with the interface. If the class is unbind, clear or any negative value, the TX queue(s) will be unbound from any current TX Scheduler Class binding. Chelsio T5/T4 Unified Wire for Linux Page 255...
  • Page 256 [root@host~]# cop -d –o <output_policy_file> <new_policy_file> iv. Apply the COP policy: [root@host~]# cxgbtool ethX policy <output_policy_file> Where, ethX: Chelsio interface For more information on additional parameters, refer cop manual by running the Note man cop command. Chelsio T5/T4 Unified Wire for Linux Page 256...
  • Page 257 Here:  sockfd : The file descriptor of the TCP socket  &cl : Pointer to the class variables  sizeof(cl) : The size of the variable v. Now, compile the application. Chelsio T5/T4 Unified Wire for Linux Page 257...

  • Page 258: Usage

    The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for offloaded connections: Load the TOE driver and bring up the interface [root@host~]# modprobe t4_tom [root@host~]# ifconfig eth0 up Chelsio T5/T4 Unified Wire for Linux Page 258...

  • Page 259: Offloaded Connections With Modified Application

    Set the appropriate rule for class 0 [root@host~]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 Chelsio T5/T4 Unified Wire for Linux Page 259...

  • Page 260: Unified Wire Manager (Um)

    XX. Unified Wire Manager (UM) XX. Unified Wire Manager (UM) Chelsio T5/T4 Unified Wire for Linux Page 260...

  • Page 261: Introduction

    Single tool with CLI and Web interface that works across Storage, Networking and Hardware. Manage all Chelsio adapters installed on the system. Tool for FAE to debug issues on the Customer front. Freedom to choose various modes of management i.e. CLI or Web GUI. Chelsio T5/T4 Unified Wire for Linux Page 261...

  • Page 262: Reference Architecture

    The Management Client can be used to connect to agents and manage them. Once connected you can view and configure Chelsio HBAs and related networking, storage and hardware properties. You can use either the CLI or Web GUI client to manage agents based on your Chelsio T5/T4 Unified Wire for Linux Page 262...

  • Page 263: Authentication And Encryption

    CLI client or the Web GUI and the agent, are encrypted using SSL. This ensures that the data cannot be accessed when it is being transmitted over the network. Chelsio T5/T4 Unified Wire for Linux Page 263...

  • Page 264: Hardware And Software

    T520-LL-CR T520-CR T580-LP-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX S302E S302E-C S310E-CR S310E-CR-C S310E-CXA S310E-SR+ S310E-SR S310E-BT S320E-CR S320E-LP-CR S320E-CXA S320EM-BS S320EM-BCH N320E-G2-CR N320E N320E-CXA N320E-BT N310E N310E-CXA Chelsio T5/T4 Unified Wire for Linux Page 264...

  • Page 265: Platform/Component Matrix

    T3 : 2.0.0.1 (RHEL 5.8,6.3) T4,T5: 2.12.0.2 Bypass T4: 2.12.0.2 T3 : 2.0.0.1 Bonding T4: 3.7.1 T3: 5.2.0-0560 iSCSI Target T4,T5: 2.12.0.2-1180 Open iSCSI Initiator T3,T4,T5: 2.0-873 iWARP T3,T4,T5: 2.12.0.2 FCoE Initiator T4: 2.12.0.2 Chelsio T5/T4 Unified Wire for Linux Page 265...

  • Page 266: Installing Unified Wire Manager

    Chelsio Unified Wire has been designed to install Unified Wire Manager (UM) by default. All the three UM components, i.e. Management Agent, Client and Station, will be installed on selecting any of the Terminator 4/Terminator 5 configuration tuning options during installation. Hence, no separate installation is required. Chelsio T5/T4 Unified Wire for Linux Page 266...

  • Page 267: Verifying Um Components Status

    You can also execute the following comand to determine if Management Agent is running: [root@chelsio]# /etc/init.d/chelsio-uwire_mgmtd status The above command should display the following result: Unified Wire Manager Agent : Running Chelsio T5/T4 Unified Wire for Linux Page 267...

  • Page 268: Verifying Management Client

    While accessing the Web Management Interface, if an error “Management Note Station is not running” is displayed at the log-in page, follow the above steps to restart Management Station and try again. Chelsio T5/T4 Unified Wire for Linux Page 268...

  • Page 269: Management Agent

    Execute the following command to list the service configuration. [root@host~]# chkconfig --list chelsio-uwire_mgmtd Execute the following command to enable/disable the service to start at system runlevel 5. [root@host~]# chkconfig --level 5 chelsio-uwire_mgmtd on/off Chelsio T5/T4 Unified Wire for Linux Page 269...

  • Page 270: Firewall

    If there is a firewall appliance / software protecting the network that the system is on, and you wish to connect to the system from a different network, using the client, the firewall appliance also needs to be configured appropriately. Chelsio T5/T4 Unified Wire for Linux Page 270...

  • Page 271: Cli Client

    Also, two different Web GUI or CLI clients cannot connect to a management agent at the same time. There is no mechanism to allow this scenario. Chelsio T5/T4 Unified Wire for Linux Page 271...

  • Page 272: Web Gui Client

    [root@host~]# openssl rsa -in server.key.org -out server.key [root@host~]# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt [root@host~]# cp server.crt ./ssl.crt [root@host~]# cp server.key ./ssl.key ii. Start/Restart Apache services [root@host~]# rcapache2 [start|restart] Chelsio T5/T4 Unified Wire for Linux Page 272...

  • Page 273: Accessing Web Management Interface

    The security certificate used by the web server is a generic one. It may cause the following types of prompts in different browsers. You will need to select the correct option to continue. Security Certificate prompt in Internet Explorer Figure 7.2 (a) - Chelsio T5/T4 Unified Wire for Linux Page 273...
  • Page 274 XX. Unified Wire Manager (UM) Security Certificate prompt in Mozilla Firefox Figure 7.2 (b) - Security Certificate prompt in Apple Safari Figure 7.2 (c) - Security Certificate prompt in Google Chrome Figure 7.2 (d) - Chelsio T5/T4 Unified Wire for Linux Page 274...
  • Page 275 Figure 7.2 (e) - Web GUI Login page Not performing any operation/action for 5 minutes will result in session timeout. Note You will have to re-login and connect to the Agents again. Chelsio T5/T4 Unified Wire for Linux Page 275...

  • Page 276: Layout And Navigation

    Navigation Pane. The panels can be expanded and collapsed by clicking on the panel heading.  The Bottom bar has the About link on the right and copyright details on the left. Figure 7.3 – Web Management Interface Chelsio T5/T4 Unified Wire for Linux Page 276...

  • Page 277: Home Page

    The passwords are encrypted, but it is still advisable to store passwords only if the system you are running the GUI client on, is secure. Figure 7.4.1 (a) - Bookmarks and history module Chelsio T5/T4 Unified Wire for Linux Page 277...
  • Page 278 Once removed, the system will no longer appear in the Bookmarks and History Note module. If you wish to manage that system again, you will have to use the Add system option. Chelsio T5/T4 Unified Wire for Linux Page 278...
  • Page 279 The Add Agents button adds the selected system to the list of discovered agents in the Bookmarks and History module. The Clear Agents button resets the list of discovered agents. Figure 7.4.1 (b) - Service Discovery module Chelsio T5/T4 Unified Wire for Linux Page 279...
  • Page 280 Installation section. 4. Download the appropriate driver from Chelsio’s Download Center, service.chelsio.com. 5. Locate the driver package. 6. Click Install button to install the driver. Figure 7.4.1 (c) – Bulk Driver Installation module Chelsio T5/T4 Unified Wire for Linux Page 280...
  • Page 281 If you deselect the ‘Remember Password’ option, you will be asked to enter password every time you try to connect to the system. Figure 7.4.2 - Adding a UM Agent Chelsio T5/T4 Unified Wire for Linux Page 281...
  • Page 282 Recipients field separated a comma. Enter Mail server details and ensure that the “Enable email Alerts” field is enabled. Select the Agent(s), for which you want to receive alerts and Click on Save. Chelsio T5/T4 Unified Wire for Linux Page 282...
  • Page 283 Various configurable parameters have been categorized into several modules like Boot Configuration module to install and erase option ROM, Network Configuration module to set MTU and VLAN ID, Card Configuration module to change driver parameters, etc. Chelsio T5/T4 Unified Wire for Linux Page 283...
  • Page 284 SNO field and finally click Delete Member(s). Figure 7.4.6 (a) – Creating a group Chelsio T5/T4 Unified Wire for Linux Page 284...
  • Page 285 Using this module, you can install option ROM or erase option ROM on Chelsio network devices. The Set Default Boot Settings button will reset the adapter to factory boot settings. Figure 7.4.6 (c) – Boot Configuration module Chelsio T5/T4 Unified Wire for Linux Page 285...
  • Page 286 MTU can be set between 1500-9000 bytes. VLAN id can be set for an adapter within the range 0-4094 (enter 0 to disable it). Figure 7.4.6 (d) – Network Configuration module Chelsio T5/T4 Unified Wire for Linux Page 286...
  • Page 287 Offload. Offload settings are only available when using the TOE capable drivers (t4_tom and toecore for T5 and T4 adapters; t3_tom and toecore for T3 adapters). Figure 7.4.6 (e) – Card Configuration module Chelsio T5/T4 Unified Wire for Linux Page 287...
  • Page 288 In the Port Configuration module, you can set various port settings like enabling Tx checksum and TCP segmentation offload, setting Link speed and link duplex mode, etc. The settings depend on the device driver installed. Figure 7.4.6 (f) - Port Configuration module Chelsio T5/T4 Unified Wire for Linux Page 288...
  • Page 289 Use the Bypass Configuration module to configure Chelsio’s bypass adapters like B420-SR and B404-BT. For more information on different bypass modes and configurational parameters, Bypass Driver chapter. Figure 7.4.6 (g) - Bypass Configuration module Chelsio T5/T4 Unified Wire for Linux Page 289...

  • Page 290: System Page

    The system page contains generic system and support modules which are discussed below: 7.5.1. System Summary This module lists the system Hostname, Operating System, platform and also gives the count of the Chelsio cards found. Figure 7.5.1 - System Summary module Chelsio T5/T4 Unified Wire for Linux Page 290...
  • Page 291 Using this module, one can install various Chelsio drivers for different operating systems. You can choose the configuration file type (Linux Agents only). Figure 7.5.2 (a) - Drivers Installation module connected to Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 291...
  • Page 292 XX. Unified Wire Manager (UM) Figure 7.5.2 (b) - Drivers Installation module connected to FreeBSD Agent Chelsio T5/T4 Unified Wire for Linux Page 292...
  • Page 293 To reload a driver select Reload. Finally click Load/Unload Driver button. Click Refresh if changes are not reflected immediately. To reject the load/unload option selected, click Discard Changes. Figure 7.5.3 - Driver Details module Chelsio T5/T4 Unified Wire for Linux Page 293...
  • Page 294 (type of test; you can run more than one test at a time) and click Run Test. After the tests are completed, the results will be displayed in a tabular format. Figure 7.5.4 - System Diagnostics module for a T4 CNA Chelsio T5/T4 Unified Wire for Linux Page 294...
  • Page 295 KVM modules (kvm and kvm_intel/kvm_amd) are loaded. If not done already, access Agent CLI and follow these steps to do so: If any or all the aforementioned drivers are already loaded, unload them before proceeding: [root@host~]# rmmod <kvm_intel/kvm_amd> [root@host~]# rmmod kvm [root@host~]# rmmod cxgb4 Chelsio T5/T4 Unified Wire for Linux Page 295...
  • Page 296 The VM Configurations module allows you to view UUID and domain state of Virtual Machines. You can perform various system power options like start or resume (if VM is paused), turn off, restart or suspend (pause) a VM. Chelsio T5/T4 Unified Wire for Linux Page 296...
  • Page 297 Now, click on any of the system power actions provided at the bottom. Figure 7.5.7 (a) – VM Configurations module Figure 7.5.7 (b) – VM Configurations module: viewing properties for multiple VMs Chelsio T5/T4 Unified Wire for Linux Page 297...
  • Page 298 The VF Configurations module lists all the VMs, Virtual Functions mapped to each Virtual Machine and all the available VFs. You can also add and remove VFs for a particular VM. Figure 7.5.8 - VF Configurations module Chelsio T5/T4 Unified Wire for Linux Page 298...
  • Page 299 Now, click on any of the system power actions provided at the bottom. Figure 7.5.9 (a) – Xen Configurations module Figure 7.5.9 (b) – Xen Configurations module: viewing properties for multiple VMs Chelsio T5/T4 Unified Wire for Linux Page 299...
  • Page 300 Virtual Functions (VFs) you want to create for the physical ports. If VFs are not listed, unload the driver, reload it and create VFs again. Chelsio T5/T4 Unified Wire for Linux Page 300...
  • Page 301 Use the Get Logs button to retrieve, and Hide Logs button to clear the log entries. The Delete Logs button will remove the logs permanently from the agent. Figure 7.5.11 (a) - Managed System Application Logs module for Windows Agent Chelsio T5/T4 Unified Wire for Linux Page 301...

  • Page 302: Network Page

    The modules available on this page are as below: Card summary This module provides PCI, firmware and other details of the card. The card's serial number and factory MAC address are also provided for inventory purposes. Chelsio T5/T4 Unified Wire for Linux Page 302...
  • Page 303 XX. Unified Wire Manager (UM) Figure 7.6.2 (a) - Card Summary module Chelsio T5/T4 Unified Wire for Linux Page 303...
  • Page 304 Highlight the system item in the tree menu on the left, and click Refresh, to refresh data from the system, in case the updated settings are not being shown. Figure 7.6.2 (b) – TCP Offload Settings module for a Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 304...
  • Page 305 Figure 7.6.2 (c) - TCP Offload Settings module for a FreeBSD Agent Device Driver settings (Windows) The device driver settings applicable to the card are shown here. For Chelsio T5 and T4 adapters, only the MaxVMQueues field will be displayed. On changing the settings, the changed settings may not reflect immediately on refreshing the data.
  • Page 306 TCP/IP offload capability. The statistics are for TCP and IP protocol processing done in the card's hardware. These statistics may only be applicable if the card is TOE enabled. Figure 7.6.2 (e) - Card Statistics module for a T3 HBA Chelsio T5/T4 Unified Wire for Linux Page 306...
  • Page 307 The modules available on this page are as below: Port summary Port details such as the Ethernet adapter nam, link status, etc are shown in this module. Figure 7.6.2.1 (a) - Port Summary of T5/T4 CNA on Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 307...
  • Page 308 Port settings such as MTU, Link speed and others can be set in this module. The settings depend on the device driver installed. Figure 7.6.2.1 (b) - Port Settings of T4/T5 CNA on Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 308...
  • Page 309 Network teams are also indicated with an appropriate icon. The primary IP address (IPv4) can be set for the adapter, when it is selected. There is an option to add/modify/delete additional IP Chelsio T5/T4 Unified Wire for Linux Page 309...
  • Page 310 0-4094 (enter 0 to disable it). You can use the View/Set IP addresses option to add, modify or delete IP aliases. Figure 7.6.3 (a) - System network configuration module Chelsio T5/T4 Unified Wire for Linux Page 310...
  • Page 311 XX. Unified Wire Manager (UM) Figure 7.6.3 (b) – Managing IP aliases Chelsio T5/T4 Unified Wire for Linux Page 311...
  • Page 312 Table view in the Advanced NIC characteristics. The polling time field sets the average time (in seconds) based on which the table/graph updates the report. Figure 7.6.3 (c) - System network statistics module Chelsio T5/T4 Unified Wire for Linux Page 312...
  • Page 313 XX. Unified Wire Manager (UM) Figure 7.6.3 (d) - Network Throughput Vs Time instant Graph Figure 7.6.3 (e) - Network Throughput Vs Time instant Table Chelsio T5/T4 Unified Wire for Linux Page 313...
  • Page 314 Windows, you may set multiple default gateways. Use the option to set multiple default gateways with caution, since it may cause the system to stop communicating with external networks. Figure 7.6.3 (f) - Default gateway and DNS configuration module for a FreeBSD Agent Chelsio T5/T4 Unified Wire for Linux Page 314...
  • Page 315 Chelsio cards. Do not mix third party cards and offload-enabled Chelsio cards in a single team. Figure 7.6.3 (g) - Create a network team/bond device module Chelsio T5/T4 Unified Wire for Linux Page 315...
  • Page 316 Use the GetConnections utility to view currently active TCP/UDP connections. Offload status for each connection is also displayed if protocol offload hardware is available. This is useful for troubleshooting any connectivity issues for clients to various services. Figure 7.6.3 (h) - Ping Utility Chelsio T5/T4 Unified Wire for Linux Page 316...
  • Page 317 XX. Unified Wire Manager (UM) Figure 7.6.3 (i) - TraceRoute Utility Chelsio T5/T4 Unified Wire for Linux Page 317...
  • Page 318 To delete a bridge, click on the bridge name and then click “Delete Bridge”. This will also delete the virtual interface attached to that bridge and the VM(s) to which the interface was assigned. Chelsio T5/T4 Unified Wire for Linux Page 318...
  • Page 319 To delete a bridge, click on the bridge name and then click “Delete Bridge”. If there are virtual network interfaces attached, you will have to detach them first. Chelsio T5/T4 Unified Wire for Linux Page 319...
  • Page 320 To assign interfaces on the host to a bridge, specify the bridge name and the host interface in the “Add Interface to Bridge” section. Click “Add”. You can add multiple interfaces to the same bridge using the aforementioned method. The Cost and Priority parameters are optional. Chelsio T5/T4 Unified Wire for Linux Page 320...
  • Page 321 XX. Unified Wire Manager (UM) Figure 7.6.3.1 (c) – Creating Bridge Figure 7.6.3.1 (d) – Adding Bridge to VM Figure 7.6.3.1 (e) – Adding interfafe to Bridge Chelsio T5/T4 Unified Wire for Linux Page 321...
  • Page 322 You can attach upto 64 virtual interfaces to a particular bridge. Device Number, Bridge Name and VM Name are the mandatory fields while creating a virtual interface. Figure 7.6.3.1 (f) – Creating Bridge Figure 7.6.3.1 (g) – Creating and attaching virtual interface Chelsio T5/T4 Unified Wire for Linux Page 322...
  • Page 323 This type of virtual network does is not bound to any physical network adapter and no access to external networks is provided. Chelsio T5/T4 Unified Wire for Linux Page 323...
  • Page 324 Once created, you can manage the virtual networks in the Virtual Network Manager module. Figure 7.6.3.1 (i) – Creating external virtual network Figure 7.6.3.1 (j) – Creating internal virtual network Figure 7.6.3.1 (k) – Creating private virtual network Chelsio T5/T4 Unified Wire for Linux Page 324...
  • Page 325 On Windows Agents, only T3 HBAs are supported currently. Parameters can be set per port. On FreeBSD Agents, only T4 CNAs are supported. iWARP parameter settings can be set only when iw_cxgbe driver is loaded. Chelsio T5/T4 Unified Wire for Linux Page 325...
  • Page 326 XX. Unified Wire Manager (UM) Figure 7.6.4 (a) – iWARP settings for T4 CNA for FreeBSD Agent Chelsio T5/T4 Unified Wire for Linux Page 326...
  • Page 327 XX. Unified Wire Manager (UM) Figure 7.6.4 (b) – iWARP settings for T4 CNA for Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 327...
  • Page 328 The WD-UDP module lists the process ids (pid) of UDP traffic running on the agent and displays the corresponding statistics and attributes. Please ensure that WD-UDP traffic is running on the agent before accessing this Note module. Figure 7.6.5 (a) – WD-UDP Process Statistics Chelsio T5/T4 Unified Wire for Linux Page 328...
  • Page 329 XX. Unified Wire Manager (UM) Figure 7.6.5 (b) – WD-UDP Process Attributes Chelsio T5/T4 Unified Wire for Linux Page 329...
  • Page 330 Please ensure that WD-TOE traffic is running on the agent before accessing this Note module. Figure 7.6.5 (c) – WD-TOE Process Statistics Figure 7.6.5 (d) – WD-TOE Process Attributes Chelsio T5/T4 Unified Wire for Linux Page 330...

  • Page 331: Storage Page

    This module provides details about the driver installed; such as driver name and its version. The module also gives information about the number of FCoE enabled cards that are present on the machine. Figure 7.7.1 (a) – FCoE Initiator Summary module for Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 331...
  • Page 332 Figure 7.7.1.1 (a) – FCoE Card Summary module FCoE Attributes Information such as Interrupt modes (MSI/MSI-X/INTx), SCSI mode and the card state are provided in this module. Figure 7.7.1.1 (b) – FCoE Attributes module Chelsio T5/T4 Unified Wire for Linux Page 332...
  • Page 333 FCoE Port Summary The SCSI adapter name and the underlying ENODE MAC address of the physical port can be found here. Figure 7.7.1.2 (a) – FCoE Port Summary module for Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 333...
  • Page 334 (10G/1G) being used. Note that only class 3 service is supported by the initiator for now and the frame size is fixed to 2128 bytes as per spec. Figure 7.7.1.2 (b) – FCoE Port Attributes module for Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 334...
  • Page 335 To delete a virtual port, select the option Delete and select the virtual port WWPN which you want to delete and click on delete. Figure 7.7.1.2 (c) – FCoE NPIV management module Chelsio T5/T4 Unified Wire for Linux Page 335...
  • Page 336 This module provides details about the discovered target such as target’s FC ID, WWPN and WWNN so that the user can identify the discovered target accordingly. Figure 7.7.1.3 (a) – FCoE Remote Port Attributes module Chelsio T5/T4 Unified Wire for Linux Page 336...
  • Page 337 For Linux, the SCSI address is displayed in H:C:T:L (Host:Channel:Target:Lun) format and for Windows, it is displayed in P:B:T:L(SCSI Port:Bus:Target:Lun) format. Figure 7.7.1.3 (b) – FCoE Remote Port Lun Details module Chelsio T5/T4 Unified Wire for Linux Page 337...
  • Page 338 FCoE Virtual Port Summary The SCSI adapter name and the underlying ENODE MAC address of the physical port can be found here. Figure 7.7.1.4 (a) – FCoE Virtual Port Summary module for Linux Agent Chelsio T5/T4 Unified Wire for Linux Page 338...
  • Page 339 (10G/1G) being used. Note that only class 3 service is supported by the initiator for now and the frame size is fixed to 2128 bytes as per spec. Figure 7.7.1.4 (b) – FCoE Virtual Port Attributes module Chelsio T5/T4 Unified Wire for Linux Page 339...
  • Page 340 LUN, SCSI address, and LUN address are provided. For Linux, the SCSI address is displayed in H:C:T:L (Host:Channel:Target:Lun) format and for Windows, it is displayed in P:B:T:L(SCSI Port:Bus:Target:Lun) format. Figure 7.7.1.4 (d) - FCoE Remote Port Lun Details module Chelsio T5/T4 Unified Wire for Linux Page 340...
  • Page 341 The CHAP authentication secret should be between 12 and 16 characters in length, and the initiator's IQN name should start with "iqn." Figure 7.7.2 (a) - Open iSCSI initiator Chelsio T5/T4 Unified Wire for Linux Page 341...
  • Page 342 XX. Unified Wire Manager (UM) Figure 7.7.2 (b) - Microsoft iSCSI initiator Chelsio T5/T4 Unified Wire for Linux Page 342...
  • Page 343 XX. Unified Wire Manager (UM) Figure 7.7.2 (c) - Chelsio iSCSI initiator Chelsio T5/T4 Unified Wire for Linux Page 343...
  • Page 344 (the IP address and TCP Port specified when discovering the target). If there are any pre-existing iSCSI sessions established to the target, deletion of Note the target Portal from the discovered targets list will fail. Figure 7.7.2 (d) - Discover targets module Chelsio T5/T4 Unified Wire for Linux Page 344...
  • Page 345 XX. Unified Wire Manager (UM) Figure 7.7.2 (e) - Discover targets module Chelsio T5/T4 Unified Wire for Linux Page 345...
  • Page 346 In Windows, you can specify the initiator HBA to use and its IP address while logging in. Figure 7.7.2 (f) - Targets module after logging in Chelsio T5/T4 Unified Wire for Linux Page 346...
  • Page 347 You can also view details like link id, status, enode mac, etc of all the ports of the selected adapter. Figure 7.7.3 (a) - Full Offload iSCSI Hardware Information module Chelsio T5/T4 Unified Wire for Linux Page 347...
  • Page 348 The Port Up and Port Down buttons will enable and disable the selected port respectively. The Clear IP button deletes values set for the IP Type, IP, Subnet Mask and Gateway properties and resets them. Figure 7.7.3 (b) - FO iSCSI Manage Ports module Chelsio T5/T4 Unified Wire for Linux Page 348...
  • Page 349 In the FO iSCSI Initiator Properties module, you can configure FO iSCSI Initiator by setting different properties like enabling/disabling CHAP authentication, setting Header and Data digest, etc. Figure 7.7.3 (c) - FO iSCSI Initiator Properties Chelsio T5/T4 Unified Wire for Linux Page 349...
  • Page 350 In this module, you can set upto 8 instances. Configurable parameters include initiator node name (IQN), alias (friendly) name, Initiator (CHAP) Username and password. Figure 7.7.3 (d) - FO iSCSI Manage Instances module Chelsio T5/T4 Unified Wire for Linux Page 350...
  • Page 351 (target) IP. Finally, click Discover. After successful discovery, all the discovered targets will appear in the Discovered Targets section. To view more details, click on the Target name. Figure 7.7.3 (e) - FO iSCSI Discover Details module Chelsio T5/T4 Unified Wire for Linux Page 351...
  • Page 352 CHAP for a secure iSCSI connection. One-way (target authenticates the initiator) and Mutual (target and initiator authenticate each other) authentication methods are supported. Figure 7.7.3 (f) - FO iSCSI Session Details module: Login Chelsio T5/T4 Unified Wire for Linux Page 352...
  • Page 353 Established sessions section. Select the Adapter and session id. Details of the selected session will be displayed. To end the session, click Logout. Figure 7.7.3 (g) - FO iSCSI Session Details module: Established Sessions Chelsio T5/T4 Unified Wire for Linux Page 353...
  • Page 354 Properties such as Target name and Alias, Max Data Receive Length, Authentication mode related to a specific iSCSI target can be viewed and modified here. iSCSI targets can be started/stopped or deleted. Chelsio T5/T4 Unified Wire for Linux Page 354...
  • Page 355 XX. Unified Wire Manager (UM) Figure 7.7.4 (b) - Target properties module Chelsio T5/T4 Unified Wire for Linux Page 355...
  • Page 356 XX. Unified Wire Manager (UM) Session details Details including Session ID, Initiator IQN and Connections List of all discovered and currently connected iSCSI targets are listed here. Figure 7.7.4 (c) - Session Details module Chelsio T5/T4 Unified Wire for Linux Page 356...
  • Page 357 XX. Unified Wire Manager (UM) New Target Creation New iSCSI target can be created here by specifying the Target IQN and Target Alias name. Figure 7.7.4 (d) - New Target Creation module Chelsio T5/T4 Unified Wire for Linux Page 357...
  • Page 358 View/Edit iSCSI Target LUNs This module displays various Logical Units created in an iSCSI Target. Selected LUNs can be deleted. Figure 7.7.5 (a) - View/Edit iSCSI Target LUNs module Chelsio T5/T4 Unified Wire for Linux Page 358...
  • Page 359 Device and RAM Disk Size etc. RW (Read-Write) and RO (Read Only) are the two kinds of permissions that can be set. If Ram Disk is selected, then a minimum of 16 MB should be provided. Figure 7.7.5 (b) - Adding a new LUN Chelsio T5/T4 Unified Wire for Linux Page 359...
  • Page 360 Portal List on the left displays details of the portal group on which an iSCSI target is listening and the related info is displayed on the right under Portal Details. Selected portals can be deleted. Figure 7.7.6 (a) - View/Edit iSCSI Target Portals module Chelsio T5/T4 Unified Wire for Linux Page 360...
  • Page 361 XX. Unified Wire Manager (UM) Add Portal New Portals can be added here by choosing the specific target and Portal IP address. The Port number should be 3260. Figure 7.7.6 (b) - Adding a new Portal Chelsio T5/T4 Unified Wire for Linux Page 361...
  • Page 362 View/Edit iSCSI Target ACLs This module displays details for all the ACLs configured for an iSCSI Target. Selected ACLs can be deleted. Figure 7.7.7 (a) - Target ACL operations module Chelsio T5/T4 Unified Wire for Linux Page 362...
  • Page 363 XX. Unified Wire Manager (UM) Add ACL New ACLs can be configured by specifying Target name, initiator IQN name, IP address and permission type. Figure 7.7.7 (b) - Adding new ACL Chelsio T5/T4 Unified Wire for Linux Page 363...

  • Page 364: Hardware Features

    Protocol field are tcp, udp and frag. The Priority field can be used to determine the priority of a filter when Vlan ids are same. Insert at position features allows user to add a filter at a specified position. Chelsio T5/T4 Unified Wire for Linux Page 364...
  • Page 365 T5/T4 CARD fields are mandatory. After providing appropriate values for the parameters click “Save Changes”. For a detailed explanation regarding different fields, please refer cxgbtool Note manual by running man cxgbtool command on Management Agent CLI. Chelsio T5/T4 Unified Wire for Linux Page 365...
  • Page 366 Policies can be added and deleted. Policy Details displays the primitives (maximum 8) and actions which can be modified. For more details on creating policies, please refer to COP man pages. Chelsio T5/T4 Unified Wire for Linux Page 366...
  • Page 367 /ChelsioUwire- x.xx.x.x/src/network/firmware. For instance, to select Low latency Networking for T4 adapter, locate the file, t4-config.txt, in /ChelsioUwire-x.xx.x.x/src/network/firmware/low_latency_config directory. Chelsio T5/T4 Unified Wire for Linux Page 367...
  • Page 368 Figure 7.8.3 (a) – T4/T5 Save Config File module T5/T4 Boot Option ROM management This module allows managing the PXE and FCoE boot capability for Chelsio T5 and T4 adapters. The Option ROM (PXE and FCoE) may be installed to or erased from the card. The version of Option ROM flashed can be viewed here.
  • Page 369 Using the Boot Devices option you can set various iSCSI target properties. Enable Option ROM only if you are planning to boot the system via PXE or Note install the operating system on discovered iSCSI or FCoE LUN. Chelsio T5/T4 Unified Wire for Linux Page 369...
  • Page 370 XX. Unified Wire Manager (UM) Figure 7.8.3 (c) - PXE Boot configuration for T4 CNAs Chelsio T5/T4 Unified Wire for Linux Page 370...
  • Page 371 XX. Unified Wire Manager (UM) Figure 7.8.3 (d) - FCoE Boot configuration for T4 CNAs: Function parameter Figure 7.8.3 (e) - FCoE Boot configuration for T4 CNAs: Boot parameter Chelsio T5/T4 Unified Wire for Linux Page 371...
  • Page 372 XX. Unified Wire Manager (UM) Figure 7.8.3 (f) - FCoE Boot configuration for T4 CNAs: Show WWPN parameter Figure 7.8.3 (g) - iSCSI Boot configuration for T4 CNAs: Function parameter Chelsio T5/T4 Unified Wire for Linux Page 372...
  • Page 373 XX. Unified Wire Manager (UM) Figure 7.8.3 (h) - iSCSI Boot configuration for T4 CNAs: Initiator parameter Chelsio T5/T4 Unified Wire for Linux Page 373...
  • Page 374 XX. Unified Wire Manager (UM) Figure 7.8.3 (i) - iSCSI Boot configuration for T4 CNAs: Network parameter Chelsio T5/T4 Unified Wire for Linux Page 374...
  • Page 375 Current bypass mode. The Watchdog timer is used to ensure that if there is a software failure, the switch will enter the default state. The Watchdog timeout value should be provided in milliseconds. Chelsio T5/T4 Unified Wire for Linux Page 375...
  • Page 376 Purge table: Delete all the rules present in the selected table. This action will not delete the table.  Activate table: Enable the selected table.  Deactivate table: Disable the selected table. Chelsio T5/T4 Unified Wire for Linux Page 376...
  • Page 377 To delete a rule, select the corresponding checkbox and click Delete a Filter. Finally, click Save Changes. You can delete multiple rules using this method. For more information on Redirection, see Bypass Driver chapter. Figure 7.8.4 (b) - Redirect Configuration module: Table configuration tab Chelsio T5/T4 Unified Wire for Linux Page 377...
  • Page 378 XX. Unified Wire Manager (UM) Figure 7.8.4 (c) - Redirect Configuration module: Rules configuration tab Chelsio T5/T4 Unified Wire for Linux Page 378...
  • Page 379 Limiting, Class Weighted Round Robin and Channel Rate Limiting). Based on the parameters specified, different scheduler levels can be configured. To know more about the levels and related parameters, please refer Traffic Management chapter (Click here). 7.8.5 (b) – Egress Packet Scheduler module Chelsio T5/T4 Unified Wire for Linux Page 379...

  • Page 380: Uninstalling Unified Wire Manager

    1. Use the following query command to determine the name of the client RPM: [root@host~]# rpm –qa | grep chelsio-uwire_mgmt-client 2. Now, execute the following command with the result from the above query to uninstall Management Client: E.g. for RHEL 6.3: [root@host~]# rpm –e chelsio-uwire_mgmt-client-rhel6u3-2.2-xyz.x86_64 Chelsio T5/T4 Unified Wire for Linux Page 380...

  • Page 381: Uninstalling Management Station

    1. Use the following query command to determine the name of the Management Station RPM: [root@host~]# rpm –qa | grep chelsio-uwire_mgmt-station 2. Now, execute the following command with the result from the above query to uninstall Management Station: E.g. for RHEL 6.3: [root@host~]# rpm –e chelsio-uwire_mgmt-station-rhel6u3-2.2-xyz.x86_64 Chelsio T5/T4 Unified Wire for Linux Page 381...

  • Page 382: Unified Boot

    Chapter XXI. Unified Boot XXI. Unified Boot Chelsio T5/T4 Unified Wire for Linux Page 382...

  • Page 383: Introduction

    * If system BIOS version is lower than 1.5 and both Legacy and uEFI are enabled, system will hang during POST. Please upgrade the BIOS version to 1.5 or higher to avoid this issue. Chelsio T5/T4 Unified Wire for Linux Page 383...
  • Page 384 Following are the currently shipping Chelsio Adapters that are compatible with Chelsio Unified Boot software: T580-OCP-SO T520-OCP-SO T520-BT T580-CR T520-LL-CR T520-SO-CR T520-CR T540-CR T580-LP-CR T580-SO-CR T420-CR T440-CR T422-CR T404-BT T420-BCH T420-SO-CR T440-LP-CR T420-LL-CR T420-BT Chelsio T5/T4 Unified Wire for Linux Page 384...

  • Page 385: Software Requirements

    No separate DUD required for SLES 12 OS installation, as inbox drivers can be Note used. FCoE and iSCSI Boot are not supported with the current release. Support will be Note added in upcoming releases. Chelsio T5/T4 Unified Wire for Linux Page 385...

  • Page 386: Flashing Firmware And Option Rom

    Plug-in the USB flash drive in the system on which the Chelsio CNA is installed. vii. Reboot the system and go into the BIOS setup. viii. Make the USB flash drive as the primary boot device. ix. Save the changes. Chelsio T5/T4 Unified Wire for Linux Page 386...

  • Page 387: Legacy

    Once the system boots from the USB flash drive, change your working directory to CHELSIO directory: C:\>cd CHELSIO iii. Run the following command to list all Chelsio CNAs present on the system. The list displays a unique index for each CNA found. C:\CHELSIO>cfut4 –l Chelsio T5/T4 Unified Wire for Linux Page 387...
  • Page 388 Run the following command to flash the appropriate firmware (t5fw-x.xx.xx.x.bin for T5 adapters; t4fw-x.xx.xx.x.bin for T4 adapters). C:\CHELSIO>cfut4 -d <idx> -uf <firmware_file>.bin firmware_file Here, is the firmware image file present in the CHELSIO directory. Chelsio T5/T4 Unified Wire for Linux Page 388...
  • Page 389 Flash the unified option ROM onto the Chelsio CNA using the following command: C:\CHELSIO>cfut4 -d <idx> -ub cubt4.bin Here, cubt4.bin is the unified option ROM image file present in the CHELSIO directory. ix. Flash the default boot configuration file. C:\CHELSIO>cfut4 -d <idx> -uc bootcfg Chelsio T5/T4 Unified Wire for Linux Page 389...

  • Page 390: Uefi

    Reboot the system for changes to take effect. 2.3. uEFI In BIOS, configure the system having Chelsio CNA to boot in uEFI mode. For Supermicro systems, enable Network Stack as well before proceeding. Note Chelsio T5/T4 Unified Wire for Linux Page 390...
  • Page 391 Chelsio uEFI driver is loaded. If the driver is loaded (as shown in the image below), continue to step (iv) If the driver is not loaded, load the uEFI driver (ChelsioUD.efi) found in the CHELSIO directory, and try again. Chelsio T5/T4 Unified Wire for Linux Page 391...
  • Page 392 If Chelsio T4/T5 is not listed, please ensure that Chelsio uEFI driver is loaded Note correctly as mentioned here in the Flashing Firmware and Option ROM section. vi. Highlight the Chelsio adapter to be configured and press [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 392...
  • Page 393 Erase or update firmware using the methods explained below:  Erase existing firmware a. Select [Erase] as Flash Operation b. Select [FW File] as Flash File Type c. Select Update/Erase d. Press [Y] to confirm. e. Reboot system. Chelsio T5/T4 Unified Wire for Linux Page 393...
  • Page 394 Press [Enter] e. Select Update/Erase Press [Y] to confirm. g. Reboot system Similarly, you can use the above method to update/erase Option ROM, (T5/T4) Boot Configuration, Hardware Configuration and Phy Firmware file. Chelsio T5/T4 Unified Wire for Linux Page 394...

  • Page 395: Configuring Pxe Server

    (Use port # 67, set PXE option and provide bootable file name in settings)  http://unattended.sourceforge.net/pxe-win2k.html Chelsio Communications does not take any responsibility regarding contents given Note in above mentioned links. Those are given for example purposes only. Chelsio T5/T4 Unified Wire for Linux...

  • Page 396: Pxe Boot Process

    Press [Alt+C] when the message Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2015 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 397 Enable the Adapter BIOS using arrow keys if not already enabled. Hit [ENTER]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose PXE from the list to configure. Hit [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 397...
  • Page 398 Chapter XXI. Unified Boot vii. Use the arrow keys to highlight the appropriate function among the supported NIC functions and hit [Enter] to select. viii. Enable NIC function bios if not already enabled. Chelsio T5/T4 Unified Wire for Linux Page 398...
  • Page 399 Hit [F10] or [Esc] and then [Y] to save configuration changes. xi. Reboot the system. xii. Allow the Chelsio option ROM to initialize and setup PXE devices. DO NOT PRESS ALT-S to skip Chelsio option ROM. Chelsio T5/T4 Unified Wire for Linux Page 399...
  • Page 400 In the system setup, choose any of the Chelsio PXE devices as the first boot device. xiv. Reboot. DO NOT PRESS ALT-S to skip Chelsio option ROM, during POST. xv. Hit [F12] key when prompted to start PXE boot. Chelsio T5/T4 Unified Wire for Linux Page 400...

  • Page 401: Uefi Pxe Boot

    Chelsio HII should be listed as Chelsio T4/T5. Highlight it and press [Enter]. If Chelsio T4/T5 is not listed, please ensure that Chelsio uEFI driver is loaded Note correctly as mentioned here in the Flashing Firmware and Option ROM section. Chelsio T5/T4 Unified Wire for Linux Page 401...
  • Page 402 Chapter XXI. Unified Boot iii. Select the Chelsio adapter to be configured and press [Enter]. iv. Select Configuration Utility and press [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 402...
  • Page 403 Enable adapter BIOS if not already enabled. It is highly recommended that you use the Save Changes option every time a Note parameter/option is changed. vi. Select Chelsio Protocol Selection and press [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 403...
  • Page 404 0 for PXE, enabling PXE Function 1 will enable port 1 and so on, for NIC function. Please refer the table below: NIC Function enabled Ports enabled PXE Function 0 PXE Function 1 PXE Function 2 PXE Function 3 Chelsio T5/T4 Unified Wire for Linux Page 404...
  • Page 405 Chapter XXI. Unified Boot ix. Select Save Changes and press [Enter] Chelsio T5/T4 Unified Wire for Linux Page 405...
  • Page 406 Reboot the system and choose any of the available Chelsio PXE devices for PXE boot. xi. Reboot and hit [F12] key when prompted to start PXE boot. xii. Chelsio option ROM will now initialize and setup PXE devices. Chelsio T5/T4 Unified Wire for Linux Page 406...

  • Page 407: Fcoe Boot Process

    Reboot the system. Press [Alt+C] when the message “Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2015 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS” appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 408 Enable the Adapter BIOS if not already enabled. Hit [ENTER]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose FCoE from the list to configure and hit [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 408...
  • Page 409 Chapter XXI. Unified Boot vii. Choose the first option, Configure function parameters, from the list of parameter type and hit [Enter]. viii. Enable FCoE BIOS if not already enabled. Chelsio T5/T4 Unified Wire for Linux Page 409...
  • Page 410 Chapter XXI. Unified Boot ix. Choose the order of the ports to discover FCoE targets. Set discovery timeout to a suitable value. Recommended value is >= 30. Chelsio T5/T4 Unified Wire for Linux Page 410...
  • Page 411 Chapter XXI. Unified Boot xi. Hit [F10] or [Esc] and then [Y] to save the configuration. xii. Choose Configure boot parameters. Chelsio T5/T4 Unified Wire for Linux Page 411...
  • Page 412 Select the first boot device and hit [Enter] to discover FC/FCoE targets connected to the switch. Wait till all reachable targets are discovered. xiv. List of discovered targets will be displayed. Highlight a target using the arrow keys and hit [Enter] to select. Chelsio T5/T4 Unified Wire for Linux Page 412...
  • Page 413 Chapter XXI. Unified Boot xv. From the list of LUNs displayed for the selected target, choose one on which operating system has to be installed. Hit [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 413...
  • Page 414 Chapter XXI. Unified Boot xvi. Hit [F10] or [Esc] and then [Y] to save the configuration. xvii. Reboot the machine. xviii. During POST, allow the Chelsio option ROM to discover FCoE targets. Chelsio T5/T4 Unified Wire for Linux Page 414...
  • Page 415 Enter BIOS setup and choose FCoE disk discovered via Chelsio adapter as the first boot device. xx. Reboot and boot from the FCoE disk or install the required OS using PXE. Chelsio T5/T4 Unified Wire for Linux Page 415...

  • Page 416: Uefi Fcoe Boot

    Select Chelsio T4/T5 and press [Enter] If Chelsio T4/T5 is not listed, please ensure that Chelsio uEFI driver is loaded Note correctly as mentioned here in the Flashing Firmware and Option ROM section. Chelsio T5/T4 Unified Wire for Linux Page 416...
  • Page 417 Chapter XXI. Unified Boot iii. Select the Chelsio adapter to be configured and press [Enter]. iv. Select Configuration Utility and press [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 417...
  • Page 418 Chapter XXI. Unified Boot Enable adapter BIOS if not already enabled. It is highly recommended that you use the Save Changes option every time a Note parameter/option is changed. Chelsio T5/T4 Unified Wire for Linux Page 418...
  • Page 419 Chapter XXI. Unified Boot vi. Select Chelsio Protocol Selection and press [Enter]. vii. Select FCoE and press [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 419...
  • Page 420 Chapter XXI. Unified Boot viii. Under Function Parameters, enable FCoE BIOS, if not already enabled. Chelsio T5/T4 Unified Wire for Linux Page 420...
  • Page 421 Chapter XXI. Unified Boot ix. Set discovery timeout to a suitable value. Recommended value is >= 30 Chelsio T5/T4 Unified Wire for Linux Page 421...
  • Page 422 Chapter XXI. Unified Boot Choose the order of the ports to discover FCoE targets. Chelsio T5/T4 Unified Wire for Linux Page 422...
  • Page 423 Under the first boot device, select Discover Target and press [Enter] to discover FC/FCoE targets connected to the switch. Wait till all reachable targets are discovered. xii. List of discovered targets will be displayed. Highlight a target to select it and hit [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 423...
  • Page 424 List of LUNs for the selected target will be displayed. Highlight a LUN to select it and hit [Enter]. xiv. Select Save Changes and press [Enter]. xv. Reboot the system for changes to take effect. Chelsio T5/T4 Unified Wire for Linux Page 424...
  • Page 425 The discovered LUN should appear in the Boot Configuration section and system BIOS section. xvii. Select the LUN as the first boot device and exit from BIOS. xviii. Either boot from the LUN or install the required OS. Chelsio T5/T4 Unified Wire for Linux Page 425...

  • Page 426: Iscsi Boot Process

    Reboot the system. Press [Alt+C] when the message “Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2015 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS” appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 427 Enable the Adapter BIOS if not already enabled. Hit [Enter]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose iSCSI from the list to configure and hit [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 427...
  • Page 428 Enable iSCSI BIOS if not already enabled. iBFT (iSCSI Boot Firmware Table) will be selected by default. You can also configure the number of iSCSI login attempts (retries) in case the network is unreachable or slow. Chelsio T5/T4 Unified Wire for Linux Page 428...
  • Page 429 Chapter XXI. Unified Boot ix. Choose the order of the ports to discover iSCSI targets. Set discovery timeout to a suitable value. Recommended value is >= 30. Chelsio T5/T4 Unified Wire for Linux Page 429...
  • Page 430 Chapter XXI. Unified Boot xi. Hit [Esc] and then [Y] to save the configuration. xii. Go back and choose Configure Initiator Parameters to configure initiator related properties. Chelsio T5/T4 Unified Wire for Linux Page 430...
  • Page 431 Initiator properties like IQN, Header Digest, Data Digest, etc will be displayed. Change the values appropriately or continue with the default values. Hit [F10] to save. xiv. CHAP authentication is disabled by default. To enable and configure, go back and choose Configure CHAP Parameters Chelsio T5/T4 Unified Wire for Linux Page 431...
  • Page 432 Finally, provide Initiator and Target CHAP credentials according to the authentication method selected. Hit [F10] to save. xvi. Go back and choose Configure Network Parameters to configure iSCSI Network related properties. Chelsio T5/T4 Unified Wire for Linux Page 432...
  • Page 433 Select the port using which you want to connect to the target. Hit [Enter]. xviii. Select Yes in the Enable DHCP field to configure port using DHCP or No to manually configure the port. Hit [F10] to save. Chelsio T5/T4 Unified Wire for Linux Page 433...
  • Page 434 If you want to discover target using DHCP, select Yes in the Discover Boot Target via DHCP field. To discover target via static IP, select No and provide the target IP and Hit [F10] to save. The default TCP port selected is 3260. Chelsio T5/T4 Unified Wire for Linux Page 434...
  • Page 435 Chapter XXI. Unified Boot xxi. Go back and choose Discover iSCSI Target (s) to connect to a target. xxii. Select the portal group on which iSCSI service is provided by the target. Chelsio T5/T4 Unified Wire for Linux Page 435...
  • Page 436 A list of available targets will be displayed. Select the target you wish to connect to and hit [Enter]. xxiv. A list of LUNs configured on the selected target will be displayed. Select the LUN you wish to connect to and hit [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 436...
  • Page 437 Chapter XXI. Unified Boot xxv. Hit [Esc] and then [Y] to save the configuration. xxvi. Reboot the machine. xxvii. During POST, allow the Chelsio option ROM to discover iSCSI targets. Chelsio T5/T4 Unified Wire for Linux Page 437...
  • Page 438 Enter BIOS setup and choose iSCSI target LUN discovered via Chelsio adapter as the first boot device. xxix. Reboot and boot from the iSCSI Target LUN or install the required OS using PXE. Chelsio T5/T4 Unified Wire for Linux Page 438...

  • Page 439: Uefi Iscsi Boot

    Select Chelsio T4/T5 and press [Enter] If Chelsio T4/T5 is not listed, please ensure that Chelsio uEFI driver is loaded Note correctly as mentioned here in the Flashing Firmware and Option ROM section. Chelsio T5/T4 Unified Wire for Linux Page 439...
  • Page 440 Chapter XXI. Unified Boot iii. Select the Chelsio adapter to be configured and press [Enter]. iv. Select Configuration Utility and press [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 440...
  • Page 441 Chapter XXI. Unified Boot Enable adapter BIOS if not already enabled. It is highly recommended that you use the Save Changes option every time a Note parameter/option is changed. Chelsio T5/T4 Unified Wire for Linux Page 441...
  • Page 442 Chapter XXI. Unified Boot vi. Select Chelsio Protocol Selection and press [Enter]. vii. Select iSCSI and press [Enter] Chelsio T5/T4 Unified Wire for Linux Page 442...
  • Page 443 Chapter XXI. Unified Boot viii. Under Function Parameters, enable iSCSI BIOS, if not already enabled. ix. Set discovery timeout to a suitable value. Recommended value is >= 30 Chelsio T5/T4 Unified Wire for Linux Page 443...
  • Page 444 Choose the order of the ports to discover iSCSI targets. xi. Under Initiator Parameters, iSCSI Initiaitor properties like IQN, Header Digest, Data Digest, etc will be displayed. Change the values appropriately or continue with the default values. Chelsio T5/T4 Unified Wire for Linux Page 444...
  • Page 445 Under the first port, select Enable DHCP field, hit [Enter] and select Enabled. This will configure port using DHCP. Select Disabled to manually configure the port. xiii. Under Target Parameters, select Enabled for the Boot Target via DHCP parameter to discover target using DHCP. Chelsio T5/T4 Unified Wire for Linux Page 445...
  • Page 446 To discover target via static IP, select Disabled and provide the target IP. xiv. CHAP authentication is disabled by default. To enable and configure, highlight CHAP Policy and hit [Enter]. Select the policy type from the corresponding pop-up and hit [Enter] again. Chelsio T5/T4 Unified Wire for Linux Page 446...
  • Page 447 Provide Initiator and Target CHAP credentials according to the CHAP policy selected. xvi. Select Discover Target and press [Enter] to discover iSCSI targets connected to the switch. Wait till all reachable targets are discovered. Chelsio T5/T4 Unified Wire for Linux Page 447...
  • Page 448 A list of available targets will be displayed. Select the target you wish to connect to and hit [Enter]. xviii. A list of LUNs configured on the selected target will be displayed. Select the LUN you wish to connect to and hit [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 448...
  • Page 449 Chapter XXI. Unified Boot xix. Select Save Changes and press [Enter] xix. Reboot the system for changes to take effect. Chelsio T5/T4 Unified Wire for Linux Page 449...
  • Page 450 The discovered LUN should appear in the Boot Configuration section and system BIOS. xxi. Select the LUN as the first boot device and exit from BIOS. xxii. Either boot from the LUN or install the required OS. Chelsio T5/T4 Unified Wire for Linux Page 450...

  • Page 451: Creating Driver Update Disk (Dud)

    For RHEL 7.X, use Chelsio-DriverUpdateDisk-RHEL7.X-x86_64-x.xx.x.x.iso Note 7.2. Creating DUD for Suse Enterprise Linux If you haven’t done already, download ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center, service.chelsio.com ii. Untar the package, [root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x.tar.gz Chelsio T5/T4 Unified Wire for Linux Page 451...
  • Page 452 [root@host~]# mkfs.vfat /dev/sda1 Depending on the distribution to be installed, copy the corresponding image file to the USB stick. For example, execute the following command for SLES11sp3. [root@host~]# dd if=/root/ChelsioUwire-x.xx.x.x/Uboot/LinuxDUD/Chelsio- DriverUpdateDisk-SLES11sp3-x86_64-x.x.x.x-y.img of=/dev/sda1 Chelsio T5/T4 Unified Wire for Linux Page 452...

  • Page 453: Os Installation

    Network. There may be built-in Chelsio driver in these distributions. The driver may or may not work with Chelsio adapters, depending on the adapter in use, and the version of the driver that shipped in that particular distribution. Please flash the firmware provided in the package. Chelsio T5/T4 Unified Wire for Linux Page 453...
  • Page 454 SCSI disks in Linux. Enter the index number of the device to be used and hit [Enter]. iii. The installer will search and display DUD image files found in the selected device. Enter the index number of the file to be used and hit [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 454...
  • Page 455 “C” to skip and start the loading process. vii. After the drivers are successfully loaded, OS installation will commence. Proceed as usual. Chelsio T5/T4 Unified Wire for Linux Page 455...
  • Page 456 The dd option specifies that you will be providing a Driver Update Disk during the installation. ii. The installer will load and prompt you for the driver update disk. Select “Yes” and hit [Enter] to proceed. Chelsio T5/T4 Unified Wire for Linux Page 456...
  • Page 457 Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. Chelsio T5/T4 Unified Wire for Linux Page 457...
  • Page 458 The installer will ask if you wish to load more drivers. Choose “Yes” to load if you have any other drivers to load. Otherwise choose “No”. vi. Select the required language from the list. Chelsio T5/T4 Unified Wire for Linux Page 458...
  • Page 459 Select the type of keyboard you have from the list. viii. In this step, you can choose the source which contains the OS installation ISO image. In this case, select “NFS directory”. Chelsio T5/T4 Unified Wire for Linux Page 459...
  • Page 460 Here you can specify if you want to configure your network interfaces using DHCP or manually using IPv4. IPv6 is currently not supported. Hence disable IPv6 before proceeding. Chelsio T5/T4 Unified Wire for Linux Page 460...
  • Page 461 USB drive with DUD image is inserted. No DUD required for SLES12 PXE and iSCSI, as inbox drivers can be used. Note ii. Select the appropriate entry from the PXE menu and press [Enter]. Chelsio T5/T4 Unified Wire for Linux Page 461...
  • Page 462 Chapter XXI. Unified Boot iii. Select “Start Installation” and then “Start Installation or Update”. Chelsio T5/T4 Unified Wire for Linux Page 462...
  • Page 463 Linux kernel and drivers to determine the name of NIC interface by pressing [Alt] + [F3] or [Alt] + [F4]. Press [Alt] + [F1] to get back to the list. Chelsio T5/T4 Unified Wire for Linux Page 463...

  • Page 464: Installation On Fcoe Lun

    Type e and then dd at the boot prompt for RHEL 7. For RHEL 6 and SLES distributions, press Tab and then dd. iv. Load Chelsio Driver Update Disk depending on the Linux distribution (Click here for RHEL 7.x; Click here for RHEL6.x; Click here for SLES11 SPx/SLES12). Chelsio T5/T4 Unified Wire for Linux Page 464...
  • Page 465 SLES11 SPx/ SLES12) section to load Chelsio Driver Update Disk. After successfully loading Chelsio DUD, follow the procedure mentioned below to continue installation, based on the distribution. 8.2.1. RHEL 7.x Choose your installation language and click Continue Chelsio T5/T4 Unified Wire for Linux Page 465...
  • Page 466 Chapter XXI. Unified Boot Click INSTALLATION DESTINATION under SYSTEM. Chelsio T5/T4 Unified Wire for Linux Page 466...
  • Page 467 The discovered FC/FCoE LUNs will appear as local storage in the Local Standard Disks section. Select the LUN which was saved as boot device in system BIOS. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation. Chelsio T5/T4 Unified Wire for Linux Page 467...
  • Page 468 Under Other Storage Options, you can either chose to configure partition automatically or manually. Select the appropriate option and click Done. Then proceed with the installation as usual. 8.2.2. RHEL 6.x Installation Click Next when the graphical installer screen appears. Chelsio T5/T4 Unified Wire for Linux Page 468...
  • Page 469 Select the FC/FCoE LUN which was saved as boot device in system BIOS and click Next. Then proceed with the installation as usual. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation. Chelsio T5/T4 Unified Wire for Linux Page 469...
  • Page 470 Chapter XXI. Unified Boot 8.2.3. SLES11 SPx Installation Choose installation language and Keyboard layout type. Select the checkbox I Agree to the License terms and click Next. Chelsio T5/T4 Unified Wire for Linux Page 470...
  • Page 471 Chapter XXI. Unified Boot ii. Select New Installation to perform a fresh installation and click Next. Chelsio T5/T4 Unified Wire for Linux Page 471...
  • Page 472 Chapter XXI. Unified Boot iii. Choose from the available base scenarios and click Next. Chelsio T5/T4 Unified Wire for Linux Page 472...
  • Page 473 Chapter XXI. Unified Boot iv. Configure Clock and Time Zone settings. Click Next. Chelsio T5/T4 Unified Wire for Linux Page 473...
  • Page 474 The discovered FC/FCoE LUNs will appear in the Preparing Hard Disk screen. Select the LUN which was saved as boot device in system BIOS. Click Next. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation. Chelsio T5/T4 Unified Wire for Linux Page 474...
  • Page 475 Overview tab and advanced settings can be changed in the Expert tab. To change, click on one of the headlines or click Change and select the category. Finally, click Next. Chelsio T5/T4 Unified Wire for Linux Page 475...
  • Page 476 Chapter XXI. Unified Boot vii. The Expert Partitioner screen displays the partition setup suggested by the installer. Click on the device selected in step (v) and click Accept. viii. Proceed with installation as usual Chelsio T5/T4 Unified Wire for Linux Page 476...
  • Page 477 Choose installation language and keyboard layout type. Select the checkbox I Agree to the License terms and click Next. ii. During system probe, the YaST installer will detect Chelsio FCoE driver csiostor present in the DUD and prompt for confirmation to load/activate. Press [OK]. Chelsio T5/T4 Unified Wire for Linux Page 477...
  • Page 478 Enter the registration or evaluation code for your copy of SLES12 and email id associated with your Suse Customer Care (SCC) account and click Next. To bypass registration, click Skip Registration and then Yes on the pop-up window that appears. Chelsio T5/T4 Unified Wire for Linux Page 478...
  • Page 479 The next screen will display a list of add-ons and extensions available for SLES12. To install, select the checkbox I would like to install an additional Add On Product, then select the radio button for the add-on/extension you wish to install, and click Next. Chelsio T5/T4 Unified Wire for Linux Page 479...
  • Page 480 On the Suggested Partitioning screen, YaST generated partition setup will be displayed. To change the suggested settings click Edit Proposal Settings. To select the disk on which to apply the proposed settings, click Create Partition Setup. To change the partition setup click Expert Partitioner. Chelsio T5/T4 Unified Wire for Linux Page 480...
  • Page 481 Click Create Partition Setup and select the LUN which was saved as boot device in system BIOS. Click Next. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation. Chelsio T5/T4 Unified Wire for Linux Page 481...
  • Page 482 To use the entire LUN for installation, click User Entire Hard Disk. Please note that this will delete all the existing partitions. To install operating system on an existing partition, select the partition from the list displayed. Click Next. Chelsio T5/T4 Unified Wire for Linux Page 482...
  • Page 483 Chapter XXI. Unified Boot viii. The selected LUN should now appear in the Suggested Partitioning screen. Click Next. ix. Proceed with installation as usual. Chelsio T5/T4 Unified Wire for Linux Page 483...

  • Page 484: Installation On Iscsi Lun

    RHEL 7.x; Click here for RHEL6.x; Click here SLES11 SPx) section to load Chelsio Driver Update Disk. No separate DUD required for SLES 12 OS installation, as inbox drivers can be Note used. Chelsio T5/T4 Unified Wire for Linux Page 484...
  • Page 485 After successfully loading Chelsio DUD, follow the procedure mentioned below to continue installation, based on the distribution. 8.3.1. RHEL 7.x On the installer welcome screen, choose your installation language and click Continue Chelsio T5/T4 Unified Wire for Linux Page 485...
  • Page 486 Chapter XXI. Unified Boot ii. Click Installation Destination under SYSTEM. Chelsio T5/T4 Unified Wire for Linux Page 486...
  • Page 487 Chapter XXI. Unified Boot iii. Click Add a disk Chelsio T5/T4 Unified Wire for Linux Page 487...
  • Page 488 The discovered iSCSI LUNs will appear in the Search tab. Select it and click Done. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation. Chelsio T5/T4 Unified Wire for Linux Page 488...
  • Page 489 Under Other Storage Options, you can either chose to configure partition automatically or manually. Select the appropriate option and click Done. Then proceed with the installation as usual. 8.3.2. RHEL 6.x Click Next when the graphical installer screen appears. Chelsio T5/T4 Unified Wire for Linux Page 489...
  • Page 490 BIOS and click Next. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation. iv. Proceed with the installation as usual. Chelsio T5/T4 Unified Wire for Linux Page 490...
  • Page 491 Chapter XXI. Unified Boot 8.3.3. SLES11 SPx installation Choose installation language and Keyboard layout type. Select the checkbox I Agree to the License terms and click Next. Chelsio T5/T4 Unified Wire for Linux Page 491...
  • Page 492 Chapter XXI. Unified Boot ii. Click Configure iSCSI Disks in the Disk Activation screen. Chelsio T5/T4 Unified Wire for Linux Page 492...
  • Page 493 Chapter XXI. Unified Boot iii. The discovered LUNs will appear in the Connected Targets tab. Select the LUN which was saved as boot device in system BIOS and click OK. Chelsio T5/T4 Unified Wire for Linux Page 493...
  • Page 494 Chapter XXI. Unified Boot iv. Select New Installation to perform a fresh installation and click Next. Chelsio T5/T4 Unified Wire for Linux Page 494...
  • Page 495 Chapter XXI. Unified Boot v. Configure Clock and Time Zone settings. Click Next. Chelsio T5/T4 Unified Wire for Linux Page 495...
  • Page 496 Chapter XXI. Unified Boot vi. Choose from the available server base scenarios and click Next. Chelsio T5/T4 Unified Wire for Linux Page 496...
  • Page 497 Overview tab and advanced settings can be changed in the Expert tab. To change, click on one of the headlines or click Change and select the category. Finally, click Next. viii. Proceed with installation as usual. Chelsio T5/T4 Unified Wire for Linux Page 497...
  • Page 498 Chapter XXI. Unified Boot 8.3.4. SLES12 Installation Choose installation language and Keyboard layout type. Select the checkbox I Agree to the License terms and click Next. Chelsio T5/T4 Unified Wire for Linux Page 498...
  • Page 499 Chapter XXI. Unified Boot ii. Click Configure iSCSI Disks in the Disk Activation screen. Chelsio T5/T4 Unified Wire for Linux Page 499...
  • Page 500 BIOS and click OK. Make sure the same LUN discovered at the Option ROM stage is selected for OS Note installation. iv. Proceed with the installation as usual. Chelsio T5/T4 Unified Wire for Linux Page 500...

  • Page 501: Lustre File System

    Chapter XXII. Lustre File System XXII. Lustre File System Chelsio T5/T4 Unified Wire for Linux Page 501...

  • Page 502: Introduction

    Metadata servers (MDS), which manage the names and directories in the file system. Systems in a Lustre cluster 1.1. Hardware Requirements 1.1.1. Supported Adapters Currently Lustre File System is supported on following Chelsio Adapters: T520-BT T580-CR T580-LP-CR T520-LL-CR Chelsio T5/T4 Unified Wire for Linux Page 502...

  • Page 503: Software Requirements

    Chelsio Unified Wire currently supports Lustre-2.6.0 1.2.2. Linux Requirements Currently Lustre File System is supported on following distributions: RHEL 6.5, 2.6.32-431.el6 (Limited QA performed) Other kernel versions have not been tested and are not guaranteed to work. Chelsio T5/T4 Unified Wire for Linux Page 503...

  • Page 504: Creating/Configuring Lustre File System

    Enable and assign IP address to Chelsio interface. Edit modprobe.conf file with appropriate interface name options lnet networks=tcp0(ethX),tcp1(ethY) //For NIC/TOE options lnet networks=o2ib0(ethX),o2ib1(ethY) //For iWARP where, ethX and ethY represent Chelsio interfaces. Chelsio T5/T4 Unified Wire for Linux Page 504...
  • Page 505 [root@host~]# mkfs.lustre --ost --fsname=<fsname> --mgsnode=<NID> <block device name> On Client node, follow steps (i)-(vii). Mount the Lustre file system on the client node by running the following command: xii. [root@host~]# mount -t lustre <MGS node>:/<fsname> <mount_point> Chelsio T5/T4 Unified Wire for Linux Page 505...

  • Page 506: Xxiii. Appendix A

    Chapter XXIII. Appendix XXIII. Appendix A Chelsio T5/T4 Unified Wire for Linux Page 506...

  • Page 507: Troubleshooting

    Always enable LLDP on the interfaces as FCoE link won’t come up until and unless a successful LLDP negotiation happens. priority-flow-control mode on the switch On the switch, make sure priority-flow-control mode is always set to auto and flow control is disabled. Chelsio T5/T4 Unified Wire for Linux Page 507...
  • Page 508 Ethernet interface part of both Ethernet VLAN and FCoE VLAN. Cisco nexus switch reporting “pauseRateLimitErrDisable” If in any case the switch-port on the Cisco nexus switch is reporting “pauseRateLimitErrDisable”, then perform an Ethernet port shut/no shut. Chelsio T5/T4 Unified Wire for Linux Page 508...

  • Page 509: Chelsio End-User License Agreement (Eula)

    TERMINATE INSTALLATION PROCESS. 1. License. Chelsio Communications, Inc. ("Chelsio") hereby grants you, the Licensee, and you hereby accept, a limited, non-exclusive, non-transferable license to install and use the Software with one or more Chelsio network adapters on a single server computer for use in communicating with one or more other computers over a network.
  • Page 510 ADP Schedule contract. If the Software is acquired under the terms of a DoD or civilian agency contract, use, duplication or disclosure by the Government is subject Chelsio T5/T4 Unified Wire for Linux Page 510...
  • Page 511 Agreement. Should you have any questions concerning this Agreement, you may contact Chelsio by writing to: Chelsio Communications, Inc. 209 North Fair Oaks Avenue, Sunnyvale, CA 94085 U.S.A Chelsio T5/T4 Unified Wire for Linux Page 511...

This manual is also suitable for:

Chelsio t4

Table of Contents