1. Manuals
  2. Brands
  3. Chelsio Communications Manuals
  4. Computer Hardware
  5. Chelsio T5
  6. Installation and user manual

Chelsio Communications Chelsio T5 Installation And User Manual

Unified wire for linux
Hide thumbs Also See for Chelsio T5:
1
2
3
Table Of Contents
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
Table of Contents

Advertisement

Quick Links

Download this manual See also: User Manual, Installer and User Manual

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Chelsio T5 and is the answer not in the manual?

Questions and answers

Related Manuals for Chelsio Communications Chelsio T5

Summary of Contents for Chelsio Communications Chelsio T5

  • Page 2 For all support related questions please send email to support@chelsio.com Copyright © 2015.Chelsio Communications. All Rights Reserved. Chelsio ® is a registered trademark of Chelsio Communications. All other marks and names mentioned herein may be trademarks of their respective companies. Chelsio T5/T4 Unified Wire For Linux...

  • Page 3: Document History

    1.0.3 03/26/2013 1.0.4 04/12/2013 1.0.5 06/20/2013 1.0.6 08/17/2013 1.0.7 10/22/2013 1.0.8 03/08/2013 1.0.9 05/15/2013 1.1.0 07/26/2013 1.1.1 08/14/2013 1.1.2 12/06/2013 1.1.3 12/19/2013 1.1.4 03/13/2014 1.1.5 05/02/2014 1.1.6 06/30/2014 1.1.7 10/22/2014 1.1.8 11/04/2014 1.1.9 02/05/2015 Chelsio T5/T4 Unified Wire For Linux...

  • Page 4: Table Of Contents

    Loading in TOE mode (with full offload support) Software/Driver Unloading 3.1. Unloading the NIC driver 3.2. Unloading the TOE driver Software/Driver Configuration and Fine-tuning 4.1. Instantiate Virtual Functions (SR-IOV) 4.2. Performance tuning III. VIRTUAL FUNCTION NETWORK (VNIC) Introduction 1.1. Hardware Requirements Chelsio T5/T4 Unified Wire For Linux...
  • Page 5 Software/Driver Configuration and Fine-tuning 4.1. Accelerating UDP Socket communications WD-TOE Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Running the application VII. ISCSI PDU OFFLOAD TARGET Introduction Chelsio T5/T4 Unified Wire For Linux...
  • Page 6 Auto login from cxgb4i initiator at OS bootup DATA CENTER BRIDGING (DCB) Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Configuring Cisco Nexus 5010 switch 4.2. Configuring the Brocade 8000 switch Chelsio T5/T4 Unified Wire For Linux...
  • Page 7 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Offloading TCP traffic over a bonded interface XIII. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) Introduction 1.1. Hardware Requirements 1.2. Software Requirements Chelsio T5/T4 Unified Wire For Linux...
  • Page 8 Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Starting ba server 4.2. Bypass API (CLI) XVII. WD SNIFFING AND TRACING Theory of Operation 1.1. Hardware Requirements 1.2. Software Requirements Installation and Usage Chelsio T5/T4 Unified Wire For Linux viii...
  • Page 9 Traffic Management Rules 4.2. Configuring Traffic Management Usage 5.1. Non-Offloaded Connections 5.2. Offloaded Connections 5.3. Offloaded Connections with Modified Application UNIFIED WIRE MANAGER (UM) Introduction 1.1. Features 1.2. Reference Architecture 1.3. Unified Wire Manager Components Chelsio T5/T4 Unified Wire For Linux...
  • Page 10 Uninstalling Management Client 8.3. Uninstalling Management Station XXI. UNIFIED BOOT Introduction 1.1. Hardware Requirements 1.2. Software Requirements Flashing firmware and option ROM 2.1. Using Flash Utility 2.2. Using cxgbtool Configuring PXE Server PXE boot process Chelsio T5/T4 Unified Wire For Linux...
  • Page 11 8.2. Installation on FCoE LUN 8.3. Installation on iSCSI LUN XXII. LUSTRE FILE SYSTEM Introduction 1.1. Hardware Requirements 1.2. Software Requirements Creating/Configuring Lustre File System XXIII. APPENDIX A Troubleshooting Chelsio End-User License Agreement (EULA) Chelsio T5/T4 Unified Wire For Linux...

  • Page 12: Chelsio Unified Wire

    Chapter I. Chelsio Unified Wire I. Chelsio Unified Wire Chelsio T5/T4 Unified Wire For Linux Page 12...

  • Page 13: Introduction

    Chapter I. Chelsio Unified Wire 1. Introduction Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip, single firmware cards provide enterprises and data centers with high performance solutions for various Network and Storage related requirements.

  • Page 14: Hardware Requirements

    1.2. Hardware Requirements The Chelsio Unified Wire software supports Chelsio T5 and T4 Series of Unified Wire Adapters. To know more about the list of adapters supported by each driver, please refer to their respective sections. 1.3.
  • Page 15 Management Station configuration files.  chsetup: The chsetup tool loads NIC,TOE and iWARP drivers, and creates WD- UDP configuration file.  chstatus: This utility provides status information on any Chelsio NIC in the system. Chelsio T5/T4 Unified Wire For Linux Page 15...
  • Page 16 The OptionROM directory contains Unified Boot Option ROM image (cubt4.bin) and a Legacy Flash Utility (cfut4.exe), which can be used to flash Unified Boot Option ROM onto Chelsio's T5 and T4 Converged Network Adapters (CNAs). The LinuxDUD directory contains image (.img) files required to update drivers for Linux (RHEL and SLES) distributions.

  • Page 17: Hardware Installation

    13. Verify if the adapter was installed successfully by using the lspci command a. For T5 adapters : [root@host]# lspci |grep -i Chelsio 07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified...
  • Page 18 Storage Controller 03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000 For Chelsio T5/T4 adapters, the physical functions are currently assigned as: Physical functions 0 - 3: for the SR-IOV functions of the adapter Physical function 4: for all NIC functions of the adapter...
  • Page 19 Note following convention: the port farthest from the motherboard will appear as the first network interface. However, for T5 40G and T420-BT adapters, the association of physical Ethernet ports and their corresponding network device names is opposite. For these adapters, the port nearest to the motherboard will appear as the first network interface.

  • Page 20: Software/Driver Installation

    UDP Seg.Offload NIC/TOE, IPv6, USO, number of UDP Segmentation Offload & Pacing UM(Agent,Client,WebGUI), Filtering, TM connections. T5 Wire Direct Configures T5 adapters to provide low Wire NIC/TOE,iWARP,WD-UDP,WD-TOE, Latency* Direct latency. UM(Agent,Client,WebGUI) Configures adapters to establish a large NIC/TOE,WD-UDP,WD-TOE, High Capacity WD number of WD-UDP connections.

  • Page 21: Pre-Requisites

    GUI installer: [root@host]# ./install.py iv. If Dialog utility is present, you can skip to step (v). If not, press ‘y’ to install it when the installer prompts for input. Chelsio T5/T4 Unified Wire For Linux Page 21...
  • Page 22 Select Enable IPv6-Offload to install drivers with IPv6 Offload support or Disable IPv6- offload to continue installation without IPv6 offload support. vii. Select the required T5/T4 configuration tuning option: The tuning options may vary depending on the Linux distribution.
  • Page 23 Retain IRQ balance daemon: Do not disable IRQ balance daemon. c. TX-Coalasce: Write tx_coal=2 to modprobe.d/conf. For more information on the Performance tuning options, please refer to Note Software/Driver Configuration and Fine-tuning section of the Network (NIC/TOE) chapter. Chelsio T5/T4 Unified Wire For Linux Page 23...
  • Page 24 To install OFED-3.12 choose the Install-OFED option. To skip this step, select Skip-OFED. This step will be prompted only for OFED supported platforms. Note xi. The selected components will now be installed: xii. After successful installation, summary of installed components will be displayed. Chelsio T5/T4 Unified Wire For Linux Page 24...
  • Page 25 Change your current working directory to Chelsio Unified Wire package directory and run the following script to start the GUI installer: [root@host]# ./install.py ii. Select “Yes” to continue with the installation on the updated kernel or “No” to exit. Chelsio T5/T4 Unified Wire For Linux Page 25...
  • Page 26 If you already have the required version of OFED software installed, you can skip this step. To install OFED-3.12 choose the Install-OFED option.To skip this step, select Skip-OFED. This step will be prompted only for OFED supported platforms. Note Chelsio T5/T4 Unified Wire For Linux Page 26...
  • Page 27 Now, execute the following command: [root@host] # ./install.py -C -m <machinefilename> iii. Select the required T5/T4 configuration tuning option. The tuning options may vary depending on the Linux distribution. iv. Select the required Cluster Configuration. v. If you already have the required version of OFED software installed, you can skip this step.
  • Page 28 CLI mode (individual drivers) for instructions on installing them. To view the different configuration tuning options, view help by typing Note [root@host]#make help vi. Reboot your machine for changes to take effect. Chelsio T5/T4 Unified Wire For Linux Page 28...
  • Page 29 To build and install NIC driver without offload support : [root@host]# make nic [root@host]# make nic_install  To build and install NIC driver with offload support and Offload drivers: [root@host]# make toe [root@host]# make toe_install Chelsio T5/T4 Unified Wire For Linux Page 29...
  • Page 30 To build and install WD-TOE and WD-UDP drivers together: [root@host]# make wdtoe_wdudp [root@host]# make wdtoe_wdudp_install  To build and install all drivers with DCB support: [root@host]# make dcbx=1 [root@host]# make dcbx=1 install Chelsio T5/T4 Unified Wire For Linux Page 30...
  • Page 31 To build and install FCoE Target drivers: [root@host]# make fcoe_pdu_offload_target [root@host]# make fcoe_pdu_offload_target_install  The default T5/T4 configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host]# make CONF=<T5/T4 configuration> <Build Target> [root@host]# make CONF=<T5/T4 configuration> <Install Target>...

  • Page 32: Installing Chelsio Unified Wire From Rpm

    1. Unified Wire: Install all the drivers in the Unified Wire software package. This option will not install OFED and drivers built against OFED. 2. T5 Wire Direct Latency: Install Wire Direct Latency drivers needed for Low latency applications. 3. Custom: Customize the installation. Use this option to install drivers/software and related components (like OFED-3.12) according to the tunining option selected.
  • Page 33 To install OFED and Chelsio drivers built against OFED, run the above command with -o option. [root@host]# ./install.py –i <Installation mode> -c <Configuration> -o vi. Reboot your machine for changes to take effect. Chelsio T5/T4 Unified Wire For Linux Page 33...

  • Page 34: Firmware Update

    Reboot your machine for changes to take effect. 3.4. Firmware update The T5 and T4 firmwares are installed on the system, typically in /lib/firmware/cxgb4 , and the driver will auto-load the firmwares if an update is required. The kernel must be configured to enable userspace firmware loading support: Device Drivers ->...

  • Page 35: Software/Driver Uninstallation

    GUI installer: [root@host]# ./install.py ii. Select “uninstall” , Under “Choose an action” iii. Select “all” to uninstall all the installed drivers, libraries and tools or select “custom” to remove specific components. Chelsio T5/T4 Unified Wire For Linux Page 35...
  • Page 36 The selected components will now be uninstalled. v. After successful uninstalltion, summary of the uninstalled components will be displayed. vi. Select “View log” to view uninstallation log or “Exit” to continue. Chelsio T5/T4 Unified Wire For Linux Page 36...
  • Page 37 4.1.3.1. iWARP driver uninstallation on Cluster nodes To uninstal iWARP drivers on multiple Cluster nodes with a single command, run the following command: [root@host]# ./install.py -C -m <machinefilename> -u all Chelsio T5/T4 Unified Wire For Linux Page 37...
  • Page 38 To uninstall iWARP driver: [root@host]# make iwarp_uninstall  To uninstall Bypass driver: [root@host]# make bypass_uninstall  To uninstall UDP Segmentation Offload driver: [root@host]# make udp_offload_uninstall  To uninstall WD-TOE driver: [root@host]# make wdtoe_uninstall Chelsio T5/T4 Unified Wire For Linux Page 38...

  • Page 39: Uninstalling Chelsio Unified Wire From Rpm

    [root@host]# ./uninstall.py –h by typing for more information. Uninstalling Unified Wire package will not uninstall Unified Wire Manager. Refer Note to the Unified Wire Manager (UM) chapter to remove the software manually (Click here). Chelsio T5/T4 Unified Wire For Linux Page 39...
  • Page 40 To uninstal iWARP drivers on multiple Cluster nodes with a single command, run the following: [root@host] # ./install.py -C -m <machinefilename> -u The above command will remove Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. Chelsio T5/T4 Unified Wire For Linux Page 40...

  • Page 41: Configuring Chelsio Network Interfaces

    In order to test Chelsio adapters’ features it is required to use two machines both with Chelsio’s (T5, T4 or both) network adapters installed. These two machines can be connected directly without a switch (back-to-back), or both connected to a switch. The interfaces have to be declared and configured.
  • Page 42 Select mode for adapter (1,2,3): Reload the network driver for changes to take effect. [root@host]# rmmod cxgb4 [root@host]# modprobe cxgb4 In case of T580-SO-CR adapters, reboot the machine for changes to take effect. Note Chelsio T5/T4 Unified Wire For Linux Page 42...

  • Page 43: Configuring Network-Scripts

    (for the moment it does not make any difference whether we are using NIC-only or the TOE-enabling driver). The output of ifconfig should display the T5/T4 interfaces as: [root@host]# ifconfig -a | grep HWaddr eth0 Link encap:Ethernet HWaddr 00:30:48:32:6A:AA...

  • Page 44: Checking Link

    First, bring up the interface you want to test using ifup eth1. You should now be able to ping any other machine from your network provided it has ping response enabled. Chelsio T5/T4 Unified Wire For Linux Page 44...

  • Page 45: Software/Driver Update

    For any distribution specific problems, please check README and Release Notes included in the release for possible workaround. Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various software/drivers. You can also subscribe to our newsletter for the latest software updates. Chelsio T5/T4 Unified Wire For Linux Page 45...

  • Page 46: Network (Nic/Toe)

    Chapter II. Network (NIC/TOE) II. Network (NIC/TOE) Chelsio T5/T4 Unified Wire For Linux Page 46...

  • Page 47: Introduction

    Chapter II. Network (NIC/TOE) 1. Introduction Chelsio’s T5 and T4 series of Unified Wire Adapters provide extensive support for NIC operation, including all stateless offload mechanisms for both IPv4 and IPv6 (IP, TCP and UDP checksum offload, LSO - Large Send Offload aka TSO - TCP Segmentation Offload, and assist mechanisms for accelerating LRO - Large Receive Offload).

  • Page 48: Software Requirements

    Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 48...

  • Page 49: Software/Driver Loading

    To enable full offload support, run the following command: [root@host]# modprobe t4_tom Offload support needs to be enabled upon each reboot of the system. This can Note be done manually as shown above. Chelsio T5/T4 Unified Wire For Linux Page 49...

  • Page 50: Software/Driver Unloading

    3. Software/Driver Unloading 3.1. Unloading the NIC driver To unload the NIC driver, run the following command: [root@host]# rmmod cxgb4 3.2. Unloading the TOE driver Please reboot the system to unload the TOE driver. Chelsio T5/T4 Unified Wire For Linux Page 50...

  • Page 51: Software/Driver Configuration And Fine-Tuning

    [root@host]# t4_perftune.sh Receiver Side Scaling (RSS) Receiver Side Scaling enables the receiving network traffic to scale with the available number of processors on a modern networked computer. RSS enables parallel receive processing and Chelsio T5/T4 Unified Wire For Linux Page 51...
  • Page 52 Writing 8 in /proc/irq/65/smp_affinity eth7 now up and tuned Because there are 4 CPUs on the system, 4 entries of interrupts are assigned. For other T5/T4 network interfaces, you should see similar output message. Now the receiving traffic is dynamically assigned to one of the system’s CPUs through a T5/T4 queue.
  • Page 53 (queue 3) Now interrupts from eth6 are evenly distributed among the 4 CPUs. Without T5/T4’s RSS support, the interrupts caused by network traffic may be distributed unevenly over CPUs. For your information, the traffic produced by the same iperf commands...
  • Page 54 -w net.core.wmem_default=16777216 sysctl -w net.core.optmem_max=16777216 sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216' sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216' iii. Bring up the network interfaces and run the following command: [root@host]# ethtool -C ethXX adaptive-rx on Chelsio T5/T4 Unified Wire For Linux Page 54...
  • Page 55 Please note that if your Linux system has IP forwarding enabled, i.e. acting as a bridge or router, the LRO needs to be disabled. This is due to a known kernel issue. Chelsio’s T5/T4 card supports both hardware assisted GRO/LRO and Linux-based GRO/LRO. t4_tom is the kernel module that enables the hardware assisted GRO/LRO.
  • Page 56 [root@host]# lsmod | grep t4_tom t4_tom 88378 0 [permanent] toecore 21618 1 t4_tom cxgb4 225342 1 t4_tom Then T5/T4’s hardware GRO/LRO implementation is enabled. t4_tom If you would like to use the Linux GRO/LRO for any reason, first the kernel module needs to be removed from kernel module list.
  • Page 57 Chapter II. Network (NIC/TOE) When Linux’s GRO is enabled, Chelsio’s T5/T4 driver provides two GRO-related statistics. They are displayed using the following command: [root@host]# ethtool -S eth6 GROPackets : 0 GROMerged : 897723 GROPackets is the number of held packets. Those are candidate packets held by the kernel to be processed individually or to be merged to larger packets.

  • Page 58: Virtual Function Network (Vnic)

    Chapter III. Virtual Function Network (vNIC) III. Virtual Function Network (vNIC) Chelsio T5/T4 Unified Wire For Linux Page 58...

  • Page 59: Introduction

    Other benefits of Virtualization include improved disaster recovery, server portability, cloud computing, Virtual Desktop Infrastructure (VDI), etc. Chelsio’s T5 and T4 Unified Wire family of Adapters deliver increased bandwidth, lower latency and lower power with virtualization features to maximize cloud scaling and utilization. The adapters support SFP+ and 10BASE-T media.

  • Page 60: Software Requirements

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 60...

  • Page 61: Software/Driver Loading

    The vNIC driver must be loaded or unloaded on the Guest OS by the root user. Any attempt to load the driver as a regular user will fail. 2.1. Loading the driver To load the driver, run the following command: [root@host]# modprobe cxgb4vf Chelsio T5/T4 Unified Wire For Linux Page 61...

  • Page 62: Software/Driver Unloading

    Chapter III. Virtual Function Network (vNIC) 3. Software/Driver Unloading 3.1. Unloading the driver To unload the driver, execute the following command: [root@host]# rmmod cxgb4vf Chelsio T5/T4 Unified Wire For Linux Page 62...

  • Page 63: Software/Driver Configuration And Fine-Tuning

    Chapter III. Virtual Function Network (vNIC) 4. Software/Driver Configuration and Fine-tuning 4.1. Instantiate Virtual Functions To instantiate Chelsio Virtual Functions, please refer to the Network (NIC/TOE) section (click here) Chelsio T5/T4 Unified Wire For Linux Page 63...

  • Page 64: Iwarp (Rdma)

    Chapter IV. iWARP (RDMA) IV. iWARP (RDMA) Chelsio T5/T4 Unified Wire For Linux Page 64...

  • Page 65: Introduction

    The iWARP RDMA operation benefits from the virtualization, traffic management and QoS mechanisms provided by T5/T4 engine. It is possible to ACL process iWARP RDMA packets. It is also possible to rate control the iWARP traffic on a per-connection or per-class basis, and to give higher priority to QPs that implement distributed locking mechanisms.
  • Page 66 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 66...

  • Page 67: Software/Driver Loading

    [root@host]# make install  To load the iWARP driver we need to load the NIC driver and core RDMA drivers first. Run the following commands: [root@host]# modprobe cxgb4 [root@host]# modprobe iw_cxgb4 [root@host]# modprobe rdma_ucm Chelsio T5/T4 Unified Wire For Linux Page 67...

  • Page 68: Software/Driver Unloading

    Chapter IV. iWARP (RDMA) 3. Software/Driver Unloading To unload the iWARP driver, run the following command: [root@host]# rmmod iw_cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 68...

  • Page 69: Software/Driver Configuration And Fine-Tuning

    Load the NIC, iWARP & core RDMA modules as mentioned in Software/Driver Loading section. After which, you will see two or four ethernet interfaces for the T5/T4 device. Configure them with an appropriate ip address, netmask, etc. You can use the Linux ping command to test basic connectivity via the T5/T4 interface.

  • Page 70: Enabling Various Mpis

    Create authorization file [root@host]# cd .ssh [root@host]# cat *.pub > authorized_keys [root@host]# chmod 600 authorized_keys v. Copy directory .ssh to all systems in the cluster [root@host]# cd [root@host]# scp -r /root/.ssh remotehostname-or-ipaddress: Chelsio T5/T4 Unified Wire For Linux Page 70...
  • Page 71 Contact Intel for obtaining their MPI with DAPL support. xi. To run Intel MPI applications: mpdboot -n <no_of_nodes_in_cluster> -r ssh mpdtrace mpiexec -ppn -n 2 /opt/intel/impi/3.1/tests/IMB-3.1/IMB-MPI1 The performance is best with NIC MTU set to 9000 bytes. Chelsio T5/T4 Unified Wire For Linux Page 71...
  • Page 72 ("" == "`echo $path | grep /usr/mpi/gcc/openmpi-x.y.z/bin`") then set path=(/usr/mpi/gcc/openmpi-x.y.z/bin $path) endif # LD_LIBRARY_PATH if ("1" == "$?LD_LIBRARY_PATH") then if ("$LD_LIBRARY_PATH" !~ */usr/mpi/gcc/openmpi-x.y.z/lib64*) then setenv LD_LIBRARY_PATH /usr/mpi/gcc/openmpi- x.y.z/lib64:${LD_LIBRARY_PATH} endif else setenv LD_LIBRARY_PATH /usr/mpi/gcc/openmpi-x.y.z/lib64 endif # MPI_ROOT setenv MPI_ROOT /usr/mpi/gcc/openmpi-x.y.z Chelsio T5/T4 Unified Wire For Linux Page 72...
  • Page 73 Register OpenMPi with MPI-selector: [root@host]# mpi-selector --register openmpi --source-dir /usr/mpi/gcc/openmpi-x.y.z/bin ix. Verify if it is listed in mpi-selector: [root@host]# mpi-selector --l x. Set OpenMPI: [root@host]# mpi-selector --set openmpi –yes xi. Logut and log back in. Chelsio T5/T4 Unified Wire For Linux Page 73...
  • Page 74 ("" == "`echo $path | grep /usr/mpi/gcc/mvapich2-x.y/bin`") then set path=(/usr/mpi/gcc/mvapich2-x.y/bin $path) endif # LD_LIBRARY_PATH if ("1" == "$?LD_LIBRARY_PATH") then if ("$LD_LIBRARY_PATH" !~ */usr/mpi/gcc/mvapich2-x.y/lib64*) then setenv LD_LIBRARY_PATH /usr/mpi/gcc/mvapich2- x.y/lib64:${LD_LIBRARY_PATH} endif else setenv LD_LIBRARY_PATH /usr/mpi/gcc/mvapich2-x.y/lib64 endif # MPI_ROOT setenv MPI_ROOT /usr/mpi/gcc/mvapich2-x.y Chelsio T5/T4 Unified Wire For Linux Page 74...
  • Page 75 Add the following entries in .bashrc file: export MVAPICH2_HOME=/usr/mpi/gcc/mvapich2-x.y/ export MV2_USE_IWARP_MODE=1 export MV2_USE_RDMA_CM=1 viii. Register MPI: [root@host]# mpi-selector --register mvapich2 --source-dir /usr/mpi/gcc/mvapich2-x.y/bin/ xii. Verify if it is listed in mpi-selector: [root@host]# mpi-selector --l Chelsio T5/T4 Unified Wire For Linux Page 75...
  • Page 76 On each node, create /etc/mv2.conf with a single line containing the IP address of the local T4/T5 interface. This is how MVAPICH2 picks which interface to use for RDMA traffic. 4.2.4. Building MPI tests Download Intel’s MPI Benchmarks from http://software.intel.com/en-us/articles/intel-mpi-...
  • Page 77 Workaround: Increase the OpenMPI rdma route resolution timeout. The default is 1000, or 1000ms. Increase it to 30000 with this parameter: --mca btl_openib_connect_rdmacm_resolve_timeout 30000  Run MVAPICH2 application as : mpirun_rsh -ssh -np 8 -hostfile mpd.hosts $MVAPICH2_HOME/tests/imb/IMB-MPI1 Chelsio T5/T4 Unified Wire For Linux Page 77...

  • Page 78: Setting Up Nfs-Rdma

    Include the rdma port 2050 into this file as: [root@host]# echo rdma 2050 > /proc/fs/nfsd/portlist vii. Run exportfs to make local directories available for Network File System (NFS) clients to mount. [root@host]# exportfs Now the NFS-RDMA server is ready. Chelsio T5/T4 Unified Wire For Linux Page 78...

  • Page 79: Performance Tuning

    In order to maximize your system’s performance, Chelsio recommends installing the tools which will copy t4_perftune.sh script to /sbin directory. Also, disable virtualization, c-state technology, VT-d, Intel I/O AT and SR-IOV in the BIOS settings and then run the following command: [root@host]# t4_perftune.sh Chelsio T5/T4 Unified Wire For Linux Page 79...

  • Page 80: Wd-Udp

    Chapter V. WD-UDP V. WD-UDP Chelsio T5/T4 Unified Wire For Linux Page 80...

  • Page 81: Introduction

    It features software modules that enable direct wire access from user space to the Chelsio T5/T4 network adapter with complete bypass of the kernel, which results in an ultra-low latency 40/10Gb Ethernet solution for high frequency trading and other delay-sensitive applications.
  • Page 82 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 82...

  • Page 83: Software/Driver Loading

    RDMA core modules from the OFED package should be loaded before proceeding. To load the WD-UDP driver, use the following commands which will automatically load RDMA core modules: [root@host]# modprobe cxgb4 [root@host]# modprobe iw_cxgb4 [root@host]# modprobe rdma_ucm Chelsio T5/T4 Unified Wire For Linux Page 83...

  • Page 84: Software/Driver Unloading

    Chapter V. WD-UDP 3. Software/Driver Unloading To unload the WD-UDP driver, run the following command: [root@host]# rmmod iw_cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 84...

  • Page 85: Software/Driver Configuration And Fine-Tuning

    Unicast and Multicast datagrams can be accelerated, but the datagrams must meet the following criteria: Non fragmented. In other words, they fit in a single IP datagram that is <= the T5/T4 device MTU. Routed through the T5/T4 acceleration device. If the ingress datagram arrives via a device other than the T5/T4 acceleration device, then it will not utilize the acceleration path.
  • Page 86 Chapter V. WD-UDP egress, if the destination IP address will not route out via the T5/T4 device, then it too will not be accelerated. 4.1.2. Using libcxgb4_sock The libcxgb4_sock library utilizes the Linux RDMA Verbs subsystem, and thus requires the RDMA modules be loaded.
  • Page 87 # endpoints that bind to port 0 (requesting the host allocate a port) # can be accelerated with port=0: # endpoint {interface=eth1 port=0} Assume your T5/T4 interface is eth2. To accelerate all applications that preload libcxgb4_sock using eth2, you only need one entry in /etc/libcxgb4_sock.conf...
  • Page 88 The udp benchmark from the hpcbench suite can be used to show the benefits of libcxgb4_sock. The hpcbench suite can be found at: Source: http://hpcbench.sourceforge.net/index.html Sample: http://hpcbench.sourceforge.net/udp.html The nodes in this example, r9 and r10, have T5/T4 eth1 configured and the ports are connected point-to-point. Chelsio T5/T4 Unified Wire For Linux Page 88...
  • Page 89 Then on r9, we run udptest to port 9000 to see the host stack UDP latency: [root@r9 ~]# /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9000 Running the same test with libcxgb4_sock: [root@r9 ~]# LD_PRELOAD=libcxgb4_sock.so /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9001 Chelsio T5/T4 Unified Wire For Linux Page 89...
  • Page 90 To see if the application is being offloaded, open a window on one of the machines, and run tcpdump against the Chelsio interface. If you see minimal UDP output on the interface, then the UDP traffic is being properly offloaded. Chelsio T5/T4 Unified Wire For Linux Page 90...

  • Page 91: Wd-Toe

    Chapter VI. WD-TOE VI. WD-TOE Chelsio T5/T4 Unified Wire For Linux Page 91...

  • Page 92: Introduction

    It features software modules that enable direct wire access from user space to the Chelsio T5/T4 network adapter with complete bypass of the kernel, which results in a low latency 10Gb Ethernet solution for high frequency trading and other delay-sensitive applications.
  • Page 93 Chapter VI. WD-TOE Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6 Kernel.org linux-3.7 Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 93...

  • Page 94: Software/Driver Loading

    Before proceeding, please ensure that the offload drivers are installed with WD-TOE support. They are installed by default with Low Latency Networking, T5 Wire Direct Latency or High Capacity WD configuration tuning Options. With any other configuration tuning option, the installation needs to be customized.

  • Page 95: Software/Driver Unloading

    Chapter VI. WD-TOE 3. Software/Driver Unloading Reboot the system to unload the driver. Chelsio T5/T4 Unified Wire For Linux Page 95...

  • Page 96: Software/Driver Configuration And Fine-Tuning

    Start netserver at the PEER, using the following command: [root@host]# PROT=TCP wdload netserver –D -4 ii. On the Test Machine, run the following command to run netperf application. [root@host]# PROT=TCP wdload netperf -H <PEER_IP> -t TCP_RR -l 10 Chelsio T5/T4 Unified Wire For Linux Page 96...

  • Page 97: Iscsi Pdu Offload Target

    Chapter VII. iSCSI PDU Offload Target VII. iSCSI PDU Offload Target Chelsio T5/T4 Unified Wire For Linux Page 97...

  • Page 98: Introduction

    The Chelsio iSCSI PDU Offload Target software provides the following high level features: Expanded NIC Support  Chelsio TCP Offload Engine (TOE) Support  T5/T4 Based HBAs (T5/T4xx Series cards) Chelsio Terminator ASIC Support  Offloads iSCSI Fast Data Path with Direct Data Placement (DDP) ...

  • Page 99: Hardware Requirements

    1.2.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with iSCSI PDU Offload Target software: T502-BT T580-CR T520-LL-CR T520-CR T522-CR T580-LP-CR T540-CR T420-CR T440-CR T422-CR T404-BT T420-BCH Chelsio T5/T4 Unified Wire For Linux Page 99...
  • Page 100 For deployment in a production environment a more sophisticated system would be needed. That typically consists of a system with one or more storage controllers with multiple disk drives attached running software or hardware based RAID. Chelsio T5/T4 Unified Wire For Linux Page 100...

  • Page 101: Software Requirements

    RHEL5 as an example, or a complete tree. The source tree needs to be configured and the header files need to be compiled. Additionally, the Linux kernel must be configured to use modules. Chelsio T5/T4 Unified Wire For Linux Page 101...
  • Page 102 The key file will be sent back and must be installed on the system in order to unlock the software. Details on this mechanism can be found in the next section on software installation. Chelsio T5/T4 Unified Wire For Linux Page 102...

  • Page 103: Software/Driver Loading

    It also provides control for iSCSI global settings. 1. Loading the Kernel module For RHEL distributions, run modprobe as follows: [root@host]# modprobe chiscsi_t4 For SLES11 spX, run modprobe as follows: [root@host]# modprobe chiscsi_t4 --allow-unsupported Chelsio T5/T4 Unified Wire For Linux Page 103...
  • Page 104 It can also be used to stop the targets, and restart/reload configuration. For the script to execute properly, make sure the following flag is set on all Note kernel.org kernels. # CONFIG_MODULE_FORCE_LOAD=y Chelsio T5/T4 Unified Wire For Linux Page 104...

  • Page 105: Generating Single Rpm For T3 And T4 Adapters

    7. Next, build the iSCSI PDU offload target driver (chiscsi_t3) for T3 adapters using the Chelsio Unified Wire source package: [root@host]# cd ChelsioUwire-x.xx.x.x/build/src/chiscsi [root@host]# make CXGB3TOE_SRC=<path to T3 NIC package>/src 8. Now, generate the RPM: [root@host]# make iscsi_pdu_target_rpm Chelsio T5/T4 Unified Wire For Linux Page 105...

  • Page 106: Obtaining The Iscsi Software License

    PCI vendor ID MAC address Link speed Privacy notice: The above information is used by Chelsio Communications strictly for iSCSI license management. Chelsio Communications, Inc. does not share, sell, rent, or lease the information to any third party. Chelsio T5/T4 Unified Wire For Linux...
  • Page 107 ******************************************************* SUMMARY: Interface with MAC Address 00:07:43:00:00:10 (10000 Mbps) is selected Please send info file "/etc/xyz_chelsio_infofile" to Chelsio Communications at support@chelsio.com Chelsio Support will send back the keyfile. The keyfile unlocks the software as long as the above interface is present...

  • Page 108: Software/Driver Unloading

    Chapter VII. iSCSI PDU Offload Target 3. Software/Driver Unloading Use the following command to unload the module: [root@host]# rmmod chiscsi_t4 Chelsio T5/T4 Unified Wire For Linux Page 108...

  • Page 109: Software/Driver Configuration And Fine-Tuning

    There are only a few global configuration items that can be changed. An example is “iscsi_offload_mode” which specifies the hardware protocol offload method that should be used (e.g. “TOE”, “ULP”, or “AUTO” modes) for all iSCSI targets. Chelsio T5/T4 Unified Wire For Linux Page 109...

  • Page 110: A Quick Start Guide For Target

    TargetDevice – A device served up by the associated target. A device can be:  A block device (for example, /dev/sda)  A virtual block device (for example, /dev/md0)  A RAM disk  A regular file Chelsio T5/T4 Unified Wire For Linux Page 110...
  • Page 111 To start a specific target execute [root@host]# iscsictl –f /etc/chelsio-iscsi/chiscsi.conf –S target=iqn.2006-02.com.chelsio.diskarray.san1 with “ ” option followed by iscsictl Stop Target: To stop the all the iSCSI target(s), execute “ ”. target=ALL [root@host]# iscsictl –s target=ALL Chelsio T5/T4 Unified Wire For Linux Page 111...
  • Page 112 ” iscsictl Change Global Settings: To change Chelsio global settings, execute option. [root@host]# iscsictl –G iscsi_offload_mode=AUTO with “ ” option. iscsictl View Help: To print help to stdout, execute [root@host]# iscsictl –h Chelsio T5/T4 Unified Wire For Linux Page 112...

  • Page 113: The Iscsi Configuration File

    Chelsio System Wide Global Entity Parameters pass system control information to the iSCSI software which affects all targets in the same way. More detail of these parameters below can be found in a later section entitled “System Wide Parameters”. Chelsio T5/T4 Unified Wire For Linux Page 113...
  • Page 114 Entity Parameters pass iSCSI protocol control information to the Chelsio iSCSI module. This information is unique for each entity block. The parameters follow the IETF iSCSI standard RFC 3720 in both definition and syntax. The descriptions below are mostly from this RFC. Chelsio T5/T4 Unified Wire For Linux Page 114...
  • Page 115 - 1) iSCSI PDU. OFMarker “Yes” “No” To turn on or off the initiator to target “No” markers on the connection. Chelsio only supports “No”. Chelsio T5/T4 Unified Wire For Linux Page 115...
  • Page 116 NOTE: The double quotes are required as part of the format. Auth_CHAP_Initiator “<user id>” CHAP user id and secret for the :“<secret>” initiator. <user id> must be less than 256 characters. Commas “,” Chelsio T5/T4 Unified Wire For Linux Page 116...
  • Page 117 ShadowMode “Yes” “No” To turn ShadowMode on or off for “No” iSCSI Target Redirection Chelsio T5/T4 Unified Wire For Linux Page 117...
  • Page 118 16MB (16 Megabytes). The minimum value of x is 1 (1MB) and the maximum value is limited by system memory. ScsiID=xxxxxx is a 16 character unique value set for multipath aware iSCSI initiator host. Chelsio T5/T4 Unified Wire For Linux Page 118...
  • Page 119 IP addresses that the iSCSI target is listening on (i.e., the target portal IP addresses). NOTE: when configuring an ACL at least one of the above three must be provided:  iname, and/or Chelsio T5/T4 Unified Wire For Linux Page 119...
  • Page 120 # Chelsio iSCSI Global Settings global: iscsi_offload_mode=AUTO iscsi_auth_order=ACL # an iSCSI Target “iqn.2006-02.com.chelsio.diskarray.san1” # being served by the portal group "5". Setup as a RAM Disk. target: TargetName=iqn.2006-02.com.chelsio.diskarray.san1 Chelsio T5/T4 Unified Wire For Linux Page 120...
  • Page 121 # Local block devices being served up # lun 0 is pointed to /dev/sda # lun 1 is pointed to /dev/sdb TargetDevice=/dev/sda,ScsiID=aabbccddeeffgghh,WWN=aaabbbcccdddeeef TargetDevice=/dev/sdb # Portal groups served this target PortalGroup=1@102.50.50.25:3260 PortalGroup=2@102.60.60.25:3260 # CHAP configuration Chelsio T5/T4 Unified Wire For Linux Page 121...

  • Page 122: Challenge-Handshake Authenticate Protocol (Chap)

    4.5.1. Oneway CHAP authentication With one-way CHAP (also called unidirectional CHAP) the target uses CHAP to authenticate the initiator. The initiator does not authenticate the target. This method is the default method. Chelsio T5/T4 Unified Wire For Linux Page 122...
  • Page 123 CHAP is used. The default setting of AuthMethod=None,CHAP On an iSCSI node, with Auth_CHAP_Policy=Oneway , Chelsio iSCSI target will accept a relevant initiator if it does a) no CHAP b) CHAP Oneway or c) CHAP Mutual Chelsio T5/T4 Unified Wire For Linux Page 123...

  • Page 124: Target Access Control List (Acl) Configuration

    The format for ACL rule is as follows: ACL=[iname=<initiator name>][;<sip=<source ip addresses>] [;dip=<destination ip addresses>][;lun=<lun_list>:<permissions>] target: TargetName=iqn.2006-02.com.chelsio.diskarray.san1 TargetDevice=/dev/sda PortalGroup=1@102.50.50.25:3260 PortalGroup=2@102.60.60.25:3260 # initiator “iqn.2006-02.com.chelsio.san1” is allowed # full read-write access to this target Chelsio T5/T4 Unified Wire For Linux Page 124...

  • Page 125: Target Storage Device Configuration

    This option is usefuly for measuring network performance. Chelsio T5/T4 Unified Wire For Linux Page 125...
  • Page 126 “SYNC” can be used with FILE mode to make sure the data is flushed to the storage device before the Target responds back to the Initiator. FILE To configure a FILE storage device specify as the device mode: TargetDevice=<path to the storage device>[,FILE][,SYNC] Chelsio T5/T4 Unified Wire For Linux Page 126...
  • Page 127 TargetDevice=<path to the storage device>,BLK <path> Where: Is the path to the actual storage device, such as /dev/sdb. The path must exist in the system. target: #<snip> # /dev/sdb is accessed in BLK mode TargetDevice=/dev/sdb,BLK #<snip> Chelsio T5/T4 Unified Wire For Linux Page 127...

  • Page 128: Target Redirection Support

    Below is an example with ShadowMode disabled: target: #<snip> # any login requests received on 10.193.184.81:3260 will be # redirected to 10.193.184.85:3261 PortalGroup=1@10.193.184.81:3260,[2] PortalGroup=2@10.193.184.85:3261 # the PortalGroup “2” IS present on the same system ShadowMode=No #<snip> Chelsio T5/T4 Unified Wire For Linux Page 128...

  • Page 129: The Command Line Interface Tools "Iscsictl" & "Chisns

    4.9.2. chisns chisns is the command line tool for controlling the iSNS client. This is a simple tool that starts the iSNS client with a client and server parameter. Chelsio T5/T4 Unified Wire For Linux Page 129...
  • Page 130 Chelsio iSCSI targets(s) configuration(s) will be displayed. If target=ALL is specified or no parameters are specified the output will be abbreviated. Specify specific targets to get detailed configuration data. Chelsio T5/T4 Unified Wire For Linux Page 130...
  • Page 131 | TOE | ULP> iscsi_auth_order=<ACL | CHAP> iscsi_target_vendor_id=<string of max. 8 characters> Example: iscsictl -G iscsi_auth_order=ACL The var=const parameter(s) are mandatory. If the var=const parameter is not specified, the command will be denied. Chelsio T5/T4 Unified Wire For Linux Page 131...
  • Page 132 All non-active targets specified will not be loaded / started. For Rules 1-3, if the specified targets are currently active (running), they will get reloaded. For Rules 1 & 2, if the specified targets are not Chelsio T5/T4 Unified Wire For Linux Page 132...
  • Page 133 It defaults to <hostname>. query=<query interval> where query is the initiator query interval (in seconds). It defaults to 60 seconds. Examples: chisns server=192.0.2.10 chisns server=192.0.2.10:3205 id=isnscln2 query=30 Chelsio T5/T4 Unified Wire For Linux Page 133...

  • Page 134: Rules Of Target Reload (I.e. "On The Fly" Changes)

    The following parameter CAN be changed by reloading of the configuration file. The new value will NOT affect any connections and sessions that already completed login phase: TargetAlias MaxConnections InitialR2T ImmediateData FirstBurstLength MaxBurstLength MaxOutstandingR2T HeaderDigest DataDigest MaxRecvDataSegmentLength AuthMethod Auth_CHAP_Initiator Auth_CHAP_Target Chelsio T5/T4 Unified Wire For Linux Page 134...

  • Page 135: System Wide Parameters

    It will select ULP mode when the connection is going through a Chelsio iSCSI acceleration enabled HBA. The purpose of this mode is to optimize the performance of the Chelsio acceleration hardware. Chelsio T5/T4 Unified Wire For Linux Page 135...

  • Page 136: Performance Tuning

    [root@host]# t4_perftune.sh Next, if not already done, load the iSCSI PDU offload target driver (chiscsi_t4) by following the instructions given here. Next, run the chiscsi_set_affinity.sh script to map iSCSI worker threads Chelsio T5/T4 Unified Wire For Linux Page 136...
  • Page 137 Chapter VII. iSCSI PDU Offload Target to different CPUs. [root@host]# chiscsi_set_affinity.sh Chelsio T5/T4 Unified Wire For Linux Page 137...

  • Page 138: Iscsi Pdu Offload Initiator

    Chapter VIII. iSCSI PDU Offload Initiator VIII. iSCSI PDU Offload Initiator Chelsio T5/T4 Unified Wire For Linux Page 138...

  • Page 139: Introduction

    Chapter VIII. iSCSI PDU Offload Initiator 1. Introduction The Chelsio T5/T4 series Adapters support iSCSI acceleration and iSCSI Direct Data Placement (DDP) where the hardware handles the expensive byte touching operations, such as CRC computation and verification, and direct DMA to the final host memory destination:...

  • Page 140: Software Requirements

    Kernel.org linux-3.6 Kernel.org linux-3.7* Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 140...

  • Page 141: Software/Driver Loading

    If loading the driver fails, then follow the steps mentioned below and try again. Run the following command to view all the loaded iSCSI modules [root@host]# lsmod | grep iscsi Chelsio T5/T4 Unified Wire For Linux Page 141...
  • Page 142 Chapter VIII. iSCSI PDU Offload Initiator ii. Now, unload them using the following command: [root@host]# rmmod <modulename> Chelsio T5/T4 Unified Wire For Linux Page 142...

  • Page 143: Software/Driver Unloading

    Chapter VIII. iSCSI PDU Offload Initiator 3. Software/Driver Unloading To unload the driver, execute the following commands: [root@host]# rmmod cxgb4i [root@host]# rmmod libcxgbi Chelsio T5/T4 Unified Wire For Linux Page 143...

  • Page 144: Software/Driver Configuration And Fine-Tuning

    Create an interface file located under iface directory for the new transport class cxgb4i in the following format: iface.iscsi_ifacename = <iface file name> iface.hwaddress = <MAC address> iface.transport_name = cxgb4i iface.net_ifacename = <ethX> iface.ipaddress = <iscsi ip address> Chelsio T5/T4 Unified Wire For Linux Page 144...
  • Page 145 Start Daemon from /sbin by using the following command: [root@host]# iscsid -f -d 3 iscsid is already running, then kill the service and start it as shown above Note after installing the Chelsio Unified Wire package. Chelsio T5/T4 Unified Wire For Linux Page 145...
  • Page 146 -m node -T <iqn name of target> -p <target ip address>:<target port no> -I <cxgb4i iface file name> -u E.g.:- [root@host]# iscsiadm -m node -T iqn.2004-05.com.chelsio.target1 -p 102.2.2.155:3260,1 -I cxgb4i.00:07:43:04:5b:da -u Other options can be found by typing iscsiadm --help Note Chelsio T5/T4 Unified Wire For Linux Page 146...

  • Page 147: Auto Login From Cxgb4I Initiator At Os Bootup

    -q iscsi_tcpmodprobe -q ib_iser modprobe -q cxgb4i modprobe -q cxgb3i modprobe -q bnx2i modprobe -q be2iscsi daemon brcm_iscsiuio daemon $prog retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval Chelsio T5/T4 Unified Wire For Linux Page 147...

  • Page 148: Data Center Bridging (Dcb)

    Chapter IX. Data Center Bridging (DCB) IX. Data Center Bridging (DCB) Chelsio T5/T4 Unified Wire For Linux Page 148...

  • Page 149: Introduction

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Kernel.org linux-3.4* Kernel.org linux-3.6 Kernel.org linux-3.7* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 149...

  • Page 150: Software/Driver Loading

    Run the following commands: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom [root@host]# modprobe csiostor Once the storage and networking traffic are started simultaneously, they will honor DCB settings defined on the switch. Chelsio T5/T4 Unified Wire For Linux Page 150...

  • Page 151: Software/Driver Unloading

    3. Software/Driver Unloading To disable DCB feature, unload FCoE Initiator and Network drivers: [root@host]# rmmod csiostor [root@host]# rmmod cxgb4 If t4_tom is loaded, please reboot machine after unloading FCoE Initiator and Note Network drivers. Chelsio T5/T4 Unified Wire For Linux Page 151...

  • Page 152: Software/Driver Configuration And Fine-Tuning

    0 switch(config-cmap-qos)# class-map type qos class-fcoe switch(config-cmap-qos)# match cos 3 iii. Configure queuing class-maps. switch(config)#class-map type queuing class-nic switch(config-cmap-que)#match qos-group 2 iv. Configure network-qos class-maps. switch(config)#class-map type network-qos class-nic switch(config-cmap-nq)#match qos-group Chelsio T5/T4 Unified Wire For Linux Page 152...
  • Page 153 4.1.2. Configuring the FCoE/FC ports In this procedure, you may need to adjust some of the parameters to suit your environment, such as VLAN IDs, Ethernet interfaces, and virtual Fibre Channel interfaces Chelsio T5/T4 Unified Wire For Linux Page 153...
  • Page 154 2 switch(config-if)#no shutdown switch(config)#exit iv. Enabling DCB: switch(config)# interface ethernet 1/13 switch(config-if)# priority-flow-control mode auto switch(config-if)# flowcontrol send off switch(config-if)# flowcontrol receive off switch(config-if)# lldp transmit switch(config-if)# lldp receive switch(config-if)# no shutdown Chelsio T5/T4 Unified Wire For Linux Page 154...

  • Page 155: Configuring The Brocade 8000 Switch

    2 weight 60 switch(conf-cee-map)#priority-table 2 2 2 1 2 2 2 2 iii. Configure the CEE interface as a Layer 2 switch port. Example of configuring the switch port as a 10-Gigabit Ethernet interface. Chelsio T5/T4 Unified Wire For Linux Page 155...
  • Page 156 Unlike cisco, only one VLAN ID can carry FCoE traffic for now on Brocade 8000. Note It is their limitation. v. Save the Configuration switch#copy running-config startup-config Chelsio T5/T4 Unified Wire For Linux Page 156...

  • Page 157: Fcoe Pdu Offload Target

    Chapter X. FCoE PDU Offload Target X. FCoE PDU Offload Target Chelsio T5/T4 Unified Wire For Linux Page 157...

  • Page 158: Introduction

    The license is tied to the selected Chelsio NIC present in the system. The license file will be generated depending on your requirement for a Chelsio FCoE Target. To obtain a license key file, which could be either a production or an evaluation version, please follow the steps below. Chelsio T5/T4 Unified Wire For Linux Page 158...
  • Page 159 For the selected NIC, the following information is gathered: PCI vendor ID MAC address Link speed Privacy notice: The above information is used by Chelsio Communications strictly for license management. Chelsio Communications, Inc. does not share, sell, rent, or lease the information to any third party. [root@host]# chinfotool Scanning System for network devices..
  • Page 160 Chelsio, the user will receive a license file tailored to the system. Copy the file /etc/chelsio-fcoe/ [root@host]# cp -f chfcoe.key /etc/chelsio-fcoe/ The FCoE target software is now unlocked, ready to be configured and used. Chelsio T5/T4 Unified Wire For Linux Page 160...

  • Page 161: Software/Driver Loading

    [root@host]# modprobe chfcoe chfcoe_fip_mode=1  To load the driver in VN2VN mode with VLAN support, run the following command: [root@host]# modprobe chfcoe chfcoe_fip_mode=1 chfcoe_vlanid=<vlan_id> Alternatively, you can edit /etc/modprobe.d/chfcoe.conf for specifying VN2VN and VLAN options. Chelsio T5/T4 Unified Wire For Linux Page 161...

  • Page 162: Software/Driver Configuration And Fine-Tuning

    Configure qos class-maps and set the traffic priorities: NIC uses priority 0 and FcoE uses priority 3. switch(config)#class-map type qos class-nic switch(config-cmap-qos)# match cos 0 switch(config-cmap-qos)# class-map type qos class-fcoe switch(config-cmap-qos)# match cos 3 iii. Configure queuing class-maps. switch(config)#class-map type queuing class-nic switch(config-cmap-que)#match qos-group 2 Chelsio T5/T4 Unified Wire For Linux Page 162...
  • Page 163 Apply the new policy (PFC on NIC and FcoE traffic) to the entire system. switch(config)#system qos switch(config-sys-qos)#service-policy type qos input policy-test switch(config-sys-qos)#service-policy type queuing output policy-test switch(config-sys-qos)#service-policy type queuing input policy-test switch(config-sys-qos)#service-policy type network-qos policy-test Chelsio T5/T4 Unified Wire For Linux Page 163...
  • Page 164 VLAN and FCoE VLAN. iii. Assign VLAN ID to the Ethernet port on which FCoE service was enabled in step1. switch(config)# interface ethernet 1/13 switch(config-if)#switchport mode trunk switch(config-if)#switchport trunk allowed vlan 2 switch(config-if)#no shutdown switch(config)#exit Chelsio T5/T4 Unified Wire For Linux Page 164...

  • Page 165: Collecting Port Information

    You can use cxgbtool to view both target and initiator WWPNs. The command requires an adapter device file to be specified. After loading chfcoe driver, device files will be created for each adapter in /dev/chfcoeX, where X is index of the adapter. [root@host]# ls /dev/chfcoe* /dev/chfcoe0 Chelsio T5/T4 Unified Wire For Linux Page 165...
  • Page 166 If FCoE PDU Offload Target is operating in FCF mode, then the local node information will be available only after the target completes FLOGI to the swtich. In order to identify Chelsio target’s wwpn from other vendors’, the WWPN always Note begins with 0x5000743. Chelsio T5/T4 Unified Wire For Linux Page 166...
  • Page 167 Chapter X. FCoE PDU Offload Target 3.2.2. Verifying remote ports To verify remote ports (fabric, name server, initiator ports etc.), use the following command: [root@host] # cxgbtool stor -a <adapter device file> --show-rnode wwn=<target wwpn> Chelsio T5/T4 Unified Wire For Linux Page 167...
  • Page 168 Chapter X. FCoE PDU Offload Target Chelsio T5/T4 Unified Wire For Linux Page 168...

  • Page 169: Configuring Luns On Target

    Ensure that SCST handler modules, which are used in configuration (eg: scst_vdisk), are loaded before proceeding. iv. Configure LUNs on target by running the following command. [root@host]# scstadmin -config <LUN Config file> Chelsio T5/T4 Unified Wire For Linux Page 169...
  • Page 170 Chapter X. FCoE PDU Offload Target The following configuration file adds three LUNs(ram disk, physical disk & nullio disk) for the target specified. Only initiators present in the group will be able to access the LUNs. Chelsio T5/T4 Unified Wire For Linux Page 170...

  • Page 171: Configuring Persistent Target

    Place the configuration file created in the previous step, scst.conf, under /etc/chelsio-fcoe/ directory. iii. If any handler other than scst_vdisk is used in SCST configuration, please add module load entry for the same under chfcoe_start() section of /etc/init.d/chfcoe file. Chelsio T5/T4 Unified Wire For Linux Page 171...

  • Page 172: Verifying Initiators Connected To The Target

    Once connected to the target, initiators will be able to access the LUNs configured on the target. Execute the following command to list the initiators which are connected to the target: [root@host] # scstadmin –list_session Chelsio T5/T4 Unified Wire For Linux Page 172...

  • Page 173: Removing Luns

    If hyperthreading is enabled, use both thread siblings of a core for binding either workers or irqs. [root@host]# cat /sys/devices/system/cpu/cpu0/topology/threads_siblings_list 0,16 Use both 0 and 16 CPUs for either workers or irqs. Chelsio T5/T4 Unified Wire For Linux Page 173...
  • Page 174 19161's new affinity list: 19 chfcoeworker_8: pid 19162's new affinity list: 20 chfcoeworker_9: pid 19163's new affinity list: 21 chfcoeworker_10: pid 19164's new affinity list: 22 chfcoeworker_11: pid 19165's new affinity list: 23 Chelsio T5/T4 Unified Wire For Linux Page 174...
  • Page 175 Writing 1 in /proc/irq/165/smp_affinity Writing 2 in /proc/irq/166/smp_affinity Writing 10000 in /proc/irq/167/smp_affinity Writing 20000 in /proc/irq/168/smp_affinity Writing 1 in /proc/irq/169/smp_affinity Writing 2 in /proc/irq/170/smp_affinity Writing 10000 in /proc/irq/171/smp_affinity Writing 20000 in /proc/irq/172/smp_affinity Chelsio T5/T4 Unified Wire For Linux Page 175...

  • Page 176: Software/Driver Unloading

    Chapter X. FCoE PDU Offload Target 4. Software/Driver Unloading To unload the driver, run the following command: [root@host]# modprobe –r chfcoe Chelsio T5/T4 Unified Wire For Linux Page 176...

  • Page 177: Fcoe Full Offload Initiator

    Chapter XI. FCoE Full Offload Initiator XI. FCoE Full Offload Initiator Chelsio T5/T4 Unified Wire For Linux Page 177...

  • Page 178: Introduction

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Kernel.org linux-3.4* Kernel.org linux-3.6 Kernel.org linux-3.7* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 178...

  • Page 179: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver, execute the following: [root@host]# modprobe csiostor To load the driver on any SLES distribution, execute the following command: Note [root@host]# modprobe csiostor --allow Chelsio T5/T4 Unified Wire For Linux Page 179...

  • Page 180: Software/Driver Unloading

    3. Software/Driver Unloading To unload the driver: [root@host]# modprobe -r csiostor If multipath services are running, unload of FCoE driver is not possible. Stop the Note multipath service and then unload the driver. Chelsio T5/T4 Unified Wire For Linux Page 180...

  • Page 181: Software/Driver Configuration And Fine-Tuning

     Inorder to identify chelsio fc_host from other vendor fc_host, the WWPN always begins with 0x5000743 Alternatively, the local port information can also be found using: [root@host]# cxgbtool stor –a <adapter_no> --show-lnode Chelsio T5/T4 Unified Wire For Linux Page 181...
  • Page 182 Chapter XI. FCoE Full Offload Initiator Chelsio T5/T4 Unified Wire For Linux Page 182...
  • Page 183 R can correspond to NameServer, Management Server and other initiator ports Note logged in to the switch and targets. Alternatively, the local ports can also be found using cxgbtool [root@host]# cxgbtool stor –a <adapter no> --show-lnode Chelsio T5/T4 Unified Wire For Linux Page 183...
  • Page 184 Chapter XI. FCoE Full Offload Initiator After finding out the wwpn of the local node, to verify the list of discovered targets, use the following command. [root@host]# cxgbtool stor –a <adapter_no> --show-rnode --wwn=<wwpn of lnode> Chelsio T5/T4 Unified Wire For Linux Page 184...

  • Page 185: Formatting The Luns And Mounting The Filesystem

    Chapter XI. FCoE Full Offload Initiator 4.3. Formatting the LUNs and Mounting the Filesystem Use lsscsi –g to list the LUNs discovered by the initiator [root@host]# lsscsi –g Chelsio T5/T4 Unified Wire For Linux Page 185...
  • Page 186 Chapter XI. FCoE Full Offload Initiator Alternatively, the LUNs discovered by the Chelsio FCoE initiators can be accessed via easily- identifiable ‘udev’ path device files like: [root@host]# ls /dev/disk/by-path/pci-0000:04:00.0-csio-fcoe <local_wwpn>:<remote_wwpn>:<lun_wwn> Chelsio T5/T4 Unified Wire For Linux Page 186...

  • Page 187: Creating Filesystem

    Chapter XI. FCoE Full Offload Initiator 4.4. Creating Filesystem Create an ext3 filesystem using the following command: [root@host]# mkfs.ext3 /dev/sdx Chelsio T5/T4 Unified Wire For Linux Page 187...

  • Page 188: Mounting The Formatted Lun

    Chapter XI. FCoE Full Offload Initiator 4.5. Mounting the formatted LUN The formatted LUN can be mounted on the specified mountpoint using the following command: [root@host]# mount /dev/sdx /mnt Chelsio T5/T4 Unified Wire For Linux Page 188...

  • Page 189: Offload Bonding Driver

    Chapter XII. Offload Bonding driver XII. Offload Bonding driver Chelsio T5/T4 Unified Wire For Linux Page 189...

  • Page 190: Introduction

    It also provides redundancy in case one of link fails. The traffic running over the bonded interface can be fully offloaded to the T5/T4 Adapter, thus freeing the CPU from TCP/IP overhead.
  • Page 191 Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 191...

  • Page 192: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host]# modprobe bonding Chelsio T5/T4 Unified Wire For Linux Page 192...

  • Page 193: Software/Driver Unloading

    Chapter XII. Offload Bonding driver 3. Software/Driver Unloading To unload the driver, run the following command: [root@host]# rmmod bonding Chelsio T5/T4 Unified Wire For Linux Page 193...

  • Page 194: Software/Driver Configuration And Fine-Tuning

    Assign IPv4/IPv6 address to the bonded interface: [root@host]# ifconfig bond0 X.X.X.X/Y [root@host]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up All TCP traffic will be offloaded over the bonded interface now. Chelsio T5/T4 Unified Wire For Linux Page 194...

  • Page 195: Offload Multi-Adapter Failover (Mafo)

    Chapter XIII. Offload Multi-Adapter Failover (MAFO) XIII. Offload Multi-Adapter Failover (MAFO) Chelsio T5/T4 Unified Wire For Linux Page 195...

  • Page 196: Introduction

    Chapter XIII. Offload Multi-Adapter Failover (MAFO) 1. Introduction Chelsio’s T5 and T4-based adapters offer a complete suite of high reliability features, including adapter-to-adapter failover. The patented offload Multi-Adapter Failover (MAFO) feature ensures all offloaded traffic continue operating seamless in the face of port failure.

  • Page 197: Software Requirements

    Multi-Adapter failover feature will work for Link Down events caused by:  Cable unplug on bonded interface.  Bringing corresponding switch port down. The feature will not work if the bonded interfaces are administratively taken Note down. Chelsio T5/T4 Unified Wire For Linux Page 197...

  • Page 198: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host]# modprobe bonding Chelsio T5/T4 Unified Wire For Linux Page 198...

  • Page 199: Software/Driver Unloading

    Chapter XIII. Offload Multi-Adapter Failover (MAFO) 3. Software/Driver Unloading To unload the driver, run the following command: [root@host]# rmmod bonding Chelsio T5/T4 Unified Wire For Linux Page 199...

  • Page 200: Software/Driver Configuration And Fine-Tuning

    [root@host]# ifconfig bond0 X.X.X.X/Y [root@host]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up All TCP traffic will be offloaded over the bonded interface now and fail-over will happen in case of link-down event. Chelsio T5/T4 Unified Wire For Linux Page 200...

  • Page 201: Network Device Configuration

    Some operating systems may attempt to auto-configure the detected hardware Note and some may not detect all ports on a multi-port adapter. If this happens, please refer to the operating system documentation for manually configuring the network device. Chelsio T5/T4 Unified Wire For Linux Page 201...

  • Page 202: Udp Segmentation Offload And Pacing

    Chapter XIV. UDP Segmentation Offload and Pacing XIV. UDP Segmentation Offload and Pacing Chelsio T5/T4 Unified Wire For Linux Page 202...

  • Page 203: Introduction

    Chapter XIV. UDP Segmentation Offload and Pacing 1. Introduction Chelsio’s T5/T4 series of adapters provide UDP segmentation offload and per-stream rate shaping to drastically lower server CPU utilization, increase content delivery capacity, and improve service quality. Tailored for UDP content, UDP Segmentation Offload (USO) technology moves the processing required to packetize UDP data and rate control its transmission from software running on the host to the network adapter.

  • Page 204: Software Requirements

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6 Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Performed. Chelsio T5/T4 Unified Wire For Linux Page 204...

  • Page 205: Software/Driver Loading

    [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Though normally associated with the Chelsio TCP Offload engine, the t4_tom module is required in order to allow for the proper redirection of UDP socket calls. Chelsio T5/T4 Unified Wire For Linux Page 205...

  • Page 206: Software/Driver Unloading

    Chapter XIV. UDP Segmentation Offload and Pacing 3. Software/Driver Unloading Reboot the system to unload the driver. Chelsio T5/T4 Unified Wire For Linux Page 206...

  • Page 207: Software/Driver Configuration And Fine-Tuning

    This value needs to be in the range of 0 to 14 for T4 series of adapters and a range of 0 to 15 for T5 series of adapters. The application will function according to the parameters set for that traffic class.
  • Page 208 3. In step (iv), define a new socket option: setsockopt (sockfd,17,UDP_RTPHEADERLEN,&rtp_header_size, sizeof(rtp_header_size)); Here,  &rtp_header_size : pointer to the RTP header length variable  sizeof(rtp_header_size) : the size of the RTP header length variable Chelsio T5/T4 Unified Wire For Linux Page 208...

  • Page 209: Configuring Udp Pacing

    (Kbps) for this UDP stream. This value should be in the range of 50 Kbps to 50 Mbps for T4 adapters. For T5 adapters, the value should be in the range of 100 kbps to 1 Gbps.
  • Page 210 10240 UDP connections, it can check the return code of ENOSPC from a send() or sendto() call and close this socket and open a new one that uses the kernel UDP stack. Chelsio T5/T4 Unified Wire For Linux Page 210...

  • Page 211: Offload Ipv6 Driver

    Chapter XV. Offload IPv6 driver XV. Offload IPv6 driver Chelsio T5/T4 Unified Wire For Linux Page 211...

  • Page 212: Introduction

    Internet Protocol version 6 (IPv6) is a version of the Internet Protocol (IP) designed to succeed the Internet Protocol version 4 (IPv4). Chelsio’s Offload IPv6 feature provides support to fully offload IPv6 traffic to the T5/T4 adapter. 1.1. Hardware Requirements 1.1.1.
  • Page 213 Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 213...

  • Page 214: Software/Driver Loading

    (t4_tom) drivers. The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom All the IPv6 traffic will be offloaded now. Chelsio T5/T4 Unified Wire For Linux Page 214...

  • Page 215: Software/Driver Unloading

    To disable Offload IPv6 feature, unload NIC and TOE drivers: 3.1. Unloading the NIC driver To unload the NIC driver, run the following command: [root@host]# rmmod cxgb4 3.2. Unloading the TOE driver Please reboot the system to unload the TOE driver. Chelsio T5/T4 Unified Wire For Linux Page 215...

  • Page 216: Bypass Driver

    Chapter XVI. Bypass Driver XVI. Bypass Driver Chelsio T5/T4 Unified Wire For Linux Page 216...

  • Page 217: Introduction

    B420, port 0 to port 1 and vice versa; and in B404, port 0 to port 1 and port 2 to port 3 and vice versa. Normal Mode The Bypass Adapters can be programmed to function as a NIC. In this mode all the packets are redirected to the host. Chelsio T5/T4 Unified Wire For Linux Page 217...

  • Page 218: Hardware Requirements

    Finally, the input action directs the packet to the host where it can be processed by the application. 1.2. Hardware Requirements 1.2.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with Chelsio Bypass driver: B404-BT B420-SR Chelsio T5/T4 Unified Wire For Linux Page 218...

  • Page 219: Software Requirements

    Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Kernel.org linux-3.9* Kernel.org linux-3.10* Kernel.org linux-3.11* Kernel.org linux-3.12* Kernel.org linux-3.13* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed. Chelsio T5/T4 Unified Wire For Linux Page 219...

  • Page 220: Software/Driver Loading

    The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. Run the following command to load the Bypass driver: [root@host]# modprobe cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 220...

  • Page 221: Software/Driver Unloading

    Chapter XVI. Bypass Driver 3. Software/Driver Unloading Run the following command to unload the Bypass driver: [root@host]# rmmod cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 221...

  • Page 222: Software/Driver Configuration And Fine-Tuning

    The redirect management element will interface with the SDK server to take the appropriate actions to manage tables and rules. 4.2.1. Bypass Management The bypass CLI will have the following syntax: [root@host]# bypass ethX get|set --key [value] Chelsio T5/T4 Unified Wire For Linux Page 222...
  • Page 223 4.2.3. Bypass State To obtain the current state of the switch: [root@host]# bypass ethX get --current_state To set the state of the switch: [root@host]# bypass ethX set --current_state [bypass|disconnect|normal] Chelsio T5/T4 Unified Wire For Linux Page 223...
  • Page 224 When a rule matches the action associated with it, that rule is performed and higher number rules are not evaluated. Rules are not evaluated until a table is activated. Chelsio T5/T4 Unified Wire For Linux Page 224...
  • Page 225 Redirect rule srcaddr6 IPv6 Source IP address dstaddr6 IPv6 Destination IP address srcmask6 IPv6 Source IP address mask (Default: FFFF:FFFF:FFFF:FFFF:FFF F:FFFF:FFFF:FFFF) dstmask6 IPv6 Destination IP address mask (Default:FFFF:FFFF:FFFF:F FFF:FFFF:FFFF:FFFF:FFFF) Chelsio T5/T4 Unified Wire For Linux Page 225...
  • Page 226 [root@host]# redirect ethX create_table --table 1 Activate the table: [root@host]# redirect ethX activate_table --table 1 Add a rule to the newly created table: [root@host]# redirect ethX add --table 1 --proto icmp –-action drop –-port 0 Chelsio T5/T4 Unified Wire For Linux Page 226...
  • Page 227 IPv4 mode. For IPv6, the ba_server needs to be started manually. [root@host]# service bad start To stop the service: [root@host]# service bad stop To restart the service: [root@host]# service bad restart Chelsio T5/T4 Unified Wire For Linux Page 227...

  • Page 228: Wd Sniffing And Tracing

    Chapter XVII. WD Sniffing and Tracing XVII. WD Sniffing and Tracing Chelsio T5/T4 Unified Wire For Linux Page 228...

  • Page 229: Theory Of Operation

    Connect Client A and B to ports 0 and 1 or ports 2 and 3. d) Enable tracing. In either mode the targeted traffic bypasses the kernel TCP/IP stack and is delivered directly to user space by means of an RX queue. Chelsio T5/T4 Unified Wire For Linux Page 229...

  • Page 230: Hardware Requirements

    Schematic diagram of T4 sniffer and tracer 1.1. Hardware Requirements 1.1.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with the tools: T420-CR T440-CR T422-CR T404-BT T420-BCH Chelsio T5/T4 Unified Wire For Linux Page 230...

  • Page 231: Software Requirements

    Suse Linux Enterprise Server 11 SP2 kernel (SLES11SP2), 3.0.13-0.27 * Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Chelsio T5/T4 Unified Wire For Linux Page 231...

  • Page 232: Installation And Usage

    2.3. Using Tracer (wd_tcpdump_trace) 1. Setup: Wire tapping requires 3 systems with one machine having a T4 card with two or more ports. The machines should be setup in the following manner: Chelsio T5/T4 Unified Wire For Linux Page 232...
  • Page 233 [root@host]# wd_tcpdump_trace -i <iface> Try ping or ssh between machines A and B. The traffic should successfully make it from end to end and wd_tcpdump_trace on the DUT should show the tapped traffic. Chelsio T5/T4 Unified Wire For Linux Page 233...

  • Page 234: Xviii.classification And Filtering

    Chapter XVIII. Classification and Filtering XVIII. Classification and Filtering Chelsio T5/T4 Unified Wire For Linux Page 234...

  • Page 235: Introduction

    The following are the currently shipping Chelsio Adapters that are compatible with the Classification and Filtering feature: T502-BT T580-CR T520-LL-CR T520-SO-CR T520-CR T522-CR T540-CR T580-LP-CR T580-SO-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX Chelsio T5/T4 Unified Wire For Linux Page 235...

  • Page 236: Software Requirements

    Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8 Kernel.org linux-3.9 Kernel.org linux-3.10 Kernel.org linux-3.11 Kernel.org linux-3.12 Kernel.org linux-3.13 Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Peformed. Chelsio T5/T4 Unified Wire For Linux Page 236...

  • Page 237: Usage

    Creating Filter Rules Network driver (cxgb4) must be installed and loaded before setting the filter rule. If you haven’t done already, run the Unified Wire Installer with the appropriate T5/T4 configuration tuning option to install the Network Driver. Next, run the following command to load the network driver:...
  • Page 238 [root@host]# cxgbtool ethX filter 0 action switch iport 0 eport 1 vlan 3 The above filter rule will route all ingress packets that match VLAN id 3 from port 0 of Chelsio adapter to port 1.Remaining packets will be sent to the host. Chelsio T5/T4 Unified Wire For Linux Page 238...

  • Page 239: Listing Filter Rules

    For more information on additional parameters, refer cxgbtool manual by running Note man cxgbtool command. prio argument is not supported for LE-TCAM filters when T5 Hash Filter config Note file is used. 2.3. Listing Filter Rules To list the filters set, run the following command: [root@host]# cxgbtool ethX filter show 2.4.

  • Page 240: Layer 3 Example

    Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:14372 errors:0 dropped:0 overruns:0 frame:0 TX packets:62203 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1585952 (1.5 MiB) TX bytes:4798122 (4.5 MiB) Interrupt:16 Chelsio T5/T4 Unified Wire For Linux Page 240...
  • Page 241 Chapter XVIII. Classification and Filtering Setup a static OR default route towards T5 router to reach 102.1.2.0/24 network [root@host]# route add -net 102.1.2.0/24 gw 102.1.1.250  Follow these steps on Node 2 Configure IP address and enable the 3 interfaces: [root@host]# ifconfig eth0 102.1.2.1/24 up...
  • Page 242 [root@host]# cxgbtool eth0 filter 0 lip 102.1.2.0/24 hitcnts 1 action switch eport 1 smac 00:07:43:04:96:48 dmac 00:07:43:12:D4:88 smac dmac Where, is the MAC address of eth1 interface on T5 adapter machine and is the MAC address of eth0 interface on Node 2. Chelsio T5/T4 Unified Wire For Linux Page 242...

  • Page 243: Layer 2 Example

    [root@host]# cxgbtool eth0 filter 1 lip 102.1.1.0/24 hitcnts 1 action switch eport 0 smac 00:07:43:04:96:40 dmac 00:07:43:04:7D:50 smac dmac Where, is the MAC address of eth0 interface on T5 adapter machine and is the MAC address of eth0 interface on Node 1. 2.6. Layer 2 example Here’s an example on how to achieve L2 switching functionality.
  • Page 244 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:1961 errors:0 dropped:2 overruns:0 frame:0 TX packets:141 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:218606 (213.4 KiB) TX bytes:17483 (17.0 KiB) Interrupt:17 Chelsio T5/T4 Unified Wire For Linux Page 244...
  • Page 245 = fragmentation, macmatch, mpshittype, protocol, tos, port, fcoe Unload and re-load the cxgb4 driver. iii. Enable promiscuous mode on both the interfaces on T5 adapter machine: [root@host]# ifconfig eth0 up promisc [root@host]# ifconfig eth1 up promisc Build and install latest iproute2 package...
  • Page 246 Create a filter to match incoming packet's dst-mac 00:07:43:04:7d:50 with particular mac-idx and switch it out via eport 0: [root@host]# cxgbtool eth0 filter 1 macidx 7 action switch eport 0 hitcnts 1 Chelsio T5/T4 Unified Wire For Linux Page 246...

  • Page 247: Hash/Ddr Filters

    The default (Unified Wire) configuration tuning option allows you to create LE-TCAM filters, which has a limit of 496 filter rules. If you wish to create more, select T5 Hash Filter configuration tuning option during installation which allows you to create HASH/DDR filters with a capacity of ~0.5 million filter rules.

  • Page 248: Listing Filter Rules

    To list the Hash/DDR filters set, run the following command: [root@host]# cat /proc/drivers/cxgb4/<bus-id>/hash_filters  To list the both LE-TCAM and Hash/DDR filters set, run the following command: [root@host]# cxgbtool ethX filter show Chelsio T5/T4 Unified Wire For Linux Page 248...

  • Page 249: Removing Filter Rules

    3.4. Swap MAC feature Chelsio T5’s Swap MAC feature swaps packet source MAC and destination MAC addresses. This is applicable only for switch filter rules. Here’s an example: [root@host]# cxgbtool eth2 filter 1 action switch lip 102.2.2.1 fip 102.2.2.2 lport 5001 fport 14000...
  • Page 250 Configure the RSS Queue corresponding to trace0 filter configured above. Determine the RspQ ID of the queues by looking at Trace QType in /sys/kernel/debug/cxgb4/<bus- id>/sge_qinfo file. [root@host]# cxgbtool ethX reg 0x09808=<Trace-Queue0- RspQ ID > Chelsio T5/T4 Unified Wire For Linux Page 250...
  • Page 251 [root@host]# echo rx1 snaplen=40 > /proc/driver/cxgb4/<bus_id>/trace3 [root@host]# cxgbtool ethX reg 0x09ff4=<Trace-Queue1- RspQ ID > [root@host]# cxgbtool ethX reg 0x09ffc=<Trace-Queue2- RspQ ID > [root@host]# cxgbtool ethX reg 0x0a004=<Trace-Queue3- RspQ ID > Use “snaplen=60” in case of IPV6. Note Chelsio T5/T4 Unified Wire For Linux Page 251...

  • Page 252: Traffic Management

    XIX. Traffic Management XIX. Traffic Management Chelsio T5/T4 Unified Wire For Linux Page 252...

  • Page 253: Introduction

    XIX. Traffic Management 1. Introduction Traffic Management capabilities built-in to Chelsio T5/T4 CNAs can shape transmit data traffic through the use of sophisticated queuing and scheduling algorithms built-in to the ASIC hardware which provides fine-grained software control over latency and bandwidth parameters such as packet rate and byte rate.

  • Page 254: Software Requirements

    Suse Linux Enterprise Server 11 SP3 kernel (SLES11SP3), 3.0.76-0.11 Ubuntu 12.04, 3.2.0-23 Ubuntu 12.04.2, 3.5.0-23* Kernel.org linux-3.4 Kernel.org linux-3.6* Kernel.org linux-3.7 Kernel.org linux-3.8* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA Peformed. Chelsio T5/T4 Unified Wire For Linux Page 254...

  • Page 255: Software/Driver Loading

    The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. Run the following commands to load the drivers: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Chelsio T5/T4 Unified Wire For Linux Page 255...

  • Page 256: Software/Driver Unloading

    XIX. Traffic Management 3. Software/Driver Unloading Reboot the system to unload the driver. Chelsio T5/T4 Unified Wire For Linux Page 256...

  • Page 257: Software/Driver Configuration And Fine-Tuning

     Channel No. is the port on which data is flowing (0-3).  scheduler-class-index is the TCP traffic class (0-14 for T4 series of adapters and 0-15 for T5 series of adapters).  maximum-rate is the bit rate (Kbps) for this TCP stream.
  • Page 258  Channel No. is the port on which data is flowing (0-3).  scheduler-class-index is the TCP traffic class (0-14 for T4 series of adapters and 0-15 for T5 series of adapters).  weight is the weight to be used for a weighted-round-robin scheduling hierarchy. Possible values include 1 to 99.

  • Page 259: Configuring Traffic Management

    If the TX queue is all, * or any negative value, the binding will apply to all of the TX Note queues associated with the interface. If the class is unbind, clear or any negative value, the TX queue(s) will be unbound from any current TX Scheduler Class binding. Chelsio T5/T4 Unified Wire For Linux Page 259...
  • Page 260 [root@host]# cop -d –o <output_policy_file> <new_policy_file> 4. Apply the COP policy: [root@host]# cxgbtool ethX policy <output_policy_file> Where, ethX: Chelsio interface For more information on additional parameters, refer cop manual by running the Note man cop command. Chelsio T5/T4 Unified Wire For Linux Page 260...
  • Page 261 Here:  sockfd : The file descriptor of the TCP socket  &cl : Pointer to the class variables  sizeof(cl) : The size of the variable 5. Now, compile the application. Chelsio T5/T4 Unified Wire For Linux Page 261...

  • Page 262: Usage

    The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for offloaded connections: 1. Load the Network driver in TOE mode [root@host]# modprobe t4_tom Chelsio T5/T4 Unified Wire For Linux Page 262...

  • Page 263: Offloaded Connections With Modified Application

    2. Modify the application as mentioned in the Configuring Traffic Management section. 3. Set the appropriate rule for class 0 [root@host]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 Chelsio T5/T4 Unified Wire For Linux Page 263...

  • Page 264: Unified Wire Manager (Um)

    XX. Unified Wire Manager (UM) XX. Unified Wire Manager (UM) Chelsio T5/T4 Unified Wire For Linux Page 264...

  • Page 265: Introduction

    Chelsio FCoE initiators available on Linux and Windows operating systems can be managed. Users can manage Option ROM (PXE and iSCSI for T3; PXE, FCoE and iSCSI for T5 and T4) capability for Chelsio cards using various tools available in the software.

  • Page 266: Reference Architecture

    The Management Client can be used to connect to agents and manage them. Once connected you can view and configure Chelsio HBAs and related networking, storage and hardware properties. You can use either the CLI or Web GUI client to manage agents based on your Chelsio T5/T4 Unified Wire For Linux Page 266...

  • Page 267: Authentication And Encryption

    CLI client or the Web GUI and the agent, are encrypted using SSL. This ensures that the data cannot be accessed when it is being transmitted over the network. Chelsio T5/T4 Unified Wire For Linux Page 267...

  • Page 268: Hardware And Software

    T520-CR T522-CR T580-LP-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX S302E S302E-C S310E-CR S310E-CR-C S310E-CXA S310E-SR+ S310E-SR S310E-BT S320E-CR S320E-LP-CR S320E-CXA S320EM-BS S320EM-BCH N320E-G2-CR N320E N320E-CXA N320E-BT N310E N310E-CXA Chelsio T5/T4 Unified Wire For Linux Page 268...

  • Page 269: Platform/Component Matrix

    T3 : 2.0.0.1 (RHEL 5.8,6.3) T4,T5: 2.10.1.2 Bypass T4: 2.10.1.2 T3 : 2.0.0.1 Bonding T4: 3.7.1 T3: 5.2.0-0560 iSCSI Target T4,T5: 2.10.1.2-813 Open iSCSI Initiator T3,T4,T5: 2.0-872 iWARP T3,T4,T5: 2.10.1.2 FCoE Initiator T4: 2.10.1.2 Chelsio T5/T4 Unified Wire For Linux Page 269...

  • Page 270: Installing Unified Wire Manager

    Chelsio Unified Wire has been designed to install Unified Wire Manager (UM) by default. All the three UM components, i.e. Management Agent, Client and Station, will be installed on selecting any of the Terminator 4/Terminator 5 configuration tuning options during installation. Hence, no separate installation is required. Chelsio T5/T4 Unified Wire For Linux Page 270...

  • Page 271: Verifying Um Components Status

    2. You can also execute the following comand to determine if Management Agent is running: [root@chelsio]# /etc/init.d/chelsio-uwire_mgmtd status The above command should display the following result: Unified Wire Manager Agent : Running Chelsio T5/T4 Unified Wire For Linux Page 271...

  • Page 272: Verifying Management Client

    While accessing the Web Management Interface, if an error “Management Note Station is not running” is displayed at the log-in page, follow the above steps to restart Management Station and try again. Chelsio T5/T4 Unified Wire For Linux Page 272...

  • Page 273: Management Agent

    Execute the following command to list the service configuration. [root@host]# chkconfig --list chelsio-uwire_mgmtd Execute the following command to enable/disable the service to start at system runlevel 5. [root@host]# chkconfig --level 5 chelsio-uwire_mgmtd on/off Chelsio T5/T4 Unified Wire For Linux Page 273...

  • Page 274: Firewall

    If there is a firewall appliance / software protecting the network that the system is on, and you wish to connect to the system from a different network, using the client, the firewall appliance also needs to be configured appropriately. Chelsio T5/T4 Unified Wire For Linux Page 274...

  • Page 275: Cli Client

    Also, two different Web GUI or CLI clients cannot connect to a management agent at the same time. There is no mechanism to allow this scenario. Chelsio T5/T4 Unified Wire For Linux Page 275...

  • Page 276: Web Gui Client

    [root@host]# openssl rsa -in server.key.org -out server.key [root@host]# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt [root@host]# cp server.crt ./ssl.crt [root@host]# cp server.key ./ssl.key 2. Start/Restart Apache services [root@host]# rcapache2 [start|restart] Chelsio T5/T4 Unified Wire For Linux Page 276...

  • Page 277: Accessing Web Management Interface

    2. The security certificate used by the web server is a generic one. It may cause the following types of prompts in different browsers. You will need to select the correct option to continue. Security Certificate prompt in Internet Explorer Figure 7.2 (a) - Chelsio T5/T4 Unified Wire For Linux Page 277...
  • Page 278 XX. Unified Wire Manager (UM) Security Certificate prompt in Mozilla Firefox Figure 7.2 (b) - Security Certificate prompt in Apple Safari Figure 7.2 (c) - Security Certificate prompt in Google Chrome Figure 7.2 (d) - Chelsio T5/T4 Unified Wire For Linux Page 278...
  • Page 279 Figure 7.2 (e) - Web GUI Login page Not performing any operation/action for 5 minutes will result in session timeout. Note You will have to re-login and connect to the Agents again. Chelsio T5/T4 Unified Wire For Linux Page 279...

  • Page 280: Layout And Navigation

    Navigation Pane. The panels can be expanded and collapsed by clicking on the panel heading.  The Bottom bar has the About link on the right and copyright details on the left. Figure 7.3 – Web Management Interface Chelsio T5/T4 Unified Wire For Linux Page 280...

  • Page 281: Home Page

    The passwords are encrypted, but it is still advisable to store passwords only if the system you are running the GUI client on, is secure. Figure 7.4.1 (a) - Bookmarks and history module Chelsio T5/T4 Unified Wire For Linux Page 281...
  • Page 282 Once removed, the system will no longer appear in the Bookmarks and History Note module. If you wish to manage that system again, you will have to use the Add system option. Chelsio T5/T4 Unified Wire For Linux Page 282...
  • Page 283 The Add Agents button adds the selected system to the list of discovered agents in the Bookmarks and History module. The Clear Agents button resets the list of discovered agents. Figure 7.4.1 (b) - Service Discovery module Chelsio T5/T4 Unified Wire For Linux Page 283...
  • Page 284  Installing Driver 1. In the Choose the card fields, select T3 or T4/T5 depending on the chip revision of the network card. 2. Select the operating system for which drivers are to be installed in the Choose the OS Type field.
  • Page 285 If you deselect the ‘Remember Password’ option, you will be asked to enter password every time you try to connect to the system. Figure 7.4.2 - Adding a UM Agent Chelsio T5/T4 Unified Wire For Linux Page 285...
  • Page 286 Recipients field separated a comma. Enter Mail server details and ensure that the “Enable email Alerts” field is enabled. Select the Agent(s), for which you want to receive alerts and Click on Save. Chelsio T5/T4 Unified Wire For Linux Page 286...
  • Page 287 Various configurable parameters have been categorized into several modules like Boot Configuration module to install and erase option ROM, Network Configuration module to set MTU and VLAN ID, Card Configuration module to change driver parameters, etc. Chelsio T5/T4 Unified Wire For Linux Page 287...
  • Page 288 SNO field and finally click Delete Member(s). Figure 7.4.6 (a) – Creating a group Chelsio T5/T4 Unified Wire For Linux Page 288...
  • Page 289 Using this module, you can install option ROM or erase option ROM on Chelsio network devices. The Set Default Boot Settings button will reset the adapter to factory boot settings. Figure 7.4.6 (c) – Boot Configuration module Chelsio T5/T4 Unified Wire For Linux Page 289...
  • Page 290 MTU can be set between 1500-9000 bytes. VLAN id can be set for an adapter within the range 0-4094 (enter 0 to disable it). Figure 7.4.6 (d) – Network Configuration module Chelsio T5/T4 Unified Wire For Linux Page 290...
  • Page 291 The Card Configuration module allows you to set various adapter settings including TCP Offload. Offload settings are only available when using the TOE capable drivers (t4_tom and toecore for T5 and T4 adapters; t3_tom and toecore for T3 adapters). Figure 7.4.6 (e) – Card Configuration module...
  • Page 292 In the Port Configuration module, you can set various port settings like enabling Tx checksum and TCP segmentation offload, setting Link speed and link duplex mode, etc. The settings depend on the device driver installed. Figure 7.4.6 (f) - Port Configuration module Chelsio T5/T4 Unified Wire For Linux Page 292...
  • Page 293 Use the Bypass Configuration module to configure Chelsio’s bypass adapters like B420-SR and B404-BT. For more information on different bypass modes and configurational parameters, Bypass Driver chapter. Figure 7.4.6 (g) - Bypass Configuration module Chelsio T5/T4 Unified Wire For Linux Page 293...

  • Page 294: System Page

    The system page contains generic system and support modules which are discussed below: 7.5.1. System Summary This module lists the system Hostname, Operating System, platform and also gives the count of the Chelsio cards found. Figure 7.5.1 - System Summary module Chelsio T5/T4 Unified Wire For Linux Page 294...
  • Page 295 Using this module, one can install various Chelsio drivers for different operating systems. You can choose the configuration file type (Linux Agents only). Figure 7.5.2 (a) - Drivers Installation module connected to Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 295...
  • Page 296 XX. Unified Wire Manager (UM) Figure 7.5.2 (b) - Drivers Installation module connected to FreeBSD Agent Chelsio T5/T4 Unified Wire For Linux Page 296...
  • Page 297 To reload a driver select Reload. Finally click Load/Unload Driver button. Click Refresh if changes are not reflected immediately. To reject the load/unload option selected, click Discard Changes. Figure 7.5.3 - Driver Details module Chelsio T5/T4 Unified Wire For Linux Page 297...
  • Page 298 (type of test; you can run more than one test at a time) and click Run Test. After the tests are completed, the results will be displayed in a tabular format. Figure 7.5.4 - System Diagnostics module for a T4 CNA Chelsio T5/T4 Unified Wire For Linux Page 298...
  • Page 299 KVM modules (kvm and kvm_intel/kvm_amd) are loaded. If not done already, access Agent CLI and follow these steps to do so: If any or all the aforementioned drivers are already loaded, unload them before proceeding: [root@host]# rmmod <kvm_intel/kvm_amd> [root@host]# rmmod kvm [root@host]# rmmod cxgb4 Chelsio T5/T4 Unified Wire For Linux Page 299...
  • Page 300 The VM Configurations module allows you to view UUID and domain state of Virtual Machines. You can perform various system power options like start or resume (if VM is paused), turn off, restart or suspend (pause) a VM. Chelsio T5/T4 Unified Wire For Linux Page 300...
  • Page 301 Now, click on any of the system power actions provided at the bottom. Figure 7.5.7 (a) – VM Configurations module Figure 7.5.7 (b) – VM Configurations module: viewing properties for multiple VMs Chelsio T5/T4 Unified Wire For Linux Page 301...
  • Page 302 The VF Configurations module lists all the VMs, Virtual Functions mapped to each Virtual Machine and all the available VFs. You can also add and remove VFs for a particular VM. Figure 7.5.8 - VF Configurations module Chelsio T5/T4 Unified Wire For Linux Page 302...
  • Page 303 Now, click on any of the system power actions provided at the bottom. Figure 7.5.9 (a) – Xen Configurations module Figure 7.5.9 (b) – Xen Configurations module: viewing properties for multiple VMs Chelsio T5/T4 Unified Wire For Linux Page 303...
  • Page 304 Virtual Functions (VFs) you want to create for the physical ports. If VFs are not listed, unload the driver, reload it and create VFs again. Chelsio T5/T4 Unified Wire For Linux Page 304...
  • Page 305 Use the Get Logs button to retrieve, and Hide Logs button to clear the log entries. The Delete Logs button will remove the logs permanently from the agent. Figure 7.5.11 (a) - Managed System Application Logs module for Windows Agent Chelsio T5/T4 Unified Wire For Linux Page 305...

  • Page 306: Network Page

    7.6.1. Network summary The Network Summary module provides the total number of Chelsio adapters present, including the number of T5, T4 and T3 adapters. It also provides the total number of Network interfaces including corporate and Chelsio interfaces and VLANs.
  • Page 307 XX. Unified Wire Manager (UM) Figure 7.6.2 (a) - Card Summary module Chelsio T5/T4 Unified Wire For Linux Page 307...
  • Page 308 TOE capable drivers (t3_tom and toecore for T3 adapters; t4_tom and toecore for T4 and T5 adapters). On changing the settings, the changed settings may not reflect immediately on refreshing the data. Highlight the system item in the tree menu on the left, and click Refresh, to refresh data from the system, in case the updated settings are not being shown.
  • Page 309 Figure 7.6.2 (c) - TCP Offload Settings module for a FreeBSD Agent Device Driver settings (Windows) The device driver settings applicable to the card are shown here. For Chelsio T5 and T4 adapters, only the MaxVMQueues field will be displayed. On changing the settings, the changed settings may not reflect immediately on refreshing the data.
  • Page 310 TCP/IP offload capability. The statistics are for TCP and IP protocol processing done in the card's hardware. These statistics may only be applicable if the card is TOE enabled. Figure 7.6.2 (e) - Card Statistics module for a T3 HBA Chelsio T5/T4 Unified Wire For Linux Page 310...
  • Page 311 The modules available on this page are as below: Port summary Port details such as the Ethernet adapter nam, link status, etc are shown in this module. Figure 7.6.2.1 (a) - Port Summary of T5/T4 CNA on Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 311...
  • Page 312 Port settings such as MTU, Link speed and others can be set in this module. The settings depend on the device driver installed. Figure 7.6.2.1 (b) - Port Settings of T4/T5 CNA on Linux Agent Chelsio T5/T4 Unified Wire For Linux...
  • Page 313 Port statistics Ethernet statistics and additional hardware statistics for the port are displayed in this module. Figure 7.6.2.1 (c) - Port Statistics of T4/T5 CNA on Linux Agent 7.6.3. Networking Management page The system networking and teaming / bonding configurations are shown on this page. IP addresses, MTU, VLAN Ids, DNS and default gateway settings can be viewed and modified here.
  • Page 314 0-4094 (enter 0 to disable it). You can use the View/Set IP addresses option to add, modify or delete IP aliases. Figure 7.6.3 (a) - System network configuration module Chelsio T5/T4 Unified Wire For Linux Page 314...
  • Page 315 XX. Unified Wire Manager (UM) Figure 7.6.3 (b) – Managing IP aliases Chelsio T5/T4 Unified Wire For Linux Page 315...
  • Page 316 Table view in the Advanced NIC characteristics. The polling time field sets the average time (in seconds) based on which the table/graph updates the report. Figure 7.6.3 (c) - System network statistics module Chelsio T5/T4 Unified Wire For Linux Page 316...
  • Page 317 XX. Unified Wire Manager (UM) Figure 7.6.3 (d) - Network Throughput Vs Time instant Graph Figure 7.6.3 (e) - Network Throughput Vs Time instant Table Chelsio T5/T4 Unified Wire For Linux Page 317...
  • Page 318 Windows, you may set multiple default gateways. Use the option to set multiple default gateways with caution, since it may cause the system to stop communicating with external networks. Figure 7.6.3 (f) - Default gateway and DNS configuration module for a FreeBSD Agent Chelsio T5/T4 Unified Wire For Linux Page 318...
  • Page 319 Chelsio cards. Do not mix third party cards and offload-enabled Chelsio cards in a single team. Figure 7.6.3 (g) - Create a network team/bond device module Chelsio T5/T4 Unified Wire For Linux Page 319...
  • Page 320 Use the GetConnections utility to view currently active TCP/UDP connections. Offload status for each connection is also displayed if protocol offload hardware is available. This is useful for troubleshooting any connectivity issues for clients to various services. Figure 7.6.3 (h) - Ping Utility Chelsio T5/T4 Unified Wire For Linux Page 320...
  • Page 321 XX. Unified Wire Manager (UM) Figure 7.6.3 (i) - TraceRoute Utility Chelsio T5/T4 Unified Wire For Linux Page 321...
  • Page 322 To delete a bridge, click on the bridge name and then click “Delete Bridge”. This will also delete the virtual interface attached to that bridge and the VM(s) to which the interface was assigned. Chelsio T5/T4 Unified Wire For Linux Page 322...
  • Page 323 To delete a bridge, click on the bridge name and then click “Delete Bridge”. If there are virtual network interfaces attached, you will have to detach them first. Chelsio T5/T4 Unified Wire For Linux Page 323...
  • Page 324 To assign interfaces on the host to a bridge, specify the bridge name and the host interface in the “Add Interface to Bridge” section. Click “Add”. You can add multiple interfaces to the same bridge using the aforementioned method. The Cost and Priority parameters are optional. Chelsio T5/T4 Unified Wire For Linux Page 324...
  • Page 325 XX. Unified Wire Manager (UM) Figure 7.6.3.1 (c) – Creating Bridge Figure 7.6.3.1 (d) – Adding Bridge to VM Figure 7.6.3.1 (e) – Adding interfafe to Bridge Chelsio T5/T4 Unified Wire For Linux Page 325...
  • Page 326 You can attach upto 64 virtual interfaces to a particular bridge. Device Number, Bridge Name and VM Name are the mandatory fields while creating a virtual interface. Figure 7.6.3.1 (f) – Creating Bridge Figure 7.6.3.1 (g) – Creating and attaching virtual interface Chelsio T5/T4 Unified Wire For Linux Page 326...
  • Page 327 This type of virtual network does is not bound to any physical network adapter and no access to external networks is provided. Chelsio T5/T4 Unified Wire For Linux Page 327...
  • Page 328 Once created, you can manage the virtual networks in the Virtual Network Manager module. Figure 7.6.3.1 (i) – Creating external virtual network Figure 7.6.3.1 (j) – Creating internal virtual network Figure 7.6.3.1 (k) – Creating private virtual network Chelsio T5/T4 Unified Wire For Linux Page 328...
  • Page 329 These settings can be set only when iWARP driver (iw_cxgb4 for T4 and T5; iw_cxgb3 for T3) is loaded. If you set any parameter for a T5 adapter, it applies for all the T5 adapters present. Same applies for T4 and T3 adapters.
  • Page 330 XX. Unified Wire Manager (UM) Figure 7.6.4 (a) – iWARP settings for T4 CNA for FreeBSD Agent Chelsio T5/T4 Unified Wire For Linux Page 330...
  • Page 331 XX. Unified Wire Manager (UM) Figure 7.6.4 (b) – iWARP settings for T4 CNA for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 331...
  • Page 332 The WD-UDP module lists the process ids (pid) of UDP traffic running on the agent and displays the corresponding statistics and attributes. Please ensure that WD-UDP traffic is running on the agent before accessing this Note module. Figure 7.6.5 (a) – WD-UDP Process Statistics Chelsio T5/T4 Unified Wire For Linux Page 332...
  • Page 333 XX. Unified Wire Manager (UM) Figure 7.6.5 (b) – WD-UDP Process Attributes Chelsio T5/T4 Unified Wire For Linux Page 333...
  • Page 334 Please ensure that WD-TOE traffic is running on the agent before accessing this Note module. Figure 7.6.5 (c) – WD-TOE Process Statistics Figure 7.6.5 (d) – WD-TOE Process Attributes Chelsio T5/T4 Unified Wire For Linux Page 334...

  • Page 335: Storage Page

    This module provides details about the driver installed; such as driver name and its version. The module also gives information about the number of FCoE enabled cards that are present on the machine. Figure 7.7.1 (a) – FCoE Initiator Summary module for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 335...
  • Page 336 Figure 7.7.1.1 (a) – FCoE Card Summary module FCoE Attributes Information such as Interrupt modes (MSI/MSI-X/INTx), SCSI mode and the card state are provided in this module. Figure 7.7.1.1 (b) – FCoE Attributes module Chelsio T5/T4 Unified Wire For Linux Page 336...
  • Page 337 FCoE Port Summary The SCSI adapter name and the underlying ENODE MAC address of the physical port can be found here. Figure 7.7.1.2 (a) – FCoE Port Summary module for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 337...
  • Page 338 (10G/1G) being used. Note that only class 3 service is supported by the initiator for now and the frame size is fixed to 2128 bytes as per spec. Figure 7.7.1.2 (b) – FCoE Port Attributes module for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 338...
  • Page 339 To delete a virtual port, select the option Delete and select the virtual port WWPN which you want to delete and click on delete. Figure 7.7.1.2 (c) – FCoE NPIV management module Chelsio T5/T4 Unified Wire For Linux Page 339...
  • Page 340 This module provides details about the discovered target such as target’s FC ID, WWPN and WWNN so that the user can identify the discovered target accordingly. Figure 7.7.1.3 (a) – FCoE Remote Port Attributes module Chelsio T5/T4 Unified Wire For Linux Page 340...
  • Page 341 For Linux, the SCSI address is displayed in H:C:T:L (Host:Channel:Target:Lun) format and for Windows, it is displayed in P:B:T:L(SCSI Port:Bus:Target:Lun) format. Figure 7.7.1.3 (b) – FCoE Remote Port Lun Details module Chelsio T5/T4 Unified Wire For Linux Page 341...
  • Page 342 FCoE Virtual Port Summary The SCSI adapter name and the underlying ENODE MAC address of the physical port can be found here. Figure 7.7.1.4 (a) – FCoE Virtual Port Summary module for Linux Agent Chelsio T5/T4 Unified Wire For Linux Page 342...
  • Page 343 (10G/1G) being used. Note that only class 3 service is supported by the initiator for now and the frame size is fixed to 2128 bytes as per spec. Figure 7.7.1.4 (b) – FCoE Virtual Port Attributes module Chelsio T5/T4 Unified Wire For Linux Page 343...
  • Page 344 LUN, SCSI address, and LUN address are provided. For Linux, the SCSI address is displayed in H:C:T:L (Host:Channel:Target:Lun) format and for Windows, it is displayed in P:B:T:L(SCSI Port:Bus:Target:Lun) format. Figure 7.7.1.4 (d) - FCoE Remote Port Lun Details module Chelsio T5/T4 Unified Wire For Linux Page 344...
  • Page 345 7.7.2. iSCSI initiator (Linux, Windows) All supported iSCSI initiators can be managed from this page. The supported initiators on Windows are Microsoft and Chelsio iSCSI initiator (T5/T4 adapters). On Linux, Open iSCSI initiator is supported. The modules available on this page are: Initiator nodes This module lists the initiator nodes / virtual adapters configured in the initiator stack.
  • Page 346 XX. Unified Wire Manager (UM) Figure 7.7.2 (b) - Microsoft iSCSI initiator Chelsio T5/T4 Unified Wire For Linux Page 346...
  • Page 347 XX. Unified Wire Manager (UM) Figure 7.7.2 (c) - Chelsio iSCSI initiator Chelsio T5/T4 Unified Wire For Linux Page 347...
  • Page 348 (the IP address and TCP Port specified when discovering the target). If there are any pre-existing iSCSI sessions established to the target, deletion of Note the target Portal from the discovered targets list will fail. Figure 7.7.2 (d) - Discover targets module Chelsio T5/T4 Unified Wire For Linux Page 348...
  • Page 349 XX. Unified Wire Manager (UM) Figure 7.7.2 (e) - Discover targets module Chelsio T5/T4 Unified Wire For Linux Page 349...
  • Page 350 In Windows, you can specify the initiator HBA to use and its IP address while logging in. Figure 7.7.2 (f) - Targets module after logging in Chelsio T5/T4 Unified Wire For Linux Page 350...
  • Page 351 You can also view details like link id, status, enode mac, etc of all the ports of the selected adapter. Figure 7.7.3 (a) - Full Offload iSCSI Hardware Information module Chelsio T5/T4 Unified Wire For Linux Page 351...
  • Page 352 The Port Up and Port Down buttons will enable and disable the selected port respectively. The Clear IP button deletes values set for the IP Type, IP, Subnet Mask and Gateway properties and resets them. Figure 7.7.3 (b) - FO iSCSI Manage Ports module Chelsio T5/T4 Unified Wire For Linux Page 352...
  • Page 353 In the FO iSCSI Initiator Properties module, you can configure FO iSCSI Initiator by setting different properties like enabling/disabling CHAP authentication, setting Header and Data digest, etc. Figure 7.7.3 (c) - FO iSCSI Initiator Properties Chelsio T5/T4 Unified Wire For Linux Page 353...
  • Page 354 In this module, you can set upto 8 instances. Configurable parameters include initiator node name (IQN), alias (friendly) name, Initiator (CHAP) Username and password. Figure 7.7.3 (d) - FO iSCSI Manage Instances module Chelsio T5/T4 Unified Wire For Linux Page 354...
  • Page 355 (target) IP. Finally, click Discover. After successful discovery, all the discovered targets will appear in the Discovered Targets section. To view more details, click on the Target name. Figure 7.7.3 (e) - FO iSCSI Discover Details module Chelsio T5/T4 Unified Wire For Linux Page 355...
  • Page 356 CHAP for a secure iSCSI connection. One-way (target authenticates the initiator) and Mutual (target and initiator authenticate each other) authentication methods are supported. Figure 7.7.3 (f) - FO iSCSI Session Details module: Login Chelsio T5/T4 Unified Wire For Linux Page 356...
  • Page 357 Established sessions section. Select the Adapter and session id. Details of the selected session will be displayed. To end the session, click Logout. Figure 7.7.3 (g) - FO iSCSI Session Details module: Established Sessions Chelsio T5/T4 Unified Wire For Linux Page 357...
  • Page 358 Properties such as Target name and Alias, Max Data Receive Length, Authentication mode related to a specific iSCSI target can be viewed and modified here. iSCSI targets can be started/stopped or deleted. Chelsio T5/T4 Unified Wire For Linux Page 358...
  • Page 359 XX. Unified Wire Manager (UM) Figure 7.7.4 (b) - Target properties module Chelsio T5/T4 Unified Wire For Linux Page 359...
  • Page 360 XX. Unified Wire Manager (UM) Session details Details including Session ID, Initiator IQN and Connections List of all discovered and currently connected iSCSI targets are listed here. Figure 7.7.4 (c) - Session Details module Chelsio T5/T4 Unified Wire For Linux Page 360...
  • Page 361 XX. Unified Wire Manager (UM) New Target Creation New iSCSI target can be created here by specifying the Target IQN and Target Alias name. Figure 7.7.4 (d) - New Target Creation module Chelsio T5/T4 Unified Wire For Linux Page 361...
  • Page 362 View/Edit iSCSI Target LUNs This module displays various Logical Units created in an iSCSI Target. Selected LUNs can be deleted. Figure 7.7.5 (a) - View/Edit iSCSI Target LUNs module Chelsio T5/T4 Unified Wire For Linux Page 362...
  • Page 363 Device and RAM Disk Size etc. RW (Read-Write) and RO (Read Only) are the two kinds of permissions that can be set. If Ram Disk is selected, then a minimum of 16 MB should be provided. Figure 7.7.5 (b) - Adding a new LUN Chelsio T5/T4 Unified Wire For Linux Page 363...
  • Page 364 Portal List on the left displays details of the portal group on which an iSCSI target is listening and the related info is displayed on the right under Portal Details. Selected portals can be deleted. Figure 7.7.6 (a) - View/Edit iSCSI Target Portals module Chelsio T5/T4 Unified Wire For Linux Page 364...
  • Page 365 XX. Unified Wire Manager (UM) Add Portal New Portals can be added here by choosing the specific target and Portal IP address. The Port number should be 3260. Figure 7.7.6 (b) - Adding a new Portal Chelsio T5/T4 Unified Wire For Linux Page 365...
  • Page 366 View/Edit iSCSI Target ACLs This module displays details for all the ACLs configured for an iSCSI Target. Selected ACLs can be deleted. Figure 7.7.7 (a) - Target ACL operations module Chelsio T5/T4 Unified Wire For Linux Page 366...
  • Page 367 XX. Unified Wire Manager (UM) Add ACL New ACLs can be configured by specifying Target name, initiator IQN name, IP address and permission type. Figure 7.7.7 (b) - Adding new ACL Chelsio T5/T4 Unified Wire For Linux Page 367...

  • Page 368: Hardware Features

    Protocol field are tcp, udp and frag. The Priority field can be used to determine the priority of a filter when Vlan ids are same. Insert at position features allows user to add a filter at a specified position. Chelsio T5/T4 Unified Wire For Linux Page 368...
  • Page 369 You can create filter rules for any combination in the list. However, filter rule created only for the Active Combination will apply. To create a new rule, select the adapter type (T5 or T4) in the Select a chip type drop-down, select a combination and click “Add a Filter row”. The FILTERID and T5/T4 CARD fields are mandatory.
  • Page 370 7.8.2. Traffic Management (Linux) Using this page, you can add/delete/modify offload policies only in the presence of offload driver (t3_tom for T3 adapters; t4_tom for T5 and T4 adapters). Traffic Management configuration The Chelsio Card section on the left displays all the cards available in the server and their corresponding policies on the right.
  • Page 371 7.8.3. Boot T4/T5 Save Config File (Linux) This module displays the current T5/T4 configuration tuning option selected. You can also change the tuning option by selecting the config file for each option located in /ChelsioUwire- x.xx.x.x/src/network/firmware. For instance, to select Low latency Networking for T4 adapter, locate the file, t4-config.txt, in /ChelsioUwire-x.xx.x.x/src/network/firmware/low_latency_config...
  • Page 372 Figure 7.8.3 (a) – T4/T5 Save Config File module T5/T4 Boot Option ROM management This module allows managing the PXE and FCoE boot capability for Chelsio T5 and T4 adapters. The Option ROM (PXE and FCoE) may be installed to or erased from the card. The version of Option ROM flashed can be viewed here.
  • Page 373 Using the Boot Devices option you can set various iSCSI target properties. Enable Option ROM only if you are planning to boot the system via PXE or Note install the operating system on discovered iSCSI or FCoE LUN. Chelsio T5/T4 Unified Wire For Linux Page 373...
  • Page 374 XX. Unified Wire Manager (UM) Figure 7.8.3 (c) - PXE Boot configuration for T4 CNAs Chelsio T5/T4 Unified Wire For Linux Page 374...
  • Page 375 XX. Unified Wire Manager (UM) Figure 7.8.3 (d) - FCoE Boot configuration for T4 CNAs: Function parameter Figure 7.8.3 (e) - FCoE Boot configuration for T4 CNAs: Boot parameter Chelsio T5/T4 Unified Wire For Linux Page 375...
  • Page 376 XX. Unified Wire Manager (UM) Figure 7.8.3 (f) - FCoE Boot configuration for T4 CNAs: Show WWPN parameter Figure 7.8.3 (g) - iSCSI Boot configuration for T4 CNAs: Function parameter Chelsio T5/T4 Unified Wire For Linux Page 376...
  • Page 377 XX. Unified Wire Manager (UM) Figure 7.8.3 (h) - iSCSI Boot configuration for T4 CNAs: Initiator parameter Chelsio T5/T4 Unified Wire For Linux Page 377...
  • Page 378 XX. Unified Wire Manager (UM) Figure 7.8.3 (i) - iSCSI Boot configuration for T4 CNAs: Network parameter Chelsio T5/T4 Unified Wire For Linux Page 378...
  • Page 379 Current bypass mode. The Watchdog timer is used to ensure that if there is a software failure, the switch will enter the default state. The Watchdog timeout value should be provided in milliseconds. Chelsio T5/T4 Unified Wire For Linux Page 379...
  • Page 380 Purge table: Delete all the rules present in the selected table. This action will not delete the table.  Activate table: Enable the selected table.  Deactivate table: Disable the selected table. Chelsio T5/T4 Unified Wire For Linux Page 380...
  • Page 381 To delete a rule, select the corresponding checkbox and click Delete a Filter. Finally, click Save Changes. You can delete multiple rules using this method. For more information on Redirection, see Bypass Driver chapter. Figure 7.8.4 (b) - Redirect Configuration module: Table configuration tab Chelsio T5/T4 Unified Wire For Linux Page 381...
  • Page 382 XX. Unified Wire Manager (UM) Figure 7.8.4 (c) - Redirect Configuration module: Rules configuration tab Chelsio T5/T4 Unified Wire For Linux Page 382...
  • Page 383 XX. Unified Wire Manager (UM) 7.8.5. T4 Egress Class Schedulers Schedulers can be set only when T5/T4 network driver (cxgb4) is loaded. Egress Queue Map Using this module, you can bind (map) NIC (non-offloaded) Tx queues to Tx Scheduler classes.

  • Page 384: Uninstalling Unified Wire Manager

    1. Use the following query command to determine the name of the client RPM: [root@host]# rpm –qa | grep chelsio-uwire_mgmt-client 2. Now, execute the following command with the result from the above query to uninstall Management Client: E.g. for RHEL 6.3: [root@host]# rpm –e chelsio-uwire_mgmt-client-rhel6u3-2.2-xyz.x86_64 Chelsio T5/T4 Unified Wire For Linux Page 384...

  • Page 385: Uninstalling Management Station

    1. Use the following query command to determine the name of the Management Station RPM: [root@host]# rpm –qa | grep chelsio-uwire_mgmt-station 2. Now, execute the following command with the result from the above query to uninstall Management Station: E.g. for RHEL 6.3: [root@host]# rpm –e chelsio-uwire_mgmt-station-rhel6u3-2.2-xyz.x86_64 Chelsio T5/T4 Unified Wire For Linux Page 385...

  • Page 386: Unified Boot

    Chapter XXI. Unified Boot XXI. Unified Boot Chelsio T5/T4 Unified Wire For Linux Page 386...

  • Page 387: Introduction

    * If system BIOS version is lower than 1.5 and both Legacy and uEFI are enabled, please upgrade to 1.5 or higher. Otherwise the system will hang during POST. 1.1.2. Supported Switches Following is the list of network switches supported by Chelsio Unified Boot software: Chelsio T5/T4 Unified Wire For Linux Page 387...

  • Page 388: Software Requirements

    The Chelsio Driver Update Disk driver has been developed to run on 64-bit Linux platforms. Following is the list of Drivers/Software and supported Linux distributions: Linux Distribution Driver/Software (DUDs) RHEL6.5, 2.6.32-431.el6 PXE,FCoE,iSCSI SLES11SP3, 3.0.76-0.11 PXE,FCoE,iSCSI Chelsio T5/T4 Unified Wire For Linux Page 388...

  • Page 389: Flashing Firmware And Option Rom

    Make the USB flash drive as the primary boot device. Save the changes and exit. ix. Once the system boots from the USB flash drive, change your working directory to CHELSIO directory: C:\>cd CHELSIO Chelsio T5/T4 Unified Wire For Linux Page 389...
  • Page 390 CNA index found in step x (0 in this case) xii. Delete any previous firmware using the following command: C:\CHELSIO>cfut4 -d <idx> -xh –xf Here, is the CNA index found in step x (0 in this case) Chelsio T5/T4 Unified Wire For Linux Page 390...
  • Page 391 Chapter XXI. Unified Boot xiii. Run the following command to flash the appropriate firmware (t5fw-x.xx.xx.x.bin for T5 adapters; t4fw-x.xx.xx.x.bin for T4 adapters). C:\CHELSIO>cfut4 -d <idx> -uf <firmware_file>.bin firmware_file Here, is the CNA index found in step x (0 in this case) and is the firmware image file present in the CHELSIO folder.
  • Page 392 Delete any previous Option ROM settings: C:\CHELSIO>cfut4 -d <idx> -xc Here, is the CNA index found in step x (0 in this case) xvi. Reboot the system for changes to take effect. Chelsio T5/T4 Unified Wire For Linux Page 392...

  • Page 393: Using Cxgbtool

    Flash Option ROM using the following command: [root@host]# cxgbtool ethX loadboot /lib/firmware/cxgb4/cubt4.bin vi. Flash appropriate firmware using the following command (t4fw-x.xx.xx.x.bin for T4 adapters;t5fw-x.xx.xx.x.bin for T5 adapters): [root@host]# cxgbtool ethX loadfw <firmware_file>.bin Where, ethX is the Chelsio interface. vii. Reboot for changes to take effect.

  • Page 394: Configuring Pxe Server

    (Use port # 67, set PXE option and provide bootable file name in settings)  http://unattended.sourceforge.net/pxe-win2k.html Chelsio Communications does not take any responsibility regarding contents Note given in above mentioned links. Those are given for example purposes only. Chelsio T5/T4 Unified Wire For Linux...

  • Page 395: Pxe Boot Process

    Press [Alt+C] when the message Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2014 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 396 Enable the Adapter BIOS using arrow keys if not already enabled. Hit [ENTER]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose PXE from the list to configure. Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 396...
  • Page 397 Please note that enabling NIC Func 00 will enable port 0 for PXE, enabling NIC Func 01 will enable port 1 and so on for NIC function. Please refer the table below: Chelsio T5/T4 Unified Wire For Linux Page 397...
  • Page 398 Hit [F2] or [DEL] or any other key as mentioned during system startup to enter the system setup. xiii. Allow the Chelsio option ROM to initialize and setup PXE devices. DO NOT PRESS ALT-S to skip Chelsio option ROM. Chelsio T5/T4 Unified Wire For Linux Page 398...
  • Page 399 In the system setup, choose any of the Chelsio PXE devices as the first boot device. xv. Reboot. DO NOT PRESS ALT-S to skip Chelsio option ROM, during POST. xvi. Hit [F12] key when prompted to start PXE boot. Chelsio T5/T4 Unified Wire For Linux Page 399...

  • Page 400: Uefi Pxe Boot

    After configuring the PXE server, make sure the PXE server works. Then reboot the client machine. Configure the system having Chelsio CNA to boot in UEFI mode. For Supermicro systems, enable Network Stack as well before proceeding. Note iii. Boot to EFI Shell. Chelsio T5/T4 Unified Wire For Linux Page 400...
  • Page 401 Chapter XXI. Unified Boot drivers iv. Issue command to note down the handle# of Chelsio driver. Then issue command drvcfg -s <handle#> to invoke the configuration utility. Chelsio T5/T4 Unified Wire For Linux Page 401...
  • Page 402 Hit [Enter]. vii. Choose config utility if you want to configure Chelsio adapter for PXE boot, else choose flash utility if you want to flash/erase firmware, option ROM image or configuration parameters. Chelsio T5/T4 Unified Wire For Linux Page 402...
  • Page 403 If config utility is chosen, go to step (ix), else goto step (xvii). ix. Enable the Adapter BIOS if not already enabled. Hit [Enter]. Choose PXE as the function to use and configure. Chelsio T5/T4 Unified Wire For Linux Page 403...
  • Page 404 Please note that enabling NIC Func 00 will enable port 0 for PXE, enabling NIC Func 01 will enable port 1 and so on for NIC function. Please refer the table below: Chelsio T5/T4 Unified Wire For Linux Page 404...
  • Page 405 NIC Func00 NIC Func01 NIC Func02 NIC Func03 xiv. Hit [Esc] and then [Y] to save the configuration. xv. Reboot the system and choose any of the Chelsio PXE devices for PXE boot. Chelsio T5/T4 Unified Wire For Linux Page 405...
  • Page 406 To confirm if uEFI Chelsio PXE has loaded successfully, please boot to uEFI shell and drivers issue the command . Now, note that the type field should be B as shown below. Chelsio T5/T4 Unified Wire For Linux Page 406...
  • Page 407 Chapter XXI. Unified Boot xvii. If flash utility is chosen, choose any of available options and Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 407...
  • Page 408 If you have chosen to update option ROM image, then enter the full path to the file. E.g.:- If the pen drive is device fs0 and the option ROM image file is in fs0:\CHELSIO\cubt4.bin, CHELSIO\cubt4.bin then enter xix. Reboot the system for changes to take effect. Chelsio T5/T4 Unified Wire For Linux Page 408...

  • Page 409: Fcoe Boot Process

    Reboot the system. Press [Alt+C] when the message Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2014 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 410 Enable the Adapter BIOS if not already enabled. Hit [ENTER]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose FCoE from the list to configure and hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 410...
  • Page 411 Chapter XXI. Unified Boot vii. Choose the first option (function parameters) from the list of parameter type and hit [Enter]. viii. Enable FCoE BIOS if not already enabled. Chelsio T5/T4 Unified Wire For Linux Page 411...
  • Page 412 Chapter XXI. Unified Boot ix. Choose the order of the ports to discover FCoE targets. Set discovery timeout to a suitable value. Recommended value is >= 30. Chelsio T5/T4 Unified Wire For Linux Page 412...
  • Page 413 Chapter XXI. Unified Boot xi. Hit [F10] or [Esc] and then [Y] to save the configuration. xii. Choose Configure boot parameters. Chelsio T5/T4 Unified Wire For Linux Page 413...
  • Page 414 Select the first boot device and hit [Enter] to discover FC/FCoE targets connected to the switch. Wait till all reachable targets are discovered. xiv. List of discovered targets will be displayed. Highlight a target using the arrow keys and hit [Enter] to select. Chelsio T5/T4 Unified Wire For Linux Page 414...
  • Page 415 Chapter XXI. Unified Boot xv. From the list of LUNs displayed for the selected target, choose one on which operating system has to be installed. Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 415...
  • Page 416 Chapter XXI. Unified Boot xvi. Hit [F10] or [Esc] and then [Y] to save the configuration. xvii. Reboot the machine. xviii. During POST, allow the Chelsio option ROM to discover FCoE targets. Chelsio T5/T4 Unified Wire For Linux Page 416...
  • Page 417 Enter BIOS setup and choose FCoE disk discovered via Chelsio adapter as the first boot device. xx. Reboot and boot from the FCoE disk or install the required OS using PXE. Chelsio T5/T4 Unified Wire For Linux Page 417...

  • Page 418: Iscsi Boot Process

    Reboot the system. Press [Alt+C] when the message Chelsio Unified Boot BIOS vX.X.X.XX, Copyright (C) 2003-2014 Chelsio Communications Press <Alt-C> to Configure T4/T5 Card(s). Press <Alt- S> to skip BIOS appears on the screen to enter the configuration utility. iii. The configuration utility will appear as below: iv.
  • Page 419 Enable the Adapter BIOS if not already enabled. Hit [Enter]. Use the default values for Boot Mode, EDD and EBDA Relocation parameters, Note unless instructed otherwise. vi. Choose iSCSI from the list to configure and hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 419...
  • Page 420 For Windows, select IBFT, since Microsoft iSCSI Initiator will be used. You can also configure the number of iSCSI login attempts (retries) incase the network is unreachable or slow. Chelsio T5/T4 Unified Wire For Linux Page 420...
  • Page 421 Chapter XXI. Unified Boot ix. Choose the order of the ports to discover iSCSI targets. Set discovery timeout to a suitable value. Recommended value is >= 30. Chelsio T5/T4 Unified Wire For Linux Page 421...
  • Page 422 Chapter XXI. Unified Boot xi. Hit [Esc] and then [Y] to save the configuration. xii. Go back and choose Configure Initiator Parameters to configure initiator related properties. Chelsio T5/T4 Unified Wire For Linux Page 422...
  • Page 423 Initiator properties like IQN, Header Digest, Data Digest, etc will be displayed. Change the values appropriately or continue with the default values. Hit [F10] to save. xiv. CHAP authentication is disabled by default. To enable and configure, go back and choose Configure CHAP Parameters Chelsio T5/T4 Unified Wire For Linux Page 423...
  • Page 424 Finally, provide Initiator and Target CHAP credentials according to the authentication method selected. Hit [F10] to save. xvi. Go back and choose Configure Network Parameters to configure iSCSI Network related properties. Chelsio T5/T4 Unified Wire For Linux Page 424...
  • Page 425 Select the port using which you want to connect to the target. Hit [Enter]. xviii. Select Yes in the Enable DHCP field to configure port using DHCP or No to manually configure the port. Hit [F10] to save. Chelsio T5/T4 Unified Wire For Linux Page 425...
  • Page 426 If you want to discover target using DHCP, select Yes in the Discover Boot Target via DHCP field. To discover target via static IP, select No and provide the target IP and Hit [F10] to save. The default TCP port selected is 3260. Chelsio T5/T4 Unified Wire For Linux Page 426...
  • Page 427 Chapter XXI. Unified Boot xxi. Go back and choose Discover iSCSI Target (s) to connect to a target. xxii. Select the portal group on which iSCSI service is provided by the target Chelsio T5/T4 Unified Wire For Linux Page 427...
  • Page 428 A list of available targets will be displayed. Select the target you wish to connect to and hit [Enter]. xxiv. A list of LUNs configured on the selected target will be displayed. Select the LUN you wish to connect to and hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 428...
  • Page 429 Chapter XXI. Unified Boot xxv. Hit [Esc] and then [Y] to save the configuration. xxvi. Reboot the machine. xxvii. During POST, allow the Chelsio option ROM to discover iSCSI targets. Chelsio T5/T4 Unified Wire For Linux Page 429...
  • Page 430 Enter BIOS setup and choose iSCSI target LUN discovered via Chelsio adapter as the first boot device. xxi. Reboot and boot from the iSCSI Target LUN or install the required OS using PXE. Chelsio T5/T4 Unified Wire For Linux Page 430...

  • Page 431: Creating Driver Update Disk (Dud)

    7. Creating Driver Update Disk (DUD) The following section describes the procedure to create Driver Update Disks for RHEL and SLES distributions for T5 adapters. In case of T4 adapters, you can skip this step and use inbox drivers to install the operating system.
  • Page 432 [root@host]# mkfs.vfat /dev/sda Depending on the distribution to be installed, copy the corresponding image file to the USB stick. For example, execute the following command for SLES11sp3. [root@host]# dd if=/root/ ChelsioUwire-x.xx.x.x/Uboot/LinuxDUD/Chelsio- DriverUpdateDisk-SLES11sp3-x86_64-x.x.x.x-y.img of=/dev/sda Chelsio T5/T4 Unified Wire For Linux Page 432...

  • Page 433: Os Installation

    Network. There may be built-in Chelsio driver in these distributions. The driver may or may not work with Chelsio adapters, depending on the adapter in use, and the version of the driver that shipped in that particular distribution. Please flash the firmware provided in the package. Chelsio T5/T4 Unified Wire For Linux Page 433...
  • Page 434 Driver Update Disk during the installation. ii. Driver disk prompt: The installer will load and prompt you for the driver update disk. Now select “Yes” and hit [Enter] to proceed. Chelsio T5/T4 Unified Wire For Linux Page 434...
  • Page 435 Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. Chelsio T5/T4 Unified Wire For Linux Page 435...
  • Page 436 The installer will ask if you wish to load more drivers. Choose “Yes” to load if you have any other drivers to load. Otherwise choose “No”. vi. Choose language and Keyboard type: Select the required language from the list. Chelsio T5/T4 Unified Wire For Linux Page 436...
  • Page 437 Select the type of keyboard you have from the list. viii. Select Installation method: In this step, you can choose the source which contains the OS installation ISO image. In this case, select “NFS directory”. Chelsio T5/T4 Unified Wire For Linux Page 437...
  • Page 438 Configure TCP/IP settings: Here you can specify if you want to configure your network interfaces using DHCP or manually using IPv4. IPv6 is currently not supported. Hence disable IPv6 before proceeding. Chelsio T5/T4 Unified Wire For Linux Page 438...
  • Page 439 Type dd at the boot prompt for the installation media. The dd option specifies that you will be providing a Driver Update Disk during the installation. The DUD gets detected and the driver is automatically loaded by installer. Chelsio T5/T4 Unified Wire For Linux Page 439...
  • Page 440 Chapter XXI. Unified Boot ii. Start Installation Select “Start Installation” and then “Start Installation or Update”. Chelsio T5/T4 Unified Wire For Linux Page 440...
  • Page 441 Select method of install: Select “Network” as the source of medium to install the SLES Operating System. iv. Select the Network protocol: Select the desired Network protocol from the list presented. Chelsio T5/T4 Unified Wire For Linux Page 441...
  • Page 442 [Alt] + [F3] or [Alt] + [F4]. Press [Alt] + [F1] to get back to the list. vi. Configure DHCP IP Select “Yes” to configure the network interface selected in the previous step using DHCP. Chelsio T5/T4 Unified Wire For Linux Page 442...

  • Page 443: Installation On Fcoe Lun

    Insert the OS installation disc into your CD/DVD ROM. ii. On the Grub menu, choose Install or upgrade an existing system option if not already selected. iii. Press Tab and type at the boot prompt. Chelsio T5/T4 Unified Wire For Linux Page 443...
  • Page 444 Chapter XXI. Unified Boot iv. Driver disk prompt: The installer will load and prompt you for the driver update disk. Now select “Yes” and hit [Enter] to proceed. Chelsio T5/T4 Unified Wire For Linux Page 444...
  • Page 445 Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. Chelsio T5/T4 Unified Wire For Linux Page 445...
  • Page 446 The installer will ask if you wish to load more drivers. Choose “Yes” to load if you have any other drivers to load. Otherwise choose “No” and hit [Enter]. viii. Testing Installation media If you want to test the installation media, choose OK or else Skip. Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 446...
  • Page 447 Chapter XXI. Unified Boot ix. Graphical Installer Red Hat graphical installer will now start. Click Next. Select Specialized Storage Devices radio button and click Next. Chelsio T5/T4 Unified Wire For Linux Page 447...

  • Page 448: Installation On Iscsi Lun

    Insert the OS installation disc into your CD/DVD ROM. ii. On the Grub menu, choose Install or upgrade an existing system option if not already selected. iii. Press Tab and type at the boot prompt. Chelsio T5/T4 Unified Wire For Linux Page 448...
  • Page 449 Chapter XXI. Unified Boot iv. Driver disk prompt: The installer will load and prompt you for the driver update disk. Now select “Yes” and hit [Enter] to proceed. Chelsio T5/T4 Unified Wire For Linux Page 449...
  • Page 450 Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. Chelsio T5/T4 Unified Wire For Linux Page 450...
  • Page 451 The installer will ask if you wish to load more drivers. Choose “Yes” to load if you have any other drivers to load. Otherwise choose “No” and hit [Enter]. xii. Testing Installation media If you want to test the installation media, choose OK or else Skip. Hit [Enter]. Chelsio T5/T4 Unified Wire For Linux Page 451...
  • Page 452 Chapter XXI. Unified Boot xiii. Graphical Installer Red Hat graphical installer will now start. Click Next. Chelsio T5/T4 Unified Wire For Linux Page 452...
  • Page 453 Select Specialized Storage Devices radio button and click Next. xv. The discovered LUNs will appear in the Basic Devices tab and you can proceed with installation as usual. If not, follow the steps mentioned below to add target. Chelsio T5/T4 Unified Wire For Linux Page 453...
  • Page 454 Click +Add Advanced Target button and select the Add iSCSI target radio button. Click +Add drive button. xvii. Enter the target IP. Initiator IQN will be autogenerated by the installer. Click Start Discovery. Chelsio T5/T4 Unified Wire For Linux Page 454...
  • Page 455 Select the checkbox corresponding to the LUN on which you wish to install operating system and click Login. Make sure the same LUN discovered at the Option ROM stage is selected for Note OS installation. Chelsio T5/T4 Unified Wire For Linux Page 455...
  • Page 456 Chapter XXI. Unified Boot xx. Next, choose the CHAP authentication method and click Login. xxi. On successful login, a dialog box will display the results along with the connected LUN name. Click OK. Chelsio T5/T4 Unified Wire For Linux Page 456...
  • Page 457 Chapter XXI. Unified Boot xxii. Select the LUN under the Other SAN Devices tab, and click Next. xxiii. Proceed with the installation as usual. Chelsio T5/T4 Unified Wire For Linux Page 457...

  • Page 458: Lustre File System

    Chapter XXII. Lustre File System XXII. Lustre File System Chelsio T5/T4 Unified Wire For Linux Page 458...

  • Page 459: Introduction

    Metadata servers (MDS), which manage the names and directories in the file system. Systems in a Lustre cluster 1.1. Hardware Requirements 1.1.1. Supported Adapters Currently Lustre File System is supported on following Chelsio Adapters: T502-BT T580-CR T580-LP-CR T520-LL-CR Chelsio T5/T4 Unified Wire For Linux Page 459...

  • Page 460: Software Requirements

    Currently Lustre File System is supported on following distributions: Redhat Enterprise Linux 6 update 5 kernel (RHEL6.5), 2.6.32-431.el6* *Limited QA performed Other kernel versions have not been tested and are not guaranteed to work. Chelsio T5/T4 Unified Wire For Linux Page 460...

  • Page 461: Creating/Configuring Lustre File System

    Enable and assign IP address to Chelsio interface. Edit modprobe.conf file with appropriate interface name options lnet networks=tcp0(ethX),tcp1(ethY) //For NIC/TOE options lnet networks=o2ib0(ethX),o2ib1(ethY) //For iWARP where, ethX and ethY represent Chelsio interfaces. Chelsio T5/T4 Unified Wire For Linux Page 461...
  • Page 462 [root@host]# mkfs.lustre --ost --fsname=<fsname> --mgsnode=<NID> <block device name> On Client node, follow steps (i)-(vii). Mount the Lustre file system on the client node by running the following command: xii. [root@host]# mount -t lustre <MGS node>:/<fsname> <mount_point> Chelsio T5/T4 Unified Wire For Linux Page 462...

  • Page 463: Xxiii. Appendix A

    Chapter XXIII. Appendix XXIII. Appendix A Chelsio T5/T4 Unified Wire For Linux Page 463...

  • Page 464: Troubleshooting

    Always enable LLDP on the interfaces as FCoE link won’t come up until and unless a successful LLDP negotiation happens. priority-flow-control mode on the switch On the switch, make sure priority-flow-control mode is always set to auto and flow control is disabled. Chelsio T5/T4 Unified Wire For Linux Page 464...
  • Page 465 Ethernet interface part of both Ethernet VLAN and FCoE VLAN. Cisco nexus switch reporting “pauseRateLimitErrDisable” If in any case the switch-port on the Cisco nexus switch is reporting “pauseRateLimitErrDisable”, then perform an Ethernet port shut/no shut. Chelsio T5/T4 Unified Wire For Linux Page 465...

  • Page 466: Chelsio End-User License Agreement (Eula)

    TERMINATE INSTALLATION PROCESS. 1. License. Chelsio Communications, Inc. ("Chelsio") hereby grants you, the Licensee, and you hereby accept, a limited, non-exclusive, non-transferable license to install and use the Software with one or more Chelsio network adapters on a single server computer for use in communicating with one or more other computers over a network.
  • Page 467 ADP Schedule contract. If the Software is acquired under the terms of a DoD or civilian agency contract, use, duplication or disclosure by the Government is subject Chelsio T5/T4 Unified Wire For Linux Page 467...
  • Page 468 Agreement. Should you have any questions concerning this Agreement, you may contact Chelsio by writing to: Chelsio Communications, Inc. 370 San Aleso Ave. Sunnyvale, CA 94085 Chelsio T5/T4 Unified Wire For Linux Page 468...

This manual is also suitable for:

Chelsio t4

Table of Contents