Cisco 9134 - MDS Multilayer Fabric Switch Troubleshooting Manual
  1. Manuals
  2. Brands
  3. Cisco Manuals
  4. Switch
  5. 9134 - MDS Multilayer Fabric Switch
  6. Troubleshooting manual

Cisco 9134 - MDS Multilayer Fabric Switch Troubleshooting Manual

Mds 9000 family
Hide thumbs Also See for 9134 - MDS Multilayer Fabric Switch:
1
2
Table Of Contents
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
Table of Contents

Advertisement

Quick Links

See also: Configuration Manual , User Manual
S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m
Cisco MDS 9000 Family
Troubleshooting Guide, Release 3.x
Cisco MDS SAN-OS for Release 3.3(1)
May 2008
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
Text Part Number: OL-9285-05

Advertisement

Table of Contents

Troubleshooting

loading

Related Manuals for Cisco 9134 - MDS Multilayer Fabric Switch

Summary of Contents for Cisco 9134 - MDS Multilayer Fabric Switch

  • Page 1 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Cisco MDS SAN-OS for Release 3.3(1) May 2008 Americas Headquarters Cisco Systems, Inc.
  • Page 2 OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

  • Page 3: Table Of Contents

    Verifying SAN Element Registration Fibre Channel End-to-End Connectivity Fabric Issues Port Issues Primary Troubleshooting Flowchart System Messages 1-10 System Message Text 1-10 Syslog Server Implementation 1-10 Implementing Syslog with Fabric Manager 1-11 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 4 Recovering One Supervisor Module With Corrupted Bootflash 2-23 Recovering Both Supervisor Modules With Corrupted Bootflash 2-24 Recognizing Error States 2-25 Switch or Process Resets 2-26 Recoverable System Restarts 2-27 Unrecoverable System Restarts 2-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 5 Troubleshooting the Power Supplies Troubleshooting Fan Issues Fan Is Not Spinning Fan Is Spinning; Fan LED is Red Troubleshooting a Fan Failure Using Device Manager Troubleshooting a Fan Failure Using the CLI 4-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 6 Reinitializing a Failed Module Using the CLI 4-37 Module Resets 4-38 Troubleshooting Mixed Generation Hardware C H A P T E R Overview Port Groups Port Speed Mode Dynamic Bandwidth Management Out-of-Service Interfaces Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 7 6-10 License Listed as Missing 6-11 Troubleshooting Cisco Fabric Services C H A P T E R Overview Initial Troubleshooting Checklist Verifying CFS Using Fabric Manager Verifying CFS Using the CLI Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 8 Port Remains in a Link Failure or Not Connected State 8-13 Troubleshooting Port Problems 8-15 Port Remains in Initializing State 8-16 Troubleshooting Port Registration Issues Using the CLI 8-17 Unexpected Link Flapping Occurs 8-21 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x viii OL-9285-05...
  • Page 9 10-3 PortChannel Issues 10-3 Cannot Configure a PortChannel 10-3 Newly Added Interface Does Not Come Online In a PortChannel 10-4 Configuring Port Channel Modes Using Fabric Manager 10-4 Trunking Issues 10-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 10 Domain Issues 11-17 Domain ID Conflict Troubleshooting 11-17 Switch Cannot See Other Switches in a VSAN 11-18 FC Domain ID Overlap 11-18 Assigning a New Domain ID Using Fabric Manager 11-19 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 11 C H A P T E R Overview 13-1 Configuration Guidelines 13-1 Transit VSANs 13-2 Border Switches 13-2 Limitations and Restrictions 13-2 Initial Troubleshooting Checklist 13-3 Verifying IVR Configuration Using Fabric Manager 13-3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 12 Resolving an Out of Sync Full Zone Database Using the CLI 14-11 Mismatched Default Zone Policy 14-12 Resolving Mismatched Default Zone Policies Using Fabric Manager 14-12 Resolving Mismatched Default Zone Policies Using the CLI 14-13 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 13 15-15 Troubleshooting FICON 16-1 C H A P T E R FICON Overview 16-1 FICON Port Numbering 16-2 Default FICON Port Numbering Scheme 16-2 Reserved FICON Port Numbering Scheme 16-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xiii OL-9285-05...
  • Page 14 Verifying RADIUS Server Groups Using Fabric Manager 17-9 Verifying RADIUS Server Groups Using the CLI 17-9 Verifying TACACS+ Server Groups Using Fabric Manager 17-9 Verifying TACACS+ Server Groups Using the CLI 17-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 15 Fabric Binding Overview 19-2 Initial Troubleshooting Checklist 19-2 Common Troubleshooting Tools in Fabric Manager 19-3 Common Troubleshooting Commands in the CLI 19-3 FC-SP Issues 19-4 Switch or Host Blocked from Fabric 19-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 16 C H A P T E R Overview 20-1 iSCSI Restrictions 20-2 iSLB Restrictions 20-2 Initial Troubleshooting Checklist 20-3 Common Troubleshooting Tools in Fabric Manager 20-3 Common Troubleshooting Commands in the CLI 20-3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 17 Displaying a Time Stamp Acceptable Difference Failure Using the CLI 20-28 FCIP Special Frame Tunnel Creation and Monitoring 20-30 Configuring and Displaying an FCIP Tunnel with Special Frame Using the CLI 20-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xvii OL-9285-05...
  • Page 18 Configuration, Commit, or Merge Failed—”Duplicate Node Name” 20-60 iSLB Configuration Failed—”Pending iSLB CFS Config Has Reached Its Limit...” 20-61 iSCSI Disable Failed—”Cannot Disable Iscsi - Large Iscsi Config Present...” 20-61 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xviii OL-9285-05...
  • Page 19 22-3 IPsec Allowed Transforms 22-4 Initial Troubleshooting Checklist 22-4 Common Troubleshooting Tools in Fabric Manager 22-4 Common Troubleshooting Commands in the CLI 22-5 IPsec Issues 22-5 Verifying IKE Configuration Compatibility 22-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 20 Import and Export Support for Certificates and Associated Key Pairs 24-2 PKI Enrollment Support 24-2 Maximum Limits 24-3 Initial Troubleshooting Checklist 24-3 Common Troubleshooting Tools in Fabric Manager 24-3 Common Troubleshooting Commands in the CLI 24-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 21 Not Receiving Syslog-based Call Home Alerts 25-8 Periodic Inventory Notification Does Not Reflect Current Inventory 25-9 Troubleshooting Fabric Manager 26-1 C H A P T E R Overview 26-1 Guidelines 26-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 22 Fabric Manager Upgrade Without Losing Map Settings 26-5 Restrictions When Using Fabric Manager Across FCIP 26-5 Running Cisco Fabric Manager with Network Multiple Interfaces 26-5 Specifying an Interface for Fabric Manager Server 26-5 Specifying an Interface for Fabric Manager Client or Device Manager...
  • Page 23 B-17 Device Manager: RMON Threshold Manager B-17 Fibre Channel Name Service B-18 SCSI Target Discovery B-19 SNMP and RMON Support B-19 Using RADIUS B-21 Using Syslog B-21 Logging Levels B-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxiii OL-9285-05...
  • Page 24 B-28 Fibre Channel Protocol Analyzers B-28 Using Host Diagnostic Tools B-28 Configuration Limits for Cisco MDS SAN-OS Release 3.x A P P E N D I X N D E X Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxiv...

  • Page 25: New And Changed Information

    Cisco MDS SAN-OS Release 3.x software. The Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x is updated to address each new and changed guideline in the Cisco MDS SAN-OS Release 3.x software. The latest version of this document is available at the following Cisco Systems website: http://www.cisco.com/en/US/products/ps5989/prod_troubleshooting_guides_list.html...
  • Page 26 IP Access Lists Describes troubleshooting IP Access Control Chapter 21, “Troubleshooting IP Lists (ACLs). Access Lists” Users and Roles Added troubleshooting users and roles based Chapter 18, “Troubleshooting access. Users and Roles” Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxvi OL-9285-05...

  • Page 27: Document Organization

    This document is intended to provide guidance for troubleshooting issues that may appear when deploying a storage area network (SAN) using the Cisco MDS 9000 Family of switches. This document introduces tools and methodologies to recognize a problem, determine its cause, and find possible solutions.
  • Page 28 Describes procedures used to troubleshoot Fabric Manager. Appendix A Before Contacting Technical Describes the steps to perform before calling for Support technical support with any Cisco MDS 9000 Family product. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxviii OL-9285-05...

  • Page 29: Document Conventions

    Means reader take note. Notes contain helpful suggestions or references to material not covered in the Note manual. Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxix OL-9285-05...

  • Page 30: Related Documentation

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Related Documentation The documentation set for the Cisco MDS 9000 Family includes the following documents. To find a document online, use the Cisco MDS SAN-OS Documentation Locator at this website: http://www.cisco.com/en/US/docs/storage/san_switches/mds9000/roadmaps/doclocater.htm...

  • Page 31: Cisco Fabric Manager

    • Command-Line Interface Cisco MDS 9000 Family Software Upgrade and Downgrade Guide • Cisco MDS 9000 Family Storage Services Module Software Installation and Upgrade Guide • Cisco MDS 9000 Family CLI Quick Configuration Guide • Cisco MDS 9000 Family CLI Configuration Guide •...
  • Page 32 For information on obtaining documentation, obtaining support, providing documentation feedback, security guidelines, and also recommended aliases and general Cisco documents, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at: http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html...

  • Page 33: Overview Of The Troubleshooting Process

    C H A P T E R Troubleshooting Overview This chapter introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when configuring and using the Cisco MDS 9000 Family of multilayer directors and fabric switches. This chapter includes the following sections: Overview of the Troubleshooting Process, page 1-1 •...

  • Page 34: Best Practices

    This section provides a series of questions that may be useful when troubleshooting a problem with a Cisco MDS 9000 Family switch or connected devices. Use the answers to these questions to plan a course of action and to determine the scope of the problem. For example, if a host can only access some, but not all, of the logical unit numbers (LUNs) on an existing subsystem, then fabric-specific issues (such as FSPF, ISLs, or FCNS) do not need to be investigated.

  • Page 35: C H A P T E R 1 Troubleshooting Overview

    Fabric configuration— In Fabric Manager, click the Fabric Configuration Analysis icon. • Module status—In Device Manager, choose Physical > Modules. • Cisco SAN-OS version—In Device Manager, choose Physical > System. • View logs—In Device Manager, choose Logs > FM Server or Logs > Switch Resident. •...

  • Page 36: Common Cli Commands

    • show accounting log Use the show running interface CLI command to view the interface configuration in Cisco SAN-OS Note Release 3.0(1) or later. The interface configuration as seen in the show running-config CLI command is no longer consolidated.

  • Page 37: Verifying San Element Registration

    FC ID is listed as a device. This can validate that FSPF is working correctly. Fabric Issues Answering the following questions will help to determine the status of the fabric configuration: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 38: Port Issues

    Yellow box: In Device Manager, a port has been selected. • Gray box: The port is administratively disabled. • • Black box: An SFP is not present. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 39 For ISLs, the IP address of the connected switch • Speed • Frames transmitted and received • Percentage utilization for the CPU, dynamic memory, and Flash memory • Figure 1-2 Device Manager: Summary View Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 40 Device Manager: Oversubscription Information Device Manager provides oversubscription information for supported switching modules and line cards. Use the Device View to view an individual module and then right-click to select from the following tasks: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 41: Primary Troubleshooting Flowchart

    Port group oversubscription is supported on 24-port and 48-port 4-Gbps Fibre Channel switching Note modules, the 32-port 2-Gbps Fibre Channel switching module, the Cisco MDS 9120 20-port 1/2-Gbps Fibre Channel module, and the Cisco MDS 9140 40-port 1/2-Gbps Fibre Channel module.

  • Page 42: System Messages

    Syslog Server Implementation The syslog facility allows the Cisco MDS 9000 Family platform to send a copy of the message log to a host for more permanent storage. This can be useful if the logs need to be examined over a long period of time or when the Cisco MDS switch is not accessible.

  • Page 43: Implementing Syslog With Fabric Manager

    To do this the Cisco MDS switch must be configured to send syslog messages to your local PC and a syslog server must be running on that PC to receive those messages. These messages can be categorized into four classes: Hardware—Line card or power supply problems...

  • Page 44: Implementing Syslog With The Cli

    UNIX workstations have a built-in syslog server. You must have root access (or run the Cisco syslog server as setuid to root) to stop the built-in syslog daemon and start the Cisco syslog server.

  • Page 45: Troubleshooting With Logs

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Troubleshooting with Logs Cisco SAN-OS generates many types of system messages on the switch and sends them to a syslog server. These messages can be viewed using Fabric Manager or the CLI to determine what events may have led up to the current problem condition you are facing.

  • Page 46: Viewing The Log From The Supervisor

    2005 Sep 16 15:45:41 172.20.150.82 %PLATFORM-2-MOD_PWRUP_XBAR: Modules powered up due to xbar availability 2005 Sep 18 15:12:07 172.20.150.82 %MODULE-2-MOD_FAIL: Initialization of module 14 (serial: JAB092501FC) failed Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-14 OL-9285-05...

  • Page 47: Contacting Customer Support

    “Obtaining Documentation, Obtaining Support, and Security Guidelines” section on page xxxii. For more information on steps to take before calling Technical Support, see the “Before Contacting Technical Support” section on page A-1. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-15 OL-9285-05...
  • Page 48 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 49: Overview

    Miscellaneous Software Image Issues, page 2-32 • Overview Each Cisco MDS 9000 switch ships with an operating system (Cisco SAN-OS) that consists of two images—the kickstart image and the system image. There is also a module image if the Storage Services Module (SSM) is present.

  • Page 50: C H A P T E R 2 Troubleshooting Installs, Upgrades, And Reboots

    It is normal for the active supervisor to become the standby supervisor during an upgrade. Note Follow these guidelines when upgrading or downgrading Cisco SAN-OS software images: Read the Cisco SAN-OS Release Notes for the release you are upgrading or downgrading to. Cisco • SAN-OS Release Notes are available at the following website: http://cisco.com/en/US/products/ps5989/prod_release_notes_list.html...
  • Page 51 Where possible, choose to do a nondisruptive upgrade. In general, you can nondisruptively upgrade • to Cisco SAN-OS Release 3.x software from any Cisco SAN-OS software Release 2.x or later. Review the upgrade table in the Cisco SAN-OS Release Notes for the version you will be – installing.

  • Page 52: Guidelines For Reboots

    These modules use a rolling upgrade install mechanism where the modules are upgraded in sequence. After the first module upgrade finishes, and before the next module upgrade begins, Cisco SAN-OS introduces a time delay to ensure that all applications in the module reach a steady state. The IPS modules require a five-minute delay before the next IPS module upgrade can guarantee a stable state.

  • Page 53: Troubleshooting Fabric Manager Installations

    Device Manager will not Device Manager proxied through Uncheck the Proxy SNMP through FM Server start. Fabric Manager Server. check box in the Device Manager startup dialog box, and restart Device Manager. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 54: Verifying Cisco San-Os Software Installations

    -- SUCCESS Extracting “loader” version from image bootflash:/b-1.3.0.104. -- SUCCESS switch# show install all status This is the log of last installation. <----------------- log of last install Verifying image bootflash:/b-1.3.0.104 -- SUCCESS Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 55: Troubleshooting Cisco San-Os Software Upgrades And Downgrades

    Warning: The startup config contains commands not supported by the system image; as a result, some resources might become unavailable after an install. Do you wish to continue? (y/ n) [y]: n Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 56 Message 2 indicates that the Fibre Channel tunnel feature is not supported in the new image. The RSPAN feature uses Fibre Channel tunnels. 2) Feature Index : 119 , Capability : CAP_FEATURE_FC_TUNNEL_CFG Description : fc-tunnel is enabled Capability requirement : STRICT Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 57: Software Installation Ends With Error

    SAN-OS Software Using Fabric Manager” section on page 2-10 or the “Installing Cisco SAN-OS Software from the CLI” section on page 2-11. Or, use the install module CLI command to upgrade the failed module. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 58: Installing San-Os Software Using Fabric Manager

    TFTP server is running or because access to the TFTP port 69 has been denied for security reasons (the default setting on LINUX). In these cases, you cannot transfer files from the local host to the switch. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-10 OL-9285-05...

  • Page 59: Installing Cisco San-Os Software From The Cli

    [####################] 100% -- SUCCESS Extracting "ips" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "svclc" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "system" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-11 OL-9285-05...
  • Page 60 [####################] 100% -- SUCCESS Syncing image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin to standby. [####################] 100% -- SUCCESS Setting boot variables. [####################] 100% -- SUCCESS Performing configuration copy. [####################] 100% -- SUCCESS Module 5: Waiting for module online. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-12 OL-9285-05...

  • Page 61: Troubleshooting Cisco San-Os Software System Reboots

    BIOS image. See the “Recovery Using BIOS Setup for Supervisor-1” section on page 2-16. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-13 OL-9285-05...

  • Page 62: Corrupted Bootflash Recovery

    Access this utility only when needed to recover a corrupted internal disk. The BIOS changes explained in this section are required only to recover a corrupted bootflash. Caution Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-14 OL-9285-05...
  • Page 63 Hanging Switch (boot)# Recovery configuration state prompt state prompt 3 = Kickstart Power on 3 = Kickstart Power on 4 = System image and Ctrl-C image and Esc image Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-15 OL-9285-05...

  • Page 64: Recovery Using Bios Setup For Supervisor-1

    Your navigating options are provided at the bottom of the screen. Note Tab = Jump to next field Ctrl-E = Down arrow Ctrl-X = Up arrow Ctrl-H = Erase (Backspace might not work if your terminal is not configured properly.) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-16 OL-9285-05...
  • Page 65 The file name must be entered exactly as it is displayed on your TFTP server. For example, if you have Caution a file named MDS9500-kiskstart_mzg.10, then enter this name using the exact uppercase characters and file extensions as shown on your TFTP server. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-17 OL-9285-05...
  • Page 66 Be sure that you have made a backup of the configuration files before you issue this command. Caution Follow the procedure specified in the “Recovery from the switch(boot)# Prompt” section on page 2-21. Step 15 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-18 OL-9285-05...

  • Page 67: Recovery From The Loader> Prompt On Supervisor-2 Modules

    Be sure that you have made a backup of the configuration files before you issue this command. Step 6 Follow the procedure specified in the “Recovery from the switch(boot)# Prompt” section on page 2-21. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-19 OL-9285-05...

  • Page 68: Recovery From The Loader> Prompt On Supervisor-1 Modules

    INIT: Sending processes the TERM signal Sending all processes the TERM signal... done. Sending all processes the KILL signal... done. Entering single-user mode... INIT: Going single user INIT: Sending processes the TERM signal switch(boot)# Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-20 OL-9285-05...

  • Page 69: Recovery From The Switch(Boot)# Prompt

    If you believe there are file system problems, issue the init system check-filesystem command. As of Cisco MDS SAN-OS Release 2.1(1a), this command checks all internal file systems and fixes any errors that are encountered. This command takes considerable time to complete.

  • Page 70: Recovery For Switches With Dual Supervisor Modules

    Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license.

  • Page 71: Recovering One Supervisor Module With Corrupted Bootflash

    After the init system command completes on the standby supervisor module, issue the system no standby manual-boot command in EXEC mode on the active supervisor module. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-23 OL-9285-05...

  • Page 72: Recovering Both Supervisor Modules With Corrupted Bootflash

    Boot the kickstart image file from the required server. Step 4 loader> boot tftp://172.16.10.100/kickstart-latest Address: 172.16.1.2 Netmask: 255.255.255.0 Server: 172.16.10.100 Gateway: 172.16.1.1 Booting: /kick-282 console=ttyS0,9600n8nn quiet loader_ver= “2.1(2)”..........Image verification OK Starting kernel... Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-24 OL-9285-05...

  • Page 73: Recognizing Error States

    Figure 2-7 Figure 2-8, follow the procedure specified in the “Recovery Using BIOS Setup for Supervisor-1” section on page 2-16. Figure 2-7 Error State if Powered On and Ctrl-C Is Entered Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-25 OL-9285-05...

  • Page 74: Switch Or Process Resets

    A clock module failed. Verify that a clock module failed. See the “Troubleshooting Clock Module Issues” section on page 4-12. Replace the failed clock module during the next maintenance window. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-26 OL-9285-05...

  • Page 75: Recoverable System Restarts

    Enter the following command to check the syslog file to see which process restarted and why it restarted: Step 1 switch# show log logfile | include error For information about the meaning of each message, refer to the Cisco MDS 9000 Family System Messages Reference. The system output looks like the following example: Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor"...
  • Page 76 EFL 00000246 ESP 7FFFFC5C XSS 0000002B Stack: 128 bytes. ESP 7FFFFC5C, TOP 7FFFFD10 0x7FFFFC5C: 0804F990 0804C416 00000003 0804E994 ....0x7FFFFC6C: 00000008 0804BF95 2AC451E0 2AAC24A4 ..Q.*.$.* 0x7FFFFC7C: 7FFFFD14 2AC2C581 0804E6BC 7FFFFCA8 ..*..Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-28 OL-9285-05...
  • Page 77 Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2) Exit code: signal 6 (core dumped) CWD: /var/sysmgr/work Virtual Memory: CODE 08048000 - 080FB060 DATA 080FC060 - 080FCBA8 081795C0 - 081EC000 STACK 7FFFFCF0 TOTAL 20952 KB Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-29 OL-9285-05...
  • Page 78 See also the “Troubleshooting Supervisor Issues” section on page 4-14 or the “Troubleshooting Switching and Services Modules” section on page 4-21. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-30 OL-9285-05...

  • Page 79: Unrecoverable System Restarts

    The show system reset-reason CLI command displays the following information: • In a Cisco MDS 9500 Series switch, the last four reset-reason codes for the supervisor module in slot 5 and slot 6 are displayed. If either supervisor module is absent, the reset-reason codes for that supervisor module are not displayed.

  • Page 80: Recovering The Administrator Password

    You forgot the administrator password for You can recover the password using a local console connection. For the accessing a Cisco MDS 9000 Family switch. latest instructions on password recovery, refer to the Cisco MDS 9000 CLI Family Configuration Guide at the following website: http://cisco.com/en/US/products/ps5989/products_installation_and_conf iguration_guides_list.html...

  • Page 81: All Ports Down Because Of System Health Failure

    Cisco MDS SAN-OS Release 2.0(1b) to module module-number command, where come up when running Release 2.0(3) and following the configuration of module-number is a specific module. on an MPS-14/2 a new FCIP link. module. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-33 OL-9285-05...

  • Page 82: Cannot Create, Modify, Or Delete Admin Role

    Following an upgrade from Cisco SAN-OS Reconfigure the FC IDs as necessary. resets. Release 1.1 to Cisco SAN-OS Release 1.3 or later, with persistent FC ID enabled, the FC IDs for the storage arrays might change after a link flap.

  • Page 83: Chapter 3 Managing Storage Services Modules

    The 32-port Fibre Channel Storage Services Module (SSM) for the Cisco MDS 9000 Family supports up to 32 Fibre Channel ports and provides distributed intelligent storage services. Cisco MDS 9500 Series switches running Cisco MDS SAN-OS Release 2.0(2b) or later support the SSM Note module.
  • Page 84 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Table 3-1 lists the features supported for the SSM Table 3-1 Cisco MDS SAN-OS Feature Support for the SSM. Cisco MDS SAN-OS Release 2.0(2b), 2.0(3), 2.0(4), Module 2.0(1b)

  • Page 85: Initial Troubleshooting Checklist

    Verify that the SSI image is present and pointed to by the SSI boot variable. Verify that the EPLD version is 2.1(2) or later for nondisruptive layer-2 upgrades. Verify that you have configured all SSM ports prior to upgrading to Cisco SAN-OS Release 3.0(1) or later. Do not use port mode auto.

  • Page 86: Ssm Fails To Boot

    A newly installed SSM initially operates in Fibre Channel switching mode by default. Note If you downgrade to a Cisco MDS SAN-OS release that does not support the SSM, you must power down the module. The boot variables for the SSM are lost.

  • Page 87: Verifying The Ssi Boot Image

    3-9. Verifying the SSI Boot Image To verify that you have the correct Cisco MDS SAN-OS release and SSI boot image file on your switch, follow these steps: Log in to the switch through the console port, an SSH session, or a Telnet session.

  • Page 88: Using The Install Ssi Command

    Download the SSI software image file from Cisco.com to your FTP server. If your SSM boots, then verify that you have enough free space available on the modflash: on the SSM using the dir modflash://slot-1/ command. The download site on Cisco.com shows the size of the boot image file in bytes.
  • Page 89 Note the slot number for later reference. Verify the Cisco MDS SAN-OS release that is running on the switch and verify the location and name Step 3 of the SSI boot image that is on the switch by following the procedure described in the “Verifying the...

  • Page 90: Recovering A Replacement Ssm

    * this terminal session Recovering a Replacement SSM In Cisco MDS SAN-OS Release 2.1(2) and later, you use the CompactFlash memory (modflash:) on the SSM to store the SSI image. If the SSM is replaced, the new SSM might not initialize.

  • Page 91: Ssm Upgrade Is Disruptive

    Use the no ssm enable feature CLI command. Installing EPLD Images on Modules Refer to the Cisco MDS SAN-OS Release Notes for Cisco MDS 9000 EPLD Images to verify whether or not the EPLD has changed for the Cisco SAN-OS image version being used.
  • Page 92 Verify that you have enough free space available on the active and standby supervisor memory devices that you plan to use, either bootflash: or slot0:. The download site on Cisco.com shows the size of the EPLD image file in bytes.
  • Page 93 12288 Jan 01 00:01:06 1980 lost+found/ 14765056 Mar 21 15:35:06 2005 m9500-sf1ek9-kickstart-mz.2.1.1.bin 15944704 Apr 06 16:46:04 2005 m9500-sf1ek9-kickstart-mz.2.1.1a.bin 48063243 Mar 21 15:34:46 2005 m9500-sf1ek9-mz.2.1.1.bin 48036239 Apr 06 16:45:41 2005 m9500-sf1ek9-mz.2.1.1a.bin Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-11 OL-9285-05...
  • Page 94 The switch software prompts you to continue after reporting the module state. When you confirm your intention to continue, the upgrade continues. switch# install module 2 epld bootflash:m9000-epld-2.1.2.img <------------------------------------------------------------progress twirl Module 2 EPLD upgrade is successful Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-12 OL-9285-05...
  • Page 95 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m When you upgrade the EPLD module on Cisco MDS 9100 Series switches, you receive the...
  • Page 96 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 97: Troubleshooting Hardware

    C H A P T E R Troubleshooting Hardware This chapter describes how to identify and resolve problems that might occur in the hardware components of the Cisco MDS 9000 Family. It includes the following sections: Overview, page 4-1 •...

  • Page 98: C H A P T E R 4 Troubleshooting Hardware

    If you have a redundant supervisor module, refer to the following website for the latest Cisco MDS 9000 Family configuration guides for descriptions of how the redundant supervisor module comes online and how the software images are handled: http://www.cisco.com/univercd/cc/td/doc/product/sn5000/mds9000/index.htm.

  • Page 99: Troubleshooting Power Supply Issues

    “Troubleshooting Supervisor Issues” section on page 4-14. If you have a redundant supervisor module, refer to the following website for the latest Cisco MDS 9000 Family configuration guides for descriptions of the supervisor module LEDS, how the redundant supervisor module comes online, and how the software images are handled: http://www.cisco.com/univercd/cc/td/doc/product/sn5000/mds9000/index.htm.

  • Page 100: All Power Supply Leds Are Off

    CLI command or similar Fabric Manager/Device Manager command to collect more information. Error Message PLATFORM-5-PS_REMOVE: Power supply [dec] removed (Serial No. [chars]). Explanation Power supply has been removed. No action is required. Recommended Action Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 101: Power Supply Input Ok Led Is Red

    If both power supplies are the same capacity or the mode is combined, Cisco SAN-OS never shuts down a power supply. Power supply is not operational.

  • Page 102: Power Supply Output Failed Led Is On

    If the temperature sensors are near or over a threshold value, you should replace the power supply. Power supply is not operational. Troubleshoot the power supplies. See the “Troubleshooting the Power Supplies” section on page 4-7. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 103: Troubleshooting The Power Supplies

    If you purchased Cisco support through a Cisco reseller, contact the reseller directly. If you Note purchased support directly from Cisco, contact Cisco Technical Support at this URL: http://www.cisco.com/warp/public/687/Directory/DirTAC.shtm...

  • Page 104: Troubleshooting Fan Issues

    Tighten all captive screws, and then restart the system. Fan module has failed. Troubleshoot the Fan Module. See the “Troubleshooting a Fan Failure Using the CLI” section on page 4-10. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 105: Troubleshooting A Fan Failure Using Device Manager

    Enter the show platform internal info command or similar Fabric Manager/Device Manager command to collect more information. If the OperStatus is absent, the fan module has been removed. As soon as the fan module is removed, Cisco Step 3 SAN-OS starts a five-minute countdown.

  • Page 106: Troubleshooting A Fan Failure Using The Cli

    Enter the show platform internal info command to collect more information. Recommended Action If the fan status is absent, the fan module has been removed. As soon as the fan module is removed, Cisco Step 3 SAN-OS starts a five-minute countdown.

  • Page 107: Temperature Threshold Violations

    Intake The intake sensor, located at the airflow intake on the module, is the most critical indicator of module temperature. All Cisco SAN-OS actions are taken when the major threshold of an intake sensor is exceeded. A minor threshold violation or a major threshold violation on an outlet sensor results in the following...

  • Page 108: Troubleshooting Clock Module Issues

    Enter the show environment temperature CLI command or similar Fabric Manager/Device Manager command to collect more information. Troubleshooting Clock Module Issues A Cisco MDS 9500 Series director has two clock modules: A and B. Use the show environment clock CLI command to view the clock module status. (See Example 4-4.)

  • Page 109: Troubleshooting Other Hardware Issues

    ********* Exception info for module 8 ******** <---Possible failed module exception information --- exception instance 1 ---- device id: device errorcode: 0x80000080 system time: (1127843531 ticks) Tue Sep 27 17:52:11 2005 error type: FATAL error <------------------- Error Type field Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-13 OL-9285-05...

  • Page 110: Troubleshooting Supervisor Issues

    After all components on the standby are synchronized with those of the active supervisor, the standby supervisor is up. Cisco SAN-OS maintains debug information during runtime. When a supervisor reboots, much of the debug information is lost. However, all critical information is stored in NVRAM and can be used to reconstruct the failure.

  • Page 111: Active Supervisor Reboots

    2005 Sep 27 18:58:06 172.20.150.204 %SYSMGR-3-SERVICE_CRASHED: Service "xbar" (PID 2349) hasn't caught signal 9 (no core). 2005 Sep 27 18:58:06 172.20.150.204 %SYSMGR-3-SERVICE_CRASHED: Service "xbar" (PID 2352) hasn't caught signal 9 (no core). Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-15 OL-9285-05...
  • Page 112 2005 Sep 28 14:17:47 172.20.150.204 %XBAR-5-XBAR_STATUS_REPORT: Module 6 reported status for component 12 code 0x60a02. 2005 Sep 28 14:17:59 172.20.150.204 %PORT-5-IF_UP: Interface mgmt0 on slot 5 is up 2005 Sep 28 14:18:00 172.20.150.204 %CALLHOME-2-EVENT: SUP_FAILURE Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-16 OL-9285-05...

  • Page 113: Standby Supervisor Not Recognized By Active Supervisor

    Telnet to the standby supervisor console port and verify that it is in standby mode. (See Example 4-11.) Example 4-11 Verify Standby Supervisor Mode runlog>telnet sw4-ts 2004 Trying 172.22.22.55... Connected to sw4-ts.cisco.com (172.22.22.55). Escape character is '^]'. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-17 OL-9285-05...
  • Page 114 Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2005, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license.

  • Page 115: Standby Supervisor Stays In Powered-Up State

    Powered-Up State Using Device Manager” section on state. page 4-20 or the “Verifying That a Standby Supervisor Is in Powered-Up State Using the CLI” section on page 4-20. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-19 OL-9285-05...

  • Page 116: Verifying That A Standby Supervisor Is In The Powered-Up State Using Device Manager

    This prevents the switch from attempting to fail over to an unavailable module. This section provides a workaround for a failed supervisor under certain conditions. An example situation is used to describe the problem and the workaround. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-20 OL-9285-05...

  • Page 117: Troubleshooting Switching And Services Modules

    Module-Type Model Status ----- -------------------------------- ------------------ ------------ IP Storage Services Module DS-X9308-SMIP World-Wide-Name(s) (WWN) ----------- ------ -------------------------------------------------- 2.1(2) 0.206 21:c1:00:05:30:00:8f:5e to 21:c8:00:05:30:00:8f:5e MAC-Address(es) Serial-Num -------------------------------------- ---------- 00-05-30-00-9e-fa to 00-05-30-00-9f-06 JAB064704LH Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-21 OL-9285-05...

  • Page 118: Module Initialization Overview

    Most of the module related failures (such as the module not coming up, the module getting reloaded, and so on) can be analyzed by looking at the logs stored on the switch. Use the following CLI commands to view this information: show system reset-reason module • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-22 OL-9285-05...

  • Page 119: Module Bootup

    Error Message MODULE-2-MOD_DNLD_FAIL: Image download failed for module [dec]. The module failed to download a new image from the supervisor module. Explanation Collect module information by entering the show module internal all module Recommended Action <dec> command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-23 OL-9285-05...

  • Page 120: Runtime Diagnostics

    The supervisor continues to monitor the module periodically to verify correct operation. The following events are monitored: • Heartbeat message—Sent between the supervisor and the module to verify that the module is running. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-24 OL-9285-05...

  • Page 121: Analyzing The Logs

    To isolate a module problem, follow these steps: Verify that all Status LEDs are green. If any status LED is red or off, the module might have shifted out Step 1 of its slot. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-25 OL-9285-05...

  • Page 122: Troubleshooting Powered-Down Modules

    Step 4 If the Status LED on a switching module is orange, the module might be busy or disabled. Refer to the following website for the latest Cisco MDS 9000 Family configuration guides to configure or enable the interfaces: http://www.cisco.com/univercd/cc/td/doc/product/sn5000/mds9000/index.htm.
  • Page 123 Enter the show module, show platform internal all module[dec] and show module internal all module [dec] CLI command to collect more information if you suspect module has been powered down due to errors. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-27 OL-9285-05...

  • Page 124: Diagnosing A Powered-Down Module

    Ports Module-Type Model Status ----- -------------------------------- ------------------ ------------ Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby Supervisor/Fabric-1 DS-X9530-SF1-K9 active * IP Storage Services Module powered-dn World-Wide-Name(s) (WWN) ----------- ------ ------------------------------------ 2.1(2) 2.1(2) 0.602 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-28 OL-9285-05...
  • Page 125 (1127834890 ticks) Tue Sep 27 15:28:10 2005 error type: Warning Number Ports went bad: none exception information --- exception instance 4 ---- device id: device errorcode: 0x40000002 system time: (1127834823 ticks) Tue Sep 27 15:27:03 2005 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-29 OL-9285-05...
  • Page 126 Instance:3, Seq Id:0xc, Ret:success [E_MTS_RX] Src:MTS_SAP_XBAR_MANAGER(48), Opc:MTS_OPC_LC_REMOVED(1082) 110) FSM:<ID(3): Slot 8, node 0x0802> Transition at 716643 usecs after Tue Sep 2 7 15:30:23 2005 Previous state: [LCM_ST_CHECK_REMOVAL_SEQUENCE] Triggered event: [LCM_EV_ALL_LC_REMOVED_RESP_RECEIVED] Next state: [LCM_ST_LC_FAILURE] Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-30 OL-9285-05...

  • Page 127: Troubleshooting Reloaded Modules

    In this example, you can conclude that module is not coming up, because the XBAR Manager is failing during the insertion of the module. Troubleshooting Reloaded Modules Module is automatically reloaded. Symptom The following system messages may be present if a module reloads: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-31 OL-9285-05...
  • Page 128 Port loop-back test failure. Explanation No action is required. Recommended Action Error Message SYSTEMHEALTH-2-OHMS_MOD_SNAKE_TEST_FAILED: Module [dec] has failed snake loopback tests. Snake test failure. Explanation No action is required. Recommended Action Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-32 OL-9285-05...

  • Page 129: Diagnosing A Reloaded Module

    0x40730017 system time: (1127843486 ticks) Tue Sep 27 17:51:26 2005 error type: FATAL error Number Ports went bad: 1,2,3,4,5,6,7,8 exception information --- exception instance 4 ---- device id: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-33 OL-9285-05...

  • Page 130: Troubleshooting Modules In An Unknown State

    1) Event:E_DEBUG, length:37, at 370073 usecs after Thu Sep 29 17:22:48 2005 [103] unable to init lc sprom 0 mod 8 switch# show platform internal event-history module 8 Inside pfm_show_eventlog Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-34 OL-9285-05...

  • Page 131: Troubleshooting Modules Not Detected By The Supervisor

    “Installing current version of Cisco SAN-OS on SAN-OS Software Using Fabric Manager” section on page 2-10 the switch. “Installing Cisco SAN-OS Software from the CLI” section on page 2-11. Diagnosing a Module Not Detected by the Supervisor To diagnose a module that has not been detected by the supervisor, follow these steps:...

  • Page 132: Reinitializing A Failed Module Using Fabric Manager

    When a module is inserted into the switch, the supervisor module reads the SPROM contents of the module. If the module is supported by the current version of Cisco SAN-OS, the module will be powered-up by the supervisor module. If the power status does not show that the module has powered up properly, the module information is not relayed to the supervisor.

  • Page 133: Reinitializing A Failed Module Using The Cli

    If the module is not powered on, remove and reseat the module and then power on the module. Step 7 switch# config t switch(config)# no poweroff module 2 switch(config)# exit switch# If the module is still not operating, reload the entire switch. Step 8 switch# reload Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-37 OL-9285-05...

  • Page 134: Module Resets

    The front panel reset button on the supervisor module was pressed. • Any hardware failure that caused the processor, dynamic memory, or I/O to reset or hang. • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-38 OL-9285-05...

  • Page 135: Overview

    Troubleshooting Mixed Generation Hardware This chapter describes how to identify and resolve problems that might occur when you combine Generation 1 and Generation 2 hardware components of the Cisco MDS 9000 Family. It includes the following sections: Overview, page 5-1 •...

  • Page 136: C H A P T E R 5 Troubleshooting Mixed Generation Hardware

    Gigabit Ethernet ports in fixed slot1 Note Generation 2 Fibre Channel switching modules are not supported on the Cisco MDS 9216 switch; however, they are supported by both the Supervisor-1 module and the Supervisor-2 module. The 4-port 10-Gbps Fibre Channel switching module supports 10-Gbps port rates. The rest of the Generation 2 modules support 1-Gbps, 2-Gbps, 4-Gbps, or autosensing port rates.

  • Page 137: Port Speed Mode

    2 Gbps 1 Gbps 1 Gbps 10 Gbps 1. Available only on the 4-port 10-Gbps switching module. Table 5-5 shows the bandwidth reserved based on port speed for ports in shared mode. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 138: Out-Of-Service Interfaces

    Port Index Availability Each chassis in the Cisco MDS 9000 Series has a hardware-based maximum port availability based on internally assigned port indexes. When the maximum number of port indexes is reached in a chassis, any modules remaining or added to the chassis will not boot up.
  • Page 139 Using any combination of modules that include a Generation 1 module or a Supervisor-1 module limits the port index availability to 252 on all Cisco MDS 9500 Series directors. Generation 1 modules also require contiguous port indexes where the system assigns a block of port index numbers contiguously...
  • Page 140 Using any combination of modules that include a Generation 1 module and a Supervisor-2 module limits the port index availability to 252 on all Cisco MDS 9500 Series directors. The Generation 1 modules can use any contiguous block of port indexes that start on the first port index reserved for any slot in the range 0-252.

  • Page 141: Combining Modules And Supervisors

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Combining Modules and Supervisors All the existing Generation 1 and Generation 2 switching modules are supported by Cisco MDS SAN-OS Release 3.0(1) and later. However, there are limitations to consider when combining the various modules and supervisors in the Cisco MDS 9500 Series platform chassis.

  • Page 142: Module Does Not Come Online

    | (None) 0- 255| | (None) ----- | 253-255 In some cases, the sequence in which switching modules are inserted into the chassis determines if one or more modules is powered up. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 143: Verifying Port Index Allocation Using The Cli

    Startup module index distribution: ------------------------------------------------------+ Slot | Allowed | Alloted indices info range | Total | Index values -----|---------|-------|------------------------------| 0- 255| | 64-79 0- 255| | 0-11 ----- | 253-255 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 144: Cannot Configure Port In Dedicated Mode

    “Verifying Bandwidth Utilization in a Port Group port in dedicated port group. Using Device Manager” section on page 5-11 or the mode. “Verifying Bandwidth Utilization in a Port Group Using the CLI” section on page 5-12. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-10 OL-9285-05...

  • Page 145: Verifying Bandwidth Utilization In A Port Group Using Device Manager

    Right-click a port and select Configure. Lower the port speed. See the “Dynamic Bandwidth Management” section on page 5-3 for the minimum bandwidth requirements for port rate modes and port speeds. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-11 OL-9285-05...

  • Page 146: Verifying Bandwidth Utilization In A Port Group Using The Cli

    Use the swtichport speed command on one or more ports to change the port speed to a lower port speed. See the “Dynamic Bandwidth Management” section on page 5-3 for the minimum bandwidth requirements for port rate modes and port speeds. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-12 OL-9285-05...

  • Page 147: Cannot Enable A Port

    Selecting the Correct Software Images The Supervisor-1 and Supervisor-2 modules supported by Cisco MDS 9100, 9200 and 9500 Series switches require different system and kickstart images. You can determine which images to use on your switch by the naming conventions shown in Table 5-12.
  • Page 148 9216, 9216A or 9216i Supervisor-1 module Filename begins with m9200-s1k9 9506 or 9509 Supervisor-1 module Filename begins with m9500-sf1ek9 Supervisor-2module Filename begins with m9500-sf2ek9 9513 Supervisor-2 module Filename begins with m9500-sf2ek9 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-14 OL-9285-05...

  • Page 149: Chapter 6 Troubleshooting Licensing

    Multiprotocol Services (MPS-14/2) module. If you install a module with IP ports in the empty slot on the Cisco MDS 9216i, a separate SAN Extension over IP license is required to enable related features on the IP ports of the additional module.

  • Page 150: Chassis Serial Numbers

    120 day grace period to evaluate the feature. You must purchase and install the number of licenses required for that feature before the grace period ends or Cisco SAN-OS will disable the feature at the end of the grace period. If you try to use an unlicensed feature, you may see the following system...

  • Page 151: Initial Troubleshooting Checklist

    To view license information using Fabric Manager, follow these steps: Step 1 Select Switches > Licenses from the Physical Attributes pane. You see the license information in the Information pane, one line per feature. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 152: Displaying License Information Using Device Manager

    Grace 79D 16H MAINFRAME_PKG Unused Grace expired ENTERPRISE_PKG InUse never DMM_FOR_SSM_PKG Unused SAN_EXTN_OVER_IP Unused PORT_ACTIVATION_PKG Unused SME_FOR_IPS_184_PKG Unused Grace 86D 5H SAN_EXTN_OVER_IP_18_4 Unused SAN_EXTN_OVER_IP_IPS2 Unused never 1 license(s) missing SAN_EXTN_OVER_IP_IPS4 Unused 10G_PORT_ACTIVATION_PKG Unused Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 153 Displays the Contents of a Specified License Key File switch# show license file Permanent.lic Permanent.lic: SERVER this_host ANY VENDOR cisco INCREMENT MAINFRAME_PKG cisco 1.0 permanent uncounted \ HOSTID=VDH=FOX0646S017 \ NOTICE=”<LicFileID></LicFileID><LicLineID>0</LicLineID> \ <PAK>dummyPak</PAK>” SIGN=EE9F91EA4B64 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 154: Licensing Installation Issues

    A common problem with licenses stems from not using the correct chassis serial number when ordering your license. To obtain the correct chassis serial number using Fabric Manager, follow these steps: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 155: Rma Chassis Errors Or License Transfers Between Switches

    Note If you purchased Cisco support through a Cisco reseller, contact the reseller directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support at this URL: http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml...

  • Page 156: Incorrect Number Of Licenses In Use For Multiple Modules

    Copy both license files to one file: Step 2 Example SERVER this_host ANY VENDOR cisco INCREMENT SAN_EXTN_OVER_IP_IPS2 cisco 1.0 permanent 1 \ VENDOR_STRING=<LIC_SOURCE>MDS_SWIFT</LIC_SOURCE><SKU>M9500EXT12EK9=</SKU> \ HOSTID=VDH=FOXYYYYYYY \ NOTICE="<LicFileID>2005082204514XXXX</LicFileID><LicLineID>1</LicLineID> \ <PAK>MDS-1X-JAB-0F1A81</PAK>" SIGN=F0652E02XXXX INCREMENT SAN_EXTN_OVER_IP_IPS2 cisco 1.0 permanent 1 \ Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 157: Grace Period Alerts

    6-10. The Cisco SAN-OS license counter keeps track of all licenses on a switch. If you are evaluating a feature and the grace period has started, you will receive console messages, SNMP traps, system messages, and Call Home messages on a daily basis.

  • Page 158: Checking In The Fabric Manager Server License From Device Manager

    Because of Caveat CSCeg23889, you might still receive Call Home or system messages for an unused Note FM_SERVER_PKG license. This caveat describes how extraneous messages are sent after a Fabric Manager Server license is checked in. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-10 OL-9285-05...

  • Page 159: License Listed As Missing

    Supervisor module was replaced after Reinstall the license. missing. license was installed. Supervisor bootflash: is corrupted. See the “Corrupted Bootflash Recovery” section on page 2-14 to recover from corrupted bootflash:. Reinstall the license. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-11 OL-9285-05...
  • Page 160 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 161: Overview

    C H A P T E R Troubleshooting Cisco Fabric Services This chapter describes procedures used to troubleshoot Cisco Fabric Services (CFS) problems in the Cisco MDS 9000 Family multilayer directors and fabric switches. It includes the following sections: Overview, page 7-1 •...

  • Page 162: C H A P T E R 7 Troubleshooting Cisco Fabric Services

    Span across some or all of the switches in the topology, within the physical scope of the application. All switches in the fabric must be CFS capable. A Cisco MDS 9000 Family switch is CFS capable if it is running Cisco SAN-OS Release 2.0(1b) or later. Switches that are not CFS capable do not receive distributions and result in part of the fabric not receiving the intended distribution.

  • Page 163: Verifying Cfs Using Fabric Manager

    To verify that an application is listed and enabled, issue the show cfs application command to all Step 1 switches. An example of the show cfs application command follows: Switch# show cfs application ------------------------------------------- Application Enabled Scope ------------------------------------------- Physical Physical dpvm Physical Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 164 :Logical [VSAN 2] ----------------------------------------------------------- Domain Switch WWN IP Address ----------------------------------------------------------- 20:00:00:0e:d7:00:3c:9e 10.76.100.52 [Local] 20:00:00:05:30:00:6b:9e 10.76.100.167 20:00:00:0d:ec:06:55:c0 10.76.100.205 Total number of entries = 3 Scope :Logical [VSAN 3] ----------------------------------------------------------- Domain Switch WWN IP Address Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 165: Merge Failure Troubleshooting

    CLI command. If there are any locks, then the merge will not proceed. Commit the changes or clear the session lock so that the merge can proceed. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 166: Recovering From A Merge Failure With Fabric Manager

    20:00:00:0d:ec:04:99:c0 COMMIT admin 3849 Fri Aug 24 04:30:19 2007 20:00:00:0d:ec:04:99:c0 LOCK_RELEASE_REQUEST admin 3848 Fri Aug 24 04:30:19 2007 20:00:00:0d:ec:04:99:c0 LOCK_RELEASED admin 3848 Fri Aug 24 04:33:07 2007 20:00:00:0d:ec:04:99:c0 LOCK_REQUEST admin 3868 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 167: Lock Failure Troubleshooting

    Set the Config Action drop-down menu on the master switch to commit or abort and click Apply Step 2 Changes to restore all peers in the fabric to the same configuration database and free the CFS lock. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 168: Resolving Lock Failure Issues Using The Cli

    An example of the application-name commit command follows: Switch# config terminal Switch(config)# ntp commit Switch(config)# An example of the application-name abort command follows: Switch# config terminal Switch(config)# ntp abort Switch(config)# Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 169: System State Inconsistent And Locks Being Held

    In the CLI, use the show cfs lock name application-name command to determine if a distribution is in progress on the fabric. If the application does not show in the output, the distribution has completed. Example command output follows: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 170: Cfs Regions Troubleshooting

    Switch# show cfs application name device-alias Enabled : Yes Timeout : 20s Merge Capable : Yes <<<<<< Application is capable of being merged. Scope : Physical-fc Region <<<<<< Application is in Region 1 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-10 OL-9285-05...

  • Page 171: Regions For Conditional Service

    “Merge Failure Troubleshooting” section on page 7-5 to identify and resolve the conflicts. Note When an application is moved from one region to another (including the default region), it loses all histories. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-11 OL-9285-05...
  • Page 172 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 173: Chapter 8 Troubleshooting Ports

    C H A P T E R Troubleshooting Ports This chapter describes how to identify and resolve problems that can occur with ports in the Cisco MDS 9000 Family of multilayer directors and fabric switches. It includes the following sections: Overview, page 8-1 •...
  • Page 174 Verify that no ports on a Generation 2 module are out of service. Use the show running interface CLI command to view the interface configuration in Cisco SAN-OS Note Release 3.0(1) or later. The interface configuration as seen in the show running-config CLI command is no longer consolidated.
  • Page 175 The fcdomain feature is disabled. manager disabled Isolation due to zone merge The zone merge operation failed. failure Isolation due to VSAN The VSANs at both ends of an ISL are different. mismatch Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 176: Limitations And Restrictions

    The interface is operating in TE mode when a port’s operational state is trunking. Overview of the FC-MAC Driver and the Port Manager This section describes the internal details of port related components in Cisco SAN-OS. Use this section to understand the underlying functions that may be causing port related problems.

  • Page 177: Port Manager Overview

    You must use the attach module CLI command to access these FC-MAC show commands. Note The FLOGI server is a separate application that handles the FLOGI processing for Nx ports. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 178: Troubleshooting Port States With The Device Manager

    For N ports, the port world-wide name (pWWN) and Fibre Channel ID (FC ID) of the connected • device For ISLs, the IP address of the connected switch • Speed • Frames transmitted and received • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 179: Device Manager: Port Selection

    Trunk Config—View and configure trunk mode and allowed VSANs. • Trunk Failure—Failure cause for ISLs. • Physical—Configure beaconing; view SFP information. • Capability—View current port capability for hold-down timers, BB credits, maximum receive buffer • size. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 180: Device Manager: Port Monitoring

    Class 2 Errors—View error statistics for Class 2 traffic, including busy frame responses and port • rejects. FICON—View FICON error statistics, including pacing, disparity, EOF, OOF, and order sets errors. • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 181: Isolating Port Issues Using Device Manager

    Step 3 If either of the ports fails to remain in the online state, then you may have a faulty GBIC, cabling or HBA/subsystem port. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 182: Troubleshooting Port States From The Cli

    Use the show hardware internal debug-info interface fc CLI command to debug ports. To issue commands with the internal keyword, you must have an account that is a member of the Note network-admin group. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-10 OL-9285-05...

  • Page 183: Useful Commands At The Fc-Mac Level

    Gives all non-zero statistics for the port. statistics show hardware internal fc-mac2 port port statistics show hardware internal fc-mac port port Displays the current state of the SFP. gbic-info show hardware internal fc-mac2 port port gbic-info Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-11 OL-9285-05...

  • Page 184: Isolating Port Issues Using The Cli

    -------------------- ---- -------- ----------------------- ----------------------- fc2/5 1 0x7e0200 21:00:00:e0:8b:08:d3:20 20:00:00:e0:8b:08:d3:20 fc2/7 1 0x7e0300 20:00:00:e0:69:41:98:93 10:00:00:e0:69:41:98:93 fc2/11 1 0x7e0100 21:00:00:e0:8b:07:ca:39 20:00:00:e0:8b:07:ca:39 fc2/14 1 0x7e0002 50:06:04:82:c3:a0:98:53 50:06:04:82:c3:a0:98:53 fc8/31 1 0x7e0000 50:06:04:82:c3:a0:98:42 50:06:04:82:c3:a0:98:42 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-12 OL-9285-05...

  • Page 185: Common Problems With Port Interfaces

    Nx port. This problem may be the result of one or more of the possible causes listed in Table 8-3. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-13 OL-9285-05...
  • Page 186 We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-14...

  • Page 187: Troubleshooting Port Problems

    MAC state indicates a loss of synchronization. In auto – Sync not acquired mode, this state is not necessarily an error. In any case, check the speed capabilities and configuration at both ends. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-15 OL-9285-05...

  • Page 188: Port Remains In Initializing State

    We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs Note crossing the EISL instead of just the VSAN experiencing the problem. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-16 OL-9285-05...

  • Page 189: Troubleshooting Port Registration Issues Using The Cli

    We recommend that you do not disable and then enable a T or TE port. This would affect all the Note VSANs crossing the EISL instead of just the VSAN experiencing the problem. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-17 OL-9285-05...
  • Page 190 /* Program the NameServer with wwn and FCID */ 6) FSM:<[99]21:00:00:e0:8b:07:a4:36> Transition at 330381 usecs after Sun Feb 1 04:18:15 1980 Previous state: [FLOGI_ST_PERFORM_CONFIG] Triggered event: [FLOGI_EV_ZS_CFG_RESPONSE] Next state: [FLOGI_ST_PERFORM_CONFIG] /* Response from ZoneServer */ Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-18 OL-9285-05...
  • Page 191 Feb 17 04:42:54 fcns: vsan 99: No configuration present for this portname Feb 17 04:42:54 fcns: vsan 99: No configuration present for this nodename /* Port is now registered in nameserver, will send out RSCN to it */ Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-19 OL-9285-05...
  • Page 192 SAN-OS debug facility. The command show fcns database detail vsan X displays a detailed list of all devices registered in the Note fabric. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-20 OL-9285-05...

  • Page 193: Unexpected Link Flapping Occurs

    ELP and to try to initialize as an xE port, even if an end device is physically connected to that interface. In this case, a port reason code of isolation because of ELP failure can be displayed even if an ISL is not present. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-21 OL-9285-05...
  • Page 194 We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs Note crossing the EISL instead of just the VSAN experiencing the problem. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-22 OL-9285-05...

  • Page 195: Link Initialization Flow

    They include: Active state. Link recovery state (LR): LR transmit substate (LR1) LR receive substate (LR2) LRR receive substate (LR3) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-23 OL-9285-05...
  • Page 196 NOS receive substate (LF1) NOS transmit substate (LF2) The Cisco MDS 9000 Family switch maintains port counters for link initialization ordered sets, including OLS, LRR, and NOS for fabric connections, as well as primitives for arbitrated loop connections on FL ports and TL ports. Understanding the link initialization flow and viewing the port counters using show interface can be useful when you troubleshoot port initialization problems.

  • Page 197: Viewing Port Counters

    Tx path from the switch to the Rx input on the N port interface is properly connected. A faulty transmitter on the switch’s SFP or a faulty receiver on the N port’s SFP could also cause the issue. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-25 OL-9285-05...

  • Page 198: Port Bounces Between Initializing And Offline States

    Analyze the event history provided by the Port Manager after using incompatibility in handling the the show port internal event-history CLI command. See the ELP process. “Troubleshooting ELP Issues Using the CLI” section on page 8-27. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-26 OL-9285-05...

  • Page 199: Troubleshooting Elp Issues Using The Cli

    To issue commands with the internal keyword, you must have an account that is a member of Note the network-admin group. switch# show port internal info interface fc2/1 fc2/1 - if_index: 1080000 Admin Config - state(up), mode(Auto), speed(auto), trunk(no trunk) beacon(off), snmp trap(on), tem(false) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-27 OL-9285-05...

  • Page 200: E Port Bounces Remains Isolated After A Zone Merge

    “Troubleshooting E port Isolation using Fabric Manager” section on The active zone set on both switches contains a zone with page 8-29. the same name but with different zone members. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-28 OL-9285-05...

  • Page 201: Troubleshooting E Port Isolation Using Fabric Manager

    Using the Zone Merge Analysis tool in Fabric Manager, the compatibility of two active zone sets in two switches can be checked before actually merging the two zone sets. Refer to the Cisco MDS 9000 Fabric Manager Configuration Guide for more information.

  • Page 202: Troubleshooting E Port Isolation Using The Cli

    We recommend that you do not disable and then enable a T or TE port. This would affect all the Note VSANs crossing the EISL instead of just the VSAN experiencing the isolation problem. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-30 OL-9285-05...

  • Page 203: Port Cycles Through Up And Down States

    Use the show interface command to verify that the switch detected a problem and disabled the port. Step 1 Check cables, SFPs, and optics. mds# show interface fc1/14 fc1/14 is down (errDisabled) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-31 OL-9285-05...

  • Page 204: Troubleshooting Fx Port Failure

    Typical end-user questions that lead to Fx port troubleshooting include: Why is no storage visible on my newly installed server? • Why is previously assigned storage not visible to my server after reboot? • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-32 OL-9285-05...
  • Page 205 Storage FLOGI to thier physical ports respective switch? Troubleshoot Server and storage registered Check VSAN with Name Server membership in all switches? Check zoning config and LUN masking Verify data path Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-33 OL-9285-05...
  • Page 206 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 207: Chapter 9 Troubleshooting N-Port Virtualization

    NPV mode do not join a fabric; rather, they pass traffic between NPV core switch links and end-devices, which eliminates the domain IDs for these edge switches. NPV is available only for the following Cisco MDS 9000 switches: the Cisco MDS 9124 Multilayer Note Fabric Switch, the Cisco MDS 9134 Fabric Switch, the Cisco Fabric Switch for HP c-Class BladeSystem and the Cisco Fabric Switch for IBM BladeCenter.

  • Page 208: Initial Troubleshooting Checklist

    Because the output is based on name server database information, the show fcns database npv Note commands can be run from any MDS switch running SAN-OS 3.2(1) or later. The switch does not need to be NPV enabled. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 209 For details about a specific node, including FCID, type and pWWN, enter the show fcns database npv node_wwn node_wwn command. switch# show fcns database npv node_wwn 20:00:00:0d:ec:3d:42:40 VSAN 1: -------------------------------------------------------------------------- FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------- 0x330f00 N 2f:ff:00:06:2b:10:c7:b2 (LSI) scsi-fcp:init 0x331000 N 2f:ff:00:06:2b:10:c7:b3 (LSI) scsi-fcp:init Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 210: Common Problems With Npv

    You can move the login of an end device from its existing uplink to the one with least number of logins by performing the following tasks: Step 1 Enter the shutdown command on the server interface that needs to be migrated. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 211: Npiv Is Not Enabled

    Number of External Interfaces: 1 Step 2 If the state is “Failed” with the reason “Mismatch in VSAN for this upstream port” then the external interface is configured to have different VSAN on both ends. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 212: Core Npv Device Is Not A Switch

    If the state is “Other,” verify that physical link to the NPV core switch or core port is in F port mode or is in administrative shutdown state. Server Interface is Down If the server interface is down: Check to ensure that the port is properly connected. Step 1 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 213: Waiting On Flogi From The Server Or Target

    Number of Server Interfaces: 7 If the State is “Waiting for External Interface,” check the status of external link. At least one external Step 2 link must be up for server interface to be up. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 214 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 215: Portchannel Overview

    265 PortChannels. A PortChannel number refers to the unique (to each switch) identifier associated with each channel group. This number ranges from of 1 to 256. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 10-1 OL-9285-05...

  • Page 216: C H A P T E R 10 Troubleshooting Portchannels And Trunking

    When trunking is operational on an E port, that E port becomes a TE port. A TE port is specific to switches in the Cisco MDS 9000 Family. An industry standard E port can link to other vendor switches and is referred to as a nontrunking interface.

  • Page 217: Common Troubleshooting Commands In The Cli

    Disable autocreation if you want to manually configure PortChannel. PortChannels. In Device Manager, select Interfaces > FC ALL..., select the Other tab, uncheck the AutoChannelCreate check box, and click Apply. Use the no channel-group auto CLI command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 10-3 OL-9285-05...

  • Page 218: Newly Added Interface Does Not Come Online In A Portchannel

    Step 3 Trunking Issues This section describes common trunking issues and includes the following topics: Cannot Configure Trunking, page 10-5 • VSAN Traffic Does Not Traverse Trunk, page 10-5 • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 10-4 OL-9285-05...

  • Page 219: Cannot Configure Trunking

    Switches > Interfaces > FC Logical, select the Trunk Config tab, and set the Allowed VSANs field. Click Apply Changes. Use the switchport trunk allowed vsan CLI command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 10-5 OL-9285-05...
  • Page 220 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 221: Overview

    VSANs, zoning, FCdomain, admin issues, or switch-specific or fabric-specific issues. Fabric Manager provides the configuration consistency check tool. Refer to the Cisco MDS 9000 Fabric Manager Configuration Guide for more information about this tool. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 222: C H A P T E R 11 Troubleshooting Vsans, Domains, And Fspf

    Use the following CLI commands to display VSAN, FC domain, and FSPF information: • show vsan • show vsan vsan-id show vsan membership • show interface fc slot/port trunk vsan-id • show vsan-id membership • show vsan membership interface fc slot/port • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-2 OL-9285-05...

  • Page 223: Vsan Issues

    “E Port Is Isolated in a VSAN” section on is isolated. page 11-5. Host and storage are not in the same See the “Zone and Zone Set Issues” section on page 14-4. zone. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-3 OL-9285-05...

  • Page 224: Verifying Vsan Membership Using Fabric Manager

    Step 3 to transport the VSAN found in Step switch# show interface fc2/14 fc2/14 is trunking Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e Port mode is TE Speed is 2 Gbps Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-4 OL-9285-05...

  • Page 225: E Port Is Isolated In A Vsan

    Using the CLI” section on page 11-9. Port parameters misconfigured. See the “Common Problems with Port Interfaces” section on page 8-13. Zoning mismatch. Chapter 14, “Troubleshooting Zones and Zone Sets.” Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-5 OL-9285-05...

  • Page 226: Resolving An Isolated E Port Using Fabric Manager

    This sample output shows that all the interfaces on the switch belong to VSAN 3, with the exception of interface fc2/5 and fc2/13, which are part of VSAN 4. Use the vsan database vsan vsan-id interface command to move the ports into the same VSAN. Step 3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-6 OL-9285-05...

  • Page 227: Resolving An Isolated Isl Using Fabric Manager

    Received 30 OLS, 21 LRR, 18 NOS, 53 loop inits The example shows the output of the show interface command with one or more isolated VSANs. Here, the TE port has one VSAN isolated. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-7 OL-9285-05...
  • Page 228 “DPVM Config Database Not Activating” section on page 11-14 domain misconfiguration problems. Use the vsan vsan-id interface command to correct the VSAN misconfiguration problems. Repeat this procedure for all isolated VSANs on this TE port. Step 5 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-8 OL-9285-05...

  • Page 229: Resolving Fabric Timer Issues Using Fabric Manager

    VSAN to maintain fabric topology when a host or storage device connection is moved between two switches or between ports on the same switch. It retains the configured VSAN regardless of where a device is connected or moved. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-9 OL-9285-05...

  • Page 230: Troubleshooting Dpvm Using Fabric Manager

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Verify the following requirements when using DPVM: The interface through which the dynamic device connects to the Cisco MDS switch must be •...

  • Page 231: Troubleshooting Dpvm Using The Cli

    CLI command to verify that DPVM is not enabled. Set the Status field to enable in Fabric Manager and then click Apply Changes or use the dpvm enable CLI command to enable DPVM. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-11 OL-9285-05...

  • Page 232: Dpvm Database Not Distributed

    CLI commands to create the DPVM active database. When DPVM distribution is enabled, you must do an explicit commit for DPVM activate and autolearn Note to take effect. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-12 OL-9285-05...

  • Page 233: No Autolearn Entries In Active Database

    Or use the show dpvm pending CLI command to determine if there are uncommitted changes. Use the dpvm database and dpvm commit CLI commands to commit any pending changes. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-13 OL-9285-05...

  • Page 234: Dpvm Config Database Not Activating

    Fabric Manager and then click Apply Changes. Or use the dpvm activate and dpvm commit CLI commands to create the DPVM active database. Then copy the active database again. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-14 OL-9285-05...

  • Page 235: Port Suspended Or Disabled After Dpvm Activation

    DPVM configuration in both fabrics. Manually reconcile any differences before attempting to merge the fabrics. Use the show cfs merge status name dpvm CLI command to verify the merge status. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-15 OL-9285-05...

  • Page 236: Dpvm Process Terminates

    Use the dpvm commit command in config mode to commit the changes to the config database. Step 2 Step 3 Use the no dpvm enable command in config mode to disable DPVM on the switch. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-16 OL-9285-05...

  • Page 237: Domain Issues

    In Cisco SAN-OS Release 2.1(1a) and later releases, when powering up a new switch in a • multi-switch fabric, a BF occurs and the switch with the better priority becomes the principal switch.

  • Page 238: Switch Cannot See Other Switches In A Vsan

    Use the fcdomain domain domain-id [static | preferred] vsan vsan-id CLI command or similar Fabric Manager procedure to change the domain ID for one of the overlapping domain IDs. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-18 OL-9285-05...

  • Page 239: Assigning A New Domain Id Using Fabric Manager

    0 EOF abort, 0 fragmented, 0 unknown class 231 frames output, 3709 bytes, 16777216 discards Received 28 OLS, 19 LRR, 16 NOS, 48 loop inits Transmitted 62 OLS, 22 LRR, 25 NOS, 30 loop inits Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-19 OL-9285-05...

  • Page 240: Using Fabric Reconfiguration For Domain Id Assignments

    (RCF) occurs. The RCF functionality would automatically force a new principal switch selection and cause new domain IDs to be assigned to the different switches. A disruptive reconfiguration might affect data traffic. Caution Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-20 OL-9285-05...
  • Page 241 Configured domain ID: 0x64(100) (preferred) Principal switch run time information: Running priority: 2 Interface Role RCF-reject ---------------- ------------- ------------ fc2/1 Downstream Enabled fc2/2 Downstream Disabled fc2/7 Upstream Disabled ---------------- ------------- ------------ Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-21 OL-9285-05...

  • Page 242: Cfs Distribution Of Domain Id List Fails

    ID list. Compare this to any other switches in the VSAN to determine what domain IDs are missing. Use the fcdomain allowed CLI command to add any missing domain IDs. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-22 OL-9285-05...

  • Page 243: Allowed Domain Id List Incorrect After A Vsan Merge

    Restart drop-down menu and click Apply Changes. If CFS is enabled, then select the CFS tab and select commit from the ConfigAction drop-down menu and click Apply Changes. Or use the fcdomain restart CLI command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-23 OL-9285-05...

  • Page 244: Fspf Issues

    Metric 1000 Metric 1000 0x00010001 0x00010003 Port 3 Port 4 Index Index 0x00010002 Domain_ID 0x00010003 Switch5 For the purpose of this example, assume that all interfaces are located in VSAN 1. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-24 OL-9285-05...

  • Page 245: Troubleshooting Fspf Using Device Manager

    1000 <-----5 0x00010003 0x00010002 1000 <-----6 FSPF Link State Database for VSAN 2 Domain 237 <-----------LSR for another switch LSR Type Advertising domain ID = 237 <-----7 LSR Age = 185 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-25 OL-9285-05...
  • Page 246 <-----2 Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5 s <-----3 FSPF State is FULL <-----4 Neighbor Domain Id is 1, Neighbor Interface index is 0x00010002 <-----5 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-26 OL-9285-05...
  • Page 247 The next hop (238) has two interfaces. This indicates that both paths will be used during load sharing. Up to sixteen paths can be used by FSPF with a Cisco MDS 9000 Family switch. With the implementation of VSANs used with Cisco MDS 9000 Family switches, a separate instance of FSPF runs within each VSAN, and each instance is independent of the others.

  • Page 248: Loss Of Two-Way Communication

    Step 1 to determine the value of the hello interval on the adjacent switch. Step 2 Fill in the Hello field to change the hello interval and click Apply. Step 3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-28 OL-9285-05...

  • Page 249: Resolving A Wrong Hello Interval On An Isl Using The Cli

    Hello timer on the adjacent switch. switch2# show fspf v 1 interface fc2/16 FSPF interface fc2/16 in VSAN 1 FSPF routing administrative state is active Interface cost is 500 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-29 OL-9285-05...

  • Page 250: Resolving A Mismatched Retransmit Interval On An Isl Using Device Manager

    The retransmit interval is not set to the default, so you should check the neighbor configuration to make sure it matches. FSPF is not in FULL state, indicating a problem. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-30 OL-9285-05...

  • Page 251: Resolving A Mismatch In Dead Intervals On An Isl Using Fabric Manager

    If the debug output overwhelms the current session, you can use the second session to enter the undebug all command to stop the debug message output. Use the undebug all command to turn off debugging. Step 2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-31 OL-9285-05...

  • Page 252: Resolving A Region Mismatch Using Fabric Manager

    /* This is the region */ SPF hold time is 0 msec MinLsArrival = 1000 msec , MinLsInterval = 5000 msec Local Domain is 0x78(120) Number of LSRs = 2, Total Checksum = 0x000133de Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-32 OL-9285-05...
  • Page 253 Use the fspf config vsan command to enter the FSPF configuration mode and use the region command to change the region. The region must match on all switches in the VSAN. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-33 OL-9285-05...
  • Page 254 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 255: Chapter 12 Troubleshooting San Device Virtualization

    C H A P T E R Troubleshooting SAN Device Virtualization This chapter describes how to troubleshoot and resolve SAN device virtualization (SDV) configuration issues in the Cisco MDS 9000 Family of multilayer directors and fabric switches. It includes the following sections: Overview, page 12-1 •...

  • Page 256: Debugging And Verifying Sdv Configuration Using The Cli

    Ensure that the virtual devices are zoned correctly. Activate the zone set. For a host/virtual device connected to a Cisco MDS 9124 Fabric Switch, ensure that there is a rewrite-capable SDV-enabled director switch in the path. This section includes the following topics: Debugging and Verifying SDV Configuration Using the CLI, page 12-2 •...

  • Page 257: Sdv Issues

    There must be at least one rewrite-capable SDV-enabled MDS switch located between the server and • the target that is being virtualized. The Cisco MDS 9124 Fabric Switch is not a rewrite-capable switch. In other words, SDV does not work when real devices and primary virtual devices are connected to the same Cisco MDS 9124 Fabric Switch.

  • Page 258: Sdv Commit Partially Fails

    View the syslog and identify the reason for the already in use. failure; rectify the problem and reissue the commit command. FCID cannot be assigned because the domain cannot be reserved. CFS distribution failure. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 12-4 OL-9285-05...

  • Page 259: Host Cannot Locate Disk

    If the host or virtual device is connected to the Cisco MDS 9124 Fabric Switch, check the director at the next hop to confirm that the rewrite and capture entries are programmed correctly on all the trunk ports to the switch.

  • Page 260: Sdv Merge Fails When Isl Comes Up

    If a zone activation fails in a SDV zone, enter the show zone internal sdv-table command to view the physical-virtual mapping maintained in the zone server. Devices should be zoned so that they cannot communicate to both the real device and its virtual component. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 12-6 OL-9285-05...

  • Page 261: Troubleshooting Ivr

    Configuration problems with IVR can prevent devices from communicating properly. Prior to Cisco MDS SAN-OS Release 2.1(1a), IVR required unique domain IDs for all switches in the fabric. As of Cisco MDS SAN-OS Release 2.1(1a), you can enable IVR Network Address Translation (NAT) to allow non-unique domain IDs.

  • Page 262: Chapter 13 Troubleshooting Ivr

    The VSAN topology configuration must be updated before a border switch is added or removed. • Limitations and Restrictions The following limitations apply to IVR: IVR is not supported on the Cisco MDS 9124 Fabric Switch, the Cisco Fabric Switch for HP c-Class • BladeSystem, and the Cisco Fabric Switch for IBM BladeCenter. •...

  • Page 263: Initial Troubleshooting Checklist

    Verify that you have added IVR virtual domains to the allowed domain ID list if you have a Cisco SN5428 storage router or a Cisco MDS 9020 switch in your fabric. If you change any FSPF link cost, ensure that the FSPF path cost (that is, the sum of the link costs on the path) of any IVR path is less than 30,000.

  • Page 264: Verifying Ivr Configuration Using The Cli

    Shows IVR virtual domains for all local VSANs. show ivr Shows IVR virtual fcdomain status. virtual-fcdomain-add-st atus show ivr vsan-topology Verifies the configured IVR topology. show ivr zoneset Verifies the IVR zone set configuration. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-4 OL-9285-05...

  • Page 265: Ivr Enhancements By Cisco San-Os Release

    Show IVR VSAN Topology graph internal debug information zone-fsm Show ivr zone fsm internals IVR Enhancements by Cisco SAN-OS Release Table 13-2 lists the IVR enhancements by Cisco SAN-OS release. Table 13-2 IVR Enhancements by Cisco SAN-OS Release Cisco SAN-OS Release IVR Enhancement Release 3.3(1)

  • Page 266: Ivr Issues

    IVR-enabled switch in your fabric. Table 13-3 shows which license to purchase based on the IVR feature you are using and the module or chassis you have enabled IVR on. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-6 OL-9285-05...

  • Page 267: Cannot Enable Ivr

    SAN_EXTN_OVER_IPS4 1. Cisco MDS 9216i enables the SAN_EXTENSION features without a license for the two Gigabit Ethernet ports on the integrated supervisor card. If you are using IVR over FCIP and Fibre Channel, you need the ENTERPRISE_PKG as well as the...

  • Page 268: Ivr Network Address Translation Fails

    ID. header. If this same destination ID appears inside the message payload, Cisco SAN-OS may not detect it and IVR NAT fails. Disable IVR NAT and ensure that all domain IDs are unique. Refer to the Cisco MDS 9000 Family configuration guides at...
  • Page 269 No zone set has been activated. See the “Troubleshooting Zone Set Activation” section on page 14-8 to activate a zone set on an IVR-enabled switch, or use the force option when activating the IVR zone set. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-9 OL-9285-05...

  • Page 270: Border Switch Fails

    Or use the ivr vsan topology auto CLI command to automatically reconfigure the IVR topology, or use the ivr vsan topology database CLI command to manually reconfigure the IVR topology. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-10 OL-9285-05...

  • Page 271: Traffic Does Not Traverse Ivr Path

    Or use the ivr service-group CLI command. Use the ivr service-group activate CLI command to activate this change. If CFS is enabled, use the ivr commit CLI command to commit this change. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-11 OL-9285-05...

  • Page 272: Link Isolated

    CLI command to map the pWWN to an appropriate FC ID that matches the virtual domain ID. Refer to the Cisco MDS 9000 Family configuration guides for the related procedure to configure Persistent FC IDs for IVR.

  • Page 273: Lun Configuration Failure In Ivr Zoning

    Possible Cause Solution LUN configuration One or more switches in the VSAN Upgrade to the Cisco SAN-OS release required for the IVR failed in IVR zoning. are not running Cisco MDS SAN-OS features you want to use. See Table 13-1 Chapter 2, Release 2.1(1a) or later.

  • Page 274: Cfs Merge Failed

    If you purchased Cisco support through a Cisco reseller, contact the reseller Recommended Action directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support. Cisco MDS SAN-OS Release 2.0(1b). Introduced Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 275: Troubleshooting The Ivr Wizard

    The IVR wizard in Fabric Manager simplifies the process of configuring IVR across your fabric. The IVR wizard automatically checks for the appropriate Cisco SAN-OS version across the switches in the VSAN and determines which IVR features the switches are capable of. (See Table 13-1.)

  • Page 276: Warning: Not All Switches Are Ivr Nat Capable Or Are Unmanageable

    Possible Cause Solution Warning: Not all One or more switches in the fabric Upgrade to the Cisco SAN-OS release required for the IVR switches are IVR are not running Cisco MDS SAN-OS features you want to use. See Table 13-1...

  • Page 277: Error: Pending Action/ Pending Commits

    Changes. Error: Fabric Is Changing. Please Retry the Request Later This error may occur if there are different versions of Cisco SAN-OS on the IVR-enabled switches. You should upgrade all IVR-enabled switches to the same version of Cisco SAN-OS. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...
  • Page 278 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 279: Chapter 14 Troubleshooting Zones And Zone Sets

    Troubleshooting Zones and Zone Sets This chapter describes how to identify and resolve problems that might occur while implementing zones and zone sets on switches in the Cisco MDS 9000 Family. It includes the following sections: Overview, page 14-1 •...

  • Page 280: Troubleshooting Zone Configuration Issues With Fabric Manager

    For zone configuration problems, use the following helpful tools: Cisco Fabric Analyzer. (See the “Cisco Fabric Analyzer” section on page B-25.) • Cisco Fabric Manager and CLI system messages. (See the System Messages, page 1-10.) • Log messages (See the “Troubleshooting with Logs”...
  • Page 281 Number of devices zoned in vsan: 0/2 (Unzoned: 2) Number of zone members resolved: 0/2 (Unresolved: 2) Num zones: 1 Number of IVR zones: 0 Number of IPS zones: 0 Formattted size: 38 bytes / 2048 Kb Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-3 OL-9285-05...

  • Page 282: Zone And Zone Set Issues

    Num attribute groups: 0 Formattted size: 20 bytes / 2048 Kb See the Cisco MDS 9000 Family Command Reference for the description of the information displayed in the command output. The debug zone change CLI command followed by the zone name in question can help you get started debugging zones for protocol errors, events, and packets.

  • Page 283: Host Cannot Communicate With Storage

    If the zone is not in the active zone set, see the “Resolving Zone is Not in Active Zone Set Using Fabric Manager” section on page 14-6. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-5 OL-9285-05...

  • Page 284: Resolving Host Not Communicating With Storage Using The Cli

    Using the CLI” section on page 11-4. Configure zoning, if necessary, by using the show zone status vsan-id command to determine if the Step 2 default zone policy is set to deny. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-6 OL-9285-05...
  • Page 285 Use the show zone command to verify that host and storage are now in the same zone. Step 2 switchA# show zone zone name NewZoneName vsan 2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-7 OL-9285-05...

  • Page 286: Troubleshooting Zone Set Activation

    A zone that is part of an active zone set is called an active zone. Two main problems can occur with activating a zone set: No zone set is active. • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-8 OL-9285-05...

  • Page 287: Troubleshooting Zone Activation Using Fabric Manager

    Zone activation can fail if a new switch joins the fabric. When a new switch joins the fabric, it acquires the existing zone sets. Also, large zone sets may experience timeout errors in Cisco MDS SAN-OS Release 1.3(4a) and earlier.

  • Page 288: Troubleshooting Zone Activation Using The Cli

    Optionally, use the zoneset name ActiveZonesetName vsan-id command and the member NewZone Step 5 command to add the zone to the active zone set in the VSAN. switch(config)# zoneset name ZoneSet1 vsan 2 switch(config-zoneset)# member NewZoneAdded Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-10 OL-9285-05...

  • Page 289: Troubleshooting Full Zone Database Synchronization Across Switches

    Troubleshooting Full Zone Database Synchronization Across Switches All switches in the Cisco MDS 9000 Family distribute active zone sets when new E port links come up or when a new zone set is activated in a VSAN. The zone set distribution takes effect while sending merge requests to the adjacent switch or while activating a zone set.

  • Page 290: Mismatched Default Zone Policy

    Choose Fabricxx > VSANxx and view the Release field to verify that all switches are capable of working in the enhanced mode. All switches must have Cisco MDS SAN-OS Release 2.0(1b) or later. If one or more switches are not capable of working in enhanced mode, then your request to move to enhanced mode is rejected.

  • Page 291: Resolving Mismatched Default Zone Policies Using The Cli

    All switches must have Cisco MDS SAN-OS Release 2.0(1b) or later. If one or more switches are not capable of working in enhanced mode, then your request to move to enhanced mode is rejected.
  • Page 292 Full zoning databases are inconsistent between two switches connected by the Explanation interface. Databases are not merged. Compare full zoning database with the adjacent switch, correct the difference Recommended Action and flap the link. Cisco MDS SAN-OS Release 1.2(2a). Introduced Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-14 OL-9285-05...

  • Page 293: Recovering From Link Isolation

    Using the Zone Merge Analysis tool in Fabric Manager, the compatibility of two active zone sets in two switches can be checked before actually merging the two zone sets. Refer to the Cisco MDS 9000 Fabric Manager Configuration Guide for more information.

  • Page 294: Resolving A Link Isolation Because Of A Failed Zone Merge Using The Cli

    The import option of the command overwrites the local switch’s active zone set with that of the remote switch. The export option overwrites the remote switch’s active zone set with the local switch’s active zone set. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-16 OL-9285-05...

  • Page 295: Mismatched Active Zone Sets Within The Same Vsan

    VSAN. Step 2 A mismatched active zone set may include zones with the same name but different members, or a missing zone within the zone set. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-17 OL-9285-05...

  • Page 296: Resolving Mismatched Active Zone Sets Within The Same Vsan Using The Cli

    5 minutes input rate 120 bits/sec, 15 bytes/sec, 0 frames/sec 5 minutes output rate 88 bits/sec, 11 bytes/sec, 0 frames/sec 10845 frames input, 620268 bytes, 0 discards 0 CRC, 0 unknown class 0 too long, 0 too short Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-18 OL-9285-05...
  • Page 297 Use the show interface fcx/y trunk vsan-id command to verify that VSAN 99 is no longer isolated: Step 6 Switch1# show interface fc1/5 trunk vsan 99 fc1/5 is trunking Vsan 99 is up, FCID is 0x780102 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-19 OL-9285-05...

  • Page 298: Deactivating A Zone Set And Restarting The Zone Merge Process Using Fabric Manager

    Use the no zoneset activate name zoneset-name vsan-id command to deactivate the zone set Step 1 configuration from the switch: This will disrupt traffic and cause the MDS 9000 switch to lose connectivity with the network. Caution Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-20 OL-9285-05...

  • Page 299: Enhanced Zoning Issues

    VSAN, that switch will lock the fabric to prevent others from making zoning changes. The user must issue a commit to make the changes active and release the fabric wide lock. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-21...
  • Page 300 Another user on a different switch is holding the enhanced zoning configuration lock. If you are using the CLI, you see a message stating that the lock is currently busy. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-22 OL-9285-05...

  • Page 301: Resolving Enhanced Zoning Lock Issues With Fabric Manager

    If problems persist, use the clear zone lock command to remove the lock from the switch. This should Step 4 only be done on the switch that holds the lock. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-23 OL-9285-05...
  • Page 302 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 303: Chapter 15 Troubleshooting Distributed Device Alias Services

    • Overview When you configure different features for a Cisco MDS 9000 Family switch, such as zoning, DPVM, or port security, you must specify the correct device name each time. An inaccurate device name can cause unexpected results. You can circumvent this problem by defining and using device aliases.

  • Page 304: Merge Failure Messages

    VSAN Expln: Example 15-3 Message on Switch Rejecting the Validation 2007 Apr 10 00:00:06 switch-2 %DEVICE-ALIAS-3-MERGE_VALIDATION_REJECTED: Failed SAP: 110 Reason: inter-VSAN zone member cannot be in more than one VSAN Expln: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-2 OL-9285-05...

  • Page 305: Commit Failure Messages

    The following limitations and restrictions are associated with the use of device aliases: Enhanced mode operation is only supported on switches running SAN-OS Release 3.1(1) or later. • • Interop mode VSANs do not accept enhanced mode (native alias-based) configurations. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-3 OL-9285-05...

  • Page 306: Merge Failure Issues

    15-9. The application is busy handling its own merge. The application database is locked due to a defect in the application. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-4 OL-9285-05...

  • Page 307: Resolving Duplicate Device Alias Names

    A1 pwwn 21:01:01:01:01:01:01:02 Total number of entries = 1 Make the appropriate changes to the device alias database for one of the fabrics. Refer to the Cisco MDS Step 4 9000 Family CLI Configuration Guide for details.

  • Page 308: Resolving Mapping A Pwwn To Different Device Alias Names

    A1 pwwn 21:01:01:01:01:01:01:02 Total number of entries = 1 Step 4 Make the appropriate changes to the device alias database for one of the fabrics. Refer to the Cisco MDS 9000 Family CLI Configuration Guide for details. Resolving Mode Mismatch The device alias feature can operate in either basic or enhanced mode.

  • Page 309: Resolving Merge Failures In Mixed Fabric

    Refer to the Cisco MDS 9000 Family CLI Configuration Guide for details.

  • Page 310: Resolving A Validation Failure

    Validation timer: Per SAP Info Table: =================== SAPS: MTS Buffer Array Details: ========================= Buffers: Local Status: ============= Num Reqs Sent: 0 20:00:00:0d:ec:04:99:40 Num SAPs Done: Failed SAP Status: success Expln: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-8 OL-9285-05...

  • Page 311: Resolving Merge In Progress Issues

    When a merge occurs, the merged device alias database is validated with the registered applications on each switch in both of the fabrics. During this process, an application database may be in a locked state. Common causes are listed in Table 15-2. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-9 OL-9285-05...
  • Page 312 20:00:00:0d:ec:04:99:40 (switch-2) has rejected the validation. The explanation indicates that the IVR application has rejected the validation because it is busy. switch-1# show device-alias internal validation-info Validation timer: Per SAP Info Table: =================== SAPS: MTS Buffer Array Details: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-10 OL-9285-05...

  • Page 313: Validation And Commit Failure Issues

    The commit process includes validation of the database change by all registered applications on all switches in the fabric. This section identifies the common problems that may cause validation and commit failures. See Table 15-3. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-11 OL-9285-05...

  • Page 314: Resolving Database Conflicts

    2007 Apr 10 19:13:08 switch-2 %DEVICE-ALIAS-3-VALIDATION_REJECTED: Failed SAP: 110 Reason: inter-VSAN zone member cannot be in more than one VSAN ==>SAP and reason Expln: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-12 OL-9285-05...

  • Page 315: Resolving Application Busy Situations

    2007 Apr 10 02:30:55 switch-2 %DEVICE-ALIAS-3-VALIDATION_FAILED: Failed SAP: ==>Status 110 Reason: Some of the registered modules are busy handling other requests. ==>Reason Please retry the command after some time. Expln: there is a pending action. ==>Resolution Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-13 OL-9285-05...

  • Page 316: Resolving Database Size Issues

    Some of the switches in the fabric are running a version of software which cannot support either the issued command or maximum device-alias limits. Please fix those switches and retry the command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-14 OL-9285-05...

  • Page 317: Resolving Mode Issues

    Some of the switches in the fabric are running a version of software which cannot support enhanced mode. Please fix those switches and retry the command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-15 OL-9285-05...
  • Page 318 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 319: Chapter 16 Troubleshooting Ficon

    • FICON Overview The Cisco MDS 9000 Family supports the Fibre Channel, FICON, iSCSI, and FCIP capabilities within a single, high-availability platform. Fibre Channel and FICON are different FC4 protocols and their traffic are independent of each other. If required, devices using these protocols can be isolated using VSANs.

  • Page 320: Ficon Port Numbering

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m The FICON feature is not supported on Cisco MDS 9120, 9124 or 9140 switches, the 32-port Fibre...
  • Page 321 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Table 16-1 Default FICON Port Numbering in the Cisco MDS 9000 Family Implemented Port Allocation...

  • Page 322: Reserved Ficon Port Numbering Scheme

    You can use the ficon slot assign port-numbers CLI command to use any excess ports by assigning Note numbers to the slots. Before you assign numbers to the slots however, we recommend that you review the default port number assignments for Cisco MDS 9000 switches shown in Table 16-1.

  • Page 323: Ficon Configuration Files

    The file format is proprietary to IBM. These files can be read and written by IBM hosts using the in-band CUP protocol. Additionally, you can use the Cisco MDS CLI or Fabric Manager applications to modify these FICON configuration files. When you enable the FICON feature in a VSAN, the switches always use the startup FICON configuration file, called IPL.

  • Page 324: Cup In-Band Management

    The CUP specification is proprietary to IBM. Note CUP is supported by switches and directors in the Cisco MDS 9000 Family. The CUP function allows the mainframe to manage the Cisco MDS switches. Host communication includes control functions, such as blocking and unblocking ports, as well as monitoring and error reporting functions.

  • Page 325: Initial Troubleshooting Checklist

    Begin troubleshooting FICON issues by checking the following issues: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that you enabled in-order delivery for the FICON-enabled VSAN. Verify that you have enabled fabric binding in all switches in the FICON fabric.

  • Page 326: Common Troubleshooting Tools In Fabric Manager Or Device Manager

    For example, if a port with BB_credits as 25 is being swapped with an OSM port for which a maximum of 12 BB_credits is allowed (not a configurable parameter), the port swapping operation is rejected. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-8 OL-9285-05...

  • Page 327: Swapping Ports

    Cannot Enable FICON Port, page 16-11 • • Cannot Configure FCIP or PortChannel for FICON, page 16-12 • FCIP fails for FICON, page 16-12 • FICON Tape Acceleration Not Working, page 16-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-9 OL-9285-05...

  • Page 328: Cannot Enable Ficon

    Switch is configured to disallow SNMP Use the snmp port control CLI command in FICON Device Manager control over FICON. submode to enable SNMP control for FICON. cannot configure FICON. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-10 OL-9285-05...

  • Page 329: Mainframe Cannot Configure Ficon

    Choose FICON > Port Numbers in Device Manager. Alternatively, use the ficon slot slot number assign port-numbers CLI command to change the port number. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-11 OL-9285-05...

  • Page 330: Cannot Configure Fcip Or Portchannel For Ficon

    Choose Switches > ISLs > FCIP in Fabric Manager, working. then select the Tunnels (Advanced) tab and check Write Accelerator. Or use the ficon-tape-accelerator vsan CLI command on each FCIP interface. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-12 OL-9285-05...

  • Page 331: Chapter 17 Troubleshooting Radius And Tacacs

    The authentication, authorization, and accounting (AAA) mechanism verifies the identity of, grants access to, and tracks the actions of users managing a switch. All Cisco MDS 9000 Family switches use the Remote Access Dial-In User Service (RADIUS) and Terminal Access Controller Access Control System Plus (TACACS+) protocols to provide solutions using remote AAA servers.

  • Page 332: Common Troubleshooting Tools In Fabric Manager

    User Cannot Access Certain Features, page 17-11 • Switch Does Not Communicate with AAA Server Multiple misconfigurations can result in an AAA server that the Cisco SAN-OS switch does not communicate with. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...
  • Page 333 For TACACS+ servers, see the “Verifying TACACS+ Server Monitor Configuration Using Fabric Manager” section on page 17-7 or the “Verifying TACACS+ Server Monitor Configuration Using the CLI” section on page 17-7. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-3 OL-9285-05...

  • Page 334: Verifying Radius Configuration Using Fabric Manager

    RADIUS servers before the switch declares a timeout failure. Use the radius commit command to commit any changes and distribute to all switches in the fabric. Step 6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-4 OL-9285-05...

  • Page 335: Verifying Tacacs+ Configuration Using Fabric Manager

    TACACS+ servers before the switch declares a timeout failure. Use the tacacs commit command to commit any changes and distribute to all switches in the fabric. Step 5 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-5 OL-9285-05...

  • Page 336: Verifying Radius Server Monitor Configuration Using Fabric Manager

    Use the radius-server deadtime command to configure the time that the switch waits before retesting Step 3 a dead server. Use the radius commit command to commit any changes and distribute to all switches in the fabric. Step 4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-6 OL-9285-05...

  • Page 337: Verifying Tacacs+ Server Monitor Configuration Using Fabric Manager

    Use the tacacs-server deadtime command to configure the time that the switch waits before retesting a dead server. Use the tacacs commit command to commit any changes and distribute to all switches in the fabric. Step 4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-7 OL-9285-05...

  • Page 338: User Authentication Fails

    17-9. For TACACS+ servers, see the “Verifying TACACS+ Server Groups Using Fabric Manager” section on page 17-9 or the “Verifying TACACS+ Server Groups Using the CLI” section on page 17-10. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-8 OL-9285-05...

  • Page 339: Verifying Radius Server Groups Using Fabric Manager

    Set the Server List field to a comma-separated list of TACACS+ servers. Step 3 Step 4 Set the Deadtime field to configure the time that the switch waits before retesting a dead server. and click Apply to save these changes. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-9 OL-9285-05...

  • Page 340: Verifying Tacacs+ Server Groups Using The Cli

    Cisco-AVPair = shell:roles=" rolename1 rolename2" For TACACS+, configure the attribute and value pair on the server for the role using: roles=" rolename1 rolename2" Verify that all roles are defined on the switch. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-10 OL-9285-05...

  • Page 341: User Cannot Access Certain Features

    Choose User Setup > User Data Configuration to verify that the user is configured. Step 2 View the Cisco IOS/PIX RADIUS Attributes setting for a user. Verify that the user is assigned the correct Step 3 roles in the AV-pairs. For example, shell:roles=”network-admin”...
  • Page 342 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Refer to the User guide for Cisco Secure ACS at the following website for more information: http://cisco.com/en/US/products/sw/secursw/ps2086/products_user_guide_list.html...

  • Page 343: Chapter 18 Troubleshooting Users And Roles

    Overview The CLI and SNMP use common roles in all switches in the Cisco MDS 9000 Family. You can use the CLI to modify a role that was created using SNMP and vice versa. A user configured through the CLI can access the switch using SNMP (for example, Fabric Manager or Device Manager) and vice versa.

  • Page 344: Role-Based Authorization

    • If a password is trivial (short, easy-to-decipher), your password configuration is rejected. Passwords are case-sensitive. The default password for any Cisco MDS 9000 Family switch is no longer “admin”. You must explicitly configure a strong password. Clear text passwords can only contain alphanumeric characters. Special characters such as the dollar sign Note ($) or the percent sign (%) are not allowed.

  • Page 345: Rules And Features For Each Role

    If you had swapped these two rules and issued the deny config feature fspf rule first and issued the permit config rule next, you would be allowing the user to perform all configuration commands because the second rule globally overrode the first rule. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-3 OL-9285-05...

  • Page 346: Initial Troubleshooting Checklist

    Begin troubleshooting user and role issues by checking the following issues: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that the passwords for all users follow the guidelines for strong passwords. Verify that no usernames are reserved words or all numeric.

  • Page 347: User Cannot Log Into Switch

    Step 1 Select debug from the Severity Level drop-down menu for auth, authPriv, and aaad. Click Apply. Step 2 This sets the switch to log debug information for these facilities. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-5 OL-9285-05...

  • Page 348: Verifying User Login With System Messages Using The Cli

    This sets the switch to save system messages at the debug level or above in the TestFile log file. At this point, all future login attempts are tracked in the log file. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-6...

  • Page 349: User Cannot Create Roles

    If you have logged in as a network-admin using Note MDS authentication, Device Manager and Fabric Manager automatically provide the appropriate encryption for this task, even if you did not specify a specific privacy password. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-7 OL-9285-05...

  • Page 350: User Cannot Access Certain Features

    Right-click a role and select Rules to view or modify the rules assigned to a role. Step 7 Check the feature check boxes for the features that you want this role to access and click Apply to save Step 8 these changes. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-8 OL-9285-05...

  • Page 351: Verifying Roles Using The Cli

    Use the role command to modify the rules assigned to a role. Step 4 switch# role name sangroup switch(config-role)# no rule 4 switch(config-role)# rule 4 deny exec feature fcping Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-9 OL-9285-05...

  • Page 352: User Has Too Much Access

    Check the Scope Enable check box to make the role VSAN-restricted. Step 2 Add the range of VSANs that you want to allow this role to configure in the Scope VSAN Id List field. Step 3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-10 OL-9285-05...

  • Page 353: Verifying Vsan-Restricted Roles Using The Cli

    User is assigned a VSAN-restricted See the “Verifying VSAN-Restricted Roles Using Fabric E ports. role. Manager” section on page 18-10 or the “Verifying VSAN-Restricted Roles Using the CLI” section on page 18-11. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-11 OL-9285-05...

  • Page 354: Unexpected User Displayed In Logs

    Choose User Setup > User Data Configuration to verify that the user is configured. Step 2 View the Cisco IOS/PIX RADIUS Attributes setting for a user. Verify that the user is assigned the correct Step 3 roles in the AV-pairs. For example, shell:roles=”network-admin”...

  • Page 355: Chapter 19 Troubleshooting Fc-Sp, Port Security, And Fabric Binding

    Troubleshooting FC-SP, Port Security, and Fabric Binding This chapter describes procedures used to troubleshoot Fibre Channel Security Protocol (FC-SP), port security, and fabric binding in Cisco MDS 9000 Family products. It includes the following sections: • FC-SP Overview, page 19-1 Port Security Overview, page 19-2 •...

  • Page 356: Port Security Overview

    Begin troubleshooting FC-SP issues by checking the following issues: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that your installed HBAs support FC-SP. Verify that you have configured MD5 for the hash algorithm if you are authenticating through a RADIUS or TACACS+ server.

  • Page 357: Common Troubleshooting Tools In Fabric Manager

    Us e the following CLI commands to troubleshoot fabric binding issues: show fabric-binding status • show fabric-binding database vsan • show fabric-binding database active vsan • show fabric-binding violations • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-3 OL-9285-05...

  • Page 358: Fc-Sp Issues

    FC-SP tab, set the Mode field to autoActive or autoPassive, and click Apply Changes in Fabric Manager. Or use the fcsp auto-active or fcsp auto-passive CLI command in interface mode to set the DHCHAP mode. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-4 OL-9285-05...

  • Page 359: Verifying Fc-Sp Configuration Using Fabric Manager

    Choose Switches > Interfaces > FC Logical and select the FLOGI tab to find the pWWN for the host Step 3 that you want to add to the FC-SP local database. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-5 OL-9285-05...

  • Page 360: Verifying Local Fc-Sp Database Using The Cli

    Total number of flogi = 1 Use the fcsp dhchap devicename command to add a host or switch to the local database. Step 4 switch(config)# fcsp dhchap devicename 20:00:00:33:8b:00:00:00 password rtp9509 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-6 OL-9285-05...

  • Page 361: Authentication Fails When Using Cisco Acs

    Use the show fcsp asciiwwn sWWN CLI command to get an ASCII equivalent of the sWWN. On the Cisco ACS server, choose User Setup. Search for the ASCII equivalent of the sWWN in the User column of the User List.

  • Page 362: Device Does Not Log Into A Switch When Autolearn Is Disabled

    See the “Verifying Port Security Violations Using Fabric Manager” section on page 19-10 or the “Verifying Port Security Violations Using the CLI” section on page 19-11. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-8 OL-9285-05...

  • Page 363: Verifying The Active Port Security Database Using Fabric Manager

    20:11:33:11:00:2a:4a:66 swwn 20:00:00:0c:85:90:3e:80 interface fc1/13 Use the port-security activate command to copy the configure database to the active database and Step 4 reactivate port security. switch(config)# port-security activate vsan 1 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-9 OL-9285-05...

  • Page 364: Verifying Port Security Violations Using Fabric Manager

    Apply Changes to copy the configure database to the active database and reactivate port security. Select the CFS tab, if CFS is enabled, and select commit from the ConfigAction drop-down menu to distribute these changes to all switches in the fabric. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-10 OL-9285-05...

  • Page 365: Verifying Port Security Violations Using The Cli

    Use the no shutdown command in interface mode to bring the port back online. Step 3 Optionally, remove the device from the switch and use the no shutdown command to bring the port back online. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-11 OL-9285-05...

  • Page 366: Cannot Activate Port Security

    19-15. Autolearn is enabled. Disable autolearn. See the “Disabling Autolearn Using Fabric Manager” section on page 19-13 or the “Disabling Autolearn Using the CLI” section on page 19-13. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-12 OL-9285-05...

  • Page 367: Disabling Autolearn Using Fabric Manager

    “Disabling Autolearn Using Fabric Manager” were lost after a configure database and to startup section on page 19-13 or the “Disabling Autolearn Using reboot. configuration. the CLI” section on page 19-13. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-13 OL-9285-05...

  • Page 368: Merge Fails

    Copy the running configuration to the startup configuration, using the fabric option. This saves the port Step 12 security configure database to the startup configuration on all switches in the fabric. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-14 OL-9285-05...

  • Page 369: Configuring Port Security With Autolearn Using The Cli

    After correcting a fabric binding configuration issue, you do not have to disable the interface and Note reenable it. The port comes up automatically after a fabric binding reactivation if the problem was fixed. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-15 OL-9285-05...

  • Page 370: Switch Cannot Attach To The Fabric

    Admin Status drop-down menu to bring the port back online. Click Apply Changes. You may need to set the interface down and then up to bring it back online. Note Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-16 OL-9285-05...

  • Page 371: Verifying Fabric Binding Violations Using The Cli

    Use the fabric-binding activate command to copy the configure database to the active database and reactivate fabric binding. switch(config)# fabric-binding activate vsan 3 Use the no shutdown command in interface mode to bring the port back online. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-17 OL-9285-05...

  • Page 372: Cannot Activate Fabric Binding

    3 switch(config-port-security)# no swwn 20:00:00:0c:85:90:3e:80 Step 4 Use the fabric-binding activate command to copy the configure database to the active database and reactivate fabric binding. switch(config)# fabric-binding activate vsan 1 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-18 OL-9285-05...

  • Page 373: Unauthorized Switch Gains Access To Fabric

    Copy the running configuration to the startup configuration, using the fabric option. This saves the port Step 6 security configure database to the startup configuration on all switches in the fabric. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-19 OL-9285-05...

  • Page 374: Configuring Fabric Binding Using The Cli

    Step 5 Copy the running configuration to the startup configuration, using the fabric option. This saves the fabric binding configure database to the startup configuration on all switches in the fabric. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-20 OL-9285-05...

  • Page 375: Chapter 20 Troubleshooting Ip Storage Services

    VSANs, security, and traffic management. The IPS module can be used in any Cisco MDS 9000 Family switch and has eight Gigabit Ethernet ports. Each port can run the FCIP and iSCSI protocols simultaneously.

  • Page 376: Iscsi Restrictions

    User profile 28 User profile 28 Peer info IP addr 10.10.10.2 Peer info IP addr 10.10.11.2 The iSCSI feature is specific to the IPS module and is available in Cisco MDS 9200 Switches or Note • Cisco MDS 9500 Directors.

  • Page 377: Initial Troubleshooting Checklist

    Begin troubleshooting IP storage services issues by checking the following issues: Checklist Check off Verify licensing requirements. See the Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that you are not configuring IPsec with IPv6. Verify that auto-zone and CFS distribution are enabled for iSLB.
  • Page 378 [assignment | interface | metric | session]—Displays the internal • data structures for the iSLB load balancing feature. Use the following debug commands to gather more information for iSLB: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-4 OL-9285-05...

  • Page 379: Ip Issues

    If you configure secondary VRRP IPv6 addresses on an IPFC VSAN interface, before a downgrading to a release prior to Cisco Release 3.0(1), you must remove the secondary VRRP IPv6 addresses. This is required only when you configure IPv6 addresses.
  • Page 380 0 multicast frames, 0 compressed 0 input errors, 0 frame, 0 overrun 0 fifo 144401 packets output, 7805631 bytes, 0 underruns 0 output errors, 0 collisions, 0 fifo 0 carrier errors Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-6 OL-9285-05...

  • Page 381: Verification Of Switch Connectivity

    20:c8:00:05:30:00:86:5f 0x6a(106) 20:c8:00:05:30:00:f8:e3 Concatenate the domain ID with FFFC to obtain the domain controller address. For example, if the Step 2 domain ID is 0xda(218), the concatenated ID is 0xfffcda. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-7 OL-9285-05...

  • Page 382: Verification Of Static Ip Routing

    Default gateway is 172.17.8.1 C 172.17.8.0/24 is directly connected, mgmt0 S 11.2.36.0/22 via 11.3.36.1, gigabitethernet8/7 C 11.3.36.0/22 is directly connected, gigabitethernet8/7 C 11.3.56.0/22 is directly connected, gigabitethernet8/8 S 11.2.56.0/22 via 11.3.56.1, gigabitethernet8/8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-8 OL-9285-05...

  • Page 383: Cannot Assign Ip Address To An Interface

    FCIP Tape acceleration does not work if the FCIP port is part of a PortChannel or if there are multiple Note paths between the initiator and the target port. Such a configuration might cause SCSI discovery failure or broken write or read operations. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-9 OL-9285-05...

  • Page 384: One-To-One Fcip Tunnel Creation And Monitoring

    The interface FCIP can be any number between 1 – 255 and does not need to be the same as the profile number. In this example the same number is used for simplicity. Specify a profile to use. Step 9 MDS1(config-if)# use-profile 28 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-10 OL-9285-05...

  • Page 385: Displaying The Default Values Using The Cli

    The following example shows the configuration of a switch (MDS2) with debug mode activated. To activate debug mode for this situation, run the debug ips flow fcip command on a separate terminal. MDS2(config)# fcip profile 28 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-11 OL-9285-05...
  • Page 386 Mar 10 21:42:23 ips: (ips_demux) Mts Opcode is 1905, id is 7304 Mar 10 21:42:23 ips: FCIP28: Processing Pull Config Request Mar 10 21:42:23 ips: FCIP28: Bound to entity 28 port: 3225 ip: 10.10.11.2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-12 OL-9285-05...

  • Page 387: Displaying The Debug Output From Fcip Tunnel Supervisor Using The Cli

    Mar 10 22:59:46 ips: Hndlr MTS_OPC_PM_LOGICAL_PORT_STATE_CHANGE_RANGE (mts_opc 3114 msg_id 47540) Mar 10 22:59:46 ips: fu_fsm_execute_all: match_msg_id(0), log_already_open(0) Mar 10 22:59:46 ips: fu_fsm_execute_all: null fsm_event_list Mar 10 22:59:46 ips: fu_fsm_engine: mts msg MTS_OPC_PM_LOGICAL_PORT_STATE_CHANGE_RANGE(msg_id 47540) dropped Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-13 OL-9285-05...

  • Page 388: Displaying The Debug Output From The Fcip Tunnel Ips Module Using The Cli

    2715:FCIP28: *** Received eisl frame in E mode Mar 13 19:18:20 port8: 2716:FCIP28: SUP-> Set trunk mode: 2 Mar 13 19:18:20 port8: 2717:FCIP28: Change the operational mode to TRUNK Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-14 OL-9285-05...

  • Page 389: Verifying The Configuration Of The Profiles Using The Cli

    Maximum number of TCPconnections is 2 (The default is 2 TCP connections being used, one for class F and the other for class 2 and 3.) Time Stamp is disabled Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-15 OL-9285-05...
  • Page 390 (These are the frames that averaged over 5 minutes and the total count of frames since the last clear counters command was issued, or since the last tunnel up.) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-16 OL-9285-05...

  • Page 391: Verifying The Establishment Of Default Tcp Connections For Each Configured Fcip Tunnel Using The Cli

    Hardware Ingress Counters (Verify good increments on the active tunnel.) 2312 Good, 0 protocol error, 0 header checksum error 0 FC CRC error, 0 iSCSI CRC error, 0 parity error Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-17 OL-9285-05...

  • Page 392: Ethereal Screen Captures Of The Tcp Connection And Fcip Tunnels

    Figure 20-3 First Capture of TCP Connection TCP connection Figure 20-4 shows more of the trace, with frame 13 being the first FCIP frame. This frame carries the FC Standard ELP. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-18 OL-9285-05...
  • Page 393 Second Capture of TCP Connection exchange link parameter Figure 20-5 shows the FC portion of the EISL initialization over the FCIP tunnel. Figure 20-5 Third Capture of TCP Connection Cisco Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-19 OL-9285-05...

  • Page 394: One-To-Three Fcip Tunnel Creation And Monitoring

    MDS1(config)# interface fcip 21 MDS1(config-if)# use-profile 21 MDS1(config-if)# peer-info ipaddr 10.10.11.2 MDS1(config-if)# no shutdown MDS1(config-if)# exit MDS1(config)# ip route 10.10.11.0 255.255.255.0 10.10.10.1 MDS1(config)# ip route 10.10.11.0 255.255.255.0 interface gigabitethernet 2/1 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-20 OL-9285-05...

  • Page 395: Creating The Fcip Interface For The Second Tunnel Using The Cli

    MDS22# show fcip profile 21 FCIP Profile 21 Internet Address is 34.34.34.34 (interface GigabitEthernet2/5) (In this line, the Gigabit Ethernet port is now shown and the FCIP profile is bound to a physical port.) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-21 OL-9285-05...

  • Page 396: Cli

    (Change the TCP listen port on switch MDS2.) MDS2(config-profile)# end MDS2(config)# interface fcip 21 MDS2(config-if)# passive-mode (Put interface fcip 21 in passive mode to guarantee MDS1 initiates a TCP connection.) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-22 OL-9285-05...
  • Page 397 2515 received, 2342 sent, 0 retransmitted 0 bad segments received, 0 reset sent TCP Active Connections Local Address Remote Address State Send-Q Recv-Q 10.10.11.2:13 10.10.10.2:65188 ESTABLISH (The port is 13 as configured.) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-23 OL-9285-05...

  • Page 398: Fcip Interface Misconfiguration Examples

    0 Class 2/3 frames input, 0 bytes 0 Error frames 0 frames output, 0 bytes 0 Class F frames output, 0 bytes 0 Class 2/3 frames output, 0 bytes 0 Error frames 0 reass frames Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-24 OL-9285-05...

  • Page 399: Displaying The Fcip Interface As Administratively Shut Down Using The Cli

    MDS2# show interface fcip 21 fcip21 is down (Link failure or not-connected) Hardware is GigabitEthernet Port WWN is 20:42:00:0b:5f:d5:9f:c0 Admin port mode is auto, trunk mode is on vsan is 1 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-25 OL-9285-05...

  • Page 400: Displaying The Debug Output From The Second Switch Using The Cli

    Peer port WWN is 20:42:00:05:30:00:59:de Admin port mode is auto, trunk mode is on Port mode is TE vsan is 1 Trunk vsans (allowed active) (1-2) Trunk vsans (operational) (1-2) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-26 OL-9285-05...

  • Page 401: Displaying Passive Mode Set On Both Sides Of The Fcip Tunnel With The Cli

    (Both sides are set to passive mode. You must change one or both sides to no passive-mode under the FCIP interface.) Special Frame is disabled MDS2(config)# interface fcip 21 MDS2(config-if)# no passive-mode Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-27 OL-9285-05...

  • Page 402: Displaying A Time Stamp Acceptable Difference Failure Using The Cli

    (The TCP connection begins trying to reestablish the connection.) Mar 15 00:01:50 port1: 3307:FCIP21: Tunnel is not ADMIN UP state, reject new TCP connection from 10.10.10.2:65066 Mar 15 00:01:50 port1: 3308:FCIP21: Received new TCP connection from peer: 10.10.10.2:65064 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-28 OL-9285-05...
  • Page 403 Peer Internet address is 10.10.10.2 and port is 3225 Special Frame is disabled Maximum number of TCP connections is 2 Time Stamp is enabled, acceptable time difference 2000 ms B-port mode disabled TCP Connection Information Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-29 OL-9285-05...

  • Page 404: Fcip Special Frame Tunnel Creation And Monitoring

    “One-to-Three FCIP Tunnel Creation and Monitoring” section on page 20-20) must be completed before adding the FCIP special frame configuration. This section describes how to correctly configure and show an FCIP tunnel with a special frame. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-30 OL-9285-05...

  • Page 405: Configuring And Displaying An Fcip Tunnel With Special Frame Using The Cli

    MDS2# show interface fcip 21 fcip21 is trunking Hardware is GigabitEthernet Port WWN is 20:42:00:0b:5f:d5:9f:c0 Peer port WWN is 20:42:00:05:30:00:59:de Admin port mode is auto, trunk mode is on Port mode is TE Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-31 OL-9285-05...
  • Page 406 Use the show wwn switch command on the remote switch to verify the peer switch WWN. Figure 20-9 shows a trace of an FCIP tunnel with a special frame. Figure 20-9 Trace of FCIP Tunnel with a Special Frame Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-32 OL-9285-05...

  • Page 407: Special Frame Misconfiguration Example

    855325:FCIP21: Delete the DE object [1] 0xd802d080 Jan 14 15:14:31 port1: 855326:FCIP21: Received new TCP connection from peer: 10.10.10.2:64048 Jan 14 15:14:31 port1: 855327:FCIP21: Create a DE 0xd802d200 for this tunnel Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-33 OL-9285-05...

  • Page 408: Troubleshooting Fcip Link Flaps

    For example, if you have a 200 Mbps physical link and you have configured two FCIP tunnels across that link, each with 155 Mbps traffic, link failures will occur because the physical connection cannot handle the traffic. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-34 OL-9285-05...

  • Page 409: Troubleshooting Fcip And Compression

    Troubleshooting FCIP and Compression If you have an FCIP tunnel between an IPS module and a Cisco MDS 14/2, 9221i or MDS 18/4 module, use the same compression mode on both sides of the FCIP tunnel. In this specific configuration, avoid compression mode 1, because the Cisco MDS module could send compressed traffic faster than the IPS module could process.

  • Page 410: Displaying Iscsi Authentication Using Fabric Manager

    Whenever you experience a login failure, use the show authentication command to see if the iSCSI authentication is correctly defined. This is an example of local authentication: switch# show authentication authentication method:none console:not enabled telnet/ssh:not enabled authentication method:radius console:not enabled telnet/ssh:not enabled Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-36 OL-9285-05...

  • Page 411: Troubleshooting User Name And Password Configuration

    Note The iSCSI password must be at least 16 characters. switch# show user-account iscsi username:iscsi secret:1234567812345678 username:iscsiuser secret:1234567812345678 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-37 OL-9285-05...

  • Page 412: Radius Configuration Troubleshooting

    RADIUS server. The following example shows the output of the debug security radius command, if the iSCSI client logs in successfully. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-38 OL-9285-05...
  • Page 413 649 2003y3m14d 15h12m48s User (user002) authenticate OK. 650 2003y3m14d 15h12m54s ------------------------------------------------ 651 2003y3m14d 15h12m54s Message Type=Access_Request 652 2003y3m14d 15h12m54s ID=60, Length=90 653 2003y3m14d 15h12m54s User name=user002 654 2003y3m14d 15h12m54s NAS IP address=2887147911 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-39 OL-9285-05...

  • Page 414: Troubleshooting Radius Routing Configuration

    5 00:51:33 securityd: no response from RADIUS server for authentication user002 5 00:51:33 securityd: doing local chap authentication for user002 5 00:51:33 securityd: local chap authentication result for user002:user not present Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-40 OL-9285-05...

  • Page 415: Troubleshooting Dynamic Iscsi Configuration

    Fabric Manager to allow iSCSI targets to be discovered by the logged-in iSCSI initiators. Use the iscsi import target fc CLI command to allow iSCSI targets to be discovered by the – logged-in iSCSI initiators. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-41 OL-9285-05...

  • Page 416: Useful Show Commands To Debug Dynamic Iscsi Configuration

    FULLMOON Node WWN is 20:0c:00:0b:be:77:72:42 (dynamic) Member of vsans: 1 Number of Virtual n_ports: 1 Virtual Port WWN is 20:0d:00:0b:be:77:72:42 (dynamic) Interface iSCSI 2/7, Portal group tag: 0x86 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-42 OL-9285-05...

  • Page 417: Virtual Target Access Control

    The show commands in this section are used to debug static iSCSI configuration. The following command output indicates correctly established iSCSI sessions. Run these commands on your switch and compare the output with these samples to help identify possible issues. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-43 OL-9285-05...
  • Page 418 FC target: Up, Reorder PDU: No, Marker send: No (int 0) Received MaxRecvDSLen key: Yes Session #3 (index 86) Target iqn.com.domainname.IPS-TEST.02-08.gw.2200002037c52356 VSAN 5, ISID 00023d000056, TSID 135, Status active, no reservation Type Normal, ExpCmdSN 1356, MaxCmdSN 1366, Barrier 0 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-44 OL-9285-05...
  • Page 419 5 minutes input rate 3336 bits/sec, 417 bytes/sec, 0 frames/sec 5 minutes output rate 120 bits/sec, 15 bytes/sec, 0 frames/sec iSCSI statistics 4113028 packets input, 4022586092 bytes 303140 Command pdus, 3740200 Data-out pdus, 3816015476 Data-out bytes, 0 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-45 OL-9285-05...
  • Page 420 5 VSAN 5: -------------------------------------------------------------------------- FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------- 0x610002 20:0b:00:0b:be:77:72:42 scsi-fcp:init isc..w 0x6101e1 22:00:00:20:37:c5:2d:6d (Seagate) scsi-fcp:target 0x6101e2 22:00:00:20:37:c5:2e:2e (Seagate) scsi-fcp:target 0x6101e4 22:00:00:20:37:c5:23:56 (Seagate) scsi-fcp:target 0x6101e8 22:00:00:20:37:c5:26:0a (Seagate) scsi-fcp:target Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-46 OL-9285-05...
  • Page 421 Bytes: TX: 1344, RX: 0 Number of connection: 1 TCP parameters Connection Local 10.1.29.100:3260, Remote 10.1.29.101:1048 Path MTU 1500 bytes Current retransmission timeout is 300 ms Round trip time: Smoothed 165 ms, Variance: 35 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-47 OL-9285-05...

  • Page 422: Iscsi Tcp Performance Issues

    Sizes on both TCP endpoints, RTT (round trip time), actual available bandwidth between the TCP peers, the MSS (maximum segment size), and the support for higher MTUs between the peers. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-48 OL-9285-05...

  • Page 423: Cli Commands Used To Access Performance Data

    The second point of performance tuning is to increase the TCP window size of the iSCSI endpoints. Depending on the latency between the iSCSI client and the IPS, this will need fine tuning. The switch’s iSCSI configuration defines the TCP window size in kilobytes. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-49 OL-9285-05...

  • Page 424: Lab Setup

    (This is the iSCSI target IP address for the Windows iSCSI client.) no shutdown interface iscsi2/1 tcp pmtu-enable tcp window-size 1024 (To increase the receive window size of the IPS module (in kilobytes).) tcp sack-enable Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-50 OL-9285-05...

  • Page 425: Verifying Connectivity Between Client And Ips Iscsi Service

    20:41:00:0c:30:57:5e:c0 to 20:48:00:0c:30:57:5e:c0 MAC-Address(es) Serial-Num -------------------------------------- ---------- 00-0b-be-f8-7f-00 to 00-0b-be-f8-7f-04 JAB070804Q3 00-05-30-00-a8-56 to 00-05-30-00-a8-62 JAB070205am * this terminal session MDS_BOTTOM# show iscsi remote iSCSI Node name is iqn.1987-05.com.cisco:02.75af2f95624c.shark-nas iSCSI alias name: SHARK-NAS Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-51 OL-9285-05...
  • Page 426 1072305 Command pdus, 0 Data-out pdus, 0 Data-out bytes, 0 fragments 53430805 packets output, 72837086312 bytes 1072273 Response pdus (with sense 9), 0 R2T pdus 52358444 Data-in pdus, 70272402880 Data-in bytes MDS_BOTTOM# show iscsi remote initiator iqn.1987-05.com.cisco:02.75af2f95624c.shark-nas iscsi tcp Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-52 OL-9285-05...
  • Page 427 (This is the window size set on the Windows client. See Figure 20-14.) Peer receive window: Current: 1000 KB, Maximum: 1000 KB, Scale: 4 (This is the window size set on the IPS iSCSI interface. See Figure 20-14.) Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-53 OL-9285-05...

  • Page 428: Tcp Parameter Changes

    Internet address is 10.48.69.251/26 MTU 1500 bytes, BW 1000000 Kbit Port mode is IPS Speed is 1 Gbps Beacon is turned off 5 minutes input rate 3957384 bits/sec, 494673 bytes/sec, 6716 frames/sec Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-54 OL-9285-05...
  • Page 429 24 entries, 24 connections completed, 0 entries timed out 0 dropped due to overflow, 0 dropped due to RST 0 dropped due to ICMP unreach, 0 dropped due to bucket overflow Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-55 OL-9285-05...
  • Page 430 1 iSCSI sessions share this FC session Target: shark_nas Negotiated parameters RcvDataFieldSize 2048 our_RcvDataFieldSize 1392 MaxBurstSize 0, EMPD: FALSE Random Relative Offset: FALSE, Sequence-in-order: Yes Statistics: PDU: Command: 0, Response: 1612007 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-56 OL-9285-05...

  • Page 431: Verifying That The Host Is Configured For High Mtu Or Mss With The Cli

    715535 segments, 943511612 bytes 712704 data, 2831 ack only packets 0 control (SYN/FIN/RST), 0 probes, 0 window updates 0 segments retransmitted, 0 bytes 0 retransmitted while on ethernet send queue, 345477 packets split Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-57 OL-9285-05...

  • Page 432: Islb Issues

    Symptom Possible Cause Solution iSLB configuration Not all switches are running Cisco Update switches to Cisco SAN-OS Release 3.0(1) or later. not distributed to all SAN-OS Release 3.0(1) or later. switches in the fabric. CFS distribution is not enabled for Enable CFS distribution for iSLB.

  • Page 433: Iscsi Initiator And Virtual Target Configuration Not Distributed

    ID is not configured on a switch. ID is not yet configured. Use the vsan database vsan vsan-id CLI command to add the VSAN ID, or remove the VSAN ID from the initiator configuration. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-59 OL-9285-05...

  • Page 434: Islb Configuration, Commit, Or Merge Failed-"Failed To Allocate Wwn

    CLI commands for details on the specific initiator in error. Duplicate node name To fix the problem, use a different node name. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-60 OL-9285-05...

  • Page 435: Islb Configuration Failed-"Pending Islb Cfs Config Has Reached Its Limit

    Check the output of the show islb cfs-session status CLI timeout. it is possible for the iSLB commit to command to get the status of the commit. take a long time. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-61 OL-9285-05...

  • Page 436: Session Down-"Pwwn In Use At Remote Switch

    Use the debug ips islb vrrp flow CLI command to check if the redirection is performing correctly. Use the show islb vrrp summary CLI command to see if the initiator to the interface mapping is set up. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-62 OL-9285-05...

  • Page 437: Islb Zones Not Present In Active Zone Set

    Activate the zone set from a switch that has both IVR and iSLB). or IVR) must be done from the switch iSLB enabled. that has IVR configured. Use the islb zoneset activate CLI command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-63 OL-9285-05...

  • Page 438: Vrrp Master Overutilized

    Use the show islb status, islb commit, or islb abort CLI command to view the status, to commit the changes or to discard the changes, respectively. Also, verify that no zone or IVR zone changes are pending. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-64 OL-9285-05...

  • Page 439: Islb Cfs Commit Fails

    The iSLB configuration on other switches will be overwritten. A commit after a merge failure Note synchronizes the fabric configuration to the running- config of the switch where the commit was performed. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-65 OL-9285-05...
  • Page 440 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 441: Chapter 21 Troubleshooting Ip Access Lists

    Each switch in the Cisco MDS 9000 Family can have a maximum of 64 IP-ACLs and each IP-ACL can have a maximum of 256 filters.

  • Page 442: Address Information

    (less than) option, or the range (range of ports) option. Table 21-1 displays the port numbers recognized by the Cisco SAN-OS software for associated TCP and UDP ports for IPv4. IPv6-ACL CLI commands do not support TCP or UDP port names. Note...

  • Page 443: Icmp Information

    ICMP Type Value ICMP Type Code echo echo-reply unreachable redirect time exceeded traceroute ToS Information IPv4 packets can be filtered based on the ToS conditions—delay, monetary-cost, normal-service, reliability, and throughput. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-3 OL-9285-05...

  • Page 444: Initial Troubleshooting Checklist

    Begin troubleshooting IP-ACLs by checking the following issues: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that the access list has been applied to the interface. Verify that the access list is not empty.

  • Page 445: All Packets Are Blocked

    Click Apply Changes to save these changes. Click the IP ACL wizard icon. You see the IP-ACL wizard dialog box. Step 4 Add the IP-ACL name in the name field and click Add. Step 5 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-5 OL-9285-05...

  • Page 446: Re-Creating Ip-Acls Using The Cli

    Step 3 associated with it. switch(config)# no ip access-list TCPAlow We recommend deleting an ACL and re-creating it because you cannot change the order of filters in an Note ACL. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-6 OL-9285-05...

  • Page 447: No Packets Are Blocked

    Permit filter is too high in the access Delete the access list and re-create. See the “Re-creating list order. IP-ACLs Using Fabric Manager” section on page 21-5 “Re-creating IP-ACLs Using the CLI” section on page 21-6. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-7 OL-9285-05...

  • Page 448: Portchannel Not Working With Acl

    Incorrect ACL on mgmt0 interface. Connect to console port locally and delete the ACL. Use connect to switch. the no ip access-group or the no ipv6 traffic-filter CLI command in interface mode. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-8 OL-9285-05...

  • Page 449: Chapter 22 Troubleshooting Ipsec

    Cisco MDS 9216i Switch with the MPS-14/2 capability in the integrated supervisor module. Refer • to the Cisco MDS 9200 Series Hardware Installation Guide for more information on the Cisco MDS 9216i Switch. The IPsec feature is not supported on the management interface.

  • Page 450: Supported Ipsec And Ike Algorithms For Microsoft Windows And Linux Platforms

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m IPsec and IKE are not supported by the Cisco Fabric Switch HP c-Class BladeSystem and the Cisco Note Fabric Switch for IBM BladeCenter.

  • Page 451: Ike Allowed Transforms

    SHA-1 (HMAC variant) SHA-1 MD5 (HMAC variant) Authentication method Preshared keys Preshared keys RSA signatures in digital certificates DH group identifier 768-bit DH 768-bit DH (1) 1024-bit DH 1536-bit DH Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-3 OL-9285-05...

  • Page 452: Ipsec Allowed Transforms

    Begin troubleshooting IPsec issues by checking the following issues: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that IKE has been configured for IPsec. Verify the digital certificates configuration if it is enabled for IPsec. See Chapter 24, “Troubleshooting Digital Certificates.”...

  • Page 453: Common Troubleshooting Commands In The Cli

    Verifying Security Policy Databases Compatibility, page 22-8 • Verifying Interface Status Using Fabric Manager, page 22-9 • Verifying Interface Status Using the CLI, page 22-9 • Verifying Security Associations, page 22-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-5 OL-9285-05...

  • Page 454: Verifying Ike Configuration Compatibility

    Select the Interfaces tab and verify that the crypto map set is applied to the correct interface on both Step 3 switches. In Device Manager, choose IP > ACLs and verify that the ACLs used in the crypto map in Step 1 Step 4 compatible on both switches. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-6 OL-9285-05...

  • Page 455: Verifying Ipsec Configuration Compatibility Using The Cli

    Ensure that the transform sets are compatible in the show crypto transform-set domain ipsec command Step 4 outputs for both switches. Ensure that the PFS settings in the show crypto map domain ipsec command outputs are configured Step 5 the same on both switches. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-7 OL-9285-05...

  • Page 456: Verifying Security Policy Databases Compatibility

    Source port :*, Destination port :500 Protocol UDP Physical port:0/0, Vlan_id:0/0 Action cleartext Inbound Policy 2 : Source IP Address :10.10.100.232/255.255.255.255 Destination IP Address :10.10.100.231/255.255.255.255 Source port :*, Destination port :* Protocol * Physical port:0/1, Vlan_id:0/4095 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-8 OL-9285-05...

  • Page 457: Verifying Interface Status Using Fabric Manager

    FCIP tunnels are compatible. Verifying Interface Status Using the CLI To verify the status of the interfaces using the CLI, follow these steps: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-9 OL-9285-05...
  • Page 458 Trunk vsans (initializing) Using Profile id 1 (interface GigabitEthernet7/1) Peer Information Peer Internet address is 10.10.100.232 and port is 3225 FCIP tunnel is protected by IPSec Write acceleration mode is off Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-10 OL-9285-05...
  • Page 459 Special Frame is disabled Maximum number of TCP connections is 2 Time Stamp is disabled QOS control code point is 0 QOS data code point is 0 B-port mode disabled TCP Connection Information Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-11 OL-9285-05...

  • Page 460: Verifying Security Associations

    MDSC# show crypto sad domain ipsec interface:GigabitEthernet1/2 Crypto map tag:cmap-01, local addr. 10.10.100.232 protected network: local ident (addr/mask):(10.10.100.232/255.255.255.255) remote ident (addr/mask):(10.10.100.231/255.255.255.255) current_peer:10.10.100.231 local crypto endpt.:10.10.100.232, remote crypto endpt.:10.10.100.231 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-12 OL-9285-05...
  • Page 461 Encrypt algorithm is DES/3DES Auth algorithm is MD5 Source ip address 10.10.100.231/255.255.255.255 Destination ip address 10.10.100.232/255.255.255.255 Physical port 1, mask:0x1 Misc select 0 mask:0x0 Vlan 0 mask:0xfff Protocol 0 mask:0x0 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-13 OL-9285-05...
  • Page 462 Hard limit expiry 1100652419 secs (since January 1, 1970), remaining 206 4 secs Soft limit expiry 1100652397 secs (since January 1, 1970), remaining 204 2 secs Outbound MAC table index:125 Sequence number:7123 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-14 OL-9285-05...

  • Page 463: Security Associations Do Not Re-Key

    The show crypto global domain ipsec command output displays statistics for all SAs. Command output follows: MDSA# show crypto global domain ipsec IPSec global statistics: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-15 OL-9285-05...
  • Page 464 Example command output follows: MDSA# show crypto global domain ipsec interface gigabitethernet 7/1 IPSec interface statistics: IKE transaction stats:0 num Inbound SA stats:1 num, 512 max Outbound SA stats:1 num, 512 max Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-16 OL-9285-05...

  • Page 465: Chapter 23 Troubleshooting Santap

    SANTap is an Intelligent Storage Services feature supported on the Storage Services Module (SSM). The SSM supports SANTap in Cisco MDS SAN-OS Release 2.0(2b) and later. The SANTap feature allows third-party data storage applications, such as long distance replication and continuous backup, to be integrated into the SAN.

  • Page 466: Definitions

    SANTap. An initiator port on the appliance sends out SANTap Control Protocol requests to the SANTap process. When the request is processed, the response is sent back by the Cisco VI (virtual initiator) to a target port on the appliance.

  • Page 467: Interface Restrictions

    SANTap components. Begin your troubleshooting activity as follows: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that SANTap is enabled on the SSM module of the selected switch.

  • Page 468: Common Troubleshooting Tools In Fabric Manager

    Example 23-3 Display SANTap DVT LUN Information switch# show santap module 2 dvtlun DVT LUN Information : dvt pwwn = 22:00:00:20:37:88:20:ef dvt lun = 0x0 xmap id Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 23-4 OL-9285-05...
  • Page 469 2 avtlun AVT LUN Information : avt pwwn = 2a:4b:00:05:30:00:22:25 avt lun = 0x0 xmap id = 16 avt id = 12 tgt lun = 0x0 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 23-5 OL-9285-05...

  • Page 470: Messages, Logs And Databases

    Review FCNS and FLOGI databases, using the show fcns and show flogi CLI commands. SANTap Issues This section includes the following topics: Host Login Problems, page 23-7 • ITL Problems, page 23-7 • Common Mismatch Problems, page 23-7 • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 23-6 OL-9285-05...

  • Page 471: Host Login Problems

    Use the show isapi dpp 4 queue command to display DPP queue information. Step 1 Verify that the number of ITLs on a DPP is within the limitations for the version of Cisco SAN-OS and Step 2 SSI in use. Use the show isapi dpp 4 queue incl LUN and show isapi dpp 4 queue count commands.
  • Page 472 When a host HBA logs into both DVTs, an attempt is made to create two VIs with the same WWN in the same back-end VSAN. This results in non-deterministic behavior. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 23-8 OL-9285-05...
  • Page 473 LUNs. In this situation, perform a purge to clear one of the 16 entries after removing the host. Then you can add the new host to the DVT. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 23-9 OL-9285-05...
  • Page 474 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 475: Chapter 24 Troubleshooting Digital Certificates

    • Overview Public Key Infrastructure (PKI) support provides the means for the Cisco MDS 9000 Family of switches to obtain and use digital certificates for secure communication in the network. PKI support provides manageability and scalability for IPsec/IKE and SSH.

  • Page 476: Rsa Key Pairs And Identity Certificates

    CRLs are maintained by CAs to give information of prematurely revoked certificates, and the CRLs are published in a repository. Cisco MDS SAN-OS allows the manual configuration of pre-downloaded CRLs for the trusted CAs, and then caches them in the switch bootflash (cert-store). During the verification of a peer certificate by IPsec or SSH, the issuing CA’s CRL is consulted only if the CRL has already been cached locally and...

  • Page 477: Maximum Limits

    Receive the issued certificate back from the CA, signed with the CA’s private key. Write the certificate into a nonvolatile storage area on the switch (bootflash). Cisco MDS SAN-OS supports certificate retrieval and enrollment using a manual cut-and-paste method. Cut-and-paste enrollment literally means you must cut and paste the certificate requests and resulting certificates between the switch (using a console, Telnet, or SSH connection) and the CA, as follows: Create an enrollment certificate request, which is displayed in base64-encoded text form.

  • Page 478: Common Troubleshooting Commands In The Cli

    URL field and enter the challenge password in the Password field. Click Apply Changes. Or use the crypto ca enroll CLI command and enter a challenge password during enrollment. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-4 OL-9285-05...

  • Page 479: Cannot Export Identity Certificate In Pkcs#12 Format

    Choose Switches > Security > PKI and select the RSA Key-Pair tab. Click Create Row and set the name and size field. Check the Exportable check box and click Create. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-5 OL-9285-05...
  • Page 480 Click Apply Changes to save the changes. Request an identity certificate from the CA. Step 9 Note The CA may require manual verification before issuing the identity certificate. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-6 OL-9285-05...

  • Page 481: Configuring Certificates On The Mds Switch Using The Cli

    Vegas-1(config)# do show crypto key mypubkey rsa key label: myKey key size: 1024 exportable: yes Vegas-1(config)# Associate the RSA key pair to the trust point. Step 5 Vegas-1(config)# crypto ca trustpoint myCA Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-7 OL-9285-05...
  • Page 482 CA Administrator in order to revoke your certificate. For security reasons your password will not be saved in the configuration. Please make a note of it. Password:nbv123 The subject name in the certificate will be: Vegas-1.cisco.com Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-8 OL-9285-05...
  • Page 483 Vegas-1(config)# do show crypto ca certificates Trustpoint: myCA certificate: subject= /CN=Vegas-1.cisco.com issuer= /emailAddress=amandke@cisco.com/C=IN/ST=Karnataka/L=Bangalore/O=Cisco/OU =netstorage/CN=Aparna CA serial=0A338EA1000000000074 notBefore=Nov 12 03:02:40 2005 GMT notAfter=Nov 12 03:12:40 2006 GMT MD5 Fingerprint=3D:33:62:3D:B4:D0:87:A0:70:DE:A3:87:B3:4E:24:BF purposes: sslserver sslclient ike Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-9 OL-9285-05...

  • Page 484: Pki Fails After Reboot

    Choose Switches > Security > PKI in Fabric Manager. failed for. Right-click the RSA key pair that you want to delete and click Delete Row. Or use the no crypto key zeroize rsa CLI command Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-10 OL-9285-05...

  • Page 485: Importing Certificate And Rsa Key Pairs From Backup Using Fabric Manager

    Optionally, use the delete ca-certificate command in trust point config submode to remove the CA Step 2 certificate from the trust point. switch(config)# crypto ca trustpoint myCA switch(config-trustpoint)# delete ca-certificate Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-11 OL-9285-05...
  • Page 486 Step 6 Use the crypto ca import command to import the certificates and RSA key pairs to the trust point. switch(config)# crypto ca import admin-ca pkcs12 bootflash:adminid.p12 nbv123 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-12 OL-9285-05...

  • Page 487: Chapter 25 Troubleshooting Call Home

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m C H A P T E R Troubleshooting Call Home This chapter describes how to troubleshoot the Call Home feature in the Cisco MDS 9000 Family. It includes the following sections: Overview, page 25-1 •...

  • Page 488: Alert Groups

    Alert Groups An alert group is a predefined subset of Call Home alerts supported in all switches in the Cisco MDS 9000 Family. Different types of Call Home alerts are grouped into different alert groups depending on their type.

  • Page 489: Initial Troubleshooting Checklist

    Verify that you have configured the contact name, phone, and street address on the switch. Verify that the switch has IP connectivity to your e-mail server. If Cisco AutoNotify is used, verify that you have an active service contract that covers the device being configured.

  • Page 490: Call Home Issues

    Step 3 Click an alert group to select it for association. You see a check next to that alert group. To deselect it and remove the check, click it again. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-4 OL-9285-05...

  • Page 491: Configuring An Alert Group Using The Cli

    5 and above for the user-defined profile (test1). Removes a previously configured urgency level switch(config-callhome)# no destination-profile oldtest message-level 7 and reverts it to the default of 0 (all messages are sent). Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-5 OL-9285-05...

  • Page 492: Configuring An E-Mail Server For Call Home Using Fabric Manager

    See the “Configuring Call Home Contact Information configured switches Using Fabric Manager” section on page 25-7 or the “Configuring Call Home Contact Information Using the CLI” section on page 25-7. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-6 OL-9285-05...

  • Page 493: Configuring Call Home Contact Information Using Fabric Manager

    Assigns the customer’s e-mail address. Up to 128 switch(config-callhome)# email-contact username@company.com alphanumeric characters are accepted in e-mail address format. Note You can use any valid e-mail address. You cannot use spaces. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-7 OL-9285-05...

  • Page 494: Receiving Too Many Call Home Alerts

    Verify the syslog message level configured on the switch. Choose Switches > Events >Syslog in Fabric Manager and click the Severity Level tab. Or use the show logging level CLI command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-8 OL-9285-05...

  • Page 495: Periodic Inventory Notification Does Not Reflect Current Inventory

    Inventory change occurred after the last The periodic inventory for Call Home is updated when the notification does not system reboot. switch restarts. Initiate a nondisruptive reboot to update reflect the current inventory notification. inventory Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-9 OL-9285-05...
  • Page 496 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 497: Chapter 26 Troubleshooting Fabric Manager

    Troubleshooting Device Manager, page 26-13 • Overview Cisco Fabric Manager is a Java and SNMP-based network fabric and device management tool with a GUI that displays real-time views of your network fabric, including Cisco MDS 9000 and third-party switches, hosts, and storage devices.

  • Page 498: Guidelines

    Java version. Verify that the necessary ports are open in your firewall if Fabric Manager Server is installed behind a firewall. Refer to the Cisco MDS 9000 Family Fabric Manager Configuration Guide for details on running Fabric Manager behind a firewall.

  • Page 499: Cannot Log Into Fabric Manager

    2-32. Cannot Upgrade Fabric Manager If you attempt to upgrade Fabric Manager by pointing your web browser at a switch running Cisco SAN-OS 3.0(1) through SAN-OS 3.1(3), you may encounter an issue where the upgrade does not complete. You should open the Java Web Start application on your desktop and disable HTTP proxy. If you are using Microsoft Windows, open Java Web Start and choose File >...

  • Page 500: Red Line Through The Switch

    Fabric Manager Upgrade Without Losing Map Settings, page 26-5 • Restrictions When Using Fabric Manager Across FCIP, page 26-5 • • Running Cisco Fabric Manager with Network Multiple Interfaces, page 26-5 • Configuring a Proxy Server, page 26-6 • Clearing Topology Maps, page 26-6 Using Fabric Manager in a Mixed Software Environment, page 26-7 •...

  • Page 501: Fabric Manager Upgrade Without Losing Map Settings

    For that switch, it will display a red slash through an FCIP device because of a timeout error. It will still see all targets, initiators, and ISLs attached to a Cisco SN5428 (or any other switch) as long as they appear in the name server or FSPF.

  • Page 502: Specifying An Interface For Fabric Manager Client Or Device Manager

    To clear information from topology maps, follow these steps: Step 1 Click on the Refresh Map icon in the Fabric pane. This clears the information from the client. Step 2 From the Server menu, click Purge Down Elements. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-6 OL-9285-05...

  • Page 503: Using Fabric Manager In A Mixed Software Environment

    Using Fabric Manager in a Mixed Software Environment You can use Fabric Manager version 2.x to manage a mixed fabric of Cisco MDS 9000 Family switches. Certain 2.x feature tabs will be disabled for any switches running a software version that does not support those features.

  • Page 504: Allowing Network-Open Users To Download Fabric Manager Web Client

    Fabric Manager Web Client by following these steps: Open the server.properties file in the Fabric Manager installation directory. On a Windows platform, this Step 1 file is in C:\Program Files\Cisco Systems\MDS 9000 by default. Set web.allowDownload4All=true. By default, this property is not set. Step 2 Save and exit the file.

  • Page 505: Verifying Tcp Port For Fabric Manager Web Client

    Environment. Cisco recommends Java version 1.5 or message displays. above, which includes Java Web Start. Refer to the installation instructions in the Cisco MDS 9000 Family Fabric Manager Configuration Guide, or the Release Notes for your specific Fabric Manager release.

  • Page 506: Enabling The Java Runtime Console

    If the web site is not encrypted with SSL, clear the Require server verification (https:) for all sites in this zone check box. Step 5 Click Add. The URL or IP address appears in the Websites list. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-10 OL-9285-05...

  • Page 507: Clearing Java Web Start Cache

    Step 1 Go to the Web Client installation directory and cd to the bin directory. Step 2 Enter the following line to create a user: addUser <userName> <password> Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-11 OL-9285-05...

  • Page 508: Setting Fabric Manager Authentication Method

    Performance Manager gathers network device statistics historically and provides this information graphically using a web browser. It presents recent statistics in detail and older statistics in summary. Performance Manager also integrates with external tools such as Cisco Traffic Analyzer. The Performance Manager has three operational stages: •...

  • Page 509: Performance Manager Generates Java Error

    While Fabric Manager provides real-time views of your network fabric, Device Manager provides a real-time graphic representation of a Cisco MDS 9000 Family switch chassis, including the installed switching modules, the supervisor modules, the status of each port within each module, the power supplies, and the fan assemblies.

  • Page 510: Wrong Version Of Device Manager Is Launched

    Wrong version of Java Web Start cache contains multiple Clear the Java Web Start cache. See Clearing Java Web Device Manager is versions of Device Manager software. Start Cache, page 26-11. opened. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-14 OL-9285-05...

  • Page 511: Appendix

    Steps to Perform Before Calling TAC At some point, you may need to contact your customer support representative or Cisco TAC for some additional assistance. This section outlines the steps that the you should perform prior to contacting your next level of support, as this will reduce the amount of time spent resolving the issue.

  • Page 512: A P P E N D I X A Before Contacting Technical Support

    Step 4 On which switch, host bus adapter (HBA), or storage port is the problem occurring? • Which Cisco SAN-OS software, driver versions, operating systems versions and storage device • firmware are in your fabric? What is the network topology? (In Fabric Manager, go to Tools > Show Tech Support and check •...

  • Page 513: Copying Files To Or From The Switch

    Step 3 Set the server address and the file that you want to copy. Step 4 Select Apply to copy the file. Step 5 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 514: Copying Files Using The Cli

    A short script could be written to be run on the MDS to perform a save and then backup of the configuration. The script only needs to contain two commands: copy running-configuration startup-configuration and then copy startup-configuration tftp://server/name. To execute the script use: run-script filename. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 515: Using Core Dumps

    Use the system cores CLI command to set up core dumps on your switch. switch# system cores tftp://10.91.51.200/jsmith_cores switch# show system cores Cores are transferred to tftp://10.91.51.200/jsmith_cores Note The file name (indicated by jsmith_cores) must exist in the TFTP server directory. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...
  • Page 516 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 517: Appendix

    A P P E N D I X Troubleshooting Tools and Methodology This appendix describes the troubleshooting tools and methodology available for the Cisco MDS 9000 Family multilayer directors and fabric switches. It includes the following sections: Using Cisco MDS 9000 Family Tools, page B-1 •...

  • Page 518: A P P E N D I X B Troubleshooting Tools And Methodology

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Command-Line Interface Troubleshooting Commands The command-line interface (CLI) lets you configure and monitor a Cisco MDS 9000 Family switch using a local console or remotely using a Telnet or SSH session. The CLI provides a command structure ®...
  • Page 519 [FLOGI_ST_PERFORM_CONFIG] Dec 10 23:40:26 flogi: current event [FLOGI_EV_RIB_RESPOSE] Dec 10 23:40:26 flogi: next state [FLOGI_ST_PERFORM_CONFIG] The following is a summary of some of the common debug commands available Cisco SAN-OS: Table B-1 Debug Commands Debug command Purpose Enables AAA debugging.

  • Page 520: Fc Ping And Fc Traceroute

    IP routed network. The traceroute utility operates in a similar fashion, but can also determine the specific path that a frame takes to its destination on a hop-by-hop basis. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 521: Using Fc Ping

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m These tools have been migrated to Fibre Channel for use with the Cisco MDS 9000 Family switches and are called FC ping and FC traceroute.
  • Page 522 2000000c306c2440(0xfffceb) --> MDS which connects directly to the traced FCID (0xeb01e8) Latency 0 msec 2000000c306c2440(0xfffceb) -->idem, but looped around Latency 0 msec 2000000c30575ec0(0xfffced) --> first hop MDS on the return path from traced FCID to originor switch# Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 523: Monitoring Processes And Cpus

    The sum of all dynamically allocated memory that this process has received from the system; this • includes memory that may have been returned to the system The amount of CPU time the process has used, in microseconds • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 524: Using The Show Processes Cli Command

    R = runnable (on run queue). • • S = sleeping. • T = traced or stopped. • Z = defunct (“zombie”) process. NR = not-running. • ER = should be running but currently not-running. • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 525: Viewing Cpu Time In Device Manager

    Use the show processes cpu command to display CPU utilization. The command output includes: • Runtime(ms) = CPU time the process has used, expressed in milliseconds. • Invoked = number of times the process has been invoked. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05...

  • Page 526: Using The Show System Resource Cli Command

    This on-board failure logging (OBFL) feature stores failure and environmental information in nonvolatile memory on the module. The information will help in post-mortem analysis of failed cards. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-10 OL-9285-05...

  • Page 527: Configuring Obfl For The Switch

    Enables the OBFL miscellaneous switch(config)# hw-module logging onboard miscellaneous-error information. Enables the boot uptime, device version, switch(config)# hw-module logging onboard obfl-log and OBFL history. Disables all OBFL features. switch(config)# no hw-module logging onboard Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-11 OL-9285-05...

  • Page 528: Configuring Obfl For A Module

    Use the show logging onboard status command to display the configuration status of OBFL. switch# show logging onboard status Switch OBFL Log: Enabled Module: 6 OBFL Log: Enabled error-stats Enabled exception-log Enabled miscellaneous-error Enabled obfl-log (boot-uptime/device-version/obfl-history) Enabled system-health Enabled stack-trace Enabled Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-12 OL-9285-05...

  • Page 529: Displaying Obfl Logs

    Alerts and Alarms, page B-17 • Device Manager: RMON Threshold Manager, page B-17 For detailed information about using Cisco Fabric Manager, refer to the Cisco MDS 9000 Family Fabric Note Manager Configuration Guide. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 530: Fabric Manager And Device Manager

    Analyzing Switch Device Health Choose the Switch Health option from the Fabric Manager Tools menu to determine the status of the components of a specific switch. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-14 OL-9285-05...

  • Page 531: Analyzing End-To-End Connectivity

    No paths exist. • Only one unique path exists. • VSAN does not have an active zone set. • Average time... micro secs—The latency value was more than the threshold supplied. • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-15 OL-9285-05...

  • Page 532: Analyzing Switch Fabric Configuration

    Analyzing the Results of Merging Zones Cisco Fabric Manager provides a very useful tool for troubleshooting problems that occur when merging zones configured on different switches. Select the Zone Merge option on the Fabric Manager Tools menu to determine if two connected switches have compatible zone configurations.

  • Page 533: Alerts And Alarms

    Management Protocol (SNMP), Remote Monitor (RMON), Syslog, and Call Home alarms and notifications. SNMP provides a set of preconfigured traps and informs that are automatically generated and sent to the destinations (trap receivers) chosen by the user. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-17 OL-9285-05...

  • Page 534: Fibre Channel Name Service

    SCSI target devices attach to the fabric, they register themselves with the name service, which is then distributed among all participating fabric switches. This information can then be used to help determine the identity and topology of nodes connected to the fabric. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-18 OL-9285-05...

  • Page 535: Scsi Target Discovery

    (LUN) details including the number of LUNs, the LUN IDs, and the sizes of the LUNs. This information is then compiled and made available to through CLI commands, through the Cisco Fabric Manager, and also via an embedded SNMP MIB which allows the information to be easily retrieved by an upstream management application.
  • Page 536 Cisco-Proprietary Storage and Storage Network MIBs (for example, NAME-SERVER-MIB) • These MIBs were written by Cisco to help expose information that is discovered within a fabric to management applications not connected to the fabric itself. In addition to exposing configuration details for features like zoning and Virtual SANs (VSANs) via MIBs, discovered information from sources like the FC-GS-3 Name Server can be pulled via a MIB.

  • Page 537: Using Radius

    Authentication refers to the authentication of users for access to a specific device. You can use RADIUS to manage user accounts for access to Cisco MDS 9000 Family switches. When you try to log into a switch, the switch validates you with information from a central RADIUS server.

  • Page 538: Logging Levels

    S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m A unique feature within the Cisco MDS 9000 Family switches is the ability to send RADIUS accounting records to the Syslog service.

  • Page 539: Using Fibre Channel Span

    In Ethernet networks, this problem can be solved using the SPAN utility, which is provided with the Cisco Catalyst Family of Ethernet switches. SPAN has also been implemented with the Cisco MDS 9000 Family switches for use in Fibre Channel networks. SPAN lets you take a copy of all traffic and direct it to another port within the switch.

  • Page 540: Using Cisco Network Management Products

    The Cisco MDS 9000 Family Port Analyzer Adapter is a stand-alone adapter card that converts Fibre Channel frames to Ethernet frames by encapsulating each Fibre Channel frame into an Ethernet frame. This product is meant to be used for analyzing SPAN traffic from a Fibre channel port on a Cisco MDS 9000 Family switch.

  • Page 541: Cisco Fabric Analyzer

    With the Cisco Fabric Analyzer, Cisco has brought Fibre Channel protocol analysis within a storage network to a new level of capability. Using Cisco Fabric Analyzer, you can capture Fibre Channel control traffic from a switch and decode it without having to disrupt any connectivity, and without having to be present locally at the point of analysis.
  • Page 542 The Ethereal application allows remote access to Fibre Channel control traffic and does not require a Fibre Channel connection on the remote workstation. The Cisco Fabric Analyzer lets you capture and decode Fibre Channel traffic remotely over Ethernet. It captures Fibre Channel traffic, encapsulates it in TCP/IP, and transports it over an Ethernet network to the remote client.

  • Page 543: Ip Network Simulator

    • Reordering packets • For more information about using the IP Network Simulator, refer to the Cisco MDS 9000 Family CLI Configuration Guide. Using Other Troubleshooting Products This section describes products from other vendors that you might find useful when troubleshooting problems with your storage network and connected devices.

  • Page 544: Using Host Diagnostic Tools

    An external protocol analyzer (for example from Finisar), is capable of capturing and decoding link level issues and the fibre channel ordered sets which comprise the fibre channel frame. The Cisco MDS 9000 Family Port Analyzer Adapter, does not capture and decode at the ordered set level.
  • Page 545 GUI utility, the dd utility, or a third-party utility like Extreme SCSI. Every UNIX version provides similar utilities, but this guide only provides examples for Solaris. Refer to the documentation for your specific operating system for details. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-29 OL-9285-05...
  • Page 546 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

  • Page 547: Appendix

    A P P E N D I X Configuration Limits for Cisco MDS SAN-OS Release 3.x The features supported by Cisco MDS SAN-OS have maximum configuration limits. For some of the features, we have verified configurations that support limits less that the maximum. Table C-1 lists the Cisco verified limits and maximum limits for switches running Cisco MDS SAN-OS Release 3.x.
  • Page 548 1. Certain design considerations must be met to reach this limit. We recommend that you have the large Fabric design validated by Cisco Advanced Services. 2. This is the number of trunking-enabled ISL ports multiplied by the number of VSANs in the switch.
  • Page 549 BIOS setup (procedure) 2-16 debug commands recovery with dual supervisors 2-23 clock modules 4-12 commit fails border switch fails 13-10 device alias 15-3, 15-11 buffer-to-buffer credits 12-3 See BB_credits Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-1 OL-9285-05...
  • Page 550 Fabric Manager 11-10 merge limitations 15-4 merging mixed fabrics 15-7 troubleshooting checklist 15-1 using blank commit 15-7 validation fails 15-11 EFMD 19-2 VSAN limitations fabric binding 15-3 16-6 Device Manager EISLs Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-2 OL-9285-05...
  • Page 551 26-2 Fibre Connection. See FICON recommened JRE version (table) FICON troubleshooting tools configuration files 16-5 using over FCIP 26-5 configuring 16-1 using with multiple NICs 26-5 CUP in-band management 16-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-3 OL-9285-05...
  • Page 552 SPD compatibility 16-8 22-8 statistics 22-15 supported platforms (table) 22-3 troubleshooting 22-5 verifying configuration 22-6, 22-7 hardware IP security. See IPsec overview IP services 20-5 startup issues iSCSI Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-4 OL-9285-05...
  • Page 553 13-13 module-based overview 13-1 one-click install fails persistent FC IDs 13-12 serial numbers release-specific support (table) 13-5 transfer between switches restriction 23-2 unexpected grace period warnings. restrictions 13-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-5 OL-9285-05...
  • Page 554 8-26 NPIV cycles through up and down states 8-31 restrictions dedicated mode bandwidth (table) VSAN mismatches DPVM membership not in database 11-13 ELP issues 8-27 error disabled 8-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-6 OL-9285-05...
  • Page 555 Fabric Manager tools 23-4 verifying violations using Fabric Manager host login problems 23-7 (procedure) 19-10 initial checklist 23-3 verifying violations using the CLI (procedure) 19-11 interface restrictions 23-3 port swapping Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-7 OL-9285-05...
  • Page 556 2-26 unrecoverable restart See system messages 2-31 upgrading best practices system health failure 2-33 verifying installation system images software images, selecting for supervisor modules selecting for supervisor modules 5-13 5-13 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-8 OL-9285-05...
  • Page 557 FCIP connections 20-9 VSAN trunking. See trunking flowchart FSPF issues 11-24 hardware problems 4-13 IP services 20-5 WWNs, suspended connections iSCSI issues 20-35 modules 4-21 overview power supplies SSM recovery zone Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-9 OL-9285-05...
  • Page 558 14-12 port isolation 8-28 troubleshooting checklist 14-1 troubleshooting with CLI 14-2 troubleshooting with Fabric Manager 14-2 zone set activation 14-8 zone sets maximum number in a switch Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-10 OL-9285-05...
  • Page 559 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...
  • Page 560 S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x...

This manual is also suitable for:

Ds-c9124-k9Ds-c9124ap-k9Ds-c9134-1k9Ds-c9134-k99124 - mds multilayer fabric switchDs-c9124 ... Show all

Table of Contents