Brocade Communications Systems StoreFabric SN6500B Troubleshooting Manual
  1. Manuals
  2. Brands
  3. Brocade Communications Systems Manuals
  4. Software
  5. StoreFabric SN6500B
  6. Troubleshooting manual

Brocade Communications Systems StoreFabric SN6500B Troubleshooting Manual

Brocade fabric os troubleshooting and diagnostics guide v7.1.0 (53-1002751-01, march 2013)
Hide thumbs Also See for StoreFabric SN6500B:
Table of Contents

Advertisement

Quick Links

Download this manual See also: Administrator's Manual, Reference Manual
53-1002751-01
®
®
14 December 2012
Fabric OS
Troubleshooting and Diagnostics Guide
Supporting Fabric OS v7.1.0

Advertisement

Table of Contents
loading

Related Manuals for Brocade Communications Systems StoreFabric SN6500B

Summary of Contents for Brocade Communications Systems StoreFabric SN6500B

  • Page 1 53-1002751-01 ® ® 14 December 2012 Fabric OS Troubleshooting and Diagnostics Guide Supporting Fabric OS v7.1.0...
  • Page 2 Copyright © 2008-2012 Brocade Communications Systems, Inc. All Rights Reserved. Brocade, the B-wing symbol, BigIron, DCX, Fabric OS, FastIron, IronPoint, IronShield, IronView, IronWare, JetCore, NetIron, SecureIron, ServerIron, StorageX, and TurboIron are registered trademarks, and Brocade Network Advisor (formerly Data Center Fabric Manager or DCFM), Extraordinary Networks, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.
  • Page 3 Document History Title Publication number Summary of changes Date Fabric OS Troubleshooting and 53-0000853-01 First released edition. March 2008 Diagnostics Guide Fabric OS Troubleshooting and 53-1001187-01 Added support for Virtual Fabrics, fcPing, November 2008 Diagnostics Guide pathInfo, and additional troubleshooting tips.
  • Page 4 Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

  • Page 5: Table Of Contents

    Contents About This Document How this document is organized ......xi Supported hardware and software ......xii What’s new in this document .
  • Page 6 Switch boot ..........13 Rolling Reboot Detection .
  • Page 7 Brocade configuration form ....... . 48 Chapter 5 Firmware Download Errors Blade troubleshooting tips .
  • Page 8 Chapter 9 Zoning Overview of corrective action ....... 73 Verifying a fabric merge problem ......73 Verifying a TI zone problem .
  • Page 9 System message log ........110 Displaying the system message log, with no page breaks .
  • Page 10 Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

  • Page 11: About This Document

    About This Document In this chapter • How this document is organized ........xi •...

  • Page 12: Supported Hardware And Software

    Supported hardware and software In those instances in which procedures or parts of procedures documented here apply to some switches but not to others, this guide identifies which switches are supported and which are not. Although many different software and hardware configurations are tested and supported by Brocade Communications Systems, Inc.

  • Page 13: Document Conventions

    Document conventions This section describes text formatting conventions and important notice formats used in this document. TEXT FORMATTING The narrative-text formatting conventions that are used are as follows: bold text Identifies command names Identifies the names of user-manipulated GUI elements Identifies keywords and operands Identifies text to enter at the GUI or CLI italic text...

  • Page 14: Notes, Cautions, And Warnings

    NOTES, CAUTIONS, AND WARNINGS The following notices and statements are used in this manual. They are listed below in order of increasing severity of potential hazards. NOTE A note provides a tip, guidance, or advice, emphasizes important information, or provides a reference to related information.

  • Page 15: Other Industry Resources

    White papers, online demonstrations, and data sheets are available through the Brocade website http://www.brocade.com/products-solutions/products/index.page For additional Brocade documentation, visit the Brocade website: http://www.brocade.com Release notes are available on the MyBrocade website and are also bundled with the Fabric OS firmware. OTHER INDUSTRY RESOURCES For additional resource information, visit the Technical Committee T11 website.

  • Page 16: Document Feedback

    • Brocade 300, 5100, and 5300 — On the switch ID pull-out tab located on the bottom of the port side of the switch. • Brocade 6505, 6510, and 6520— On the switch ID pull-out tab located inside the chassis on the port side on the left.

  • Page 17: Introduction

    Chapter Introduction In this chapter • Troubleshooting overview ......... 1 •...

  • Page 18: Most Common Problem Areas

    Most common problem areas Most common problem areas Table 1 identifies the most common problem areas that arise within SANs and identifies tools to use to resolve them. TABLE 1 Common troubleshooting problems and tools Problem area Investigate Tools • •...
  • Page 19 Questions for common symptoms TABLE 2 Common symptoms Symptom Areas to check Chapter or Document Blade is faulty Firmware or application download Chapter 2, “General” Hardware connections Chapter 5, “Firmware Download Errors” Chapter 7, “Virtual Fabrics” Blade is stuck in the “LOADING” state Firmware or application download Chapter 5, “Firmware Download Errors”...
  • Page 20 Questions for common symptoms TABLE 2 Common symptoms (Continued) Symptom Areas to check Chapter or Document License issues Licensing Chapter 2, “General” LSAN is slow or times-out LSAN tagging Chapter 2, “General” Marginal link Links Chapter 3, “Connectivity” No connectivity between host and storage Cables Chapter 3, “Connectivity”...

  • Page 21: Gathering Information For Your Switch Support Provider

    Gathering information for your switch support provider TABLE 2 Common symptoms (Continued) Symptom Areas to check Chapter or Document User is unable to change switch settings RBAC settings Chapter 6, “Security” Account settings Virtual Fabric does not form FIDs Chapter 7, “Virtual Fabrics” Zone configuration mismatch Effective configuration Chapter 9, “Zoning”...

  • Page 22: Capturing Output From A Console

    Gathering information for your switch support provider 2. Type the appropriate supportSave command based on your needs: • If you are saving to an FTP or SCP server, use the following syntax: supportSave When invoked without operands, this command goes into interactive mode. The following operands are optional: -n Does not prompt for confirmation.

  • Page 23: Capturing Command Output

    Building a case for your switch support provider 3. Set the utility to capture output from the screen. Some utilities require this step to be performed prior to opening up a session. Check with your utility vendor for instructions. 4. Type the command or start the process to capture the required data on the console. Capturing command output 1.

  • Page 24: Detailed Problem Information

    Building a case for your switch support provider • Do you have IPsec installed on the switch’s Ethernet interface? Yes or no. • Use the ipsecConfig show command to determine the answer. • Do you have Inband Management installed on the switches GigE ports? Yes or no. •...

  • Page 25: Gathering Additional Information

    Building a case for your switch support provider • Host information: OS version and patch level HBA type HBA firmware version HBA driver version Configuration settings • Storage information: Disk/tape type Disk/tape firmware level Controller type Controller firmware level Configuration settings Storage software (such as EMC Control Center, Veritas SPC, etc.) •...
  • Page 26 Building a case for your switch support provider Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

  • Page 27: General

    Chapter General In this chapter • Licenses ............11 •...

  • Page 28: Frame Viewer

    Frame Viewer Frame Viewer When a frame is unable to reach its destination due to timeout, it is discarded. You can use Frame Viewer to find out which flows contained the dropped frames, which can help you determine which applications might be impacted. Using Frame Viewer, you can see exactly what time the frames were dropped.

  • Page 29: Switch Boot

    Switch boot Symptom MQ errors are appearing in the switch log. Probable cause and recommended action An MQ error is a message queue error. Identify an MQ error message by looking for the two letters MQ followed by a number in the error message: 2004/08/24-10:04:42, [MQ-1004], 218,, ERROR, ras007, mqRead, queue = raslog-test- string0123456-raslog, queue I D = 1, type = 2...

  • Page 30: Rolling Reboot Detection

    Switch boot Rolling Reboot Detection A rolling reboot occurs when a switch or enterprise-class platform has continuously experienced unexpected reboots. This behavior is continuous until the rolling reboot is detected by the system. Once the Rolling Reboot Detection (RRD) occurs, the switch is put into a stable state so that a minimal supportSave can be collected and sent to your service support provider for analysis.
  • Page 31 Switch boot This is an HA bootup-related issue and happens when switch is unable to recover to a stable state. HASM log contains more detail and specific information on this type of failure, such as one of the following: Failover recovery failed: This occurs when failover recovery failed and has to reboot the CP. Failover when standby CP unready: Occurs when the active CP has to failover, but the standby CP is not ready to take over mastership.

  • Page 32: Fc-Fc Routing Connectivity

    FC-FC routing connectivity * Fabric OS has detected frequent switch reboot condition. * * Following actions can be taken to recover the switch: * * - take off or replace the bad blades. * * - use supportsave to collect supportsave data. * ************************************************************** Please change passwords for switch default accounts now.

  • Page 33: Superping

    FC-FC routing connectivity received reply from 0x020800: 12 bytes time:1010 usec 5 frames sent, 5 frames received, 0 frames rejected,0 frames timeout Round-trip min/avg/max = 1006/1044/1159 usec Regardless of the device’s zoning configuration, the fcPing command sends the ELS frame to the destination port.
  • Page 34 FC-FC routing connectivity ATTENTION There are some devices that do not support the ELS ECHO request. In these cases, the device either does not respond to the request or send an ELS reject. When a device does not respond to the ELS request, further debugging is required;...
  • Page 35 FC-FC routing connectivity In the following example, superping is invoked using the fcPing --allpaths command to destination domain 165. The following example displays each hop as (Domain1/Index1-> Domain2/Index2) format. To reach destination domain 165 from source domain 3 there are 2 unique end-to-end paths.

  • Page 36: Route And Statistical Information

    FC-FC routing connectivity Restrictions • Fabric reconfiguration cannot occur while using the superping tool. It is assumed that the fabric is stable before the fcPing --allpaths command is executed. • The control path for interswitch communication should be available, even if the data path for device to device communication may have resource starvation.

  • Page 37: Performance Issues

    FC-FC routing connectivity switch:admin> pathinfo 5 In Port Domain ID (Name) Out Port Cost ----------------------------------------------------------------------------- 1 (sw0) 2 (sw0) 3 (sw0) 4 (sw0) 10000 7 (switch_3) 5 (switch_3) To display basic and extended statistics in interactive mode: switch:admin> pathinfo Max hops: (1..127) [25] Fabric Id: (1..128) [-1] Domain|Wwn|Name: [] 8 Source port: (0..15) [-1]...
  • Page 38 FC-FC routing connectivity Symptom Host application times out. Probable cause and recommended action The FCR tends to take a long time, more than 5 seconds, to present and setup paths for the proxy devices. Certain hosts are able to do discovery much faster as a result they end up timing out. Use the speed tag feature to always present target proxy to the host and import them faster.

  • Page 39: Connectivity

    Chapter Connectivity In this chapter • Port initialization and FCP auto-discovery process ....23 • Link issues ........... . 25 •...
  • Page 40 Port initialization and FCP auto-discovery process • VEX_Port—A virtual EX_Port. It connects a Fibre Channel router to an edge fabric. From the point of view of a switch in an edge fabric, a VEX_Port appears as a normal VE_Port. It follows the same Fibre Channel protocol as other VE_Ports.

  • Page 41: Link Issues

    Link issues Link issues Symptom Port LEDs are flashing. Probable cause and recommended action Depending on the rate of the flash and the color of the port LED this could mean several things. To determine what is happening on either your port status LED or power status LED, refer to that switch’s model hardware reference manual.

  • Page 42: Checking The Logical Connection

    Connection problems Checking the logical connection 1. Enter the switchShow command. 2. Review the output from the command and determine if the device successfully logged in to the switch. • A device that is logically connected to the switch is registered as an F_, L_, E_, EX_, VE_, VEX_, or N_Port.
  • Page 43 Connection problems 021a00; 2,3;20:00:00:e0:69:f0:07:c6;10:00:00:e0:69:f0:07:c6; 895 Fabric Port Name: 20:0a:00:60:69:10:8d:fd 051edc; 3;21:00:00:20:37:d9:77:96;20:00:00:20:37:d9:77:96; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee0; 3;21:00:00:20:37:d9:73:0f;20:00:00:20:37:d9:73:0f; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee1; 3;21:00:00:20:37:d9:76:b3;20:00:00:20:37:d9:76:b3; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee2;...

  • Page 44: Link Failures

    Link failures Link failures A link failure occurs when a server, storage, or switch device is connected to a switch, but the link between the devices does not come up. This prevents the devices from communicating to or through the switch. If the switchShow command or LEDs indicate that the link has not come up properly, use one or more of the following procedures.

  • Page 45: Checking For A Loop Initialization Failure

    Link failures Checking for a loop initialization failure 1. Verify the port is an L_Port. a. Enter the switchShow command. b. Check the last field of the output to verify that the switch port indicates an L_Port. If a loop device is connected to the switch, the switch port must be initialized as an L_Port.

  • Page 46: Correcting A Port That Has Come Up In The Wrong Mode

    Marginal links After becoming an active port, the port becomes an F_Port or an E_Port depending on the device on the opposite side. If the opposite device is a host or target device, the port becomes an F_Port. If the opposite device is another switch, the port becomes an E_Port. If there is a problem with the host or target device, enter portCfgGPort to force the port to try to come up as point-to-point only.

  • Page 47: Troubleshooting A Marginal Link

    Marginal links Only one frame is transmitted and received at any given time. An external cable is not required to run this test. The port LEDs flicker green rapidly while the test is running. Table 5 shows the different loopback modes you can use when using portLoopbackTest to test a marginal link.
  • Page 48 Marginal links switch:admin> porterrshow frames enc disc link loss loss frjt fbsy err g_eof shrt long eof out c3 fail sync sig ============================================================================ 665k 7.0k 665k 7.4k (output truncated) 3. If you suspect a marginal link, isolate the areas by moving the suspected marginal port cable to a different port on the switch.

  • Page 49: Device Login Issues

    Device login issues Device login issues A correct login is when the port type matches the device type that is plugged in. In the following example, it shows that the device connected to Port 1 is a fabric point-to-point device and it is correctly logged in an F_Port.
  • Page 50 Device login issues In some cases, you may find that the port has been locked as an L_Port and the device attached is a fabric point-to-point device such as a host or switch. This would be an incorrect configuration for the device and therefore the device cannot log into the switch. To correct this type of problem, remove the Lock L_Port configuration using the portCfgDefault command.

  • Page 51: Media-Related Issues

    Media-related issues ------------------------------------------------- Thu Nov 6 16:52:39 2008 16:52:39.066 PORT 00010004,4302000f,02000000 16:52:39.066 PORT ce3dfab0,d9672800,00000002 16:52:39.066 PORT ce3dfab0,d9672800,00000080 16:52:39.066 PORT 00000000,00000000,00000002 16:52:39.066 PORT 00010004,4302000f,00000002 16:52:39.066 PORT 00010004,4302000f,02000000 16:52:39.071 PORT ioctl 88010004 1,0 * 4 16:52:42.311 SPEE 00000000,00000000,00000000 16:52:42.558 SPEE 00000000,00000000,00000000 16:52:42.558 SPEE 00000000,00000000,00000000 16:52:42.558 SPEE 00000001,00000000,00000000...

  • Page 52: Testing A Port's External Transmit And Receive Path

    Segmented fabrics Testing a port’s external transmit and receive path 1. Connect to the switch and log in as admin. 2. Connect the port you want to test to any other switch port with the cable you want to test. 3.

  • Page 53: Reconciling Fabric Parameters Individually

    Segmented fabrics • Incompatible zoning configuration (see Chapter 9, “Zoning”). • Domain ID conflict (see “Reconciling fabric parameters individually” on page 37). • Fabric ID conflict (see Chapter 7, “Virtual Fabrics”). • Incompatible security policies. • Incorrect fabric mode. • Incorrect policy distribution.

  • Page 54: Downloading A Correct Configuration

    Segmented fabrics Alternatively, you can reconcile fabric parameters by entering the configUpload command for each switch and upload a known-good configuration file. If you do this option, the two switches must be the same model. Downloading a correct configuration You can restore a segmented fabric by downloading a previously saved correct backup configuration to the switch.

  • Page 55: Reconciling Incompatible Software Features

    Segmented fabrics Fabric parameters (yes, y, no, n): [no] y Domain: (1..239) [1] 89 WWN Based persistent PID (yes, y, no, n): [no] Allow XISL Use (yes, y, no, n): [yes] R_A_TOV: (4000..120000) [10000] E_D_TOV: (1000..5000) [2000] WAN_TOV: (0..30000) [0] MAX_HOPS: (7..19) [7] Data field size: (256..2112) [2112] Sequence Level Switching: (0..1) [0]...

  • Page 56: Port Mirroring

    Port mirroring Port mirroring With port mirroring, you can configure a switch port to mirror the traffic between a specific source and destination port. This is only supported between F_Ports. This is a useful way to troubleshoot a problem port without bringing down the host and destination links to insert an inline analyzer. Port mirroring captures traffic between two devices.

  • Page 57: Supported Platforms

    Port mirroring • Using the firmware download procedure to downgrade to previous Fabric OS releases that do not support port mirroring requires that you remove all the port mirroring connections. If you downgrade to a previous versions of Fabric OS, you cannot proceed until the mirroring connections are removed.

  • Page 58: Maximum Mirror Connections

    Port mirroring Maximum mirror connections Table 9 shows the maximum number of mirror connections you can add to a mirror port. TABLE 9 Maximum number of mirror connections Model Maximum Number of Mirror Connections (chassis-wide) Brocade 300 Brocade 5100 Brocade 5300 Brocade 5410 Brocade 5450 Brocade 5460...

  • Page 59: Deleting A Port Mirror Connection

    Port mirroring The configuration database keeps information about the number of port mirror connections configured on a switch, the number of chunks of port mirroring data that are stored, and the chunk number. When removing a mirror connection, always use this method to ensure that the data is cleared.
  • Page 60 Port mirroring Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

  • Page 61: Configuration

    Chapter Configuration In this chapter • Configuration upload and download issues ......45 • Brocade configuration form ........48 Configuration upload and download issues It is important to maintain consistent configuration settings on all switches in the same fabric because inconsistent parameters (such as inconsistent PID formats) can cause fabric...
  • Page 62 Configuration upload and download issues • The FTP or SCP server’s IP address cannot be contacted. Verify that you can connect to the FTP server. Use your local PC to connect to the FTP server or ping the FTP server. Example of a successful ping C:\>...
  • Page 63 Configuration upload and download issues • On a Virtual Fabrics-enabled switch, you do not have the chassis role permission set on your user account. Implement one change at a time, then issue the command again. By implementing one change at a time, you are able to determine what works and what does not work.

  • Page 64: Gathering Additional Information

    Brocade configuration form Symptom Configuration did not seem to change after the configuration download process finished. Probable cause and recommended action Verify that the switch was rebooted by checking the system log. If you are doing this on a enterprise-class platform, verify that both CPs rebooted by checking the system log. If any error occurs during the download, such as an error about a particular key, it is important to issue the configDefault command and attempt to repeat the configDownload command.
  • Page 65 Brocade configuration form TABLE 10 Brocade configuration and connection (Continued) Brocade configuration settings Value Domain ID Switch name Ethernet IP address Ethernet subnet mask Total number of local devices (nsShow) Total number of devices in fabric (nsAllShow) Total number of switches in the fabric (fabricShow) Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...
  • Page 66 Brocade configuration form Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

  • Page 67: Chapter 5 Firmware Download Errors

    Chapter Firmware Download Errors In this chapter • Blade troubleshooting tips ........51 •...

  • Page 68: Firmware Download Issues

    Firmware download issues Symptom The blade is faulty (issue slotShow to confirm). Probable cause and recommended action If the port or application blade is faulty, enter the slotPowerOff and slotPowerOn commands for the port or application blade. If the port or application blade still appears to be faulty, remove it and re-insert it into the chassis.
  • Page 69 Firmware download issues Symptom Cannot upgrade directly to v6.3.0. Probable cause and recommended action If the switch is running Fabric OS v6.1.0 or earlier, you are not allowed to upgrade directly to v6.3.0 because of the “one-version” rule. Upgrade your switch to Fabric OS version v6.2.0 before upgrading to v6.3.0.

  • Page 70: Troubleshooting With The Firmwaredownload Command

    Troubleshooting with the firmwareDownload command Network Protocol(1-auto-select, 2-FTP, 3-SCP) [1]: 2 Password: <hidden> Server IP: 192.168.168.115, Protocol IPv4 Checking system settings for firmwaredownload... Cannot download the requested firmware because the firmware doesn't support this platform. Please enter another firmware path. Symptom Cannot download on a switch with Interop mode turned on.

  • Page 71: Gathering Additional Information

    USB error handling If the firmware download fails in a director or enterprise-class platform, the CPs may end up with different versions of firmware and are unable to achieve HA synchronization. In such cases, issue the firmwareDownload -s command on the standby CP; the single mode (-s) option allows you to upgrade the firmware on the standby CP to match the firmware version running on the active CP.

  • Page 72: Considerations For Downgrading Firmware

    Considerations for downgrading firmware Considerations for downgrading firmware The pre-installation check of the firmwareDownload command detects all of the blocking conditions that can prevent a successful downgrade, and warns you about all these conditions. The error messages displayed by the firmwareDownload command states the blocking conditions and the corresponding commands to correct them.
  • Page 73 Considerations for downgrading firmware Checking version compatibility... Version compatibility check passed.. The following items must be addressed before downloading the specified firmware: FC8-32E and FC8-48E are not supported by the targeted firmware. Please use slotshow to determine which of these are installed and remove them before continuing.

  • Page 74: Blade Types

    Considerations for downgrading firmware Downgrade is not allowed because R-RDY flow control ports are configured as D-Ports. Please use "portdporttest --show all" to view the port list and "portcfgdport --disable" to disable before downgrading. Downgrade is not allowed because D-Port is configured with DWDM mode. Please use "portcfgshow"...

  • Page 75: Platform

    Considerations for downgrading firmware Message Cannot upgrade directly to v6.3.0. Upgrade your switch to v6.2.0 first before upgrading to the requested version. Probable cause and recommended action If the switch is running v6.1.0 or earlier, you are not allowed to upgrade directly to v6.3.0 because of the “two-version”...
  • Page 76 Considerations for downgrading firmware Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

  • Page 77: Chapter 6 Security

    Chapter Security In this chapter • Passwords ........... . 61 •...

  • Page 78: Password Recovery Options

    Device authentication Symptom Unable to log into the boot PROM. Probable cause and recommended action To recover a lost boot PROM password, contact your switch service provider. You must have previously set a recovery string to recover the boot PROM password. This does not work on lost or forgotten passwords in the account database.

  • Page 79: Protocol And Certificate Management

    Protocol and certificate management Symptom Switch is unable to form an F_Port. Probable cause and recommended action Regardless of the device authentication policy mode on the switch, the F_Port is disabled if the DH-CHAP protocol fails to authenticate. If the HBA sets the FC-SP bit during FLOGI and the switch sends a FLOGI accept with FC-SP bit set, then the switch expects the HBA to start the AUTH_NEGOTIATE.

  • Page 80: Snmp

    SNMP • If you think this is a policy-related issue, FCS switch or other security server-related issue then use supportSave -n to collect data from the Primary FCS switch and all affected switches. • If login-related, then also include the following information: Does login problem appear on a Serial, CP IP, or Switch IP address connection? Is it CP0 or CP1? Is the CP in active or standby?

  • Page 81: General Virtual Fabrics Troubleshooting

    Chapter Virtual Fabrics In this chapter • General Virtual Fabrics troubleshooting ......65 •...

  • Page 82: Fabric Identification Issues

    Fabric identification issues • VE_Ports cannot exist in a logical switch that has XISL use turned on. Although VE_Ports are allowed in a base switch, Fabric OS v6.2.0 and later do not support the use of VE_Ports to carry traffic for logical fabrics using XISLs. They can be used to carry FCR traffic through EX_Ports and VEX_Ports.

  • Page 83: Base Switch Issues

    Base switch issues Symptom The switch with domain <domain> with firmware version <fw version> has joined the FID <fid> fabric and may not be compatible with XISL use. Probable cause and recommended action This message indicates the specified switch in the logical fabric that is using XISLs is running an incompatible firmware version and must be upgraded to Fabric OS v6.2.0 or later.

  • Page 84: Logical Switch Issues

    Logical switch issues Logical switch issues CAUTION When a logical switch is created, all configuration for the logical switch is set to factory defaults. When a logical switch is deleted, all configuration for the logical switch is deleted permanently and is not recoverable. Symptom The indicated slot is empty.

  • Page 85: Switch Configuration Blade Compatibility

    Switch configuration blade compatibility Symptom A non-GE blade is within the slot range. Probable cause and recommended action You are attempting to configure a GE port on a slot that does not contain GE ports. Symptom A port or ports is already in the current switch. Probable cause and recommended action You may not move a port to the same switch.

  • Page 86: Gathering Additional Information

    Gathering additional information SW BLADE FC16-32 ENABLED SW BLADE FC16-48 ENABLED SW BLADE FC16-48 ENABLED CORE BLADE CR16-8 ENABLED CP BLADE ENABLED CP BLADE ENABLED CORE BLADE CR16-8 ENABLED SW BLADE FC8-32 ENABLED SW BLADE FC16-48 ENABLED AP BLADE FX8-24 ENABLED SW BLADE FC16-32...

  • Page 87: Chapter 8 Isl Trunking

    Chapter ISL Trunking In this chapter • Link issues ........... . 71 •...

  • Page 88: Buffer Credit Issues

    Buffer credit issues • Trunking is not supported in switch interoperability mode. • Port trunking is disabled. • The port is not an E_Port. • The port is not 2 Gbps, 4 Gbps, or 8 Gbps. • The port connects to a switch other than the one you want it to. To correct this issue, connect additional ISLs to the switch with which you want to communicate.

  • Page 89: Overview Of Corrective Action

    Chapter Zoning In this chapter • Overview of corrective action ........73 •...

  • Page 90: Segmented Fabrics

    Segmented fabrics If you enter the cfgShow command to display information about all zones, the TI zones appear in the defined zone configuration only and do not appear in the effective zone configuration. 1. Connect to the switch and log in as admin. 2.

  • Page 91: Zone Conflicts

    Zone conflicts Symptom Zone conflict appears in logs and fabric is segmented. Probable cause and recommended action This issue is usually caused by incompatible zoning configurations. Verify the following are true: • The effective cfg (zone set) on each end of the segmented ISL is identical. •...

  • Page 92: Correcting A Fabric Merge Problem Quickly

    Zone conflicts TABLE 14 Commands for debugging zoning (Continued) Command Function cfgEnable Use to enable and activate (make effective) the specified configuration. cfgSave Use to save changes to the zone configuration database. cfgTransAbort Use to abort the current zoning transaction without committing it. cfgTransShow Use to display the ID of the current zoning transaction defZone...

  • Page 93: Changing The Default Zone Access

    Zone conflicts Changing the default zone access A switch is not allowed to merge with another switch that has an active effective configuration if the default zone is set to “no access”. Before the switch can join, the default zone setting has to be set to “all access”.

  • Page 94: Checking For Fibre Channel Connectivity Problems

    Zone conflicts Checking for Fibre Channel connectivity problems Enter the fcPing command (refer to the Fabric OS Command Reference for more information on this command), which checks the zoning configuration for the two ports specified by: • Generates an Extended Link Service (ELS) frame ECHO request to the source port specified and validates the response.

  • Page 95: Checking For Zoning Problems

    Zone conflicts received reply from 0x020800: 12 bytes time:1006 usec received reply from 0x020800: 12 bytes time:1008 usec received reply from 0x020800: 12 bytes time:1038 usec received reply from 0x020800: 12 bytes time:1010 usec 5 frames sent, 5 frames received, 0 frames rejected, 0 frames timeout Round-trip min/avg/max = 1006/1044/1159 usec Pinging 22:00:00:04:cf:75:63:85 [0x217d9] with 12 bytes of data: Request timed out...

  • Page 96: Gathering Additional Information

    Gathering additional information Gathering additional information Collect the data from a supportSave -n command. Then collect the data from the cfgTransShow command. For the port having the problem, collect the data from the filterPortShow <port> command. Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

  • Page 97: Diagnostic Features

    Chapter Diagnostic Features In this chapter • About Fabric OS diagnostics ........81 •...

  • Page 98: Power-On Self Test

    Power-on self test Most information can be captured using the supportSave command and downloaded by FTP off the switch, but when you are collecting information from specialized commands, such as supportShow, this information has to be captured using a Telnet client. To save a set of files that customer support technicians can use to further diagnose the switch condition, enter the supportSave command.
  • Page 99 Power-on self test modprobe: modprobe: Can't open dependencies file /lib/modules/2.4.19/modules.dep (No such file or directory) INIT: version 2.78 booting INIT: Entering runlevel: 3 eth0: Link status change: Link Up. 100 Mbps Full duplex Auto (autonegotiation complete). INITCP: CPLD Vers: 0x95 Image ID: 0x19 uptime: 2008;...

  • Page 100: Disabling Post

    Switch status Disabling POST A reboot is not required for this command to take effect. 1. Connect to the switch and log in with a user account that has admin privileges with the chassis-role permission. 2. Enter the diagDisablePost command. This disables POST1 and POST2.

  • Page 101: Displaying Switch Information

    Switch status Faulty ports monitor HEALTHY Missing SFPs monitor HEALTHY All ports are healthy For more information on how the overall switch status is determined, refer to the switchStatusPolicySet command in the Fabric OS Command Reference. Displaying switch information Table 15 lists the switch summary information TABLE 15 Switch summary information...

  • Page 102: Displaying The Uptime For A Switch

    Switch status The switchShow command also displays the following information for ports on the specified switch: TABLE 17 switchShow command output Variable Definition Index Index follows Area up to 255. Then it continues to the maximum port of the platform. Index identifies the port number relative to the switch.

  • Page 103: Using The Spinfab And Porttest Commands

    Using the SpinFab and portTest commands Using the SpinFab and portTest commands The spinFab command is an online diagnostics command to verify the ISL links between switches at the maximum speed. It is done by setting up the routing functionality in the hardware such that the test frames received by E_Port are retransmitted on the same E_Port.

  • Page 104: Debugging Spinfab Errors

    Using the SpinFab and portTest commands Debugging spinFab errors Link errors and tx/rx errors are seen when the spinFab test fails. Link errors Once the frame is sent out of the port, the spinFab command monitors the link errors in the ASIC. If any of the error counters are non-zero, spinFab reports ERROR and the test fails on the port.

  • Page 105: Clearing The Error Counters

    Using the SpinFab and portTest commands • Further isolation can be done by running the portLoopbackTest command (Offline test) on the failing port in-order to check whether the blade internal ports are having some problems. The -lb_mode 1 operand verifies that the SFP is working within normal operating parameters.

  • Page 106: Disabling A Port

    Diagnostic Port (D_Port) Disabling a port 1. Connect to the switch and log in using an account with admin permissions. 2. Enter the appropriate command based on the current state of the port and on whether it is necessary to specify a slot number:. •...

  • Page 107: Understanding D_Port

    Diagnostic Port (D_Port) Understanding D_Port The D_Port does not carry any user traffic, and is designed to run only specific diagnostics tests on it for identifying link-level faults or failures. Basically, in order to bring up a port in D_Port mode, you must configure both ends of the link between a given pair of switches (or switches configured as Access Gateways), and you must disable the existing port before you can configure it as a D_Port.
  • Page 108 Diagnostic Port (D_Port) • Size of test frames • Duration of test • User-defined test payload • Predefined pattern for use in test payload • Testing with forward error correction (FEC) on or off (default is off) • Testing with credit recovery (CR) on or off (default is off) D_Port test initiation modes and test start behavior Table 20 summarizes D_Port test initiation modes and test start behavior.

  • Page 109: Supported Topologies

    Diagnostic Port (D_Port) Supported topologies The following supported topologies illustrate at a high level how D_Port functionality can be used: • “Topology 1: ISLs” on page 93 • “Topology 2: ICLs” on page 93 • “Topology 3: Access Gateways” on page 94 •...
  • Page 110 Diagnostic Port (D_Port) FIGURE 5 ICLs connecting chassis blades For configuration details see “Using D_Port without HBAs” on page 95. Topology 3: Access Gateways Figure 6 illustrates a switch configured as a single Access Gateway connected to a fabric switch. N and F represent, respectively, an N_Port and an F_Port to be configured as D_Ports.

  • Page 111: Using D_Port Without Hbas

    Diagnostic Port (D_Port) Topology 4: HBA to switch Figure 8 illustrates connectivity between an HBA and a switch. F represents an F_Port to be configured as a D_Port. This topology supports dynamic D_Port mode. In dynamic mode, the HBA port does not need to be configured explicitly as a D_Port. It comes up in D_Port mode if the corresponding switch port has been configured as a D_Port.
  • Page 112 Diagnostic Port (D_Port) 1. Disable Port 1 on Switch A, by using the portDisable [slot/]port command. switchA:admin> portdisable 1 2. Configure Port 1 on Switch A as a D_Port, by using portCfgDport - -enable [slot/]port. switchA:admin> portcfgdport --enable 1 3. Repeat steps 1 and 2 for the corresponding port (in this example Port 2) on Switch B. switchB:admin>...

  • Page 113: Using D_Port With Hbas

    Diagnostic Port (D_Port) To display a summary of the D_Port, use the portDportTest [slot/]port command with the - -show all operand. switch:admin> portdporttest --show all Port State SFP Capabilities Test Result ============================================= 24 ONLINE E,O PASSED 26 ONLINE E,O RESPONDER 33 OFFLINE --- RESPONDER 8.
  • Page 114 Diagnostic Port (D_Port) HBAs also support testing in dynamic mode, described as follows. If D_Port is enabled on the switch only, it forces the connected adapter port into D_Port mode. The switch initiates and stops tests on the adapter port as specified by the switch configuration. Testing is started by means of BCU commands or HCM options.
  • Page 115 Diagnostic Port (D_Port) BCU D_Port commands The following are BCU commands that you can use for D_Port configuration and control: • bcu diag dportenable — Enables D_Port on a specific port, sets the test pattern, and sets the frame count for testing. •...

  • Page 116: Controlling Testing

    Diagnostic Port (D_Port) Controlling testing You can stop and start D_Port testing on a port by using the following respective commands: • portdporttest - -stop [slot/]port • portdporttest - -start [slot/]port Available suboptions to the - --start option are as follows: •...
  • Page 117 Diagnostic Port (D_Port) 4. Install an SFP and attempt to enable the D_Port. switch:admin> bcu diag --dportenable 1/0 ERROR: Switch port is not D_Port capable or D_Port is disabled 5. Connect to the HBA without the SFP and disable the native port. switch:admin>...
  • Page 118 Diagnostic Port (D_Port) The default value 1 for 1 million frames. -framesize - Specifies the size of test frames that are generated to run the test. The Range is minimum 36 bytes and maximum 2112 bytes The size of the frames should be given in multiples of 4 otherwise the nearest higher multiple of 4 value will be taken as frame size.
  • Page 119 Diagnostic Port (D_Port) Example The following example shows the results of a portdporttest - -show output where the electrical and optical tests pass but the link test fails. switch:admin> portdporttest --show 10/39 D-Port Information: =================== Slot: Port: Remote WWNN: 10:00:00:05:33:7e:69:c4 Remote port: Mode: Manual...
  • Page 120 Diagnostic Port (D_Port) Example Use the switchShow command to see D_Port information as in the following example. switch:admin> switchshow switchName: switch_10 switchType: 109.1 switchState: Online switchMode: Native switchRole: Principal switchDomain: switchId: fffc01 switchWwn: 10:00:00:05:33:13:2f:b4 zoning: switchBeacon: FC Router: Allow XISL Use: ON LS Attributes: [FID: 10, Base Switch: No, Default Switch: No, Address Mode 0] Index Port Address Media Speed State...

  • Page 121: Port Information

    Port information Port information Use the following instructions to view information about ports and to help diagnose if your switch is experiencing port problems. Viewing the status of a port 1. Connect to the switch and log in as admin. 2.

  • Page 122: Displaying The Port Statistics

    Port information Displaying the port statistics 1. Connect to the switch and log in as admin. 2. At the command line, enter the portStatsShow command. Port statistics include information such as the number of frames received, number of frames sent, number of encoding errors received, and number of class 2 and class 3 frames received.

  • Page 123: Displaying A Summary Of Port Errors For A Switch

    Port information Displaying a summary of port errors for a switch 1. Connect to the switch and log in as admin. 2. Enter the portErrShow command. Refer to the Fabric OS Command Reference for additional portErrShow command information. switch:admin> porterrshow frames enc disc link loss loss frjt fbsy err g_eof shrt long...

  • Page 124: Equipment Status

    Equipment status TABLE 21 Error summary description (Continued) Error type Description too shrt Frames shorter than minimum too long Frames longer than maximum bad eof Frames with bad end-of-frame delimiters enc out Encoding error outside of frames disc c3 Class 3 frames discarded link fail Link failures (LF1 or LF2 states) loss sync...

  • Page 125: Checking The Status Of A Power Supply

    Equipment status Fan 1 is Absent Fan 2 is Ok, speed is 6553 RPM Fan 3 is Ok, speed is 6367 RPM The possible status values are: • OK—Fan is functioning correctly. • Absent—Fan is not present. • Below minimum—Fan is present but rotating too slowly or stopped. •...

  • Page 126: System Message Log

    System message log Information displays for each temperature sensor in the switch. The possible temperature status values are: • OK—Temperature is within acceptable range. • FAIL—Temperature is outside of acceptable range. Damage might occur. Refer to the hardware reference manual for your switch to determine the normal temperature range.

  • Page 127: Displaying The System Message Log One Message At A Time

    Port log Displaying the system message log one message at a time 1. Connect to the switch and log in as admin. 2. Enter the errShow command. Clearing the system message log 1. Connect to the switch and log in as admin. 2.
  • Page 128 Port log 16:48:47.263 PORT 02fffffd,00fffffd,02e4ffff,14000000 16:48:47.263 PORT c0fffffd,00fffffd,02e401c2,00000001 16:48:47.263 FCPH read 02fffffd,00fffffd,c0000000,00000000,02e401c2 16:48:47.263 FCPH 22380000,02e401c2,00000c1e,0000001c,00000000 <output truncated> Use the commands summarized in Table 22 to view and manage port logs. Refer to the Fabric OS Command Reference for additional information about these commands. TABLE 22 Commands for port log management Command...

  • Page 129: Syslogd Configuration

    Syslogd configuration 20:29:20.806 FCPH read 02fffffd,00fffffd,d2000000,00000000,02ee09a8 20:29:20.806 FCPH 22380000,02ee09a8,00000608,0000001c,00000000 20:29:32.638 FCPH write 00fffffd,00fffffd,00000000,00000000,00000000 20:29:32.638 FCPH 00300000,00000000,000005f4,00020182,00000000 20:29:32.638 PORT 02fffffd,00fffffd,09a9ffff,14000000 20:29:32.638 FCPH write 40 00fffffd,00fffffd,00000000,00000000, 00000000 20:29:32.638 FCPH 00300000,00000000,000005f4,00020182,00000000 20:29:32.639 PORT 02fffffd,00fffffd,09aaffff,14000000 <output truncated> Syslogd configuration The system logging daemon (syslogd) is an IP-based service for logging system messages made available by default on UNIX and Linux operating systems.

  • Page 130: Configuring The Switch

    Syslogd configuration local7.info /var/adm/swinfo local7.debug /var/adm/debug7 If you prefer to map Fabric OS severities to a different UNIX local7 facility level, see “Setting the facility level” on page 114. Configuring the switch Configuring the switch involves specifying syslogd hosts and, optionally, setting the facility level. You can also remove a host from the list of syslogd hosts.

  • Page 131: Automatic Trace Dump Transfers

    Automatic trace dump transfers 3. Verify the IP address was deleted using the syslogdIpShow command. Automatic trace dump transfers You can set up a switch so that diagnostic information is transferred automatically to a remote server. If a problem occurs, you can then provide your customer support representative with the most detailed information possible.

  • Page 132: Saving Comprehensive Diagnostic Files To The Server

    Automatic trace dump transfers Saving comprehensive diagnostic files to the server 1. Connect to the switch and log in as admin. 2. Enter the supportSave -c command and respond to the prompts. switch:admin> supportsave -c This command will collect RASLOG, TRACE, supportShow, core file, FFDC data and other support information and then transfer them to a FTP/SCP server or a USB device.

  • Page 133: Appendix A Switch Type And Blade Id

    Appendix Switch Type and Blade ID The switchType is a displayed field listed when you run the switchShow command. When you are gathering information to give to your switch support provider, you may be asked the switch model. If you do not know the model, you can use this chart to convert the switchType to a B-Series model number.
  • Page 134 Switch Type and Blade ID TABLE 24 switchType to B-Series model converter (Continued) switchType B-Series switch model Base switch speed Brocade Encryption Switch 8 Gb 16-port encryption switch 5410 8 Gb 12-port embedded switch 8 Gb 16-port switch 5480 8 Gb 24-port embedded switch 5470 8 Gb 20-port embedded switch M5424...
  • Page 135 Switch Type and Blade ID TABLE 25 B-series blade model descriptions (Continued) Blade ID B-series blade model Description FCOE10-24 24-FC ports on an application blade that provides Converged Enhanced Ethernet to bridge a Fibre Channel and Ethernet SAN. FX8-24 24-FC port with 10 1-GbE and two 10-GbE ports Fibre Channel routing and FCIP blade FC8-64 8 Gb 64-FC ports blade...
  • Page 136 Switch Type and Blade ID Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

  • Page 137: Appendix B Hexadecimal Conversion

    Appendix Hexadecimal Conversion Hexadecimal overview Hexadecimal, also known as hex, is a numeral system with a base of 16, usually written by means of symbols 0–9 and A–F (or a–f). Its primary purpose is to represent the binary code that computers interpret in a format easier for humans to remember.

  • Page 138: Decimal-To-Hexadecimal Conversion Table

    Hexadecimal Conversion Decimal-to-hexadecimal conversion table TABLE 26 Decimal-to-hexadecimal conversion table Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...
  • Page 139 Hexadecimal Conversion TABLE 26 Decimal-to-hexadecimal conversion table (Continued) Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...
  • Page 140 Hexadecimal Conversion Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...
  • Page 141 Index correcting device login issues link failures account management marginal links lost password recovery options crc_err counter recovering forgotten passwords crc_g_eof counter unable to modify switch settings user forgot password user unable to change switch settings D_Port (diagnostic port) saving port mappings topologies blade errors viewing information...
  • Page 142 enabling firmwareDownload errors port AP blade type 24 is inserted enc_out errors blade application firmware failed blade application firrmware failed equipment status, viewing blade is faulty EX_Ports blade is stuck in the ‘LOADING’ state cannot download the requested firmware cannot download with interop mode turned on cannot upgrade directly to v6.0 command fails F_Port...
  • Page 143 L_Port passwords, recovering forgotten LEDs pathInfo flashing performance problems no light PLOGI steady License issues enabling ports licensing issues point-to-point initialization failure link port intermittent connectivty bypassed marginal disabled no port LED light enabling port LEDs flashing in wrong mode port LEDs steady initialization logical connection...
  • Page 144 resolving zone conflicts test Rolling Reboot Detection a port a switch TI zone problem time is not in-sync troubleshooting certificates SCSI corrupt certificate retry errors invalid certificate timeout errors trunk security, gathering additional information bounces segmentation failed to form segmented fabrics trunk goes offline and online (bounces) setting up automatic trace dump transfers slow-down in FCR performance...
  • Page 145 zone configuration configuration mismatch content mismatch resolving conflicts troubleshooting type mismatch Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...
  • Page 146 Fabric OS Troubleshooting and Diagnostics Guide 53-1002751-01...

This manual is also suitable for:

Fabric os v7.1.0

Table of Contents