1. Manuals
  2. Brands
  3. Brocade Communications Systems Manuals
  4. Software
  5. 8
  6. Troubleshooting and diagnostics manual

Brocade Communications Systems 8 Troubleshooting And Diagnostics Manual

Fabric os troubleshooting and diagnostics guide
Hide thumbs Also See for 8:
Table of Contents

Advertisement

Quick Links

Download this manual See also: Administrator's Manual, Reference Manual
53-1001769-01
®
30 March 2010
Fabric OS
Troubleshooting and Diagnostics Guide
Supporting Fabric OS v6.4.0

Advertisement

Table of Contents
loading

Related Manuals for Brocade Communications Systems 8

Summary of Contents for Brocade Communications Systems 8

  • Page 1 53-1001769-01 ® 30 March 2010 Fabric OS Troubleshooting and Diagnostics Guide Supporting Fabric OS v6.4.0...

  • Page 2: Document History

    Copyright © 2008-2010 Brocade Communications Systems, Inc. All Rights Reserved. Brocade, the B-wing symbol, BigIron, DCX, Fabric OS, FastIron, IronPoint, IronShield, IronView, IronWare, JetCore, NetIron, SecureIron, ServerIron, StorageX, and TurboIron are registered trademarks, and DCFM, Extraordinary Networks, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.

  • Page 3: Table Of Contents

    Basic information ........7 Detailed problem information ......8 Gathering additional information .
  • Page 4 Time issues ..........11 Switch message logs.
  • Page 5 Chapter 4 Configuration Issues In this chapter ......... . . 47 Configupload and download issues .
  • Page 6 Chapter 8 ISL Trunking Issues In this chapter ......... . . 75 Link issues.
  • Page 7 Equipment status ........97 Checking the temperature, fan, and power supply .
  • Page 8 viii Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...

  • Page 9: About This Document

    • Chapter 7, “Virtual Fabrics”, provides procedures to troubleshooting Virtual Fabrics. • Chapter 8, “ISL Trunking Issues,” provides procedures for resolving trunking issues. • Chapter 9, “Zone Issues,” provides preparations and procedures for performing firmware downloads, as well troubleshooting information.

  • Page 10: Supported Hardware And Software

    Supported hardware and software In those instances in which procedures or parts of procedures documented here apply to some switches but not to others, this guide identifies exactly which switches are supported and which are not. Although many different software and hardware configurations are tested and supported by Brocade Communications Systems, Inc.

  • Page 11: Document Conventions

    • Information that was changed: All commands have been updated. • Information that was deleted: All obsolete information. This information was obsoleted because it was no longer supported in the current version of firmware. The iSCSI chapter has been removed and put in the iSCSI Administrator’s Guide. For further information about documentation updates for this release, refer to the release notes.

  • Page 12: Command Examples

    Repeat the previous element, for example “member[;member...]” value Fixed values following arguments are printed in plain font. For example, show WWN Boolean. Elements are exclusive. Example: show mode egress | ingress Command examples This book describes how to perform configuration tasks using the Fabric OS command line interface, but does not describe the commands in detail.

  • Page 13: Additional Information

    Additional information This section lists additional Brocade and industry-specific documentation that you might find helpful. Brocade resources To get up-to-the-minute information, go to http://my.brocade.com and register at no cost for a user ID and password. For practical discussions about SAN design, implementation, and maintenance, you can obtain Building SANs with Brocade Fabric Switches through: http://www.amazon.com White papers, online demos, and data sheets are available through the Brocade Web site at:...

  • Page 14: Document Feedback

    • Description of any troubleshooting steps already performed and the results • Serial console and Telnet session logs • syslog message logs 2. Switch Serial Number The switch serial number and corresponding bar code are provided on the serial number label, as illustrated below.: *FT00X0054E9* FT00X0054E9...

  • Page 15: In This Chapter

    Chapter Introduction to Troubleshooting In this chapter • Troubleshooting overview ......... 1 •...

  • Page 16: Most Common Problem Areas

    Most common problem areas Most common problem areas Table 1 identifies the most common problem areas that arise within SANs and identifies tools to use to resolve them. TABLE 1 Common troubleshooting problems and tools Problem area Investigate Tools • •...
  • Page 17 Chapter 2, “General Issues” Marginal links Chapter 3, “Connectivity Issues” Intermittent connectivity Links Chapter 3, “Connectivity Issues” Trunking Chapter 8, “ISL Trunking Issues” Buffer credits FCIP tunnel Fibre Channel over IP Administrator’s Guide LEDs are flashing Links Chapter 3, “Connectivity Issues”...
  • Page 18 Links Chapter 3, “Connectivity Issues” No connectivity between host and storage Cables Chapter 3, “Connectivity Issues” SCSI timeout errors Chapter 8, “ISL Trunking Issues” SCSI retry errors Chapter 9, “Zone Issues” Zoning Fibre Channel over IP Administrator’s Guide No connectivity between switches Licensing Chapter 2, “General Issues”...

  • Page 19: Gathering Information For Your Switch Support Provider

    1. Connect to the switch and log in using an account assigned to the admin role. 2. Type the supportFtp command and respond to the prompts. Example of supportFTP command switch:admin> supportftp -s Host IP Addr[1080::8:800:200C:417A]: User Name[njoe]: userFoo Password[********]: <hidden> Remote Dir[support]:...

  • Page 20: Capturing Output From A Console

    Gathering information for your switch support provider 2. Type the appropriate supportSave command based on your needs: • If you are saving to an FTP or SCP server, use the following syntax: supportSave When invoked without operands, this command goes into interactive mode. The following operands are optional: -n Does not prompt for confirmation.

  • Page 21: Capturing Command Output

    Building a case for your switch support provider 3. Set the utility to capture output from the screen. Some utilities require this step to be performed prior to opening up a session. Check with your utility vendor for instructions. 4. Type the command or start the process to capture the required data on the console. Capturing command output 1.

  • Page 22: Detailed Problem Information

    Building a case for your switch support provider • Do you have IPsec installed on the switch’s Ethernet interface? Yes or no. Use the ipsecConfig show command to determine the answer. • Do you have Inband Management installed on the switches GigE ports? Yes or no. User the portShow iproute geX command to determine the answer.

  • Page 23: Gathering Additional Information

     Storage software (such as EMC Control Center, Veritas SPC, etc.)  8. If this is a Brocade 48000, Brocade DCX or DCX-4S enterprise-class platform, are the CPs in-sync? Yes or no. Use the haShow command to determine the answer.
  • Page 24 Building a case for your switch support provider Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...

  • Page 25: Chapter 2 General Issues

    Chapter General Issues In this chapter • Licensing issues ..........11 •...

  • Page 26: Switch Message Logs

    Switch message logs Switch message logs Switch message logs (RAS logs) contain information on events that happen on the switch or in the fabric. This is an effective tool in understanding what is going on in your fabric or on your switch. Weekly review of the RAS logs is necessary to prevent minor problems from becoming larger issues, or in catching problems at an early stage.

  • Page 27: Switch Boot Issues

    Switch boot issues Symptom I2C bus errors are appearing in the switch log. Probable cause and recommended action C bus errors generally indicate defective hardware or poorly seated devices or blades; the specific item is listed in the error message. See the Fabric OS Message Reference for information specific to the error that was received.
  • Page 28 Switch boot issues • reboot • haFailover • fastBoot • firmwareDownload The RRD feature is activated and halts rebooting when an unexpected reboot reason is shown continuously in the reboot history within a certain period of time. The period of time is switch dependent.

  • Page 29: Fc-Fc Routing Connectivity

    FC-FC routing connectivity Restrictions The following restrictions are applicable on the RRD feature: • The RRD works only on CFOS based systems and is not available on AP blades. • If FIPS mode is enabled, then the RRD feature works in record-only mode. •...
  • Page 30 FC-FC routing connectivity The fcPing command sends a Fibre Channel ELS ECHO request to a pair of ports. It performs a zone check between the source and destination. In addition, two Fibre Channel Extended Link Service (ELS) requests will be generated. The first ELS request is from the domain controller to the source port identifier.

  • Page 31: Superping

    FC-FC routing connectivity Example of one device that accepts the request and another device that rejects the request: switch:admin> fcping 10:00:00:00:c9:29:0e:c4 21:00:00:20:37:25:ad:05 Source: 10:00:00:00:c9:29:0e:c4 Destination: 21:00:00:20:37:25:ad:05 Zone Check: Not Zoned Pinging 10:00:00:00:c9:29:0e:c4 [0x20800] with 12 bytes of data: received reply from 10:00:00:00:c9:29:0e:c4: 12 bytes time:1162 usec received reply from 10:00:00:00:c9:29:0e:c4: 12 bytes time:1013 usec received reply from 10:00:00:00:c9:29:0e:c4: 12 bytes time:1442 usec received reply from 10:00:00:00:c9:29:0e:c4: 12 bytes time:1052 usec...
  • Page 32 FC-FC routing connectivity • Determines the specific ISLs and internal links with failures. • Exercises all ISL links in the base fabric for a logical fabric configuration. The number of actual paths covered when using the superping tool depends on two other parameters that you can optionally specify.
  • Page 33 Consider the example below in which a few errors are recorded on ISLs 3/205-->2/25, 3/204-->2/27, 2/42-->101/3, and 2/1-->101/8. But a maximum of 100 percentage errors are recorded on internal port 0/284 on domain 2, which is the potential faulty link.

  • Page 34: Route And Statistical Information

    To display basic and extended statistics in interactive mode: switch:admin> pathinfo Max hops: (1..127) [25] Fabric Id: (1..128) [-1] Domain|Wwn|Name: [] 8 Source port: (0..15) [-1] Destination port: (0..255) [-1] Basic stats (yes, y, no, n): [no] y Extended stats (yes, y, no, n): [no] y...

  • Page 35: Performance Issues

    FC-FC routing connectivity Timeout: (1..30) [5] Target port is Embedded Hop In Port Domain ID (Name) Out Port BW Cost --------------------------------------------------------- (web226) 1G 1000 Port ----------------------------------------------- B/s (1s) B/s (64s) Txcrdz (1s) Txcrdz (64s) F/s (1s) F/s (64s) 2743 Words 2752748 2822763 Frames...
  • Page 36 Symptom Host application times out. Probable cause and recommended action The FCR tends to take a long time, more than 5 seconds, to present and setup paths for the proxy devices. Certain hosts are able to do discovery much faster as a result they end up timing out. Use the speed tag feature to always present target proxy to the host and import them faster.

  • Page 37: Connectivity Issues

    Chapter Connectivity Issues In this chapter • Port initialization and FCP auto discovery process ....23 • Link issues ........... . 25 •...
  • Page 38 Port initialization and FCP auto discovery process • VEX_Port—A virtual EX_Port. It connects a Fibre Channel router to an edge fabric. From the point of view of a switch in an edge fabric, a VEX_Port appears as a normal VE_Port. It follows the same Fibre Channel protocol as other VE_Ports.

  • Page 39: Link Issues

    Link issues Link issues Symptom Port LEDs are flashing. Probable cause and recommended action Depending on the rate of the flash and the color of the port LED this could mean several things. To determine what is happening on either your port status LED or power status LED, refer to that switch’s model hardware reference manual.

  • Page 40: Checking The Logical Connection

    Connection problems Checking the logical connection 1. Enter the switchShow command. 2. Review the output from the command and determine if the device successfully logged into the switch. • A device that is logically connected to the switch is registered as an F_, L_, E_, EX_, VE_, VEX_, or N_Port.
  • Page 41 Connection problems Type Pid PortName NodeName TTL(sec) 021a00; 2,3;20:00:00:e0:69:f0:07:c6;10:00:00:e0:69:f0:07:c6; 895 Fabric Port Name: 20:0a:00:60:69:10:8d:fd 051edc; 3;21:00:00:20:37:d9:77:96;20:00:00:20:37:d9:77:96; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee0; 3;21:00:00:20:37:d9:73:0f;20:00:00:20:37:d9:73:0f; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee1; 3;21:00:00:20:37:d9:76:b3;20:00:00:20:37:d9:76:b3; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b...

  • Page 42: Link Failures

    2. Enter the switchShow command to determine if the port has module light. 3. Enter the portCfgSpeed command to change the port speed to 1, 2, 4 or 8 Gbps, depending on what speed can be used by both devices. This should correct the negotiation by setting to one speed.

  • Page 43: Checking For A Loop Initialization Failure

    Link failures Checking for a loop initialization failure 1. Verify the port is an L_Port. a. Enter the switchShow command. b. Check the last field of the output to verify that the switch port indicates an L_Port. If a loop device is connected to the switch, the switch port must be initialized as an L_Port.

  • Page 44: Correcting A Port That Has Come Up In The Wrong Mode

    Marginal links After becoming an active port, the port becomes an F_Port or an E_Port depending on the device on the opposite side. If the opposite device is a host or target device, the port becomes an F_Port. If the opposite device is another switch, the port becomes an E_Port. If there is a problem with the host or target device, enter the portCfgGPort to force the port to try to come up as point-to-point only.

  • Page 45: Troubleshooting A Marginal Link

    Check the cables to and from the switch, patch panel, and other devices. Check the SFP by swapping it with a known good working SFP. If you see this issue on an 8 Gbps blade, use the portCfgFillWord command to reduce EMI. •...
  • Page 46 Marginal links switch:admin> porterrshow frames enc disc link loss loss frjt fbsy err g_eof shrt long eof out c3 fail sync sig ============================================================================ 665k 7.0k 665k 7.4k (output truncated) 3. If you suspect a marginal link, isolate the areas by moving the suspected marginal port cable to a different port on the switch.

  • Page 47: Device Login Issues

    Device login issues Device login issues A correct login is when the port type matches the device type that is plugged in. In the example below, it shows that the device connected to Port 1 is a fabric point-to-point device and it is correctly logged in an F_Port.
  • Page 48 PRESENT U_PORT LED 5 Offline No_Light PRESENT U_PORT LED 6 Offline No_Module PRESENT U_PORT LED 7 Offline No_Module PRESENT U_PORT LED 8 Offline No_Light PRESENT U_PORT LED 9 Offline No_Light PRESENT U_PORT LED 10 Offline No_Module PRESENT U_PORT LED 11 Offline...

  • Page 49: Media-Related Issues

    15 Online In_Sync PRESENT ACTIVE E_PORT G_PORT U_PORT SEGMENTED LOGIN 6. Enter the portLogDumpPort portid command where the port ID is the port number; then, view the device-to-switch communication. switch:admin> portlogdumpport 8 | more time task event port cmd args...

  • Page 50: Testing A Port's External Transmit And Receive Path

    Media-related issues TABLE 6 Component test descriptions Test name Operands Checks portTest [-ports itemlist] [-iteration count] Used to isolate problems to a single [-userdelay time] [-timeout time] replaceable element and isolate [-pattern pattern] [-patsize size] problems to near-end terminal [-seed seed] [-listtype porttype] equipment, far-end terminal equipment, or transmission line.

  • Page 51: Segmented Fabrics

    Verifies that the on chip SRAM located in the 4 and 8 Gbps ASIC is using the Turbo-Ram BIST circuitry. This allows the BIST controller to perform the SRAM write and read operations at a much faster rate.

  • Page 52: Reconciling Fabric Parameters Individually

    6. Connect to the segmented switch after the discrepancy is identified. Disable the switch by entering the switchDisable command. 8. Enter the configure command to edit the appropriate fabric parameters for the segmented switch. 9. Enable the switch by entering the switchEnable command.
  • Page 53 6. Enter the configure command. When the Fabric Parameters prompt displays, type y. 8. When the Domain prompt displays, type in the new number. 9. Press enter on all prompts to accept their default settings. 10. Enter the switchEnable command.

  • Page 54: Reconciling Incompatible Software Features

    4 Gbps can only support up to 4 Gbps of traffic. A normal 4 Gbps F_Port is bi-directional and can support up to 8 Gbps (4 Gbps transmit and 4 Gbps receive) of traffic. If the mirror port bandwidth is exceeded, no credits will be returned to the receiver port and thus those devices involved in mirror connection see a degraded level of performance.

  • Page 55: Port Mirroring Considerations

    • Port Mirroring is supported with Virtual Fabrics with the limitation that you cannot have FCR enabled within the same 8-port group. • If you have NPIV or 10-bit address mode enabled on a Brocade 300, 5300, 5410, 5450, 5460,...

  • Page 56: Supported Platforms

    Port Mirroring is supported only in FC ports. In general, a platform or blade supporting port mirroring supports both the mirror ports and the mirror connections. Exemptions are listed in Table TABLE 8 Port Mirroring platform supportability Brocade Model Fabric OS v6.2.0 Fabric OS v6.1.1_enc...

  • Page 57: Configuring A Port To Be A Mirror Port

    Port mirroring TABLE 9 Maximum number of mirror connections Model Maximum Number of Mirror Connections (chassis-wide) Brocade 5000 Brocade 5100 Brocade 5300 Brocade 5410 Brocade 5450 Brocade 5460 Brocade 5470 Brocade 5480 Brocade 7500 Brocade 7500E Brocade 7600 Brocade 7800 Brocade DCX Brocade DCX-4S Brocade Encryption Switch...

  • Page 58: Adding A Port Mirror Connection

    Port mirroring Adding a port mirror connection 1. Connect to the switch and log in using an account assigned to the admin role. 2. Enter the portMirror add slotnumber/portnumber SourceID DestID command. The configuration database keeps information about the number of port mirror connections configured on a switch, the number of chunks of port mirroring data that are stored, and the chunk number.

  • Page 59: Displaying Port Mirror Connections

    Port mirroring Displaying port mirror connections 1. Connect to the switch and log in using an account assigned to the admin role. 2. Enter the portMirror show command. You should see output similar to the following: switch:admin> portmirror --show Number of mirror connection(s) configured: 4 Mirror_Port State ----------------------------------------...
  • Page 60 Port mirroring Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...

  • Page 61: Chapter 4 Configuration Issues

    Chapter Configuration Issues In this chapter • Configupload and download issues ....... . 47 •...
  • Page 62 Configupload and download issues • The FTP or SCP server’s IP address cannot be contacted. Verify that you can connect to the FTP server. Use your local PC to connect to the FTP server or ping the FTP server. Example of a successful ping C:\>...
  • Page 63 Configupload and download issues • On a Virtual Fabric-enabled switch, you do not have the chassis role permission set on your user account. Implement one change at a time, then issue the command again. By implementing one change at a time, you will be able to determine what works and what does not work.

  • Page 64: Gathering Additional Information

    Configupload and download issues Probable cause and recommended action Verify that the switch was rebooted by checking the system log. If you are doing this on a enterprise-class platform, verify that both CPs rebooted by checking the system log. If any error occurs during the download, such as an error about a particular key, it is important to issue the configDefault command and attempt to repeat the configDownload command.

  • Page 65: Brocade Configuration Form

    Brocade configuration form Brocade configuration form Use this form as a hard copy reference for your configuration information. In the hardware reference manuals for the Brocade 48000, DCX, and DCX-4S modular switches there is a guide for FC port setting tables. The tables can be used to record configuration information for the various blades.
  • Page 66 Brocade configuration form Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...

  • Page 67: Chapter 5 Firmwaredownload Errors

    CORE8 and CR4S-8 core blades provide ICL functionality between two Brocade DCX Backbones. CORE8 blades can be inserted only into slots 5 and 8 on the Brocade DCX. CR4S-8 blades can be inserted only into slots 3 and 6 on the Brocade DCX-4S.

  • Page 68: Firmware Download Issues

    Firmware download issues Symptom The blade is faulty (issue slotShow to confirm). Probable cause and recommended action If the port or application blade is faulty, enter the slotPowerOff and slotPowerOn commands for the port or application blade. If the port or application blade still appears to be faulty, remove it and re-insert it into the chassis.
  • Page 69 Firmware download issues Symptom Firmwaredownload times out. Probable cause and recommended action This can be caused by an excessively slow network. If it takes more than 30 minutes to download firmware on a switch, or on each CP in a director, the firmware download process times out. If a timeout occurs on a switch, the firmware download process synchronizes the two partitions on the switch by starting a firmware commit operation.
  • Page 70 Firmware download issues Password: <hidden> Server IP: 192.126.168.115, Protocol IPv4 Checking system settings for firmwaredownload... Firmware access timeout. The server is inaccessible or firmware path is invalid. Please make sure the server name or IP address, the user/password and the firmware path are valid. Symptom Cannot download the requested firmware.

  • Page 71: Troubleshooting Firmwaredownload

    Troubleshooting firmwareDownload Troubleshooting firmwareDownload A network diagnostic script and preinstallation check is a part of the firmwareDownload procedure. The script and preinstallation check performs troubleshooting and automatically checks for any blocking conditions. If the firmware download fails, see the Fabric OS Message Reference for details about error messages.

  • Page 72: Considerations For Downgrading Firmware

    Considerations for downgrading firmware TABLE 11 USB error handling Scenario under which download fails Error handling Action An access error occurs during Firmwaredownload will timeout None. firmwaredownload because the and commit will be started to removal of the USB device, or USB repair the partitions of the CPUs device hardware failure, etc.
  • Page 73 Considerations for downgrading firmware File Name: /home/userfoo/v6.3.0 Network Protocol (1-auto-select, 2-FTP, 3-SCP) [1]: Password: <hidden> Checking System Settings... Version compatibility check passed. Checking version compatibility... Version compatibility check passed.. The following items need to be addressed before downloading the specified firmware: Downgrade is not allowed because there are more than 2 FX8-24 blades plugged in to the chassis.

  • Page 74: Blade Types

    Considerations for downgrading firmware IPSec on FCIP tunnels requires version v6.3.1 or higher for 7800, and v6.4.0 or higher for FX8-24. Please use "portcfg fciptunnel" to disable IPSec for all FCIP tunnels and try again. Downgrade is not allowed because Persistent PID is enabled. Please use "ag --persistentpidenable 0"...

  • Page 75: Firmware Versions

    Considerations for downgrading firmware Message The FS8-18 (type 43) blade is not supported by the target firmware. Please use slotshow to find out which slot it is in and remove it first. Probable cause and recommended action The firmware download operation was attempting to downgrade a system to Fabric OS v6.1.1_enc or earlier with one or more Brocade FC8-18 AP blades (blade ID 43) in the system.

  • Page 76: Routing

    Considerations for downgrading firmware Message Downgrade is not allowed because VF is enabled. Please run "lscfg --config" and "lscfg --delete" commands to remove the non-default LS first, then run "fosconfig --disable vf" to disable VF before proceeding. Probable cause and recommended action You cannot downgrade because Virtual Fabrics are enabled.

  • Page 77: Chapter 6 Security Issues

    Chapter Security Issues In this chapter • Password issues ..........63 •...

  • Page 78: Password Recovery Options

    Device authentication issues Symptom Unable to log into the boot PROM. Probable cause and recommended action To recover a lost boot PROM password, contact your switch service provider. You must have previously set a recovery string to recover the boot PROM password. This does not work on lost or forgotten passwords in the account database.

  • Page 79: Protocol And Certificate Management Issues

    Protocol and certificate management issues Symptom Switch is unable to form an F_Port. Probable cause and recommended action Regardless of the device authentication policy mode on the switch, the F_Port is disabled if the DH-CHAP protocol fails to authenticate. If the HBA sets the FC-SP bit during FLOGI and the switch sends a FLOGI accept with FC-SP bit set, then the switch expects the HBA to start the AUTH_NEGOTIATE.

  • Page 80: Snmp Issues

    SNMP issues • If you think this is a policy-related issue, FCS switch or other security server-related issue then use supportSave -n to collect data from the Primary FCS switch and all affected switches. • If login-related, then also include the following information: Does login problem appear on a Serial, CP IP, or Switch IP address connection? Is it CP0 or CP1? Is the CP in active or standby?
  • Page 81 FIPS issues Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...
  • Page 82 FIPS issues Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...

  • Page 83: In This Chapter

    Chapter Virtual Fabrics In this chapter • General Virtual Fabric troubleshooting ......69 •...

  • Page 84: Fabric Identification Issues

    Fabric identification issues • VE_Ports cannot exist in a logical switch that has XISL use turned on. Although VE_Ports are allowed in a base switch, Fabric OS v6.2.0 and later do not support the use of VE_Ports to carry traffic for logical fabrics using XISLs. They can be used to carry FCR traffic through EX_ and VEX_Ports.

  • Page 85: Base Switch Issues

    Base switch issues Symptom The switch with domain <domain> with firmware version <fw version> has joined the FID <fid> fabric and may not be compatible with XISL use. Probable cause and recommended action This message indicates the specified switch in the logical fabric using XISLs is running an incompatible firmware version and must be upgraded to Fabric OS v6.2.0.

  • Page 86: Logical Switch Issues

    Logical switch issues Logical switch issues CAUTION When a logical switch is created, all configuration for the logical switch is set to factory defaults. When a logical switch is deleted, all configuration for the logical switch is deleted permanently and is not recoverable. Symptom The indicated slot is empty.

  • Page 87: Switch Configuration Blade Compatibility

    Switch configuration blade compatibility Symptom Unable to remove ports from their current switch. Probable cause and recommended action When moving ports to a switch, they are first removed from the switch in which they reside. This error message is displayed if this step fails. Symptom A non-GE blade is within the slot range.

  • Page 88: Gathering Additional Information

    Switch configuration blade compatibility Gathering additional information For Virtual Fabric-related issues, use the following guidelines to gather additional data for your switch support provider: • Perform the supportSave command. • If not sure about the problem area, perform the supportSave command on all chassis and logical switches in the fabric.

  • Page 89: Chapter 8 Isl Trunking Issues

    Chapter ISL Trunking Issues In this chapter • Link issues ........... . 75 •...

  • Page 90: Buffer Credit Issues

    Local or remote ports are not in the same port group. Move all ISLs to the same port group. The port groups begin at port 0 and are in groups of 4 or 8, depending on the switch model. Until this is done, the ISLs will not trunk. •...

  • Page 91: Chapter 9 Zone Issues

    Chapter Zone Issues In this chapter • Overview of corrective action ........77 •...

  • Page 92: Segmented Fabrics

    Configured Status: Activated / Failover-Enabled Enabled Status: Deactivated / Failover-Enabled TI Zone Name: bluezone: Port List: 9,2; 9,3; 8,3; 8,5; Configured Status: Deactivated / Failover-Disabled Enabled Status: Activated / Failover-Enabled Segmented fabrics This section discusses fabric segmentation. Fabric segmentation occurs when two or more switches are joined together by ISLs and do not communicate to each other.

  • Page 93: Zone Conflicts

    Zone conflicts Symptom Zone conflict appears in logs and fabric is segmented. Probable cause and recommended action This issue is usually caused by incompatible zoning configurations. Verify the following are true: • The effective cfg (zone set) on each end of the segmented ISL is identical. •...

  • Page 94: Correcting A Fabric Merge Problem Quickly

    Zone conflicts TABLE 14 Commands for debugging zoning (Continued) Command Function cfgEnable Use to enable and activate (make effective) the specified configuration. cfgSave Use to save changes to the zone configuration database. cfgTransAbort Use to abort the current zoning transaction without committing it. cfgTransShow Use to display the ID of the current zoning transaction defZone...

  • Page 95: Changing The Default Zone Access

    Zone conflicts Changing the default zone access A switch is not allowed to merge with another switch that has an active effective configuration if the default zone is set to “no access”. Before the switch can join, the default zone setting has to be set to "all access".

  • Page 96: Checking For Fibre Channel Connectivity Problems

    Zone conflicts Checking for Fibre Channel connectivity problems Enter the fcPing command (refer to the Fabric OS Command Reference for more information on this command), which checks the zoning configuration for the two ports specified by: • Generates an ELS (Extended Link Service frame) ECHO request to the source port specified and validates the response.

  • Page 97: Checking For Zoning Problems

    Zone conflicts received reply from 0x020800: 12 bytes time:1006 usec received reply from 0x020800: 12 bytes time:1008 usec received reply from 0x020800: 12 bytes time:1038 usec received reply from 0x020800: 12 bytes time:1010 usec 5 frames sent, 5 frames received, 0 frames rejected, 0 frames timeout Round-trip min/avg/max = 1006/1044/1159 usec Pinging 22:00:00:04:cf:75:63:85 [0x217d9] with 12 bytes of data: Request timed out...

  • Page 98: Gathering Additional Information

    Gathering additional information Gathering additional information Collect the data from a supportSave -n command. Then collect the data from the cfgTransShow command. For the port having the problem, collect the data from the filterPortShow <port> command. Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...

  • Page 99: In This Chapter

    Chapter Working With Diagnostic Features In this chapter • About Fabric OS diagnostics ........85 •...

  • Page 100: Power-On Self Test

    Power-on self test To save a set of files that customer support technicians can use to further diagnose the switch condition, enter the supportSave command. The command prompts for an FTP server, packages the following files, and sends them to the specified server: •...
  • Page 101 Power-on self test eth0: Link status change: Link Up. 100 Mbps Full duplex Auto (autonegotiation complete). INITCP: CPLD Vers: 0x95 Image ID: 0x19 uptime: 2008; sysc_qid: 0 Fabric OS (Paulsa45) Paulsa45 console login: 2005/03/31-20:12:42, [TRCE-5000], 0,, INFO, ?, trace:, trace_buffer.c, line: 1170 2005/03/31-20:12:42, [LOG-5000], 0,, INFO, SW4100_P45, Previous message repeat 1 time(s), trace_ulib.c, line: 540 2005/03/31-20:12:43, [HAM-1004], 219,, INFO, SW4100_P45, Processor rebooted -...

  • Page 102: Disabling Post

    Switch status Disabling POST A reboot is not required for this command to take effect. 1. Connect to the switch and log in with a user account that has admin privileges with the chassis-role permission. 2. Enter the diagDisablePost command. This disables POST1 and POST2.

  • Page 103: Displaying Switch Information

    Switch status Faulty ports monitor HEALTHY Missing SFPs monitor HEALTHY All ports are healthy For more information on how the overall switch status is determined, see the switchStatusPolicySet command in the Fabric OS Command Reference. Displaying switch information 1. Connect to the switch and log in as admin. 2.

  • Page 104: Displaying The Uptime For A Switch

    SpinFab and portTest The switchShow command also displays the following information for ports on the specified switch: Index Index follows Area up to 255. Then it continues to the maximum port of the platform. Index identifies the port number relative to the switch. Index column is only displayed on enterprise-class platforms.

  • Page 105: Debugging Spinfab Errors

    SpinFab and portTest CAUTION During the spinFab testing, the switch remains in normal operation. However, some performance degradation will occur due to the ISL links being saturated with the test frames. This test should be run with caution on a live fabric. Table 15 lists the supported ports for the specified version of Fabric OS when using the spinFab command.
  • Page 106 SpinFab and portTest Pt0/3(33) Ch0/33 Enc_out Error Counter is 187725412 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1, Pt0/3(33) Ch0/33 TruncFrm Error Counter is 32 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1, Pt0/3(33) Ch0/33 FrmTooLong Error Counter is 32 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1, Pt0/3(33) Ch0/33 BadOrdSet Error Counter is 32 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1,...

  • Page 107: Clearing The Error Counters

    SpinFab and portTest The following are debugging procedures: • Check whether the same port is reporting Link Errors as discussed in “Link Errors” on page 91. If yes, follow the same set of debugging procedures as discussed in “Link Errors” on page 91.

  • Page 108: Port Information

    Port information Port information Use the following instructions to view information about ports and to help diagnose if your switch is experiencing port problems. Viewing the status of a port 1. Connect to the switch and log in as admin. 2.

  • Page 109: Displaying The Port Statistics

    Time R_RDY high priority tim_txcrd_z Time TX Credit Zero (2.5Us ticks) time_txcrd_z_vc 0- 3: time_txcrd_z_vc 4- 7: time_txcrd_z_vc 8-11: time_txcrd_z_vc 12-15: er_enc_in Encoding errors inside of frames er_crc Frames with CRC errors er_trunc Frames shorter than minimum er_toolong Frames longer than maximum...

  • Page 110: Displaying A Summary Of Port Errors For A Switch

    Port information Displaying a summary of port errors for a switch 1. Connect to the switch and log in as admin. 2. Enter the portErrShow command. See the Fabric OS Command Reference for additional portErrShow command information. switch:admin> porterrshow frames enc disc link loss loss frjt fbsy err g_eof shrt long c3 fail sync...

  • Page 111: Equipment Status

    Equipment status TABLE 16 Error summary description (Continued) Error type Description too shrt Frames shorter than minimum too long Frames longer than maximum bad eof Frames with bad end-of-frame delimiters enc out Encoding error outside of frames disc c3 Class 3 frames discarded link fail Link failures (LF1 or LF2 states) loss sync...

  • Page 112: Checking The Status Of A Power Supply

    Equipment status Fan 1 is Absent Fan 2 is Ok, speed is 6553 RPM Fan 3 is Ok, speed is 6367 RPM The possible status values are: • OK—Fan is functioning correctly. • Absent—Fan is not present. • Below minimum—Fan is present but rotating too slowly or stopped. •...

  • Page 113: System Message Log

    System message log Information displays for each temperature sensor in the switch. The possible temperature status values are: • OK—Temperature is within acceptable range. • FAIL—Temperature is outside of acceptable range. Damage might occur. Refer to the hardware reference manual for your switch to determine the normal temperature range.

  • Page 114: Displaying The System Message Log One Message At A Time

    Port log Displaying the system message log one message at a time 1. Connect to the switch and log in as admin. 2. Enter the errShow command. Clearing the system message log 1. Connect to the switch and log in as admin. 2.
  • Page 115 Port log <output truncated> Use the commands summarized in Table 17 to view and manage port logs. See the Fabric OS Command Reference for additional information about these commands. TABLE 17 Commands for port log management Command Description portLogClear Clear port logs for all or particular ports. portLogDisable Disable port logs for all or particular ports.

  • Page 116: Syslogd Configuration

    Syslogd configuration Syslogd configuration The system logging daemon (syslogd) is an IP-based service for logging system messages made available by default on Unix and Linux operating systems. It is available as a third-party application for Windows operating systems. Fabric OS can be configured to use a UNIX-style syslogd process to forward system events and error messages to log files on a remote host system.

  • Page 117: Automatic Trace Dump Transfers

    3. Verify that the IP address was entered correctly, using the syslogdIpShow command. The syslogdIpadd command accepts IPv4 and IPv6 addresses. You can specify up to six host IP addresses for storing syslog messages, as shown in this example: switch:admin> syslogdipadd 1080::8:800:200C:417A switch:admin> syslogdipadd 1081::8:800:200C:417A switch:admin> syslogdipadd 1082::8:800:200C:417A switch:admin>...

  • Page 118: Specifying A Remote Server

    Automatic trace dump transfers • Setting up a periodic checking of the remote server so that you are alerted if the server becomes unavailable and you can correct the problem. After the setup is complete, you can run the supportSave -c command to save RASLog, TRACE, supportShow, core file, FFDC data and other diagnostic support information to the server without specifying server details.

  • Page 119: Appendix A Switch Type And Blade Id

    24000 BLOOMII 2 Gb 128-port core fabric switch 3850 BLOOMII 2 Gb 16-port switch with switch limit 3250 BLOOMII 2 Gb 8-port switch with switch limit 4012 GoldenEye 2 Gb 12-port embedded switch 4100 Condor 4 Gb 32-port switch 200E...
  • Page 120 4 Gb 32-FC ports blade FC8-16 8 Gb 16-FC ports blade FR4-18i Condor 16 FC-ports with 2 GbE ports router blade FC4-16IP Condor 8 FC ports and 8 GbE copper RJ-45 ports iSCSI blade Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...
  • Page 121 Application Services blade FC4-48 4 Gb 48-FC ports blade FC10-6 Condor 10 Gb 6-FC ports blade FC8-48 Condor2 8 Gb 48-FC ports blade FC8-32 8 Gb 32-FC ports blade FS8-18 Condor2 8 Gb 16-port encryption blade FCOE10-24 Condor2 24-FC ports on an application blade that provides Converged Enhanced Ethernet to bridge a Fibre Channel and Ethernet SAN.
  • Page 122 Switch Type and Blade ID Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...

  • Page 123: Appendix B Hexadecimal

    Appendix Hexadecimal Hexadecimal, or simply hex, is a numeral system with a base of 16, usually written using unique symbols 0–9 and A–F, or a–f. Its primary purpose is to represent the binary code that computers interpret and represent in a format easier for humans to read. It acts as a form of shorthand, in which one hexadecimal digit stands in place of four binary bits.
  • Page 124 Hexadecimal TABLE 21 Decimal to Hex conversion table Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...
  • Page 125 Hexadecimal TABLE 21 Decimal to Hex conversion table (Continued) Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...
  • Page 126 Hexadecimal Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01...
  • Page 127 Index correcting device login issues link failures account management marginal links lost password recovery options crc_err counter recovering forgotten passwords crc_g_eof counter unable to modify switch settings user forgot password user unable to change switch settings devices login issues missing blade errors diagnostics AP blade type 24 is inserted...
  • Page 128 FL_Port FLOGI frames tx and rx F_Port functional tests fabric issues merge fails parameters parameters, reconcile segments G_Port fabric merge problem gathering fabric segmentation basic switch information configuration mismatch detailed information content mismatch information for technical support default zone access fcPing Fibre Channel connectivity type mismatch...
  • Page 129 LSAN is slow or times-out port mirroring adding port connections considerations deleting port connections port type E_Port M_Port EX_Port marginal links F_Port media-related issues FL_Port functional tests G_Port structural tests L_Port message logs M_Port missing devices U_Port VE_Port VEX_Port portErrShow crc_err counter errors Name Server, (See also NS) crc_g_eof counter errors...
  • Page 130 storage user forgot password devices user is unable to change switch settings management applications using fcPing structural tests superping tool supportSave supportSave timeout value switch VE_Port configuration VEX_Port constantly reboots viewing disabling port and saving diagnostic information panic equipment status reboots during configup/download fan status system status...

This manual is also suitable for:

Poweredge m1000ePoweredge m600Poweredge m605Poweredge m610Poweredge m710Poweredge m710hd ... Show all

Table of Contents