Page 1 Sun StorEdge 3900 and 6900 ™ Series Troubleshooting Guide Sun Microsystems, Inc. 4150 Network Circle Santa Clara, CA 95054 U.S.A. 650-960-1300 Part No. 816-4290-11 March 2002, Revision A Send comments about this document to: docfeedback@sun.com...
Page 2 Sun, Sun Microsystems, le logo Sun, AnswerBook2, Sun StorEdge, StorTools, docs.sun.com, Sun Enterprise, Sun Fire, SunOS, Netra, et Solaris sont des marques de fabrique ou des marques déposées, ou marques de service, de Sun Microsystems, Inc. aux Etats-Unis et dans d’autres pays.
Page 3: Table Of Contents
Contents Introduction 1 Predictive Failure Analysis Capabilities 2 General Troubleshooting Procedures 3 Troubleshooting Overview Tasks 3 Multipathing Options in the Sun StorEdge 6900 Series 7 Alternatives to Sun StorEdge Traffic Manager 8 To Quiesce the I/O 8 To Unconfigure the c2 Path 8 To Suspend the I/O 10 To Return the Path to Production 10 To View the VxDisk Properties 11...
Page 4 To Isolate the A3/B3 FC Link 39 A4/B4 Fibre Channel (FC) Link 40 To Verify the Data Host 42 Sun StorEdge 3900 Series 42 Sun StorEdge 6900 Series 42 FRU tests available for the A4/B4 FC Link Segment 44 To Isolate the A4/B4 FC Link 44...
Page 5 To Verify Configuration Settings 47 To Clear the Lock File 50 Troubleshooting Host Devices 53 Host Event Grid 53 Using the Host Event Grid 53 Replacing the Master, Alternate Master, and Slave Monitoring Host 57 To Replace the Master Host 57 To Replace the Alternate Master or Slave Monitoring Host 58 Conclusion 59 Troubleshooting Sun StorEdge FC Switch-8 and Switch-16 Devices 61...
Page 6 Virtualization Engine LEDs 72 Power LED Codes 73 Interpreting LED Service and Diagnostic Codes 73 Back Panel Features 74 Ethernet Port LEDs 74 Fibre Channel Link Error Status Report 75 To Check Fibre Channel Link Error Status Manually 76 Translating Host Device Names 78 To Display the VLUN Serial Number 79 Devices That Are Not Sun StorEdge Traffic Manager-Enabled 79 Sun StorEdge Traffic Manager-Enabled Devices 80...
Page 7 Troubleshooting the T1/T2 Data Path 102 Notes 102 T1/T2 Notification Events 103 Sun StorEdge T3+ Array Storage Service Processor Verification 106 T1/T2 FRU Tests Available 107 Notes 108 T1/T2 Isolation Procedures 108 Sun StorEdge T3+ Array Event Grid 109 Using the Sun StorEdge T3+ Array Event Grid 109 Replacing the Master Midplane 122 To Replace the Master Midplane 122 Conclusion 122...
Page 8 viii Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 9: List Of Figures
List of Figures Sun StorEdge 3900 Series Fibre Channel Link Diagram 16 FIGURE 2-1 Sun StorEdge 6900 Series Fibre Channel Link Diagram 17 FIGURE 2-2 Data Host Notification of Intermittent Problems 23 FIGURE 3-1 Data Host Notification of Severe Link Error 24...
Page 10 Path Failure —I/O Routed through Both HBAs 94 FIGURE 7-6 Virtualization Engine Event Grid 95 FIGURE 7-7 Storage Service Processor Event 103 FIGURE 8-1 Virtualization Engine Alert 105 FIGURE 8-2 Manage Configuration Files Menu 106 FIGURE 8-3 Example Link Test Text Output from the Storage Automated Diagnostic Environment 107 FIGURE 8-4 Sun StorEdge T3+ array Event Grid 109 FIGURE 8-5...
Page 11 Preface The Sun StorEdge 3900 and 6900 Series Troubleshooting Guide provides guidelines for isolating problems in supported configurations of the Sun StorEdge 3900 and 6900 series. For detailed configuration information, refer to the Sun StorEdge 3900 and 6900 Series Reference Manual. The scope of this troubleshooting guide is limited to information pertaining to the components of the Sun StorEdge 3900 and 6900 series, including the Storage Service Processor and the virtualization engines in the Sun StorEdge 6900 series.
Page 12 Chapter 7 provides detailed information for troubleshooting the virtualization engines. Chapter 8 describes how to troubleshoot the Sun StorEdge T3+ array devices. Also included in this chapter is information about the Explorer Data Collection Utility. Chapter 9 discusses ethernet hub troubleshooting. Information associated with the 3COM Ethernet hubs is limited in this guide, however, as this is third-party information.
Page 13: Typographic Conventions
Typographic Conventions Typeface Meaning The names of commands, files, AaBbCc123 and directories; on-screen computer output What you type, when AaBbCc123 contrasted with on-screen computer output AaBbCc123 Book titles, new words or terms, words to be emphasized Command-line variable; replace with a real name or value Shell Prompts Shell C shell...
Page 14: Related Documentation
Related Documentation Product Title Late-breaking News • Sun StorEdge 3900 and 6900 Series Release Notes Sun StorEdge 3900 and 6900 • Sun StorEdge 3900 and 6900 Series Site Preparation Guide series hardware information • Sun StorEdge 3900 and 6900 Series Regulatory and Safety Compliance Manual •...
Page 15: Accessing Sun Documentation Online
Accessing Sun Documentation Online A broad selection of Sun system documentation is located at: http://www.sun.com/products-n-solutions/hardware/docs A complete set of Solaris documentation and many other titles are located at: http://docs.sun.com Sun Welcomes Your Comments Sun is interested in improving its documentation and welcomes your comments and suggestions.
Page 16 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 17: Introduction
C H A P T E R Introduction The Sun StorEdge 3900 and 6900 series storage subsystems are complete preconfigured storage solutions. The configurations for each of the storage subsystems are shown in TABLE 1-1 Series System Sun StorEdge Sun StorEdge 3900 series 3910 system Sun StorEdge...
Page 18: Predictive Failure Analysis Capabilities
Predictive Failure Analysis Capabilities The Storage Automated Diagnostic Environment software provides the health and monitoring functions for the Sun StorEdge 3900 and 6900 series systems. This software provides the following predictive failure analysis (PFA) capabilities. FC links—Fibre Channel links are monitored at all end points using the link FC- ELS link counters.
Page 19: General Troubleshooting Procedures
C H A P T E R General Troubleshooting Procedures This chapter contains the following sections: “Troubleshooting Overview Tasks” on page 3 “Multipathing Options in the Sun StorEdge 6900 Series” on page 7 “Fibre Channel Links” on page 15 “Storage Automated Diagnostic Environment Event Grid” on page 21 Troubleshooting Overview Tasks This section lists the high-level steps to isolate and troubleshoot problems in the Sun StorEdge 3900 and 6900 series.
Page 20 1. Discover the error by checking one or more of the following messages or files: Storage Automated Diagnostic Environment alerts or email messages /var/adm/messages Sun StorEdge T3+ array syslog file Storage Service Processor messages /var/adm/messages.t3 messages /var/adm/log/SEcfglog file 2. Determine the extent of the problem by using one or more of the following methods: Storage Automated Diagnostic Environment Topology view Storage Automated Diagnostic Environment Revision Checking (manual patch or...
Page 21 4. Check the status of the Sun StorEdge FC network switch-8 and switch-16 switches using the following tools: Storage Automated Diagnostic Environment device monitoring reports Run the SEcfg script, which displays and shows the Sun StorEdge T3+ array configuration LED Status (online/offline, POST error codes found in the Sun StorEdge network FC switch-8 and switch-16 switch Installation and Configuration Guide) Explorer Data Collection Utility output (located on the Storage Service Processor) SANsurfer GUI...
Page 22 8. Verify the fix using the following tools: Storage Automated Diagnostic Environment GUI Topology View and Diagnostic Tests /var/adm/messages on the data host 9. Return the path to service by using one of the following methods: Multipathing software Restarting the application Sun StorEdge 3900 and 6900 Series Troubleshooting Guide •...
Page 23: Multipathing Options In The Sun Storedge 6900 Series
Multipathing Options in the Sun StorEdge 6900 Series Using the virtualization engines presents several challenges in how multipathing is handled in the Sun StorEdge 6900 series. Unlike Sun StorEdge T3+ array and Sun StorEdge network FC switch-8 and switch- 16 switch installations, which present primary and secondary pathing options, the virtualization engines present only primary pathing options to the data host.
Page 24: Alternatives To Sun Storedge Traffic Manager
Note that in the Class and State fields, the virtualization engines are presented as two primary/ONLINE devices. The current Sun StorEdge Traffic Manager design does not enable you to manually halt the I/O (that is, you cannot perform a failover to the secondary path) when only primary devices are present.
Page 25 2. Using Storage Automated Diagnostic Environment Topology GUI, determine which virtualization engine is in the path you need to disable. 3. Use the world wide name (WWN) of the virtualization engine that is in the unconfigure command, as follows: # cfgadm -c unconfigure c2::2b000060220041f4 # cfgadm -al Ap_Id c0::dsk/c0t0d0...
Page 26: To Suspend The I/O
To Suspend the I/O Use one of the following methods to suspend the I/O while the failover occurs: 1. Stop all customer applications that are accessing the Sun StorEdge T3+ array. 2. Manually pull the link from the Sun StorEdge T3+ array to the switch and wait for a Sun StorEdge T3+ array LUN failover.
Page 27: To View The Vxdisk Properties
To View the VxDisk Properties 1. Type the following: # vxdisk list Disk_1 Device: Disk_1 devicetag: Disk_1 type: sliced hostid: diag.xxxxx.xxx.COM disk: name=t3dg02 id=1010283311.1163.diag.xxxxx.xxx.com group: name=t3dg id=1010283312.1166.diag.xxxxx.xxx.com flags: online ready private autoconfig nohotuse autoimport imported pubpaths: block=/dev/vx/dmp/Disk_1s4 char=/dev/vx/rdmp/Disk_1s4 privpaths: block=/dev/vx/dmp/Disk_1s3 char=/dev/vx/rdmp/Disk_1s3 version: iosize: min=512 (bytes) max=2048 (blocks)
Page 28 2. Use the luxadm(1M) command to display further information about the underlying LUN. # luxadm display /dev/rdsk/c20t2B000060220041F4d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/c20t2B000060220041F4d0s2 Status(Port A): Vendor: Product ID: WWN(Node): WWN(Port A): Revision: Serial Num: Unformatted capacity: 102400.000 MBytes Write Cache: Read Cache: Minimum prefetch: Maximum prefetch: Device Type:...
Page 29: To Quiesce The I/O On The A3/B3 Link
To Quiesce the I/O on the A3/B3 Link 1. Determine the path you want to disable. 2. Disable the path by typing the following: # vxdmpadm disable ctlr=<c#> 3. Verify that the path is disabled: # vxdmpadm listctlr all Steps 1 and 2 halt I/O only up to the A3/B3 link. I/O will continue to move over the T1 &...
Page 30: To Return The Path To Production
To Return the Path to Production 1. Type: # vxdmpadm enable ctlr=<c#> 2. Verify that the path has been re-enabled by typing: # vxdmpadm listctlr all Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 31: Fibre Channel Links
The following diagrams provide troubleshooting information for the basic components and Fibre Channel links specific to the Sun StorEdge 3900 series, shown , and the Sun StorEdge 6900 series, shown in FIGURE 2-1...
Page 32: Fibre Channel Link Diagrams
A1 to B1—HBA to Sun StorEdge FC network switch-8 and switch-16 switch link A4 to B4—Sun StorEdge FC network switch-8 and switch-16 switch to Sun StorEdge T3+ array link Sw1a Sun StorEdge 3900 Series Fibre Channel Link Diagram FIGURE 2-1 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 HOST...
Page 33: Figure 2-2 Sun Storedge 6900 Series Fibre Channel Link Diagram
shows the basic components and the Fibre Channel links for a Sun FIGURE 2-2 StorEdge 6900 series system: A1 to B1—HBA to Sun StorEdge network FC switch-8 and switch-16 switch link A2 to B2—Sun StorEdge network FC switch-8 and switch-16 switch to virtualization engine link on the host side A3 to B3—Sun StorEdge network FC switch-8 and switch-16 switch to the virtualization engine link on the device side...
Page 34: Host Side Troubleshooting
Storage Automated Diagnostic Environment, running on the Storage Service Processor, detects. You can find these messages by monitoring the following Sun StorEdge 3900 series and the Sun StorEdge 6900 series components: Sun StorEdge network FC switch-8 and switch-16 switches...
Page 35: Command Line Test Examples
Command Line Test Examples To run a single Sun StorEdge diagnostic test from the command line rather than through the Storage Automated Diagnostic Environment interface, you must log into the appropriate Host or Slave for testing the components. The following two tests, the qlctest(1M) and the switchtest(1M) are provided as examples.
Page 36: Switchtest(1M)
switchtest(1M) switchtest(1M) is used to diagnose the Sun StorEdge network FC switch-8 and switch-16 switch devices. The switchtest process also provides command line access to switch diagnostics. switchtest supports testing on local and remote switches. switchtest runs the port diagnostic on connected switch ports. While switchtest is running, the port statistics are monitored for errors, and the chassis status is checked.
Page 37: Storage Automated Diagnostic Environment Event Grid
Storage Automated Diagnostic Environment Event Grid The Storage Automated Diagnostic Environment generates component-specific event grids that describe the severity of an Event, whether action is required, a description of the event, and recommended action. Refer to Chapters 5 through 9 of this troubleshooting guide for component-specific event grids.
Page 38 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 39: Troubleshooting The Fibre Channel Links
A1/B1 Fibre Channel (FC) Link If a problem occurs with the A1/B1 FC link: In a Sun StorEdge 3900 series system, the Sun StorEdge T3+ array will fail over. In a Sun StorEdge 6900 series system, no Sun StorEdge T3+ array will fail over, but a severe problem can cause a path to go offline.
Page 40: Figure 3-2 Data Host Notification Of Severe Link Error
Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category : Message EventType: LogEvent.driver.MPXIO_offline EventTime: 01/08/2002 14:48:02 Found 2 ’driver.MPXIO_offline’ warning(s) in logfile: /var/adm/messages on diag.xxxxx.xxx.com (id=80fee746): Jan 8 14:47:07 779286 kern.info] /scsi_vhci/ssd@g29000060220041f96257354230303053 (ssd19) multipath status: degraded, path /pci@6,4000/SUNW,qlc@3/fp@0,0 (fp1) to target address: 2b000060220041f9,1 is offline Jan 8 14:47:07 779286 kern.info] /scsi_vhci/ssd@g29000060220041f96257354230303052...
Page 41: To Verify The Data Host
To Verify the Data Host An error in the A1/B1 FC link can cause a path to go offline in the multipathing software. luxadm(1M) Display CODE EXAMPLE 3-1 # luxadm display /dev/rdsk/c6t29000060220041F96257354230303052d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/ c6t29000060220041F96257354230303052d0s2 Status(Port A): O.K.
Page 42: Fru Tests Available For A1/B1 Fc Link Segment
An error in the A1/B1 FC link can also cause a device to enter the “unusable” state in cfgadm. In this case, the output for luxadm -e port will show that a device that was “connected” changed to an “unconnected” state. CODE EXAMPLE 3-2 # cfgadm -al Ap_Id...
Page 43 switchtest(1M) called with options CODE EXAMPLE 3-3 # ./switchtest -v -o "dev=2:192.168.0.30:0" "switchtest: called with options: dev=2:192.168.0.30:0" "switchtest: Started." "Testing port: 2" "Using ip_addr: 192.168.0.30, fcaddr: 0x0 to access this port." "Chassis Status for Device: Switch Power: OK Temp: OK 23.0c Fan 1: OK Fan 2: OK "...
Page 44: To Isolate The A1/B1 Fc Link
To Isolate the A1/B1 FC Link 1. Quiesce the I/O on the A1/B1 FC link path. 2. Run switchtest or qlctest to test the entire link. 3. Break the connection by uncabling the link. 4. Insert a loopback connector into the switch port. 5.
Page 45: A2/B2 Fibre Channel (Fc) Link
A2/B2 Fibre Channel (FC) Link If a problem occurs with the A2/B2 FC link: In a Sun StorEdge 3900 series system, the Sun StorEdge T3+ array will fail over. In a Sun StorEdge 6900 series system, no Sun StorEdge T3+ array will fail over, but a severe problem can cause a path to go offline.
Page 46: Figure 3-5 A2/B2 Fc Link Storage Service Processor Side Event
Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category : Switch EventType: StateChangeEvent.X.port.1 EventTime: 01/08/2002 17:38:32 ’port.1’ in SWITCH diag-sw1b (ip=192.168.0.31) is now Unknown (status- state changed from ’Online’ to ’Admin’): ---------------------------------------------------------------- Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal...
Page 47: To Verify The Host Side
To Verify the Host Side An error in the A2/B2 FC link can result in a device being listed as in an “unusable” state in cfgadm, but no HBAs are listed as in the “unconnected” state in luxadm output. The multipathing software will note an OFFLINE path. Chapter 3 Troubleshooting the Fibre Channel Links For Internal Use Only...
Page 48 CODE EXAMPLE 3-4 # cfgadm -al Ap_Id <snip> # luxadm -e port Found path to 2 HBA ports /devices/pci@6,4000/SUNW,qlc@2/fp@0,0:devctl /devices/pci@6,4000/SUNW,qlc@3/fp@0,0:devctl # luxadm display /dev/rdsk/c6t29000060220041F96257354230303052d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/ c6t29000060220041F96257354230303052d0s2 Status(Port A): Status(Port B): Vendor: Product ID: WWN(Node): WWN(Port A): WWN(Port B): Revision: Serial Num:...
Page 49: To Verify The A2/B2 Fc Link
Note – You can find procedures for restoring virtualization engine settings in the Sun StorEdge 3900 and 6900 Series Reference Manual. To Verify the A2/B2 FC Link You can check the A2/B2 FC link using the Storage Automated Diagnostic Environment, Diagnose—Test from Topology functionality. The Storage Automated Diagnostic Environment’s implementation of diagnostic tests verifies the operation of user-selected components.
Page 50 5. If the switch or the GBIC show no errors, replace the remaining components in the following order: a. Replace the virtualization engine-side GBIC, recable the link, and monitor the link for errors. b. Replace the cable, recable the link, and monitor the link for errors. c.
Page 51: A3/B3 Fibre Channel (Fc) Link
A3/B3 Fibre Channel (FC) Link If a problem occurs with the A3/B3 FC link: In a Sun StorEdge 3900 series system, the Sun StorEdge T3+ array will fail over. In a Sun StorEdge 6900 series system, no Sun StorEdge T3+ array will fail over, but a severe problem can cause a path to go offline.
Page 52: Figure 3-7 A3/B3 Fc Link Storage Service Processor-Side Event
Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category : Switch EventType: StateChangeEvent.M.port.1 EventTime: 01/08/2002 18:28:38 ’port.1’ in SWITCH diag-sw1a (ip=192.168.0.30) is now Not-Available (status-state changed from ’Online’ to ’Offline’): Info: A port on the switch has logged out of the fabric and gone offline Action: 1.
Page 53: To Verify The Host Side
To Verify the Host Side An error in the A3/B3 FC link results in a device being listed as in an “unusable” state in cfgadm, but no HBAs are listed as in the “unconnected” state in luxadm output. The multipathing software will note an “offline” path. Devices in the “connected”...
Page 54: To Verify The Storage Service Processor
CODE EXAMPLE 3-6 8 18:26:38 diag.xxxxx.xxx.com vxdmp: [ID 619769 kern.notice] NOTICE: vxdmp: Path failure on 118/0x1f8 8 18:26:38 diag.xxxxx.xxx.com vxdmp: [ID 997040 kern.notice] NOTICE: vxvm:vxdmp: disabled path 118/0x1f8 belonging to the dmpnode 231/0xd0 To Verify the Storage Service Processor You can check the A3/B3 FC link using the Storage Automated Diagnostic Environment, Diagnose—Test from Topology functionality.
Page 55: To Isolate The A3/B3 Fc Link
To Isolate the A3/B3 FC Link 1. Quiesce the I/O on the A3/B3 FC link path. 2. Break the connection by uncabling the link. 3. Insert the loopback connector into the switch port. 4. Run switchtest: a. If the test fails, replace the GBIC and rerun switchtest. b.
Page 56: A4/B4 Fibre Channel (Fc) Link
A4/B4 Fibre Channel (FC) Link If a problem occurs with the A4/B4 FC link: In a Sun StorEdge 3900 series system, the Sun StorEdge T3+ array will fail over. In a Sun StorEdge 6900 series system, no Sun StorEdge T3+ array will fail over, but a severe problem can cause a path to go offline.
Page 57: Figure 3-10 Storage Service Processor Notification
Site : FSDE LAB Broomfield CO Source : diag Severity : Warning Category : Switch DeviceId : switch:100000c0dd0061bb EventType: LogEvent.MessageLog EventTime: 01/29/2002 14:25:05 Change in Port Statistics on switch diag-sw1b (ip=192.168.0.31): Port-1: Received 16289 ’InvalidTxWds’ in 0 mins (value=365972 ) ---------------------------------------------------------------------- Site : FSDE LAB Broomfield CO...
Page 58: To Verify The Data Host
To Verify the Data Host A problem in the A4/B4 FC Link appears differently on the data host, depending on if the array is a Sun StorEdge 3900 series or a Sun StorEdge 6900 seriesdevice. Sun StorEdge 3900 Series In a Sun StorEdge 3900 series device, the data host multipathing software is...
Page 59 To verify the failover luxadm display can be used, the failed path will be marked OFFLINE, as shown in CODE EXAMPLE 3-7 CODE EXAMPLE 3-7 Failed Path marked OFFLINE # luxadm display /dev/rdsk/c26t60020F200000644> DEVICE PROPERTIES for disk: /dev/rdsk/ c26t60020F20000064433C3352A60003E82Fd0s2 Status(Port A): O.K.
Page 60: Fru Tests Available For The A4/B4 Fc Link Segment
CODE EXAMPLE 3-8 # cfgadm -al Ap_Id ac0:bank0 ac0:bank1 c1::dsk/c1t6d0 c21::50020f2300006355 FRU tests available for the A4/B4 FC Link Segment The switchtest can only be run from the Storage Service Processor The linktest will be able to isolate the switch and the GBIC on the switch. It will not be able to isolate the cable or the Sun StorEdge T3+ array controller.
Page 61 5. Rerun switchtest. a. If switchtest fails, replace the GBIC and rerun switchtest. b. If the test fails again, replace the switch. 6. If switchtest passes, assume that the suspect components are the cable and the Sun StorEdge T3+ array controller. a.
Page 62 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 63: Configuration Settings
C H A P T E R Configuration Settings This chapter contains the following sections: “Verifying Configuration Settings” on page 47 “To Clear the Lock File” on page 50 For a complete listing of SUNWsecfg Error Messages and recommended action, refer to Appendix B.
Page 64 Note – For cluster configurations and systems that are attached to Windows NT, the default configurations may not match the current installed configuration. Be aware of this when running the verification scripts. Certain items may be flagged as FAIL in these special circumstances. CODE EXAMPLE 4-1 # /opt/SUNWsecfg/checkdefaultconfig Checking all accessible components...
Page 65 10. If anything is marked FAIL, check the /var/adm/log/SEcfglog file for the details of the failure. Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : ---------- -SAVED CONFIGURATION--------------. Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : blocksize : 16k. Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO :...
Page 66: To Clear The Lock File
11. Fix the FAIL condition, and then verify the settings again. # /opt/SUNWsecfg/bin/checkt3config -n t3b0 Checking : t3b0 If you interrupt any of the SUNWsecfg scripts (by typing a Control-C default font, for example), a lock file might remain in the /opt/SUNWsecfg/etc directory, causing subsequent commands to fail.
Page 67 savevemap output CODE EXAMPLE 4-2 Tue Jan 29 16:12:34 MST 2002 savevemap: v1 ENTER. Tue Jan 29 16:12:34 MST 2002 checkslicd: v1 ENTER. Tue Jan 29 16:12:42 MST 2002 checkslicd: v1 EXIT. Tue Jan 29 16:14:01 MST 2002 savevemap: v1 EXIT. When savevemap: <ve-pair>...
Page 68 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide — March 2002...
Page 69: Troubleshooting Host Devices
C H A P T E R Troubleshooting Host Devices This chapter describes how to troubleshoot components associated with a Sun StorEdge 3900 or 6900 series Host. This chapter contains the following sections: “Using the Host Event Grid” on page 53 “To Replace the Master Host”...
Page 70: Figure 5-1 Host Event Grid
Host Event Grid FIGURE 5-1 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 71 lists all the host events in the Storage Automated Diagnostic Environment. TABLE 5-1 Storage Automated Diagnostic Environment Event Grid for the Host TABLE 5-1 Category Component EventType host Alarm+ host Alarm- host lun.t300 Alarm- For Internal Use Only Action Description Yellow [ Info ] status of hba /devices/...
Page 72 Storage Automated Diagnostic Environment Event Grid for the Host (Continued) TABLE 5-1 host lun.VE Alarm- host ifptest Diagnostic Test- host qlctest Diagnostic Test- host socaltest Diagnostic Test- host enclosure PatchInfo host enclosure backup Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 [ Info ] The state of lun.VE.c14t50020 F2300003EE5d0s2.
Page 73: Replacing The Master, Alternate Master, And Slave Monitoring Host
Replacing the Master, Alternate Master, and Slave Monitoring Host The following procedures are a high-level overview of the procedures that are detailed in the Storage Automated Diagnostic Environment User’s Guide. Follow these procedures when replacing a master, alternate master, or slave monitoring host. Note –...
Page 74: To Replace The Alternate Master Or Slave Monitoring Host
5. Choose Utilities -> System -> Recover Config. Refer to Chapter 7 of the Storage Automated Diagnostic Environment User’s Guide for detailed instructions. a. In the Recover Config window, enter the IP address of any alternate master or slave monitoring host (all hosts keep a copy of the configuration). b.
Page 75: Conclusion
7. Choose Maintenance -> General Maintenance -> Maintain Hosts. Refer to Chapter 3, “Maintenance,” of the Storage Automated Diagnostic User’s Guide for detailed instructions. 8. In the Maintain Hosts window, select the new host. 9. Configure the options as needed. 10.
Page 76 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 77: Troubleshooting Sun Storedge Fc Switch-8 And Switch-16 Devices
The switches are paired to provide redundancy. Two switches are used in each Sun StorEdge 3900 series, and four switches are used in each Sun StorEdge 6900 series. Each Sun StorEdge network FC switch-8 and switch-16 switch is connected by way of an Ethernet to the service network for management and service from the Storage Service Processor.
Page 78: To Diagnose And Troubleshoot Switch Hardware
These switches can be monitored through the SANSurfer GUI, which is available on the Storage Service Processor. You configure and modify the switches using the Configuration Utilities. Do not configure or modify the switches using any method other than the SUNWsecfg tools. To Diagnose and Troubleshoot Switch Hardware 1.
Page 79: Figure 6-1 Switch Event Grid
Switch Event Grid FIGURE 6-1 Chapter 6 Troubleshooting Sun StorEdge FC Switch-8 and Switch-16 Devices For Internal Use Only...
Page 80 lists the switch events. TABLE 6-1 Storage Automated Diagnostic Environment Event Grid for Switches TABLE 6-1 Component EventType switch port statistics switch chassis.fan Alarm switch chassis.power Alarm switch chassis.temp Alarm switch chassis.zone Alarm Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Action Description Yellow...
Page 81 Storage Automated Diagnostic Environment Event Grid for Switches (Continued) TABLE 6-1 Component EventType switch enclosure Audit switch Comm_ Established switch Comm_Lost switch switchtest Diagnostic Test- Chapter 6 For Internal Use Only Action Description Auditing a new switch called ras d2-swb1 (ip=xxx.0.0.41) 10002000007a609 Communication...
Page 82 Storage Automated Diagnostic Environment Event Grid for Switches (Continued) TABLE 6-1 Component EventType switch enclosure Discovery switch enclosure LocationChan Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Action Description [ Info ] Discovered a new switch called ras d2-swb1 (ip=xxx.0.0.41) 10002000007a609...
Page 83 Storage Automated Diagnostic Environment Event Grid for Switches (Continued) TABLE 6-1 Component EventType switch port StateChange+ switch port StateChange- switch enclosure Statistics Chapter 6 For Internal Use Only Action Description [ Info/Action ] port.1 in SWITCH diag185 (ip= xxx.20.67.185) is now Available (status- state changed from OFFLINE to ONLINE)
Page 84: Replacing The Master Midplane
Replacing the Master Midplane Follow this procedure when replacing the master midplane in a Sun StorEdge network FC switch-8 or switch-16 switch or a Brocade Silkworm switch. This procedure is detailed in the Storage Automated Diagnostic Environment User’s Guide. To Replace the Master Midplane 1.
Page 85: Troubleshooting Virtualization Engine Devices
C H A P T E R Troubleshooting Virtualization Engine Devices This chapter describes how to troubleshoot the virtualization engine component of a Sun StorEdge 6900 series system. This chapter contains the following sections: “Virtualization Engine Description” on page 69 “Translating Host Device Names”...
Page 86: Virtualization Engine Diagnostics
Virtualization Engine Diagnostics The virtualization engine monitors the following components: Virtualization engine router Sun StorEdge T3+ array Cabling among the router and storage Service Request Numbers The service request numbers are used to inform the user of storage subsystem activities. Service and Diagnostic Codes The virtualization engine’s service and diagnostic codes inform the user of subsystem activities.
Page 87: To Display Log Files And Retrieve Srns
To Display Log Files and Retrieve SRNs Use the /opt/svengine/sduc/sreadlog command to display log files and retrieve the Service Request Numbers (SRN) for errors that need action. Data is returned in the following format: TimeStamp:nnn:Txxxxx.uuuuuuuu SRN=mmmmm TimeStamp:nnn:Txxxxx.uuuuuuuu SRN=mmmmm TimeStamp:nnn:Txxxxx.uuuuuuuu SRN=mmmmm Item Description TimeStamp...
Page 88: To Clear The Log
Item TimeStamp uuuuuuuu SRN=mmmmm To Clear the Log Use the /opt/svengine/sduc/sclrlog command. Virtualization Engine LEDs describes the LEDs on the back of the virtualization engine.. TABLE 7-1 Virtualization Engine LEDs TABLE 7-1 Power Status Fault 1 The Status LED will blink a service code when the Fault LED is Solid on. Sun StorEdge 3900 and 6900 Series Troubleshooting Guide —...
Page 89: Power Led Codes
Power LED Codes The virtualization engine LEDs are shown in Virtualization Engine Front Panel LEDs FIGURE 7-1 Interpreting LED Service and Diagnostic Codes The Status LED communicates the status of the virtualization engine in decimal numbers. Each decimal number is represented by number of blinks, followed by a medium duration (two seconds) of LED off.
Page 90: Back Panel Features
Back Panel Features The back panel of the virtualization engine contains the Sun StorEdge network FC switch-8 or switch-16 switches and a socket for the AC power input, and various data ports and LEDs. Ethernet Port LEDs The Ethernet port LEDs indicate the speed, activity, and validity of the link, shown TABLE 7-3 Speed, Activity, and Validity of the Link TABLE 7-3...
Page 91: Fibre Channel Link Error Status Report
Fibre Channel Link Error Status Report The virtualization engine’s host-side and device-side interfaces provide statistical data for the counts listed in TABLE 7-4 Virtualization Engine Statistical Data TABLE 7-4 Count Type Description Link Failure Count The number of times the virtualization engine’s frame manager detects a non-operational state or other failure of N_Port initialization protocol.
Page 92: To Check Fibre Channel Link Error Status Manually
To Check Fibre Channel Link Error Status Manually The Storage Automated Diagnostic Environment, which runs on the Storage Service Processor, monitors the Fibre Channel link status of the virtualization engine. The virtualization engine must be power-cycled to reset the counters. Therefore, you should manually check the accumulation of errors between a fixed period of time.
Page 93 Fibre Channel Link Error Status Example CODE EXAMPLE 7-1 # /opt/svengine/sduc/svstat -d v1 I00001 Host Side FC Vital Statistics: Link Failure Count Loss of Sync Count Loss of Signal Count Protocol Error Count Invalid Word Count Invalid CRC Count I00001 Device Side FC Vital Statistics: Link Failure Count Loss of Sync Count Loss of Signal Count...
Page 94: Translating Host Device Names
Translating Host Device Names You can translate host device names to VLUN, disk pool, and physical Sun StorEdge T3+ array LUNs. The luxadm output for a host device, shown in the unique VLUN serial number that is needed to identify this LUN. CODE EXAMPLE 7-2 # luxadm display /dev/rdsk/c4t2B00006022004186d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/c4t2B00006022004186d0s2...
Page 95: To Display The Vlun Serial Number
To Display the VLUN Serial Number Devices That Are Not Sun StorEdge Traffic Manager-Enabled 1. Use the format -e command. 2. Type the disk on which you are working at the format prompt. 3. Type inquiry at the scsi prompt. 4.
Page 96: Sun Storedge Traffic Manager-Enabled Devices
Sun StorEdge Traffic Manager-Enabled Devices 1. If the devices support the Sun StorEdge Traffic Manager software, you can use this shortcut. 2. Type: # luxadm display /dev/rdsk/c6t29000060220041956257334B30303148d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/ c6t29000060220041956257334B30303148d0s2 Status(Port A): Status(Port B): Vendor: Product ID: WWN(Node): WWN(Port A): WWN(Port B):...
Page 97: To View The Virtualization Engine Map
To View the Virtualization Engine Map The virtualization engine map is stored on the Storage Service Processor. 1. To view the virtualization engine map, type: # showvemap -n v1 -f VIRTUAL LUN SUMMARY Disk pool VLUN Serial Number --------------------------------------------------------------------------- t3b00 6257334B30303148 t3b00 6257334B30303149...
Page 98 2. You can optionally establish a telnet connection to the virtualization engine and run the runsecfg utility to poll a live snapshot of the virtualization engine map. Refer to “To Replace a Failed Virtualization Engine” on page 84 for telnet instructions.
Page 99: To Failback The Virtualization Engine
To Failback the Virtualization Engine In the event of a Sun StorEdge T3+ array LUN failover, use the following procedure to fail the LUN back to its original controller. 1. From the Storage Service Processor, type: # /opt/svengine/sduc/mpdrive failback -d v1 -j 60020F2000006DFA where: Virtualization engine pair on which to run the command Controller serial number, which corresponds to the Sun...
Page 100: To Replace A Failed Virtualization Engine
For detailed information about the SUNWsecfg scripts, refer to the Sun StorEdge 3900 and 6900 Series Reference Manual. To Replace a Failed Virtualization Engine 1. Replace the old (failed) virtualization engine unit with a new unit. 2. Identify the MAC address of the new unit and replace the old MAC address with the new one in the /etc/ethers file: 8:0:20:7d:82:9e virtualization engine-name 3.
Page 101 11. Enable the switch port: # /opt/SUNWsecfg/flib/setveport -v virtualization engine-name -e 12. Reset the virtualization engine: # resetve -n virtualization engine-name 13. Find the initiator number for the new and old number: # showvemap -n virtualization engine-pairname -l The new unit will not have any zones defined. 14.
Page 102: To Manually Clear The San Database
To Manually Clear the SAN Database It is occasionally necessary to manually clear the SAN database on the virtualization engine routers. Caution – This procedure will wipe out the SAN database and will remove the configuration of disk pools, Multipath drives, Zoning, and VLUNs. After performing this procedure, the virtualization map must be restored to the virtualization engine pair using /opt/SUNWsecfg/bin/restorevemap.
Page 103: Stopping And Restarting The Slic Daemon
Stopping and Restarting the SLIC Daemon Follow this procedure to restart the SLIC daemon if the SLIC daemon becomes unresponsive, or if messages such as the following are displayed: connect: Connection refused or Socket error encountered.. To Restart the SLIC Daemon 1.
Page 104 3. Remove the segments by typing the following: # ipcrm -m 301 -m 302 -m 303 -s 196608 -s 196609 -s 196610 Check the ipcrm(1m) man page for details. 4. Restart the SLIC daemon # /opt/SUNWsecfg/bin/startslicd -n v1 (or v2, depending on configuration) 5.
Page 105: Brick (2 Luns Total)
Sun StorEdge 6900 Series Multipathing Example One Sun StorEdge T3+ array partner pair with 1 500GB RAID 5 LUN per brick (2 LUNs total) Currently, there is one 10GB VLUN created from each physical LUN, for a total of two VLUNs. In a Sun StorEdge 6900 series, there are four possible physical paths to each Sun StorEdge T3+ array Volume (LUN).
Page 106: Figure 7-2 Sun Storedge 6900 Series Logical View
In the event of a path failure after the second tier of Sun StorEdge network FC switch-8 and switch-16 switches (or in the event of both T Ports failing between the switches), the virtualization engines force a LUN failover of the affected Sun StorEdge T3+ array and routes all I/O to its secondary path.
Page 107: Figure 7-3 Primary Data Paths To The Alternate Master
Primary Data Paths to the Alternate Master FIGURE 7-3 Chapter 7 Troubleshooting Virtualization Engine Devices For Internal Use Only...
Page 108: Figure 7-4 Primary Data Paths To The Master Sun Storedge T3+ Array
Primary Data Paths to the Master Sun StorEdge T3+ Array FIGURE 7-4 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide — March 2002...
Page 109: Figure 7-5 Path Failure-Before The Second Tier Of Switches
Path Failure—Before the Second Tier of Switches FIGURE 7-5 Chapter 7 Troubleshooting Virtualization Engine Devices For Internal Use Only...
Page 110: Figure 7-6 Path Failure -I/O Routed Through Both Hbas
Path Failure —I/O Routed through Both HBAs FIGURE 7-6 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide — March 2002...
Page 111: Virtualization Engine Event Grid
Virtualization Engine Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort virtualization engine events by component, category, or event type. The Storage Automated Diagnostic Environment GUI displays an event grid that describes the severity of the event, whether action is required, a description of the event, and the recommended action.
Page 112 lists the Virtualization Engine Events. TABLE 7-5 Storage Automated Diagnostic Environment Event Grid for Virtualization Engine TABLE 7-5 Category Component EventType virtualization enclosure Alarm engine virtualization enclosure Alarm.log engine virtualization enclosure Audit engine virtualization Comm_ engine Established virtualization Comm_ engine Lost Sun StorEdge 3900 and 6900 Series Troubleshooting Guide —...
Page 113 Storage Automated Diagnostic Environment Event Grid for Virtualization Engine (Continued) TABLE 7-5 Category Component EventType virtualization ve_diag Diagnostic engine Test- virtualization veluntest Diagnostic engine Test- virtualization enclosure Discovery engine For Internal Use Only Action Description ve_diag (diag240) on ve-1 (ip=xxx.20.67.213) failed veluntest (diag240) on ve-1...
Page 114 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide — March 2002...
Page 115: Troubleshooting The Sun Storedge T3+ Array Devices
C H A P T E R Troubleshooting the Sun StorEdge T3+ Array Devices This chapter contains the following sections: “Explorer Data Collection Utility” on page 99 “Sun StorEdge T3+ Array Event Grid” on page 109 Explorer Data Collection Utility The Explorer Data Collection Utility script is included on the Storage Service Processor in the /export/packages directory.
Page 116 Do not accept automatic emailing of the Explorer Data Collection Utility output, unless the Storage Service Processor is properly set up to handle mail correctly. Automatic Email Submission Would you like all explorer output to be sent to: explorer-database-americas@sun.com at the completion of explorer when -mail or -e is specified? [y,n] n Before running the Explorer Data Collection Utility, make sure that the switch and Sun StorEdge T3+ array information is added to the proper /opt/SUNWexplo/etc...
Page 117 Editing Sun StorEdge T3+ array information using vi CODE EXAMPLE 8-2 # vi t3input.txt # Input file for extended data collection # Format is HOST PASSWORD t3b0 XXXX t3b2 XXXX t3b3 XXXX :wq! Note – xxxx represents Sun StorEdge T3+ array passwords. You can now run /opt/SUNWexplo/bin/explorer to collect information about the Storage Service Processor operating system, the Sun StorEdge network FC switch-8 or switch-16 switch, and Sun StorEdge T3+ array information, which...
Page 118: Troubleshooting The T1/T2 Data Path
Troubleshooting the T1/T2 Data Path Notes There are two T Port links for redundancy. If one of the two links is lost, no Sun StorEdge T3+ array LUN failover will occur, and no pathing failures will be noted. If both T Port links fail, there will be a Sun StorEdge T3+ array LUN failover, as one of the virtualization engines take control of the I/O operations.
Page 119: T1/T2 Notification Events
T1/T2 Notification Events The example below shows a typical port failure event Site : Lab 3286 - DSQA1 Broomfield Source : diag.xxxxx.xxx.com Severity : Error (Actionable) Category : Switch DeviceId : switch:100000c0dd00b682 EventType: StateChangeEvent.M.port.8 EventTime: 01/30/2002 11:17:22 ’port.8’ in SWITCH diag209-sw2a (ip=192.168.0.32) is now Not-Available (status-state changed from ’Online’...
Page 120 If both T Ports go offline, you might see messages like the following. Note the virtualization engine Event alerting the LUN failover. Site : Lab 3286 - DSQA1 Broomfield Source : diag.xxxxx.xxx.com Severity : Warning (Actionable) Category : Ve DeviceId : ve:6257335A-30303142 EventType: AlarmEvent.volume EventTime: 01/30/2002 11:49:05 Volume T49152 on diag209-v1a changed from 6257335A-30303142(active=50020F23-...
Page 121: Figure 8-2 Virtualization Engine Alert
...continued from previous page... ---------------------------------------------------------------------- Site : Lab 3286 - DSQA1 Broomfield Source : diag.xxxxx.xxx.com Severity : Warning Category : Message DeviceId : message:diag.xxxxx.xxx.com EventType: LogEvent.driver.Fabric_Warning EventTime: 01/30/2002 11:50:07 Found 1 ’driver.Fabric_Warning’ warning(s) in logfile: /var/adm/messages on diag.xxxxx.xxx.com (id=809f76b4): INFORMATION: Fabric warning Jan 30 11:46:37 WWN:2b00006022004186 kern.warning] WARNING: fp(2): N_x Port with D_ID=108000,...
Page 122: Sun Storedge T3+ Array Storage Service Processor Verification
Sun StorEdge T3+ Array Storage Service Processor Verification 1. Run port listmap on the Sun StorEdge T3+ array to see the failover event. # t3b0:/:<1>port listmap port targetid u1p1 u1p1 u2p1 u2p1 2. Compare the virtualization engine configuration to a saved configuration by running /opt/SUNWsecfg/runsecfg and choosing Verify Virtualization Engine Map.
Page 123: T1/T2 Fru Tests Available
T1/T2 FRU Tests Available Switch - switchtest Link - linktest Running linktest from the Storage Automated Diagnostic Environment GUI will guide the Service Engineer to discover the failed FRU. Once the test has completed its run, an email message, similar to the following message, will be sent to the Email recipient that was specified in linktest.
Page 124: Notes
Notes When inserting a loopback connector into the T Port, there will be NO green light indicating a proper insertion. However, the test will run and be valid. There is currently an RFE to address this issue. If only one of the links has failed and the I/O is travelling over the remaining link, once the failed link is replaced and recabled, I/O will be automatically be routed over the repaired link by the switch.
Page 125: Sun Storedge T3+ Array Event Grid
Sun StorEdge T3+ Array Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort Sun StorEdge T3+ array events by component, category, or event type. The Storage Automated Diagnostic Environment GUI displays an event grid that describes the severity of the event, whether action is required, a description of the event, and the recommended action.
Page 126 The following table lists all of the events for the Sun StorEdge T3+ array. Category Component EventType power.temp Alarm+ disk.port Alarm- interface. Alarm- loopcard.cab Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Action Description The state of power.u1pcu1.PowTe mp on diag213 (ip=xxx.20.67.213) is...
Page 127 Category Component EventType power. Alarm- battery power.fan Alarm- For Internal Use Only Action Description [ Info/Action ] The state of power.u1pcu1.BatStat e on diag213 (ip=xxx.20.67.213) is Fault Possible causes are: 1. Voltage level on power supply and battery have moved out of acceptable thresholds.
Page 128 Category Component EventType power. Alarm- output power.temp Alarm- enclosure Alarm.log Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Action Description [ Info/Action ] The state of power.u1pcu1.PowOu tput on diag213 (ip=xxx.20.67.21 3) is Fault [ Info/Action ] The state of power.u1pcu1.
Page 129 Category Component EventType enclosure Alarm. time Discrepancy enclosure Audit Comm_ Established Comm_ Established For Internal Use Only Action Description Yello [ Action ] Time of T3 diag213 (ip=xxx.20.67.213) is different from host: T3=Fri Oct 26 10:16:17 200, Host=2001-10-26 12:21:04 [ Info ] Auditing a new Sun StorEdge T3+ array called ras d2-t3b1...
Page 130 Category Component EventType Comm_Lost Comm_Lost Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Action Description Down [ Info/Action ] Lost communication (InBandwithdiag213 (ip=xxx.20.67.21 3) ( last reboot was 2001-09-27 15:22:00) Information: InBand. This event is established using luxadm.
Page 131 Category Component EventType t3ofdg Diagnostic Test- t3test Diagnostic Test- t3volverify Diagnostic Test- enclosure Discovery controller Insert Component disk Insert Component interface. Insert loopcard Component For Internal Use Only Action Description t3ofdg (diag240) on diag213 (ip=xxx.20.67.213) failed t3test (diag240) on diag213 (ip=xxx.20.67.213) failed t3volverify (diag240)
Page 132 Category Component EventType power Insert Component enclosure Location Change enclosure QuiesceEnd enclosure QuiesceStart enclosure Removal controller Remove Component Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Action Description [ Info ] ’power.u1pcu2’(TE CTROL-CAN.300- 1454- 01(50).008275) was added to T3 diag213 (ip=xxx.20.67.21 Location of t3...
Page 133 Category Component EventType disk Remove Component interface. Remove loopcard Component power Remove Component controller State Change+ For Internal Use Only Action Description [ Info/Action ] disk.u2d3(SEAGAT E.ST318203FSUN18 G.LRG07139) was removed from diag158 (ip=xxx.20.67.158) [ Info/Action ] Information: The Sun StorEdge T3+ array has reported that a loopcard has been removed from...
Page 134 Category Component EventType disk State Change+ interface. State loopcard Change+ volume State Change+ power State Change+ Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Action Description disk.u1d5 in Sun StorEdge T3+ array rasd3-t3b1 (ip=xxx.0.0.41) is now Available (status-state changed from fault- disabled to ready-...
Page 135 Category Component EventType controller State Change- disk StateChange- For Internal Use Only Action Description [ Info/Action ] controller.u1ctr in T3 diag213 (ip=xxx.20.67.213) is now Not-Available (status-state changed from unknown to ready-disabled) Information: The Sun StorEdge T3+ array controller has been disabled.
Page 136 Category Component EventType interface. StateChange- loopcard Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Action Description [ Info/Action ] Information: The Sun StorEdge T3+ array has indicated that the loopcard is no longer in an optimal state. Information Recommended action: 1.
Page 137 Category Component EventType volume StateChange- power StateChange- enclosure Statistics For Internal Use Only Action Description [ Info/Action ] [ Info/Action ] power.u1pcu2(TECT ROL-CAN.300- 1454- 01(50).008275) in T3 rasd2-t3b1 (ip=xxx.0.0.41) is now Not-Available (status-state changed from ready-enabled to ready-disable). Statistics about T3 d2-t3b1 (ip=xxx.0.0.41) Chapter 8...
Page 138: Replacing The Master Midplane
Replacing the Master Midplane Follow this procedure when replacing the master midplane in a Sun StorEdge T3+ array. This procedure is detailed in the Storage Automated Diagnostic Environment User’s Guide. To Replace the Master Midplane 1. Choose Maintenance --> General Maintenance -- > Maintain Devices. Refer to Chapter 3 of the Storage Automated Diagnostic Environment User’s Guide.
Page 139: Troubleshooting Ethernet Hubs
C H A P T E R Troubleshooting Ethernet Hubs The Sun StorEdge 3900 and 6900 series uses an Ethernet hub as the backbone for the internal service network. The allocation of Ethernet ports are as follows: 1—Storage Service Processor (per subsystem) 1—for each Fibre Channel Switch 1—for each Virtualization Engine 2—for each Sun StorEdge T3+ array partner group...
Page 140 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 141 A P P E N D I X Virtualization Engine References This Appendix contains the following Tables: Table A-1 “SRN and SNMP Reference” Table A-2 “ SRN/SNMP Single Point of Failure Table Table A-3 “Port Communication” Table A-4 “Service Codes” provides an explanation of Service Request Numbers for the virtualization TABLE A-1 engine.
Page 142 SRN and SNMP Reference TABLE A-1 Description 70005 Write error is detected by master. If the initiator is master, then it has detected a write error on a member within a mirror drive. 70006 virtualization engine-to-virtualization engine communication has failed. 70007 Rebuild is aborted with write error.
Page 143 SRN and SNMP Reference TABLE A-1 Description 7009A Read degrade recorded. A mirror drive was written to, causing it to enter the degrade state. 7009B Write degrade recorded. If a spare drive is available, it will be brought in and used to replace the failed drive.
Page 144 SRN and SNMP Reference TABLE A-1 Description 72005 Failed to check for SAN changes. 72006 Failed to read SAN event log. 72007 SLIC daemon connection is down. SRN/SNMP Single Point of Failure Table TABLE A-2 SNMP Description 70020 • SAN topology has changed 70030 •...
Page 145 Port Communication TABLE A-3 Port Daemon Daemon Daemon virtualization engine provides service codes for the virtualization engine. TABLE A-4 Service Codes TABLE A-4 Code Number PCI bus parity error. The attempt to report one error resulted in another error. Corrupt database Corrupt database Zone mapping database This message indicates that an attempt to...
Page 146 Service Codes TABLE A-4 Unauthorized cabling configuration. Too many HBAs attempting to log in. Node mapping table cleared using SW2. Improper SW2 setting. Too many virtualization engines in SAN. Heartbeat connection between virtualization engines is down. FC device-side type code invalid. Too many elastic store errors to continue.
Page 147 A P P E N D I X SUNWsecfg Error Messages The Sun StorEdge 3900 and 6900 Series Reference Manual lists and defines the command utilities that configure the various components of the Sun StorEdge 3900 and 6900 series storage systems. The information in this appendix expands on that information by providing recommendations for corrective action, should you encounter errors with the command utilities.
Page 148 Virtualization Engine SUNWsecfg Error Messages TABLE B-1 Message Description and Cause of Error Common to Invalid virtualization engine pair name virtualization engines $vepair, or virtualization engine is unavailable. Confirm that the configuration locks are set. This is usually due to the savevemap command running.
Page 149 Virtualization Engine SUNWsecfg Error Messages (Continued) TABLE B-1 Message Description and Cause of Error Common to 1. Device-side operating mode is not virtualization engine set properly. 2. Device-side UID reporting scheme is not set properly. 3. Host-side operating mode is not set properly.
Page 150 Virtualization Engine SUNWsecfg Error Messages (Continued) TABLE B-1 Message Description and Cause of Error Invalid WWN $wwn on $vepair createvezone initiator $init, or virtualization engine is unavailable. No virtualization engines are available. listavailable They are either not found, or the configuration lock is set.
Page 151 Sun StorEdge Network FC Switch-8 and Switch-16 Switch SUNWsecfg Error Messages TABLE B-2 Message Description and Cause of Error Common Switch Sun StorEdge system type entered, ${cab_type}, does not match system type discovered, ${boxtype}. Common Switch 1. Unable to obtain lock on switch ${switch}.
Page 152 Sun StorEdge Network FC Switch-8 and Switch-16 Switch SUNWsecfg Error Messages TABLE B-2 Message Description and Cause of Error Invalid flash file $flashfile. setswitchflash Check the number of ports on switch $switch. ${switch} timed out after reset. The setswitchflash switch took longer than two minutes to reset after a configuration change.
Page 153 Sun StorEdge T3+ Array SUNWsecfg Error Messages TABLE B-3 Message Description and Cause of Error Common to Sun Present configuration does not match StorEdge T3+ array Reference configurations Common to Sun 1. Could not mount volume $vol. StorEdge T3+ array 2.
Page 154 Sun StorEdge T3+ Array SUNWsecfg Error Messages (Continued) TABLE B-3 Message Description and Cause of Error Snapshot configuration files are not checkt3config present. Unable to check configuration. 1. The $lun status reported a bad or checkt3mount nonexistent LUN. 2. While checking the configuration using the showt3 -n command, operations abort.
Page 155 Sun StorEdge T3+ Array SUNWsecfg Error Messages (Continued) TABLE B-3 Message Description and Cause of Error Error while the block size compare restoret3config command is executing. The $BRICK_IP{$IPADD} command is aborted. $LUN configuration failed to restore restoret3config and the force option was used to reinitialize, without success $LUN configuration is not found in the restoret3config...
Page 156 In this example, the getcabinet script might determine the device to be a Sun StorEdge 3900 series when, in reality, it is a Sun StorEdge 6900 series. Could not determine the Sun StorEdge system type.
Page 157: Setupswitch Exit Values
setupswitch Exit Values lists the setupswitch exit values. The associated messages are logged in the TABLE 9-1 /var/adm/log/SEcfglog log file. setupswitch Exit Values TABLE 9-1 Severity Level Message Type INFO ERROR WARNING WARNING WARNING NOTE: If multiple solutions are connected to a switch, the switch settings might not match the default settings. Message Meaning All switch settings are properly set.
Page 158 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...
Page 159 Index accessing documentation online, xv checkswitch used to diagnose and troubleshooting switch, 62 comments sending documentation comments, xv configuration settings, 47 veriﬁcation of, 47 data host verification for Sun StorEdge 39x0 series, 42 for Sun StorEdge 69x0 series, 42 diagrams ﬁbre channel link, 15, 16 documentation how book is organized, xi...
Page 160 health functions for Sun StorEdge 3900 and 6900 series, 2 host device names translating, 78 host devices troubleshooting, 53 host event grid, 53 host side troubleshooting, 18 suspension of, 10, 13 isolation procedures for A2/B2 link, 33 link error example of severe data host error, 24 lock file how to clear, 50 luxadm(1M)
Page 161 notiﬁcation events, 103 T1/T2 data path troubleshooting, 102 test examples command line, 19 qlctest(1M), 19 switchtest(1M), 20 thresholds used in PFA, 2 troubleshooting broad steps, 3 check status of Sun StorEdge T3+ array, 4 check status of the Sun StorEdge FC Network Switch-8 and Switch-16, 5 check status of the virtualization engine, 5 determine extent of the problem, 4...
Page 162 Index 146 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002...

This manual is also suitable for:

Storedge 6900 series

Sun Microsystems StorEdge 3900 Series Troubleshooting Manual

1 Introduction

2 General Troubleshooting Procedures

3 Troubleshooting the Fibre Channel Links

4 Configuration Settings

5 Troubleshooting Host Devices

6 Troubleshooting Sun Storedge FC Switch-8 and Switch-16 Devices

7 Troubleshooting Virtualization Engine Devices

8 Troubleshooting the Sun Storedge T3+ Array Devices

9 Troubleshooting Ethernet Hubs

Quick Links

Troubleshooting

Need help?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Sun Microsystems StorEdge 3900 Series

Summary of Contents for Sun Microsystems StorEdge 3900 Series

This manual is also suitable for:

Table of Contents