Virtual fabric 10gb switch module for ibm bladecenter (89 pages)
Summary of Contents for IBM RS/6000 SP
Page 1
RS/6000 SP SP Switch Service Guide GA22-7443-04...
Page 3
RS/6000 SP SP Switch Service Guide GA22-7443-04...
Page 4
Fifth Edition (April 2002) This book replaces GA22-7443-03. IBM welcomes your comments. A form for readers’ comments may be provided at the back of this publication or you may address your comments to the following address: International Business Machines Corporation...
The following is a list of all safety notices (in English only) pertaining to SP hardware maintenance tasks from this and other RS/6000 SP hardware publications. Translations of each of the safety notices into other languages are included in RS/6000 SP: Safety Information .
Page 12
During an electrical storm, do not connect or disconnect any cable that has a conductive outer surface or a conductive connector. ( SPSFD005 ) DANGER Switch off power and unplug the machine power cable from the power receptacle, before removing or installing any part that is connected to primary power. ( SPSFD006 ) DANGER To prevent possible electrical shock during machine installation, relocation, or reconfiguration, connect the primary power cable only after connecting all electrical signal cables.
Before disconnecting the power cables from the power receptacles, ensure that the customer’s branch distribution circuit breakers (customer power source circuit breakers) are Off and tagged with DO NOT OPERATE tags, S229-0237. Refer to “Lockout safety procedures” in RS/6000 SP: System Service Guide , before proceeding. ( SPSFD014 )
Page 14
The wide node weight may exceed 32 Kg (70.5 lbs). ( SPSFC006 ) CAUTION: Do not open more than one wide node or switch assembly drawer at a time. ( SPSFC007 ) CAUTION: Make sure the stability foot and wheel chocks are installed on the frame. These are required to maintain frame balance and position during service operations.
( SPSFC017 ) CAUTION: All IBM laser modules are designed so that there is never any human access to laser radiation above a class 1 level during normal operation, user maintenance, or prescribed service conditions.
Page 16
Recycling facilities might not be available in your area. In the United States, IBM has established a collection process for reuse, recycling, or proper disposal of used sealed lead-acid, nickel-cadmium and nickel metal hydride batteries and battery packs from IBM equipment.
If you are attempting to isolate an SP system failure, use the Maintenance Analysis Procedures (MAPs) beginning with the Start MAP in RS/6000 SP: System Service Guide (GA22-7442). For a listing of the complete RS/6000 SP hardware service library, see “Related information”.
(for example, a page number or table number). v Fill out one of the forms at the back of this book and return it by mail, by fax, or by giving it to an IBM representative.
Adobe Acrobat Reader. GA22-7443-00 First edition of the restructured RS/6000 SP hardware service library. This publication, along with the other SP service publications (see “Related information” on page xv), replaces The Maintenance Information Manuals Volumes 1–4 (GA22-7375, GA22-7376, GA22-7377, and GA22-7378) and makes them obsolete.
Refer to “Selecting appropriate switch clocks” on page 3-6 for these procedures. Refer to Figure 1-1 on page 1-2 for a high-level view of the RS/6000 SP Switch. Switch connection types Standard node Processor nodes in 9076 SP frames are attached to the SP Switches with switch cables.
Switch description and problem determination (MAP 0590) FRUs include: Fans, circuit breaker/LED card, switch supervisor card, switch power card(s), inner chassis cable, front chassis cable, complete assembly. SPS-8 There are only 8 ports. All clocks are distributed through data cables.
Page 23
-a -N sphwlog | pg (For SSP code levels 1.02 and higher) v Refer to the ″Starting a service call (MAP 0100)″ in RS/6000 SP: System Service Guide for more information on the pg command Note: In a frame with processor nodes, entries for the switch will refer to “node17”...
2. Advanced cable wrap tests will not run successfully for 15- and 20-meter SPS data cables. If 15- or 20-meter data cables are used, swap/change the suspected cable. RS/6000 SP: SP Switch Service Guide Action Go to “SP Switch environment (MAP 0600)”...
This MAP provides diagnostic information for switch problems that are related to the operating environment. Note: Refer to “Service position procedures” on page 3-9 for placing a switch into the service position or for removing the switch from the service position.
Voltage out of range: −5 V “shutdownN5” 2. Have the customer remove the SP Switch from the active configuration and power off the SP Switch. 3. Put the switch into service position. Refer to “Service position procedures” on page 3-9.
Page 27
Warning Fan: “fanwarning1”, “fanwarning2”, ..., “fanwarning5” v Shutdown Fan: “fanfail1”, “fanfail2”, ..., “fanfail5” 2. Have customer remove the SP Switch from the active configuration and power off the SP Switch. 3. Set switch circuit breaker to the Off position.
3. Install front panel assembly and reinstall front retaining screws (see “Service position procedures” on page 3-9). 4. Connect only power (J1) and supervisor (J2) cables at the rear of the SP Switch. 5. Put the SP Switch’s circuit breaker into the On (‘1’) position.
Page 29
Temperature is out of specified range; however, no serious electrical current or fan speed problems have been detected. 1. Check for airflow blockage at air intakes and exhaust of the SP Switch and system frame. Also, check air temperatures around the frame, looking for sources of abnormally high temperatures (above 40C or 104F).
Page 30
“...PS2Fail...” v “...PSFuseGood...” 2. Have the customer remove the SP Switch from the active configuration and power off the SP Switch. 3. Switch off the circuit breaker. 4. Unplug cables connected to J1 and J2 at rear of the SP Switch.
Page 31
2. Remove the switch supervisor card (refer to “Removing the switch supervisor card” on page 4-4). 3. Using a digital multimeter, measure resistance at the planar connection for the supervisor card, between pins 12A and 12B. v The resistance should be in a range of 4 to 20 ohms.
Page 32
After replacing the power supply card you still have a PSxFail condition. 1. Put the circuit breaker in the Off (‘0’) position. 2. Remove the power cable (J1) and supervisor cable (J2) from the rear of the SP Switch. 3. Remove the front panel assembly.
Purpose of this MAP This MAP provides diagnostic information for problems related to the SP Switch power supply. Note: Refer to “Service position procedures” on page 3-9 for placing the switch into the service position or removing it from the service position.
Page 34
1-36. v If the Power LED does not stay lit, the SP Switch is not responding to the command. a. Go to ″Frame supervisor not responding (MAP 0110)″ in RS/6000 SP: System Service Guide . Step 0610-004 From either a Perspectives window or by looking at the SP Switch, you determined that the Power (green) LED is Off.
Page 35
If yes, go to “Step 0610-009”. v If no, you have a problem with 48 V dc power distribution. a. Go to ″Open in 48 V dc distribution (MAP 0560)″ in RS/6000 SP: System Service Guide . Step 0610-009 Only one SP Switch is showing the circuit breaker on but the Power (green) LED is not lit.
Return to “Step 0610-006” on page 1-15. Step 0610-010 You have an SP type switch that is either tripping the circuit breaker to the Off position or the circuit breaker is on but the Power (green) LED is not lit.
This MAP provides diagnostic information for problems related to the SP Switch function. Notes: 1. Refer to “Appendix A” in RS/6000 SP: Installation and Relocation for frame-to-frame or switch-to-switch cables. Logical frame to physical frame translation information can also be found in this appendix.
Page 38
Notes: a. Verify that the timestamp on the out.top file is current (using ls -l /var/adm/SPlogs/css/out.top). b. Refer to ″Starting a service call (MAP 0100)″ in RS/6000 SP: System Service Guide for information on using the pg command. 8. Problems are indicated at the end of a text line by the following sequence: −# ErrorMessage...
4. Log into processor node and enter: lscfg | grep css 5. If you get no result, try swapping in another switch adapter. Repeat step 3. If you get a result, the problem is probably resolved; repeat “Step 0620-001” on page 1-17 for next problem or go to “Step 0620-043”...
Page 40
(SPS) Link status: Link has been removed from network, fenced 1-20 RS/6000 SP: SP Switch Service Guide Description and action Description: Possible hardware problem. Action: Go to “Step 0620-004” on page 1-21. Description: Switch network not wired as specified in switch topology or problem with connection between switch and device.
Page 41
Where applicable, frame and SP Switch jack numbers are indicated on each line of the out.top file. (Refer to “Format structure” on page 2-1 for more information on the nomenclature.)
If no, go to “Step 0620-006”. Step 0620-006 No cable connected at the indicated jack on the SP Switch (refer to the tables in “Switch data cables” on page 2-13). 1. Should there be a cable connecting the indicated jack to a processor node? v If yes: a.
Page 43
The cable appears to be fully connected at both the SP Switch and at the processor node. 1. From the front of the RS/6000 SP frame or from the control workstation, check the Power (green) LED on the processor node to make sure it is lit.
After using Table 1-9, go to “Step 0620-012” on page 1-26. v If no: a. The diagnostics did not detect a problem, therefore the problem is in SP Switch. b. Go to “Step 0620-041” on page 1-35. Table 1-9. Service Request Number (SRN) table for SP Switch adapters...
Page 45
Table 1-9. Service Request Number (SRN) table for SP Switch adapters (continued) Service Request Number source 763-200 through 763-299 (except 763-282) 763-282 763-2A0 through 763-2A9 763-3xx (See note) 763-4xx (See note) 763-5xx (See note) 763-6xx (See note) 763-7xx (See note)
Page 46
SP Switch function (MAP 0620) Table 1-9. Service Request Number (SRN) table for SP Switch adapters (continued) Service Request Number source 764-2A0 through 764-2A9 764-3xx (See note) 764-4xx (See note) 764-5xx (See note) 764-6xx (See note) 764-7xx (See note) 764-8xx...
Page 47
Otherwise, power off this processor node and continue service at “Step 0620-017” on page 1-28. a. Check to see if a switch clock cable is connected at switch adapter of this processor node. b. If clock cable exists, reseat switch clock cable;...
Page 48
SP Switch function (MAP 0620) Step 0620-013 An SRN listed in Table 1-9 on page 1-24 indicated that you have a switch clock problem and Priority 2 in Table 1-10 on page 1-27 directed you to this step. 1. Check the switch clock selections on the “problem” switches .
Page 49
Step 0620-021 An error message in the /var/adm/SPlogs/css/out.top file indicated that you either have an external switch connection problem or a problem with a wrapped port. Priority 3 in Table 1-8 on page 1-22 directed you to this step. 1. Check the specified jacks at the rear of the SP Switches for an obvious problem such as a loose connection.
Page 50
Go to “Step 0620-043” on page 1-36 to verify fix. Step 0620-024 Problem reported with all switch data cables from one or more frames. This may be caused by a clocking or grounding problem between frames. Note: The “master-clock” switch indicates the switch which is supplying the master clock for the system—the master clock will have the clock input multiplexor set to “internal clock”...
Page 51
If no, go to “Step 0620-028”. Step 0620-028 None of the problems reported on data cables connected to this SP Switch been resolved by selecting a different external clock. This indicates a probable problem with switch clock card or switch planar in this SP Switch.
Page 52
Ground straps are connected at bolts near the cable escapes inside the bottom rear of each frame. v For a view of a ground strap connection, refer to RS/6000 SP: Installation and Relocation . 2. Do all ground straps connecting the frames make adequate contact at both ends? v If yes, go to “Step 0620-033”...
Page 53
Do not perform cable wrap test. v Refer to ″Running diagnostics in a processor node″ in RS/6000 SP: System Service Guide . 4. Look for an SRN indicating a clock problem, such as “External clock” being listed as one of the failing components.
Page 54
Do not perform cable wrap test. v Refer to ″Running diagnostics in a processor node″ in RS/6000 SP: System Service Guide . 7. Look for an SRN indicating a clock problem, such as “External clock” being listed as one of the failing components.
Page 55
Do not perform cable wrap test. v Refer to ″Running diagnostics in a processor node″ in RS/6000 SP: System Service Guide . 3. Look for an SRN indicating a clock problem, such as “External clock” being listed as one of the failing components.
Page 56
2. If any processor nodes that lost switch clocks before or during service (due to unplugging node data/clock cable, loss of clock at the SP Switch connected to processor nodes, or so on), you will need to include them in the following command. From the control workstation, enter:...
To describe the jack 23 on the switch assembly bulkhead in the second RS/6000 SP frame in a four-frame configuration, designate as: E02-S01-BH-J23 To describe plug 1 on the power card of the any switch assembly of any RS/6000 SP frame in any size system configuration, designate as: E00-S00-PC-P1 or just PC-P1...
Front and rear views of RS/6000 SP frame Figure 2-1 shows a front view of the RS/6000 SP frame locations. “Frame (FRA)” on page 2-6 describes the assembly designations for the RS/6000 SP frame. Figure 2-1. Front view of frame locations. See notes below.
Figure 2-2 shows a front view of the RS/6000 SP multi-switch frame. SP Switch Frame Locations - Front View Main Power Switch with Left Skirt 48 V Power Modules (Front Cover and Skirt, and Air Filter Removed) Figure 2-2. Front view of multi-switch frame locations Figure 2-3 on page 2-5 shows a front view of the Model 3AX (49-inch) frame.
6. A High node or SMP High node takes up 2 shelf positions (slots). It is identified by the least odd number position of the occupied slots. 7. The switch assembly is not available in the 1.4 m frame. Figure 2-4 on page 2-6 shows a rear view of the RS/6000 SP frame locations. Chapter 2. Locations...
Note: See notes under Figure 2-1 on page 2-3 for processor node/switch assembly numbering. Frame locations Figure 2-1 on page 2-3 shows a front view of the RS/6000 SP frame locations, with numbered processor nodes, and the three phase SEPBU.
Page 63
Front door ground Rear door ground Ground Power-on switch LED card Front cover Rear cover Example: E01-FRA-G1 Chapter 2. Locations...
A i r B a f f l e Power Supervisor Power Supply 2 Card Supply 1 Fan 2 Fan 1 Fan 5 Fan 4 Fan 3 CB & LED Card Figure 2-5. SPS/SPS-8 assembly RS/6000 SP: SP Switch Service Guide...
Figure 2-6. RS/6000 SP connector details (as seen at receiving ends, not at cable ends) Cable routing Figure 2-7 on page 2-10 and Figure 2-8 on page 2-10 show back views of the RS/6000 SP frame, showing the horizontal and vertical paths of cable routing from connector-to-connector, with the depth amplified on the drawing.
2-foot service loop for node movement into service position. Figure 2-7. Frame cabling routing path in rear of RS/6000 SP frame — 1.93 m frame Figure 2-8. Frame cabling routing path in rear of RS/6000 SP frame — 2.01 m frame Note: For a multi-switch frame (F/C 2030/1), refer to Figure 2-7.
Cable routing in a multi-switch frame (F/C 2030/1) Figure 2-9 on page 2-12 and Figure 2-10 on page 2-12 show back views of the RS/6000 SP multi-switch frame with frame extension, showing the horizontal and vertical paths of cable routing from connector-to-connector.
E00-S00-BH-J15 Note: “PA” refers to connector on SPS adapter. Note: For external frame-to-frame cable locations, see RS/6000 SP: Installation and Relocation . SPS-8 data cables Table 2-3 describes the attachment locations and routing for the internal SPS-8 Switch data cables: Table 2-3.
Page 70
2-14 RS/6000 SP: SP Switch Service Guide...
Table 3-3. Tuning output files File (see note) Location daemon.stdout Primary daemon.results Primary RS/6000 SP: SP Switch Service Guide Runs on Description All nodes Monitors the switch for faults. It restarts the switch if a fault is detected. All nodes Monitors the adapter for interrupts that have not been serviced.
All nodes Note: Unless otherwise noted, the directory for these utilities is /var/adm/SPlogs/css. Using the css.snap script The css.snap script collects log files created by switch support code (device driver, worm, fault-service, diagnostic tests) into a single package. Attention: css.snap uses a number of undocumented utilities to collect information. Some of these, like read_regs and the tbXdump routines, can be destructive when used on a running system.
CSS share sinks below 30%. If successful, css.snap proceeds. If not, it exits. css.snap is called automatically from the fault-service daemon when certain serious errors are detected. It can also be issued from the command line by service personnel when a switch or adapter related problem is indicated.
2. In the Frame and Switch pane, click the icon of the switch you want to verify v A switch icon is displayed next to the frame icon only if a switch is installed in the frame 3. Click the ″Notebook″ icon on the tool bar v When the Notebook window opens, make certain that the ″Switch Status″...
To select an alternate clock topology, enter: Eclock -a filename Refer to IBM RS/6000 SP: Command and Technical Reference for more information on the correct file name to use. Determining the correct switch clock source All switch assemblies in the system must run off the same clock source. The following procedure...
2. If the primary processor node (usually in Frame 1) is connected to the switch identified in step 1, the customer must select an available processor node to be the new primary processor node. The primary node is set by the Eprimary command. Refer to IBM RS/6000 SP: Administration Guide for more information.
7. If the primary processor node was changed during the isolation procedure, the customer can change the primary processor node back to the original selection; however, this is NOT required. The primary node is set by the Eprimary command. Refer to IBM RS/6000 SP: Administration Guide , for more information.
5. If the inner chassis or supervisor power cable need to be replaced: a. Detach all cables from the rear of the switch assembly, noting where the external frame cables and wrap plugs are attached. b. Remove the inner chassis from the outer chassis sleeve by pushing on the inner chassis from the rear of the switch assembly.
Installing firmware updates on SP nodes Firmware updates (for example, IPL ROS updates for SP nodes), are available at http://www.rs6000.ibm.com/support/micro/download.html. Alternatively, you can search AIXTOOLS for the latest versions of the firmware updates. (for example, look for P2SC_IPL on AIXTOOLS for the latest version of IPL ROS on SP Nodes.)
Page 81
(Commands in following steps would be executes as in the example, but without the “dsh” prefix) Note: Refer to IBM RS/6000 SP: Administration Guide for help on using dsh. dsh -a "umount /mnt" dsh -a "mount <control wks>:/tmp/microcode /mnt"...
"ssadload -u" to update the disks. Other disks will be updated by a cfgmgr or reboot cycle. Running diagnostics on a switch port Before running diagnostic tests on a switch to diagnose node to switch problems, you must identify the port number. For example:...
2. If wrap plugs are installed, they can be plugged on the new switch. 3. Slide switch through the front of the frame and reinstall four screws that hold the switch to the frame. 4. If this is a switch drawer (in F/C 2030/1) and you are reinstalling the switch assembly, reinstall the two screws at the rear of the switch assembly.
Perform these procedures to remove a fan from an SP Switch assembly: 1. Perform “Placing a switch assembly into service position” on page 3-9 to place the switch assembly into the service position. 2. Remove the top cover of the switch assembly by removing the screws.
Perform these procedures to remove the fan control cable from an SP Switch assembly: 1. Perform “Placing a switch assembly into service position” on page 3-9 to place the switch assembly into the service position. 2. Unplug connectors P7, P8 and P9. Remove cable by unhooking retaining material along raceway, and unplugging the Fan connectors P2, P3, P4, P5 and P6.
1. Perform “Placing a switch assembly into service position” on page 3-9 to place the switch assembly into the service position. 2. Remove screws from connector P1 at the rear of the switch assembly, and retain them for new cable installation. Remove cable by unhooking retaining material along raceway. Unplug connectors P3, P4, P5 and P6.
Perform these procedures to replace the supervisor power cable from an SP Switch assembly: 1. Plug connectors P3, P4, P5 and P6. Route cable along the raceway, hooking retaining material where needed. Attach P1 connector to the rear of the switch assembly with screws retained in the removal procedure.
1. Insert power supply card. 2. Rotate card thumb locks inward to seat card. 3. Plug connector P4 (PS1) or P6 (PS2). 4. Perform “Replacing a switch assembly from service position” on page 3-9 to remove the switch assembly from the service position. Switch...
Table 5-5. Multi-switch frame (F/C 2030/1) Assembly Part number index 54G3281 93G1065 0375867 84X4761 17H5026 54G2877 54G2947 1624776 26H7198 08J5557 04H9469 11J5189 11J5191 11J5193 11J5195 21L3932 Units Description SPS Switch Assembly Screw, Phil Pan Hd M5 x 12 Shelf Assembly...
Page 102
5-12 RS/6000 SP: SP Switch Service Guide...
Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead.
Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits. IBM is not responsible for any radio or television interference caused by using other than recommended cables and connectors or by unauthorized changes or modifications to this equipment.
Dieses Gerät ist berechtigt in Übereinstimmung mit Dem deutschen EMVG vom 9.Nov.92 das EG–Konformitätszeichen zu führen. Der Aussteller der Konformitätserklärung ist die IBM Germany. Dieses Gerät erfüllt die Bedingungen der EN 55022 Klasse A. Für diese von Geräten gilt folgende Bestimmung nach dem EMVG: Geräte dürfen an Orten, für die sie nicht ausreichend entstört sind, nur mit besonderer Genehmigung des...
Please tell us how we can improve this book: Thank you for your responses. May we contact you? When you send comments to IBM, you grant IBM a nonexclusive right to use or distribute your comments in any way it believes appropriate without incurring any obligation to you.
Need help?
Do you have a question about the RS/6000 SP and is the answer not in the manual?
Questions and answers