Page 1 SPARC M8 and SPARC M7 Servers Service Manual Part No: E55215-06 September 2017...
Page 3 Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.
Page 4 Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou services émanant de tiers, sauf mention contraire stipulée dans un contrat entre vous et Oracle. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des dommages causés par l'accès à...
Page 5: Table Of Contents
Contents Using This Documentation ................ 13 Identifying Components .................. 15 Server Front Components (SPARC M8-8 and SPARC M7-8 Servers) ...... 16 Server Rear Components (SPARC M8-8 and SPARC M7-8 Servers) ....... 17 Server Front Components (SPARC M7-16) ............ 18 Server Rear Components (SPARC M7-16) ............ 19 CMIOU Chassis Front Components .............. 20 CMIOU Chassis Rear Components ..............
Page 6 ▼ Switch on PDU Circuit Breakers ............ 73 ▼ Restart the Server or Host (Oracle ILOM) .......... 74 ▼ Check the Power State of the Server or Host (Oracle ILOM) ...... 75 ▼ Return a Component to Service .............. 75 Oracle ILOM Service-Related Tools .............. 76...
Page 7 PDomain) ..................... 81 SPARC M7-16 Configurations and Failover Behavior .......... 81 Servicing CMIOUs ..................... 83 CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With Two PDomains) .. 84 CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With One PDomain) .. 85 CMIOU Configurations (SPARC M7-16) ............ 85 Mixed CMIOU Configurations ................ 86 CMIOU Redundancy ..................
Page 8 ▼ Remove a Power Supply ................ 167 ▼ Remove the Power Supply Shims (SPARC M8) ........... 170 ▼ Install the Power Supply Shims (SPARC M8) .......... 172 ▼ Install a Power Supply ................ 175 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 9 Contents Servicing Fan Modules (CMIOU Chassis) ............ 179 Fan LEDs (CMIOU Chassis) ................. 180 ▼ Remove a Fan Module (CMIOU Chassis) ........... 180 ▼ Install a Fan Module (CMIOU Chassis) ............ 182 Servicing Fan Modules (Switch Chassis) ............ 183 Fan Module LED (Switch Chassis) .............. 184 ▼ Remove a Fan Module (Switch Chassis) ............. 184 ▼...
Page 10 Servicing SP Internal Interconnect Assemblies .......... 285 ▼ Remove an SP Internal Interconnect Assembly .......... 286 ▼ Install an SP Internal Interconnect Assembly .......... 290 Servicing Internal Interconnect Assemblies ............ 297 Internal Interconnect Assembly Configuration ........... 298 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 11 Contents ▼ Remove an Internal Interconnect Assembly .......... 299 ▼ Install an Internal Interconnect Assembly ............ 302 Servicing External Interconnect Assemblies ........... 305 External Interconnect Assembly Configuration .......... 306 ▼ Remove an External Interconnect Assembly .......... 307 ▼ Install an External Interconnect Assembly ........... 311 Glossary ...................... 317 Index ........................
Page 12 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 13: Using This Documentation
Using This Documentation Overview – Describes how to troubleshoot and maintain the SPARC M8 and SPARC M7 ■ servers from Oracle Audience – Technicians, system administrators, and authorized service providers ■ Required knowledge – A basic understanding of the UNIX command line, virtualization ■...
Page 14 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 15: Identifying Components
These topics identify key components of the server, including major modules and assemblies, as well as front and rear panel features. For information about the hardware architecture and software environment, refer to “Understanding the Hardware Architecture” in SPARC M8 and SPARC M7 Servers Installation Guide in the SPARC M8 and SPARC M7 Servers Installation Guide.
Page 16: Server Front Components (Sparc M8-8 And Sparc M7-8 Servers)
Links Bottom CMIOU chassis “CMIOU Chassis Front Components” on page 20 Related Information “CMIOU Chassis Front Components” on page 20 ■ “CMIOU Chassis Rear Components” on page 21 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 17: Server Rear Components (Sparc M8-8 And Sparc M7-8 Servers)
Server Rear Components (SPARC M8-8 and SPARC M7-8 Servers) Server Rear Components (SPARC M8-8 and SPARC M7-8 Servers) Description Links Bottom CMIOU chassis “CMIOU Chassis Rear Components” on page 21 Related Information “CMIOU Chassis Rear Components” on page 21 ■...
Page 18: Server Front Components (Sparc M7-16)
Top CMIOU chassis “CMIOU Chassis Front Components” on page 20 Switch chassis “Switch Chassis Front Components (SPARC M7-16)” on page 23 Bottom CMIOU chassis “CMIOU Chassis Front Components” on page 20 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 19: Server Rear Components (Sparc M7-16)
Server Rear Components (SPARC M7-16) Related Information “Server Rear Components (SPARC M7-16)” on page 19 ■ Server Rear Components (SPARC M7-16) Identifying Components...
Page 20: Cmiou Chassis Front Components
“Servicing the Front Indicator Panel” on page 217 Fan modules (8) “Servicing Fan Modules (CMIOU Chassis)” on page 179 Bottom CMIOU chassis: ■ Left side, top to bottom: FM_6, FM_4, FM_2, FM_0 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 21: Cmiou Chassis Rear Components
CMIOU Chassis Rear Components Description Links ■ Right side, top to bottom: FM_7, FM_5, FM_3, FM_1 Top CMIOU chassis: ■ Left side, top to bottom: FM_14, FM_12, FM_10, FM_8 ■ Right side, top to bottom: FM_15, FM_13, FM_11, FM_9 Internal interconnects (5), left to right (SPARC “Servicing Internal Interconnect M8-8 or SPARC M7-8 with two PDomains): Assemblies”...
Page 22 CMIOU Chassis Rear Components Description Links Rear indicator panel (1) “Servicing Power Modules” on page 235 This component is part of the power module. ■ Bottom CMIOU chassis: ROPNL0 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 23: Switch Chassis Front Components (Sparc M7-16)
“Servicing PCIe Cards” on page 141 chassis) SPs (2), bottom CMIOU chassis, left to right “Servicing SPs” on page 117 (SPARC M8-8 and SPARC M7-8 servers): SP0,SP1 SPPs (4, SPARC M7-16 server): ■ Bottom CMIOU chassis, left to right: SPP0,SPP1 ■...
Page 24 ■ XI_N_4_T2B_S (slot IT_3, slot IT_11) ■ XI_N_4_T2B_S (slot IT_4, slot IT_13) SP internal interconnect (1): “Servicing SP Internal Interconnect Assemblies” on page 285 ■ II_CMIOU_SP (top CMIOU chassis slot, IT_20) SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 25: Switch Chassis Rear Components (Sparc M7-16)
Switch Chassis Rear Components (SPARC M7-16) Description Links ■ II_SWITCH_SP (switch chassis, slot IT_14) ■ II_CMIOU_SP (bottom CMIOU chassis slot, IT_5) “Servicing Power Power supplies (4), top to bottom: PS9–PS6 Supplies” on page 165 Related Information “Switch Chassis Rear Components (SPARC M7-16)” on page 25 ■...
Page 26 Rear indicator panel (1) “Servicing Power Modules” on page 235 This component is part of the power module. ROPNL1 AC inputs (1 strip, 4 outlets) “Servicing Power Modules” on page 235 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 27: Dimm And Eusb Locations In A Cmiou
DIMM and eUSB Locations in a CMIOU Description Links This component is part of the power module. Top to bottom: AC9–AC6 “Servicing Fan Modules (Switch Fan modules (36), left to right: FM0–FM5 per switch Chassis)” on page 183 unit “Servicing Switch Units” on page 189 Switch units (6), top to bottom: SWU_5–SWU_0 “Servicing SPs”...
Page 28: Supported Storage Devices
The server supports these storage devices through FC, iSCSI, NAS, or IB protocols: SAN attached FC arrays ■ ZFS appliances ■ The server supports solid state storage add-in-cards in addition to these types of tape backup and restore devices: TCP/IP ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 29: Understanding Server Cooling Zones
“Switch Chassis Cooling Zones” on page 31 ■ Related Information “Preparing for Cooling” in SPARC M8 and SPARC M7 Servers Installation Guide ■ CMIOU Chassis Cooling Zones The CMIOU chassis has two cooling zones. In one cooling zone, eight fan modules push air through the CMIOUs from the front of the server and exhaust it at the rear of the server.
Page 30 CMIOU power supply cooling zone “Servicing Fan Modules (CMIOU Chassis)” on page 179 “Servicing Power Supplies” on page 165 Related Information “Switch Chassis Cooling Zones” on page 31 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 31: Switch Chassis Cooling Zones
Understanding Server Cooling Zones Switch Chassis Cooling Zones The switch chassis has seven cooling zones. Six of the cooling zones contain a horizontal row of six fan modules each. Each of these zones pushes air through a single switch unit from the front of the server and exhausts it at the rear of the server.
Page 32: Component Service Task Reference
PCIe x16 card carriers, 24 per Not applicable “Servicing PCIe Cards” on page 141 CMIOU chassis PCIe cards, 3 cards per CMIOU “Servicing PCIe Cards” on page 141 /Servers/PDomains/PDomain_x/system/ PCI_Devices/Add-on/Device_x SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 33 /System/Power/Power_Supplies/Power_Supply_x chassis); 6 per chassis Power supplies (switch chassis); “Servicing Power Supplies” on page 165 /System/Power/Power_Supplies/Power_Supply_x 4 per chassis SPs (SPARC M8-8 and SPARC “Servicing SPs” on page 117 /System/Other_Removable_Devices/ M7-8 servers); 2 per CMIOU Service_Processors/Service_Processor_x chassis SPs (SPARC M7-16); 2 per “Servicing SPs”...
Page 34 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 35: Detecting And Managing Faults
If a service procedure has cleared the fault, ensure that the server is now operational. If a fault remains following a service procedure, clear it manually. Contact technical support if the problem persists. https://support.oracle.com Related Information “Identifying Components” on page 15 ■...
Page 36: Identify Faulted Components
The fmadm faulty displays a list of faults detected by PSH. You can run this command from either the host or through the Oracle ILOM fault management shell. This example shows how to check for faults through the Oracle ILOM fault management shell. You can also check for faults by typing show faulty at the Oracle ILOM prompt.
Page 37 : Server will be powered down when there are insufficient operational power supplies. Action : Please refer to the associated reference document at http://support.oracle.com/msg/SPT-8000-5X for the latest service procedures and policies regarding this diagnosis. faultmgmtsp> In this example, a fault is displayed that includes these details: Date and time of the fault (2014-10-27/10:20:25).
Page 38: Identify Disabled Components
■ Identify Disabled Components You can run the show disabled command from the Oracle ILOM prompt to identify components that have been disabled either intentionally, by a user, or because of a fault. To identify disabled components, at the Oracle ILOM prompt type: ->...
Page 39: Component Oracle Ilom State Properties
■ “Component Names Displayed by Diagnostic Software” on page 40 ■ Component Oracle ILOM State Properties The following table defines the Oracle ILOM properties seen when using the Oracle ILOM show command. Oracle ILOM Property Definition Property can be Enabled or Disabled. When Enabled, the component is requested_config_state installed and accessible by the server.
Page 40: Component Names Displayed By Diagnostic Software
DIMMs /SYS/CMIOUx/CM/CMP/BOByw/CHz/DIMM SPARC M8-8: x is 0–7, y is 0–3, w is 0 or 1, z is 0 or 1 SPARC M7-8: x is 0–7, y is 0–3, w is 0 or 1, z is 0 or 1 SPARC M7-16: x is 0–15, y is 0–3, w is 0 or 1, z is 0 or 1...
Page 41 PCIe cards /SYS/CMIOUx/PCIEy PCIe card carriers /SYS/CMIOUx/PCIEy/CAR SPARC M8-8: x is 0–7, y is 1–3 SPARC M7-8: x is 0–7, y is 1–3 SPARC M7-16: x is 0–15, y is 1–3 PDECBs /SYS/PDECBx SPARC M8-8: x is 0–7 SPARC M7-8: x is 0–7...
Page 42: Component Fault Tolerances
0–2 SPMs (CMIOU chassis) /SYS/SPx/SPMy SPARC M8-8 and SPARC M7-8 (two PDomains): x is 0, 1 and y is 0, 1 SPARC M8-8 and SPARC M7-8 (one PDomain): x is 0, 1 and y is 0 /SYS/SPPx/SPMy SPARC M7-16: x is 0–3, y is 0–1...
Page 43 If you remove any of these components while the server is powered on, wait 30 seconds Note - before installing the replacement component. Doing so ensures that Oracle ILOM has enough time to detect the new component, which is required for the software to power it on.
Page 44: Oracle Ilom Chassis View
Interpreting LEDs Oracle ILOM Chassis View The web interface of Oracle ILOM provides an interactive chassis view page to view the overall health of the chassis components. The chassis view provides a visual representation of the front and rear of the server chassis to show all serviceable components and indicate where components are absent.
Page 45: Component-Specific Leds
Interpreting LEDs Component-Specific LEDs This table lists component-specific fault LEDs that light when the component requires service. You can use these LEDs to help locate a component that the fmadm faulty command reports as faulty. Where indicated, these components also display blue Ready to Remove LEDs that must light before you can remove the component.
Page 46: Front Indicator Panel Controls And Leds
“Servicing Switch Units” on page 189 ■ Front Indicator Panel Controls and LEDs On SPARC M8-8 and SPARC M7-8 servers, the front indicator panel is located on the CMIOU chassis. On the SPARC M7-16 server, the front indicator panel is located on the switch chassis.
Page 47 Interpreting LEDs Icon or Label Description well as other server LEDs under Oracle ILOM control. The LEDs will light for 15 seconds and then return to normal operation. Component LEDs that are not under Oracle ILOM control will Note - not light up during this test.
Page 48: Rear Indicator Panel Controls And Leds
Interpreting LEDs Rear Indicator Panel Controls and LEDs On SPARC M8-8 and SPARC M7-8 servers, the rear indicator panel is located on the CMIOU chassis. On the SPARC M7-16 server, rear indicator panels are located on both CMIOU chassis and on the switch chassis.
Page 49 All lights on test: Press the Locate button three times within two seconds to test the functionality of the front and rear panel LEDs as well as other server LEDs under Oracle ILOM control. The LEDs will light for 15 seconds and then return to normal operation. Note - Component LEDs that are not under Oracle ILOM control will not light up during this test.
Page 50: Performing Advanced Troubleshooting
Display Fault Diagnostics (Message Buffer) Run the dmesg command on the host to check the system buffer for recent diagnostic messages and then display them. Log in as superuser. Type: SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 51: View Log Files (Oracle Solaris)
The error logging daemon, syslogd, automatically records various system warnings, errors, and faults in message files. These messages can alert you to system problems such as a device that is about to fail. For example, drive faults are usually captured by the Oracle Solaris message files.
Page 52: Checking Server And Component Health With Post
You can also set other Oracle ILOM properties to control various other aspects of POST operations. For example, you can specify the events that cause POST to run, the level of testing POST performs, and the amount of diagnostic information POST displays.
Page 53 The value of keyswitch_state must be normal when individual POST parameters are Note - changed. For possible values for the keyswitch_state parameter, see Oracle ILOM Administrator's Guide for Configuration and Maintenance at http://www.oracle.com/goto/ilom/docs. If the virtual keyswitch is set to normal, and you want to define level or verbosity, set the respective parameters.
Page 54: Verify That A Fault Has Been Cleared
■ Oracle ILOM Properties That Affect POST Behavior There are a number of Oracle ILOM commands that you can use to perform host diagnostic tests. For details about using these commands, refer to the chapter that describes configuring host server management actions in the Oracle ILOM Administrator's Guide for Configuration and Maintenance for your server's version of the Oracle ILOM firmware at http://www.
Page 55: Clear A Fault Manually
Clear a Fault Manually “Interpreting LEDs” on page Log in to Oracle ILOM. “Log In to Oracle ILOM (Service)” on page Verify that the server is operating normally. At the Oracle ILOM prompt, type: -> start SP/faultmgmt/shell Are you sure you want to start the faultmgmt shell (y/n)? y faultmgmtsp>...
Page 56 Clear the fault in the Oracle Enterprise Manager Ops Center software, if applicable. Clearing a fault with the fmadm acquit command does not clear that fault in the Oracle Enterprise Manager Ops Center software. You must manually clear the fault or incident. For more information, see "Marking an Incident Repaired"...
Page 57: Preparing For Service
Prevent ESD damage before you handle any server “Prevent ESD Damage” on page 61 component. “Attach an Antistatic Wrist Strap” on page 62 Properly prepare a component for return to Oracle. “Prepare a Component for Return to Oracle” on page 64 Related Information “Identifying Components”...
Page 58: Safety Symbols
No. 2 Phillips and flathead screwdrivers ■ T8, T10, T20, and T25 Torx wrench key, wrench, or driver ■ Mechanical lift (for some service procedures if only one person is present) ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 59: Filler Panels And Modules
Filler Panels and Modules Related Information “Component Service Categories” on page 59 ■ “Prevent ESD Damage” on page 61 ■ Filler Panels and Modules Depending on the configuration, the server can include the following items: CMIOU fillers are present in configurations that have fewer than the maximum number of ■...
Page 60: Find The Server Serial Number
Find the Server Serial Number If you need technical support for your server, you must to provide the server's serial number. Use one of the following options to find the serial number: SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 61: Locate The Server
Locate the Server Locate the manufacturing sticker on the front of the server or on the sticker on the side of the server. At the Oracle ILOM prompt, type: -> show /System serial_number /System Properties: serial_number = AK00239918 -> Related Information “Locate the Server”...
Page 62: Attach An Antistatic Wrist Strap
“Attach an Antistatic Wrist Strap” on page 62 for the ESD grounding jack locations on Oracle racks. If you installed a stand-alone SPARC M8-8 or SPARC M7-8 server in your own rack, refer to the rack documentation for grounding instructions.
Page 63 Attach an Antistatic Wrist Strap These grounding jacks accommodate 4-mm (0.16-in.) banana plug connectors. The Sun Rack II 1242 rack contains two ESD grounding jacks, one in the ■ front and one in the rear of the rack. Preparing for Service...
Page 64: Prepare A Component For Return To Oracle
Refer to the removal procedure for the component for any special repacking and shipping instructions. For example, refer to “Return a CMIOU to Oracle” on page 98 for detailed information about packing and shipping a CMIOU. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 65 Add any required paperwork or other documentation in the container. Except when packing CMIOUs, include any tools that were loaned to you by Oracle. Do not place tools inside a container that is being used to return a CMIOU. Close the shipping container and seal it with the packaging tape supplied by Oracle.
Page 66 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 67: Controlling The Server And Domains
Controlling the Server and Domains These topics explain how to access Oracle ILOM to service components in the server and how to power off and on the server and individual domains. Description Links Log in to Oracle ILOM. “Log In to Oracle ILOM (Service)” on page 67 Prepare a component for a hot-service “Prepare a Component for Removal”...
Page 68: Prepare A Component For Removal
CMIOUs /DCUs/DCU_x/CMIOUy SPARC M8-8 and SPARC M7-8 (two PDomains): x is 0 and y is 0–3, or x is 1 and y is 4–7 SPARC M8-8 and SPARC M7-8 (one PDomain): x is 0 and y is 0–7 SPARC M7-16: x is 0 and y is 0–3, x is 1 and y is 4–7, x is 2 and y is 8–11, or x is 3 and y is 12–15...
Page 69: Removing Power From The Server Or Domain
These topics describe different methods for removing power from the chassis. “Prepare to Power Off the Server or Domain” on page 69 ■ “Power Off the Server or Domain (Oracle ILOM)” on page 70 ■ “Switch Off PDU Circuit Breakers” on page 71 ■...
Page 70: Power Off The Server Or Domain (Oracle Ilom)
Depending on the type of problem, you might want to view system status or log files. You also might want to run diagnostics before you shut down the system. Switch from the system console to the Oracle ILOM prompt by typing the #. (Hash Period) key sequence.
Page 71: Switch Off Pdu Circuit Breakers
SPARC M8-8 or a SPARC M7-8 server in the bottom of an Oracle rack. In other configurations, where a SPARC M8-8 or SPARC M7-8 server might be installed in a different location in the rack, power is still removed first from the even numbered power supplies, and then from the odd numbered power supplies.
Page 72 AC input power cords at the SPARC M8-8 or SPARC M7-8 AC inlets. First, unplug the power cords at the even numbered AC inlets 0, 2, 4. Next, unplug the power cords at the odd numbered AC inlets 1, 3, 5.
Page 73: Returning The Server To Operation
SPARC M8-8 or SPARC M7-8 server in the bottom of an Oracle rack. In other configurations, where a SPARC M8-8 or SPARC M7-8 server might be installed in a different location in the rack, power is still restored first to the odd numbered power supplies (1, 3, 5), and then to the even numbered power supplies (0, 2, 4).
Page 74: Restart The Server Or Host (Oracle Ilom)
On multidomain servers, if one of the hosts is already running, you cannot use the Note - start /System command. The server will return a message that the target is already started. To power on a specific domain, at the Oracle ILOM prompt, type: ■ -> start /Servers/PDomains/PDomain_x/HOST If you are restarting the host following the replacement of a faulty component, verify that the fault has been resolved.
Page 75: Check The Power State Of The Server Or Host (Oracle Ilom)
Log in to Oracle ILOM. “Log In to Oracle ILOM (Service)” on page Determine if the system or domain has power. To check the power status of a domain on a single-domain server, at the Oracle ILOM ■ prompt, type: ->...
Page 76: Oracle Ilom Service-Related Tools
CMIOUs /DCUs/DCU_x/CMIOUy SPARC M8-8 and SPARC M7-8 (two PDomains): x is 0 and y is 0–3, or x is 1 and y is 4–7 SPARC M8-8 and SPARC M7-8 (one PDomain): x is 0 and y is 0–7 SPARC M7-16: x is 0 and y is 0–3, x is 1 and y is 4–7, x is 2 and y is 8–11, or x is 3 and y is 12–15...
Page 77 Oracle ILOM Service-Related Tools Oracle ILOM Command Description depending on the mode in which the Oracle Solaris OS was booted. Stops host server and generates a core dump file. set /HOST send_break_action=coredump Useful for debugging conditions when the host hangs.
Page 78 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 79: Understanding Server Configurations And Failover Behavior
These topics describe how components, particularly CMIOUs, are assigned to PDomains and DCUs, and how SPs manage system activity. “SPARC M8-8 and SPARC M7-8 Configurations and Failover Behavior (Two ■ PDomains)” on page 80 “SPARC M8-8 and SPARC M7-8 Configurations and Failover Behavior (One ■...
Page 80 /SYS/SP0/SPM1 /SYS/SP1/SPM1 Oracle ILOM identifies one of the SPMs from each SP as the Active SP to manage activity on the static PDomain. The other SPM on the SP runs Oracle ILOM, but remains inactive unless the Active SP can no longer manage the hardware. In this case, the inactive SPM assumes the Active SP role.
Page 81: Sparc M7-16 Configurations And Failover Behavior
SPARC M8-8 and SPARC M7-8 Configurations and Failover Behavior (One PDomain) In a SPARC M8-8 or a SPARC M7-8 server that is configured with a single PDomain, there are eight CMIOUs in the PDomain. Each CMIOU chassis has two redundant SPs. To facilitate failover from one SP to another, one SP serves as the Active SP, and one serves as the Standby SP.
Page 82 In this case, the inactive SPM assumes the role of the DCU SPM. Oracle ILOM also identifies one of the DCU SPMs from the pool of DCU SPMs on the same PDomain as the PDomain SPM to manage activity on that host.
Page 83: Servicing Cmious
These topics explain how to service CMIOUs. Description Links Learn about CMIOU configurations. “CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With Two PDomains)” on page 84 “CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With One PDomain)” on page 85 “CMIOU Configurations (SPARC M7-16)” on page 85 “Mixed CMIOU Configurations”...
Page 84: Cmiou Configurations (Sparc M8-8 And Sparc M7-8 With Two Pdomains)
CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With Two PDomains) Each PDomain in a SPARC M8-8 and SPARC M7-8 server that is configured with two static PDomains can consist of up to four of the eight CMIOUs in the chassis.
Page 85: Cmiou Configurations (Sparc M8-8 And Sparc M7-8 With One Pdomain)
CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With One PDomain) A SPARC M8-8 and SPARC M7-8 server that is configured as a single PDomain can have as many as eight CMIOUs in a single CMIOU chassis, as listed in the following table.
Page 86: Mixed Cmiou Configurations
The SPARC M7-8 server configured with two static PDomains is the only server that supports a mixed configuration of SPARC M7 and SPARC M8 CMIOUs. In a SPARC M7-8 server with two static PDomains, SPARC M8 CMIOUs can be installed in PDomain_1 (slots CMIOU_4 to CMIOU_7).
Page 87 SPARC M7-8 server PDomain_1 containing both SPARC M7 and SPARC M8 CMIOUs. ■ There cannot be a mix of SPARC M7 and SPARC M8 CMIOUs in PDomain_1. All of the CMIOUs in PDomain_1 must only be either SPARC M7 CMIOUs or SPARC M8 CMIOUs.
Page 88: Cmiou Redundancy
When installed in the proper slots, CMIOUs are designed to provide redundant access to the SP. Always install two CMIOUs in the first two slots of a SPARC M8-8 and SPARC M7-8 PDomain or a SPARC M7-16 DCU. Always follow these CMIOU configuration guidelines: “CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With Two...
Page 89: Sparc M8 Cmiou Label
“Install a CMIOU” on page 102 ■ SPARC M8 CMIOU Label The SPARC M8 CMIOUs contain a SPARC M8 label on the left extraction lever. SPARC M7 CMIOUs do not have a label. Related Information “SPARC M8-8 and SPARC M7-8 Configurations and Failover Behavior (Two ■...
Page 90: Cmiou Indicators And Leds
■ Steady on – CMIOU is powered on and is running in its normal operating state. No service actions are required. ECB fault (amber) Indicates that a PDECB requires service. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 91: Prepare A Cmiou For Removal
Note - When you unpack the new CMIOU, be sure to set the connector cover aside. You must install this cover on the CMIOU that you return to Oracle. Determine which CMIOU requires service. “Identify Faulted Components” on page 36 “CMIOU Indicators and...
Page 92 Prepare the installed CMIOU for removal. -> set /System/DCUs/DCU_x/CMIOU_y action=prepare_to_remove where: SPARC M8-8 or SPARC M7-8 (with two PDomains): x is 0 and y is 0–3, or x is 1 and y is ■ 4–7 SPARC M8-8 or SPARC M7-8 (with one PDomain): x is 0 and y is 0–7 ■...
Page 93: Unseat A Cmiou
A CMIOU is heavy. A fully-loaded SPARC M7 CMIOU weighs 25 lbs (11.3 kg) and Note - a fully-loaded SPARC M8 CMIOU weighs 27 lbs (12.2 kg). Use two hands when handling a CMIOU, and do not handle it by the ejectors or connectors.
Page 94 Grasp the ejector arms as close to the CMIOU as possible and pull the CMIOU one-third to halfway out of the server (panel 3). Related Information “Remove a CMIOU” on page 95 ■ “Install a CMIOU” on page 102 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 95: Remove A Cmiou
The CMIOU replacement kit includes a plastic cover to protect CMIOU connectors. Note - Install the connector cover on the CMIOU that you return to Oracle. Ensure that you have prepared the CMIOU for removal. “Prepare a CMIOU for Removal” on page Unseat the CMIOU.
Page 96 105. If you are removing a CMIOU in preparation for the replacement of another component, refer to the related replacement procedure. Component Link DIMMs “Servicing DIMMs” on page 105 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 97 “Servicing eUSB Disks” on page 203 PDECBs “Servicing PDECBs” on page 199 Related Information “Reseat a CMIOU” on page 101 ■ “Return a CMIOU to Oracle” on page 98 ■ “Install a CMIOU” on page 102 ■ Servicing CMIOUs...
Page 98: Return A Cmiou To Oracle
Install the protective connector cover, which you removed from the replacement CMIOU, on the CMIOU that you are returning to Oracle. You must install this cover on the CMIOU (panel 1). SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 99 It is very important that you properly install the connector cover to prevent additional Caution - damage during shipping and aid Oracle in determining how the component failed. Ensure that the connector cover is fully engaged and centered over connectors.
Page 100 Insert bottom center cushion on bottom of outer shipping container. Position sealed inner shipping container on top of bottom center cushion. Insert top center cushion on top of sealed inner shipping container. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 101: Reseat A Cmiou
Reseat a CMIOU Close the outer shipping container and seal it with the packaging tape supplied by Oracle. Related Information “Reseat a CMIOU” on page 101 ■ “Install a CMIOU” on page 102 ■ Reseat a CMIOU Some component replacement tasks (for example, those for interconnects and PDECBS) require you to unseat a CMIOU before you perform them.
Page 102: Install A Cmiou
If you remove this component while the server is powered on, wait 30 seconds before Caution - installing the replacement component. Doing so ensures that Oracle ILOM has enough time to detect the new component, which is required for the software to power it on.
Page 103 Install a CMIOU Install the CMIOU. Carefully slide the CMIOU less than half way into the slot, taking care to avoid bumping the connectors on the back of the CMIOU (panel 1). Open the green CMIOU levers so that they are fully open (panel 2). Insert the CMIOU back into its slot in the server until the levers begin to engage (panel 3).
Page 104 “Verify That a Fault Has Been Cleared” on page Related Information “Reseat a CMIOU” on page 101 ■ “Unseat a CMIOU” on page 93 ■ “Remove a CMIOU” on page 95 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 105: Servicing Dimms
Understand DIMM labels or names. “DIMM Names” on page 108 Locate a faulty DIMM. “Determine Which DIMM Is Faulty (Oracle ILOM)” on page 109 “Determine Which DIMM Is Faulty (LEDs)” on page 109 “DIMM Configuration Fault Messages” on page 112 Replace a DIMM.
Page 106: Dimm Configuration
CMIOU have the same capacity. Use only Oracle qualified DIMMs. ■ The SPARC M8 CMIOU DIMM locations and the buffer chip (BOB) and DDR channel (CH) numbers are as follows: SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 107 DIMM Configuration The SPARC M7 CMIOU DIMM locations and the buffer chip (BOB) and DDR channel (CH) numbers are as follows: Servicing DIMMs...
Page 108: Dimm Names
SPARC M7-8, where x is 0 to 7 ■ SPARC M7-16, where x is 0 to 15 ■ Buffer chip (BOB) number (where y is 0 to 3 and w is 0 or 1) ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 109: Determine Which Dimm Is Faulty (Oracle Ilom)
DIMMs on two different SPARC M7 CMIOUs. 2013-09-06/17:16:20 ereport.hc.dev_fault@/SYS/CMIOU1/CM/CMP/BOB11/CH1/DIMM 2013-09-06/17:16:20 ereport.hc.dev_fault@/SYS/CMIOU3/CM/CMP/BOB01/CH0/DIMM Determine Which DIMM Is Faulty (Oracle ILOM) At the Oracle ILOM prompt, type show faulty. The following output might be slightly different, depending on the server in use. -> show faulty Target...
Page 110 DIMM LED. Any faulty DIMM is identified by an amber LED next to the DIMM slot until you release the button. Note - If LEDs for two DIMMs light, replace both of the DIMMs. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 111 The following illustration shows the location of the green Fault Remind Power LED and Note - blue Fault Remind button on a SPARC M7 CMIOU. On a SPARC M8 CMIOU, the LED and button are adjacent to the bottom DIMM slot (labeled BOB21/CH0/DIMM on the board). Although their locations are different, the Fault Remind Power LED and Fault Remind Power button operate the same way on both CMIOU types.
Page 112: Dimm Configuration Fault Messages
“DIMM Configuration” on page 106. To list configuration faults that are found, you can run the fmadm faulty command from either the host or through the Oracle ILOM fault management shell. See “Identify Faulted Components” on page Remove a DIMM DIMMs can be serviced by customers.
Page 113 Remove a DIMM “Determine Which DIMM Is Faulty (LEDs)” on page 109. Remove the DIMM. Caution - Use caution when pressing the DIMM ejector latch to ensure that you do not come into contact with the adjacent DCDC power board. Doing so might loosen, or otherwise adversely impact, the DCDC board, causing the server to report system errors.
Page 114: Install A Dimm
Install the DIMM. Align the DIMM notch with the key in the connector. Ensure that the orientation is correct. The DIMM might be damaged if the orientation Caution - is reversed. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 115 Install a DIMM Push the DIMM into the connector until the ejector tabs lock the DIMM in place. If the DIMM does not easily seat into the connector, check the DIMM orientation. Install the CMIOU into the server. “Install a CMIOU” on page 102.
Page 116 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 117: Servicing Sps
Caution - servers. Never install a SPARC M8 SP in a SPARC M7 server, and never install a SPARC M7 SP in the SPARC M8-8 server. The SPARC M8 SP has a SPARC M8 label on its left extraction lever (see “SER MGT and NET MGT Port Locations and Numbering”...
Page 118: Sp Firmware And Configuration
SP boards, and is downloaded to the CMIOU when the host is started. For the server to operate correctly, the firmware in these two components must be compatible. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 119 SP Firmware and Configuration Before replacing an SP, save the configuration using the Oracle ILOM backup utility. Refer to the Oracle ILOM documentation for instructions on backing up and restoring the Oracle ILOM configuration. is initialized on a SPARC M7-8 or SPARC M7-16 server, use the tpmadm Note - failover command to backup TPM data and keys automatically to the Standby SP.
Page 120: Sp General Status Leds
No service actions are required. ■ Slow blink – A normal but transitory activity is taking place. Slow blinking might indicate that system diagnostics are running or that the system is booting. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 121: Ser Mgt And Net Mgt Port Locations And Numbering
SPARC M8 and SPARC M7 Servers Installation Guide for information about cabling the network management ports on your server. The SPARC M8 SP has a SPARC M8 label on its left extraction lever. The SPARC M7 Note - SPs do not have a label.
Page 122: Oracle Ilom Paths For Spms
SPP2/SPM0 .../Proxy_Service_Processor_3/Service_Processor_Module_0 SPP3/SPM0 DCU3 .../Proxy_Service_Processor_2/Service_Processor_Module_1 SPP2/SPM1 .../Proxy_Service_Processor_3/Service_Processor_Module_1 SPP3/SPM1 Related Information “Understanding Server Configurations and Failover Behavior” on page 79 ■ SPARC M8 and SPARC M7 Servers Administration Guide ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 123: Prevent Automatic Sp Or Spp Failover
Determine Which SP Is Managing System Activity The SP that is identified as the Active SP manages system activity. On SPARC M8-8 and SPARC M7-8 servers, this SP is located in the CMIOU chassis. On a SPARC M7-16 server, this SP is located in the switch chassis.
Page 124: Determine Which Spp Is Managing Dcu Activity (Sparc M7-16)
On an SP, type: -> set /SP/redundancy initiate_failover_action=true Related Information “Understanding Server Configurations and Failover Behavior” on page 79 ■ SPARC M8 and SPARC M7 Servers Administration Guide ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 125: Change Which Spp Is Managing Dcu Activity (Sparc M7-16)
You cannot specify which SPM will serve as the DCU SPM or PDomain SPM, but you can specify that Oracle ILOM should assign a new SPM in the pair to serve these roles. You might do this when, for example, you are replacing an SPP.
Page 126 SPM. For example, if you previously ran the state_sp command on Service_Processor_0/Service_Processor_Module_0, you now need to run it on Service_Processor_0/Service_Processor_Module_1. For SPARC M8-8 and SPARC M7-8 servers only, verify that the PCIe devices are offline. ■...
Page 127 Determine your next step based on your server Oracle ILOM version. SPARC M7 servers running Oracle ILOM version 4.0.x do not require you to prepare the SPM for removal. SPARC M8-8 servers ship with a minimum Oracle ILOM version of 4.0.x.
Page 128 PCIe card for removal by taking the devices offline for the appropriate SP/SPM slot. See “Servicing PCIe Cards” on page 141. (Oracle ILOM 3.2.x only) Verify that the SPM on the SP is ready for removal. -> show /System/Other_Removable_Devices/Service_Processors/ Service_Processor_x/Service_Processor_Module_0 health...
Page 129: Prepare To Remove An Spp (Sparc M7-16)
Prepare to Remove an SPP (SPARC M7-16) Prepare to Remove an SPP (SPARC M7-16) SPARC M7-16 servers have two SPPs in each CMIOU chassis and two SPs in the switch chassis. SPPs have two service processor modules each. SPs have one SPM each. This procedure explains how to remove an SPP.
Page 130 Determine your next step based on your server Oracle ILOM version. SPARC M8-8 and SPARC M7 servers running Oracle ILOM version 4.0.x do not require you to prepare the SPM for removal or verify that the SPMs and PCIe devices are offline.
Page 131 Prepare to Remove an SPP (SPARC M7-16) This command can take several minutes to complete. (Oracle ILOM 3.2.x only) Verify that the SPP is ready to remove by verifying that its SPMs have stopped. It can take up to two minutes for the SPMs to go offline. Once they are offline, you can prepare the SPP for removal.
Page 132: Remove An Sp Or Spp
“Prevent ESD Damage” on page Locate the lit blue Ready to Remove LED. The component is in the rear of the server. See “SP General Status LEDs” on page 120. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 133 Remove an SP or SPP If you are replacing a faulty SP, relocate any existing NET MGT and SER MGT port plugs. Remove any port plugs that are installed in the NET MGT and SER MGT ports and install them in the same ports on the replacement SP or SPP.
Page 134 8 (one PDomain)The SPs and SPPs are tightly secured in their bays. Pull firmly on the ejector latches to free the SP or SPP from the bay. Close the ejector arms. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 135 Remove an SP or SPP This protects the arms from damage when you pull the component out. Carefully remove the component from the SP tray, using two hands, and avoid bumping the rear connectors. Place the SP on a grounded antistatic mat. Set the /System/DCUs failover property back to auto.
Page 136: Install An Sp Or Spp
The SPARC M8-8 servers include SPs that are not compatible with the SPARC M7 Caution - servers. Never install a SPARC M8 SP in a SPARC M7 server, and never install a SPARC M7 SP in the SPARC M8-8 server.
Page 137 For instructions, refer to the Oracle ILOM documentation. Download the system firmware and install it on the Active SP. Follow the firmware download instructions in the Oracle ILOM documentation and refer to the SPARC M8 and SPARC M7 Servers Product Notes for the current firmware version.
Page 138 Reinitialize TPM and reset the tpmadm failover command if it was previously in use. For information about initializing TPM using the Oracle ILOM interface to enable failover, Securing Systems and Attached Devices in Oracle Solaris 11.3. Restore the TPM data and keys that were backed up to the Standby SP on a new SP you install.
Page 139 EST. Return the replaced component to Oracle. “Prepare a Component for Return to Oracle” on page Related Information “SER MGT and NET MGT Port Locations and Numbering” on page 121 ■...
Page 140 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 141: Servicing Pcie Cards
To remove a PCIe card that is assigned to an I/O domain, first remove the device Caution - from the I/O domain. For more information about making hardware changes to an I/O domain, refer to the Oracle VM for SPARC documentation at http://www.oracle.com/goto/vm-sparc/ docs.
Page 142: Pcie Card Slot Configuration And Numbering
“CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With Two ■ PDomains)” on page 84 “CMIOU Configurations (SPARC M8-8 and SPARC M7-8 With One ■ PDomain)” on page 85 “CMIOU Configurations (SPARC M7-16)” on page 85 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 143: Understanding Pcie Card Device And Service Paths
PCIe Card Paths (Bottom CMIOU Chassis) These are the CMIOU numbers, device paths, and service paths for each PCIe card slot in the SPARC M8-8 server, the SPARC M7-8 server, and the SPARC M7-16 server bottom CMIOU chassis. CMIOU No.
Page 144: Pcie Card Paths (Top Cmiou Chassis)
CMIOU12 /pci@33c/pci@1 /SYS/CMIOU12/PCIE2 pci_60 CMIOU12 /pci@33f/pci@1 /SYS/CMIOU12/PCIE1 pci_63 CMIOU11 /pci@338/pci@1 /SYS/CMIOU11/PCIE3 pci_56 CMIOU11 /pci@337/pci@1 /SYS/CMIOU11/PCIE2 pci_55 CMIOU11 /pci@33a/pci@1 /SYS/CMIOU11/PCIE1 pci_58 CMIOU10 /pci@333/pci@1 /SYS/CMIOU10/PCIE3 pci_51 CMIOU10 /pci@332/pci@1 /SYS/CMIOU10/PCIE2 pci_50 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 145: Supported Pcie Card Carriers And Pcie Cards
"x16 CAR" on the faceplate. PCIe x8 card carriers from earlier generations of Oracle servers, which are labeled with "CAR" on the faceplate, are not supported for use in SPARC M8 and M7 servers. PCIe x8 and PCIe x16 cards are supported for use in these servers.
Page 146: Card Installation Bottom Cmiou Chassis (Servers With Multiple Pdomains)
“Card Installation Top CMIOU Chassis (SPARC M7-16)” on page 148 ■ Card Installation Bottom CMIOU Chassis (Servers With Multiple PDomains) For SPARC M8-8 and SPARC M7-8 servers with two PDomains and SPARC M7-16 servers, install PCIe cards in fully populated configurations in the order shown. PDomain CMIOU...
Page 147: Card Installation Bottom Cmiou Chassis (Sparc M7 With One Pdomain)
CMIOU_7 PCI2 Card Installation Bottom CMIOU Chassis (SPARC M7 With One PDomain) For SPARC M8-8 and SPARC M7-8 servers that has one PDomain, install PCIe cards in fully populated configurations in the order shown. CMIOU PCIe Slot Card to Install...
Page 148: Card Installation Top Cmiou Chassis (Sparc M7-16)
PDomain_3 CMIOU_14 PCI2 PDomain_3 CMIOU_15 PCI2 Related Information “PCIe Card Slot Configuration and Numbering” on page 142 ■ “Understanding PCIe Card Device and Service Paths” on page 143 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 149: Pcie Card Carrier Leds And Controls
PCIe Card Carrier LEDs and Controls PCIe Card Carrier LEDs and Controls Icon Description Service Required The Service Required LED indicates that service is required. (amber) and The show faulty command provides details about any faults that cause this indicator to Locator LED light.
Page 150: Locate A Pcie Card In The Server
Locate a PCIe Card in the Server Locate a PCIe Card in the Server At the Oracle Solaris prompt, list all of the PCIe devices in the device tree: # hotplug list -l | grep PCIE This command lists dynamically reconfigurable PCIe resources and shows their operational status.
Page 151: Prepare A Pcie Card For Removal
Prepare a PCIe Card For Removal If the PCIe card is not assigned to an I/O domain, see “Remove a PCIe Card Carrier and ■ Card From the Server” on page 156. Related Information “PCIe Card Slot Configuration and Numbering” on page 142 ■...
Page 152 I/O domains. As shown in the preceding example, these virtual functions are indicated under the TYPE column as VF. You can remove virtual functions dynamically without shutting down the domains. Note - SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 153 Prepare a PCIe Card For Removal # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 ldg4 Verify that the virtual functions have been removed from the I/O domains. # ldm ls-io /SYS/CMIOU9/PCIE2 NAME TYPE DOMAIN...
Page 154 /SYS/CMIOU9/PCIE2/IOVFC.PF0. Verify that the virtual functions were removed from all physical functions of the card. Display the details for the slot, for example: SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 155 Physically remove the PCIe card from the server. “Remove a PCIe Card Carrier and Card From the Server” on page 156. For more information about making PCIe hardware changes, refer to the Oracle VM Server for SPARC documentation at http://www.oracle.com/goto/vm-sparc/docs. Related Information “Identify Faulted Components”...
Page 156: Remove A Pcie Card Carrier And Card From The Server
Remove the carrier from the slot. Pull the carrier’s extraction lever. Swing the extraction lever out 90 degrees until the far end of the lever begins to push the carrier out of the slot. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 157 Remove a PCIe Card Carrier and Card From the Server Remove the carrier from the slot. Carefully remove the PCIe card from the carrier. Press the green tab to unlock the carrier latch, and open the top of the PCIe card carrier (steps 1 and 2).
Page 158: Install A Pcie Card Into A Card Carrier
The card is correctly seated only when the tab on the top of the card (indicated in yellow in panel 1) is securely seated inside the receptacle on the carrier (see panel 2). SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 159: Install A Card Carrier Into The Server
Install only PCIe x16 hot-plug card carriers, which are physically labeled with "x16 CAR" on the faceplate, in SPARC M8 and M7 servers. Do not install PCIe x8 card carriers, which are labeled with "CAR" on the faceplate, from earlier generations of Oracle servers. See “Supported...
Page 160 The carrier’s LEDs should blink for a few seconds until the PDomain enables the PCIe card. The card’s LEDs will show activity when the card is enabled. Verify the card's installation. “Verify the PCIe Card” on page 164. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 161: Restore A Pcie Card Configuration On An I/O Domain
“Restore a PCIe Card Configuration on an I/O Domain” on page 161 ■ Restore a PCIe Card Configuration on an I/O Domain After the PCIe card has been physically replaced, verify that the card is now visible to Oracle VM Manager. # ldm ls-io pcie-slot For example: # ldm ls-io pci_45...
Page 162 “Prepare a PCIe Card For Removal” on page 151 for examples of the test file. Refer to the Oracle VM Server for SPARC documentation for the syntax of the create-vf command. Verify that the virtual functions were created. The following example shows only an excerpt of the output.
Page 163 Restore a PCIe Card Configuration on an I/O Domain /SYS/CMIOU9/PCIE2 PCIE pci_45 primary [pci@32d/pci@1] SUNW,qlc@0/fp/disk SUNW,qlc@0/fp@0,0 SUNW,qlc@0,1/fp/disk SUNW,qlc@0,1/fp@0,0/ssd@w20350080e518b846,1f fibre-channel@0,2 fibre-channel@0,3 fibre-channel@0,4 fibre-channel@0,5 /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 pci_45 ldg4 [pci@32d/pci@1/SUNW,qlc@0,2] Class properties [FIBRECHANNEL] port-wwn = 10:00:00:14:4f:f8:74:85 node-wwn = 20:00:00:14:4f:f8:74:85 bw-percent = 0 /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 pci_45 ldg4 [pci@32d/pci@1/SUNW,qlc@0,5] Class properties [FIBRECHANNEL]...
Page 164: Verify The Pcie Card
[/SYS/CMIOU0/PCIE1] ENABLED network@0 <pci.0,0> ONLINE network@0,1 <pci.0,1> ONLINE network@0,2 <pci.0,2> ONLINE network@0,3 <pci.0,3> ONLINE Related Information “Identify Faulted Components” on page 36 ■ “Interpreting LEDs” on page 44 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 165: Servicing Power Supplies
The SPARC M8-8 server includes power supplies that are not compatible with the Caution - SPARC M7 servers. Never install a SPARC M8 power supply in a SPARC M7 server, and never install a SPARC M7 power supply in the SPARC M8-8 server. The SPARC M8 power supplies include three shims installed on one side of the power supply (see “Remove the Power Supply...
Page 166: Power Supply Leds
“Preparing for Service” on page 57 ■ “Controlling the Server and Domains” on page 67 ■ “Understanding Server Configurations and Failover Behavior” on page 79 ■ Power Supply LEDs SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 167: Remove A Power Supply
Remove a Power Supply Icon Description AC Present Lights when the AC voltage is applied to the power supply. (green) OK (green) Lights when the power supply DC voltage from the power supply to the server is within tolerance. Service Required The LED lights when the power supply is faulty.
Page 168 Shown by the No. 1 arrows in the preceding illustration. Pull the extraction lever toward you to disengage the power supply. Shown by the No. 2 arrow in the preceding illustration. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 169 Remove a Power Supply Remove the power supply from the server. Slide the power supply halfway out of the server. Fold the extraction lever back toward the center of the power supply until it latches into place. This protects the lever from damage when you pull the power supply out. Carefully remove the power supply from the server.
Page 170: Remove The Power Supply Shims (Sparc M8)
Place the faulty SPARC M8 power supply on a grounded antistatic mat. A SPARC M8 power supply with have three shims installed on one of its sides. Ensure that the three shims face upward on the mat.
Page 171 Remove the shims from the side of the power supply. Place these shims in a safe location. Install the shims on the replacement power supply. Continue to “Install the Power Supply Shims (SPARC M8)” on page 172. Servicing Power Supplies...
Page 172: Install The Power Supply Shims (Sparc M8)
■ Install the Power Supply Shims (SPARC M8) Before installing a power supply into the SPARC M8-8 server, install three shims along one side of the power supply. Place the replacement SPARC M8 power supply on a grounded antistatic mat.
Page 173 Install the Power Supply Shims (SPARC M8) Ensure that the arrow labeled PSU FRONT is pointing to the front of the power supply. Note - The ESD foam tape should be facing upward. Using a T8 driver, secure the shim to the side of the power supply with one M3 screw per shim.
Page 174 Related Information “Remove the Power Supply Shims (SPARC M8)” on page 170 ■ “Remove a Power Supply” on page 167 ■ “Install a Power Supply” on page 175 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 175: Install A Power Supply
The SPARC M8-8 server includes power supplies that are not compatible with the Caution - SPARC M7 servers. Never install a SPARC M8 power supply in a SPARC M7 server, and never install a SPARC M7 power supply in the SPARC M8-8 server.
Page 176 Shown by the No. 1 arrow in the preceding illustration. Align the power supply and slide it into the empty bay. Shown by the No. 2 arrow in the preceding illustration. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 177 Verify that the fault has been cleared and the replaced component is operational. “Verify That a Fault Has Been Cleared” on page Return the replaced component to Oracle. “Prepare a Component for Return to Oracle” on page Servicing Power Supplies...
Page 178 Install a Power Supply Related Information “Power Supply LEDs” on page 166 ■ “Remove a Power Supply” on page 167 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 179: Servicing Fan Modules (Cmiou Chassis)
Servicing Fan Modules (CMIOU Chassis) A CMIOU chassis contains eight fan modules. The server can operate at full capacity with seven fan modules installed in the CMIOU chassis. The server will not operate with fewer than seven fan modules. If the server is operating with seven fan modules and one or more of those fan modules fails, the server will continue operating until defined temperature thresholds are exceeded, at which point the server will perform a graceful shutdown.
Page 180: Fan Leds (Cmiou Chassis)
■ Remove a Fan Module (CMIOU Chassis) This procedure can be performed by a customer while the server is running if there are at least seven operating fan modules. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 181 Remove a Fan Module (CMIOU Chassis) This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge. This discharge can cause server components to fail. When you unpack the replacement part, open the package on a grounded antistatic mat. Always use an antistatic wrist strap for hardware replacements.
Page 182: Install A Fan Module (Cmiou Chassis)
If you remove this component while the server is powered on, wait 30 seconds before Caution - installing the replacement component. Doing so ensures that Oracle ILOM has enough time to detect the new component, which is required for the software to power it on.
Page 183: Servicing Fan Modules (Switch Chassis)
Servicing Fan Modules (Switch Chassis) A switch chassis contains 36 fans modules. Each switch unit has 6 dedicated fan modules. The server will continue to operate at full capacity with 5 of the 6 fan modules operating for each switch unit. If the server is operating with one or more switch units that have only 5 fan modules, and one or more of those fan modules fails, the server will continue operating until defined temperature thresholds are exceeded, at which point the server will perform a graceful shutdown.
Page 184: Fan Module Led (Switch Chassis)
This procedure can be performed by a customer while the server is running if there are at least five of the six fan modules operating for each switch unit. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 185 Remove a Fan Module (Switch Chassis) If you are removing fan modules to reinstall them in a new switch unit, remove them Note - while the switch unit is installed in the chassis. This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge.
Page 186: Install A Fan Module (Switch Chassis)
If you remove a fan module while the server is powered on, wait 30 seconds before Caution - installing the replacement fan module. Doing so ensures that Oracle ILOM has enough time to detect the new component, which is required for the software to power on the new fan module.
Page 187 Install a Fan Module (Switch Chassis) “Switch Chassis Cooling Zones” on page 31 ■ Servicing Fan Modules (Switch Chassis)
Page 188 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 189: Servicing Switch Units
Servicing Switch Units In SPARC M7-16 servers, switch units are part of the scalability feature that enables a PDomain to control DCUs. Switch units are configured to work together as a single unit. If a switch unit fails, the server will operate in degraded mode. At least five switch units must be functioning for the server to operate.
Page 190: Switch Unit Leds
The SP might be running. ■ Steady on – System is powered on and is running in its normal operating state. No service actions are required. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 191: Prepare A Switch Unit For Removal
Complete the following steps before you remove a switch unit. The switch unit replacement kit includes plastic covers to protect switch unit connectors. Note - Return these covers to Oracle. Determine which switch unit requires service. “Identify Faulted Components” on page 36 “Switch Unit LEDs”...
Page 192: Unseat A Switch Unit
“Prepare a Switch Unit for Removal” on page 191. Verify that the blue Ready to Remove LED on the switch unit is on. “Switch Unit LEDs” on page 190. Unseat the switch unit. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 193 Unseat a Switch Unit If you are unseating a switch unit so you can replace it with a new one, ensure that you Note - have already removed the fan modules from the switch unit. Pull the ejector arms out to disengage the switch unit from the server. Press the arms back toward the unit to prevent them from being damaged.
Page 194: Remove A Switch Unit
“Reseat a Switch Unit” on page 196 ■ “Switch Chassis Cooling Zones” on page 31 ■ Remove a Switch Unit This procedure can be performed by a customer while the server is running. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 195 Always use an antistatic wrist strap for hardware replacements. The switch unit replacement kit includes plastic covers to protect switch unit connectors. Note - Return these covers to Oracle. Prepare the switch unit for removal. “Prepare a Switch Unit for Removal” on page 191.
Page 196: Reseat A Switch Unit
-> start /System Related Information “Install a Switch Unit” on page 197 ■ “Unseat a Switch Unit” on page 192 ■ “Switch Chassis Cooling Zones” on page 31 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 197: Install A Switch Unit
If you remove this component while the server is powered on, wait 30 seconds before Caution - installing the replacement component. Doing so ensures that Oracle ILOM has enough time to detect the new component, which is required for the software to power it on.
Page 198 “Prepare a Component for Return to Oracle” on page Related Information “Remove a Switch Unit” on page 194 ■ “Remove a Fan Module (Switch Chassis)” on page 184 ■ “Switch Chassis Cooling Zones” on page 31 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 199: Servicing Pdecbs
Servicing PDECBs There is one PDECB for each CMIOU. For the locations of the PDECBs in the CMIOU chassis, see “CMIOU Chassis Rear Components” on page There is one PDECB for the each switch unit (SPARC M7-16). For the locations of the PDECBs in the switch chassis, see “Switch Chassis Rear Components (SPARC M7-16)”...
Page 200: Remove A Pdecb
Remove the component that is installed in front of the failed PDECB. For information about preparing a CMIOU for removal and for information about removing a CMIOU, see “Servicing CMIOUs” on page SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 201 Remove a PDECB For information about preparing a switch unit for removal and for information about removing a switch unit, see “Servicing Switch Units” on page 189. Press the latch on the bottom of the PDECB to the left and slide it out of the chassis.
Page 202: Install A Pdecb
“Verify That a Fault Has Been Cleared” on page Return the replaced component to Oracle. “Prepare a Component for Return to Oracle” on page Related Information “Remove a PDECB” on page 200 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 203: Servicing Eusb Disks
Servicing eUSB Disks A single eUSB disk is installed in each CMIOU in the server. For the location of the eUSB disks inside a CMIOU, see “DIMM and eUSB Locations in a CMIOU” on page This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge.
Page 204 IPoIB for booting, and the eUSB disk in that CMIOU is the only disk in the boot pool, you must install the eUSB disk you remove in the new CMIOU. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 205: Install An Eusb Disk
Install an eUSB Disk Related Information “Install an eUSB Disk” on page 205 ■ Install an eUSB Disk Install the new eUSB disk. Position the new eUSB disk in the CMIOU (panel 1). Ensure proper pin engagement with connectors before pressing the eUSB disk in place. Tighten the green screw to secure the eUSB disk to the CMIOU (panel 2).
Page 206 Return the replaced component to Oracle. “Prepare a Component for Return to Oracle” on page Related Information “Remove an eUSB Disk” on page 203 ■ Understanding eUSB disks ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 207: Servicing A Cmiou Chassis Fan Cable Assembly
Servicing a CMIOU Chassis Fan Cable Assembly A CMIOU chassis contains two fan cable assemblies. For the location of the fan cable assembly in the CMIOU chassis, see “CMIOU Chassis Rear Components” on page This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge.
Page 208 Remove all of the interconnect assemblies from the front of the affected chassis. “Servicing Internal Interconnect Assemblies” on page 297, “Servicing External Interconnect Assemblies” on page 305 (SPARC M7-16), and “Servicing SP Internal Interconnect Assemblies” on page 285. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 209 Remove a Fan Cable Assembly Unscrew the bracket at the top of the cable assembly. Release all four panel mount connectors on each tab. Servicing a CMIOU Chassis Fan Cable Assembly...
Page 210 On the black tab, push down on the arrow while sliding the tab to the left until the pegs on the tab align with the slats in the metal brace. Press the tab back to release it. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 211 Remove a Fan Cable Assembly Remove the cable through the opening in the chassis. Related Information “Install a Fan Cable Assembly” on page 212 ■ Servicing a CMIOU Chassis Fan Cable Assembly...
Page 212: Install A Fan Cable Assembly
Install a Fan Cable Assembly Install a Fan Cable Assembly Slide the new cable through the opening in the chassis where the failed cable was previously installed. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 213 Install a Fan Cable Assembly Insert the four panel mount connectors through each of the receiving tabs and slide the black tab to the left to secure it. Servicing a CMIOU Chassis Fan Cable Assembly...
Page 214 “Returning the Server to Operation” on page Verify that the fault has been cleared and the replaced component is operational. “Verify That a Fault Has Been Cleared” on page SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 215 Install a Fan Cable Assembly Return the replaced component to Oracle. “Prepare a Component for Return to Oracle” on page Related Information “Remove a Fan Cable Assembly” on page 207 ■ Servicing a CMIOU Chassis Fan Cable Assembly...
Page 216 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 217: Servicing The Front Indicator Panel
Servicing the Front Indicator Panel The front indicator panel is located on the front of the CMIOU chassis for SPARC M8-8 and SPARC M7-8 servers, and on the front of the CMIOU chassis and switch chassis for SPARC M7-16 servers. The service procedures are the same on either chassis. For the location of the front indicator panel, see “CMIOU Chassis Front Components”...
Page 218 “Removing Power From the Server or Domain” on page Locate the indicator panel at the front of the server. “CMIOU Chassis Front Components” on page 20 “Switch Chassis Front Components (SPARC M7-16)” on page SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 219 Remove a Front Indicator Panel Remove the Torx screws and loosen the captive screws to remove the faceplate. Servicing the Front Indicator Panel...
Page 220 Remove a Front Indicator Panel With the faceplate removed, remove the No. 1 Phillips screws to free the indicator panel from the chassis. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 221 Remove a Front Indicator Panel Disconnect the small cable from the panel and remove the panel completely from the server. Related Information “Install a Front Indicator Panel” on page 222 ■ “Front Indicator Panel Controls and LEDs” on page 46 ■...
Page 222: Install A Front Indicator Panel
Install a Front Indicator Panel Install a Front Indicator Panel Connect the small cable from the chassis into the new indicator panel and position the panel in place. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 223 Install a Front Indicator Panel Install the No. 1 Phillips screws to secure the indicator panel to the chassis. Servicing the Front Indicator Panel...
Page 224 “Returning the Server to Operation” on page Verify that the fault has been cleared and the replaced component is operational. “Verify That a Fault Has Been Cleared” on page Return the replaced component to Oracle. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 225 Install a Front Indicator Panel “Prepare a Component for Return to Oracle” on page Related Information “Remove a Front Indicator Panel” on page 217 ■ “Front Indicator Panel Controls and LEDs” on page 46 ■ Servicing the Front Indicator Panel...
Page 226 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 227: Servicing The Front Indicator Panel Cable
Servicing the Front Indicator Panel Cable The front indicator panel cable connects the front indicator panel to the chassis. This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge. This discharge can cause server components to fail. Follow antistatic precautions to prevent electrostatic damage.
Page 228 If the cable you are replacing is in the CMIOU chassis, remove the left CMIOU chassis fan cable assembly. “Servicing a CMIOU Chassis Fan Cable Assembly” on page 207. Release the Molex connector. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 229 Remove the Front Indicator Panel Cable Reach through the hinging access door on the chassis divider floor where the SP tray slides to unlatch the Molex connector. Servicing the Front Indicator Panel Cable...
Page 230 Remove the Front Indicator Panel Cable Remove the two captive screws that secure the cable to the chassis. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 231 Remove the Front Indicator Panel Cable Remove the cable through the opening in the chassis. If the cable you are replacing is in the CMIOU chassis, remove the cable through the ■ opening between the fan modules and the CMIOU area (panel 1). If the cable you are replacing is in the switch chassis, remove the access panel on the left ■...
Page 232: Install The Front Indicator Panel Cable
“Servicing a CMIOU Chassis Fan Cable Assembly” on page 207. Reinstall the front indicator panel. “Servicing the Front Indicator Panel” on page 217. Reinstall the interconnect assemblies in the front of the affected chassis. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 233 If the SP is configured to synchronize with a network time server using the Network Note - Time Protocol, the Oracle ILOM clock will be reset as soon as the server is powered on and connected to the network. “Install an SP or SPP” on page 136.
Page 234 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 235: Servicing Power Modules
Servicing Power Modules SPARC M8-8 and SPARC M7-8 servers have a single power module that includes six AC power receptacles and the rear indicator panel. For the location of these power modules, see “CMIOU Chassis Rear Components” on page The SPARC M7-16 server has three power modules: one on each CMIOU chassis that each include six AC power receptacles and a rear indicator panel, and one on the switch chassis that includes four AC power receptacles and the rear indicator panel.
Page 236 “Servicing SP Trays” on page 243. Unseat the power supplies in the impacted chassis. “Servicing Power Supplies” on page 165. Remove all PDECBs from the impacted chassis. “Servicing PDECBs” on page 199. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 237 Remove a Power Module Use a T20 driver to loosen two captive screws from the top and bottom of the power module to free it from the chassis. Servicing Power Modules...
Page 238: Install A Power Module
“Install a Power Module” on page 238 ■ Install a Power Module Slide the power module part way in and position it with the slot in the side wall. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 239 Install a Power Module When the module is about 1 inch from being seated in the chassis, locate the spool standoff on the left side and the slot in the chassis side wall. Press the side wall while sliding the module into the chassis to ensure that the spool slides properly into the chassis slot.
Page 240 “Servicing SP Trays” on page 243 “Servicing SPs” on page 117. Reinstall the following components. “Servicing CMIOUs” on page 83 “Servicing Switch Units” on page 189. Return the server to operation. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 241 Verify that the fault has been cleared and the replaced component is operational. “Verify That a Fault Has Been Cleared” on page Return the replaced component to Oracle. “Prepare a Component for Return to Oracle” on page Related Information “Remove a Power Module” on page 235 ■...
Page 242 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 243: Servicing Sp Trays
Servicing SP Trays There is one SP tray in each chassis. Each SP tray contains two SPs or SPPs. For the location of the SP trays, see “CMIOU Chassis Rear Components” on page 21 “Switch Chassis Rear Components (SPARC M7-16)” on page This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge.
Page 244 Using a flat-blade screwdriver, press up on the small metal latch inside the recess of the tray to release the tray from the chassis. Repeat this step for the second latch. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 245 Unseat or Remove an SP Tray Unseat the SP tray. Disengage the SP tray (panel 1). Open and squeeze the ejector latches together. Carefully slide the tray out towards you and press the levers back together, toward the center of the SP tray (panel 2). This will keep the levers from getting damaged while you are servicing the server.
Page 246: Reseat Or Install An Sp Tray
Return the replaced component to Oracle. “Prepare a Component for Return to Oracle” on page Related Information “Unseat or Remove an SP Tray” on page 243 ■ “Servicing SPs” on page 117 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 247: Servicing Batteries
Servicing Batteries A single battery is installed in each SP and SPP in the server. This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge. This discharge can cause server components to fail. Follow antistatic precautions to prevent electrostatic damage.
Page 248 “Servicing SPs” on page 117. Release the thumb screw and slide the top cover of the SP to the rear. Lift the top cover off of the SP. Locate the system battery. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 249 Remove the Battery Description Battery location Using a flat-blade screwdriver, lift the battery out of the holder. Dispose of the battery properly in accordance with local regulations. Note - Related Information “Install the Battery” on page 250 ■ Servicing Batteries...
Page 250: Install The Battery
136. If the SP is configured to synchronize with a network time server using the Network Time Protocol, the Oracle ILOM clock will be reset as soon as the server is powered on and connected to the network. Verify that the fault has been cleared and the replaced component is operational.
Page 251 Install the Battery “Verify That a Fault Has Been Cleared” on page Related Information “Remove the Battery” on page 247 ■ Oracle ILOM documentation (http://www.oracle.com/goto/ilom/docs) ■ Servicing Batteries...
Page 252 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 253: Servicing Pdus
Servicing PDUs There are two PDUs in the rack that contains the server. From the rear of the server, PDU-A is on the left side and PDU-B is on the right side. This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge.
Page 254: Prepare A Pdu For Removal
PDU power cords receiving power from a second power grid. For example, in a SPARC M8-8 or SPARC M7-8 server, AC inputs labeled AC0, AC2, and AC4 are connected to one PDU, and AC inputs labeled AC1, AC3, and AC5 are connected to the other PDU.
Page 255 Prepare a PDU for Removal Confirm that all PDU circuit breakers on the replacement PDU are switched off. Description Circuit breaker switch lies flat in the On position. Circuit breaker switch is raised in the Off position. Related Information “Remove a PDU” on page 256 ■...
Page 256: Remove A Pdu
■ If two SPARC M8-8 or SPARC M7-8 servers are installed: L2, L1, L0; L5, L4, L3 ■ If three SPARC M8-8 or SPARC M7-8 servers are installed: L2, L1, L0; L5, L4, L3; L8, ■ L7,L6 For SPARC M7-16 servers, right PDU (PDU-B): ■...
Page 257 AC input power cords at the SPARC M8-8 or SPARC M7-8 AC inlets. First, unplug the power cords at the even numbered AC inlets 0, 2, 4. Next, unplug the power cords at the odd numbered AC inlets 1, 3, 5.
Page 258 Remove a PDU On an Oracle Rack Cabinet 1242, disconnect the two grounding straps ■ connected at the top and the bottom of the PDU. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 259 Remove a PDU On a Sun Rack II, disconnect the grounding strap connecting the top of the ■ faulty PDU to the rack. Remove the screws and washers securing the PDU to the mounting brackets. Servicing PDUs...
Page 260 Remove a PDU On an Oracle Rack Cabinet 1242, use a T25 wrench key to remove the four ■ M5 screws securing the faulty PDU to the mounting brackets. ■ On a Sun Rack II, use a T25 wrench key to remove the four M5 screws and washers securing the faulty PDU to the mounting brackets.
Page 261 Remove a PDU These screws secure the PDUs for shipping purposes. You might have already removed these screws when you installed the rack at the installation site. Carefully lift the faulty PDU up and off the mounting brackets. Remove the PDU from the rack and place it on a clean work table. Related Information “Prepare a PDU for Removal”...
Page 262: Install A Pdu
PDU's standoff bolts into the top and bottom bracket keyhole slots. Installing the PDU into an Oracle Rack Cabinet 1242: SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 263 Install a PDU Installing the PDU into a Sun Rack II: You need two people to lift and secure the PDU to the rack. Caution - The PDU is held in the rack by gravity, with the standoff bolts resting in the mounting brackets' keyhole slots.
Page 264 (two screws and washers per bracket). If you plan to ship the rack to another location, you must secure the PDU using these shipping screws. On an Oracle Rack Cabinet 1242, use a T25 Torx wrench and four M5 ■...
Page 265 Install a PDU On a Sun Rack II, use a T25 Torx wrench and four M5 shipping screws and ■ washers to secure the replacement PDU to the mounting brackets. Route the power input lead cords between the rear RETMA rail and side panel. The replacement PDU has three power input lead cords, which you must route between the side panel and the rear RETMA rail.
Page 266 ■ If two SPARC M8-8 or SPARC M7-8 servers are installed: L2, L1, L0; L5, L4, L3 ■ If three SPARC M8-8 or SPARC M7-8 servers are installed: L2, L1, L0; L5, L4, L3; L8, ■ L7,L6 For a SPARC M7-16 server, right PDU (PDU-B): ■...
Page 267 Restart the server. -> start /System Return the replaced component to Oracle. “Prepare a Component for Return to Oracle” on page Related Information “Remove a PDU” on page 256 ■ “Switch on PDU Circuit Breakers” on page 73 ■...
Page 268 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 269: Servicing The Cmiou Chassis
Servicing the CMIOU Chassis SPARC M8-8 and SPARC M7-8 servers each have a single CMIOU chassis. For the location of the chassis, see “Server Front Components (SPARC M8-8 and SPARC M7-8 Servers)” on page SPARC M7-16 servers have two CMIOU chassis and a single switch chassis. For the locations of these chassis, see “Server Front Components (SPARC M7-16)”...
Page 270 (SPARC M8-8 and SPARC M7-8) “Servicing External Interconnect Assemblies” on page 305 (SPARC M7-16) “Servicing SP Internal Interconnect Assemblies” on page 285 Fan modules “Servicing Fan Modules (CMIOU Chassis)” on page 179 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 271 From the front of the server, stabilize the rack using all anti-tilt mechanisms provided. The following steps describe how to stabilize the Sun Rack II 1242 and the Oracle Rack Cabinet 1242. If you installed a stand-alone server in your own rack, refer to the rack documentation for instructions on stabilizing the rack.
Page 272 Remove a CMIOU Chassis On a Sun Rack II, deploy the anti-tilt legs. ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 273 (panel 2). Repeat these steps for the second leg. Both legs must be deployed. On an Oracle Rack Cabinet 1242, extend the anti-tilt bar. ■ Refer to the label inside the rack for instructions on extending the anti-tilt bar. Pull the anti-tilt bar out of the rack to the fully extended position.
Page 274 Remove the chassis from the rack, and place it on an appropriate surface. Related Information “Install a CMIOU Chassis” on page 275 ■ “Servicing the Switch Chassis” on page 277 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 275: Install A Cmiou Chassis
Install a CMIOU Chassis Install a CMIOU Chassis If you have not already done so, remove the faulty CMIOU chassis. “Remove a CMIOU Chassis” on page 269. Determine your next step: If you have another person to assist you, go to Step ■...
Page 276 Verify that the fault has been cleared and the replaced component is operational. “Verify That a Fault Has Been Cleared” on page Related Information “Remove a CMIOU Chassis” on page 269 ■ “Servicing the Switch Chassis” on page 277 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 277: Servicing The Switch Chassis
Servicing the Switch Chassis There is one switch chassis in the SPARC M7-16 server. For the location of the switch chassis, “Server Front Components (SPARC M7-16)” on page 18 “Server Rear Components (SPARC M7-16)” on page This procedure requires that you handle components that are sensitive to electrostatic Caution - discharge.
Page 278 At the front of the server, remove all of these components: Component Link External interconnects “Servicing External Interconnect Assemblies” on page 305 SP interconnect “Servicing SP Internal Interconnect Assemblies” on page 285 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 279 From the front of the server, stabilize the rack using all anti-tilt mechanisms provided. The following steps describe how to stabilize the Sun Rack II 1242 and the Oracle Rack Cabinet 1242. If you installed a stand-alone server in your own rack, refer to the rack documentation for instructions on stabilizing the rack.
Page 280 Remove a Switch Chassis On a Sun Rack II, deploy the anti-tilt legs. ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 281 (panel 2). Repeat these steps for the second leg. Both legs must be deployed. On an Oracle Rack Cabinet 1242, extend the anti-tilt bar. ■ Pull the anti-tilt bar out of the rack to the fully extended position. Rotate the foot of the ant-tilt bar so that it is perpendicular to the floor.
Page 282 Remove the chassis from the rack, and place it on an appropriate surface. Related Information “Install a Switch Chassis” on page 283 ■ “Servicing the CMIOU Chassis” on page 269 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 283: Install A Switch Chassis
Install a Switch Chassis Install a Switch Chassis If you have not already done so, remove the faulty switch chassis. “Remove a Switch Chassis” on page 277. Determine your next step: If you have another person to assist you, go to Step ■...
Page 284 Verify that the fault has been cleared and the replaced component is operational. “Verify That a Fault Has Been Cleared” on page Related Information “Remove a Switch Chassis” on page 277 ■ “Servicing the CMIOU Chassis” on page 269 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 285: Servicing Sp Internal Interconnect Assemblies
Servicing SP Internal Interconnect Assemblies There is one SP internal interconnect assembly in each CMIOU chassis. These assemblies connect the CMIOUs to the SPs. There is one SP internal interconnect assembly in the switch chassis. This assembly connects switch units to the SPs. For the location of the SP internal interconnect assemblies, see “CMIOU Chassis Front Components”...
Page 286: Remove An Sp Internal Interconnect Assembly
“Servicing Internal Interconnect Assemblies” on page 297 (SPARC M8-8 and SPARC M7-8) or “Servicing External Interconnect Assemblies” on page 305 (SPARC M7-16). From the front of the server, remove the SP internal interconnect assembly. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 287 Remove an SP Internal Interconnect Assembly Loosen the captive screws on the SP internal interconnect assembly to free it from the chassis. Servicing SP Internal Interconnect Assemblies...
Page 288 If you are using a screwdriver that has interchangeable bits, the bit needs to be 2 inches long because the socket on the driver does not fit through the square holes. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 289 Remove an SP Internal Interconnect Assembly Carefully slide the SP internal interconnect assembly out of the chassis. Install the new SP internal interconnect assembly. “Install an SP Internal Interconnect Assembly” on page 290. Related Information “Install an SP Internal Interconnect Assembly” on page 290 ■...
Page 290: Install An Sp Internal Interconnect Assembly
If you notice that these springs or the connectors are damaged, do not install the interconnect into the chassis. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 291 Install an SP Internal Interconnect Assembly Insert the guide pin shield to avoid damaging the connector pins on the rear of the SP internal interconnect assembly. If you are installing an SP internal interconnect assembly in the CMIOU chassis, place the guide pin shield as shown in panel 1. If you are installing an SP internal interconnect assembly in the switch chassis, place the guide pin shield as shown in panel 2.
Page 292 Align the assembly in the chassis and install the assembly slowly to avoid bending or otherwise damaging them. Secure the SP internal interconnect assembly to the chassis. Tighten the captive screws on the face of the assembly. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 293 Install an SP Internal Interconnect Assembly Use a T20 driver to install the screws in the white, horizontal strip that extends to the left of the SP internal interconnect assembly. Servicing SP Internal Interconnect Assemblies...
Page 294 “Servicing External Interconnect Assemblies” on page 305. Reseat all CMIOUs or switch units from the impacted chassis. “Reseat a CMIOU” on page 101 “Reseat a Switch Unit” on page 196. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 295 Install an SP Internal Interconnect Assembly From the rear of the server, reinstall the SP tray and the SPs. “Servicing SP Trays” on page 243 “Servicing SPs” on page 117. Switch on the appropriate PDU circuit breakers and power on the server. “Returning the Server to Operation”...
Page 296 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 297: Servicing Internal Interconnect Assemblies
Servicing Internal Interconnect Assemblies In SPARC M8-8 and SPARC M7-8 servers, five internal interconnect assemblies in the CMIOU chassis connect the CMIOUs to each other. For the location of the internal interconnect assemblies, see “CMIOU Chassis Front Components” on page...
Page 298: Internal Interconnect Assembly Configuration
This figure shows the internal interconnect assembly slot labels and positions. Description SPARC M8-8 and SPARC M7-8 server (two PDomain) slot labels SPARC M8-8 and SPARC M7-8 server (one PDomain) slot labels SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 299: Remove An Internal Interconnect Assembly
Remove an Internal Interconnect Assembly Related Information “Remove an Internal Interconnect Assembly” on page 299 ■ “Install an Internal Interconnect Assembly” on page 302 ■ Remove an Internal Interconnect Assembly This procedure can be performed only by qualified service personnel. Remove AC power using the circuit breakers on the appropriate PDU before performing this procedure.
Page 300 Remove an Internal Interconnect Assembly When the spring-loaded screw springs out, press on the upper screw to unlatch the assembly from the chassis. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 301 Remove an Internal Interconnect Assembly Grasp the assembly and slide it out of the chassis using either the screws or center finger holds. Install the new internal interconnect assembly. “Install an Internal Interconnect Assembly” on page 302. Related Information “Install an Internal Interconnect Assembly” on page 302 ■...
Page 302: Install An Internal Interconnect Assembly
Guide the TOP end of the assembly into place first, then guide the BOTTOM end into place. Push the TOP end in until the safety latch clicks, then push in the BOTTOM end in. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 303 Install an Internal Interconnect Assembly The small connector pins on the back of the assembly are susceptible to damage. It Caution - is important that you align the assembly in the chassis and install the assembly slowly to avoid bending or otherwise damaging the connector pins. Secure the internal interconnect assembly by hand tightening the three captive screws while ensuring that the screws install smoothly into the chassis.
Page 304 Verify that the fault has been cleared and the replaced component is operational. “Verify That a Fault Has Been Cleared” on page Related Information “Remove an Internal Interconnect Assembly” on page 299 ■ “Internal Interconnect Assembly Configuration” on page 298 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 305: Servicing External Interconnect Assemblies
Servicing External Interconnect Assemblies In Oracle's SPARC M7-16 server, eight external interconnect assemblies connect the CMIOUs in the top chassis and the CMIOUs in the bottom chassis to the switch units in the switch chassis. One half of an assembly is installed in the switch chassis, and the other half of the assembly is installed in a CMIOU chassis.
Page 306: External Interconnect Assembly Configuration
External Interconnect Assembly Configuration External Interconnect Assembly Configuration This figure shows the external interconnect assembly slot labels and positions. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 307: Remove An External Interconnect Assembly
Remove an External Interconnect Assembly Related Information “Remove an External Interconnect Assembly” on page 307 ■ “Install an External Interconnect Assembly” on page 311 ■ Remove an External Interconnect Assembly This procedure can be performed only by qualified service personnel. Remove AC power using the circuit breakers on the appropriate PDU before performing this procedure.
Page 308 Remove the assembly slowly to avoid bending or otherwise damaging them. In addition, take care to avoid flexing or twisting these assemblies. Loosen the captive screws on the assembly. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 309 Remove an External Interconnect Assembly When the spring-loaded screw springs out, press on the upper screw at each end of the external interconnect to unlatch the assembly from the chassis. Servicing External Interconnect Assemblies...
Page 310 Install the new external interconnect assembly. “Install an External Interconnect Assembly” on page 311. Related Information “Install an External Interconnect Assembly” on page 311 ■ “External Interconnect Assembly Configuration” on page 306 ■ SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 311: Install An External Interconnect Assembly
Install an External Interconnect Assembly Install an External Interconnect Assembly You must return interconnect assemblies to their original slots. If you are inserting Caution - one or more previously installed assemblies to the server (for example, because you removed them to gain access to other components), you must return them to their original locations in the server.
Page 312 Guide the TOP end of the assembly into place first, then guide the BOTTOM end into place. Push the TOP end in until the safety latch clicks, then push in the BOTTOM end in. SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 313 Install an External Interconnect Assembly The small connector pins on the back of the assembly are susceptible to damage. Caution - Align the assembly in the chassis, and install the assembly slowly to avoid bending or otherwise damaging the connector pins. In addition, take care to avoid flexing or twisting these assemblies.
Page 314 “Returning the Server to Operation” on page Verify that the fault has been cleared and the replaced component is operational. “Verify That a Fault Has Been Cleared” on page SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 315 Install an External Interconnect Assembly Related Information “Remove an External Interconnect Assembly” on page 307 ■ “External Interconnect Assembly Configuration” on page 306 ■ Servicing External Interconnect Assemblies...
Page 316 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 317: Glossary
Glossary Active SP An SP selected by Oracle ILOM to manage server resources. When an Active SP can no longer serve this role, the Standby SP assumes its role. See also Standby Auto Service Request. Oracle software that provides the ability to notify Oracle Support automatically.
Page 318 The SPARC M7-16 server has four DCUs that you can assign to the host in any one of the four possible PDomains. See also PDomain. DCU SPM In SPARC M8 and SPARC M7 servers, Oracle ILOM identifies one SPM from an SPM pair to manage DCU activity. See also SPM, SPP, and DCU. DHCP Dynamic Host Configuration Protocol.
Page 319 InfiniBand A networking communications standard that features very high throughput and very low latency. I/O hub. IPMP IP network multipathing. An Oracle Solaris feature providing multipathing and load-balancing capabilities for IP network interfaces. IPoIB Internet protocol over InfiniBand. iSCSI Internet small computer system interface. An IP-based storage networking standard that enables a server to access storage across a network.
Page 320 Non-Volatile Memory Express. A specification for accessing solid-state drives attached through the PCIe bus. OpenBoot Oracle firmware that enables a PDomain to boot the Oracle Solaris OS. Provides an interface for testing hardware and software interactively. Oracle ILOM Oracle Integrated Lights Out Manager. The system management firmware that is preinstalled on the server SPs.
Page 321 The lead SPM of a PDomain. The PDomain SPM manages tasks and provides rKVMS service for that PDomain. In a SPARC M7-16 server, Oracle ILOM identifies one of the DCU SPMs from the pool of DCU SPMs on the same PDomain as the PDomain SPM to manage activity on that host.
Page 322 A device that enables the CMIOUs to communicate with each other. The SPARC M7-16 server switch contains six switch units. system In the SPARC M8 and SPARC M7 servers documentation, system refers to the /System level in the Oracle ILOM firmware. Torx A type of screw head characterized by a 6-point star-shaped pattern.
Page 323: Index
101 identifying, 123 returning to Oracle, 98 antistatic wrist strap, 62 SDM name, 32 unseating, 93 CMPs, name returned by Oracle ILOM, 40 CMs, name returned by Oracle ILOM, 40 batteries commands installing, 250 Oracle ILOM reference, 76 location, 247...
Page 324 212 failures, 109 removing, 207 Fault Remind button, 109 fan modules fault tolerance, 42 fault tolerance, 42 general error message, 112 installing installing, 114 in CMIOU chassis, 182 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 325 CMIOU chassis, 180 in switch chassis, 184 locating in CMIOU chassis, 20 card configurations in switch chassis, 25 restoring, 161 name returned by Oracle ILOM, 40 saving, 151 removing installation guidelines in CMIOU chassis, 180 DIMMs, 106 in switch chassis, 184...
Page 326 50 eUSB disks, 27 message identifier, 36 external interconnect assemblies, 23 fan modules in CMIOU chassis, 20 in switch chassis, 25 faults NAC path reference, 40 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 327 Oracle Solaris powering off, 70 clearing faults, 55 powering on, 74 static, described, 80 PDU circuit breakers switching off, 71 packing components for return to Oracle, 64 switching on, 73 PCIe card carriers PDUs installing, 159 AC connections, 253 LEDs, 149...
Page 328 SPs, 125 SP trays, 243 switch units, 191 SPPs, 132 SPs, 132 switch chassis, 277 switch units, 194 rear components requested_config_state property, 39 SPARC M7-16, 19 reseating SPARC M7-8, 17 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...
Page 329 243 returning components to service, 75 SPDBs root complexes, BUS names to assign, 143 name returned by Oracle ILOM, 42 SPMs name returned by Oracle ILOM, 42 paths supplied to Oracle ILOM, 122 safety information and symbols, 57...
Page 330 /var/adm/messages file, 51 verifying fault removal, 54 PCIe card functionality, 164 PCIe cards are offline, 131 SPM removal preparedness, 125 viewing system message log files, 51 wrist strap, installing, 62 SPARC M8 and SPARC M7 Servers Service Manual • September 2017...

This manual is also suitable for:

Sparc m7

Oracle SPARC M8 Service Manual

Using this Documentation

Identifying Components

Detecting and Managing Faults

Preparing for Service

Controlling the Server and Domains

Understanding Server Configurations and Failover Behavior

Servicing Cmious

Servicing Dimms

Servicing Sps

Servicing Pcie Cards

Servicing Power Supplies

Servicing Fan Modules (CMIOU Chassis)

Servicing Fan Modules (Switch Chassis)

Servicing Switch Units

Servicing Pdecbs

Servicing Eusb Disks

Servicing a CMIOU Chassis Fan Cable Assembly

Servicing the Front Indicator Panel

Servicing the Front Indicator Panel Cable

Servicing Power Modules

Servicing SP Trays

Servicing Batteries

Servicing Pdus

Servicing the CMIOU Chassis

Servicing the Switch Chassis

Servicing SP Internal Interconnect Assemblies

Servicing Internal Interconnect Assemblies

Servicing External Interconnect Assemblies

Glossary

Index

Quick Links

Need help?

Questions and answers

Related Manuals for Oracle SPARC M8

Summary of Contents for Oracle SPARC M8