Download
Share
Print this page
  1. Manuals
  2. Brands
  3. Oracle Manuals
  4. Server
  5. Exadata Server X11M
  6. Service manual

Oracle Exadata Server X11M Service Manual

Hide thumbs

Advertisement

Quick Links

Download this manual
Exadata Server X11M Service Manual
G18152-02
January 2025

Advertisement

loading
Need help?

Need help?

Do you have a question about the Exadata Server X11M and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Oracle Exadata Server X11M

Summary of Contents for Oracle Exadata Server X11M

  • Page 1 Exadata Server X11M Service Manual G18152-02 January 2025...
  • Page 2 The terms governing the U.S. Government's use of Oracle cloud services are defined by the applicable contract for such services. No other rights are granted to the U.S. Government.
  • Page 3 1-11 Troubleshooting and Diagnostics Diagnosing Server Component Hardware Faults Troubleshoot Hardware Faults Using Oracle ILOM CLI Troubleshoot Hardware Faults Using Oracle ILOM Web Interface Troubleshoot Power Issues Troubleshoot System Cooling Issues Troubleshoot With Diagnostic Tools Attaching Devices to the Server...
  • Page 4 Power Down the Server Gracefully Using the On/Standby Button Power Down the Server for Immediate Shutdown Using Oracle ILOM CLI Power Down the Server for Immediate Shutdown Using Oracle ILOM Web Interface Power Down the Server for Immediate Shutdown Using the On/Standby Button...
  • Page 5 Servicing Power Supplies Power Supply Overview Remove a Power Supply Install a Power Supply Servicing Fan Modules Remove a Fan Module Install a Fan Module Remove the Fan Tray Install the Fan Tray Servicing Internal M.2 Flash SSDs Remove a Flash Riser Board Install a Flash Riser Board Identify and Remove an M.2 Flash SSD Install an M.2 Flash SSD...
  • Page 6 Servicing PCIe Cards PCIe Slot Locations 11-1 Remove a Half Height PCIe Card 11-2 Install a Half Height PCIe Card 11-3 Remove a Full Height PCIe Card 11-5 Install a Full Height PCIe Card 11-6 Remove a PCIe Riser 11-8 Install a PCIe Riser 11-9 Configure PCIe Electrical Interfaces Using the PCIe Flyover Cable...
  • Page 7 Returning the Server to Operation Server Filler Panel Requirements 17-1 Remove and Install Filler Panels 17-1 Install the Server Top Cover 17-2 Remove Antistatic Measures 17-3 Reinstall the Server Into the Rack 17-3 Return the Server to the Normal Rack Position 17-4 Reconnect Power and Data Cables 17-5...
  • Page 8 480GB NVMe M.2 Solid State Drive Specification 22TB Hard Disk Drive Specification B-12 Server Installation Information Receiving and Unpacking Requirements Maintenance Space Requirements Rack Space Requirements Rack Compatibility Installation Procedure Installation Prerequisites Safety Precautions ESD Precautions Tools and Equipment For Installation Shipping Inventory Rackmounting the Server Rackmount Kit Contents...
  • Page 9 System Firmware Components Index...
  • Page 10 • Required knowledge – Advanced experience troubleshooting and replacing hardware. Product Documentation Library Documentation and resources for this Exadata Server X11M product and related products are available at Oracle Help Center.
  • Page 11 About Exadata Server X11M This section includes the server component overview. The Exadata Server X11M is a (1) or (2) processor socket, AMD® EPYC E6 Turin™ SP5 processor-based, 2U rackmount server. The X11M is available with either a Database Server [1] that supports up to four high performance NVMe drives or a Storage Server [2] that supports up to twelve high capacity SAS-3 drives.
  • Page 12 12 DDR5 32-GB RDIMMs 6400 MT/sec 384 G Storage drive configuration can comprise either solid state drives (SSDs) or hard disk drives (HDDs). Exadata Database Server Storage Exadata Server X11M Exadata Storage Server front panel storage drive bays can be provisioned Drives with: •...
  • Page 13 Configuration Exadata Database Server X11M 9-slot PCIe card back panel: Exadata Server X11M supports up to nine half-height PCIe cards. Up to nine PCIe Gen5 vertical card slots are supported when the low profile Add In Card (AIC) chassis is utilized.
  • Page 14 Setup Utility, you can configure platform-specific information, and some of the run-time and boot-time characteristics of your system using a simple graphical interface. If you are managing multiple servers, you can use Oracle Enterprise Manager Ops Center. For Oracle Enterprise Manager Ops Center 12c Release information about Ops Center, refer to Connected to x2 PCIe Gen5 links for boot drives.
  • Page 15 Chapter 1 Front Panel Components SAS Storage Drive Configuration Exadata Server X11M supports up to twelve 3.5-inch hot-pluggable SAS-3 hard disk drives (HDDs) on the front panel. Callout Description Product Serial Number (PSN) label and Radio Frequency Identification (RFID) tag...
  • Page 16 Using PCIe risers, the full height PCIe configuration supports up to four full height and two half height PCIe cards. 9-slot PCIe Card Configuration Exadata Server X11M, in 9-slot PCIe card back panel half height PCIe configuration, supports up to nine vertically mounted half height PCIe cards.
  • Page 17 Processor. SP: Reset service processor (SP). See NET 0: Network (NET) 1GbE 100/1000BASE-T RJ-45 Gigabit Ethernet (GbE) port NET MGT: Oracle Integrated Lights Out Manager (ILOM) service processor (SP) network management (NET MGT) 1GbE RJ-45 100/1000BASE-T port USB 3.1 connector System status LEDs SYSTEM: Fault-Service Required: amber;...
  • Page 18 • System-Level Status Indicators Illustrated Parts Breakdown Identify major components of Exadata Server X11M with Database Server and Exadata Server X11M with Storage Server. Exadata Server X11M with Database Server The following figure identifies Exadata Server X11M major components with Database Server.
  • Page 19 Two power supplies (PS0, PS1) (Optional) Oracle RoT card (Optional) Nine HHHL PCIe cards PCIe slots 1 through 4 are nonfunctional in single-processor systems. Exadata Server X11M with Storage Server The following figure identifies Exadata Server X11M major components with Storage Server.
  • Page 20 Chapter 1 Illustrated Parts Breakdown Callout Description Storage drives, up to twelve 3.5-inch hot-pluggable SAS-3 LFF HDDs in Storage Server Four storage drive bay cages Storage Server (DBP) System chassis Front LED indicator module and temperature sensor (FIM) Six fan modules (FM0-5) Fan tray Motherboard assembly (MB) Processors (P0, P1) and heatsinks (Single-processor systems contain only a single...
  • Page 21 Callout Description Air baffle System RTC battery PCIe card shroud (9-slot PCIe card back panel) (Optional) Oracle RoT card Nine HHHL PCIe cards PCIe slots 1 through 4 are nonfunctional in single-processor systems. Two power supplies (PS0, PS1) Replaceable Components This table lists server replaceable components and provides associated links to service replacement instructions.
  • Page 22 Provides SAS-3 signals between card located in PCIe slots and the disk backplane. Cold service Oracle Flash Accelerator F680 PCIe Card: 6.8 TB, NVMe PCIe 5.0 in Exadata Database Server X11M, Exadata Storage Server X11M, and Exadata Control Plane Server X11M 1-12...
  • Page 23 When a server hardware fault event occurs, the system lights the Fault-Service Required LED and captures the event in the Oracle ILOM event log. If you set up notifications through Oracle ILOM, you also receive an alert through the notification method you choose. When you become aware of a hardware fault, address it immediately.
  • Page 24 Service. Clear the fault in Oracle ILOM. Depending on the component, you might need to clear the fault in Oracle ILOM. Generally, components that have a FRU ID, clear the fault automatically. For details, refer to Oracle Integrated Lights Out Manager (ILOM) documentation at Oracle ILOM Documentation.
  • Page 25 To prepare the server for service, see Preparing for Service. Service the component. After servicing the component, you might need to clear the fault in Oracle ILOM. For more information, refer the service procedures for the component. See Monitoring Component...
  • Page 26 Networking PCI_Devices Firmware In the Status section of the Oracle ILOM Summary Information page, identify the server subsystem that requires service. For example, if a hardware component in the subsystem is in a fault state, the Status column notes the status as Service Required.
  • Page 27 Chapter 2 Diagnosing Server Component Hardware Faults After servicing the component, you might need to clear the fault in Oracle ILOM. For more information, refer to the service procedure for the component. For details, refer to Oracle Integrated Lights Out Manager (ILOM) documentation at Oracle ILOM Documentation.
  • Page 28 Chapter 2 Diagnosing Server Component Hardware Faults Table 2-1 (Cont.) Server Power Issues Power Description Action Prevention Issue Power The server power supplies (PS0, PS1) provide Verify that the AC cables are When a power supply fails, Supplies the necessary server voltages from the AC connected to both power replace it immediately.
  • Page 29 Chapter 2 Diagnosing Server Component Hardware Faults Table 2-2 (Cont.) Server Cooling Issues Cooling Issue Description Action Prevention Airflow The server cooling system uses fans to pull cool Inspect the server front Periodically inspect and Blockage air in from the server front intake vents and and back panel vents for clean the server vents exhaust warm air out the server back panel vents.
  • Page 30 Diagnosing Server Component Hardware Faults selection of diagnostic tools available for your server range in complexity from a comprehensive validation test suite (Oracle VTS) to a chronological event log (Oracle ILOM event Log). The selection of diagnostic tools also includes standalone software packages, firmware-based tests, and hardware-based LED indicators.
  • Page 31 Note: The serial management port does not support network connections. Attach Ethernet access to the SP through a dedicated management port (NET MGT). To connect to the Oracle ILOM service processor over the network remotely, connect an Ethernet cable to the...
  • Page 32 By default, the service processor (SP) controls the serial management (SER MGT) port and uses it to redirect the host serial console output. Using Oracle ILOM, you can assign the host console (COM1) as owner of the SER MGT port output, which allows the host console to output information directly to the SER MGT port.
  • Page 33 Connect a serial host to the server. Oracle ILOM web interface Log in to the SP Oracle ILOM web interface. Open a web browser and direct it using the IP address of the server SP. Log in as root or a user with administrator privileges.
  • Page 34 The serial management connector, labeled SER MGT, is an RJ-45 connector that can be accessed from the back panel. This port is the default connection to the server Oracle ILOM SP. Use only the SER MGT port for server management. Refer to Oracle ILOM Documentation.
  • Page 35 Chapter 2 Attaching Devices to the Server Table 2-5 Serial Management Port Signals Signal Description Signal Description Request to Send Ground Data Terminal Ready Receive Data Transmit Data Data Set Ready Ground Clear to Send Table 2-6 Default Serial Connections for Serial Port Parameter Setting Connector...
  • Page 36 SP reset button. If the Oracle ILOM SP stops running and you cannot reset it using the Oracle ILOM web interface or the Oracle ILOM CLI, use the following procedure to reset the SP from the server back panel.
  • Page 37 Once the SP has rebooted successfully, you will be able to log into Oracle ILOM. After the SP successfully boots, the LED will remain steady on. You can confirm that the SP is working by logging into Oracle ILOM for that system. For details, refer to Oracle ILOM Documentation.
  • Page 38 Accessing Oracle ILOM Prerequisites: You need to know the IP address or host name of the service processor (SP) to log in to Oracle ILOM CLI or web interface remotely using one of the network ports on the server. Note: To enable first time login and access to Oracle ILOM, a default Administrator account and its password are provided with the system.
  • Page 39 If the serial console is in use, stop and restart it using the stop /HOST/ console command followed by the start /HOST/console command. To return to the Oracle ILOM console, press Esc and then press Shift 9 to enter the open parenthesis ( character.
  • Page 40 Standby power is required for SP access, Main power and a running host is not required. Access Oracle ILOM. If this is the first time you are connecting to Oracle ILOM, do one of the following: • Access Oracle ILOM using a local serial connection to the command-line interface: From the terminal device connected to the server's SER MGT port, press Enter to obtain a prompt.
  • Page 41 Reset the Server Using Oracle ILOM For more details, see Controlling System Power and Oracle ILOM Documentation. Log in to the Oracle ILOM web interface or command-line interface (CLI) using an account with admin (a) role privileges. Reset the server: •...
  • Page 42 • From the Oracle ILOM CLI – Type exit at the CLI prompt. • Oracle ILOM web interface – Click the Log Out button in the top-right corner of the screen. Monitoring Component Health and Faults Using Oracle ILOM Oracle ILOM interfaces provide easy-to-view information about the health status of system components.
  • Page 43 Contacting Support My Oracle Support If the troubleshooting procedures in this chapter do not solve your problem, use the following table to collect information that you might need to communicate to Oracle Support. Table 2-9 System Configuration Information System Configuration...
  • Page 44 Oracle Auto Service Requests (ASR) is available at no additional cost to customers with Oracle Premier Support. Oracle ASR is the fastest way to restore system availability if a hardware fault occurs. Oracle ASR software is secure and customer installable, with the...
  • Page 45 For your protection, observe the following safety precautions when setting up your equipment: • Follow all standard cautions, warnings, and instructions marked on the equipment and described in the Oracle Server Safety and Compliance Guide and Important Safety Information for Oracle's Hardware Systems. •...
  • Page 46 FRU Key Identity Properties (KIP) Automated Update Oracle ILOM includes a key identity properties (KIP) auto-update feature that ensures product information that is used for service entitlement and warranty coverage is accurately maintained by the server at all times, including during hardware replacement activities.
  • Page 47 Power supply (PS), designated as a backup quorum member. When a server FRU that contains the KIP is removed and a replacement component is installed, the KIP of the replacement component is programmed by Oracle ILOM to contain the same KIP as the other two components.
  • Page 48 Notify affected users that the server will be powered down. Save any open files, and quit all running applications. Refer to your application documentation for specific information about these processes. Log in to the Oracle ILOM command-line interface (CLI) using an Administrator account. Accessing Oracle ILOM.
  • Page 49 Preparing the Server for Component Replacement Caution: When you power down the server using Oracle ILOM, the server enters Standby power mode. Power is still directed to the service processor remote management subsystem and power supply fans. To completely power off the server, you must disconnect the power cords from the power supplies.
  • Page 50 An immediate power-off might corrupt system data, therefore, use this procedure to power down the server only after attempting the graceful power down procedure. Log in to the Oracle ILOM command-line interface (CLI) using an Administrator account. Accessing Oracle ILOM.
  • Page 51 Server. Caution: When you power down the server using Oracle ILOM, the server enters Standby power mode. Power is still directed to the service processor and power supply fans. To completely power off the server, you must disconnect the power cords from the power supplies.
  • Page 52 To prevent the rack from tipping forward when the server is extended, extend all rack anti- tilt mechanisms. Stabilize the Rack. Also refer to the Oracle Rack Cabinet 1242 User's Guide at Oracle Rack Cabinet 1242 Documentation Library.
  • Page 53 Chapter 3 Preparing the Server for Component Replacement From the front of the server, open and hold the left and right release latch covers in the open position. When in an open position, the release latch covers engage the slide-rail release latches [1].
  • Page 54 You can use the following items as an antistatic mat: • Antistatic bag used to wrap a replacement part • Oracle ESD mat (orderable item) • Disposable ESD mat (shipped with some replacement parts or optional system components) Attach an antistatic wrist strap.
  • Page 55 Chapter 3 Preparing the Server for Component Replacement Note: Fan modules and M.2 flash SSDs are hot-pluggable and do not require AC power cables to be disconnected from the server power supplies. If the release button latch is in the locked position, use a Torx T15 screwdriver to turn the release button latch clockwise to the unlocked position.
  • Page 56 Servicing Storage Drives This section describes how to service storage drives. Storage drives are replaceable components that do not require you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable Components. Note: The procedures and illustrations in this chapter apply to both NVMe and SAS storage drives, except where noted.
  • Page 57 If you are removing an NVMe storage drive, see the procedures for your operating system. Note: NVMe storage drives are supported only on servers that are running Oracle Linux, Oracle Solaris, or Microsoft Windows Server. – Removing and Replacing an NVMe Storage Drive Using Oracle Solaris –...
  • Page 58 Chapter 4 Remove a Storage Drive – Removing and Replacing an NVMe Storage Drive Using Microsoft Windows Server Push the latch release button to open the drive latch [1, 2]. Callout Description Pressing the latch release button. Opening the latch. Caution: The latch is not an ejector.
  • Page 59 Chapter 4 Install a Storage Drive • If you are not replacing the drive, install a filler in the empty drive slot to maintain proper airflow and perform administrative tasks to configure the server to operate without the drive. See Remove and Install Filler Panels.
  • Page 60 The following sections describe how to remove and replace an NVMe storage drive on a server that is running the Oracle Linux operating system. Unmount an NVMe Storage Drive Log in to Oracle Linux that is running on the server. Remove the NVMe storage device path. To find the PCIe addresses (Bus Device Function), type:...
  • Page 61 Chapter 4 Removing and Replacing Storage Drives Using an OS /sys/bus/pci/slots/13/address:0000:b4:00 (instance nvme1, pcie slot 13, drive label nvme3) Disconnect all users from the NVMe drive and back up the NVMe drive data, as needed. Use the umount command to unmount any file systems that are mounted on the device.
  • Page 62 Chapter 4 Removing and Replacing Storage Drives Using an OS This command returns output similar to the following: 86:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:0a54] 8d:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:0a54] d9:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:0a54] e0:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:0a54] Install an NVMe Storage Drive Perform this procedure to physically install an NVMe storage drive into the server.
  • Page 63 Servicing Power Supplies This section describes how to service power supplies. Power supplies are replaceable components that do not require you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable Components. The power supplies are located at the back of the server.
  • Page 64 You also can use the Oracle ILOM show faulty command at the Oracle ILOM command-line prompt (->) to identify a power supply failure. To list all known faults in the server, log in to the Oracle ILOM Fault Management Shell from the Oracle ILOM service processor and issue the fmadm faulty command.
  • Page 65 Chapter 5 Install a Power Supply Open the power supply handle if the handle is folded flush. Open the velcro straps if present. Pull the power supply out of the chassis. Caution: When you remove a power supply, be sure to replace it with another power supply within 60 seconds;...
  • Page 66 Fault-Service Required LEDs are not lit on the front and back panels. Note: After you replace Power Supply 0, you must reset the Oracle ILOM service processor (SP) to propagate the key identity properties (KIP) data to the new power supply. For instructions on resetting the SP, refer to the Oracle Integrated...
  • Page 67 Oracle ILOM. Even if only one fan motor is faulted within the fan module, the Oracle ILOM service processor detects that two fan motors have failed to spin while the fan module is removed.
  • Page 68 Chapter 6 Install a Fan Module LED Color and State Meaning Amber – The fan module is faulty. The front Top Fan LED and the front and back panel Fault- Service Required LEDs are also lit if the system detects a fan module fault. Using your forefinger and thumb, lift the fan module straight up and out of the chassis and set it aside on an antistatic mat.
  • Page 69 Chapter 6 Remove the Fan Tray Press down on the fan module to fully seat the fan module. Verify that the fan module status indicator (LED) for the replacement fan module is not illuminated. Install the server top cover. Install the Server Top Cover.
  • Page 70 Chapter 6 Remove the Fan Tray When removing the fan tray, slightly tilt the fan tray to the left to decompress the cables, before the operation. Ensure all motherboard cables are clear of the fan tray and fan tray gasket. Lift the fan tray from the server.
  • Page 71 Chapter 6 Install the Fan Tray Install the Fan Tray Perform this procedure only when servicing Disk backplane, SAS cables, and Motherboard assembly replaceable components: Lower the fan tray into the server. Ensure all motherboard cables are clear of the fan tray and fan tray gasket. When installing the fan tray, the fan tray should first be slightly tilted to the left to compress the cables, before the operation.
  • Page 72 Chapter 6 Install the Fan Tray Install the fan modules into the fan tray. Install a Fan Module. Install the air baffle. Install the Air Baffle Continue with the next step in the service procedure.
  • Page 73 Servicing Internal M.2 Flash SSDs This section describes how to service M.2 flash solid-state drives (SSDs). M.2 flash SSDs are replaceable components that do not require you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable...
  • Page 74 Chapter 7 Install a Flash Riser Board Note: The green Power LED indicator located on the riser board is extinguished when you open the socket levers. Carefully lift the riser board straight up to remove it from the sockets. Place the riser board on an antistatic mat. If required, perform the procedures for removing the M.2 flash SSD from the flash riser board.
  • Page 75 Return the Server to the Normal Rack Position. Identify and Remove an M.2 Flash SSD In the standard Exadata Server X11M configuration, the flash riser board contains an M.2 flash SSD in a socket. Prepare the server for service. Extend the server into maintenance position.
  • Page 76 SSD contacts from the riser board socket [2]. Place the M.2 flash SSD on an antistatic mat. Install an M.2 Flash SSD In the standard Exadata Server X11M configuration, the flash riser board contains an M.2 flash SSD in a socket.
  • Page 77 Chapter 7 Install an M.2 Flash SSD Unpack the replacement M.2 flash SSD and place it on an antistatic mat. Install the M.2 flash SSD into socket SSD on the flash riser board. Align and then insert the plastic retainer clip into the slot on the M.2 flash SSD [1]. Hold the M.2 flash SSD so that the end of the card containing the plastic retainer clip is approximately one-half to one inch above the riser board, and the other end is adjacent to the riser SSD socket [1].
  • Page 78 Servicing the Air Baffle This section describes how to service the air baffle. You must power off the server before servicing the air baffle. The air baffle must be in place during system operation. Remove the air baffle to lift the Motherboard and access the motherboard handle.
  • Page 79 Chapter 8 Install the Air Baffle Set aside the air baffle. Consider your next step: • If you removed the air baffle as part of another procedure, return to that procedure. • Otherwise, continue to Install the Air Baffle. Install the Air Baffle Install the air baffle by placing it into the server and lowering it to its inserted position.
  • Page 80 Chapter 8 Install the Air Baffle the lip of the fan tray, and the others are on top of the fan tray. Consider your next step: • If you removed the air baffle as part of another procedure, return to that procedure. •...
  • Page 81 Servicing DIMMs This section describes how to service dual in-line memory modules (DIMMs). DIMMs are replaceable components that require you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable Components. Caution: These procedures require that you handle components that are sensitive to electrostatic discharge.
  • Page 82 In this scenario, you must adhere to the DIMM population rules and follow the recommended DIMM population order for optimal system performance. DIMM Population Rules The population rules for adding DIMMs to Exadata Server X11M are as follows: • The server supports: –...
  • Page 83 Chapter 9 Populating DIMMs for Optimal System Performance – Up to 3 TB memory when populating the 24 DIMM slots with 128-GB DIMMs. – 1 DIMM per channel (1DPC). Each DIMM channel is composed of a black slot. – A maximum supported memory speed of 4800 MT/s. However, the maximum attainable memory speed could be limited by the maximum speed supported by a specific processor or DIMM.
  • Page 84 Chapter 9 Populating DIMMs for Optimal System Performance Table 9-2 DIMM Memory Slot Population Requirements DIMMs per Channel DIMM Memory Slots CPU-0/1 Population Order Memory Channel: 12 DIMMs Number D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 per CPU Installed 10 DIMMs...
  • Page 85 Chapter 9 Populating DIMMs for Optimal System Performance Populating DIMMs in Dual-Processor Systems for Optimal System Performance In dual-processor systems, install DIMMs into DIMM slots starting with processor 0 (P0) D5, alternating between slots associated with processor 0 (P0) and matching slots for processor 1 (P1).
  • Page 86 Chapter 9 Using the Server Fault Remind Button P0/D6 P0/D3 P0/D8 P0/D4 P0/D7 P0/D1 P0/D10 P0/D2 P0/D9 P0/D0 P0/D11 Using the Server Fault Remind Button When you press the server Fault Remind button [1], an LED located next to the Fault Remind button lights green which indicates that there is sufficient voltage present in the fault remind circuit to light any fault LEDs that were lit due to a component failure.
  • Page 87 Chapter 9 Identify and Remove a DIMM Disconnect the power cords from the power supplies. Disconnect Cables From the Server. Extend the server into the maintenance position. Extend the Server to the Maintenance Position. Attach an antistatic wrist strap to your wrist and then to a metal area on the chassis. Take Antistatic Measures.
  • Page 88 Chapter 9 Install a DIMM Replace each failed DIMM with either another DIMM of the same rank size (quad-rank or dual-rank) or leave the slot empty. For DIMM replacement instructions, see Install a DIMM. Install a DIMM Unpack the replacement DIMM and place it on an antistatic mat. Ensure that the replacement DIMM matches the size, type, and rank of the DIMM it is replacing.
  • Page 89 Server. Verify that the power supply AC OK LED is lit. (Optional) Use Oracle ILOM to clear server DIMM faults. DIMM faults are automatically cleared after a new DIMM has been installed. If you need to manually clear DIMM faults, refer to Oracle ILOM Documentation.
  • Page 90 Servicing the Battery This section describes how to service the system battery. The system battery is a replaceable component that requires you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable Components.
  • Page 91 Chapter 10 Remove the Battery Note: Coin battery retainer types that are mounted in the RTC battery motherboard location may be different. Remove the battery from the battery holder. Review the following battery holder (Part number: 8210501) options. Callout Description 796136-1 Top-Load Battery Connector (TE) 1093TR Top-Load Battery Connector 1070TR Top-Load Battery Connector...
  • Page 92 Chapter 10 Remove the Battery Grasp connector. Support the connector by grasping the "positive" end of the connector. Note: CAUTION! The connector must be supported, otherwise damage to the soldered contacts could occur. Slide battery under lips at positive end of connector. While supporting the connector, simultaneously push the battery toward the positive end of the connector, and slightly move the lips at the negative end of the connector away from the battery.
  • Page 93 Chapter 10 Install the Battery Install the Battery Unpack the replacement RTC battery. Press the new battery into the battery retainer. • Install the battery into the Top-Load Battery Connector as follows: Each connector consists of a housing with a positive contact and a negative contact. The floor of the housing is embossed with a `+' to designate the positive contact.
  • Page 94 Chapter 10 Install the Battery Note: Insert the battery so that the positive (+) side is facing up. • Press the new battery into the battery retainer with the positive side (+) facing up. 10-5...
  • Page 95 Otherwise, proceed to the next step. If the service processor is not configured to use NTP, you must do one of the following: • Reset the Oracle ILOM clock using Oracle ILOM CLI or web interface. Refer to Oracle ILOM Documentation.
  • Page 96 Servicing PCIe Cards This section describes how to service PCIe cards and PCIe Risers. PCIe cards and PCIe risers are replaceable components that require you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable Components.
  • Page 97 Chapter 11 Remove a Half Height PCIe Card • The Full Height card chassis configuration uses two PCIe riser cards supporting two Full height 10.5 inch double width GPU cards at x16 PCIe Gen4. • Each riser also supports a PCIe Gen3 Pensando Ortano/Elba Server Adapter, DSC-200V, 2x200GbE, Dual QSFP56 Smart NIC card.
  • Page 98 Chapter 11 Install a Half Height PCIe Card Place the PCIe card on an antistatic mat. Caution: If you are not immediately inserting a replacement PCIe card into the empty slot, insert a PCIe filler panel in the slot to reduce the possibility of radiated electromagnetic interference (EMI).
  • Page 99 Chapter 11 Install a Half Height PCIe Card PCIe slots 1 through 4 are nonfunctional in single-processor systems. See Back Panel Components. If necessary, remove the PCIe filler panel from the slot. Save this filler panel in case you need to remove the PCIe card from the system. Insert the PCIe card into the correct slot [1], and rotate the PCIe locking mechanism downward to secure the PCIe card in place [2].
  • Page 100 Use Oracle ILOM to clear any server PCIe card faults. If a PCIe card fault message in Oracle ILOM is not cleared under Open Problems, you must manually clear the fault using Oracle ILOM. Refer to "Clear Faults for Undetected Replaced or Repaired Hardware Components"...
  • Page 101 Chapter 11 Install a Full Height PCIe Card While holding the PCIe Riser in one hand, use your other hand to carefully pull the PCIe card connector out of the PCIe Riser slot. If required, unplug any data cables from the PCIe card. Note the cable connections so that it's easier to properly reconnect the cables.
  • Page 102 Chapter 11 Install a Full Height PCIe Card If you are installing a x16 PCIe card in Riser 6, you must install the motherboard PCIe Flyover cable in to slot 6 to enable support for a PCIe x16 electrical interface; otherwise, slot 6 supports only a PCIe x8 electrical interface.
  • Page 103 Chapter 11 Remove a PCIe Riser Reconnect the cables to the PCIe card that you unplugged during the removal procedure. If it is difficult to reconnect the cables to the PCIe card when the card is secured into the PCIe slot, consider performing this procedure prior to inserting the PCIe card in to the PCIe Riser slot.
  • Page 104 Chapter 11 Install a PCIe Riser If required, remove the installed PCIe card from the Riser PCIe slot. Hold the Riser in one hand and use your other hand to carefully pull the PCIe card connector out of the PCIe riser slot. Leave the PCIe data and power cables attached to the PCIe card and place the PCIe card on an antistatic mat.
  • Page 105 Chapter 11 Install a PCIe Riser Reconnect the cables to the PCIe card that you unplugged during the removal procedure. If it is difficult to reconnect the cables to the PCIe card when the card is inserted into the PCIe slot, consider performing this procedure prior to inserting the PCIe card in to the PCIe Riser slot.
  • Page 106 Use Oracle ILOM to clear any server PCIe card faults. If a PCIe card fault message in Oracle ILOM is not cleared under Open Problems, you must manually clear the fault using Oracle ILOM. Refer to "Clear Faults for Undetected Replaced or Repaired Hardware Components"...
  • Page 107 Chapter 11 Remove the PCIe Flyover Cable Press and hold the button on the connector to disengage the locking mechanism, then gently pull the connector from the motherboard [1]. Callout Description PCIe Flyover cable PCIe slot 7 electrical interface connector PCIe slot 6 electrical interface connector Install the connector in the desired slot and press down on the connector until it is fully seated in the slot [2].
  • Page 108 Chapter 11 Install the PCIe Flyover Cable Disconnect the power cords from the power supplies. Disconnect Cables From the Server. Extend the server to the maintenance position. Extend the Server to the Maintenance Position. Attach an antistatic wrist strap. Take Antistatic Measures.
  • Page 109 Chapter 11 Install the PCIe Flyover Cable Plug each end of the cable into its connector until you hear an audible click. If necessary, reinstall PCI Riser 6. Install a PCIe Riser. Return the server to operation. Install the top cover. Install the Server Top Cover.
  • Page 110 See Electrostatic Discharge Safety. Remove the Motherboard Assembly Caution: Use the Oracle ILOM backup utility prior to removing the motherboard. This utility backs up the Oracle ILOM configuration of the service processor. Refer to Oracle ILOM Documentation. Prepare the server for service.
  • Page 111 (KIP) data might be lost. When a server requires service, the KIP is used by Oracle to verify that the warranty on the server has not expired. For more information on KIP, see FRU Key Identity Properties (KIP) Automated Update.
  • Page 112 Chapter 12 Remove the Motherboard Assembly Servicing the Front LED Indicator Module. Disk backplane auxiliary signal (DBP SIG) cable Servicing the Disk Backplane. Disk backplane power (DBP PWR) cable Disconnect the power cable from the motherboard and the disk backplane connections.
  • Page 113 Chapter 12 Remove the Motherboard Assembly If you are not able to remove the motherboard, loosen the third screw in the midwall motherboard handle. After removing the motherboard, tighten the same screw. With the server in the extended maintenance position, remove the motherboard from the server with all reusable components that populate the motherboard in place.
  • Page 114 Chapter 12 Remove the Motherboard Assembly Place the motherboard assembly on an antistatic mat, and next to the replacement motherboard. Remove the following reusable components from the old motherboard and install them onto the replacement motherboard. • DIMMs Identify and Remove a DIMM Install a DIMM.
  • Page 115 Chapter 12 Remove the Motherboard Assembly Note the orientation and correct handling of the External Cap. Callout Description Top of External Cap (insertion tabs are circled) Bottom of External Cap (contacts the socket) Use of the External Cap handle Processor Overview for processor frame and socket information.
  • Page 116 Chapter 12 Remove the Motherboard Assembly Lift the force frame to the open position. On the socket, open the Rail Frame. Squeeze the blue tabs to release the rail frame. 12-7...
  • Page 117 Chapter 12 Remove the Motherboard Assembly Lift the processor rail frame by holding its lift tabs and rotating the rail frame to its near vertical position. Note: The rail frame is spring-loaded. Hold on to the rail frame as it releases from the socket frame, when rotating it to the vertical position.
  • Page 118 Chapter 12 Remove the Motherboard Assembly Retain the external cap to be placed on the processor socket (in the Rail Frame) of the motherboard being replaced. This will prevent damage during shipping back to the Service Center for motherboard failure analysis. Install an External Cap on a Processor 0 or Processor 1 socket for motherboard returns.
  • Page 119 Chapter 12 Remove the Motherboard Assembly Grasping the rail frame edge [1], rotate the rail frame to the closed position [2]. Rotate the force frame to its closed position on the socket stiffener frame. Using a 12 in-lb (inch-pound) (1.35 Nm/13.5 kg-cm) driver with a Torx T20 bit, fully tighten the captive screw that secures the force frame to the socket stiffener frame.
  • Page 120 Chapter 12 Remove the Motherboard Assembly Use one hand to hold down the spring-loaded force frame until the screw is tightened. Install a processor into the socket on the replacement motherboard. Install a processor into the socket Processor 0 or Processor 1 from which you removed the processor External Cap.
  • Page 121 Chapter 12 Install the Motherboard Assembly Twist the tool back and forth to release the light pipe from the housing. Pull the light pipe away from the housing. iii. Install the light pipe on the replacement motherboard. Match the rectangular holes on the outside of the light pipe with the retaining clips on the housing.
  • Page 122 Chapter 12 Install the Motherboard Assembly motherboard and place it into the server chassis. Slide the motherboard to the back of the server to engage the raised standoffs. Caution: Be careful not to damage the rear Locate Button/LED when placing the motherboard into the chassis.
  • Page 123 (KIP) data might be lost. When a server requires service, the KIP is used by Oracle to verify that the warranty on the server has not expired. For more information on KIP, see...
  • Page 124 Chapter 12 Install the Motherboard Assembly • PCIe cards and motherboard PCIe cable Configure PCIe Electrical Interfaces Using the PCIe Flyover Cable Install a Half Height PCIe Card. Install a Full Height PCIe Card. • SAS storage drive cables (if present) Install SAS Storage Drive Cables.
  • Page 125 Servicing Processors This section describes how to service processors. Processors are replaceable components that require you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable Components. Caution: Ensure that all power is removed from the server before removing or installing a processor.
  • Page 126 Chapter 13 Processor Overview Callout Description Pin 1 orientation Force Frame Retention Screw (captive) Rail Frame (socket stiffener frame) Release Tabs (2x) Force Frame (retention frame) Processor Package (carrier frame holding preinstalled processor) Heatsink Attachment Studs (6x) External Cap The external cap is used to protect the socket pins if the processor is not installed. The following figure shows the External Cap.
  • Page 127 Chapter 13 Identify and Remove a Processor Identify and Remove a Processor • Use ESD gloves (not latex or vinyl), if possible, when handling the processor. • Obtain a torque driver set to 12 in-lb (inch-pound) (1.35 Nm/13.5 kg-cm) force with T20 Torx bit for processor and heatsink removal.
  • Page 128 Chapter 13 Identify and Remove a Processor • Never remove the processor from the carrier. Replacement processors come with a carrier package. • Whenever you remove a processor on a usable motherboard, replace it with another processor and reinstall the processor heatsink; otherwise, the server might overheat due to improper airflow.
  • Page 129 Chapter 13 Identify and Remove a Processor Note: When you press the Fault Remind button, an LED located next to the Fault Remind button illuminates green, indicating that there is sufficient voltage in the fault remind circuit to illuminate any fault LEDs that were illuminated due to a failure.
  • Page 130 Chapter 13 Identify and Remove a Processor Loosen in reverse order 6-1, then go back and disengage completely. Loosen captive screws 6,5,4,3,2,1 in the order shown in the illustration. Disengage captive screws 6,5,4,3,2,1 in the order shown in the illustration. Use a torque driver set to 12 in-lb (inch-pound) (1.35 Nm/13.5 kg-cm) force with T20 Torx bit.
  • Page 131 Chapter 13 Identify and Remove a Processor Note: Always grip the heatsink along the axis of the fins to prevent damage. Place the heatsink, facing up, on a work surface. Remove Thermal Grease. Clean off the thermal interface material (TIM) using the supplied alcohol wipes. If reusing a processor, do not remove it from the processor carrier.
  • Page 132 Chapter 13 Identify and Remove a Processor Caution: Failure to clean thermal grease from the heatsink could result in the accidental contamination of the processor socket or other components. Be careful not to get the grease on your fingers, as this could result in contamination of components.
  • Page 133 Chapter 13 Identify and Remove a Processor Using a Torx T20 screwdriver, loosen the captive Force Frame Retention Screw that secures the Force Frame to the socket. Lift the Force Frame to the open position. Apply counter-pressure to the Force Frame when opening. Open the Rail Frame.
  • Page 134 Chapter 13 Identify and Remove a Processor Squeeze the blue tabs to release the Rail Frame. Lift the Rail Frame and slide out the carrier frame package. Lift the Rail Frame by holding its lift tabs and rotating the Rail Frame to its near vertical position.
  • Page 135 Chapter 13 Identify and Remove a Processor Place the carrier frame package in an appropriate tray. Wipe the Force Frame (retention frame) with a dry wipe followed by a wipe with a small amount of isopropyl alcohol. Clean off the thermal interface material (TIM) as needed, wipe the outside and inside of the Force Frame with a dry wipe followed by a wipe with a small amount of isopropyl alcohol.
  • Page 136 Chapter 13 Install a Processor Wipe the processor lid and carrier frame with a dry wipe followed by a wipe with a small amount of isopropyl alcohol. Clean off the thermal interface material (TIM) from the processor using the supplied alcohol wipes.
  • Page 137 Chapter 13 Install a Processor Caution: The processor socket pins are very fragile. A light touch can bend the processor socket pins beyond repair. • If you are replacing a processor, ensure that the replacement processor is identical to the processor that was removed.
  • Page 138 Chapter 13 Install a Processor Callout Description Bottom of processor carrier (contacts the socket) Use of the processor carrier handle Holding the processor carrier by its handle [1], slide it into the Rail Frame. Ensure the two insertion tabs at the bottom of the processor carrier are inserted into the slots in the Rail Frame [2].
  • Page 139 Chapter 13 Install a Processor Using a 12 in-lb (inch-pound) (1.35 Nm/13.5 kg-cm) driver with a Torx T20 bit, fully tighten the captive screw that secures the Force Frame to the socket stiffener frame. Use one hand to hold down the spring-loaded Force Frame until the screw is tightened. Apply Thermal Grease.
  • Page 140 Chapter 13 Install a Processor Lower the heatsink (see the following note) on to the six socket heatsink attachment studs. Screws 1 and 2 are slightly longer than screws 3, 4, 5, and 6. Carefully lower the module. Note: The heatsink is keyed to go on only one way. The captive screws that attach to the pin 1 end of the socket stiffener frame are closer together than the captive screws that attach at the opposite end of the stiffener frame.
  • Page 141 Chapter 13 Install a Processor Using a 12 in-lbf (1.35 Nm/13.5 kg-cm) torque driver with a Torx T20 bit, tighten the heatsink module to the socket, as shown in the following illustration. Engage captive screws 1, 2, 3, 4, 5 and 6 in the order shown in the illustration. Tighten captive screws 1, 2, 3, 4, 5 and 6 in the order shown in the illustration from 12 in-lbf (13.5 kg-cm) to 13.02 in-lbf (15.0 kg-cm).
  • Page 142 Install a Processor Use Oracle ILOM to clear server processor faults. To show server faults: log in to the server as root using the Oracle ILOM CLI, and type the following command to list all known faults on the server: ->...
  • Page 143 Servicing the Disk Backplane This section describes how to service the 4-Drive backplane and 12-Drive backplane. The disk backplane (DBP) is a replaceable component that requires you to power off the server before servicing. Perform the 4-Drive backplane or 12-Drive backplane procedures. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable...
  • Page 144 Chapter 14 Remove the 4-Drive Backplane Remove a Fan Module. Remove the fan tray from the server. Remove the Fan Tray. Remove all NVMe storage drives from the server front panel and set them aside on an antistatic mat. Remove a Storage Drive.
  • Page 145 Chapter 14 Install the 4-Drive Backplane Note: Do not disconnect the DBP power cable and DBP auxiliary signal cable from the disk backplane at this time. Using a Torx T15 screwdriver, loosen the two spring-mounted screws that secure the storage drive cage in to the chassis [2]. Ensure that the DBP power cable and DBP auxiliary signal cable are free from snagging motherboard components and gently pull the storage drive cage with cables attached through the front panel of the chassis [3].
  • Page 146 Chapter 14 Install the 4-Drive Backplane Reconnect the DBP power cable and the DBP auxiliary signal cable to the disk backplane [3]. Install the storage drive cage in to the chassis. Align the storage drive cage with the opening in the front panel of the server. Pull the attached DBP power cable and DBP auxiliary signal cable through the opening while gently pushing the storage drive cage in to the chassis until it is fully seated in the chassis [4].
  • Page 147 Note: IMPORTANT: When the disk backplane is replaced, the key identity properties (KIP) of the backplane is programmed by Oracle ILOM to contain the same KIP as the other quorum member components. If you removed other quorum member components, you might need to manually program the product serial number (PSN) into the new backplane.
  • Page 148 Chapter 14 Remove the 12-Drive Backplane Remove the fan tray from the server. Remove the Fan Tray. Pull each storage drive out far enough to disengage it from the backplane. Remove a Storage Drive. Note: It is not necessary to completely remove the storage drives from the server; simply pull them out far enough to disengage them from the backplane.
  • Page 149 Chapter 14 Install the 12-Drive Backplane Disconnect the DBP auxiliary signal cable from the 12-Drive backplane [3]. See also Remove the Disk Backplane Signal Cable. Using a Torx T15 screwdriver, loosen the right-side and left-side spring-mounted screws that secure the 12-Drive backplane to the chassis [4]. Lift the 12-Drive backplane up to release it from the standoff hooks and out of the chassis [5 and 6].
  • Page 150 Chapter 14 Install the 12-Drive Backplane Using a Torx T15 screwdriver, tighten the right-side and left-side spring-mounted screws to secure the 12-Drive backplane to the chassis [3]. Reconnect the cables to the disk backplane. Reconnect the DBP auxiliary signal cable to the 12-Drive backplane [4]. See also Install the Disk Backplane Signal Cable.
  • Page 151 Note: IMPORTANT: When the disk backplane is replaced, the key identity properties (KIP) of the backplane is programmed by Oracle ILOM to contain the same KIP as the other quorum member components. If you removed other quorum member components, you might need to manually program the product serial number (PSN) into the new backplane.
  • Page 152 Chapter 14 Remove the Disk Backplane Signal Cable Locate the disk backplane signal cable. The disk backplane signal cable is connected between the 4 or 12 disk backplane and the motherboard connector, which is located between PCIe slot 4 and the motherboard handle. The signal cable is routed from the disk backplane, through the left-side cable trough, and then through the motherboard handle before connecting to the motherboard [1, 2].
  • Page 153 Chapter 14 Install the Disk Backplane Signal Cable Note: For information on disconnecting the disk backplane signal cable from the 4- disk backplane, see Remove the 4-Drive Backplane. Remove the cable from the server. Gently direct the cable through the motherboard handle and then lift the cable from the left-side cable trough [2].
  • Page 154 Chapter 14 Install the Disk Backplane Signal Cable Ensure that the disk backplane signal cable is free from snagging motherboard components and other cables in the server. You might need to remove the SAS cables from the left-side cable trough before installing the disk backplane signal cable. See Remove SAS Storage Drive Cables.
  • Page 155 Chapter 14 Install the Disk Backplane Signal Cable Note: For information on reconnecting the disk backplane signal cable to the 4-disk backplane, see Install the 4-Drive Backplane. If necessary, reinstall SAS cables in the left-side cable trough. Install SAS Storage Drive Cables.
  • Page 156 Servicing the Front LED Indicator Module This section describes how to service the front LED indicator module. The front LED indicator module is a replaceable component that requires you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable...
  • Page 157 Chapter 15 Remove the Front LED Indicator Module Note: To ease removal of the LED indicator module cable, you might need to lift the disk backplane auxiliary signal cable and SAS cable (if present) from the left-side cable trough. Remove the three No. 2 Phillips screws that secure the FIM cable cover to the chassis [1]. Lift up and remove the FIM cable cover from the FIM cable cover slot [1].
  • Page 158 Chapter 15 Install the Front LED Indicator Module Remove the two No. 2 Phillips screws that secure the LED indicator module to the server front panel. Remove the LED indicator module and cable from the server front panel. Remove the LED indicator module cable by carefully pulling the cable through the left- side cable trough.
  • Page 159 Chapter 15 Install the Front LED Indicator Module Return the server to operation. Install the server top cover. Install the Server Top Cover. Return the server to the normal rack position. Return the Server to the Normal Rack Position. Reconnect the power cords to the power supplies, and power on the server. Reconnect Power and Data Cables Power On the Server.
  • Page 160 Servicing SAS Cables This section describes how to service SAS cables. SAS interface cables are replaceable components that require you to power off the server before servicing. For more information about replaceable components, see Illustrated Parts Breakdown Replaceable Components. Caution: The system supplies power to the cables even when the server is powered off.
  • Page 161 Chapter 16 Install SAS Storage Drive Cables Remove the Fan Tray. Disconnect the SAS cables from the disk backplane [1]. Press and hold the button on the connector to disengage the locking mechanism, then gently pull the connector from the disk backplane. Disconnect the SAS cables from the Broadcom 9500-16i, 12GB SAS PCIe, 16 Port, Internal card in PCIe slot 9 [2].
  • Page 162 Chapter 16 Install SAS Storage Drive Cables Reconnect the SAS cables to the Broadcom 9500-16i, 12GB SAS PCIe, 16 Port, Internal card in PCIe slot 9 [2]. Plug each cable into its SAS connector until you hear an audible click. To ensure proper SAS cable connections, see the SAS cable connections table in Step Note:...
  • Page 163 Chapter 16 Install SAS Storage Drive Cables Install the air baffle. Install the Air Baffle. Install the server top cover. Install the Server Top Cover. Return the server to the normal rack position. Return the Server to the Normal Rack Position.
  • Page 164 Returning the Server to Operation This section describes safety considerations and provides procedures and information after replacing components in the server. After replacing components inside of the server, perform the procedures in the following sections. Server Filler Panel Requirements The server might be shipped with module-replacement filler panels for storage drives and PCIe cards.
  • Page 165 Chapter 17 Install the Server Top Cover Remove the server top cover. To remove the PCIe slot filler panel, rotate the PCIe locking mechanism in to an upright position, and lift and remove the PCIe slot filler panel where you want to install the PCIe card.
  • Page 166 Chapter 17 Remove Antistatic Measures Use a Torx T15 screwdriver to turn the release button latch counter-clockwise to the locked position. Remove Antistatic Measures • Remove any antistatic straps or conductors from the server chassis. Reinstall the Server Into the Rack After servicing the system, reinstall it into the rack.
  • Page 167 Chapter 17 Return the Server to the Normal Rack Position Rackmounting the Server. If the cable management arm (CMA) is not installed, because you removed the server completely out of the rack, install the CMA. Install the Cable Management Arm (Optional).
  • Page 168 • Press the On/Standby button on the front bezel. • Log in to the Oracle ILOM web interface, click Host Management → Power Control, and select Power On from the Select Action list. • Log in to the Oracle ILOM command-line interface (CLI), and type the following command at the Oracle ILOM prompt: ->...
  • Page 169 Chapter 17 Power On the Server When the server is powered on to Main power mode and the power-on self-test (POST) code checkpoint tests are complete, the green System OK LED status indicator on the front panel of the server lights and remains lit. 17-6...
  • Page 170 Known Service Issues This section describes important operating notes and known service issues for Exadata Server X11M. You can find detailed information about Exadata Server X11M supported hardware in Product Description. For the most updated information about supported firmware and operating systems, important operating notes, and known issues, refer to Oracle AMD-Based Cloud Servers Product Notes.
  • Page 171 Appendices: Server Reference These appendices contain Exadata Server X11M service references.
  • Page 172 Server Status Indicator LEDs These sections describe Exadata Server X11M status indicators (LEDs) located on the front and back panels, including indicators located on components and ports. System-Level Status Indicators System-level status indicators (LEDs) are located on both the server front panel and the back panel.
  • Page 173 Appendix A Power Supply Status Indicators Table A-1 (Cont.) System-Level Status Indicators Status Icon Color State and Meaning Indicator Name SP OK Green Indicates the state of the service processor. OFF – Service processor (SP) is not running. SLOW BLINK – SP is booting. Front panel STEADY ON –...
  • Page 174 Appendix A Fan Module Status Indicators Table A-2 (Cont.) Power Supply Status Indicators Status Icon Color State and Meaning Indicator Name Fault-Service Amber OFF – Normal operation. No service action is required. Required STEADY ON – The power supply (PS) detected a PS fan failure, PS overtemperature, PS overcurrent, or PS overvoltage or undervoltage.
  • Page 175 Each of the 24 DIMM slots on the motherboard has an amber fault status indicator Status Indicators (LED) associated with it. If Oracle ILOM determines that a DIMM is faulty, pressing the Fault Remind button on the motherboard I/O card signals the service processor to Servicing DIMMs.
  • Page 176 Appendix A Mellanox ConnectX-6 Dx SmartNIC Status Indicators Table A-7 (Cont.) Motherboard Status Indicators Status Description Indicator Processor Fault The motherboard includes a fault status indicator (LED) adjacent to each of the two Status Indicators processor sockets. These LEDs indicate when a processor fails. Pressing the Fault Remind button on the motherboard I/O card signals the service processor to light the Servicing fault status indicators associated with the failed processors.
  • Page 177 Appendix A Pensando Ortano DSC SmartNIC Status Indicators Callout Description Port 2 status indicator: • OFF = Link has not been established • SOLID GREEN = Valid link has been established • BLINKING GREEN = Link activity • BLINKING AMBER = 1 Hz blink indicates a beacon command is being run to locate the adapter card;...
  • Page 178 After a few minutes, the main System OK LED slowly flashes the standby blink pattern (0.1 seconds on, 2.9 seconds off), indicating that the SP (and Oracle ILOM) is ready for use. In Standby power mode, the server is not initialized or fully powered on at this point.
  • Page 179 The server uses high-line AC power. The server can operate effectively over a range of voltages and frequencies. The following table contains the power supply specifications for Exadata Server X11M. Note: The power dissipation numbers listed in the following table are the maximum rated power numbers for the power supply used in the server.
  • Page 180 Table B-3 Power Supply Specifications Parameter Power Supply Specification Exadata Server X11M Two hot-swappable and highly-redundant 1400W A271 or A271A power supplies (PS0 and PS1) A271 Power Supply A271 - Delta Electronics, Model AWF2DC1400W, Input rated 200 - 240 V~, 10A Max, 50/60 Hz.
  • Page 181 80 percent of the branch circuit AC current rating. PDU power cords for Oracle racks are 4 meters (13.12 feet) long, and 1 to 1.5 meters (3.3 to 4.9 feet) of the cord might be routed in the rack cabinet. The installation site AC power receptacle must be within 2 meters (6.6 feet) of the rack.
  • Page 182 Appendix B Environmental Requirements Table B-4 Environmental Requirements Specification Operating Nonoperating Ambient Maximum range: 5°C to 35°C (41°F to 95°F) –40°C to 68°C (–40°F to 154°F) temperature (does up to 900 meters (2,953 feet) not apply to Optimal: 21°C to 23°C (69.8°F to 73.4°F) removable media) Maximum ambient operating temperature is derated by 1 degree C per 300 meters of...
  • Page 183 Rackmountable servers and equipment, including Exadata Server X11M, draw cool air in through the front of the rack and release warm air out the back of the rack. There is no airflow requirement for the left and right sides due to front-to-back cooling.
  • Page 184 You must evaluate these restrictions. Server placement is particularly important for high-temperature environments. • Manage cables to minimize interference with the server exhaust vent. Agency Compliance Exadata Server X11M complies with the following specifications. Table B-5 Agency Compliance Category Relevant Standards...
  • Page 185 NVMe Storage Drive 8214990 specifications are listed in the following table. Table B-6 480GB, M.2, NVMe Solid State Drive 8214990 Specification Specification Value Device name Product Identifier: MICRON_7450_MTFDKBA480TFR Oracle Part Number: 8214990 Device Identification: • PCIe Vendor ID: 0x1344 • PCIe Device ID: 0x51C3 •...
  • Page 186 Appendix B 480GB NVMe M.2 Solid State Drive Specification Table B-8 Drive Reliability Attribute Value Component Design Life (Useful life) 5 years MTBF 2,000,000 hours Expected AFR (Annualized Failure 0.44% for normal 24x7 operating conditions Rate) Table B-9 Drive Capacity and Performance Attribute Value Capacity, formatted...
  • Page 187 NVMe Storage Drive 8214991 specifications are listed in the following table. Table B-13 480GB, M.2, NVMe Solid State Drive 8214991 Specification Specification Value Device name Product Identifier: SAMSUNG MZVL2480HBJD-00A07 Oracle Part Number: 8214991 Device Identification: • PCIe Vendor ID: 0x144D • PCIe Device ID: 0xA80A •...
  • Page 188 Appendix B 480GB NVMe M.2 Solid State Drive Specification Table B-13 (Cont.) 480GB, M.2, NVMe Solid State Drive 8214991 Specification Specification Value Features NVMe PCIe Gen4 Interface NVMe-MI VPD is blank “FFs” as supplied TCG Opal 2.0 compliant Product Compliance •...
  • Page 189 Appendix B 480GB NVMe M.2 Solid State Drive Specification Table B-16 (Cont.) Drive Capacity and Performance Attribute Value Capacity, raw NAND 512 GiB Random 4 KB Read 4 KB Random Read - 130K IOPS (up to) Latency - QD=1, 1 thread, 75us (Typical) Random 4 KB Write 4 KB Random Write - 20K IOPS (up to) Latency - QD=1, 1 thread, 50us (Typical)
  • Page 190 Appendix B 22TB Hard Disk Drive Specification Table B-19 (Cont.) Drive Characteristics Attribute Value Life monitoring capability Provides alerts for proactive replacement of the drive before the endurance is depleted. Provides endurance remaining in NVMe SMART logs. SSD supports the standard method defined by NVMe for Solid State Drive to report NAND wear through the “Get Log”...
  • Page 191 Appendix B 22TB Hard Disk Drive Specification Table B-21 (Cont.) 22TB HDD Drive Physical Characteristics Attribute Value Depth 147 mm, 5.787 +/-0.006 in. Weight Up to 0.670 kg, 23.6336 oz Table B-22 22TB HDD Drive Usage Usage Description Useful life 5 years Minimum Expected annualized failure rate (AFR) 0.35%...
  • Page 192 Maintenance Space Requirements The maintenance area for the rackmounted Exadata Server X11M must have the required access space. The following table lists the maintenance access requirements for the server when it is installed in a rack.
  • Page 193 Rack Compatibility Rack Compatibility The rack into which you install Exadata Server X11M must meet the requirements listed in the following table. Oracle Rack Cabinet 1242 and Sun Rack II are compatible with Exadata Server X11M. For information about the racks, see Rackmounting the Server.
  • Page 194 Depth is measured from front door handle to back door handle. Installation Procedure This topic provides an overview of the Exadata Server X11M installation procedure. Review the entire installation procedure and find links to more information about each step. The following list summarizes the tasks that you must perform to properly install Exadata Server X11M.
  • Page 195 Installation Prerequisites Familiarize yourself with safety precautions and electrostatic discharge (ESD). Before installing the server, read the safety information in Oracle Server Safety and Compliance Guide and in Important Safety Information for Oracle's Hardware Systems. Assemble the required tools and equipment for installation.
  • Page 196 Deploy your rack anti-tilt bar to prevent the rack from tipping during equipment installation. Lifting equipment: Exadata Server X11M weighs up to 76 lbs (34 kg). Two people are needed to mount the server into the rack enclosure.
  • Page 197 Before installing the server, read the safety information in Oracle Server Safety and Compliance Guide and in Important Safety Information for Oracle's Hardware Systems. Tools and Equipment For Installation To install and maintain the server, you must have the following service tools: •...
  • Page 198 Appendix C Rackmounting the Server Note: Rack Mount Instructions A) Elevated Operating Ambient - If installed in a closed or multi-unit rack assembly, the operating ambient temperature of the rack environment may be greater than room ambient. Therefore, consideration should be given to installing the equipment in an environment compatible with the maximum ambient temperature (Tma) specified by the manufacturer.
  • Page 199 Related Rack Documentation • Oracle Rack Cabinet 1242 Documentation • Oracle Sun Rack II 1242 Documentation • Oracle Sun Rack II 1042 Documentation Install Mounting Brackets on the Server To install the mounting brackets on the sides of the server:...
  • Page 200 Repeat Step 1 through Step 3 to install the other mounting bracket on the other side of the server. Mark the Rackmount Location Identify the location in the rack where you want to place the server. Exadata Server X11M requires two rack units (2U). Use the Rackmounting Template to identify the correct mounting holes for the slide-rails.
  • Page 201 Appendix C Rackmounting the Server Caution: Always load equipment into a rack from the bottom up so that the rack does not become top-heavy and tip over. Extend the rack anti-tilt bar to prevent the rack from tipping during equipment installation. Ensure that there is at least two rack units (2U) of vertical space in the rack cabinet to install the server.
  • Page 202 Appendix C Rackmounting the Server Mark the mounting holes for the front slide-rails. Mark the mounting holes for the back slide-rails. Install AC Power Cables and Slide-Rails Before you install the slide-rails into the rack, install server right-angle AC power cables into the left-side and right-side PDU electrical sockets.
  • Page 203 Appendix C Rackmounting the Server Install the slide-rails into the rack. Attach the Slide-Rails. Attach the Slide-Rails Use this procedure to attach slide-rail assemblies to the rack. C-12...
  • Page 204 Appendix C Rackmounting the Server Note: Before you install the slide-rails, be sure to install the server right-angle AC power cables (part number 7079727 - Pwrcord, Jmpr, Bulk, SR2, 2m, C14RA, 10A, C13). In the 1000 mm rack, the standard rail kit slide-rails obstruct access to the front of the 15kVA, 22kVA, and 24kVA Power Distribution Unit (PDU) electrical sockets.
  • Page 205 Appendix C Rackmounting the Server Align the front of the slide-rail assembly against the outside of the front rack rail, and push until the assembly locks into place with an audible click. Repeat Step 1 through Step 3 to attach the slide-rail assembly to the other side of the rack. Install the Server Into the Slide-Rail Assemblies Use this procedure to install the server chassis with mounting brackets into the slide-rail assemblies that are mounted to the rack.
  • Page 206 Appendix C Rackmounting the Server Insert the mounting brackets into the slide-rails, and then push the server into the rack until the mounting brackets are flush with the slide-rail stops (approximately 30 cm or 12 inches). Call Out Description Inserting mounting bracket into slide-rail Slide-rail release button Slide-rail lock Simultaneously push and hold the green slide-rail release buttons on each mounting...
  • Page 207 Appendix C Rackmounting the Server Caution: Before you install the optional cable management arm verify that the server is securely mounted in the rack and that the slide-rail locks are engaged with the mounting brackets. Install the Cable Management Arm (Optional) Follow this procedure to install the cable management arm (CMA), which you can use to manage cables connected to the back of the server.
  • Page 208 Appendix C Rackmounting the Server Call Out Description Connector A Front slide bar Velcro straps (6) Connector B Connector C Connector D Slide-rail latching bracket (used with connector D) Back slide bar Server flat cable covers Server round cable covers (optional) Prepare the CMA for installation.
  • Page 209 Appendix C Rackmounting the Server Note: Ensure that the two Velcro straps located on the front slide bar are threaded through the opening in the top of the slide bar, as shown in the illustration in Step 1. This prevents the Velcro straps from interfering with the expansion and contraction of the slide bar when the server is extended out of the rack and returned to the rack.
  • Page 210 Appendix C Rackmounting the Server Call Out Description Connector A tab Left slide-rail front slot To install CMA connector B into the right slide-rail: Insert CMA connector B into the front slot on the right slide-rail until it locks into place with an audible click [1 and 2].
  • Page 211 Appendix C Rackmounting the Server Call Out Description Connector C locking spring Insert connector C into the right slide-rail until it locks into place with an audible click [2 and 3]. Gently tug on the right side of the CMA back slide bar to verify that connector C is properly seated.
  • Page 212 Appendix C Rackmounting the Server To install CMA connector D into the left slide-rail: While holding the slide-rail latching bracket in place, insert connector D and its associated slide-rail latching bracket into the left slide-rail until connector D locks into place with an audible click [1 and 2].
  • Page 213 Appendix C Rackmounting the Server To verify that the slide-rails and the CMA are operating properly before routing cables through the CMA: Ensure that the rack anti-tilt bar is extended to prevent the rack from tipping forward when the server is extended. Caution: To reduce the risk of personal injury, stabilize the rack cabinet and extend the anti-tilt bar before extending the server from the rack.
  • Page 214 Appendix C Rackmounting the Server Continue pushing the server into the rack until the slide-rail locks (on the front of the server) engage the slide-rail assemblies. You hear a click when the server is in the normal rack position. Connect cables to the server, as required. Reconnect Power and Data Cables.
  • Page 215 Appendix C Rackmounting the Server Ensure that the secured cables do not extend above the top or below the bottom of the server to which they are attached. Otherwise, the cables might snag on other equipment installed in the rack when the server is extended from the rack or returned to the rack.
  • Page 216 Appendix C Rackmounting the Server Note: References to “left” or “right” in this procedure assume that you are facing the back of the equipment rack. To prevent the rack from tipping forward when the server is extended, ensure that the rack anti-tilt bar is extended.
  • Page 217 Appendix C Rackmounting the Server Call Description Connector D release tab (green) Slide-rail latching bracket release tab (labeled PUSH) Use your right hand to support the CMA and use your left thumb to push in (toward the left) on the slide-rail latching bracket release tab labeled PUSH (callout 2), and pull the latching bracket out of the left slide-rail and put it aside [3 and 4].
  • Page 218 Appendix C Rackmounting the Server Call Out Description Connector C release tab (labeled PUSH) To disconnect connector B: Place your right arm under the CMA to support it and grasp the back end of connector B with your right hand. Use your left thumb to pull the connector B release lever to the left, away from the right slide-rail (callout 1), and use your right hand to pull the connector out of the slide-rail [1 and 2].
  • Page 219 Configure NVMe RAID Using BRU BRU on Oracle AMD Platforms On Exadata Server X11M, BRU is employed in "Native" mode, which is based on Linux RAID metadata version 1.2. Oracle AMD platforms do not support IMSM (Intel© Matrix Storage Manager) containers and metadata, which are Intel VROC features.
  • Page 220 The Configure NVMe RAID1 Boot Devices procedure configures a RAID1 volume so that you can install Oracle Linux. Boot RAID on UEFI (BRU), implements a UEFI driver that identifies a bootable NVMe M.2 RAID1 volume in the UEFI environment. OnExadata Server X11M, BRU uses native Linux RAID1 metadata (version 1.2) and is fully compatible with mdadm utilities in...
  • Page 221 Appendix C Operating System Installation Process -> dump -destination scp://username@remote_host.us.foo.com/var/tmp/ ueficfg.xml /System/BIOS/Config Enter remote user password: ******** Dump successful. -> On the remote host, use a text editor to modify ueficfg.xml; find the 'Block RAID in UEFI' heading, change the Block_RAID_in_UEFI value from 'Disable' to 'Enable'. Change this <Block_RAID_in_UEFI>...
  • Page 222 Review the server site requirements, specifications, and components. Complete the following checklists to ensure that your site meets the physical, electrical, and environmental requirements of Exadata Server X11M before you receive the server. Review the following facility requirements before you install the server.
  • Page 223 Appendix D Table D-2 Data Center Environment Checklist Data Center Environmental Considerations Comment Does the computer room air handling meet temperature and humidity requirements? Does the installation floor layout satisfy the ventilation requirements? Will the equipment be positioned so that the exhaust air of one rack does not enter the air intake of another rack? Are the perforated floor tiles each rated at 400 CFM or greater?
  • Page 224 Does the rack have RETMA rail support? Does the rack support Oracle cable management arms (CMAs)? Does the rack support installation of Oracle vented and solid filler panels? Is there sufficient space for cable harnesses and the power distribution units (PDUs) in the rack, if required?
  • Page 225 Comment Is antistatic flooring installed? Is the floor below the raised floor free of obstacles and blockage? Complete the following checklist if you are planning to use Oracle Auto Service Request with the server. Table D-6 Auto Service Request Considerations...
  • Page 226 Appendix D Table D-7 (Cont.) Logistics Checklist Considerations Logistics Checklist Considerations Comment Is the unpacking area air-conditioned to avoid thermal shock for various hardware components? Will sufficient moving personnel be available to install the hardware? Are you prepared for uncrating and trash removal? Are there any restrictions on delivery truck length, width, or height? Are cardboard boxes and other packing material...
  • Page 227 The tables in this section identify the system components and describe the naming conventions applied to the components of Exadata Server X11M. Each section corresponds to an IPMI entity ID and lists sensors, indicators, and replaceable units related to that entity.
  • Page 228 Appendix E Power Unit Components Table E-1 (Cont.) System Chassis Components Component Name IPMI Type Description Values (if applicable) (Oracle ILOM CLI Targets) Indicator Temperature Fault LED Color: Amber /SYS/TEMP_FAULT Location: Front panel Off: Normal On: Chassis overtemp fault /SYS/OK...
  • Page 229 Appendix E Cooling Unit Components Table E-2 Power Unit Components Component Name IPMI Type Description Values (if applicable) (Oracle ILOM CLI Targets) Power distribution board FRU /SYS/PDB /SYS/PS[0-1] Power supply FRU /SYS/PS[0-1]/PRSNT Discrete Power supply is present. (Hidden) 01h-ENTITY_ABSENT sensor...
  • Page 230 Appendix E Disk Backplane Components Table E-3 (Cont.) Cooling Unit Components Component Name IPMI Type Description Values (if applicable) (Oracle ILOM CLI Targets) Threshold Fan module fan speed /SYS/MB/FM[0-5] /F[0-x]/ TACH sensor /SYS/MB/FM[0-5] / Indicator Fan Fault-Service Required LED Color: Amber...
  • Page 231 Appendix E Drive Components Table E-4 (Cont.) Disk Backplane Components Component Name IPMI Type Description Values (if applicable) (Oracle ILOM CLI Targets) Discrete Universal Serial Bus (USB) VRD error 01h-DEASSERTED /SYS/VREG_USB_ERR sensor (Hidden) 02h-ASSERTED Drive Components The following table lists the hard disk drive (HDD) and solid state drive (SSD) components.
  • Page 232 Appendix E Processor Components Table E-5 (Cont.) Drive Components Component Name IPMI Type Description Values (if applicable) (Oracle ILOM CLI Targets) Internal SSD that is NVM Express FRU /SYS/MB/SSDR[0-1]/ SSD[0-1]/NVME Discrete Indicates the SSD is PCIe (NVMExpress) 01h-ENTITY_ABSENT /SYS/MB/SSDR[0-1]/ SSD[0-1]/NVME/PRSNT...
  • Page 233 System Motherboard Components The following table lists the system motherboard components. Table E-8 System Motherboard Components Component Name IPMI Type Description Values (if applicable) (Oracle ILOM CLI Targets) General host system motherboard FRU /SYS/MB /SYS/MB/NET[0 ] Host Ethernet FRU /SYS/MB/PCIE[1-9] PCIe slot...
  • Page 234 Appendix E System Firmware Components Table E-8 (Cont.) System Motherboard Components Component Name IPMI Type Description Values (if applicable) (Oracle ILOM CLI Targets) Indicator SP OK LED /SYS/SP/OK Color: Green Location: Front panel On: SP is operating. Off: SP requires service.
  • Page 235 List of Tables Replaceable Components 1-11 Server Power Issues Server Cooling Issues Diagnostic Tool Selection Ethernet Port Transfer Rates 2-12 Serial Management Port Signals 2-13 Default Serial Connections for Serial Port 2-13 RJ-45 to DB-9 Adapter Crossovers Wiring Reference 2-13 RJ-45 to DB-25 Adapter Crossovers Wiring Reference 2-14 System Configuration Information...
  • Page 236 22TB HDD Drive Capacity and Performance B-13 Maintenance Space Requirements Rack Compatibility Oracle Rack Cabinet 1242 Rack Specifications Sun Rack II Model 1242 and Sun Rack II Model 1042 Rack Specifications Data Center Room Considerations Data Center Environment Checklist Facility Power Checklist...
  • Page 237 9-slot PCIe card configuration, chassis, aligning mounting bracket with, circuit breaker and UPS guidelines, clearance, airflow, components, access route and data center checklist, Exadata Server X11M, Access Serial Remote Host Console, 2-15 components and nomenclature, activity indicator, connecting devices to server,...
  • Page 238 Ethernet cables, connecting, Ethernet ports status indicators, Exadata Server X11M, half height PCIe configuration, Exadata Server X11M with Database Server, hard disk drive (HDD)components, Exadata Server X11M with Storage Server, hard drives, extend server, hardware faults external cables, back panel,...
  • Page 239 NVMe storage drive configuration, internal M.2 flash SSDs NVMe storage drives description, 1-11 names, inventory, ship kit, removing and replacing using Oracle Linux, known issues, 18-1 OK to Remove indicator storage drives, On/Standby LED indicator module button, 3-5, 3-7,...
  • Page 240 Index processor components, processor socket cover PCIe cards installing, 12-1 description, 1-11 removing, 12-1 installing, 11-3, 11-6 processors removing, 11-2, 11-5 components, 13-1 replacing, 11-1 description, 1-11 slot characteristics, 11-1 external cap, 13-1 slot electrical interface, 11-11 heatsink, 13-1 PCIe configuration, installing, 13-12 PCIe flyover cable...
  • Page 241 Oracle ILOM CLI, reset SP, 2-14 gracefully, using Oracle ILOM web interface, Root-of-Trust circuitry, connecting to the, 2-18 gracefully, using the On/Standby button, immediately, using Oracle ILOM CLI, immediately, using Oracle ILOM web...
  • Page 242 Index troubleshooting (continued) using LED status indicators, ventilation and cooling, using Oracle ILOM, using Oracle ILOM CLI, ventilation guidelines, unpacking guidelines, web interface, weight of server, Index-6...