Page 1
The X13OEi Platform USER'S MANUAL Revision 1.0a...
Page 2
State of California, USA. The State of California, County of Santa Clara shall be the exclusive venue for the resolution of any such disputes. Supermicro's total liability for all claims will not exceed the price paid for the hardware product.
Page 3
Socket E LGA 4677) with four UPI links (16GT/s max.) and a thermal design power (TDP) up to 350W. With support of the Intel C741 chipset, each X13OEi CPU board supports up to 8TB 3DS RDIMM/RDIMM DDR5 ECC memory with speeds up to 4800MT/s in 16 DIMMs (or up to 4400MT/s in 32 DIMMs).
Page 4
Super X13OEi Platform User's Manual Note 1: Memory speed support depends on the processors used in the system. Note 2: For TPM support, an optional part is needed. Contact Supermicro Tech Sup- port for more information. Conventions Used in the Manual...
Page 5
Super Micro Computer, Inc. 980 Rock Ave. San Jose, CA 95131 U.S.A. Tel: +1 (408) 503-8000 Fax: +1 (408) 503-8008 Email: Marketing@supermicro.com (General Information) Sales-USA@supermicro.com (Sales Inquiry: Non-Government) Government_Sales-USA@supermicro.com (Sales Inquiry: Government) Support@supermicro.com (Technical Support) RMA@supermicro.com (RMA Support) Website: www.supermicro.com Europe Address: Super Micro Computer B.V.
Page 7
Table of Contents The 4th Gen Intel Xeon Scalable Processor ..............47 Overview of the CPU Socket ....................51 Overview of the Processor Carrier Assembly (on the X13OEi-CPU Board) ....52 Overview of the Processor Heatsink Module (PHM) ............53 Creating the Processor Carrier Assembly .................54 Creating the PHM ......................58...
Page 8
Super X13OEi Platform User's Manual 4.7 Boot ..........................175 4.8 Save & Exit ........................178 Appendix A BIOS POST Codes A.1 BIOS POST Codes ......................180 Appendix B Software B.1 Microsoft Windows OS Installation ...................181 B.2 Driver Installation ......................183 B.3 BMC ..........................184 B.4 Logging into the BMC (Baseboard Management Controller) ...........184...
The X13OEi platform, which includes the X13OEi CPU boards, AOM-X13OEi-PCH, AOM- X13OEi-MP, AOM-X13OEi-BRF, AOM-X13OEi-BBR, and other related add-on modules, are intended to be used in a Supermicro proprietary 8-way or 4-way system. This chapter provides detailed information on the components installed on the X13OEi platform as well as specifications and features supported by an 8-way or 4-way system.
Super X13OEi Platform User's Manual 1.1 X13OEi-CPU (CPU Board) Your system supports two CPU modules and each CPU module supports two CPU boards (with eight CPUs total supported in the system). Each CPU board contains the following components: • Dual processors that support Intel 4th Gen Intel Xeon SPR CPUs with UPI 2.0 and 16GT/s •...
Page 11
Chapter 1: Introduction X13OEi-CPU Board Image Image of the CPU Module (Two CPU Modules in the System) Note: This system has two CPU modules and each CPU module supports two CPU boards. With two CPU modules installed, four CPU boards with eight CPUs are sup-...
Super X13OEi Platform User's Manual 1.2 AOM-X13OEi-PCH (PCH Board) Your system contains one PCH board that supports the following main components: • One Intel PCH • Two M.2 PCIe/SATA 3.0 hybrid slots in 21100/2280 • Two USB 3.0 and 2.0 connections to the rear USB 3.1 Gen1 port on the PCH board and two USB 2.0 connections for the front adapter card...
Your system contains one Middle Plane (Midplane) that supports the following: • Front side supports four X13OEi-CPU boards • Rear side supports two AOM-X13OEi-PCIe boards and one AOM-X13OEi-PCH board with six power supply connectors • Proprietary form factor in 5.70" x 9.90" (144.78 mm x 251.46 mm)
Chapter 1: Introduction Quick Reference Table (for AOM-X13OEi-MP-Front Side) Connector Description Front Control Panel (See Note 1.) Guide Post1 for CPU Board #4 Guide Post2 for CPU Board #3 Guide Post3 for CPU Board #2 Guide Post4 for CPU Board #1...
Page 26
Super X13OEi Platform User's Manual AOM-X13OEi-MP Image (Rear Side) Location of Middle Plane (One Middle Plane in the System) Middle Plane (Midplane) (Front View)
PCIe board sideband connector (See Note 1.) Guide Post5 Guide Post6 JPWR1-JPWR6 Power connectors for Supermicro proprietary 2600W power supply units (See Note 2.) JPWR_E1/JPWR_E3 Power connectors for PCIe Module#1 use (See Note 2.) JPWR_E2/JPWR_E4 Power connectors for PCIe Module#2 use (See Note 2.) Note 1: Refer to Section 2.7 in Chapter 2 for more detailed information.
Super X13OEi Platform User's Manual 1.4 X13OEi Front Bridge Board (AOM-X13OEi-BRF) Your system supports four Front Bridge boards, and each Front Bridge board supports two heatsinks • Proprietary form factor in 1.5" x 3.56" (38.10mm x 95.00mm) Locations of Front Bridge Boards (Four Front Bridge Boards in the System)
Chapter 1: Introduction Quick Reference (for AOM-X13OEi-BRF) BAR CODE AOM-X13OEi-BRF REV:1.00 BAR CODE Quick Reference Table (for AOM-X13OEi-BRF) Connector Description J1/J2 CPU UPI connectors (Refer to Section 2.7 in Chapter 2 for more information.)
Super X13OEi Platform User's Manual 1.5 AOM-X13OEi-BRR (Rear Bridge Board) Your system supports two Rear Bridge boards. • Proprietary form factor in 1.72" x 8.30" (43.69 mm x 210.82 mm) Locations of Rear Bridge Boards (Two Rear Bridge Boards in the System)
Super X13OEi Platform User's Manual Quick Reference Table (for AOM-X13OEi-BRR) Connector Description J1/J3 UPI connectors (See the note below.) J2/J4 UPI connectors (See the note below.) J5/J7 UPI connectors (See the note below.) J6/J8 UPI connectors (See the note below.)
Page 39
One Gigabit Ethernet LAN supported by Intel I210 controller Graphics (on the AOM-X13OEi-PCH Card) • Graphics controller and VGA support via ASPEED AST2600 BMC (on the AOM- X13OEi-PCH) I/O Devices (on the AOM-X13OEi-PCH Card) • One Video Display Port (DP) on the PCH card...
Page 40
• Proprietary form factor in 16.79" x 10.11" (426.47mm x 256.79mm) (X13OEi-CPU) • Proprietary form factor in 3.15" x 12.73" (80.01 mm x 323.34 mm) (AOM-X13OEi-PCH) • Proprietary form factor in 5.70" x 9.90" (144.78 mm x 251.46mm) (AOM-X13OEi-MP) •...
Page 41
Note 1: The CPU maximum thermal design power (TDP) is subject to chassis and heatsink cooling restrictions. For proper thermal management, please check the chas- sis and heatsink specifications. Note 2: For BMC configuration instructions, please refer to the Embedded BMC Con- figuration User's Guide available at http://www.supermicro.com/support/manuals/.
High Performance Computing (HPC), large scale Virtualization, in- memory database, and scientific research. The key features supported by the X13OEi platform include the following: Processor Features Supported •...
System Health sensors monitor temperatures and voltage settings of onboard processors and the system in real time via the BMC interface. Whenever the temperature of the CPU or the system exceeds Supermicro’s pre-defined threshold, the system and CPU cooling fan speed will increase to prevent the CPU or system from overheating.
Supermicro website. 1.10 Serial Port The AOM-X13OEi-PCH supports a serial port via a header on the I/O board. COM Port 1 is used for input/output. The UART provides legacy speeds with a baud rate up to 115.2 Kbps.
Touch a grounded metal object before removing the server and the components from the antistatic bag. • Handle the X13OEi CPU board and add-on cards by its edges only; do not touch compo- nents, peripheral chips, memory modules or gold contacts. •...
Thermal grease is pre-applied on a new heatsink. No additional thermal grease is needed. • Refer to the Supermicro website for updates on processor and memory support. • All graphics in this manual are for illustrations only. Your components may look different.
Page 48
Super X13OEi Platform User's Manual 1. The 4th Gen Intel Xeon Scalable Processor Processor Top View SP XCC Series CPU Key Pin 1 = Cutout = Pin 1 = CPU Key = Cutout = Pin 1 = CPU Key Processor Top View...
Page 49
Chapter 2: Installation 2. The Processor Carrier Note: The SP XCC Series CPU supports Carrier E1A. Pin 1 Carrier Top View = Cutout = Pin 1 = CPU Key = Cutout = Pin 1 = CPU Key Carrier E1A Carrier Bottom View...
Page 50
Super X13OEi Platform User's Manual 3. Heatsink 1.5U Heatsink SCALE 25.400 Note: Exercise extreme care when handling the heatsink. Pay attention to the edges of heatsink fins, which can be sharp! To avoid damaging the heatsink, please do not apply excessive force on the fins.
Super X13OEi Platform User's Manual Overview of the Processor Carrier Assembly (on the X13OEi-CPU Board) The processor carrier assembly contains a 4th Gen Intel Xeon Scalable processor and a processor carrier. Carefully follow the instructions given in the installation section to place a processor into the carrier to create a processor carrier.
Chapter 2: Installation Overview of the Processor Heatsink Module (PHM) The Processor Heatsink Module (PHM) contains a heatsink, a processor carrier, and a 4th Gen Intel Xeon Scalable processor. 1. Heatsink (Bottom View Shown Below) 2. Processor Carrier E1A 3. The 4th Gen Intel Xeon Scalable Processor (SP XCC Series) (Component Side Shown Below) 4.
Super X13OEi Platform User's Manual Creating the Processor Carrier Assembly The processor carrier assembly contains a 4th Gen Intel Xeon Scalable processor and a processor carrier. To create the processor carrier assembly, please follow the steps below: Note: Before installation, be sure to follow the instructions given on pages 1 and 2 of this chapter to properly prepare for installation.
Page 55
Chapter 2: Installation 2. First, turn over the processor carrier and locate pin 1 on the CPU and pin 1 on the carrier. Then, turn the processor over with component side (including the gold contacts) facing up and locate CPU keys on the processor. Finally, locate the CPU keys and four latches on the carrier as shown below.
Page 56
Super X13OEi Platform User's Manual 3. Locate the lever on the CPU socket and press it down as shown below. Lever Carrier E1A 4. Using pin 1 as a guide, carefully align the CPU keys (A and B) on the processor against the CPU keys on the carrier (a and b) as shown in the drawing below.
Page 57
Chapter 2: Installation 6. After the processor is placed inside the carrier, examine the four sides of the processor, making sure that the processor is properly seated on the carrier. SP XCC (Top View) (Component View)
Super X13OEi Platform User's Manual Creating the PHM After creating the processor carrier assembly, please follow the instructions below to mount the processor carrier into the heatsink to form the PHM. Note: If this is a new heatsink, the thermal grease has been pre-applied on the un- derside.
Page 59
Chapter 2: Installation CPU Carrier Assembly (CPU Component Side and Heatsink Bottom Side) SP XCC Pin 1 Processor Heatsink Module (PHM)
Super X13OEi Platform User's Manual Preparing the CPU Socket for Installation (on the CPU Board) A plastic protective cover is pre-installed on the CPU socket. Remove it from the socket by following the instructions below: 1. Press the tabs inward.
Chapter 2: Installation Preparing to Install the PHM into the CPU Socket After assembling the Processor Heatsink Module, you are ready to install it into the CPU socket. To ensure the proper installation, please follow the procedures below: 1. Locate four threaded fasteners (a, b, c, d) on the CPU socket. CPU Socket (a, b, c, d: Threaded Fasteners) CPU Socket Pin1...
Page 62
Super X13OEi Platform User's Manual 3. Check the rotating wires (1, 2, 3, 4) to make sure that they are at unlatched positions as shown in the drawing below before installing the PHM into the CPU socket. Unlatched State Side View...
Chapter 2: Installation Installing the PHM into the CPU Socket on the CPU Board 1. Align peek nut A, which is next to the triangle (pin 1) on the heatsink, against threaded fastener a on the CPU socket. Then align peek nuts B, C, and D on the heatsink against threaded fasteners b, c, and d on the CPU socket, making sure that all peek nuts on the heatsink are properly aligned with the correspondent threaded fasteners on the CPU socket.
Page 64
Super X13OEi Platform User's Manual 3. Press all four rotating wires outwards and make sure that the heatsink is securely latched onto the CPU socket. Latched State 4. With a T30 bit torque driver set to a force of 8.0 in-lbf (0.904 N-m), tighten all peek nuts in the sequence of A, B, C, and D with even pressure.
Page 65
Chapter 2: Installation 5. Tighten both thumbscrews of the heatsink onto the CPU board. 6. Examine all corners of the heatsink to ensure that the PHM is firmly attached to the CPU socket. IN-LBF 0.904 5 6 7 8 9 1 0 1 1...
Super X13OEi Platform User's Manual Removing the PHM from the CPU Socket (on the CPU Board) Before removing the PHM from theX13OEi-CPU board, be sure to shut down the system and unplug the power cables from the power supply. Then follow the steps below: 1.
Page 67
Chapter 2: Installation 3. Once the peek nuts are loosened from the CPU socket, press the rotating wires inwards to unlatch the PHM from the socket as shown in the drawings below. Unlatched State 1.5U Heatsink 4. Gently pull the PHM upwards to remove it from the CPU socket.
Super X13OEi Platform User's Manual Removing the Processor Carrier Assembly from the PHM To remove the processor carrier assembly from the PHM, please follow the steps below: 1. Detach the four plastic clips (marked a, b, c, d) on the processor carrier assembly from the four corners of the heatsink (marked A, B, C, D) as shown in the drawings below.
Page 69
Chapter 2: Installation 2. When all plastic clips are detached from the heatsink, remove the processor carrier assembly from the heatsink. 2U Heatsink (View of Component Side and Heatsink Bottom Side) SP XCC Series Heatsink (Bottom View)
Super X13OEi Platform User's Manual Removing the Processor from the Processor Carrier Assembly Once you have removed the processor carrier assembly from the PHM, you are ready to remove the processor from the processor carrier by following the steps below.
Chapter 2: Installation 2.3 Memory Support and Installation (on the CPU Board) Note: Check the Supermicro website for recommended memory modules. Important: Exercise extreme care when installing or removing memory modules to prevent any possible damage. Memory Support A CPU board supports up to 8TB 3DS RDIMM/RDIMM DDR5 (288-pin) ECC memory with speeds up to 4800MT/s in 16 DIMM slots (or 4400MT/s in 32 DIMM slots) (*Note below).
Page 72
Super X13OEi Platform User's Manual DDR5 Memory Support for 4th Gen Intel Xeon Scalable Processors DDR5 Memory Population Table for X13OEi-CPU Board (with 32 DIMMs Installed) 2 CPUs: Memory Population Sequence CPU1: P1-DIMMA1, CPU2: P2-DIMMA1 CPU1: P1-DIMME1, CPU2: P2-DIMME1 2 CPUs & 2 DIMMs...
Page 73
1. Insert the desired number of DIMMs into the memory X13OEi-CPU REV:1.02 slots based on the recommended DIMM population tables in the previous section. Locate DIMM memory slots on the X13OEi-CPU board shown on the right. CPU1 CPU2 JPWR1 MH10 JPWR2 2.
Super X13OEi Platform User's Manual 2.4 Rear I/O Connectors/Ports (on the AOM-X13OEi-PCH) Shared Ethernet LAN port and Dedicated BMC LAN Port A shared Ethernet LAN port and a dedicated BMC LAN are located at J46 and J47 on the PCH card. Both Ethernet LAN port and BMC LAN port support 1 GbE connections. The Ethernet LAN port is supported by the Intel I210 controller, and the dedicated BMC LAN, by the ASpeed AST2600 BMC (Baseboard Management Controller).
Page 75
J45. BMC LAN 1. COM1 JUSB1 2. Display Port (J45) Display Port USB0/1(3.0) Shared LAN M.2_MH2 M.2_MH4 M.2_MH1 M.2_MH3 Battery AOM-X13OEi-PCH REV:1.01 M.2-H1 M.2-H0 BAR CODE BIOS LICENSE LED1 JBT1 CMOS CLEAR MAC CODE IPMI CODE...
Page 76
Super X13OEi Platform User's Manual USB 3.2 Connector and USB 2.0 Header A USB connector with support of two USB 3.2 Gen1 ports (USB0/1) is located at JUSB1 on the PCH card. A USB 2.0 header that supports two USB 2.0 connections (USB 2/3) is located at JUSB2 on the PCH card.
A front control panel header is located at FP1 on front side of the Midplane. Front control panel header contains header pins for various buttons and LED indications with I²C support for front access. The front control panel header is designed specifically for use with Supermicro chassis.
Super X13OEi Platform User's Manual 2.5.2. Front Panel Board (FPB-FP826-TUV) The FPB-FP826-TUV supports the following components for front access: • Two USB 2.0 ports (USB1/USB2) • One VGA connector • A Power Switch (SW1) • A UID switch (SW1) •...
Page 79
Chapter 2: Installation VGA Connector (on the FPB-FP826-TUV) A VGA connector, located at JVGA1 on the FPB-FP826-TUV, provides front VGA and video display support. Refer to the layout below for the location of JVGA1. Power Switch (SW1) (on the FPB-FP826-TUV) Use the power switch located at SW1 on the FPB-FP826-TUV to turn the system power on/ off.
Page 80
Super X13OEi Platform User's Manual UID (Unit Identification) LED Switch (SW2) and Information LED (LED4) (On the FPB- FP826-TUV) A UID LED switch (SW2) is located on the front panel board. This switch can be used to identify a system unit that might be in need of service.
2.6 Connectors and Headers Power Connections System Power Supply Connectors (on the X13OEi-CPU Board) Two system power supply unit connectors, located at JPWR_E1/JPWR_E2 on the CPU board, provide main power to your system. These power connectors meet the ATX SSI EPS 12V specification and must be connected to your power supply to provide adequate power to your system.
Page 82
Super X13OEi Platform User's Manual Power Connectors for System Cooling Fans and Backplane Devices (on the X13OEi-CPU Board) An 8-pin fan power connector, located at JPWR1 on the CPU board, provides power to the system cooling fans. A 4-pin power connector, located at JPWR2, is used for backplane devices.
Page 83
Chapter 2: Installation Power Connectors for CPU Boards (on the AOM-X13OEi-MP Front Side) Eight power connectors (JPWR_S1~ JPWR_S8), located on the front side of the midplane, provide power to the CPUs on the CPU boards. Each 36-pin CPU power connector is designated for a CPU board use as specified in the table below.
Page 84
Super X13OEi Platform User's Manual PCIe Module Power Connectors (on the AOM-X13OEi-MP Rear Side) Four 36-pin power connectors, located at JPWR_E1 ~ JPWR_E4 on the rear side of the Midplane, provide power to PCIe modules. Each power connector is designated for a PCIe module use as specified below.
Page 85
Chapter 2: Installation SMC 2600W Power Supply Unit Connectors (on the AOM-X13OEi-MP Rear Side) Six Supermicro-proprietary 2600W power supply connectors are located JPWR1 ~ JPWR6 on the rear side of the Midplane. These power connectors meet the ATX SSI EPS 12V specification and must be connected to your power supply to provide adequate power to your system.
Page 86
Super X13OEi Platform User's Manual Headers SMBus Fan Header (on the X13OEi-CPU Board) A 4-pin SMBus (System Management Bus) fan header, located at JFAN1 on the CPU board, used for SMBus cooling. Fan speed control for this fan is supported by Thermal Management via the BMC 2.0 interface.
Page 87
The JTPM1 header, located on the PCH card, is used to connect a Trusted Platform Module (TPM)/Port 80, which is available from Supermicro (optional). A TPM/Port 80 connector is a security device that supports encryption and authentication in hard drives. It allows the server to deny access if the TPM-associated with the hard drive is not installed in the system.
Page 88
Super X13OEi Platform User's Manual VROC RAID Key Header (on the AOM-X13OEi-PCH) A VROC RAID Key header is located at JRK1 on the PCH card. Install a VROC RAID key on JRK1 for NVMe RAID support as shown in the illustration below. Please refer to the layout below for the location of JRK1.
Page 89
Chapter 2: Installation PCIe 3.0 M.2-H1/M.2-H2 Hybrid Slots and M.2 Mounting Holes (on the AOM-X13OEi- PCH) Two PCIe 3.0/SATA3 Hybrid M.2 slots are located at M.2-H0 (J6) and M.2-H1 (J7) on the PCH card. These M.2/SATA3 Hybrid slots support PCIe 3.0 x4 M.2 NVMe/SATA3 SSDs in the 2280 and 22110 form factors.
Page 90
Super X13OEi Platform User's Manual I-SATA0/I-SATA1 3.0 Connectors (on the X13OEi-CPU Board) Two SATA 3.0 connectors are located at SATA1/SATA2 on the CPU board. Each I-SATA connector supports eight SATA 3.0 connections. These SATA 3.0 ports are supported by the Intel C741 PCH. Connecting proper SATA cables to SATA1/SATA2 to use SATA 3.0 connections.
Three UPI 2.0 connectors supported by CPU2: P2_UPI0/ P2_UPI1/ P2_UPI3 (J4&J5, J8, J40) Note: Please refer to the X13OEi-CPU board section in Chapter 1 for the locations of PCIe connectors and UPI 2.0 connectors B. The AOM-X13OEi-MP Midplane (Rear Side) •...
Note: On two-pin jumpers, "Closed" means the jumper is on and "Open" means the jumper is off the pins. LAN Port Enable/Disable (on the AOM-X13OEi-PCH) Use Jumper JPL1, located on the PCH card, to enable or disable the shared Ethernet LAN port (J46).
Page 93
Chapter 2: Installation Watch Dog Enable Select (on the AOM-X13OEi-PCH) A Watch Dog Timer Enable jumper is located at JWD1 on the PCH card. Watch Dog Timer is a system monitor that can reboot the system when a software application hangs. Close pins 1-2 of the Watch Dog jumper (JDW1) to reset the system if an application hangs.
Page 94
Super X13OEi Platform User's Manual CMOS Clear (on the AOM-X13OEi-PCH) JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS. To Clear CMOS 1.
Page 95
Chapter 2: Installation Video Display Port Enable/Disable (on the AOM-X13OEi-PCH) Jumper JPG1, located on the PCH card, is used enable or disable the video display port (J45). The default setting is on pins 1-2 to enable the connection. Video Display Port Enable/Disable Jumper (JPG1)
Page 96
Super X13OEi Platform User's Manual ME (Manufacturing Mode) Recovery (on the AOM-X13OEi-PCH) JPME1, located on the PCH card, is used for ME Firmware Recovery mode, which will limit system resource for essential function use only without putting restrictions on power use. In the single operation mode, online upgrade will be available via Recovery mode.
Page 97
Chapter 2: Installation BIOS Recovery (on the AOM-X13OEi-PCH) A BIOS Recovery jumper is located at JBR1 on the PCH card. Use this jumper to recover BIOS settings, which will reset and revert the current BIOS settings back to the original manufacturer settings.
Super X13OEi Platform User's Manual 2.9 LED Indicators LAN LEDs (on the AOM-X13OEi-PCH) A shared Ethernet LAN port, located at 46 on the PCH card, has two LEDs. The green LED indicates activity, while the other Link LED may be green, amber, or off to indicate the speed of the connection.
Page 99
Chapter 2: Installation BMC LAN LEDs (on the AOM-X13OEi-PCH) A dedicated BMC LAN LED port is located at J47 on the PCH card on the rear side of the server. There are two LED indicators on this LAN port. The LED on the right indicates LAN activity, while the LED on the left, the speed of the LAN connection.
Page 100
Super X13OEi Platform User's Manual BMC Heartbeat LED (on the AOM-X13OEi-PCH) A BMC Heartbeat LED is located at LED1 on the PCH card. When LED1 is blinking green, the BMC is functioning normally. Refer to the layout below for the location of LED1.
Page 101
Chapter 2: Installation HDD Activity LED (on the FPB-FB826-TUV) An HDD Activity LED is located at LED1 on the FPB-FB826-TUV. When LED1 is blinking green, HDD devices are active. Refer to the layout below for the location of LED1. HDD Activity LED Indicator (LED1) LED Color Definition...
Page 102
Super X13OEi Platform User's Manual Power LED (on the FPB-FB826-TUV) LED2, located on the FPB-FB826-TUV front panel board, displays the status of your system power. When this LED is on, your system power is on. Be sure to unplug your power cable from the power supply before changing system components.
Page 103
Chapter 2: Installation Information LED (UID LED & Failure LED for OH/Fan Fail/PWR Fail) (on the FPB- FB826-TUV) The Information LED, located at LED4, has two functions. It can be used as a UID LED and a Failure LED. When used as a UID LED, LED4 will turn solid blue to indicate a unit that may be in need of service is identified.
Before Power On 1. Make sure that there are no short circuits between the X13OEi CPU board, all related add-on modules, and the chassis. 2. Disconnect all ribbon/wire cables from the server, including those for the keyboard and mouse.
Page 105
CMOS setup information. Refer to Chapter 1 for details on recommended power supplies. 2. The battery on your server, which is located on the AOM-X13OEi-PCH card, may be old. Check to verify that it still supplies ~3VDC. If it does not, replace it with a new one.
Page 106
Super X13OEi Platform User's Manual 2. Memory support: Make sure that the memory modules are supported by testing the modules using memtest86 or a similar utility. Note: Click on the "Tested Memory List" link on the product page to see a list of sup- ported memory.
Before contacting Technical Support, please take the following steps. Also, please note that as a computer manufacturer, Supermicro also sells computer systems through its channels, so it is best to first check with your distributor or reseller for troubleshooting services. They should know of any possible problems with the specific system configuration that was sold to you.
BIOS before downloading. Note 1: The SPI BIOS chip used on this server, which is located on the AOM- X13OEi -PCH card, cannot be removed. Send your machine back to our RMA Department at Supermicro for repair.
Chapter 3: Troubleshooting 3.4 Battery Removal and Installation (on the AOM- X13OEi-PCH) Battery Removal To remove the onboard battery, follow the steps below: 1. Power off your system and unplug your power cable. 2. Locate the onboard battery as shown below.
Shipping and handling charges will be applied for all orders that must be mailed when service is complete. For faster service, you can also request a RMA authorization online (http://www.supermicro.com/RmaForm/). This warranty only covers normal consumer use and does not cover damages incurred in shipping or from failure due to the alternation, misuse, abuse or improper maintenance of products.
UEFI BIOS 4.1 Introduction This chapter describes the AMIBIOS™ Setup utility for the X13OEi motherboard. The BIOS is stored on a chip and can be easily upgraded using the BMC WebUI or the SUM utility. Note: Due to periodic changes to the BIOS, some settings may have been added or deleted and might not yet be recorded in this manual.
Super X13OEi User's Manual 4.2 Main Setup When you first enter the AMI BIOS Setup utility, you will see the Main setup screen. You can always return to the Main setup screen by selecting the Main tab on the top of the screen.
Page 113
Chapter 4: UEFI BIOS Memory Information Total Memory This feature displays the total size of memory available in the system.
Super X13OEi User's Manual 4.3 Advanced Setup Configurations Use the arrow keys to select the Advanced submenu and press <Enter> to access the submenu items: Warning: Take caution when changing the Advanced settings. An incorrect value, an improper DRAM frequency, or a wrong BIOS timing setting may cause the system to malfunction. When this occurs, restore the setting to the manufacturer default setting.
Page 115
USB ports without rebooting the system. The options are Enabled, Disabled, and Enabled (Dynamic). Note: To fully utilize the functionality and features supported by Supermicro Management software and utilities, please use the Supermicro DataCenter Management Suite per Node License Key (SFT-DCMS-SINGLE), which is the license to the Supermicro’s...
Page 116
Super X13OEi User's Manual Rear USB Port(s) (This feature will display only when DCMS key is activated) Select Enabled to allow the specific type of USB devices to be used in the rear USB ports. Select Enabled (Dynamic) to allow or disallow this particular type of USB devices to be used in the rear USB ports without rebooting the system.
Page 117
Chapter 4: UEFI BIOS Processor Configuration The following CPU information is displayed: • Processor BSP Revision • Processor Socket • Processor ID • Processor Frequency • Processor Max Ratio • Processor Min Ratio • Microcode Revision • L1 Cache RAM (Per Core) •...
Page 118
Super X13OEi User's Manual CPU P State Control This feature allows you to configure the following CPU power settings: AVX P1 (Available when "SpeedStep (P-States)" is set to Enable) Use this feature to set the appropriate TDP level for the system. The Intel Advanced Vector Extensions (Intel AVX) P1 feature allows you to set the base P1 ratio for Streaming SIMD Extensions (SSE) and AVX workloads.
Page 119
Chapter 4: UEFI BIOS Hardware PM State Control Hardware P-States If this feature is set to Disable, system hardware will choose a P-state setting for the system based on an OS request. If this feature is set to Native Mode, hardware will choose a P-state setting based on the OS guidance.
Page 120
Super X13OEi User's Manual CPU1 Core Disable Bitmap ~ CPU8 Core Disable Bitmap Available Bitmap: This feature displays the available bitmap. Disable Bitmap Enter 0 to enable this feature for all CPU cores. Enter FFFFFFFFFFF to disable this feature for all CPU cores.
Page 121
Chapter 4: UEFI BIOS FB Thread Slicing Select Enable to support Fill Buffer (FB) Thread Slicing on every thread. The options are Disable and Enable. Extended APIC (Extended Advanced Programmable Interrupt Controller) Based on the Intel Hyper-Threading technology, each logical processor (thread) is assigned 256 APIC IDs (APIDs) in 8-bit bandwidth.
Page 122
Super X13OEi User's Manual Total Memory Encryption (TME) Bypass (Available when "Memory Encryption (TME)" is set to Enabled) Use this feature to disable/enable the TME function for physical memory protection. The options are Auto, Disabled, and Enabled. Total Memory Encryption Multi-Tenant (TME-MT) (Available when "Memory Encryption (TME)"...
Page 123
Chapter 4: UEFI BIOS SGX Factory Reset Use this feature to perform an SGX factory reset to delete all registration data and force an Initial Platform Establishment flow. Reboot the system for the changes to take effect. The options are Disabled and Enabled. SW Guard Extensions (SGX) Use this feature to enable Intel Software Guard Extensions (SGX) support.
Page 124
Super X13OEi User's Manual SGXLEPUBKEYHASHx Write Enable (Available when "SW Guard Extensions (SGX)" is set to Enabled) Use this feature to enable writes to SGXLEPUBKEYHASH[3..0] from OS/SW. The options are Disabled and Enabled. Only those CPUs that support Intel SGX Flexible Launch Control (FLC) feature have SGXLEPUBKEYHASH, which contains the hash of the public key for the SGX Launch Enclave (LE) to be signed with.
Page 125
Chapter 4: UEFI BIOS Chipset Configuration Warning: Setting the wrong values in the following features may cause the system to malfunc- tion. North Bridge This feature allows you to configure the following North Bridge settings. Uncore Configuration The following information is displayed. •...
Page 126
Super X13OEi User's Manual KTI Prefetch Use this feature to configure the Prefetch setting supported by Keizer Technology Interconnect (KTI), also known as Intel Ultra Path Interconnect (UPI) Technology. Select Enable for the KTI prefetcher to preload the L1 cache with data deemed relevant, which will allow the memory read to start earlier on a DDR bus in an effort to reduce latency.
Page 127
Chapter 4: UEFI BIOS Memory Configuration This feature allows you to configure the Integrated Memory Controller (iMC) settings. Enforce DDR Memory Frequency POR Select POR to enforce Plan of Record (POR) restrictions for DDR memory frequency and voltage programming. The options are POR and Disable. Memory Frequency Use this feature to set the maximum memory frequency for onboard memory modules.
Page 128
Super X13OEi User's Manual Core Ratio Value (Available when "Core Ratio" is set to Manual) Use this feature to enter the core ratio value. Mesh Ratio Use this feature to set the mesh ratio to be involved in the fADR event. The mesh ratio determines the frequency of data access between CPU cores and caches.
Page 129
Chapter 4: UEFI BIOS UEFI ARM Mirror (Available when "ADDDC Sparing" is set to Disabled and "Mirror Mode" is set to Disabled) If this feature is set to Enable, mirror mode configuration settings for UEFI-based Address Range memory will be enabled upon system boot. This will create a duplicate copy of data stored in the memory to increase memory security, but it will reduce the memory capacity into half.
Page 130
Super X13OEi User's Manual Enhanced PPR Use this feature to set advanced memory test for PPR enhancement. Select Enabled to always execute for every boot. Select Once to execute only one time. The options are Disabled, Enabled, and Once. Memory PFA Support (Available when the DCMS key is activated) Select Enabled to enable memory Predictive Failure Analysis (PFA) support.
Page 131
Chapter 4: UEFI BIOS Socket0 Port DMI/ Socket0 Port 1A / Socket0 Port 2A / Socket0 Port 3A / Socket0 Port 4A / Socket0 Port 4C / Socket0 Port 4E / Socket0 Port 4G / Socket0 Port 5A / Socket0 Port 5C / Socket0 Port 5E / Socket0 Port 5G Note: The number of slots and the slot naming vary based on your motherboard features.
Page 132
Super X13OEi User's Manual Opt-Out Illegal MSI Mitigation If this feature is set to Enable, "Illegal OxzFEE Platform Mitigation" will be opted out. The options are Enable and Disable. Interrupt Remapping (Available when "Intel® VT for Directed I/O (VT-d)" is set to Enable) Select Enable to support I/O DMA transfer remapping and device-generated interrupts.
Page 133
Chapter 4: UEFI BIOS VMD Config for IOU 0 / VMD Config for IOU 1 / VMD Config for IOU 2 / VMD Config for IOU 3 / VMD Config for IOU 4 ((Available on Socket 0 ~ Socket 7) Enable/Disable VMD Select Enable to enable the Intel VMD technology support for the root port specified.
Page 134
Super X13OEi User's Manual South Bridge The following information is displayed: • USB Module Version • USB Devices: Legacy USB Support Select Enabled to support onboard legacy USB devices. Select Auto to disable legacy support if there are no legacy USB devices present. Select Disabled to have all USB devices available for EFI applications only.
Page 135
Chapter 4: UEFI BIOS PCH SATA0 Configuration / PCH SATA1 Configuration / SATA2 Configuration When this submenu is selected, the AMI BIOS automatically detects the presence of the SATA devices that are supported by the Intel PCH chip and displays the following features. SATA Controller(s) This feature enables or disables the onboard SATA controller(s) supported by the Intel PCH chip.
Page 136
Super X13OEi User's Manual SATA Device Type Use this feature to specify if the device installed on the SATA port specified by the user should be connected to a Solid State drive or a Hard Disk Drive. The options are Hard Disk Drive and Solid State Drive.
Page 137
Chapter 4: UEFI BIOS Serial Port 2 Attribute (Available for Serial Port 2 only) Select SOL to use serial port 2 as a Serial Over LAN (SOL) port for console redirection. The options are SOL and COM. Serial Port Console Redirection ...
Page 138
Super X13OEi User's Manual Stop Bits A stop bit indicates the end of a serial data packet. Select 1 Stop Bit for standard serial data communication. Select 2 Stop Bits if slower devices are used. The options are 1 and 2.
Page 139
Chapter 4: UEFI BIOS Console Redirection Settings (Available when "Console Redirection" above is set to Enabled) Use this feature to specify how the host computer will exchange data with the client computer. Terminal Type Use this feature to select the target terminal emulation type for Console Redirection. Select VT100 to use the ASCII Character set.
Page 140
Super X13OEi User's Manual Recorder Mode Select Enabled to capture the data displayed on a terminal and send it as text messages to a remote server. The options are Disabled and Enabled. Resolution 100x31 Select Enabled for extended-terminal resolution support. The options are Disabled and Enabled.
Page 141
Chapter 4: UEFI BIOS Console Redirection Settings (Available when "Console Redirection EMS" above is set to Enabled) Out-of-Band Mgmt (Management) Port The feature selects a serial port in a client server to be used by the Microsoft Windows Emergency Management Services (EMS) to communicate with a remote host server. The options are COM1 and SOL.
Page 142
Super X13OEi User's Manual Network Configuration Network Stack Select Enabled to enable Preboot Execution Environment (PXE) or Unified Extensible Firmware Interface (UEFI) for network stack support. The options are Disabled and Enabled. IPv4 PXE Support (Available when "Network Stack" is set to Enabled) Select Enabled to enable IPv4 PXE boot support.
Page 143
Chapter 4: UEFI BIOS • Host address • Route Table • Gateway addresses • DNS addresses Interface ID Use this feature to change/enter the 64-bit alternative interface ID for the device. The string format is colon separated. The default setting is the MAC address above. DAD Transmit Count This feature displays the number of times that Duplicate Address Detection (DAD) has been performed.
Page 144
Super X13OEi User's Manual MAC:(MAC address)-IPv4 Network Configuration Note 1: This feature is available when "Onboard LAN Option ROM Type" is set to EFI. Note 2: The Ethernet controller and MAC addresses shown above are based on your system configurations.
Page 145
Chapter 4: UEFI BIOS PCIe/PCI/PnP Configuration The following information is displayed: • PCI Bus Driver Version PCI Devices Common Settings: Above 4G Decoding (Available when the system supports 64-bit PCI decoding) Select Enabled to decode a PCI device that supports 64-bit in the space above 4G Address. The options are Disabled and Enabled.
Page 146
The other option, AMI Native Support, is offered by the AMI BIOS with a generic method. (Please use the AMI Native Support option for Supermicro Security Erase Configuration support available in the Security menu.)
Page 147
Chapter 4: UEFI BIOS WHEA Support Select Enabled to support the Windows Hardware Error Architecture (WHEA) platform and provide a common infrastructure for the system to handle hardware errors within the Windows OS environment to reduce system crashes and to enhance system recovery and health monitoring.
Page 148
Super X13OEi User's Manual Pending Operation (Available when "Security Device Support" is set to Enable) Use this feature to schedule a TPM-related operation to be performed by a security (TPM) device at the next system boot to enhance system data integrity. The options are None and TPM Clear.
Page 149
Use this feature to enter the second Supermicro KMS server IPv4 address in dotted-decimal notation. Supermicro KMS TCP Port number Use this feature to enter the Supermicro KMS TCP port number. The valid range is 100 - 9999. The default setting is 5696. KMS Time Out Use this feature to enter the KMS server connecting timeout (in seconds).
Page 150
Super X13OEi User's Manual Client UserName Press <Enter> to set the client identity (UserName). The maximum length is 63 characters. Client Password Press <Enter> to set the client identity (Password). The maximum length is 31 characters. KMS TLS Certificate / Size This feature displays the Transport Layer Security (TLS) certificate and its size for CA Certificate, Client Certificate, and Client Private Key.
Page 151
Chapter 4: UEFI BIOS TPM Security Policy When this feature is set to Enabled, passwords will be installed to protect the system and storage devices as specified, and these passwords will be required to unlock the system or devices. When this feature is set to Disabled, there will be no passwords used to protect the system and storage devices.
Page 152
Super X13OEi User's Manual Intel(R) Ethernet Controller (Ethernet controller) -(MAC address) Note 1: This feature is available when "Onboard LAN Option ROM Type" is set to EFI. Note 2: The Ethernet controller and MAC addresses shown above are based on your system configurations.
Page 153
Chapter 4: UEFI BIOS Blink LEDs Use this feature to identify the physical network port by blinking the associated LED indicators. The default setting is 0 (up to 15 seconds). The following information is displayed: • UEFI Driver • Adapter PBA •...
Page 154
Super X13OEi User's Manual Discard Changes and Exit Use this feature to discard all changes and exit TLS settings. Delete Certification This feature is used to delete the certificate if a certificate has been enrolled in the system. The options are Disabled and Enabled.
Page 155
Chapter 4: UEFI BIOS Create RAID Volume Name: This feature allows you to enter the unique name of RAID volume. RAID Level: This feature allows you to select the RAID level. The options are RAID0(Stripe), RAID1(Mirror), RAID5(Parity), and RAID10(RAID0+1). Select Disks: Select the desired RAID disks one by one by setting them to X.
Page 156
Super X13OEi User's Manual Create RAID Volume Name: This feature allows you to enter the unique name of RAID volume. RAID Level: This feature allows you to select the RAID level. The options are RAID0(Stripe), RAID1(Mirror), RAID5(Parity), and RAID10(RAID0+1). Select Disks: Select the desired RAID disks one by one by setting them to X.
Page 157
Chapter 4: UEFI BIOS Create RAID Volume Name: This feature allows you to enter the unique name of RAID volume. RAID Level: This feature allows you to select the RAID level. The options are RAID0(Stripe), RAID1(Mirror), RAID5(Parity), and RAID10(RAID0+1). Select Disks: Select the desired RAID disks one by one by setting them to X.
Page 158
Super X13OEi User's Manual VLAN Configuration (MAC: 3CECEF3DD9DE) Note: The VLAN Configuration settings shown in this section are based on your system features Enter Configuration Menu Create New VLAN This feature allows the user to create a new VLAN. VLAN ID Use this feature to create a new LAN ID by using an existing VLAN or creating a new VLAN ID.
Page 159
Chapter 4: UEFI BIOS • DNS Addresses Interface ID This feature displays the Interface ID used in the network. DAD (Duplicate Address Detection) Transmit Count This feature displays the DAD Transmit Count. The default setting is 1. Policy Use this feature to select how the policy is to be configured. The options are Automatic and Manual.
Page 160
Super X13OEi User's Manual Driver Health This feature displays the health information of the drivers installed in your system, including LAN controllers, as detected by the BIOS. Select one and press <Enter> to see the details. Note: This section is provided for reference only. Driver health status differs depending on the drivers installed in your system.
Chapter 4: UEFI BIOS 4.4 Event Logs Use this feature to configure Event Logs settings. Note: After you've made any changes in this section, please be sure to reboot the system for the changes to take effect. Change SMBIOS Event Log Settings Enabling/Disabling Options SMBIOS Event Log Select Enabled to enable System Management BIOS (SMBIOS) Event Logging during system...
Page 162
Super X13OEi User's Manual SMBIOS Event Log Standard Settings Log System Boot Event (Available when "SMBIOS Event Log" is set to Enabled) Select Enabled to log system boot events. The options are Enabled and Disabled. MECI (Available when "SMBIOS Event Log" is set to Enabled) Enter the increment value for the multiple event counter.
Chapter 4: UEFI BIOS 4.5 BMC Use this feature to configure BMC settings. BMC Firmware Revision This feature indicates the BMC firmware revision used in your system. BMC STATUS This feature indicates the status of the BMC firmware installed in your system. System Event Log Enabling/Disabling Options SEL Components...
Page 164
Super X13OEi User's Manual When SEL is Full This feature allows you to determine what the BIOS should do when the system event log is full. Select Erase Immediately to erase all events in the log when the system event log is full.
Page 165
Chapter 4: UEFI BIOS Subnet Mask (Available when "Configuration Address Source" is set to Static) This feature displays the sub-network that this computer belongs to. The value of each three- digit number separated by dots should not exceed 255. Station MAC Address (Available when "Configuration Address Source" is set to Static) This feature displays the Station MAC address for this computer.
Page 166
Super X13OEi User's Manual Prefix Length (Available when "Configuration Address Source" is set to Static Configuration) This feature displays the prefix length. Press <Enter> to change the setting. Gateway IP (Available when "Configuration Address Source" is set to Static Configuration) Use this feature to enter the IPv6 gateway IP address.
BIOS Setup utility. The options are Setup and Always. Note: For more information on Security Boot configuration and Secure Erase instructions, please refer to Security Boot Configuration User's Guide and Secure Erase Instructions User's Guide posted on the web page under the link: http://www. supermicro.com/support/manuals/.
Page 168
4. Press <Del> at bootup to return to the AMI BIOS Setup Utility. 5. When AMI BIOS screen displays, using arrow keys, select Security from the BIOS menu bar on the top of the screen. The Supermicro Security Erase Configuration submenu will display on the Security menu as shown below.
Page 169
HDD User Pwd Status: This feature indicates if a password has been set as a SATA user password which will allow the user to configure Supermicro Security Erase settings on the selected HDD (SATA) device by using this SATA user password.
Page 170
Super X13OEi User's Manual Password Use this feature to set the SATA user password which allows you to configure the Supermicro Security Erase settings by using the SATA user password. Lockdown Mode (Available when the DCMS key is activated) Select Enabled to support Lockdown Mode that will prevent the existing data or keys stored in the system from being altered or changed in an effort to preserve system integrity and data security.
Page 171
Chapter 4: UEFI BIOS The following information is displayed: • System Mode • Vendor Keys • Secure Boot Secure Boot Select Enabled to configure Secure Boot settings. The options are Disabled and Enabled. Secure Boot Mode Use this feature to select the desired secure boot mode for the system. The options are Standard and Custom.
Page 172
Super X13OEi User's Manual Reset To Setup Mode (Available when any secure keys have been installed) This feature resets the system to the Setup Mode. The options are Yes and No. Export Secure Boot Variables (Available when a secure key has been installed) This feature exports the NVRAM contents of secure boot variables to a storage device.
Page 173
Chapter 4: UEFI BIOS Forbidden Signatures Use this feature to enter and configure a set of values to be used as Forbidden Signatures for the system. These values also indicate sizes, key numbers, and key sources of the forbidden signatures. Select Update to update your "Forbidden Signatures". Select Append to append your "Forbidden Signatures".
Page 174
Super X13OEi User's Manual Set Admin Password Use this feature to set the administrator password for this storage device. Device Reset Use this feature to reset the password configuration for this storage device.
Chapter 4: UEFI BIOS 4.7 Boot Use this feature to configure Boot settings: Boot Mode Select Use this feature to select the type of devices from which the system will boot. The options are Legacy, UEFI, and Dual. Note: When "Boot Mode Select" is set to Dual, all OPROM-related features will be set to Legacy.
Page 176
Super X13OEi User's Manual Add New Boot Option (Available when any storage device is detected by the BIOS) This feature allows you to add a new boot option to the boot priority features for system boot. Add boot option This feature allows you to specify the name for the new boot option.
Page 177
Chapter 4: UEFI BIOS USB Key Drive BBS Priorities Use this feature to set the system boot order of USB devices detected by the BIOS. NETWORK Drive BBS Priorities Use this feature to set the system boot order of Network devices detected by the BIOS.
Super X13OEi User's Manual 4.8 Save & Exit Select Save & Exit from the BIOS Setup screen to configure the settings below. Save Options Discard Changes and Exit Use this feature to exit from the BIOS Setup utility without making any permanent changes to the system configuration and reboot the computer.
Page 179
Chapter 4: UEFI BIOS Default Options Restore Optimized Defaults Select this feature and press <Enter> to load manufacturer optimized default settings which are intended for maximum system performance but not for maximum stability. Save as User Defaults Select this feature and press <Enter> to save the changes specified by the user as "User Default values"...
Super X13OEi Platform User's Manual Appendix A BIOS POST Codes A.1 BIOS POST Codes The AMI BIOS supplies additional checkpoint codes, which are documented online at http:// www.supermicro.com/support/manuals/ ("AMI BIOS POST Codes User's Guide"). For information on AMI updates, please refer to http://www.ami.com/products/.
USB/SATA media drive, or a USB flash drive, or the BMC KVM console. 2. Retrieve the proper RST/RSTe driver. Go to the Supermicro web page for your server and click on "Download the Latest Drivers and Utilities", select the proper driver, and copy it to a USB flash drive.
Page 182
Super X13OEi Platform User's Manual 4. During Windows Setup, continue to the dialog box where you select the drives on which to install Windows. If the disk you want to use is not listed, click on “Load driver” link at the bottom left corner.
The Supermicro website contains drivers and utilities for your system at https://www. supermicro.com/wdl/driver. Some of these must be installed, such as the chipset driver. After accessing the website, go into the CDR_Images (in the parent directory of the above link) and locate the ISO file for your server. Download this file to a USB flash drive or a media drive.
AOC-X13OEi PCH card. When logging in to the BMC for the first time, please use the unique password provided by Supermicro to log in. You can change the unique password to a user name and password of your choice for subsequent logins.
The following statements are industry standard warnings, provided to warn the user of situations where bodily injury may occur. Should you have questions or experience difficulty, contact Supermicro's Technical Support department for assistance. Only certified technicians should attempt to install or configure components.
Page 186
Super X13OEi Platform User's Manual Attention Danger d'explosion si la pile n'est pas remplacée correctement. Ne la remplacer que par une pile de type semblable ou équivalent, recommandée par le fabricant. Jeter les piles usagées conformément aux instructions du fabricant.
Page 187
Appendix C: Standardized Warning Statements Product Disposal Warning! Ultimate disposal of this product should be handled according to all national laws and regulations. 製品の廃棄 この製品を廃棄処分する場合、 国の関係する全ての法律 ・ 条例に従い処理する必要があります。 警告 本产品的废弃处理应根据所有国家的法律和规章进行。 警告 本產品的廢棄處理應根據所有國家的法律和規章進行。 Warnung Die Entsorgung dieses Produkts sollte gemäß allen Bestimmungen und Gesetzen des Landes erfolgen.
Need help?
Do you have a question about the X13OEi and is the answer not in the manual?
Questions and answers