Table of Contents

Advertisement

Quick Links

SuperWorkstation
540A-TR
USER'S MANUAL
Revision 1.0b

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the SuperWorkstation 540A-TR and is the answer not in the manual?

Questions and answers

Summary of Contents for Supermicro SuperWorkstation 540A-TR

  • Page 1 SuperWorkstation 540A-TR USER’S MANUAL Revision 1.0b...
  • Page 2 State of California, USA. The State of California, County of Santa Clara shall be the exclusive venue for the resolution of any such disputes. Supermicro's total liability for all claims will not exceed the price paid for the hardware product.
  • Page 3 If you have any questions, please contact our support team at: support@supermicro.com This manual may be periodically updated without notice. Please check the Supermicro website for possible updates to the manual revision level. Secure Data Deletion A secure data deletion tool designed to fully erase all data from storage devices can be found on our website: https://www.supermicro.com/about/policies/disclaimer.cfm?url=/wdl/utility/...
  • Page 4: Table Of Contents

    Preface Contents Contacting Supermicro ......................9 Chapter 1 Introduction 1.1 Overview ..........................10 1.2 System Features ........................11 Front View .........................11 Control Panel .........................13 Rear View ..........................14 1.3 System Block Diagram .......................15 1.4 Motherboard Layout ......................16 Quick Reference Table ......................18 Chapter 2 Installation 2.1 Overview ..........................20...
  • Page 5 Preface Chapter 3 Maintenance and Component Installation 3.1 Removing Power .......................29 3.2 Accessing the System ......................30 3.3 Static-Sensitive Devices .....................33 Precautions ........................33 Unpacking .........................33 3.4 Processor and Heatsink Installation ...................34 The 3rd Gen. Intel Xeon Scalable Processor ..............34 1. The 3rd Gen. Intel Xeon Scalable Processor ............35 2.
  • Page 6 Preface Installing Components in the 5.25" Drive Bays .............58 Removing the Empty Drive Bay ..................58 Adding a DVD-ROM Drive (optional)................58 M.2 SSD Installation ......................59 2280 M.2 NVMe SSD Module Installation ................60 2260 M.2 NVMe SSD Module Installation ................60 22110 M.2 NVMe SSD Module Installation...............61 System Cooling .........................62 System Fan Failure ......................62 Replacing System Fans ....................62...
  • Page 7 Preface Supported SSDs and Operating Sytems ................94 Additional Information .......................95 Hardware Key ........................95 Configuring NVMe RAID Manually ..................96 Status Indications ......................101 Hot-Swap Drives ......................101 Hot-unplug ........................101 Hot-plug ........................101 Related Information Links ....................101 Chapter 7 Troubleshooting and Support 7.1 Information Resources ......................102 Website ...........................102 Direct Links for the 540A-TR System ................102 Direct Links for General Support and Information .............102...
  • Page 8 Preface Returning Merchandise for Service .................116 Vendor Support Filing System ..................117 7.11 Feedback ..........................117 7.12 Contacting Supermicro ....................118 Appendix A Standardized Warning Statements for AC Systems Appendix B System Specifications Appendix C BSMI RoHS and Warnings...
  • Page 9: Contacting Supermicro

    San Jose, CA 95131 U.S.A. Tel: +1 (408) 503-8000 Fax: +1 (408) 503-8008 Email: marketing@supermicro.com (General Information) Sales-USA@supermicro.com (Sales Inquiries) Government_Sales-USA@supermicro.com (Gov. Sales Inquiries) support@supermicro.com (Technical Support) RMA@supermicro.com (RMA Support) Webmaster@supermicro.com (Webmaster) Website: www.supermicro.com Europe Address: Super Micro Computer B.V.
  • Page 10: Chapter 1 Introduction

    4U Tower/ Rackmount 18.1 x 7 x 26.5in. / 460 x 178 x 673mm (HxWxD) Notes: A Quick Reference Guide can be found on the product page of the Supermicro website. The following safety models associated with the 540A-TR have been certified as compliant...
  • Page 11: System Features

    Chapter 1: Introduction 1.2 System Features The following views of the system display the main features. Refer to Appendix B for additional specifications. Front View Control Panel Bezel Lock Drive Bays Figure 1-1. Front View 540A-TR System Features: Front Feature Description Control Panel Front control panel (see...
  • Page 12 Chapter 1: Introduction Power Supply Indicators Power Supply Condition Green LED Amber LED No AC Power to Power Supply Power Supply critical events causing a shutdown/ failure/ OCP/ OVP/ Fan Fail/ Amber LED OTP/ UVP Power Supply Warning Events Where the power supply continues to operate;...
  • Page 13: Control Panel

    Chapter 1: Introduction Control Panel NIC (LAN2) LED Power Fail LED USB 3.2 Gen 1 Line Out Information Power LED Reset LED NIC (LAN1) LED Figure 1-2. Control Panel Control Panel Features Feature Description The main power switch is used to apply or remove power from the power supply to the workstation.
  • Page 14: Rear View

    Chapter 1: Introduction Rear View Fans Fans Power Power I/O Backpanel PCIe Slots Figure 1-3. System: Rear View System Features: Rear Feature Description Power 1+1 2200W Titanium Redundant Power Supply with PMbus Fans Two 8cm PWM hot-swap rear fans I/O Backpanel Rear input/output ports (details in Chapter 4) PCIe Slots Eleven full-height, full-length PCIe slots...
  • Page 15: System Block Diagram

    Chapter 1: Introduction 1.3 System Block Diagram The block diagram below shows the connections and relationships between the subsystems and major components of the overall system. CPU1-G1/G2 CPU1-H1/H2 CPU1-E1/E2 CPU1-F1/F2 CPU1-B1/B2 CPU1-A1/A2 CPU1-D1/D2 CPU1-C1/C2 VCCP0 VPP (2.625) 5v VDDQ (1.2) 12v VR13.HC VTT_M (0.6) 7+1 PHASE...
  • Page 16: Motherboard Layout

    Chapter 1: Introduction 1.4 Motherboard Layout Below is a layout of the X12SPA-TF motherboard with jumper, connector, and LED locations shown. See the table on the following page for descriptions. For detailed descriptions, pinout information, and jumper settings, refer to Chapter 4 or the Motherboard Manual.
  • Page 17 Chapter 1: Introduction Notes: • " " indicates the location of Pin 1. • Jumpers/LED indicators not indicated are used for testing only. • Use only the correct type of onboard CMOS battery as specified by the manufacturer. Do not install the onboard battery upside down to avoid possible explosion.
  • Page 18: Quick Reference Table

    Chapter 1: Introduction Quick Reference Table Jumper Description Default Setting JBT1 Clear CMOS (Onboard) Short Pads to Clear CMOS USB11/12 Disable Pins 1-2 (Normal) USB4 Disable Pins 1-2 (Normal) JPAC1 HD Audio Enable/Disable Pins 1-2 (Enabled) PFR Operation JPFR2 Open (Normal PFR Operation) (Close: force power on without CPU installed) PFR Force Recovery JPFR3...
  • Page 19 Chapter 1: Introduction Connector Description JIPMB1 4-pin External I C Header (for an IPMI card) Chassis Intrusion Header JOH1-OH Overheat LED Header Power Supply SMBus I C Header JPWR1/3/4 +12V 8-pin CPU Power Connectors (Required) JPWR2 24-pin ATX Main Power Connector (Required) Intel VROC RAID Key Header JRK1 (Note: A VROC hardware key is required to enable an M.2 RAID card.)
  • Page 20: Chapter 2 Installation

    Chapter 2: Installation Chapter 2 Installation 2.1 Overview This chapter provides advice and instructions for rack or workstation installation. If your system is not already fully integrated with processors, system memory, etc., refer to Chapter 3 details on installing those specific components. Caution: Electrostatic Discharge (ESD) can damage electronic components.
  • Page 21: Rack Precautions

    Chapter 2: Installation • This product is not suitable for use with visual display workplace devices according to §2 of the German Ordinance for Work with Visual Display Units. Rack Precautions • Ensure that the leveling jacks on the bottom of the rack are extended to the floor so that the full weight of the rack rests on them.
  • Page 22: Airflow

    Chapter 2: Installation Airflow Equipment should be mounted into a rack so that the amount of airflow required for safe operation is not compromised. Mechanical Loading Equipment should be mounted into a rack so that a hazardous condition does not arise due to uneven mechanical loading.
  • Page 23: Preparing The Chassis For Rack Mounting

    Chapter 2: Installation 2.4 Preparing the Chassis for Rack Mounting The chassis top tower cover and feet must be removed before rack installation. Removing the Top Tower Cover 1. Locate the blue cover lock at the rear of the cover. 2.
  • Page 24: Installing The Rails

    Chapter 2: Installation 2.5 Installing the Rails This section provides a guideline for installing the rails to the chassis and to the rack with the optional rack mount kit. Identifying the Sections of the Rack Rails The chassis package includes two optional rack rail assemblies in the rack mounting kit. Each assembly consists of two sections: An inner fixed chassis rail that secures directly to the workstation chassis and an outer fixed rack rail that secures directly to the rack itself.
  • Page 25: Installing The Inner Rails To The Chassis

    Chapter 2: Installation Installing the Inner Rails to the Chassis 1. Attach the handles to the front of the chassis with three screws each. 2. Identify the left and right inner rails. They are labeled on the rails and in the figure below.
  • Page 26: Installing The Outer Rails To The Rack

    Chapter 2: Installation Installing the Outer Rails to the Rack Installing the Outer Rails 1. Attach the rear rail to the middle rail. 2. Adjust both to the proper distance so that the rails fit snugly into the rack. 3. Secure the rear rail with two M5 screws at the rear of the rack. 4.
  • Page 27: Installing The Chassis Into The Rack

    Chapter 2: Installation 2.6 Installing the Chassis into the Rack With rails attached to both the chassis and the rack, install the system into the rack. 1. Confirm that the chassis includes the inner rails and the outer rails. 2. Align the inner chassis rails with the front of the outer rack rails. 3.
  • Page 28 Chapter 2: Installation Warning: Stability hazard. The rack stabilizing mechanism must be in place, or the rack must be bolted to the floor before you slide the unit out for servicing. Failure to stabilize the rack can cause the rack to tip over. When initially installing the system to a rack, test that the rail locking tabs engage to prevent the system from being overextended.
  • Page 29: Chapter 3 Maintenance And Component Installation

    Chapter 3: Maintenance and Component Installation Chapter 3 Maintenance and Component Installation This chapter provides instructions on installing and replacing main system components. To prevent compatibility issues, only use components that match the specifications and/or part numbers given. Installation or replacement of most components requires that power first be removed from the system.
  • Page 30: Accessing The System

    Chapter 3: Maintenance and Component Installation 3.2 Accessing the System The CSE-747BTS-R2K20BP chassis features a removable top cover, side cover, and front cover which allows easy access to the inside of the chassis. Section 2.4 for instructions to remove the top cover. Top Cover Side Cover Front Cover...
  • Page 31 Chapter 3: Maintenance and Component Installation Removing the Side Cover 1. Remove power from the system as described in Section 3.1. 2. Lift the handle at the side of the tower. 3. Lift the cover from the chassis. Lift Up the Handle Lift Up the Side Cover Figure 3-2.
  • Page 32 Chapter 3: Maintenance and Component Installation Opening the Front Cover The front cover houses up to eight hot-swappable hard drives. The cover can be locked to prevent unauthorized access. The key to this lock is shipped with the system. 1. Unlock the front cover using the key shipped with the system. 2.
  • Page 33: Static-Sensitive Devices

    Chapter 3: Maintenance and Component Installation 3.3 Static-Sensitive Devices Electrostatic Discharge (ESD) can damage electronic components. To avoid damaging your motherboard, it is important to handle it very carefully. The following measures are generally sufficient to protect your equipment from ESD. Precautions •...
  • Page 34: Processor And Heatsink Installation

    Thermal grease is pre-applied on a new heatsink. No additional thermal grease is needed. • Refer to the Supermicro website for updates on processor and memory support. • All graphics in this manual are for illustrations only. Your components may look different.
  • Page 35: The 3Rd Gen. Intel Xeon Scalable Processor

    Chapter 3: Maintenance and Component Installation 1. The 3rd Gen. Intel Xeon Scalable Processor Processor Top View (3D) CPU Key Pin 1 = Cutout = Pin 1 = CPU Key Processor Top View...
  • Page 36: The Processor Carrier

    Chapter 3: Maintenance and Component Installation 2. The Processor Carrier = Cutout = Pin 1 = CPU Key Carrier Bottom View...
  • Page 37: Heatsink

    Chapter 3: Maintenance and Component Installation 3. Heatsink Note: Exercise extreme care when handling the heatsink. Pay attention to the edges of heatsink fins which can be sharp! To avoid damaging the heatsink, please do not apply excessive force on the fins when handling the heatsink.
  • Page 38: Overview Of The Cpu Socket

    Chapter 3: Maintenance and Component Installation Overview of the CPU Socket The CPU socket is protected by a plastic protective cover. Plastic Protective Cover CPU Socket...
  • Page 39: Overview Of The Processor Carrier Assembly

    Chapter 3: Maintenance and Component Installation Overview of the Processor Carrier Assembly The processor carrier assembly contains a 3rd Gen Intel Xeon Scalable processor and a processor carrier. Carefully follow the instructions given in the installation section to place a processor into the carrier to create a processor carrier.
  • Page 40: Overview Of The Processor Heatsink Module

    Chapter 3: Maintenance and Component Installation Overview of the Processor Heatsink Module The Processor Heatsink Module (PHM) contains a heatsink, a processor carrier, and the 3rd Gen Intel Xeon Scalable Processor 1. Heatsink 2. Processor Carrier 3. The 3rd Gen Intel Xeon Scalable Processor 4.
  • Page 41: Creating The Processor Carrier Assembly

    Chapter 3: Maintenance and Component Installation Creating the Processor Carrier Assembly The processor carrier assembly contains a 3rd Gen Intel Xeon Scalable processor and a processor carrier. To create the processor carrier assembly, please follow the steps below: Note: Before installation, be sure to follow the instructions given on Page 1 and Page 2 of this chapter to properly prepare yourself for installation.
  • Page 42 Chapter 3: Maintenance and Component Installation 3. Locate the lever on the CPU socket and press the lever down as shown below.Using Lever 4. Using Pin 1 as a guide, carefully align the CPU keys (A and B) on the processor against the CPU keys on the carrier (a and b) as shown in the drawing below.
  • Page 43 Chapter 3: Maintenance and Component Installation 6. After the processor is placed inside the carrier, examine the four sides of the processor, making sure that the processor is properly seated on the carrier. Processor Carrier Processor Carrier Assembly Assembly (bottom view) (Top View)
  • Page 44: Creating The Processor Heatsink Module (Phm)

    Chapter 3: Maintenance and Component Installation Creating the Processor Heatsink Module (PHM) After creating the processor carrier assembly, please follow the instructions below to mount the processor carrier into the heatsink to form the processor heatsink module (PHM). Note: If this is a new heatsink, the thermal grease has been pre-applied on the underside. Otherwise, apply the proper amount of thermal grease.
  • Page 45: Preparing The Cpu Socket For Installation

    Chapter 3: Maintenance and Component Installation Preparing the CPU Socket for Installation This motherboard comes with a plastic protective cover installed on the CPU socket. Remove it from the socket to install the Processor Heatsink Module (PHM). Gently pull up one corner of the plastic protective cover to remove it.
  • Page 46: Preparing To Install The Processor Heatsink Module (Phm) Into The Cpu Socket

    Chapter 3: Maintenance and Component Installation Preparing to Install the Processor Heatsink Module (PHM) into the CPU Socket After assembling the Processor Heatsink Module (PHM), you are ready to install it into the CPU socket. To ensure the proper installation, please follow the procedures below: 1.
  • Page 47: Installing The Processor Heatsink Module (Phm)

    Chapter 3: Maintenance and Component Installation Installing the Processor Heatsink Module (PHM) 1. Align peek nut "A", which is next to the triangle (Pin 1) on the heatsink, against threaded fastener "a" on the CPU socket. Then align peek nuts "B", "C", and "D" on the heatsink against threaded fasteners "b", "c", and "d"...
  • Page 48 Chapter 3: Maintenance and Component Installation 4. With a T30-bit screwdriver, tighten all peek nuts in the sequence of "A", "B", "C", and "D" with even pressure. To avoid damaging the processor or socket, do not use a force greater than 12 lbf-in when tightening the screws. 5.
  • Page 49: Removing The Processor Heatsink Module From The Cpu Socket

    Chapter 3: Maintenance and Component Installation Removing the Processor Heatsink Module from the CPU Socket Before removing the processor heatsink module (PHM) from the motherboard, unplug the AC power cord from all power supplies after shutting down the system. Then follow the steps below: 1.
  • Page 50 Chapter 3: Maintenance and Component Installation Removing the Processor Carrier Assembly from the Processor Heatsink Module (PHM) To remove the processor carrier assembly from the PHM, please follow the steps below: 1. Detach four plastic clips (marked a, b, c, and d) on the processor carrier assembly from the four corners of heatsink (marked A, B, C, and D) in the drawings below.
  • Page 51: Removing The Processor From The Processor Carrier Assembly

    Chapter 3: Maintenance and Component Installation Removing the Processor from the Processor Carrier Assembly Once you have removed the processor carrier assembly from the PHM, you are ready to remove the processor from the processor carrier by following the steps below. 1.
  • Page 52: Memory

    Note: Memory speed support depends on the processors used in the system. For validated memory, use our Product Resources page. Check the Supermicro website for possible updates to memory support. Memory Installation Sequence Memory for this motherboard is populated using the "Fill First" method. The DIMM slots with blue release tabs are considered the first DIMM of their channel, and those with white release tabs are the second of the channel.
  • Page 53: Guidelines Regarding Mixing Dimms

    Chapter 3: Maintenance and Component Installation DDR4 Memory Support for the 3rd Gen. Intel Xeon Scalable Processors Speed (MT/s); Voltage (V); Slots Per Channel (SPC) and DIMMs Per Channel (DPC) DIMM Capacity (GB) Ranks Per DIMM & Type 1DPC Data Width 2DPC (1-DIMM Per Chan- (2-DIMM Per Channel)
  • Page 54: Ddr4 Memory Population Guidelines

    Chapter 3: Maintenance and Component Installation DDR4 Memory Population Guidelines The following memory population table was created based on guidelines provided by Intel to support Supermicro motherboards. Memory Population Table (with 16 Slots) Number of DIMMs Memory Population Sequence 1 DIMM...
  • Page 55: Dimm Installation

    Chapter 3: Maintenance and Component Installation DIMM Installation 1. Insert the desired number of DIMMs into the slots based on the recommended DIMM population tables shown above. 2. Push the release tabs on both ends of the DIMM slot outwards to unlock it. 3.
  • Page 56: Motherboard Battery

    Chapter 3: Maintenance and Component Installation 3.6 Motherboard Battery The motherboard uses non-volatile memory to retain system information when system power is removed. This memory is powered by a lithium battery residing on the motherboard. Replacing the Battery Begin by removing power from the system.
  • Page 57: Storage Drives

    Chapter 3: Maintenance and Component Installation 3.7 Storage Drives The system supports eight hot-swap storage drive bays housed in the CSE-747BTS-R2K20BP chassis via Storage drive backplane (BPN-SAS3-747TQ-N4). For compatible storage drives, see the X12SPA-TF motherboard page. The drives are mounted in drive carriers to simplify their installation and removal from the chassis.
  • Page 58: Storage Drive Backplane

    Chapter 3: Maintenance and Component Installation Figure 3-6. Mounting a Drive in a Carrier Note: Enterprise level hard disk drives are recommended for use in Supermicro chassis and workstations. For information on recommended HDDs, visit the Supermicro website at https:// www.supermicro.com/en/products/superworkstation?pro=cpu%3D1 Storage Drive Backplane The storage drives plug into a drive backplane.
  • Page 59: Ssd Installation

    Chapter 3: Maintenance and Component Installation M.2 SSD Installation The X12SPA-TF motherboard has four M.2 PCIe 4.0 slots that support 2260, 2280, and 22110 M.2 NVMe SSD modules. 1. Loosen the screws and remove the heatsink. 2. The default positions for the 3.
  • Page 60: 2280 M.2 Nvme Ssd Module Installation

    Chapter 3: Maintenance and Component Installation 2280 M.2 NVMe SSD Module Installation 2280 2280 4.2. With the cutoff circle at the end of the 4.1. To install a 2280 M.2 NVMe module aligned with the standoff, tighten module, insert it into the slot at a 30 the screw to secure the module.
  • Page 61: 22110 M.2 Nvme Ssd Module Installation

    Chapter 3: Maintenance and Component Installation 22110 M.2 NVMe SSD Module Installation 22110 22110 4.5. To install a 22110 M.2 NVMe SSD 4.6. Go to step 5 to complete the module, insert it into the M.2 slot at a 30 installation.
  • Page 62: System Cooling

    Chapter 3: Maintenance and Component Installation System Cooling Heavy-duty fans provide cooling for the chassis. Four fans are located in the mid-section of the chassis, two fans are located in the rear, and two optional fans can be mounted on the external rear of the chassis, required for passive GPUs.
  • Page 63 Chapter 3: Maintenance and Component Installation FAN-0114L4 Mid-Fan Release Tab FAN-0138L4 Figure 3-7. Mid-System Chassis Fans FAN-0082L4 Rear Fan Release Tab Figure 3-8. Rear System Chassis Fans...
  • Page 64: Power Supply

    Chapter 3: Maintenance and Component Installation Power Supply The SuperWorkstation 540A-TR includes two 2200W redundant power supplies. These power supplies are auto-switching capable. This enables it to automatically sense and operate at a 100V to 240V input voltage. An amber light will be illuminated on the power supply when the power is off.
  • Page 65: Chapter 4 Motherboard Connections

    Chapter 4: Motherboard Connections Chapter 4 Motherboard Connections This section describes the connections on the motherboard and provides pinout definitions. Note that depending on how the system is configured, not all connections are required. The LEDs on the motherboard are also described here. The motherboard layout indicating component locations may be found in Chapter 1.
  • Page 66: Headers And Connectors

    Chapter 4: Motherboard Connections 4.2 Headers and Connectors Fan Headers There are 10 4-pin fan headers (FAN1 - FAN6, FAN A - FAN D) on the motherboard. All these 4-pin fan headers are backwards compatible with the traditional 3-pin fans. However, fan speed control is available for 4-pin fans only by Thermal Management via the IPMI 2.0 interface.
  • Page 67 The JTPM1 header is used to connect a Trusted Platform Module (TPM)/Port 80, which is available from Supermicro (optional). A TPM/Port 80 header is a security device that supports encryption and authentication in hard drives. It allows the motherboard to deny access if the TPM associated with the hard drive is not installed in the system.
  • Page 68 Chapter 4: Motherboard Connections VROC RAID Key Header A VROC RAID Key header is located at JRK1 on the motherboard. Install a VROC RAID Key on JRK1 for NVMe RAID support as shown in the illustration below. Refer to the layout below for the location of JRK1.
  • Page 69 Chapter 4: Motherboard Connections Power SMB (I C) Header The Power System Management Bus (I C) connector (JPI C1) monitors the power supply, fan, and system temperatures. Refer to the table below for pin definitions. Power SMB Header Pin Definitions Pin# Definition Clock...
  • Page 70 Chapter 4: Motherboard Connections Pump Power Header The motherboard has one +12V 4-pin header for optional CPU liquid cooling systems. When using a liquid cooling system, attach the pump power cable to the 12V_PUMP_PWR1 header. Pump Power Header Pin Definitions Pin# Definition Ground (Black)
  • Page 71: Control Panel

    These connectors are designed specifically for use with Supermicro chassis. Refer to the figure below for the descriptions of the front control panel buttons and LED indicators.
  • Page 72: Front Control Panel Leds

    Chapter 4: Motherboard Connections Front Control Panel LEDs Power Button Ground Reset Button Ground Power Fail (for LED6) 3.3V Red+ Blue+ (Blue LED_Cathode_UID) (Red OH/Fan Fail/PWR Fail for LED5/Blue UID LED) P3V3_STBY NIC2 Active LED NIC1 Active LED P3V3_STBY ID_UID/3.3V Stby HDD LED FP PWR LED 3.3V...
  • Page 73 Chapter 4: Motherboard Connections Power On & BMC/BIOS Status LED Button The Power On and BMC/BIOS Status LED button is located on pins 1 and 2 of JF1. Momentarily contacting both pins will power on/off the system or display BMC/BIOS status. Refer to the tables below for more information.
  • Page 74 Chapter 4: Motherboard Connections Power Fail LED The Power Fail LED connection is located on pins 5 and 6 of JF1. When this LED turns solid red, it indicates a power failure. Refer to the table below for pin definitions. Power Fail LED Pin Definitions (JF1) Pin#...
  • Page 75 Chapter 4: Motherboard Connections NIC1/NIC2 (LAN1/LAN2) The NIC (Network Interface Controller) LED connection for LAN port 1 is located on pins 11 and 12 of JF1, and LAN port 2 is on pins 9 and 10. Attach the NIC LED cables here to display network activity.
  • Page 76: Rear I/O Ports

    Chapter 4: Motherboard Connections 4.3 Rear I/O Ports Rear I/O Ports Description Description Description Description LAN1 (1Gb) Dedicated IPMI LAN USB6 (3.2 Gen. 1) Line In USB9 (2.0) USB7 (3.2 Gen. 1) USB4 (3.2 Gen. 2) Line Out USB10 (2.0) USB8 (3.2 Gen.
  • Page 77 Chapter 4: Motherboard Connections Universal Serial Bus (USB) Ports and Headers There are four USB 3.2 Gen. 1 ports (USB5, USB6, USB7, USB8) located on the rear I/O panel, and one USB 3.2 Gen. 1 header (USB2/3) located on the motherboard to provide front USB access.
  • Page 78 BMC on the motherboard. For more details on the UID LEDs and BMC LEDs, refer to the tables below. Also, refer to the BMC User's Guide posted on our website at http:// www.supermicro.com for more information on BMC. UID/BMC Reset Switch (UID-SW) Features &...
  • Page 79: Jumper Settings

    Chapter 4: Motherboard Connections 4.4 Jumper Settings Connector Pins How Jumpers Work Jumper To modify the operation of the motherboard, jumpers can be used to choose between optional settings. Jumpers Setting create shorts between two pins to change the function of the connector.
  • Page 80 Chapter 4: Motherboard Connections LAN Port Enable/Disable JPL1 and JPL2 allow you to enable the onboard LAN ports (LAN1 and LAN2). The default setting is pins 1-2 to enable the connections. Refer to the table below for jumper settings. LAN Enable/Disable Jumper Settings Jumper Setting Definition...
  • Page 81 Chapter 4: Motherboard Connections VGA Enable/Disable JPG1 allows you to enable the onboard VGA connector. The default setting is pins 1-2 to enable the connection. Refer to the table below for jumper settings. VGA Enable/Disable Jumper Settings Jumper Setting Definition Pins 1-2 Enabled (Default) Pins 2-3...
  • Page 82: Led Indicators

    Chapter 4: Motherboard Connections 4.5 LED Indicators LAN LEDs Two LAN ports (LAN1 and LAN2) are located on the rear I/O panel of the motherboard. Each Ethernet LAN port has two LEDs. The green LED indicates activity, while the other Link LED may be green, amber, or off to indicate the speed of the connection.
  • Page 83 Chapter 4: Motherboard Connections M.2 LEDs Four M.2 LEDs are located at LE3, LE4, LE5, and LE6 on the motherboard. When the M.2 LED is blinking, M.2 functions normally. Refer to the table below for more information. M.2 LED State LED Color Definition Green: Blinking...
  • Page 84: Storage Ports

    Chapter 4: Motherboard Connections 4.6 Storage Ports I-SATA Ports Eight SATA 3.0 ports (I-SATA0 - I-SATA7) are located on the motherboard, which are supported by the Intel® C621A chipset. These SATA ports support RAID 0, 1, 5, and 10. SATA ports provide serial-link signal connections, which are faster than the connections of Parallel ATA.
  • Page 85: Chapter 5 Software

    USB flash or media drive, or the IPMI KVM console. 2. Retrieve the proper RST/RSTe driver. Go to the Supermicro web page for your motherboard and click on "Download the Latest Drivers and Utilities", select the proper driver, and copy it to a USB flash drive.
  • Page 86 Chapter 5: Software 4. During Windows Setup, continue to the dialog where you select the drives on which to install Windows. If the disk you want to use is not listed, click on “Load driver” link at the bottom left corner. Figure 5-2.
  • Page 87: Driver Installation

    Chapter 5: Software 5.2 Driver Installation The Supermicro website contains drivers and utilities for your system at https:// www. supermicro. com/wdl/driver. Some of these must be installed, such as the chipset driver. After accessing the website, go into the CDR_Images (in the parent directory of the above link) and locate the ISO file for your motherboard.
  • Page 88: Superdoctor ® 5

    5.3 SuperDoctor ® The Supermicro SuperDoctor 5 is a program that functions in a command-line or web-based interface for Windows and Linux operating systems. The program monitors such system health information as CPU temperature, system voltages, system power consumption, fan speed, and provides alerts via email or Simple Network Management Protocol (SNMP).
  • Page 89: Bmc

    This can be found on a sticker on the chassis and a sticker on the motherboard. The sticker also displays the BMC MAC address. If necessary, the password can be reset using the Supermicro IPMICFG tool. Figure 5-5. BMC Password Label The sticker can be found on the pull-out service tag at the front of the chassis.
  • Page 90: Chapter 6 Optional Components

    Chapter 6: Optional Components Chapter 6 Optional Components This chapter describes alternate configurations and optional system components. Optional Parts Storage drive options Power options Cable Management Arm TPM security module Intel VROC RAID Key 6.1 Storage Protocols Supported The storage drive bays can support up to four 3.5" SAS3/SATA3 HDD/SSD and four 3.5" NVMe/SAS3/SATA3 storage devices.
  • Page 91: Cable Management Arm

    Chapter 6: Optional Components 6.3 Cable Management Arm The system supports a cable management arm (CMA), which keeps the rear cables organized and clear of the rail mechanisms when the system is extended out the front of the rack for maintenance.
  • Page 92 Chapter 6: Optional Components Figure 6-2. Installing the Connectors 3. Slide CMA connector #3 forward onto the two posts on the rear of the left middle rail. It snaps into place. 4. For CMA connector #4, align the metal tabs with the slots on the rear of the left outer rail and push it forward.
  • Page 93: Tpm Security Module

    It enables the motherboard to deny access if the TPM associated with the hard drive is not installed in the system. Details and installation procedures are at: http://www.supermicro.com/manuals/other/TPM.pdf. • AOM-TPM-9670V •...
  • Page 94: Intel Virtual Raid On Cpu (Vroc)

    Chapter 6: Optional Components 6.5 Intel Virtual RAID on CPU (VROC) Intel Virtual RAID on CPU (Intel VROC) is an enterprise RAID solution for NVMe SSDs ® directly attached to Intel Xeon Scalable processors. Intel Volume Management Device (VMD) is an integrated controller inside the CPU PCIe root complex. •...
  • Page 95: Additional Information

    Chapter 6: Optional Components Additional Information Additional information is available on the product page for the Supermicro add-on card and the linked manuals. www.supermicro.com/products/accessories/addon/AOC-VROCxxxMOD.cfm Hardware Key The Intel VROC hardware key is a license key that detects the Intel VROC SKU and activates the function accordingly.
  • Page 96: Configuring Nvme Raid Manually

    Chapter 6: Optional Components Configuring NVMe RAID Manually RAID for NVMe SSDs is enabled by default when Intel VROC Raid Key is populated. It may be managed manually through the UEFI BIOS. 1. Reboot the workstation and press the [DEL] key to access the BIOS options. 2.
  • Page 97 Chapter 6: Optional Components Figure 6-5. BIOS, Manual Mode (Example—your workstation may look different.) 3. Select the desired PStack# to Enable or Disable the corresponding Intel VMD controller Figure 6-6. BIOS, Enabling VMD for Pstack0...
  • Page 98 Chapter 6: Optional Components 4. Select the desired PCIe slot to Enable or Disable Intel VMD functionality according to the current hardware configuration being used. Hot Plug Capability can also be Enabled or Disabled. Figure 6-7. BIOS, Enabling VMD Functionality per Slot 5.
  • Page 99 Chapter 6: Optional Components Figure 6-9. BIOS, Enabling CPU2 Example 6. Press [F4] to save the configuration and reboot the system and press [DEL] to enter BIOS. Note: Disabling the VMD controller without first deleting the associated existing RAID volume can lead to unexpected behavior. This action is strongly not recommended. Note: The effects of physically changing or swapping a CPU on the VMD controller enablement have not yet been thoroughly tested or documented.
  • Page 100 Chapter 6: Optional Components 10. If cross-controller RAID is required, select Enable RAID spanned over VMD Controller. Figure 6-10. Created Volume without Figure 6-11. Created Volume with enabling RAID spanned over VMD enabling RAID spanned over VMD controller controller 11. Select specific disks for RAID with an [X]. •...
  • Page 101: Status Indications

    Chapter 6: Optional Components Status Indications An LED indicator on the drive carrier shows the RAID status of the drive. Drive Carrier Status LED Indicator Status State (red) Normal function Locating 4Hz blink Fault Solid ON Rebuilding 1Hz Blink IBPI SFF 8489 Defined Status LED States Hot-Swap Drives Intel VMD enables hot-plug and hot-unplug for NVMe SSDs, whether from Intel or other manufacturers.
  • Page 102: Chapter 7 Troubleshooting And Support

    Chapter 7 Troubleshooting and Support 7.1 Information Resources Website A great deal of information is available on the Supermicro website, www.supermicro.com. Menu Icon Figure 7-1. Supermicro Website • Specifications for workstations and other hardware are available by clicking the menu icon, then selecting the Products option.
  • Page 103: Intelligent Platform Management Interface (Ipmi)

    Security Center for recent security notices Supermicro Phone and Addresses 7.2 Intelligent Platform Management Interface (IPMI) The system supports the Intelligent Platform Management Interface (IPMI). IPMI is used to provide remote access, monitoring, and management. There are several BIOS settings that are related to IPMI.
  • Page 104: Troubleshooting Procedures

    Chapter 7: Troubleshooting and Support 7.3 Troubleshooting Procedures Use the following procedures to troubleshoot your system. If you have followed all of the procedures below and still need assistance, refer to the Technical Support Procedures Returning Merchandise for Service section(s) in this chapter. Power down the system before changing any non-hot-swap hardware components.
  • Page 105 Chapter 7: Troubleshooting and Support 3. Make sure that the power connector is connected to your power supply. 4. Make sure that no short circuits exist between the motherboard and chassis. 5. Disconnect all cables from the motherboard, including those for the keyboard and mouse. 6.
  • Page 106: No Video

    Chapter 7: Troubleshooting and Support No Video 1. If the power is ON but you have no video, remove all the add-on cards and cables. 2. As you try to power up the system, note any beep codes. Refer to the next section for details on beep codes.
  • Page 107 2. Memory support: Make sure that the memory modules are supported by testing the modules using memtest86 or a similar utility. Note: Refer to the product page on our website at http://www.supermicro.com for memory and CPU support and updates. 3. HDD support: Make sure that all hard disk drives (HDDs) work properly. Replace the bad HDDs with good ones.
  • Page 108: Bios Error Beep (Post) Codes

    When BIOS performs the Power On Self Test, it writes checkpoint codes to I/O port 0080h. If the computer cannot complete the boot process, a diagnostic card can be attached to the computer to read I/O port 0080h (Supermicro p/n AOC-LPC80-20). For information on AMI updates, please refer to http://www.ami.com/products/.
  • Page 109: Crash Dump Using Ipmi

    In the event of a processor internal error (IERR) that crashes your system, you may want to provide information to support staff. You can download a crash dump of status information using IPMI. The IPMI manual is available at https://www.supermicro.com/solutions/IPMI.cfm. Check IPMI Error Log 1.
  • Page 110: Uefi Bios Recovery

    Warning: Do not upgrade the BIOS unless your system has a BIOS-related issue. Flashing the wrong BIOS can cause irreparable damage to the system. In no event shall Supermicro be liable for direct, indirect, special, incidental, or consequential damages arising from a BIOS update.
  • Page 111 USB device or a writable CD/DVD. Note 1: If you cannot locate the "Super.ROM" file in your drive disk, visit our website at www.supermicro.com to download the BIOS package. Extract the BIOS binary image into a USB flash device and rename it "Super.ROM" for the BIOS recovery use.
  • Page 112 Chapter 7: Troubleshooting and Support Note: At this point, you may decide if you want to start the BIOS recovery. If you decide to proceed with BIOS recovery, follow the procedures below. 4. When the screen as shown above displays, use the arrow keys to select the item "Proceed with flash update"...
  • Page 113 Chapter 7: Troubleshooting and Support 7. Press <Del> continuously during system boot to enter the BIOS Setup utility. From the top of the toolbar, select Boot to enter the submenu. From the submenu list, select Boot Option #1 as shown below. Then, set Boot Option #1 to [UEFI AP:UEFI: Built-in EFI Shell]. Press <F4>...
  • Page 114 Chapter 7: Troubleshooting and Support Note: Do not interrupt this process until the BIOS flashing is complete. 9. The screen above indicates that the BIOS update process is complete. When you see the screen above, unplug the AC or DC power cable from the power supply, clear CMOS, and plug the AC or DC power cable in the power supply again to power on the system.
  • Page 115: Cmos Clear

    Chapter 7: Troubleshooting and Support 7.7 CMOS Clear JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS. To Clear CMOS 1.
  • Page 116: Where To Get Replacement Components

    7.9 Where to Get Replacement Components If you need replacement parts for your system, to ensure the highest level of professional service and technical support, purchase exclusively from our Supermicro Authorized Distributors/System Integrators/Resellers. A list can be found at: http://www.supermicro.com.
  • Page 117: Vendor Support Filing System

    For issues related to Red Hat Enterprise Linux, since it is a subscription based OS, contact your account representative. 7.11 Feedback Supermicro values your feedback as we strive to improve our customer experience in all facets of our business. Please email us at techwriterteam@supermicro.com to provide feedback on our manuals.
  • Page 118: Contacting Supermicro

    San Jose, CA 95131 U.S.A. Tel: +1 (408) 503-8000 Fax: +1 (408) 503-8008 Email: marketing@supermicro.com (General Information) Sales-USA@supermicro.com (Sales Inquiries) Government_Sales-USA@supermicro.com (Gov. Sales Inquiries) support@supermicro.com (Technical Support) RMA@supermicro.com (RMA Support) Webmaster@supermicro.com (Webmaster) Website: www.supermicro.com Europe Address: Super Micro Computer B.V.
  • Page 119: Appendix A Standardized Warning Statements For Ac Systems

    Should you have questions or experience difficulty, contact Supermicro's Technical Support department for assistance. Only certified technicians should attempt to install or configure components. Read this appendix in its entirety before installing or configuring components in the Supermicro chassis. T h e s e...
  • Page 120 Appendix A: Warning Statements Warnung WICHTIGE SICHERHEITSHINWEISE Dieses Warnsymbol bedeutet Gefahr. Sie befinden sich in einer Situation, die zu Verletzungen führen kann. Machen Sie sich vor der Arbeit mit Geräten mit den Gefahren elektrischer Schaltungen und den üblichen Verfahren zur Vorbeugung vor Unfällen vertraut. Suchen Sie mit der am Ende jeder Warnung angegebenen Anweisungsnummer nach der jeweiligen Übersetzung in den übersetzten Sicherheitshinweisen, die zusammen mit diesem Gerät ausgeliefert wurden.
  • Page 121 Appendix A: Warning Statements . ٌ ‫ا ك ً ف حالة و ٌ يك أى تتسبب ف اصابة جسذ ة ٌ هذا الزهز ع ٌ خطز !تحذ ز‬ ‫قبل أى تعول عىل أي هعذات،يك عىل علن بالوخاطز ال ا ٌجوة عي الذوائز‬ ٍ...
  • Page 122 Appendix A: Warning Statements Warnung Vor dem Anschließen des Systems an die Stromquelle die Installationsanweisungen lesen. ¡Advertencia! Lea las instrucciones de instalación antes de conectar el sistema a la red de alimentación. Attention Avant de brancher le système sur la source d'alimentation, consulter les directives d'installation. .‫יש...
  • Page 123 Appendix A: Warning Statements Warnung Dieses Produkt ist darauf angewiesen, dass im Gebäude ein Kurzschluss- bzw. Überstromschutz installiert ist. Stellen Sie sicher, dass der Nennwert der Schutzvorrichtung nicht mehr als: 250 V, 20 A beträgt. ¡Advertencia! Este equipo utiliza el sistema de protección contra cortocircuitos (o sobrecorrientes) del edificio.
  • Page 124 Appendix A: Warning Statements Power Disconnection Warning Warning! The system must be disconnected from all sources of power and the power cord removed from the power supply module(s) before accessing the chassis interior to install or remove system components. 電源切断の警告 システムコンポーネントの取り付けまたは取り外しのために、...
  • Page 125 Appendix A: Warning Statements ‫אזהרה מפני ניתוק חשמלי‬ ‫!אזהרה‬ ‫יש לנתק את המערכת מכל מקורות החשמל ויש להסיר את כבל החשמלי מהספק‬ ‫.לפני גישה לחלק הפנימי של המארז לצורך התקנת או הסרת רכיבים‬ ‫يجب فصم اننظاو من جميع مصادر انطاقت وإ ز انت سهك انكهرباء من وحدة امداد‬ ‫انطاقت...
  • Page 126 Appendix A: Warning Statements Attention Il est vivement recommandé de confier l'installation, le remplacement et la maintenance de ces équipements à des personnels qualifiés et expérimentés. !‫אזהרה‬ .‫צוות מוסמך בלבד רשאי להתקין, להחליף את הציוד או לתת שירות עבור הציוד‬ ‫واملدربيه...
  • Page 127 Appendix A: Warning Statements Warnung Diese Einheit ist zur Installation in Bereichen mit beschränktem Zutritt vorgesehen. Der Zutritt zu derartigen Bereichen ist nur mit einem Spezialwerkzeug, Schloss und Schlüssel oder einer sonstigen Sicherheitsvorkehrung möglich. ¡Advertencia! Esta unidad ha sido diseñada para instalación en áreas de acceso restringido. Sólo puede obtenerse acceso a una de estas áreas mediante la utilización de una herramienta especial, cerradura con llave u otro medio de seguridad.
  • Page 128 Appendix A: Warning Statements Battery Handling Warning! There is the danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer's instructions 電池の取り扱い...
  • Page 129 Appendix A: Warning Statements ‫هناك خطر من انفجار يف حالة اسحبذال البطارية بطريقة غري صحيحة فعليل‬ ‫اسحبذال البطارية‬ ‫فقط بنفس النىع أو ما يعادلها مام أوصث به الرشمة املصنعة‬ ‫جخلص من البطاريات املسحعملة وفقا لحعليامت الرشمة الصانعة‬ 경고! 배터리가 올바르게 교체되지 않으면 폭발의 위험이 있습니다. 기존 배터리와 동일하거나 제 조사에서...
  • Page 130 Appendix A: Warning Statements ¡Advertencia! Puede que esta unidad tenga más de una conexión para fuentes de alimentación. Para cortar por completo el suministro de energía, deben desconectarse todas las conexiones. Attention Cette unité peut avoir plus d'une connexion d'alimentation. Pour supprimer toute tension et tout courant électrique de l'unité, toutes les connexions d'alimentation doivent être débranchées.
  • Page 131 Appendix A: Warning Statements Backplane Voltage Warning! Hazardous voltage or energy is present on the backplane when the system is operating. Use caution when servicing. バックプレーンの電圧 システムの稼働中は危険な電圧または電力が、 バックプレーン上にかかっています。 修理する際には注意く ださい。 警告 当系统正在进行时,背板上有很危险的电压或能量,进行维修时务必小心。 警告 當系統正在進行時,背板上有危險的電壓或能量,進行維修時務必小心。 Warnung Wenn das System in Betrieb ist, treten auf der Rückwandplatine gefährliche Spannungen oder Energien auf.
  • Page 132 Appendix A: Warning Statements ‫هناك خطز مه التيار الكهزبايئ أوالطاقة املىجىدة عىل اللىحة‬ ‫عندما يكىن النظام يعمل كه حذ ر ا عند خدمة هذا الجهاس‬ 경고! 시스템이 동작 중일 때 후면판 (Backplane)에는 위험한 전압이나 에너지가 발생 합니다. 서비스 작업 시 주의하십시오. Waarschuwing Een gevaarlijke spanning of energie is aanwezig op de backplane wanneer het systeem in gebruik is.
  • Page 133 Appendix A: Warning Statements ‫תיאום חוקי החשמל הארצי‬ !‫אזהרה‬ .‫התקנת הציוד חייבת להיות תואמת לחוקי החשמל המקומיים והארציים‬ ‫تركيب املعدات الكهربائية يجب أن ميتثل للقىاويه املحلية والىطىية املتعلقة‬ ‫بالكهرباء‬ 경고! 현 지역 및 국가의 전기 규정에 따라 장비를 설치해야 합니다. Waarschuwing Bij installatie van de apparatuur moet worden voldaan aan de lokale en nationale elektriciteitsvoorschriften.
  • Page 134 Appendix A: Warning Statements Attention La mise au rebut ou le recyclage de ce produit sont généralement soumis à des lois et/ou directives de respect de l'environnement. Renseignez-vous auprès de l'organisme compétent. ‫סילוק המוצר‬ !‫אזהרה‬ .‫סילוק סופי של מוצר זה חייב להיות בהתאם להנחיות וחוקי המדינה‬ ‫التخلص...
  • Page 135 Appendix A: Warning Statements Warnung Gefährlich Bewegende Teile. Von den bewegenden Lüfterblätter fern halten. Die Lüfter drehen sich u. U. noch, wenn die Lüfterbaugruppe aus dem Chassis genommen wird. Halten Sie Finger, Schraubendreher und andere Gegenstände von den Öffnungen des Lüftergehäuses entfernt.
  • Page 136 Verbindungskabeln, Stromkabeln und/oder Adapater, die Ihre örtlichen Sicherheitsstandards einhalten. Der Gebrauch von anderen Kabeln und Adapter können Fehlfunktionen oder Feuer verursachen. Die Richtlinien untersagen das Nutzen von UL oder CAS zertifizierten Kabeln (mit UL/CSA gekennzeichnet), an Geräten oder Produkten die nicht mit Supermicro gekennzeichnet sind.
  • Page 137 ‫.قيرح وأ لطع يف ببستي دق ىرخأ تالوحمو تالباك يأ مادختسا .ميلسلا سباقلاو لصوملا مجح‬ ‫ لبق نم ةدمتعملا تالباكلا مادختسا تادعملاو ةيئابرهكلا ةزهجألل ةمالسلا نوناق رظحي‬UL ‫ وأ‬CSA ‫( ةمالع لمحت يتلاو‬UL/CSA) ‫ لبق نم ةددحملاو ةينعملا تاجتنملا ريغ ىرخأ تادعم يأ عم‬Supermicro.
  • Page 138 사항을 준수하여 제공되거나 지정된 연결 혹은 구매 케이블, 전원 케이블 및 AC 어댑터를 사용하십시오. 다른 케이블이나 어댑터를 사용하면 오작동이나 화재가 발생할 수 있습니다. 전기 용품 안전법은 UL 또는 CSA 인증 케이블 (코드에 UL / CSA가 표시된 케이블)을 Supermicro 가 지정한 제품 이외의 전기 장치에 사용하는 것을 금지합니다. Stroomkabel en AC-Adapter...
  • Page 139: Appendix B System Specifications

    Appendix B: System Specifications Appendix B System Specifications Processor Support 3rd Generation Intel® Xeon® Scalable processors in an LGA4189 socket; Supports CPU TDP up to 270W. Note: Refer to the motherboard specifications pages on our website for updates to supported processors. Chipset Intel®...
  • Page 140 Appendix B: System Specifications Operating Environment Operating Temperature: 10º to 35º C (50º to 95º F) Non-operating Temperature: -40º to 60º C (-40º to 140º F) Operating Relative Humidity: 8% to 90% (non-condensing) Non-operating Relative Humidity: 5% to 95% (non-condensing) Regulatory Compliance FCC, CE, VCCI, CSA/ UL, CB Applied Directives, Standards...
  • Page 141: Appendix C Bsmi Rohs And Warnings

    Appendix C: BSMI RoHS and Warnings Appendix C BSMI RoHS and Warnings 限 限 用 用 物 物 質 質 含 含 有 有 情 情 況 況 標 標 示 示 聲 聲 明 明 書 書 Declaration of the Presence Condition of the Restricted Substances Marking 747-R22X12 設備名稱:超級工作站/ SuperWorkstation 型號(型式):...
  • Page 142 Appendix C: BSMI RoHS and Warnings 輸 輸 入 入 額 額 定 定 : 100-127Vac, 60-50Hz, 12-11 A (x2) 200-240Vac, 60-50Hz 10-9.5 A (x2) *使 使 用 用 者 者 不 不 能 能 任 任 意 意 拆 拆 除 除 或 或 替 替 換 換 內 內 部 部 配 配 備 備 *報...

Table of Contents