Summary of Contents for Sun Microsystems Netra ft 1800
Page 1
Netra ft 1800 ™ Hardware Reference Manual Sun Microsystems, Inc. 901 San Antonio Road Palo Alto, CA 94303-4900 USA 650 960-1300 Fax 650 969-9131 Part No.: 805-4531-11 Revision A, April 1999 Send comments about this document to: docfeedback@sun.com...
Page 2
Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd. Sun, Sun Microsystems, le logo Sun, AnswerBook, Java, le logo Jave Coffee Cup, et Solaris sont des marques de fabrique ou des marques déposées, ou marques de service, de Sun Microsystems, Inc. aux Etats-Unis et dans d’autres pays. Toutes les marques SPARC sont utilisées sous licence et sont des marques de fabrique ou des marques déposées de SPARC International, Inc.
Contents Introduction 1-1 System Features 1-1 System Components 1-2 hotPCI 1-3 Common Features of Modules 1-4 1.4.1 Simple and Safe Hot Replacement 1-4 1.4.2 LED Indicators 1-4 1.4.3 Module Identity and Logs 1-5 1.4.4 Module Injection Mechanisms 1-6 1.4.5 The Maintenance Buses 1-8 1.4.6 Locations for Modules 1-8 Hot Insertion 1-9...
Page 4
Diagnostic Indicators 2-17 CPUsets 3-1 Dimensions 3-1 The CPUset 3-5 CPUset LEDs 3-7 Clock Generation 3-7 Processors and ASICs 3-8 3.5.1 UltraSPARC II Processor 3-8 3.5.2 BMX+ 3-9 3.5.3 PCIO 3-9 Netra ft 1800 Hardware Reference Manual • April 1999...
Page 5
3.5.4 U2P 3-9 UPA 3-10 PCI Buses 3-11 3.7.1 U2P ASIC 3-11 3.7.2 PCIO ASIC 3-11 Mass Storage 4-1 Drive Chassis 4-3 Hard Disk Drives 4-5 Removable Media Modules 5-1 Console, Alarms and Fans 6-1 Power Controls and Supply 6-4 Alarm Relays and Timed Resets 6-4 Serial Interfaces 6-5 6.3.1...
Page 6
DC Outputs 8-4 Troubleshooting 8-5 A. RCP Status Information A-1 Glossary Glossary-1 Index Index-1 Netra ft 1800 Hardware Reference Manual • April 1999...
Page 8
FIGURE 7-2 PCI Card Carrier Front Panel 7-4 FIGURE 7-3 Power Supply Module 8-1 FIGURE 8-1 PSU Module Locations 8-2 FIGURE 8-2 Power Supply Module Front Panel 8-4 FIGURE 8-3 viii Netra ft 1800 Hardware Reference Manual • April 1999...
Page 9
Tables Module LED States 1-5 TABLE 1-1 Module Locations TABLE 1-2 Main Maintenance Bus TABLE 2-1 Alternate Maintenance Bus Segments 2-7 TABLE 2-2 RCP Serial Connector Pinout 2-9 TABLE 2-3 Motherboard Diagnostic Indicators 2-17 TABLE 2-4 CPUset Indicator LEDs 3-7 TABLE 3-1 Clock Domains on the CPUset 3-7 TABLE 3-2...
Page 10
Netra ft 1800 Hardware Reference Manual • April 1999...
Page 11
Preface This manual describes the hardware components of the Netra ft 1800 system. Who Should Use This Guide This guide is intended to be read by installation engineers, hardware support personnel and service personnel. It is not intended for the end user of the system.
Page 12
Netra ft 1800 Reference Manual (Part No. 805-4532-10) Netra ft 1800 Developer’s Guide (Part No. 805-4530-10) Netra ft 1800 CMS API Developer’s Guide (Part No. 805-5870-10) Netra ft 1800 CMS Developer’s Guide (Part No. 805-7899-10) Netra ft 1800 Installation Guide (Part No. 805-4533-10) Netra ft 1800 Compliance and Safety Manual (Part No.
Page 13
Typographic Conventions Typographic conventions TABLE P-1 Typeface or Symbol Meaning Examples The names of commands, files, Edit your .login file. AaBbCc123 and directories; on-screen Use ls -a to list all files. computer output. % You have mail. What you type, when AaBbCc123 contrasted with on-screen Password:...
Page 14
We are interested in improving our documentation and welcome your comments and suggestions. You can email your comments to us at: docfeedback@sun.com Please include the part number of your document in the subject line of your email. Netra ft 1800 Hardware Reference Manual • April 1999...
This chapter provides a general introduction to the concept and design of the Netra ft 1800 computer system. The Netra ft 1800 is a rack-mounted server that can be configured to be a hardware fault tolerant or a high-availability system.
Because comparison of the CPUsets occurs only at I/O accesses, it does not cause the system to suffer significant performance loss. In a working Netra ft 1800, comparisons can take place millions of times per second. If a discrepancy occurs, the CPUsets verify which CPUset failed, configure the failed CPUset out of the core and continue normal processing on the remaining CPUset.
CMS. Each module belongs to one side only in a fault tolerant system See the Netra ft 1800 User’s Guide for full descriptions of the subsystems and the CMS. In addition, the system has a chassis in which the other components are mounted.
The CAF module contains an additional green Power LED which is lit when the associated motherboard is powered on, and a red Fault LED which is lit when a fault is detected on the motherboard. Netra ft 1800 Hardware Reference Manual • April 1999...
The CPUsets have additional 3 mm amber LEDs, which are described in Section 3.3 “CPUset LEDs” on page 3-7. The states of the module LEDs are summarized in TABLE 1-1 Module LED States TABLE 1-1 Fault (red) Power (green) Description Module is not in use Module is operating Faulty module...
When the latch is disengaged a red dot is exposed to facilitate identification of disengaged injectors. Open Closed Red warning dot Typical Module Injector/Ejector Lever FIGURE 1-1 Netra ft 1800 Hardware Reference Manual • April 1999...
The DSK and RMM modules have a sliding latch on an ejector handle. The latch controls the electrical connection to the motherboard. When it is closed (pushed to the right), the electrical connection is engaged; when it is open (pushed to the left), the electrical connection is disengaged.
HDD modules. Maintenance bus control of the HDD modules is implemented within the DSK module. The maintenance buses allow the processor subsystem to monitor and control the operation of the whole Netra ft 1800 system. Examples of the traffic carried on a maintenance bus are: The environmental signals from modules, which indicate their temperature.
‘off’ state can be enabled using the CAF ON switch. Because PSUs remember the state they were in the last time they were enabled, a PSU which was previously ’on’ will power on as soon as it is inserted. 1-10 Netra ft 1800 Hardware Reference Manual • April 1999...
Page 25
Whilst both motherboards can be replaced individually without stopping the system, the replacement procedure is completely different from that for the other modules, and is very precisely defined. Refer to the Netra ft 1800 User’s Guide for a full description of the procedure.
Page 26
1-12 Netra ft 1800 Hardware Reference Manual • April 1999...
C H A P T E R Motherboards The motherboard provides the communication between the modules that are connected to the slots on the motherboard. The main functional areas of the motherboard are: Bridge (two per motherboard) PCI I/O slots Interrupts Clock generation Maintenance bus...
Page 28
Split mode – the bridge arbitrates PCI address and data from the CPUsets on to the device bus. No I/O comparisons are made. The mode in which the bridge operates is controlled by the software split utility. See the Netra ft 1800 User’s Guide for information. Post-write buffer address Post-write...
Page 29
The bridge consists of three devices which in combination provide the following features: In either mode: Two CPUset PCI bus interfaces One PCI bus for I/O devices PCI bus transaction routing PCI bus error detection Post write buffer access for error diagnosis Arbitration for up to six devices on the PCI bus.
U2P ASIC in the CPUset (see Section 3.5.4 “U2P” on page 3-9) which connects to that motherboard. The same six-bit code is sent to both CPUsets. There are two further interrupts, one per bridge, which are EState interrupts and route directly to the CPUsets. Netra ft 1800 Hardware Reference Manual • April 1999...
Clock Generation The master clock, running at 24.625 MHz, is on Motherboard B. Motherboard A synchronizes to a clock provided from Motherboard B. Each motherboard provides a clock to its local CPUset. If the clock on Motherboard B is lost, the clock on Motherboard A will not drift more than 100ppm.
Page 32
Ambient temperature, fan monitor PSU (3) EEPROM Module information and status log Bit I/O Fault LED, force fans, status Bit I/O Status LM78 Ambient temperature, fan monitor, heatsink temperature, current monitor Netra ft 1800 Hardware Reference Manual • April 1999...
2.5.2 Alternate Maintenance Bus The alternate maintenance bus connects to devices on the other motherboard. The connection is carried by both CPUsets in parallel. On the other motherboard, the alternate maintenance bus connects to an 8574 Bit I/O device which controls an analog multiplexer to select and enable one of four maintenance bus segments, as shown in TABLE 2-2...
Polls the OSdog signal and the AC_OK signal from the UPS interface. The RCP provides a means of controlling the power supplies in the Netra ft 1800 system using commands sent via either of the two serial ports or the SPI (see Section 2.6.3 “SPI”...
2.6.1 RCP Serial Ports The RCP is equipped with two serial ports. One is internal to the microcontroller and the other is an external DUART. The communications protocol is N,8,1,9600, and the pinout for both ports is shown in TABLE 2-3 RCP Serial Connector Pinout TABLE 2-3 Function...
Page 36
PSU to apply power to the side B motherboard. Note that if the power on is successful, the RCP will hold the motherboard in reset while it configures the FPGAs. 2-10 Netra ft 1800 Hardware Reference Manual • April 1999...
AAoff This command causes the side A RCP to generate an active low pulse on its OFF output which causes the PSU to remove power from the side A motherboard. If the RCP is configuring the FPGAs when this command is issued, it will abort the process.
LED Indicators A green LED flashes when the RCP is running (see on page 2-15, FIGURE 2-3 on page 2-16 and Section 2.10 “LEDs and Indicators” on page 2-14). FIGURE 2-4 2-12 Netra ft 1800 Hardware Reference Manual • April 1999...
Motherboard Controller The motherboard controller provides the following registers: Determination of which CPUset is the primary (including during recovery from EState) Recording of most recent reason for reset: remote, requested by the RCP OSdog synchronization clock Mailbox interruptor Power status PLL lock status PCI slot IRQ concentrator.
System LED on the CAF, and the general purpose alarms register which controls the user-definable alarm LED indicators via relays in the CAFs. 2.10 LEDs and Indicators Refer to on page 2-15 and on page 2-16. FIGURE 2-3 FIGURE 2-4 2-14 Netra ft 1800 Hardware Reference Manual • April 1999...
Diagnostic indicator: RCP On Diagnostic indicators Power LED Fault LED Diagnostic indicators Upper Motherboard (A-MBD) LEDs and Diagnostic Indicators FIGURE 2-3 Chapter 2 Motherboards 2-15...
Diagnostic indicators Power LED Fault LED Diagnostic indicators FPGA status Config Diagnostic indicators Diagnostic indicator: RCP On Lower Motherboard (B-MBD) LEDs and Diagnostic Indicators FIGURE 2-4 2-16 Netra ft 1800 Hardware Reference Manual • April 1999...
2.10.1 LEDs The Power and Fault LEDs are always visible, and are repeated on the front of the appropriate CAFs. 2.10.2 Diagnostic Indicators The motherboards incorporate a number of diagnostic indicators which are not generally visible when the covers are fitted. It is not necessary to view these indicators;...
Page 44
2-18 Netra ft 1800 Hardware Reference Manual • April 1999...
C H A P T E R CPUsets The CPUset is the module that provides processing for a Netra ft 1800 system. A CPUset consists of one, two or four processor modules and associated connections and fans. Two CPUsets operating in lock-step form the ‘core’ of a fault tolerant system.
Lifting handle CPUset Lifting Handle Location FIGURE 3-4 The CPUset The CPUset module plugs into one of the two dedicated CPUset locations in the center of the main cabinet. With the other CPUset, it forms the self-checking and self-correcting core of the fault tolerant system. The CPUset has the following features: One to four processor modules.
Page 50
Nine BMX+ crossbar switch slices to multiplex the multiple UPA buses. Clock generator circuitry generating phase-accurate multiples of the PCI input clock for the UPA, processor and U2P clocks. Two variable-speed fans. Netra ft 1800 Hardware Reference Manual • April 1999...
CPUset LEDs In addition to the standard Power and Fault LEDs described in Section 1.4.2 “LED Indicators” on page 1-4, the CPUset has five 3 mm amber LEDs as described in TABLE 3-1 CPUset Indicator LEDs TABLE 3-1 Indicator Meaning Sync CPUset is running in synchronization with the other Split...
Three floating-point execution units Selectable little- or big-endian byte ordering 64-bit address pointers 16Kbyte non-blocking data cache 16Kbyte instruction cache Single cycle branch following Power management Software prefetch instruction support Multiple outstanding requests. Netra ft 1800 Hardware Reference Manual • April 1999...
3.5.2 BMX+ The crossbar switch (BMX+) ASICs form the hub of all data transfers in the system unit. The BMX+ ASIC permits the implementation of a high bandwidth interleaved dual bank memory system. It coordinates between memory (two buses, each 576 bits wide), the two processor UPA buses (144 bits wide) and the I/O UPA bus (72 bits wide).
UPA data bus 3 (UPA_DATA3) U2P B Refer to FIGURE 3-5 module UPA_AD0 UPA_DATA0 SC_MP module ASIC BMX+ ASIC module UPA_AD1 UPA_DATA1 module UPA_DATA2 UPA_AD2 ASIC UPA_DATA3 Data Buses on a CPUset FIGURE 3-5 3-10 Netra ft 1800 Hardware Reference Manual • April 1999...
UPA_AD0, UPA_AD1 and UPA_AD2 are full 36-bit bidirectional buses that connect the SC_MP ASIC to the CPU modules and the U2P ASICs. UPA_DATA0 and UPA_DATA1 are bidirectional 144-bit data buses (128 bits of data and 16 bits of ECC) that connect the CPU modules to the BMX+ ASIC. UPA_DATA2 and UPA_DATA3 are bidirectional 72-bit data buses (64 bits of data and eight bits of ECC) that connect the U2P ASIC to the BMX+ ASIC.
Page 56
3-12 Netra ft 1800 Hardware Reference Manual • April 1999...
C H A P T E R Mass Storage The mass storage hardware consists of a drive chassis (DSK) that contains up to six hard disk drive (HDD) modules. Both the chassis and the individual disk modules are hot-insertable. Fast-20 (Ultra-SCSI) transfers are supported. The overall dimensions of the disk chassis are 310 mm (w) x 385 mm (d) x 175 mm (h) (12.20 inches x 15.16 inches x 6.89 inches).
Hard disk drives (HDDs) (locations A-DSK0 to A-DSK5) Disk chassis A (A-DSK) Hard disk drives (HDDs) (locations B-DSK0 to B-DSK5) Disk chassis B (B-DSK) Disk Chassis Locations FIGURE 4-1 Netra ft 1800 Hardware Reference Manual • April 1999...
Drive Chassis The drive chassis accommodates up to six standard Fast-20 (UltraSCSI) 3.5-inch disks in a special shielded bracket assembly forming modules with integral ejectors that directly connect to a dedicated SCSI backplane. The chassis has associated Fault and Power indicators as described in Section 1.4.2 “LED Indicators” on page 1-4. There are also Fault and Power LEDs above each disk slot.
Hard disk drive Chassis Fault and Fault and Power Power indicator LEDs indicator LEDs Chassis injector lever Injector lock and lock position indicator Disk Chassis Front Panel FIGURE 4-3 Netra ft 1800 Hardware Reference Manual • April 1999...
This mirroring is enforced entirely by the Volume Manager software. See the Netra ft 1800 User’s Guide. When a new disk is enabled, the mirroring software copies data from the other half of the pair to the new disk while maintaining normal system operation.
Page 62
Netra ft 1800 Hardware Reference Manual • April 1999...
C H A P T E R Removable Media Modules A removable media module (RMM) contains one or two standard 8-bit half-height 5.25 inch SCSI-2 devices mounted side by side. The module is hot-replaceable, and contains Fault and Power indicators as described in Section 1.4 “Common Features of Modules”...
Removable media module A (A-RMM) Removable media module B (B-RMM) Disk and RMM Chassis Locations FIGURE 5-2 Netra ft 1800 Hardware Reference Manual • April 1999...
Indicators Tape drive CD-ROM drive Chassis injector lock indicator RMM Drive Chassis Front Panel (CD-ROM and DAT Tape Configuration) FIGURE 5-3 The drives are permanently installed in the module, and the entire module is swapped as a hot-replaceable unit. The two slots are normally occupied by a CD-ROM and a tape drive.
Page 66
Netra ft 1800 Hardware Reference Manual • April 1999...
C H A P T E R Console, Alarms and Fans The console, alarms and fans (CAF) module provides the standard system I/O interfaces, accessible from the front of the system. Console, Alarms and Fans Module FIGURE 6-1 The overall dimensions for a CAF module are 305 mm (w) x 385 mm (d) x 58 mm (h) (12 inches x 15.16 inches x 2.29 inches).
Console, Alarms and Fans module A (A-CAF) Console, Alarms and Fans module B (B-CAF) CAF Module Locations FIGURE 6-2 Netra ft 1800 Hardware Reference Manual • April 1999...
Page 69
The CAF module contains the following on its front panel (see FIGURE 6-3 One green System indicator. Three amber user-defined Alarm indicators. Fault and Power indicators for the CAF. Fault and Power indicators for the motherboard (repeaters). Power On and Stby switches. Alarm relays and timed resets.
(CMS) to the on (alarmed) state, but can be set to the off state only by the watchdog. The front-panel System indicator illuminates automatically when the system relay is in the off (non-alarmed) state. Netra ft 1800 Hardware Reference Manual • April 1999...
The remaining three alarms in each group are entirely under software control and default to the non-alarmed state when powered-on and not initialized; they also assume the non-alarmed state when power is removed. The timed-pulse reset inputs are accessible on the same connector as the alarm relays.
The port is designed so that whatever external signals are presented, the port will continue normal operation after the external signals have returned to their normal state (provided that such signals stay within the RS232 electrical specification). Netra ft 1800 Hardware Reference Manual • April 1999...
6.3.2 Modem Port The modem connection in the CAF is an RS232 port communicating with the motherboard UART. Connection is via a female DB-25 whose pinout is shown in TABLE 6-3 Modem Port Pinout TABLE 6-3 Function Description Chassis ground Output data Input data Output handshake...
The Ethernet board is provided with a dedicated, switchable +5V rail by a regulator located on the CAF baseboard. When this rail is operational, the Power indicator is illuminated on the CAF front panel. Netra ft 1800 Hardware Reference Manual • April 1999...
6.6.1 Ethernet Interfaces Connection is via RJ45 jacks which have the pinout shown in TABLE 6-5 Ethernet LAN Connector Pinout TABLE 6-5 Function Description Transmit+ Transmit- Receive+ Receive- Shield Chassis ground Pins 4, 5 and 7, 8 are connected to a common-mode termination. Chapter 6 Console, Alarms and Fans...
Page 76
6-10 Netra ft 1800 Hardware Reference Manual • April 1999...
C H A P T E R PCI Modules Standard PCI cards can be used in a Netra ft 1800 system when they are included in a PCI module. Supported cards are shown in . The PCI module consists of TABLE 7-1 the card and a PCI carrier which has the standard features of a module.
PCI modules A (A-PCI0 to A-PCI7) PCI modules B (B-PCI0 to B-PCI7) PCI Module Locations FIGURE 7-2 Netra ft 1800 Hardware Reference Manual • April 1999...
32-bit 1. Maximum of two per side. Note – The Netra ft 1800 has a PCI clock speed of 25MHz. Caution – The use of certain PCI cards can connect the Frame and Logic grounds (if they have been isolated by removal of the link on the rear of the chassis) and cannot be used in a dual grounding environment.
I/O cables coming into the host frame to mate with the SAI/P breakout box must meet the applicable local fire codes (for example, have a CL2 or CM Listing mark denoting, among other things, that it complies with vertical-tray flame test requirements). Netra ft 1800 Hardware Reference Manual • April 1999...
Caution – It is important that care is taken when inserting PCI card carriers into the Netra ft 1800 chassis. The only alignment aids are the card guides at the top and base of the slots; the connectors on the rear of the cards are complex and fragile, and can be damaged if the cards are not inserted with care.
Page 82
Netra ft 1800 Hardware Reference Manual • April 1999...
C H A P T E R Power Supply Units Each side of a Netra ft 1800 system has locations for up to three power supply units (PSUs). The power supply unit (PSU) is a module based on a number of PCBs with all required heatsinking and a cooling fan mounted internally.
–48V Dual Inlet PSU External circuit breakers must be fitted in the customer frame before the –48V and return rails are supplied to the chassis. The –48V PSU is front-replaceable. It has two diode-commoned –48V inputs, which are individually fused internally with voltage monitoring. The return rail of the two inputs is common.
V1, V2 and V3a outputs can operate in a redundant current-sharing mode. A maximum of three power supply modules can be operated in parallel. Power Supply DC Output Ratings TABLE 8-1 Output Voltage (V) Current (A) Power (W) +3.3 17.7 Netra ft 1800 Hardware Reference Manual • April 1999...
Page 87
Note – If the PSU re-enables during any of these steps, it must be reconfigured. See the Netra ft 1800 User’s Guide for information about configuring modules into the system. Chapter 8...
Page 88
Netra ft 1800 Hardware Reference Manual • April 1999...
A P P E N D I X RCP Status Information describes the status data returned to the CPUset. The status information is TABLE A-1 accessed using the AAstat command via the serial ports, and is returned as an ASCII string. There are therefore 38 characters representing the 19 bytes. In this case the first two characters are 26 (= 38 decimal).
Page 90
Serial port 1 initiated power on = 1 Serial port 1 initiated power off = 1 Serial port 0 initiated motherboard reset = 1 Serial port 1 initiated motherboard reset = 1 RCP reset by watchdog = 1 Netra ft 1800 Hardware Reference Manual • April 1999...
Page 92
Netra ft 1800 Hardware Reference Manual • April 1999...
Page 93
Glossary ASIC Application-Specific Integrated Circuit. Automatic System Recovery: reboot on system hang. BMX+ Crossbar switch ASIC. bridge The interface between the CPUsets and the I/O devices. Console, Alarms and Fans module. Configuration Management System. The software that records and monitors the modules in the system.
Page 94
(CMS) Power up and bring online a module that is already configured into the system. engineer-replaceable A module which may not indicate that it is faulty and which may require special tools for diagnosis and replacement. The Netra ft 1800 does not have any engineer-replaceable modules. ElectroStatic Discharge. EState Error limitation mode.
Page 95
A slot where a module can be inserted. Each location has a unique name location and is clearly marked on the chassis. lockstep The process by which two CPUsets work in synchronization. losing side The side of a split system which has a new identity when rebooted. Motherboard.
Page 96
The side of a split system which retains the identity of the previous fault tolerant system. UPA-to-PCI bridge (U2P) ASIC. UltraSPARC Port Architecture. Glossary-4 Netra ft 1800 Hardware Reference Manual • April 1999...
Page 97
Index NUMERICS combined mode (bridge operation), 2-2 console, alarms and fans module, See CAF –48V power module, 8-3 control panel, See CAF CPUset description, 3-1, 3-5 dimensions, 3-1 alarms, 2-14 LEDs, 3-7 description, 6-4 synchronization, 1-2 alternate maintenance bus, 2-7 crossbar switch See BMX+ ASICs BMX+, 3-9...
Page 98
2-5 maintenance bus, 8-3 power supply, 8-3 RCP, 8-3 modules power supply, See PSU Fault LED, 1-4 PSU, 8-1 firmware, 1-4 DC outputs, 8-4 front-replaceable, 1-2 LEDs, 8-3 identity, 1-5 Index-2 Netra ft 1800 Hardware Reference Manual • April 1999...
Page 99
supply rails, 1-10 UltraSPARC Port Architecture, See UPA switching, 1-10 UPA, 3-10 addressing, 3-10 RCP, 2-8 commands, 2-9 watchdog, RCP, 2-12 motherboard power on/off, 2-10 motherboard reset, 2-9 power supply, 8-3 program download, 2-11 reset, 2-12 serial port protocol and pinout, 2-9 status information, 2-11, A-1 status LEDs, 2-12 watchdog, 2-12...
Page 100
Index-4 Netra ft 1800 Hardware Reference Manual • April 1999...
Need help?
Do you have a question about the Netra ft 1800 and is the answer not in the manual?
Questions and answers