1. Manuals
  2. Brands
  3. Silicon Graphics Manuals
  4. Server
  5. ICE X
  6. User manual

Silicon Graphics ICE X User Manual

Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
X System 
SGI
ICE
®
™
Hardware User Guide
Document Number 007-5806-004

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ICE X and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Silicon Graphics ICE X

Summary of Contents for Silicon Graphics ICE X

  • Page 1 X System  ® ™ Hardware User Guide Document Number 007-5806-004...
  • Page 2 Contractor/manufacturer is SGI, 900 North McCarthy Blvd. Milpitas, CA 95035. TRADEMARKS AND ATTRIBUTIONS SGI, and the SGI logo are registered trademarks and Rackable, SGI Lustre and SGI ICE are trademarks of, Silicon Graphics International, in the United States and/or other countries worldwide.

  • Page 3: Record Of Revision

    Record of Revision Version Description -001 March, 2012 First release -002 February, 2013 Blade and rack design updates -003 June, 2014 Blade updates -004 November, 2015 cpower command and service reference updates 007-5806-004...

  • Page 5: Table Of Contents

    Contents List of Figures . . xi List of Tables . xiii Audience. . xv Important Information . . xv Chapter Descriptions . xvi Related Publications . .xvii Obtaining SGI Publications. xviii Conventions . . xix Product Support . . xix Reader Comments .
  • Page 6 System Power Status . . 23 System Overview . 25 System Models . . 26 SGI ICE X System and Blade Architectures . . 29 IP113 Blade Architecture Overview . 29 IP115 Blade Architecture Overview . 30 IP119 Blade Architecture Overview .
  • Page 7 SGI Warranty Levels . . 54 Rack Information . . 55 Overview . 55 SGI ICE X Series D-Rack (42U) . 56 ICE X D-Rack Technical Specifications . 61 SGI ICE X M-Cell Rack Assemblies . 62 M-Cell Functional Overview .
  • Page 8 Contents SGI ICE X Administration/Leader Servers . . 67 Overview . . 68 System Hierarchy . 68 Communication Hierarchy . . 69 1U Rack Leader Controller and Administration Server . . 71 1U Service Nodes . 72 C1104G-RP5 1U Service Node .
  • Page 9 Contents Replacing ICE X System Components . . 95 Removing and Replacing a Blade Enclosure Power Supply . . 95 Removing and Replacing Rear Fans (Blowers) . 98 Removing or Replacing a Fan Enclosure Power Supply . .102 Removing a Fan Assembly Power Supply .

  • Page 11: List Of Figures

    . 22 Figure 2-4 Chassis Management Controller Board Front Panel Ports and Indicators . 23 Figure 2-5 SGI ICE X Series System (Single Rack - Air Cooled Example) . 26 Figure 3-1 D-rack Blade Enclosure and Rack Components Example .
  • Page 12 List of Figures M-Cell Rack Configuration Example (Top View) . 62 Figure 4-5 SGI ICE X Multi-Cell (M-Cell) Rack Array Example . 64 Figure 4-6 Half M-Cell Rack Assembly (½-Cell) Example . . 65 Figure 4-7 SGI ICE X System Administration Hierarchy Example Diagram .
  • Page 13 . Table 1-1 cpower example command strings . . 11 Table 1-2 Tall SGI ICE X D-Rack Technical Specifications . . 61 Table 4-1 SGI ICE X M-Rack Technical Specifications . 63 Table 4-2 C2108-RP2 Control Panel Components .

  • Page 15: Audience

    To avoid problems that could void your warranty, your SGI or other approved service technician should perform all the setup, addition, or replacement of parts, cabling, and service of your SGI ICE X series system, with the exception of the following items that you can perform yourself: •...

  • Page 16: Chapter Descriptions

    Chapter 4, “Rack Information,” describes the system’s rack features. • Chapter 5, “SGI ICE X Administration/Leader Servers” describes all the controls, connectors and LEDs located on the front of the stand-alone administrative, rack leader and other support server nodes. An outline of the server functions is also provided.

  • Page 17: Related Publications

    This guide covers general operation, installation, configuration, and servicing of the 2U Rackable C2110G-RP5-P server node used in the SGI ICE X system. The 2U server can be used as a service node for login, batch, I/O gateway, MDS, or other service node purposes.

  • Page 18: Obtaining Sgi Publications

    • SGI Management Center Administration Guide for Clusters, (P/N 007-6358-00x) This document is intended for people who manage and administer the operation of SGI ICE X systems. The management center software is also used to administer other non-ICE SGI clusters or systems.

  • Page 19: Conventions

    Conventions Conventions The following conventions are used throughout this document: Convention Meaning This fixed-space font denotes literal items such as commands, files, Command routines, path names, signals, messages, and programming language structures. variable The italic typeface denotes variable entries and words or concepts being defined.

  • Page 20: Reader Comments

    About This Guide Reader Comments If you have comments about the technical accuracy, content, or organization of this document, contact SGI. Be sure to include the title and document number of the manual with your comments. (Online, the document number is located in the front matter of the manual. In printed manuals, the document number is located at the bottom of each page.) You can contact SGI in the following ways: •...

  • Page 21: Operation Procedures

    Chapter 1 Operation Procedures This chapter explains how to operate your new system in the following sections: • “Precautions” on page 1 • “Console Connections” on page 3 • “Powering the System On and Off” on page 4 • “Monitoring Your Server” on page 12 Precautions Before operating your system, familiarize yourself with the safety information in the following sections:...

  • Page 22: Safety Precautions

    1: Operation Procedures Safety Precautions Warning: Before operating or servicing any part of this product, read the “Safety Information” on page 113. Danger: Keep fingers and conductive tools away from high-voltage areas. Failure to follow these precautions will result in serious injury or death. The high-voltage areas of the system are indicated with high-voltage warning labels.

  • Page 23: Console Connections

    Console Connections Console Connections The flat panel console option (see Figure 1-1) has the following listed features: Slide Release - Move this tab sideways to slide the console out. It locks the drawer closed when the console is not in use and prevents it from accidentally sliding open. 2.

  • Page 24: Powering The System On And Off

    Administrative Controller Video Console Connection Points Powering the System On and Off This section explains how to power on and power off individual rack units, or your entire  SGI ICE X system, as follows: • “Preparing to Power On” on page 5 •...

  • Page 25: Preparing To Power On

    Powering the System On and Off When using the SGI cluster manager software, you can monitor and manage your server from a remote location. See the SGI Management Center Administration Guide for Clusters,  (P/N 007-6358-00x) for more information. Preparing to Power On To prepare to power on your system, follow these steps: Check to ensure that the cabling between the rack’s power distribution units (PDUs) and the wall power-plug receptacle is secure.

  • Page 26: Figure 1-4 Eight-Outlet Single-Phase Pdu Example

    1: Operation Procedures 4. Make sure that all PDU circuit breaker switches (see the examples in Figure 1-4, and Figure 1-5 on page 7) are turned on to provide power when the system is booted up. Power distribution unit (PDU) Power source Figure 1-4...

  • Page 27: Figure 1-5 Three-Phase Pdu Examples

    Powering the System On and Off Figure 1-5 Three-Phase PDU Examples 007-5806-004...

  • Page 28: Powering On And Off

    Console Management Power (cpower) Commands This section provides an overview of the console management power (cpower) commands for the SGI ICE X system. The cpower commands allow you to power on, power off, reset, and show the power status of multiple or single system components or individual racks.
  • Page 29 Powering the System On and Off Table 1-1 (continued) cpower option, action, target type and target list descriptions Argument Description Target_type Apply the action to a node or nodes. Nodes can be blade compute nodes node (inside a blade enclosure), administrative server nodes, rack leader controller nodes or service nodes.
  • Page 30 1: Operation Procedures Table 1-1 (continued) cpower option, action, target type and target list descriptions Argument Description halt Halts and then powers-off the target(s). Halts the target by issuing a halt command via ssh. Valid target types are: leader, node, system. If the target type is system, ICE compute nodes are halted first;...

  • Page 31: Table 1-2 Cpower Example Command Strings

    Powering the System On and Off Table 1-2 cpower example command strings Command Status/result Powers on all nodes in the system. # cpower system on or # cpower node on "r*i*n*" # cpower node status "r1i*n*" Determines the power status of all nodes in rack 1 (except CMCs).

  • Page 32: Monitoring Your Server

    These console connections enable you to view the status and error messages generated by your SGI ICE X system. You can also use these consoles to input commands to manage and monitor your system. See the section “System Power Status” on page 23, for additional information.

  • Page 33: Optional Sgi Remote Services (Sgi Rs)

    These are the main configurable I/O system interfaces for the SGI ICE X systems. See the particular server’s user guide for detailed information on installing optional I/O cards or other components.

  • Page 34: Sgi Remote Services Primary Capabilities

    1: Operation Procedures SGI Remote Services Primary Capabilities • 24x7 remote monitoring and data gathering of SGI UV customer systems • Alerts and notification on changes, failures and potential failures • Log files immediately available • Configuration fingerprint • Secure file transfer •...

  • Page 35: Figure 1-7 Sgi Remote Services Process Overview

    Optional SGI Remote Services (SGI RS) Cloud intelligence automatically reviews select Event Logs around the clock (every five minutes) to identify potential failure information. If the Cloud intelligence detects a critical Event, it notifies SGI support personnel. This monitoring requires no changes to customer systems or firewalls as long as the SGI Agent can send HTTPS messages to highly secure Cloud and Global Access Servers.

  • Page 37: System Management

    Note that the stand-alone service nodes use IPMI to monitor system “health”. Mass storage enclosures are not managed by the SGI ICE X system controller network. Figure 2-1 shows an example remote LAN-connected console used to monitor a single-rack (D-rack) SGI ICE X series system.

  • Page 38: Figure 2-1 Sgi Ice X System Network Access Example

    2: System Management Remote workstation monitor Local Area Network (LAN) SGI ICE X system Local Area Network (LAN) Cat-5 Ethernet Figure 2-1 SGI ICE X System Network Access Example 007-5806-004...

  • Page 39: Using The 1U Console Option

    Note: While the 1U console is normally plugged into the administrative controller server in the SGI ICE X system, it can also be connected to a rack leader controller server in the system for terminal access purposes.

  • Page 40: Chassis Manager Interconnects

    2: System Management Note: For an overview of how all the primary system components interact within the Ethernet network infrastructure communicate, see the section “System Hierarchy” in Chapter 5. Chassis Manager Interconnects The chassis managers in each blade enclosure connect to the system administration, rack leader and service node servers via Gigabit Ethernet switches.

  • Page 41: M-Rack Chassis Manager Interconnection

    Chassis Controller Interaction Note that the non-redundant example (shown in Figure 2-3) is a non-standard chassis management configuration with only a single virtual local area network (VLAN) connect line from each CMC to the internal LAN switch. See also “Multiple Chassis Manager Connections” in Chapter 3. 48-port GigE switch Rack 001 and 002 RLC...

  • Page 42: Chassis Management Control (Cmc) Functions

    2: System Management 48-port GigE switch Cooling M-rack RLC distribution VLAN unit LAN1 LAN2 VLAN Cooling controller VLAN CMC-3 CMC-7 Stacking cables CMC-2 CMC-6 VLAN CMC-1 CMC-5 LAN1 LAN2 LAN3 LAN4 Service node VLAN System admin node CMC-0 CMC-4 VLAN M-Cell M-rack CMCs LAN1 LAN2...

  • Page 43: Cmc Connector Ports And Indicators

    System Power Status • Monitors the On/Off power sequence • Monitors system resets • Applies a preset voltage to switch blades and fan control boards CMC Connector Ports and Indicators The ports on the CMC board are used as follows: •...
  • Page 44 2: System Management commands are communicating with BMCs using the IPMI protocol. Note that the term “IRU” represents a single blade enclosure within a blade enclosure pair. The cpower command is the main interface for all power management commands. You can request power status and power-on or power-off the system with commands entered via the administrative controller server or rack leader controller in the system rack.

  • Page 45: System Overview

    InfiniBand connectivity and I/O. You can install and operate the SGI ICE X series system in your lab or server room. Each 42U SGI rack holds one or two 21U-high (blade enclosure pairs). An enclosure pair is a sheetmetal assembly that consists of two 18-blade enclosures (upper and lower).

  • Page 46: System Models

    3: System Overview System Models Figure 3-1 shows an example configuration of an air-cooled single-rack SGI ICE X server. Figure 3-1 SGI ICE X Series System (Single Rack - Air Cooled Example) 007-5806-004...
  • Page 47 Chapter 4, “Rack Information” for more information on water-cooled ICE X M-Cell systems. The basic SGI ICE X system requires a minimum of one 42U tall rack with PDUs installed to support each blade enclosure pair and any support servers or storage units. Each rack supports two blade enclosure pairs.

  • Page 48: Figure 3-2 D-Rack Blade Enclosure And Rack Components Example

    3: System Overview 42U High Rack Service node Admin server Rack leader controller 1U Gig-E switch 1U Gig-E switch Blade enclosure pair 1U console CMC-0 CMC-1 CNSL RES HB PG Blade enclosure pair CMC-0 CMC-1 CNSL RES HB PG Figure 3-2 D-rack Blade Enclosure and Rack Components Example 007-5806-004...

  • Page 49: Sgi Ice X System And Blade Architectures

    IP113 Blade Architecture Overview An enhanced and updated multi-core version of the SGI ICE compute blade is used in the ICE X systems. The IP113 blade architecture is described in the following sections.

  • Page 50: Ip115 Blade Architecture Overview

    The IP115 compute blade cannot be plugged into and cannot be used in “previous generation” SGI Altix ICE 8200 or 8400 series blade enclosures. Usage in third-party or non SGI ICE X racks may be restricted due to thermal requirements.

  • Page 51: Ip119 Blade Architecture Overview

    The IP119 compute blade cannot be plugged into and cannot be used in “previous generation” SGI Altix ICE 8200 or 8400 series blade enclosures. Usage in third-party or non SGI ICE X racks may be restricted due to thermal requirements.

  • Page 52: Ip131 Blade Architecture Overview

    3: System Overview IP131 Blade Architecture Overview An enhanced and updated version of the SGI ICE X compute blade is used in IP131-based ICE X systems. One dual-socket node is accommodated on each IP131 blade assembly. The IP131 blade architecture is described in the following paragraphs:...

  • Page 53: Ip133 Blade Architecture Overview

    SGI Altix ICE 8200 or 8400 series blade enclosures. The IP133 is a custom liquid-cooled blade and usage in third-party or non SGI ICE X racks is unsupported at the time this document was published. Multi-generational system interconnects can be made through the InfiniBand fabric level.

  • Page 54: Quickpath Interconnect Features

    3: System Overview QuickPath Interconnect Features Each processor socket on an ICE X system node board is interconnected using two QuickPath Interconnect (QPI) links. Each QPI link consists of two point-to-point 20-bit channels - one send channel and one receive channel. Each QPI channel is capable of sending and receiving at the same time –...

  • Page 55: Blade Memory Features

    IP133 blades use DDR4 DIMMs, while IP113, IP115 and IP119 use DDR3 DIMM technology. Blade DIMM Memory Features Note that each Intel processor on an ICE X IP113, IP115 or IP119 node board uses four DDR3 memory channels with one or more memory DIMMs on each channel (depending on the configuration selected).

  • Page 56: System Infiniband Switch Blades

    For example: a single 1600 MT/s DIMM on a channel will only operate at speeds up to 1600 MT/s - not 2133 MT/s. Populating one 1600 MT/s DIMM on each channel of an ICE X system blade node board delivers a maximum of 12.8 GB/s per channel or 51.2 GB/s total memory bandwidth.

  • Page 57: Figure 3-3 Infiniband 48-Port (Premium) Fdr Switch Numbering In Blade Enclosures

    SGI sales or service representative for additional information on availability. The SGI ICE X FDR switch blade locations example is shown in Figure 3-3 on page 37. Any external switch blade ports not used to support the IB system fabric may be connected to optional service nodes or InfiniBand mass storage.

  • Page 58: System Features And Major Components

    3: System Overview System Features and Major Components The main features of the SGI ICE X series server systems are introduced in the following sections: • “Modularity and Scalability” on page 38 • “Reliability, Availability, and Serviceability (RAS)” on page 45 Modularity and Scalability The SGI ICE X series systems are modular, blade-based, scalable, high-density cluster systems.

  • Page 59: System Administration Server

    Each SGI ICE X system will have one system administration controller, at least one rack leader controller (RLC) and at least one service node. All ICE X systems require one RLC for every eight CMCs in the system. Figure 3-4 shows an overview of the SGI ICE X system management and component network interaction.

  • Page 60: Rack Leader Controller

    “backup” admin server. The second (backup) server runs the same system management image as the primary server. The server is used to install SGI ICE X system software, administer that software and monitor information from all the compute blades in the system. Check with your SGI sales or service representative for information on “cold spare”...

  • Page 61: The Rlc As Fabric Manager

    D-Rack Administration and RLC Cabling to CMCs Example The RLC as Fabric Manager In some SGI ICE X configurations the fabric management function is handled by the rack leader controller (RLC) node. The RLC is an independent server that is not part of the blade enclosure pair.

  • Page 62: Service Nodes

    3: System Overview Service Nodes The functionality of the service “nodes” listed in this subsection are all services that can technically be shared on a single hardware server unit. System scale, configuration and number of users generally determines when you add more servers (nodes) and dedicate them to these service functions.

  • Page 63: Batch Server Node

    If required, the I/O gateway server function can be hosted on an optional 1U or 2U stand-alone server within the ICE X system. One or more I/O gateway nodes are supported per system, based on system size and functional requirement.

  • Page 64: Optional Lustre Nodes Overview

    NFS. Optional Lustre Nodes Overview The nodes in the following subsections are used when the SGI ICE X system is set up as a Lustre file system configuration. In SGI ICE X installations the MDS and OSS functions are generally on separate nodes within the ICE X system and communicating over a network.

  • Page 65: Oss Node

    The object storage server (OSS node) is one of the elements of a Lustre File Storage system. The OSS is managed by the SGI ICE X management network. The OSS stores file data on one or more object storage targets (OSTs). Depending on the server’s hardware, an OSS node typically serves between two and eight OSTs, with each OST managing a single local disk filesystem.
  • Page 66 3: System Overview • Error detection and correction – External memory transfers are protected by cyclic redundancy check (CRC) error detection. If a memory packet does not checksum, it is retransmitted. – Nodes within each blade enclosure exceed SECDED standards by detecting and correcting 4-bit and 8-bit DRAM failures.

  • Page 67: System Components

    The SGI ICE X series system features the following major components: • 42U D-rack. This is a custom rack used for both the compute and I/O rack in the SGI ICE X series. Up to two blade enclosure pairs can be installed in each rack. Note that multi-rack systems will often have a dedicated I/O rack holding GigE switches, RLCs, Admin servers and additional service nodes.

  • Page 68: Figure 3-8 Sgi Ice X Series D-Rack Blade Enclosure Pair Components Example

    3: System Overview Chassis manager Switch blades CMC-0 CMC-1 CNSL RES HB PG Power supplies CMC-0 CMC-1 CNSL RES HB PG Figure 3-8 SGI ICE X Series D-Rack Blade Enclosure Pair Components Example 007-5806-004...

  • Page 69: Figure 3-9 Single-Node D-Rack Blade Enclosure Pair Component Front Diagram

    System Components CMC 1 Blade slot 8 Blade slot 17 Blade slot 7 Blade slot 16 Blade slot 6 Blade slot 15 Blade slot 5 Blade slot 14 9 Compute Blade slot 4 Blade slot 13 blade slots Blade slot 3 Blade slot 12 Blade slot 2 Blade slot 11...

  • Page 70: Figure 3-10 M-Rack Blade Enclosure Pair Components Example

    Chassis manager Switch blades Power supplies CMC-0 CMC-1 CNSL RES HB PG Rack mount shelf CMC-0 CMC-1 CNSL RES HB PG Figure 3-10 M-Rack Blade Enclosure Pair Components Example...

  • Page 71: D-Rack Unit Numbering

    (RLC) as well as the compute blades that reside in that rack. Optional System Components Availability of optional components for the SGI ICE X series of systems may vary based on new product introductions or end-of-life components. Some options are listed in this manual, others may be introduced after this document goes to production status.

  • Page 72: Sgi Remote Services Primary Capabilities

    SGI Remote Services provides a secure connection to SGI Customer Support - on demand. This can ensure business continuance with SGI systems management and optimization. SGI Remote Services Primary Capabilities • 24x7 remote monitoring and data gathering of SGI customer systems •...

  • Page 73: Sgi Remote Service Operations Overview

    Optional SGI Remote Services (SGI RS) SGI Remote Service Operations Overview An SGI Support Services Software Agent runs on each SGI system at your location, enabling remote system monitoring and secure communication to SGI Support staff. Your basic hardware and software configuration as well as system health information is captured and stored in the Cloud.

  • Page 74: Sgi Warranty Levels

    3: System Overview SGI Warranty Levels SGI Electronic Support services are available to customers who have a valid SGI Warranty or optional support contract. Additional electronic services may become available after publication of this document. To purchase a support contract that allows you to use all available SGI Electronic Support services, contact your SGI sales representative.

  • Page 75: Rack Information

    “M-Cell Functional Overview” on page 63 Overview At the time this document was published only specific SGI ICE X racks were approved for ICE X systems shipped from the SGI factory. See Figure 4-1 on page 57 and Figure 4-5 on page 62 for examples.

  • Page 76: Sgi Ice X Series D-Rack (42U)

    4: Rack Information SGI ICE X Series D-Rack (42U) The SGI tall D-Rack (shown in Figure 4-1 on page 57) has the following features and components: • Front and rear door. The front door is opened by grasping the outer end of the rectangular-shaped door piece and pulling outward.

  • Page 77: Figure 4-1 Sgi Ice X Series D-Rack Example

    Note that up to 16 power outlets may be needed to power a single blade enclosure pair and supporting servers installed in a single rack. Optional single-phase PDUs can be used in SGI ICE X racks dedicated to I/O functionality. Figure 4-1...

  • Page 78: Figure 4-2 Front Lock On Tall (42U) D-Rack

    4: Rack Information Figure 4-2 Front Lock on Tall (42U) D-Rack 007-5806-004...

  • Page 79: Figure 4-3 Optional Water-Chilled Door Panels On Rear Of Ice X D-Rack

    SGI ICE X Series D-Rack (42U) Figure 4-3 Optional Water-Chilled Door Panels on Rear of ICE X D-Rack 007-5806-004...

  • Page 80: Figure 4-4 Air-Cooled D-Rack Rear Door And Lock Example

    4: Rack Information Figure 4-4 Air-Cooled D-Rack Rear Door and Lock Example 007-5806-004...

  • Page 81: Ice X D-Rack Technical Specifications

    42U of the rack. If the top of the rack is “expanded” 2U, 4U, or 6U, to accommodate optional system components, the space in the extended zone is not water cooled. See “System-level Specifications” in Appendix A for a more complete listing of SGI ICE X system operating specifications and environmental requirements.

  • Page 82: Sgi Ice X M-Cell Rack Assemblies

    4: Rack Information SGI ICE X M-Cell Rack Assemblies Specific SGI ICE X system configurations require the use of enhanced “closed-loop” cooling and the compute rack assemblies are generally referred to as an “M-Cell”. A complete M-Cell assembly consists of four compute racks and two cooling racks, see Figure 4-5 for an example.

  • Page 83: M-Cell Functional Overview

    SGI ICE X M-Cell Rack Assemblies M-Cell Functional Overview An M-Cell consists of M-racks and cooling racks and in most configurations a special cooling distribution rack (CDU). When liquid-cooled blades are used in the system, the separate CDU supplies water that dissipates heat off the system CPUs.

  • Page 84: Figure 4-6 Sgi Ice X Multi-Cell (M-Cell) Rack Array Example

    4: Rack Information Figure 4-6 SGI ICE X Multi-Cell (M-Cell) Rack Array Example 007-5806-004...

  • Page 85: Figure 4-7 Half M-Cell Rack Assembly (½-Cell) Example

    SGI ICE X M-Cell Rack Assemblies Figure 4-7 Half M-Cell Rack Assembly (½-Cell) Example 007-5806-004...

  • Page 87: Sgi Ice X Administration/Leader Servers

    For purposes of this chapter “administration/controller server” is used as a catch-all phrase to describe the stand-alone servers that act as management infrastructure controllers. The specialized functions these servers perform within the SGI ICE X system primarily include: • Administration and management •...

  • Page 88: Overview

    The administrative controller server (also known as the system admin controller) is at the top of the distributed management infrastructure within the SGI ICE X system. The overall SGI ICE X series management is hierarchical (see the following subsection “System Hierarchy” and also Figure 5-1 on page 70), with the RLC(s) communicating with the compute nodes via CMC interconnect.

  • Page 89: Communication Hierarchy

    Overview Communication Hierarchy Communication within the overall management framework is as follows: Admin Node The Admin node communicates with the following: • All rack leader controllers (RLCs) • All cooling rack controllers (CRCs) • All cooling distribution units (CDUs) Rack Leader Controller (RLC) The RLCs within a “logical rack”...

  • Page 90: Figure 5-1 Sgi Ice X System Administration Hierarchy Example Diagram

    One BMC per compute node Compute blade Compute blade Each RLC talks to a maximum of 8 CMCs A maximum of 144 compute blades per rack leader controller Customer LAN Figure 5-1 SGI ICE X System Administration Hierarchy Example Diagram 007-5806-004...

  • Page 91: 1U Rack Leader Controller And Administration Server

    1U Rack Leader Controller (RLC) Server Front and Rear Panels The system administrative controller unit acts as the SGI ICE X system’s primary interface to the “outside world”, typically a local area network (LAN). The server is used by administrators to provision and manage cluster functions using SGI’s cluster management software.

  • Page 92: 1U Service Nodes

    5: SGI ICE X Administration/Leader Servers may also be used as a separate (non-RLC/admin) login, batch, I/O, MDS, OSS or fabric management node. See the section “Modularity and Scalability” on page 38 for a list of administration and support server types and additional functional descriptions.

  • Page 93: Figure 5-3 Sgi Rackable C1104G-Rp5 1U Service Node Front And Rear Panels

    1U Service Nodes System Four disk drive bays LEDs System Main reset power IPMI PCIe low-profile Optional GPU or expansion slot x16 full-height PCIe slot Ethernet ports ports port Figure 5-3 SGI Rackable C1104G-RP5 1U Service Node Front and Rear Panels Figure 5-4 SGI Rackable C1104G-RP5 System Control Panel and LEDs From left to right the LED indicators are:...

  • Page 94: 2U Service Nodes

    5: SGI ICE X Administration/Leader Servers 2U Service Nodes For systems using a separate login, batch, I/O, fabric management, or Luster service node; the following SGI 2U servers are also available as options. RP2 2U Service Nodes The SGI Rackable RP2 standard-depth servers are 2U rackmount sevice nodes. Each model of the server has two main subsystems: the 2U server chassis and a dual-processor serverboard.

  • Page 95: Table 5-1 C2108-Rp2 Control Panel Components

    2U Service Nodes Table 5-1 C2108-RP2 Control Panel Components Label Description System ID button with integrated LED NMI button (recessed, tool required for use) NIC-1 Activity LED NIC-3 Activity LED System Cold Reset button System Status LED Power button with integrated LED Hard Drive Activity LED NIC-4 Activity LED NIC-2 Activity LED...

  • Page 96: Rp2 Service Node Back Panel Components

    5: SGI ICE X Administration/Leader Servers RP2 Service Node Back Panel Components Figure 5-7 and Table 5-2 on page 76 identify the components on the back panel of the C2108-RP2 service node. Figure 5-7 RP2 Service Node Back Panel Components Example...

  • Page 97: C2110G-Rp5-P 2U Service Node

    2U Service Nodes C2110G-RP5-P 2U Service Node Figure 5-8 shows front and rear views of the SGI Rackable C2110G-RP5-P 2U service node. Note that this server uses up to eight DIMM memory cards, (four per processor). Separate QPI link pairs connect the two processors and the I/O hub in a network on the server’s motherboard.

  • Page 98: Table 5-3 C2110G-Rp5-P 2U Server Control Panel Functions (Listed Top To Bottom)

    5: SGI ICE X Administration/Leader Servers Figure 5-9 SGI Rackable C2110G-RP5-P 2U Service Node Control Panel Diagram Table 5-3 C2110G-RP5-P 2U Server Control Panel Functions (listed top to bottom) Functional feature Functional description button applies/ removes power from the power supplies to...

  • Page 99: Sgi Uv 20 2U Service Node

    2U Service Nodes SGI UV 20 2U Service Node The 2U-high SGI UV 20 server is available as a four-processor ICE X service node. This 2U service node uses four Intel E5-4600 Xeon processors and supports up to 48 DIMM memory modules plus multiple I/O modules and storage adapters.

  • Page 100: Figure 5-12 Sgi Uv 20 Service Node Front Control Panel Description

    5: SGI ICE X Administration/Leader Servers Figure 5-12 identifies and describes the functions of the SGI UV 20 service node’s front control panel. Figure 5-12 SGI UV 20 Service Node Front Control Panel Description For more information on the SGI UV 20 server, see the SGI UV 20 System User Guide, (P/N 007-5900-00x).

  • Page 101: Basic Troubleshooting

    Chapter 6 Basic Troubleshooting This chapter provides the following sections to help you troubleshoot your system: • “Troubleshooting Chart” on page 82 • “LED Status Indicators” on page 83 • “Blade Enclosure Pair Power Supply LEDs” on page 83 • “IP113 Compute Blade LEDs”...

  • Page 102: Troubleshooting Chart

    6: Basic Troubleshooting Troubleshooting Chart Table 6-1 lists recommended actions for problems that can occur. To solve problems that are not listed in this table, contact your SGI system support organization. Table 6-1 Troubleshooting Chart Problem Description Recommended Action The system will not power on. Ensure that the power cords of the enclosure are seated properly in the power receptacles.

  • Page 103: Led Status Indicators

    LED Status Indicators LED Status Indicators There are a number of LEDs visible on the front of the blade enclosures that can help you detect, identify and potentially correct functional interruptions in the system. The following subsections describe these LEDs and ways to use them to understand potential problem areas. Blade Enclosure Pair Power Supply LEDs Each power supply installed in a blade enclosure pair (six total) has one green and one amber status LED located at the right edge of the supply.

  • Page 104: Ip113 Compute Blade Leds

    6: Basic Troubleshooting IP113 Compute Blade LEDs Each IP113 compute blade installed in an enclosure has status LED indicators arranged in a single row behind the perforated sheetmetal of the blade. The LEDs are located in the front lower section of the compute blade and are visible through the screen of the compute blade, see Figure 6-2.

  • Page 105: Ip115 Compute Blade Status Leds

    LED Status Indicators This type of information can be useful in helping your administrator or service provider identify and more quickly correct hardware problems. See the following subsections for IP115 and IP119 blade-status LED information. IP115 Compute Blade Status LEDs Figure 6-3 identifies the locations of the IP115 board status LEDs.
  • Page 106 6: Basic Troubleshooting 6. Eth1 active (lower node) - this amber LED flashes when Eth1 is active (transmitting data). 7. BMC Heartbeat (lower node) - this green LED flashes when the node board’s BMC is functioning normally. The IP115 upper node board’s status LEDs (reference Figure 6-3 on page 85) are identified in the following numbered descriptions: 8.

  • Page 107: Ip119 Compute Blade Status Leds

    LED Status Indicators IP119 Compute Blade Status LEDs Figure 6-4 shows the location and identifies the status LEDs on the IP119 compute blade. Blade coolant access panel Figure 6-4 IP119 Blade Status LEDs Example The status LEDs are visible through the perforated front grill of the IP119 blade. The functions of the nine LED status lights are as follows: IB0 link (lower board)- this green LED is illuminated when a link has been established on the internal InfiniBand 0 port.

  • Page 108: Ip131 Compute Blade Status Leds

    6: Basic Troubleshooting IP131 Compute Blade Status LEDs Each IP131 compute blade installed in an enclosure has status LED indicators arranged in a single row behind the perforated sheetmetal of the blade. The LEDs are located in the front lower section of the compute blade and are visible through the screen of the compute blade, see Figure 6-5.

  • Page 109: Ip133 Compute Blade Status Leds

    LED Status Indicators IP133 Compute Blade Status LEDs Figure 6-6 shows the location and identifies the status LEDs visible through the perforated front grill of the IP133 compute blade. The blade consists of an interfaced lower and upper node board. Blade coolant access panel 10 11 12 13 14 Figure 6-6...

  • Page 110: Accessing Online Support Information And Services

    6: Basic Troubleshooting The IP133 upper node board’s status LEDs (reference Figure 6-6 on page 89) are identified in the following numbered descriptions: 8. UID - Unit identifier (upper node) - this blue LED is used during troubleshooting to find a specific compute node.The LED can be lit via software to aid in locating a compute node.

  • Page 111: Technical Assistance

    Accessing Online Support Information and Services • SGI Knowledgebase • Service call logging and tracking • Submission of technical questions Note that the customer document access on the SGI Customer Portal has replaced the SGI TechPubs Library. Technical Assistance This website topic area covers how and where to obtain direct technical assistance from SGI or our contracted service partners and includes: •...

  • Page 112: Optional Sgi Remote Services (Sgi Rs)

    6: Basic Troubleshooting To purchase a support contract that allows you to use the complete SGI Electronic Support services (such as SGI RS), contact your SGI sales representative. Optional SGI Remote Services (SGI RS) The optional SGI RS system automatically detects system conditions that indicate potential failure, see the section “Optional SGI Remote Services (SGI RS)”...

  • Page 113: Maintenance Procedures

    “Maintenance Precautions and Procedures” on page 93 • “Installing or Removing Internal Parts” on page 94 Note: These procedures are intended for D-rack based ICE X systems. Check with your support provider for information on M-Cell maintenance. Maintenance Precautions and Procedures This section describes how to access the system for specific types of customer approved maintenance and protect the components from damage.

  • Page 114: Preparing The System For Maintenance Or Upgrade

    7: Maintenance Procedures Preparing the System for Maintenance or Upgrade To prepare the system for maintenance, you can follow the guidelines in “Powering On and Off” in Chapter 1 and power down the affected blade enclosure pair. The section also has information on powering-up the enclosure after you have completed the maintenance/upgrade required.

  • Page 115: Replacing Ice X System Components

    Replacing ICE X System Components Replacing ICE X System Components While many of the blade enclosure components are not considered end-user replaceable, a select number of components can be removed and replaced. These include: • Blade enclosure pair power supplies (front of system) •...

  • Page 116: Figure 7-1 Removing An Enclosure Power Supply

    7: Maintenance Procedures C - 0 C - 1 Press latch to release Figure 7-1 Removing an Enclosure Power Supply 5. Align the rear of the replacement power supply with the enclosure opening. 6. Slide the power supply into the chassis until the retention latch engages. 7.

  • Page 117: Figure 7-2 Replacing An Enclosure Power Supply

    Replacing ICE X System Components C - 0 C - 1 Figure 7-2 Replacing an Enclosure Power Supply 007-5806-004...

  • Page 118: Removing And Replacing Rear Fans (Blowers)

    7: Maintenance Procedures Removing and Replacing Rear Fans (Blowers) The blade enclosure cooling fan assembly (blower enclosure) is positioned back-to-back with the blade enclosure pair. You will need to access the rack from the back to remove and replace a fan. The enclosure’s system controller issues a warning message when a fan is not running properly.

  • Page 119: Figure 7-3 Enclosure-Pair Rear Fan Assembly (Blowers)

    Replacing ICE X System Components Fan 5 Fan 11 Fan 4 Fan 10 Fan 3 Fan 9 Fan power box Fan 2 Fan 8 Fan 1 Fan 7 Fan 0 Fan 6 Figure 7-3 Enclosure-Pair Rear Fan Assembly (Blowers) 007-5806-004...

  • Page 120: Figure 7-4 Removing A Fan From The Rear Assembly

    7: Maintenance Procedures Use the following steps and illustrations to replace an enclosure fan: Using the #1 Phillips screwdriver, undo the (captive) screw (located in the middle of the blower assembly handle). The handle has a notch for the screw access, see Figure 7-4. 2.

  • Page 121: Figure 7-5 Replacing An Enclosure Fan

    Replacing ICE X System Components 3. Slide a new blower assembly completely into the open slot, see Figure 7-5. 4. Tighten the blower assembly screw to secure the new fan. Note: If you disconnected the AC power to the rear fan assembly prior to the replacement procedure, all the fans will come on and run at top speed when power is reapplied.

  • Page 122: Removing Or Replacing A Fan Enclosure Power Supply

    7: Maintenance Procedures Removing or Replacing a Fan Enclosure Power Supply The 12-fan (blower) assembly that is mounted back-to-back with the blade enclosure pair to provide cooling uses two power supplies to provide voltage to the blowers. Removal and replacement of a blower assembly power supply requires the use of a T-25 torx driver. Removing a Fan Assembly Power Supply Use the following information and illustrations to remove a power supply from the fan (blower) assembly enclosure:...

  • Page 123: Figure 7-6 Removing A Power Supply From The Fan Power Box

    Replacing ICE X System Components Pull handle Loosen screw Press latch to release Figure 7-6 Removing a Power Supply From the Fan Power Box 007-5806-004...

  • Page 124: Figure 7-7 Replacing A Power Supply In The Fan Power Box

    7: Maintenance Procedures Tighten screw Figure 7-7 Replacing a Power Supply in the Fan Power Box 007-5806-004...

  • Page 125: Overview Of Pci Express Operation

    PCI/PCI-X technology. Check with your SGI sales or service representative for more detail on PCI Express board options available with your SGI ICE X system. PCI Express is compatible with PCI/PCI-X in the following ways: •...

  • Page 126: Table 7-2 Sgi Administrative Server Pcie Support Levels

    7: Maintenance Procedures PCI Express technology uses two pairs of wires for each transmit and receive connection (4 wires total). These four wires are generally referred to as a lane or x1 connection (also called “by 1”). SGI administrative node PCIe technology uses x16, x8 and x4 connector technology in the PCI Express card slots.

  • Page 127: Technical Specifications And Pinouts

    “D-Rack System Environmental Specifications” on page 109 • “ICE X M-Rack Technical Specifications” on page 110 • “Ethernet Port Specification” on page 112 System-level Specifications Table A-1 summarizes the SGI ICE X series configuration ranges. Table A-1 SGI ICE X Series Configuration Ranges Category Minimum Maximum...

  • Page 128: D-Rack Physical And Power Specifications

    A: Technical Specifications and Pinouts D-Rack Physical and Power Specifications Table A-2 shows the physical specifications of the SGI ICE X system based on the D-Rack. Table A-2 ICE X System D-Rack Physical Specifications System Features (single rack) Specification Height 79.5 in.

  • Page 129: D-Rack System Environmental Specifications

    Ambient air or optional water cooling Heat dissipation to air Approximately 115.63 kBTU/hr maximum (based on 33.89 kW - 100% dissipation to air) Air-cooled ICE X (rack) Heat dissipation to air Approximately 5.76 kBTU/hr maximum (based on 33.89 kW - 5% dissipation to air)

  • Page 130: Ice X M-Rack Technical Specifications

    Table A-4 provides information on the individual physical specifications of the compute racks used in an M-Cell assembly. Table A-5 on page 111 lists the environmental specifications for an individual M-Rack. Table A-4 SGI ICE X M-Rack Physical Specifications Characteristic Specification Height 93 in.

  • Page 131: Table A-5 Environmental Specifications (Single M-Rack)

    System-level Specifications Table A-5 Environmental Specifications (Single M-Rack) Feature Specification +5 C (41 F) to +35 C (95 F) (up to 1500 m / 5000 ft.)  Temperature tolerance +5 C (41 F) to +30 C (86 F) (1500 m to 3000 m /5000 ft. to 10,000 ft.) (operating with 95-Watt processors) +5 C (41 F) to +28 C (82.4 F) (up to 1500 m / 5000 ft.) ...

  • Page 132: Ethernet Port Specification

    A: Technical Specifications and Pinouts Ethernet Port Specification The system auto-selects the Ethernet port speed and type (duplex vs. half-duplex) when the server is booted, based on what it is connected to. Figure A-1 shows the Ethernet port. Pin 4 Pin 5 Pin 6 Pin 3...

  • Page 133: Safety Information And Regulatory Specifications

    Appendix B Safety Information and Regulatory Specifications This appendix provides safety information and regulatory specifications for your system in the following sections: • “Safety Information” on page 113 • “Regulatory Specifications” on page 115 Safety Information Read and follow these instructions carefully: Follow all warnings and instructions marked on the product and noted in the documentation included with this product.
  • Page 134 B: Safety Information and Regulatory Specifications 9. Do not attempt to service this product yourself except as noted in this guide. Opening or removing covers of node and switch internal components may expose you to dangerous voltage points or other risks. Refer all servicing to qualified service personnel. 10.

  • Page 135: Regulatory Specifications

    Regulatory Specifications Regulatory Specifications The following topics are covered in this section: • “CMN Number” on page 115 • “CE Notice and Manufacturer’s Declaration of Conformity” on page 115 • “Electromagnetic Emissions” on page 115 • “Shielded Cables” on page 118 •...

  • Page 136: Fcc Notice (Usa Only)

    B: Safety Information and Regulatory Specifications FCC Notice (USA Only) This equipment complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: • This device may not cause harmful interference. • This device must accept any interference received, including interference that may cause undesired operation.

  • Page 137: Vcci Notice (Japan Only)

    Regulatory Specifications Cet appareil numérique német pas de perturbations radioélectriques dépassant les normes applicables aux appareils numériques de Classe A préscrites dans le Règlement sur les interferences radioélectriques établi par le Ministère des Communications du Canada. VCCI Notice (Japan Only) Figure B-1 VCCI Notice (Japan Only) Chinese Class A Regulatory Notice...

  • Page 138: Shielded Cables

    B: Safety Information and Regulatory Specifications Shielded Cables This SGI system is FCC-compliant under test conditions that include the use of shielded cables between the system and its peripherals. Your system and any peripherals you purchase from SGI have shielded cables. Shielded cables reduce the possibility of interference with radio, television, and other devices.

  • Page 139: Lithium Battery Statements

    Regulatory Specifications Warning: Advertencia: Radiación láser invisible al ser abierto. Evite exponerse a los rayos. Warning: Advarsel: Laserstråling vedåbning se ikke ind i strålen Warning: Varo! Lavattaessa Olet Alttina Lasersåteilylle Warning: Varning: Laserstrålning når denna del år öppnad ålå tuijota såteeseenstirra ej in i strålen.
  • Page 140 B: Safety Information and Regulatory Specifications Warning: Advarsel: Eksplosjonsfare ved feilaktig skifte av batteri. Benytt samme batteritype eller en tilsvarende type anbefalt av apparatfabrikanten. Brukte batterier kasseres i henhold til fabrikantens instruksjoner. Warning: Varning: Explosionsfara vid felaktigt batteribyte. Anvãnd samma batterityp eller en ekvivalent typ som rekommenderas av apparattillverkaren.

  • Page 141: Index

    Index All ICE X servers environmental specifications monitoring locations An ICE X single-rack server illustration front panel display L1 controller battery statements block diagram system laser compliance statements LED Status Indicators LEDs on the front of the IRUs lithium battery warning statements...
  • Page 142 Index monitoring locations system architecture 23, 25 optional water chilled rack cooling system block diagram system components SGI ICE X front list of physical specifications system features System Physical Specifications system overview pinouts Ethernet connector Power Supply LEDs powering on...

Table of Contents