Page 3
New Features Cray XD1™ Hardware Installation and Upgrade HR6–6402–E This rewrite of Cray XD1 Hardware Installation and Upgrade (HR6–6402) supports the general availability of release 1.4 of the Cray XD1 product. It includes the WEEE compliance statement.
Page 5
Version Description May 2006 Supports the general availability of release 1.4 of the Cray XD1 product. Includes the Waste Electrical and Electronic Equipment (WEEE) compliance statement. October 2005 Supports the general availability of release 1.3 of the Cray XD1 product. Minor changes.
United States Canada Safety Compliance: Warnings and Cautions Sicherheitshinweis VCCI-A Compliance Statement WEEE Compliance Statement Overview of Cray XD1 Installation [2] About the Cray XD1 Computer Cray XD1 Chassis Cray XD1 System Interprocessor Communications Overview of the Installation Process Required Tools and Equipment HR6–6402–E...
Page 8
To install the anti-tip foot Installing the Cabinet Trim on a 42-VU Cabinet Procedure 9: To install the cabinet trim: Connecting Cables to the Cray XD1 System [5] Cabling Guidelines General Cabling Guidelines Cable Routing Guidelines Routing Ethernet and Fibre Channel Cables Cray Private HR6–6402–E...
Page 9
Interconnecting the RapidArray Fabrics Interconnecting the Supervisory Network Overview of the Supervisory Network Connecting 100-Mbps Ethernet Ports Connecting the Cray XD1 System to External Devices PCI-X Card Connections and Numbering Connecting to a LAN via NIC Ports Procedure 10: To connect to a LAN via a NIC port...
Page 10
Cray XD1™ Hardware Installation and Upgrade Page Installing External Switches Installing an Ethernet Switch Procedure 17: To install an Ethernet switch Installing a 24-port RapidArray Switch Procedure 18: To install 24-port RapidArray switches Installing Cabinet Blanking Panels Removing a Chassis from a Cabinet...
Page 11
Figure 32. RapidArray switch and mounting hardware, rear view Figure 33. RapidArray switch with mounting bracket (detail) Figure 34. Cable troughs, side view (cabinet side panels not shown) Figure 35. Install cabinet blanking panels Figure 36. Disconnect power HR6–6402–E Cray Private...
Preface The information in this preface is common to Cray documentation provided with this software release. Accessing Product Documentation With each software release, Cray provides books and man pages, and in some cases, third-party documentation. These documents are provided in the...
Cray XD1™ Hardware Installation and Upgrade Conventions These conventions are used throughout Cray documentation: Convention Meaning This fixed-space font denotes literal items, such as file command names, pathnames, man page names, command names, and programming language elements. variable Italic typeface indicates an element that you will replace with a specific value.
Cray User Group The Cray User Group (CUG) is an independent, volunteer-organized international corporation of member organizations that own or use Cray Inc. computer systems. CUG facilitates information exchange among users of Cray systems through technical papers, platform-specific e-mail lists, workshops, and conferences.
1.2 Scope of This Manual This manual introduces you to the Cray XD1 computer and describes how to install and cable Cray XD1 hardware components. For more details on the hardware components in a Cray XD1 chassis, see Cray XD1 System Overview (S–2429).
Cray XD1™ Hardware Installation and Upgrade 1.4 Related Publications Refer to the publications in Table 1, page 2 for more information on the Cray XD1 computer and the Active Manager software. Table 1. Related publications Publication title Brief description Identifies the main new features and Cray XD1 Release Description (S–2453)
• WEEE Compliance Do not make any mechanical or electrical modifications to the equipment that are not approved by Cray Inc. Cray Inc. is not responsible for the regulatory compliance of a Cray product with unapproved mechanical or electrical modifications.
Standby (Figure 2, page 4). Power is in Standby mode. Power is not removed from the device until the power cord plug is disconnected from the power source. Figure 2. Standby symbol For your protection, adhere to the following safety precautions during installation and maintenance of the Cray XD1 computer. Cray Private HR6–6402–E...
Page 21
Caution: At least two people are required to lift the product. Keep your back as straight as possible and lift with your legs. For equipment with Fibre Channel PCI-X cards: Caution: Class 1 laser product. HR6–6402–E Cray Private...
Bereitschaft (Figure 2, page 4). Gerät ist in Bereitschaft. Spannung vorhanden bis Netzstecker gezogen ist. Befolgen Sie zu Ihrer eigenen Sicherheit die folgenden Sicherheitsanweisungen während Installation und Wartung des Cray XD1 Computers. Warning: Warnung: Diese Produkt kann nicht vom Anwender gewartet werden. Wartung nur durch qualifiziertes Personal.
1.5.5 WEEE Compliance Statement Refer to Figure 4, page 8 for the Cray XD1 system Waste Electrical and Electronic Equipment.(WEEE) compliance statement. HR6–6402–E Cray Private...
Cray XD1™ Hardware Installation and Upgrade In accordance with European Directive 2002/96/EC on Waste Electrical and Electronic Equipment (WEEE), the presence of the crossed-out wheelie bin symbol on a product or on its packaging indicates that you must not dispose of that item in the normal unsorted municipal waste stream.
Note: Each instance of the Linux operating system is a processing node in a Cray XD1 system. The term node is used in this and other Cray XD1 manuals to refer to an instance of the operating system and the hardware components that it controls.
Cray XD1 System Overview (S–2429). Figure 5, page 10 shows the front view of a Cray XD1 chassis; Figure 6, page 10 shows the rear view. Figure 5. Cray XD1 chassis, front view Figure 6.
The term system is used throughout this manual to describe either a stand-alone chassis or a network of interconnected chassis. In a typical installation, a maximum of 13 Cray XD1 chassis are mounted in a 23-in. cabinet. One or more cabinets reside in a computer room with a controlled environment.
Cray XD1 RapidArray Interconnect Topologies (HR6–6425). For multichassis systems, connect the Section 5.3.2, page 36 supervisory network. Connect cables to provide the Cray XD1 Section 5.4.2, page 39 system with LAN access. If applicable, connect Fibre Channel cables Section 5.4.3, page 40 to provide the Cray XD1 system with access to external storage.
Cray XD1™ Hardware Installation and Upgrade 2.3 Required Tools and Equipment You must provide the following tools and equipment to install a Cray XD1 system: • No. 2 Phillips screwdriver • No. 2 flat blade screwdriver • Stubby flat blade screwdriver •...
3.3 Unpacking a System Cabinet Warning: A fully populated Cray XD1 system cabinet is very heavy. Do not attempt to unpack a Cray XD1 system cabinet alone. This procedure requires at least four people to ensure personal safety and to protect the equipment.
Cray XD1™ Hardware Installation and Upgrade Use the following procedures to unpack the cabinets hipping crate and to remove the cabinet from the shipping pallet. Refer to Figure 7, page 16. Horizontal strap Wooden ramp Figure 7. Cabinet shipping crate Procedure 1: To unpack a system cabinet 1.
Page 33
• Place the ramp on the pallet so that the holes on the ramp align with the roll pins on the pallet. 11. Roll the cabinet off the pallet to the location that your site plan indicates. HR6–6402–E Cray Private...
Cray XD1™ Hardware Installation and Upgrade Figure 8. Installing the ramp Warning: The system is heavy, and momentum can carry it down the ramp quickly. Ensure that the system does not roll off the pallet or the system can be damaged.
Chapter 6, page 45. 4.1 Removing and Installing the Cabinet Side Panels For each row of Cray XD1 cabinets, you must install a pair of cabinet side panels to ensure proper airflow and a pleasing appearance. For instructions on installing blanking panels, which are required to cover external switches and unoccupied vertical units in the cabinet, refer to Section 6.6, page 58.
Cray XD1™ Hardware Installation and Upgrade 2. Lift the top panel off the bottom panel and set it aside. 3. Place your hand against the surface of the bottom panel to support it and loosen the 2 quarter-turn fasteners at the upper surface of the panel.
Figure 9. Chassis shipping bracket, right side 3. Remove the 2 screws that attach the bracket to the side of the chassis. Do not remove the slide assembly that is mounted to the lower front corner of each side of the chassis. HR6–6402–E Cray Private...
If you have a raised-floor installation, position the cabinets as follows: • Do not place the cabinets on perforated floor tiles. • Place the cabinets over the floor cutouts as shown in Figure 10, page 23. See Cray XD1 Site Planning (HR6–6401) for details on floor preparation. Cray Private HR6–6402–E...
Front Figure 10. Cabinet position relative to floor cutouts 4.4 Routing Power Circuits This section describes how to route power circuits for Cray XD1 chassis and external switches. 4.4.1 Routing Power Circuits for Cray XD1 Chassis Each Cray XD1 chassis requires its own power circuit: a 20-amp circuit for chassis with a three-phase power supply, or a 30-amp circuit for chassis with a single-phase power supply.
Page 40
Cray XD1™ Hardware Installation and Upgrade The Cray XD1 chassis has a relatively short power cord—2 feet (0.6 meters) long—because its plug is the primary power disconnect device and so must be located as close to the chassis as possible. In addition, in a multichassis system, you must be able to identify the correct plug for each chassis.
Figure 11. Power circuit routing 4.4.2 Routing Power Circuits for External Switches Depending on the customer order, Cray installs a maximum of two vertical power strips in the cabinet: one on the left side and one on the right. These power strips control the On/Off status of the external switches.
Cray XD1™ Hardware Installation and Upgrade 4.5 Installing the Ganging Kits Cray provides ganging kits (also known as joiner kits) to conceal the gaps between the front and rear ends of adjacent cabinets that have no side panels. These kits enhance both the appearance and linear alignment of a row of cabinets.
Page 43
You may need to adjust the height of the stabilizing legs to align the cabinets vertically. Refer to Figure 13, page 28. 6. Attach a joiner plate to each pair of interlocked half-joiners (front and rear) by using the black mounting hardware provided in the joiner kits. HR6–6402–E Cray Private...
If the equipment in your computer room is arranged in hot and cold aisles, ensure that the cabinet leveling feet are on solid floor tiles; refer to Cray XD1 Site Planning Cray Private...
2. Locate the 4 holes in the base structure at the front of the cabinet frame. 3. Place a flat washer on each of the four 10-32 X 0.50-in. Phillips mounting screws, insert a screw through each slot and into the mounting holes, and secure the foot to the frame. HR6–6402–E Cray Private...
Cray XD1™ Hardware Installation and Upgrade 4.8 Installing the Cabinet Trim on a 42-VU Cabinet This section explains how to add the cabinet trim to the 42-VU system cabinet. You do not need any tools for this procedure. The cabinet trim kit consists of four items: •...
Page 47
Installing and Preparing a Cabinet [4] Figure 15. Lowering the crown into the right-side vertical trim piece Figure 16, page 31 shows the installed crown piece. Figure 16. System cabinet with the crown trim installed HR6–6402–E Cray Private...
Page 48
Cray XD1™ Hardware Installation and Upgrade 4. Install any filler pieces, required only if the cabinet has RapidArray switches installed. Each filler piece covers 3 RapidArray switches; see Figure 17, page 32. Figure 17. Filler trim piece For each: • Align the trim fasteners on the back of the filler piece to the 4 brackets that are attached to the RapidArray switch mounting hardware;...
System [5] This chapter provides cabling guidelines and describes how to connect various types of cables within a Cray XD1 cabinet and between a Cray XD1 cabinet and other devices. Before you complete the procedures described here, Cray recommends that you plan the RapidArray topology for your system, as well as your LAN access and storage access requirements.
Fibre Channel plugs. See Section 5.4.3.1, page 40 for bend radius details. Route these cables to the left of the cabinet, as viewed from the rear. Cray recommends that you use a cable tray to manage the logic cables; refer to Figure 18, page 35.
Connecting Cables to the Cray XD1 System [5] Cables in tray Rear of cabinet Figure 18. Ethernet and Fibre Channel cable management 5.3 Interconnecting the Cray XD1 Internal Networks The Cray XD1 system has two internal networks: • The RapidArray interconnect provides high-speed interprocessor connections in a multichassis system.
Cray XD1 RapidArray Interconnect Topologies (HR6–6425). 5.3.2 Interconnecting the Supervisory Network The two 100-Mbps Ethernet ports on the back of the Cray XD1 chassis serve two purposes: they interconnect the supervisory network in a multichassis system and provide service access to a field engineer. The externally visible ports provide connections to an internal 100-Mbps Ethernet switch;...
5.4 Connecting the Cray XD1 System to External Devices This section describes the mapping between PCI-X cards and nodes, describes how to connect the Cray XD1 system to site networks and external storage devices, and describes how to connect a JTAG interface card to a PC for FPGA debugging purposes.
Cray XD1™ Hardware Installation and Upgrade 5.4.1 PCI-X Card Connections and Numbering The four slots for the PCI-X cards are numbered 1 to 4, left to right, viewed from the back of the chassis. All PCI-X cards connect to nodes 5 and 6; see Table 4, page 38.
Connecting Cables to the Cray XD1 System [5] 5.4.2 Connecting to a LAN via NIC Ports Ethernet cables enable you to connect the Cray XD1 system to one or more site networks (on your LAN) via ports on network interface cards (NICs). You need a CAT 5E cable with RJ–45 connectors for each connection.
Cray XD1™ Hardware Installation and Upgrade CAT 5E cable to switch Figure 21. Connect a NIC port 5.4.3 Connecting to External Storage Devices If you connect to an external storage device (such as a SAN device) via a Fibre Channel connection, the system software automatically detects the device and loads the drivers.
Connecting Cables to the Cray XD1 System [5] 5. Label both ends of the cable. 6. Repeat for all Fibre Channel connections. 7. Route the cables as described in Section 5.2, page 34. Fibre Channel cable to SAN device Figure 22. Connect a Fibre Channel port 5.4.4 Connecting the JTAG Interface Card to a Personal Computer...
(straight connection) between a port on the JTAG interface card and a personal computer that is running debugging software. Use an adapter (RJ-45 to DB25), available from Cray, to connect the cable to the parallel port on your computer. Procedure 12: To connect the JTAG interface card 1.
5.5 Connecting Power Cables Three-phase and single-phase versions of the power supply are available. For details on the electrical specifications of a Cray XD1 chassis, see Cray XD1 Site Planning (HR6–6401). The three-phase and NEMA single-phase power cord plug’s connection to the receptacle is secured by a twist-lock mechanism.
Page 60
Cray XD1™ Hardware Installation and Upgrade 8. When you are ready to power on the Cray XD1 system (after all chassis are installed and cabled), refer to Cray XD1 System Administration (S–2430) for details on the commissioning process and the startup sequences. Do not power on any chassis in the system until you are ready to commission the system.
Each 42-VU Cray XD1 cabinet holds a maximum of 13 Cray XD1 chassis in a direct-connect topology or a maximum of 12 Cray XD1 chassis and switch chassis in a switched topology.
6.1.3 Routing Power Circuits When you upgrade a cabinet, you may also need to route the power circuits for the Cray XD1 chassis and external switches; refer to Section 4.4.1, page 23 for instructions. 6.2 Installing a Chassis in a Cabinet As you install a chassis in your system, record its location.
Upgrading a Cray XD1 System [6] 6.2.2 Before You Begin Complete the following activities (if necessary) before you mount a Cray XD1 chassis in a cabinet: • Remove any blanking panels from the cabinet to make room for the new chassis.
Cray XD1™ Hardware Installation and Upgrade The lock lever, which fits through the handle on the right side of the chassis, locks the chassis in place. Slide out rails until fully extended Align center rails on chassis with rail assemblies on cabinet Figure 26.
If they arrive separately, install them before you power on the chassis. This section describes both procedures. A Cray XD1 chassis has one to three disk blades, each with one or two serial ATA (SATA) disk drives; see Figure 28, page 49. All disk drives are configured at the factory with the same software.
3 of the chassis that you plan to commission first. This is required because nodes 5 and 6 attempt to boot first when you commission the chassis. For information on the commissioning process and the startup sequence, see Cray XD1 System Administration (S–2430).
To minimize cable lengths, Cray recommends that you place Ethernet switches at the bottom of the cabinet. Install any required RapidArray switches in the middle of the cabinet between two banks of Cray XD1 chassis, unless you have a separate switch cabinet. For instructions on connecting cables between a Cray XD1 chassis and a switch, see Section 5.3.2.2, page 36.
Cray XD1™ Hardware Installation and Upgrade 6.4.1 Installing an Ethernet Switch Install the switch so that its ports face toward the rear of the cabinet. Use four 1/2-in. Phillips screws to mount the external Ethernet switches to the customized bracket below the bottom chassis. Refer to Figure 30, page 52.
You can install as many as six 24-port RapidArray switches in a single set of adapter brackets. If you are installing the RapidArray switches in the same cabinet as Cray XD1 chassis, install the switches in the middle of the cabinet between two banks of Cray XD1 chassis.
Page 70
Cray XD1™ Hardware Installation and Upgrade Air baffles Adapters Mounting brackets for 3U trim fillers (2 on each side) Mounting brackets for RA switch Figure 31. RapidArray switch and mounting hardware, front view 3. Use 4 10-32 X 0.5 PHP screws to fasten a 23-in. to 19-in. adapter to the cabinet panel and the rear edge of the air baffle.
Page 71
Upgrading a Cray XD1 System [6] Adapters Mounting brackets Figure 32. RapidArray switch and mounting hardware, rear view 5. Loosen the screws you installed in step 2 and slide a mounting bracket under a pair of screw heads in each of the 4 positions that Figure 31, page 54 illustrates.
Page 72
57. The exact location and configuration of these supports depends on local system requirements. Refer to Section 5.1.1, page 33 for general cabling guidelines. Refer to Cray XD1 RapidArray Interconnect Topologies (HR6–6425) for instructions on how to Cray Private...
Page 73
Upgrading a Cray XD1 System [6] interconnect various system topologies. Observe good cabling practices (for example, do not use a tie gun) to avoid damaging the cables. Rear of cabinet Side view Figure 34. Cable troughs, side view (cabinet side panels not shown) 12.
Cray XD1™ Hardware Installation and Upgrade 6.6 Installing Cabinet Blanking Panels After you install all Cray XD1 chassis in a cabinet, install blanking panels on the front of the cabinet to cover any unoccupied vertical units. Cabinet blanking panels are required to maintain proper airflow within the enclosure. For instructions on installing side panels, see Section 4.1, page 19.
6.7 Removing a Chassis from a Cabinet Perform the following shutdown procedure if the chassis is not yet commissioned; otherwise, see Cray XD1 System Administration (S–2430) for the correct shutdown procedure. Procedure 19: To remove a chassis from a cabinet 1.
Page 76
RapidArray topology for the remaining chassis as required for the new (smaller) topology. In either case, you must update the logical topology of the system. For RapidArray cabling instructions, see Cray XD1 RapidArray Interconnect Topologies (HR6–6425). For instructions on how to update the logical topology of the system, see the chapter on managing the RapidArray interconnect in Cray XD1 System Administration (S–2430).
Installation Checklist [A] This appendix provides a checklist for you to use when you install a Cray XD1 system. It lists tasks required to install and prepare a cabinet, install one or more chassis into a cabinet, and connect cables. Cray recommends that you make a copy of the checklist and use it to keep track of tasks as you complete them.
Page 78
Cray XD1™ Hardware Installation and Upgrade See Cray XD1 System Administration (S–2430) for instructions. Do not power on any chassis until you are ready to commission the system. Cray Private HR6–6402–E...
13 chassis can be directly connected in a Cray XD1 system. See also fat tree. disk blade A circuit board of a Cray XD1 chassis that holds one or two SATA disk drives. fabric The collection of fabric components that interconnect in the same switching plane.
Page 80
X and fabric Y, respectively. fascia One of two perforated covers on the front of a Cray XD1 chassis. Both are hinged and can be flipped down to provide access to the fans. The right-side fascia also holds the LCD.
Page 81
Glossary logical topology The set of all fabric paths in a physical topology. It enables any node in a Cray XD1 system to communicate with any other node. main board The primary circuit board in a Cray XD1 chassis. The main board has a...
Page 82
Serial Advanced Technology Attachment. Symmetric multiprocessor. supervisory network The private 100 Mbps Ethernet network internal to the Cray XD1 system that the Active Manager hardware supervisory subsystem uses to monitor and proactively manage the health of the system. Administrators or Cray field engineers can use the supervisory network to commission the Cray XD1 system or diagnose problems.
Need help?
Do you have a question about the XD1 and is the answer not in the manual?
Questions and answers