1. Manuals
  2. Brands
  3. Cray Manuals
  4. Desktop
  5. XD1
  6. Manual

Cray XD1 RapidArray Manual

Hide thumbs Also See for XD1 RapidArray:
Table of Contents

Advertisement

Quick Links

Download this manual
Cray XD1™ RapidArray
Interconnect Topologies
Private
HR6–6425–F

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the XD1 RapidArray and is the answer not in the manual?

Questions and answers

Related Manuals for Cray XD1 RapidArray

Summary of Contents for Cray XD1 RapidArray

  • Page 1 Cray XD1™ RapidArray Interconnect Topologies Private HR6–6425–F...
  • Page 2 © 2006 Cray Inc. All Rights Reserved. Unpublished Private Information. This unpublished work is protected to trade secret, copyright and other laws. Except as permitted by contract or express written permission of Cray Inc., no part of this work or its content may be used, reproduced or disclosed in any form.
  • Page 3 New Features Cray XD1™ RapidArray Interconnect Topologies HR6–6425–F This revision includes the WEEE compliance statement. This revision also incorporates information on the following new topics: • Cabling 36-chassis fat tree systems • Cabling 48-chassis fat tree systems • Configuring nonstandard fat tree topologies by depopulating standard base topologies...
  • Page 5 Record of Revision Version Description May 2006 Updates to support the general availability of release 1.4 of the Cray XD1 product. Includes the Waste Electrical and Electronic Equipment (WEEE) compliance statement. October 2005 Supports the generally available 1.3 release of the Cray XD1 product. Minor changes only.

  • Page 7: Table Of Contents

    Contents Page Preface Accessing Product Documentation Conventions Reader Comments xiii Cray User Group xiii Cray XD1 Support Introduction [1] Who Should Read This Manual Scope of This Manual How This Manual is Organized Related Publications WEEE Compliance Statement Interconnecting the RapidArray Network [2]...
  • Page 8 Cray XD1™ RapidArray Interconnect Topologies Page Cabling Direct-connect Topologies [3] Overview Direct-connect Topology Equipment Required Time Required Installation Order Direct-connect Topology: Two Chassis Direct-connect Topology: Three Chassis Direct-connect Topology: Four Chassis Direct-connect Topology: Five Chassis Direct-connect Topology: Six Chassis Direct-connect Topology: Seven Chassis...
  • Page 9 Cabling a 36-Chassis Fat Tree [7] Overview Cable Inventory Port Groups (Fabrics) Installation Order Port Mapping Details Fabric A Fabric B Fabric C Fabric D Fabric E Fabric F Cabling a 48-Chassis Fat Tree [8] Overview Cable Inventory HR6–6425–F Cray Private...
  • Page 10 Depopulating Cables Depopulating 24-port Switch Chassis Minimum Number of 144-port Switch Chassis Depopulating Leaf Blades in 144-port Switch Chassis Minimum Number of Cray XD1 Chassis Systems with 24-port Switches Example 4: Minimum number of chassis with 24-port switches Systems with 144-port Switches...
  • Page 11 RapidArray cables for a 12-chassis fat tree system Table 20. Fat tree topology port groups: 12 chassis Table 21. Port mapping for a 12-chassis fat tree system—Fabric A Table 22. Port mapping for a 12-chassis fat tree system—Fabric B HR6–6425–F Cray Private...
  • Page 12 Cray XD1™ RapidArray Interconnect Topologies Page Table 23. Port mapping for a 12-chassis fat tree system—Fabric C Table 24. Port mapping for a 12-chassis fat tree system—Fabric D Table 25. RapidArray cables for a 24-chassis fat tree system Table 26.
  • Page 13 Figure 23. Logical view of 48-chassis fat tree configuration Figure 24. Cabinet layout of 48-chassis system (front view) Figure 25. Cabinet layout of 48-chassis system (rear view) Figure 26. Port groups and port mapping rules for 48-chassis system HR6–6425–F Cray Private...

  • Page 15: Preface

    Preface The information in this preface is common to Cray documentation provided with this software release. Accessing Product Documentation With each software release, Cray provides books and man pages, and in some cases, third-party documentation. These documents are provided in the...

  • Page 16: Conventions

    Cray XD1™ RapidArray Interconnect Topologies Conventions These conventions are used throughout Cray documentation: Convention Meaning This fixed-space font denotes literal items, such as file command names, pathnames, man page names, command names, and programming language elements. variable Italic typeface indicates an element that you will replace with a specific value.

  • Page 17: Reader Comments

    Cray User Group The Cray User Group (CUG) is an independent, volunteer-organized international corporation of member organizations that own or use Cray Inc. computer systems. CUG facilitates information exchange among users of Cray systems through technical papers, platform-specific e-mail lists, workshops, and conferences.

  • Page 18: Cray Xd1 Support

    Cray XD1™ RapidArray Interconnect Topologies Cray XD1 Support Obtain support for the Cray XD1 product in either of the following ways: Telephone: 1–888–279–2729 (Cray XD1 Customer Support Center) Through the CRInform website: http://crinform.cray.com/xd/ Note: Use the contact information provided here if you have a support agreement with Cray.

  • Page 19: Introduction [1]

    1.2 Scope of This Manual This manual describes how to physically interconnect all the chassis of one Cray XD1 system to implement the supported system topologies. It gives general instructions for handling RapidArray cables, preparing to cable the system, and routing and securing cables. It includes complete details for cabling every supported direct-connect topology and for cabling the supported base fat tree topologies.

  • Page 20: Related Publications

    Provides a Chassis ID form and a Switch Chassis ID form for you to record your chassis and switch chassis IDs. You will need this information when you commission your system. 1.4 Related Publications Table 1, page 3 lists the Cray XD1 publications that supplement the contents of this manual. Cray Private HR6–6425–F...

  • Page 21: Weee Compliance Statement

    Information about resolved issues and known issues for a particular release of the product. 1.5 WEEE Compliance Statement Refer to Figure 1, page 4 for the Cray XD1 system Waste Electrical and Electronic Equipment (WEEE) compliance statement. HR6–6425–F Cray Private...

  • Page 22: Figure 1. Weee Compliance Statement

    Cray XD1™ RapidArray Interconnect Topologies In accordance with European Directive 2002/96/EC on Waste Electrical and Electronic Equipment (WEEE), the presence of the crossed-out wheelie bin symbol on a product or on its packaging indicates that you must not dispose of that item in the normal unsorted municipal waste stream.

  • Page 23: Interconnecting The Rapidarray Network [2]

    Cray services are available. Topology planning determines the number of RapidArray cables that you need and the length of each; cables are available from Cray Inc. Before you connect any cables, review the bend radius information provided in the next section.

  • Page 24: Bend Radii

    Cray XD1™ RapidArray Interconnect Topologies 2.2.2 Bend Radii The minimum bend radius of a RapidArray cable is based on the gauge of the wire. The RapidArray cable shows the manufacturer name, wire gauge, and technical data about the cable at approximately 18-in. intervals along the cable.

  • Page 25: Disconnecting Rapidarray Cables

    RapidArray switch Figure 3. RapidArray cable connection Note: If you connect a RapidArray cable to a port on a Cray XD1 chassis or switch chassis, and the amber and/or green port lights flash rapidly for several seconds, disconnect and reattach the cable. This condition may result from attaching the cable connector at an angle to the chassis/switch chassis connector so that the connectors mate on one side before the other.

  • Page 26: Cabling The Topology

    These labels help you reconnect the cables if you replace a chassis or temporarily disconnect one from the topology. 3. Record the ID and position of each Cray XD1 chassis and each switch chassis (if any) in the topology. You need this information when you commission the system;...

  • Page 27: Routing And Securing The Cables For Your System

    1. Work the cable over its length to get the correct bend into the cable. Do not force a new bend in the cable. 2. Attach one end of the cable to its source: either the Cray XD1 chassis or a switch chassis.

  • Page 28: Figure 4. Rapidarray Cable Bundles

    Cray XD1™ RapidArray Interconnect Topologies Figure 4. RapidArray cable bundles Cray Private HR6–6425–F...

  • Page 29: Cabling Direct-Connect Topologies [3]

    3.1.1 Direct-connect Topology The current release supports the direct-connect topology. In such a topology, each Cray XD1 chassis connects directly to all other chassis in the system. This topology, which is also known as all-to-all mesh, requires no external switches (switch chassis) and incurs the minimum number of hops during interprocessor communication.

  • Page 30: Equipment Required

    Cray XD1™ RapidArray Interconnect Topologies Figure 5. Direct-connect topology–high-level 3.1.2 Equipment Required Cray XD1 systems—which have 12 RapidArray connectors on each chassis– require the following equipment for cabling direct-connect topologies: • RapidArray cables; see the appropriate topology in this chapter for the cable lengths and quantities your system requires •...

  • Page 31: Table 3. Port Mapping For A 2-Chassis Direct-Connect System

    (3.3 feet). Table 3, page 13 lists the port mapping for the two-chassis direct-connect system. Each row in this table represents one cable. Table 3. Port mapping for a 2-chassis direct-connect system From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...

  • Page 32: Direct-Connect Topology: Three Chassis

    Chassis Port Chassis Port Feet Meters Figure 6, page 14 shows two Cray XD1 chassis that interconnect in a direct connect topology; each chassis has a single internal RapidArray switch. CHASSIS 2 Port number Bundle cable number CHASSIS 1 Figure 6. Direct-connect topology: 2 chassis 3.3 Direct-connect Topology: Three Chassis...

  • Page 33: Table 4. Port Mapping For A 3-Chassis Direct-Connect System

    Cabling Direct-connect Topologies [3] Table 4. Port mapping for a 3-chassis direct-connect system From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...
  • Page 34 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Figure 7, page 17 shows three Cray XD1 chassis that interconnect in a direct connect topology; each chassis has a single internal RapidArray switch. Cray Private HR6–6425–F...

  • Page 35: Direct-Connect Topology: Four Chassis

    Note that if the chassis are not adjacent, they will require different cable lengths than this section recommends. Port mapping remains the same. The wiring in this system consists of bundles of four cables each (24 cables total). HR6–6425–F Cray Private...

  • Page 36: Table 5. Port Mapping For A 4-Chassis Direct-Connect System

    Cray XD1™ RapidArray Interconnect Topologies The cables are all the same length—one meter (3.3 feet). Table 5, page 18 lists the port mapping for the four-chassis direct-connect system. Each row in this table represents one cable. Table 5. Port mapping for a 4-chassis direct-connect system...
  • Page 37 Cabling Direct-connect Topologies [3] From Cable length Chassis Port Chassis Port Feet Meters Figure 8, page 20 shows four Cray XD1 chassis that interconnect in a direct connect topology; each chassis has a single internal RapidArray switch. HR6–6425–F Cray Private...

  • Page 38: Direct-Connect Topology: Five Chassis

    Cray XD1™ RapidArray Interconnect Topologies CHASSIS 4 CHASSIS 3 CHASSIS 2 CHASSIS 1 Figure 8. Direct-connect topology: 4 chassis 3.5 Direct-connect Topology: Five Chassis A five-chassis direct-connect system consists of five adjacent chassis that interconnect without intermediate switches. Three RapidArray cables connect each pair of chassis.

  • Page 39: Table 6. Port Mapping For A 5-Chassis Direct-Connect System

    The cables are all the same length—one meter (3.3 feet). Table 6, page 21 lists the port mapping for the five-chassis direct-connect system. Each row in this table represents one cable. Table 6. Port mapping for a 5-chassis direct-connect system From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...
  • Page 40 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Cray Private HR6–6425–F...
  • Page 41 Cabling Direct-connect Topologies [3] From Cable length Chassis Port Chassis Port Feet Meters Figure 9, page 24 shows five Cray XD1 chassis that interconnect in a direct connect topology; each chassis has a single internal RapidArray switch. HR6–6425–F Cray Private...

  • Page 42: Figure 9. Direct-Connect Topology: 5 Chassis

    Cray XD1™ RapidArray Interconnect Topologies CHASSIS 5 CHASSIS 4 CHASSIS 3 CHASSIS 2 CHASSIS 1 Figure 9. Direct-connect topology: 5 chassis Cray Private HR6–6425–F...

  • Page 43: Direct-Connect Topology: Six Chassis

    Table 8, page 25 lists the port mapping for the six-chassis direct-connect system. Each row in this table represents one cable. Table 8. Port mapping for a 6-chassis direct-connect system From Cable length Port Port Feet Meters Chassis Chassis HR6–6425–F Cray Private...
  • Page 44 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Cray Private HR6–6425–F...
  • Page 45 Cabling Direct-connect Topologies [3] From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...
  • Page 46 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Figure 10, page 29 shows six Cray XD1 chassis that interconnect in a direct connect topology; each chassis has a single internal RapidArray switch. Cray Private HR6–6425–F...

  • Page 47: Figure 10. Direct-Connect Topology: 6 Chassis

    Cabling Direct-connect Topologies [3] CHASSIS 6 CHASSIS 5 CHASSIS 4 CHASSIS 3 CHASSIS 2 CHASSIS 1 Figure 10. Direct-connect topology: 6 chassis HR6–6425–F Cray Private...

  • Page 48: Direct-Connect Topology: Seven Chassis

    Cray XD1™ RapidArray Interconnect Topologies 3.7 Direct-connect Topology: Seven Chassis A seven-chassis direct-connect system consists of seven adjacent chassis that interconnect without intermediate switches. Two RapidArray cables connect the pair of chassis. Note that if the chassis are not adjacent, they will require different cable lengths than this section recommends.
  • Page 49 Cabling Direct-connect Topologies [3] From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...
  • Page 50 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Cray Private HR6–6425–F...
  • Page 51 Cabling Direct-connect Topologies [3] From Cable length Chassis Port Chassis Port Feet Meters Figure 11, page 34 shows seven Cray XD1 chassis that interconnect in a direct connect topology; each chassis has a single internal RapidArray switch. HR6–6425–F Cray Private...

  • Page 52: Figure 11. Direct-Connect Topology: 7 Chassis

    Cray XD1™ RapidArray Interconnect Topologies CHASSIS 7 CHASSIS 6 CHASSIS 5 CHASSIS 4 CHASSIS 3 CHASSIS 2 CHASSIS 1 Figure 11. Direct-connect topology: 7 chassis Cray Private HR6–6425–F...

  • Page 53: Direct-Connect Topology: Eight To Thirteen Chassis

    For the 8- through 12-chassis systems, you cable the Set A connections for only those chassis installed. For example, if you are cabling an eight-chassis direct-connect system, connect the cables for Chassis 1 through 8 of the 13-chassis system. HR6–6425–F Cray Private...

  • Page 54: Table 12. Port Mapping For A 13-Chassis Direct-Connect System-Set A

    Cray XD1™ RapidArray Interconnect Topologies Table 12. Port mapping for a 13-chassis direct-connect system–Set A From Cable length Chassis Port Chassis Port Feet Meters Cray Private HR6–6425–F...
  • Page 55 Cabling Direct-connect Topologies [3] From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...
  • Page 56 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Cray Private HR6–6425–F...
  • Page 57 Cabling Direct-connect Topologies [3] From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...
  • Page 58 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Cray Private HR6–6425–F...
  • Page 59 From Cable length Chassis Port Chassis Port Feet Meters Figure 12, page 42 shows the Set A connections of 13 Cray XD1 chassis that interconnect in a direct-connect topology; each chassis has a single internal RapidArray switch. HR6–6425–F Cray Private...

  • Page 60: Figure 12. Direct-Connect Topology: 8 To 13 Chassis-Set A

    Cray XD1™ RapidArray Interconnect Topologies CHASSIS 13 CHASSIS 12 CHASSIS 11 CHASSIS 10 CHASSIS 9 CHASSIS 8 CHASSIS 7 CHASSIS 6 CHASSIS 5 CHASSIS 4 CHASSIS 3 CHASSIS 2 CHASSIS 1 Figure 12. Direct-connect topology: 8 to 13 chassis—Set A Cray Private HR6–6425–F...

  • Page 61: Set B Connections

    Table 13 lists the port mapping for the Set B connections of the 8-chassis direct-connect system. Each row in this table represents one cable. Table 13. Port mapping for an 8-chassis direct-connect system—Set B From Cable length Port Port Feet Meters Chassis Chassis HR6–6425–F Cray Private...
  • Page 62 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Table 14 lists the port mapping for the Set B connections of the 9-chassis direct-connect system. Each row in this table represents one cable. Cray Private...

  • Page 63: Table 14. Port Mapping For A 9-Chassis Direct-Connect System-Set B

    Cabling Direct-connect Topologies [3] Table 14. Port mapping for a 9-chassis direct-connect system—Set B From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...

  • Page 64: Table 15. Port Mapping For A 10-Chassis Direct-Connect System-Set B

    Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Table 15 lists the port mapping for the Set B connections of the 10-chassis direct-connect system. Each row in this table represents one cable. Table 15. Port mapping for a 10-chassis direct-connect system—Set B...

  • Page 65: Table 16. Port Mapping For An 11-Chassis Direct-Connect System-Set B

    Table 16 lists the port mapping for the Set B connections of the 11-chassis direct-connect system. Each row in this table represents one cable. Table 16. Port mapping for an 11-chassis direct-connect system—Set B From Cable length Chassis Port Chassis Port Feet Meters HR6–6425–F Cray Private...
  • Page 66 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Chassis Port Feet Meters Table 17 lists the port mapping for the Set B connections of the 12-chassis direct-connect system. Each row in this table represents one cable. Cray Private...

  • Page 67: Table 17. Port Mapping For A 12-Chassis Direct-Connect System-Set B

    Cabling Direct-connect Topologies [3] Table 17. Port mapping for a 12-chassis direct-connect system—Set B From Cable length Chassis Port Chassis Port Feet Meters Figure 13, page 50 shows the Set B connections for the 8- through 12-chassis systems. HR6–6425–F Cray Private...

  • Page 68: Figure 13. Direct-Connect Topology: 8 To 13 Chassis-Set B

    Cray XD1™ RapidArray Interconnect Topologies Applies to Set B = Port 12 partition size 11,12 RA link 10,11,12 10,11 9,10 Chassis number Set B fabric partition Figure 13. Direct-connect topology: 8 to 13 chassis—Set B Cray Private HR6–6425–F...

  • Page 69: Cabling Fat Tree Topologies: Introduction [4]

    first layer of switching. You need to purchase additional external switches only for the second and higher layers of switching. As shown in Figure 14, page 51, in a fully populated topology, each Cray XD1 chassis connects to each external switch chassis with one or more links. All 12 main ports of the Cray XD1 chassis connect to switch chassis.

  • Page 70: Base Configurations And Supported Configurations

    4.2 Base Configurations and Supported Configurations The current release of the Cray XD1 product can support fat tree configurations that contain up to 48 Cray XD1 chassis. The supported fat tree configurations use only the main RapidArray fabric, not the expansion fabric.

  • Page 71: Number Of Switch Chassis

    Figure 16, page 54 shows the numbering of the leaf blades and ports on a 144-port switch chassis. These ports are identified by the combination of their leaf blade number and port number within the leaf blade. HR6–6425–F Cray Private...

  • Page 72: Equipment Required

    Cray XD1™ RapidArray Interconnect Topologies Port 1 Port 2 Port 3 Port 4 Port 5 Port 6 Port 7 Port 8 Port 9 Port 10 Port 11 Port 12 Leaf Port 1 Port 2 Port 3 Port 4 Port 5 Port 6 Port 7 Port 8 Port 9 Port 10 Port 11 Port 12...

  • Page 73: Cabling A 12-Chassis Fat Tree [5]

    A 12-chassis fat tree system consists of a single cabinet. The cabinet holds 2 groups of 6 chassis that are separated by 6 switch chassis. Two RapidArray cables connect each Cray XD1 chassis and each switch chassis. Figure 17, page 56 shows the configuration.

  • Page 74: Figure 17. Fat Tree Topology: 12 Chassis

    Cray XD1™ RapidArray Interconnect Topologies 1-12 1..6 1-6..1 P13 7..12 1-6..1 P19 1-11 1..6 1-6..1 P14 7..12 1-6..1 P20 1-10 Cray 1..6 1-6..1 P15 7..12 1-6..1 P21 Chassis 1..6 1-6..1 P16 7..12 1-6..1 P22 1..6 1-6..1 P17 7..12 1-6..1 P23 1..6...

  • Page 75: Port Groups (Fabrics)

    The tables in this section provide the mapping for each of the port groups. 5.4.1 Fabric A Table 21, page 58 lists the port mapping for Fabric A (ports 19 to 24). Each row in this table represents one cable. HR6–6425–F Cray Private...

  • Page 76: Table 21. Port Mapping For A 12-Chassis Fat Tree System-Fabric A

    Cray XD1™ RapidArray Interconnect Topologies Table 21. Port mapping for a 12-chassis fat tree system—Fabric A From Cable length Chassis Port Switch Port Feet Meters Cray Private HR6–6425–F...

  • Page 77: Fabric B

    Table 22, page 59 lists the port mapping for Fabric B (ports 7 to 12). Each row in this table represents one cable. Table 22. Port mapping for a 12-chassis fat tree system—Fabric B From Cable length Chassis Port Switch Port Feet Meters HR6–6425–F Cray Private...
  • Page 78 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Port Feet Meters Cray Private HR6–6425–F...

  • Page 79: Fabric C

    Table 23, page 61 lists the port mapping for Fabric C (ports 13 to 18). Each row in this table represents one cable. Table 23. Port mapping for a 12-chassis fat tree system—Fabric C From Cable length Port Port Feet Meters Chassis Switch HR6–6425–F Cray Private...

  • Page 80: Fabric D

    Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Port Feet Meters 5.4.4 Fabric D Table 24, page 62 lists the port mapping for Fabric D (ports 1 to 6). Each row in this table represents one cable.
  • Page 81 Cabling a 12-Chassis Fat Tree [5] From Cable length Chassis Port Switch Port Feet Meters HR6–6425–F Cray Private...
  • Page 82 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Port Feet Meters Cray Private HR6–6425–F...

  • Page 83: Cabling A 24-Chassis Fat Tree [6]

    6.6 ft (2 m) 8.2 ft (2.5 m) 9.8 ft (3 m) Figure 18, page 66 shows 2 cabinets, each with 2 groups of 6 Cray XD1 chassis that are separated by 6 switch chassis and interconnect in a fat tree topology. HR6–6425–F...

  • Page 84: Figure 18. Fat Tree Topology: 24 Chassis

    Cray XD1™ RapidArray Interconnect Topologies Cabinet 2 Cabinet 1 2-12 1-12 1..6 1-6..1 P13 7..12 2-6..1 P19 1..6 2-6..1 P13 7..12 1-6..1 P19 2-11 1-11 1..6 1-6..1 P14 7..12 2-6..1 P20 1..6 2-6..1 P14 7..12 1-6..1 P20 2-10 1-10 Cray 1..6...

  • Page 85: Port Groups (Fabrics)

    6.4 Port Mapping Details The tables in this section provide the mapping for each of the port groups. 6.4.1 Fabric A Table 27, page 68 lists the port mapping for Fabric A (ports 19 to 24). Start and HR6–6425–F Cray Private...

  • Page 86: Table 27. Port Mapping For A 24-Chassis Fat Tree System-Fabric A

    Cray XD1™ RapidArray Interconnect Topologies end points for Fabric A are in the same cabinet. Each row in this table represents one cable. Table 27. Port mapping for a 24-chassis fat tree system—Fabric A From Cable length Port Port Feet...

  • Page 87: Fabric B

    Fabric B are in the same cabinet. Each row in this table represents one cable. Table 28. Port mapping for a 24-chassis fat tree system—Fabric B From Cable length Port Port Feet Meters Chassis Switch HR6–6425–F Cray Private...
  • Page 88 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Port Feet Meters Cray Private HR6–6425–F...

  • Page 89: Table 29. Port Mapping For A 24-Chassis Fat Tree System-Fabric C1

    Cabinet 2 and ends on the chassis in Cabinet 1. Each row in this table represents one cable. Table 29. Port mapping for a 24-chassis fat tree system—Fabric C1 From Cable length Port Port Feet Meters Cabinet Chassis Cabinet Switch HR6–6425–F Cray Private...
  • Page 90 Cray XD1™ RapidArray Interconnect Topologies From Cable length Cabinet Chassis Port Cabinet Switch Port Feet Meters 6.4.4 Fabric C2 Table 30, page 73 lists the port mapping for Fabric C2 (ports 13 to 18). Start and end points for Fabric C2 are in the adjacent cabinets. Fabric C2 starts on the switches in Cabinet 1 and ends on the chassis in Cabinet 2.

  • Page 91: Table 30. Port Mapping For A 24-Chassis Fat Tree System-Fabric C2

    Cabling a 24-Chassis Fat Tree [6] Table 30. Port mapping for a 24-chassis fat tree system—Fabric C2 From Cable length Cabinet Chassis Port Cabinet Switch Port Feet Meters HR6–6425–F Cray Private...

  • Page 92: Table 31. Port Mapping For A 24-Chassis Fat Tree System-Fabric D1

    Cray XD1™ RapidArray Interconnect Topologies From Cable length Cabinet Chassis Port Cabinet Switch Port Feet Meters 6.4.5 Fabric D1 Table 31, page 74 lists the port mapping for Fabric D1 (ports 1 to 6). Start and end points for Fabric D1 are in the adjacent cabinets. Fabric D1 starts on the switches in Cabinet 2 and ends on the chassis in Cabinet 1.
  • Page 93 Cabling a 24-Chassis Fat Tree [6] From Cable length Cabinet Chassis Port Cabinet Switch Port Feet Meters HR6–6425–F Cray Private...

  • Page 94: Table 32. Port Mapping For A 24-Chassis Fat Tree System-Fabric D2

    Cray XD1™ RapidArray Interconnect Topologies 6.4.6 Fabric D2 Table 32, page 76 lists the port mapping for Fabric D2 (ports 1 to 6). Start and end points for Fabric D2 are in the adjacent cabinets. Fabric D2 starts on the switches in Cabinet 1 and ends on the chassis in Cabinet 2.
  • Page 95 Cabling a 24-Chassis Fat Tree [6] From Cable length Cabinet Chassis Port Cabinet Switch Port Feet Meters HR6–6425–F Cray Private...
  • Page 96 Cray XD1™ RapidArray Interconnect Topologies Cray Private HR6–6425–F...

  • Page 97: Cabling A 36-Chassis Fat Tree [7]

    Figure 20, page 80 shows a front view of the physical arrangement of cabinets and the cabinet numbers. This cabinet numbering scheme allows the system to grow to 4 cabinets of Cray XD1 chassis without renumbering the original 3 cabinets.

  • Page 98: Figure 20. Cabinet Layout Of 36-Chassis System (Front View)

    Cray XD1™ RapidArray Interconnect Topologies Switch Cabinet XD1 Cabinet #3 XD1 Cabinet #1 XD1 Cabinet #2 Figure 20. Cabinet layout of 36-chassis system (front view) Figure 21, page 81 shows a rear view of the cabinet layout. It also shows the switch chassis numbers.

  • Page 99: Cable Inventory

    Table 33. RapidArray cables for a 36-chassis fat tree system Cable length Quantity 4.9 ft (1.5 m) 6.6 ft (2 m) 8.2 ft (2.5 m) 9.8 ft (3 m) 13.1 ft (4 m) 16.4 ft (5 m) HR6–6425–F Cray Private...

  • Page 100: Port Groups (Fabrics)

    This information applies to each of the switch chassis. At the Cray XD1 chassis end, all the ports (1 to 12) on each chassis belong to the same port group. The switch chassis leaf and port numbers are as defined in Figure 16, page 54.

  • Page 101: Installation Order

    Figure 22, page 82 fully defines a general rule that determines the mapping of Cray XD1 ports to switch chassis ports in a 36-chassis system. For convenience, the tables in this section provide the mapping details for each of the port groups.

  • Page 102: Fabric A

    Cray XD1™ RapidArray Interconnect Topologies 7.5.1 Fabric A Table 35, page 84 lists the port mapping for Fabric A. Each row in this table represents one cable. All chassis in Fabric A are in cabinet 1. Table 35. Port mapping for a 36-chassis fat tree system—Fabric A...
  • Page 103 Cabling a 36-Chassis Fat Tree [7] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis HR6–6425–F Cray Private...
  • Page 104 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis Cray Private HR6–6425–F...

  • Page 105: Fabric B

    Table 36, page 87 lists the port mapping for Fabric B. Each row in this table represents one cable. All chassis in Fabric B are in cabinet 1. Table 36. Port mapping for a 36-chassis fat tree system—Fabric B From Cable length Port Port Feet Meters Chassis Switch Leaf chassis HR6–6425–F Cray Private...
  • Page 106 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis Cray Private HR6–6425–F...
  • Page 107 Cabling a 36-Chassis Fat Tree [7] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 HR6–6425–F Cray Private...

  • Page 108: Fabric C

    Cray XD1™ RapidArray Interconnect Topologies 7.5.3 Fabric C Table 37, page 90 lists the port mapping for Fabric C. Each row in this table represents one cable. All chassis in Fabric C are in cabinet 2. Table 37. Port mapping for a 36-chassis fat tree system—Fabric C...
  • Page 109 Cabling a 36-Chassis Fat Tree [7] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis HR6–6425–F Cray Private...
  • Page 110 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis Cray Private HR6–6425–F...

  • Page 111: Fabric D

    Table 38, page 93 lists the port mapping for Fabric D. Each row in this table represents one cable. All chassis in Fabric D are in cabinet 2. Table 38. Port mapping for a 36-chassis fat tree system—Fabric D From Cable length Port Port Feet Meters Chassis Switch Leaf chassis HR6–6425–F Cray Private...
  • Page 112 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis Cray Private HR6–6425–F...
  • Page 113 Cabling a 36-Chassis Fat Tree [7] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis HR6–6425–F Cray Private...

  • Page 114: Fabric E

    Cray XD1™ RapidArray Interconnect Topologies 7.5.5 Fabric E Table 39, page 96 lists the port mapping for Fabric E. Each row in this table represents one cable. All chassis in Fabric E are in cabinet 3. Table 39. Port mapping for a 36-chassis fat tree system—Fabric E...
  • Page 115 Cable length Chassis Port Switch Leaf Port Feet Meters chassis 16.4 16.4 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 16.4 16.4 16.4 16.4 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 HR6–6425–F Cray Private...
  • Page 116 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 Cray Private...

  • Page 117: Fabric F

    Table 40. Port mapping for a 36-chassis fat tree system—Fabric F From Cable length Port Port Feet Meters Chassis Switch Leaf chassis 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 16.4 16.4 16.4 16.4 16.4 16.4 HR6–6425–F Cray Private...
  • Page 118 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 16.4 16.4 13.1 13.1 13.1 13.1 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 13.1 13.1 13.1 13.1 16.4 16.4 16.4 16.4 16.4 16.4 16.4...
  • Page 119 Cabling a 36-Chassis Fat Tree [7] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 13.1 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 HR6–6425–F Cray Private...
  • Page 120 Cray XD1™ RapidArray Interconnect Topologies Cray Private HR6–6425–F...

  • Page 121: Cabling A 48-Chassis Fat Tree [8]

    In the standard configuration, all five cabinets are adjacent, and the switching cabinet is the central one. Figure 24, page 104 shows a front view of the physical arrangement of cabinets and the cabinet numbers. This cabinet numbering scheme is consistent with that of the 36-chassis system. HR6–6425–F Cray Private...

  • Page 122: Figure 24. Cabinet Layout Of 48-Chassis System (Front View)

    Cray XD1™ RapidArray Interconnect Topologies Switch Cabinet XD1 Cabinet #3 XD1 Cabinet #1 XD1 Cabinet #2 XD1 Cabinet #4 Figure 24. Cabinet layout of 48-chassis system (front view) Figure 25, page 104 shows a rear view of the cabinet layout. It also shows the switch chassis numbers.

  • Page 123: Cable Inventory

    Fabric A, Fabric B, Fabric C, Fabric D, Fabric E, Fabric F, Fabric G, and Fabric H. Figure 25, page 104 shows the Cray XD1 chassis in each port group, and Figure 26, page 106 shows the switch chassis ports in each port group.

  • Page 124: Table 42. Fat Tree Topology Port Groups: 48 Chassis

    This information applies to each of the switch chassis. At the Cray XD1 chassis end, all the ports (1 to 12) on each chassis belong to the same port group. The switch chassis port numbers are as defined in Figure 16, page 54.

  • Page 125: Installation Order

    Figure 26, page 106 fully defines a general rule that determines the mapping of Cray XD1 ports to switch chassis ports in a 48-chassis system. For convenience, the tables in this section provide the mapping details for each of the port groups.

  • Page 126: Fabric A

    Cray XD1™ RapidArray Interconnect Topologies 8.5.1 Fabric A Table 43, page 108 lists the port mapping for Fabric A. Each row in this table represents one cable. All chassis in Fabric A are in cabinet 1. Table 43. Port mapping for a 48-chassis fat tree system—Fabric A...
  • Page 127 Cabling a 48-Chassis Fat Tree [8] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 HR6–6425–F Cray Private...
  • Page 128 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis Cray Private HR6–6425–F...

  • Page 129: Fabric B

    All chassis in Fabric B are in cabinet 1. Table 44. Port mapping for a 48-chassis fat tree system—Fabric B From Cable length Port Port Feet Meters Chassis Switch Leaf chassis 13.1 13.1 13.1 HR6–6425–F Cray Private...
  • Page 130 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 Cray Private HR6–6425–F...
  • Page 131 Cabling a 48-Chassis Fat Tree [8] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 HR6–6425–F Cray Private...

  • Page 132: Fabric C

    Cray XD1™ RapidArray Interconnect Topologies 8.5.3 Fabric C Table 45, page 114 lists the port mapping for Fabric C. Each row in this table represents one cable. All chassis in Fabric C are in cabinet 2. Table 45. Port mapping for a 48-chassis fat tree system—Fabric C...
  • Page 133 Cabling a 48-Chassis Fat Tree [8] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis HR6–6425–F Cray Private...
  • Page 134 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis Cray Private HR6–6425–F...

  • Page 135: Fabric D

    Table 46, page 117 lists the port mapping for Fabric D. Each row in this table represents one cable. All chassis in Fabric D are in cabinet 2. Table 46. Port mapping for a 48-chassis fat tree system—Fabric D From Cable length Port Port Feet Meters Chassis Switch Leaf chassis HR6–6425–F Cray Private...
  • Page 136 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis Cray Private HR6–6425–F...
  • Page 137 Cabling a 48-Chassis Fat Tree [8] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis HR6–6425–F Cray Private...

  • Page 138: Fabric E

    Cray XD1™ RapidArray Interconnect Topologies 8.5.5 Fabric E Table 47, page 120 lists the port mapping for Fabric E. Each row in this table represents one cable. All chassis in Fabric E are in cabinet 3. Table 47. Port mapping for a 48-chassis fat tree system—Fabric E...
  • Page 139 Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 16.4 16.4 16.4 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 HR6–6425–F Cray Private...
  • Page 140 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 Cray Private...

  • Page 141: Fabric F

    Table 48. Port mapping for a 48-chassis fat tree system—Fabric F From Cable length Port Port Feet Meters Chassis Switch Leaf chassis 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 HR6–6425–F Cray Private...
  • Page 142 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 13.1 13.1 16.4 16.4 16.4 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 13.1 13.1 13.1...
  • Page 143 Cabling a 48-Chassis Fat Tree [8] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 HR6–6425–F Cray Private...

  • Page 144: Fabric G

    Cray XD1™ RapidArray Interconnect Topologies 8.5.7 Fabric G Table 49, page 126 lists the port mapping for Fabric G. Each row in this table represents one cable. All chassis in Fabric G are in cabinet 4. Table 49. Port mapping for a 48-chassis fat tree system—Fabric G...
  • Page 145 Cable length Chassis Port Switch Leaf Port Feet Meters chassis 19.7 19.7 19.7 19.7 19.7 16.4 16.4 16.4 16.4 16.4 16.4 19.7 19.7 19.7 19.7 19.7 19.7 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 HR6–6425–F Cray Private...
  • Page 146 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 16.4 16.4 16.4 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 16.4 16.4 16.4 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 Cray Private...

  • Page 147: Fabric H

    Table 50. Port mapping for a 48-chassis fat tree system—Fabric H From Cable length Port Port Feet Meters Chassis Switch Leaf chassis 16.4 16.4 16.4 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 13.1 16.4 16.4 16.4 13.1 13.1 13.1 13.1 13.1 13.1 16.4 HR6–6425–F Cray Private...
  • Page 148 Cray XD1™ RapidArray Interconnect Topologies From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 13.1 13.1 13.1 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 13.1 13.1 13.1 16.4 16.4...
  • Page 149 Cabling a 48-Chassis Fat Tree [8] From Cable length Chassis Port Switch Leaf Port Feet Meters chassis 16.4 16.4 16.4 16.4 16.4 13.1 13.1 13.1 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 16.4 HR6–6425–F Cray Private...
  • Page 150 Cray XD1™ RapidArray Interconnect Topologies Cray Private HR6–6425–F...

  • Page 151: Configuring Nonstandard Fat Trees [9]

    Choose as your starting point the first standard base configuration that has at least as many Cray XD1 chassis as the size to which you expect the system to grow. This minimizes the number of new components to buy and the effort to change the cabling as you expand the system.

  • Page 152: Example 2: 28-Chassis System: Removing Chassis From 48-Chassis Configuration

    9.2.3 Depopulating Cables Remove all cables that, in the standard base configuration, connect on at least one end to a Cray XD1 chassis or a switch chassis that you remove. 9.2.4 Depopulating 24-port Switch Chassis In configurations that use the 24-port switch chassis, remove switch chassis in pairs.

  • Page 153: Minimum Number Of Cray Xd1 Chassis

    Configuring Nonstandard Fat Trees [9] This occurs if you remove all 12 Cray XD1 chassis in a cabinet. For systems that are based on the standard 36-chassis configuration, 4 leaf blades become unused in each switch chassis for each cabinet that you completely depopulate. For systems that are based on the standard 48-chassis configuration, 3 leaf blades...

  • Page 154: Systems With 144-Port Switches

    Example 5: Minimum number of chassis with 144-port switches If a system contains 2 144-port switch chassis, and each switch chassis has 9 leaf blades, the minimum number of Cray XD1 chassis is 9. 9.2.8 Cabling Remaining Components Cable the components that remain in the depopulated system exactly as specified in this manual for the corresponding components of the corresponding standard base configuration.

  • Page 155: Appendix A Id Information

    As you cable your system, record all the chassis and switch chassis IDs. You will need this information when you commission the system; see the chapter about setting up the system in Cray XD1 System Administration (S–2430). A.1.1 Chassis IDs The chassis ID, its default IP address, and serial number are on a sticker behind the right-side fascia.

  • Page 156: Table 52. Chassis Ids (Fat Tree Systems)

    Cray XD1™ RapidArray Interconnect Topologies Use Table 52, page 138 to record chassis ID information for a fat tree system. Table 52. Chassis IDs (fat tree systems) Chassis Number Chassis ID Location 1-10 1-11 1-12 2-10 2-11 Cray Private HR6–6425–F...

  • Page 157: Switch Chassis Ids

    4-11 4-12 A.1.2 Switch Chassis IDs For 24-port switch chassis, the switch chassis ID is on a sticker on the right side of the switch chassis. For 144-port switch chassis, the switch chassis ID is on a HR6–6425–F Cray Private...
  • Page 158 Cray XD1™ RapidArray Interconnect Topologies sticker on the front of the switch chassis. On these stickers, the chassis ID is the value shown for SysGuid. Use Table 53, page 140 to record switch chassis ID information. Table 53. Switch Chassis IDs...

  • Page 159: Glossary

    Also known as "all-to-all mesh." A switchless multichassis topology in which every Cray XD1 chassis in the system connects to every other chassis. A maximum of 13 chassis can be directly connected in a Cray XD1 system. See also fat tree.
  • Page 160 The fabric expansion card connects to the main board. fascia One of two perforated covers on the front of a Cray XD1 chassis. Both are hinged and can be flipped down to provide access to the fans. The right-side fascia also holds the LCD.
  • Page 161 See RapidArray link. logical topology The set of all fabric paths in a physical topology. It enables any node in a Cray XD1 system to communicate with any other node. See also accepted topology, expected topology, and present topology.
  • Page 162 RapidArray switch A full-crossbar nonblocking switch in the RapidArray fabric. The base configuration includes one 24-port RapidArray switch in each Cray XD1 chassis. The optional fabric expansion card adds a second RapidArray switch. Equivalent external RapidArray switches are available for implementing fat tree (switched) topologies.
  • Page 163 12 8 to 13 chassis (Set A), 35 five chassis, 21 four chassis, 18 equipment seven chassis, 30 required for Cray XD1 systems six chassis, 25 direct-connect topology, 12 three chassis, 14 fat tree topology, 54 two chassis, 13 fat tree topology HR6–6425–F...
  • Page 164 Cray XD1™ RapidArray Interconnect Topologies 12 chassis, 57 related publications, 2 24 chassis, 68 36 chassis, 84 48 chassis, 107 switch chassis ports IDs, 139 connecting numbering, 8 RapidArray, 5 publications related, 2 topologies direct-connect, described, 11 fat tree, described, 51...

Table of Contents