1. Manuals
  2. Brands
  3. Oracle Manuals
  4. Switch
  5. Sun Datacenter InfiniBand Switch 648
  6. Installation manual

Oracle Sun Datacenter InfiniBand Switch 648 Installation Manual

Hide thumbs Also See for Sun Datacenter InfiniBand Switch 648:
Table of Contents

Advertisement

Quick Links

Download this manual
Sun Datacenter InfiniBand
Switch 648
Installation Guide
Part No. 820-7738-12
May 2010, Revision A

Advertisement

Table of Contents
loading

Related Manuals for Oracle Sun Datacenter InfiniBand Switch 648

Summary of Contents for Oracle Sun Datacenter InfiniBand Switch 648

  • Page 1 Sun Datacenter InfiniBand Switch 648 Installation Guide Part No. 820-7738-12 May 2010, Revision A...
  • Page 2 Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.

  • Page 3: Table Of Contents

    Contents Using This Documentation ix Preparing the Site 1 Understanding Switch Specifications 1 Physical Specifications 2 Environmental Requirements 3 Acoustic Noise Emissions 3 Electrical Specifications 3 Network Management Connector and Pins 4 Serial Management Connector and Pins 5 CXP Connector and Pins 5 Routing Service Cables 7 Power Cord Requirements 8 Management Cable Requirements 9...
  • Page 4 Inspecting the Midplane for Installation 45 ▼ Identify the Midplane Fabric Card Connectors 46 ▼ Inspect the Pins on the Rear Side of the Midplane 47 ▼ Identify the Midplane Line Card Connectors 48 Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 5 ▼ Inspect the Pins on the Front Side of the Midplane 50 Inspecting the Fabric Cards for Installation 51 ▼ Identify the Fabric Cards for Installation 52 ▼ Inspect the Fabric Card Chassis for Installation 53 ▼ Inspect the Fabric Card Retainer Bolts for Installation 53 ▼...
  • Page 6 Display the Line Card FRU ID Information 107 Connecting InfiniBand Cables 108 ▼ Attach Cables to the CXP Connectors 109 ▼ Check the CXP Connector Status LEDs 115 ▼ Check Link Status 116 Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 7 ▼ Identify the CXP Connector Addresses 117 ▼ Display the InfiniBand Cable FRU ID Information 118 Verifying the InfiniBand Fabric 119 ▼ Discover the InfiniBand Fabric Topology 120 ▼ Perform Diagnostics on the InfiniBand Fabric 121 ▼ Validate the InfiniBand Fabric and Report Errors 123 Index 125 Contents...
  • Page 8 Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 9: Using This Documentation

    Using This Documentation This installation guide provides detailed procedures that describe the preparation ® and installation of the Sun Datacenter InfiniBand Switch 648 from Oracle . This document is written for technicians, system administrators, authorized service providers, and users who have advanced experience installing computing hardware.
  • Page 10 Submit comments about this document by clicking the Feedback[+] link at (http://docs.sun.com). Include the title and part number of your document with your feedback: Sun Datacenter InfiniBand Switch 648 Installation Guide, part number 820-7738-12. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 11: Preparing The Site

    Preparing the Site Preparing your site for the switch is essential for a proper installation and operation. These topics describe information you should know before you install the switch. “Understanding Switch Specifications” on page 1 ■ “Routing Service Cables” on page 7 ■...

  • Page 12: Physical Specifications

    19.25 in. (489 mm) Weight 410.0 lbs (186 kg) Related Information “Remove the Shipping Carton Contents” on page 25 ■ “Install the Chassis in the Rack” on page 39 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 13: Environmental Requirements

    Environmental Requirements Parameter Operating Ambient temperature 41˚F to 95˚F (5˚C to 35˚C) Relative humidity 5% to 90% noncondensing, 80˚F (27˚C) maximum wet bulb Elevation (Sun requirement) Maximum 9840 feet (3000 meters) at 104˚F (40˚C) Acoustic Noise Emissions Parameter Operating Idling Acoustic power LWAd (1B=10dB) 7.1 B 7.2 B...

  • Page 14: Network Management Connector And Pins

    Not used Related Information “Management Cable Requirements” on page 9 ■ “Attach the Management Cables” on page 76 ■ “Access the CMC From the Network Management Port” on page 84 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 15: Serial Management Connector And Pins

    Serial Management Connector and Pins The following table lists the pinout of the serial management connector. Signal Related Information “Management Cable Requirements” on page 9 ■ “Attach the Management Cables” on page 76 ■ “Access the CMC From the Serial Management Port” on page 82 ■...
  • Page 16 RX7p RX6p TX4n TX5n RX7n RX6n TX6p TX7p RX5p RX4p TX6n TX7n RX5n RX4n TX8p TX9p RX11p RX10p TX8n TX9n RX11n RX10n TX10p TX11p RX9p RX8p TX10n TX11n RX9n RX8n Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 17: Routing Service Cables

    Signal Signal Signal Signal VCC3.3-Tx PRSNT VCC3.3-Rx VCC12-Tx Int_L/Reset VCC12-Rx The following table provides descriptions of the CXP signals. Signal Description Ground for both signal and power return VCC3.3-Tx 3.3V standby power from the power supplies VCC3.3-Rx 3.3V standby power return VCC12-Tx 12V power for cable export and future enhancements VCC12-Rx...

  • Page 18: Power Cord Requirements

    Caution – Only install and route power cabling in a matter that complies to federal, state, and local electrical codes. Related Information “Electrical Specifications” on page 3 ■ “Attach the Power Cords” on page 78 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 19: Management Cable Requirements

    Management Cable Requirements Management of the switch is done at the management console, which is either a serial device attached to the SER MGT port of the CMC or a 10/100 Ethernet connection at the NET MGT port of the CMC. Typically, the SER MGT connection to the serial device is only used for the initial startup and configuration of the switch.

  • Page 20: Infiniband Cable Cautions

    Do not twist the cable to open a Do not pack the cable to fit a tight kink. If it is not severe, open the space. Use an alternative cable kink by unlooping the cable. route. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 21: Infiniband Cable Guidelines

    Do not hang the cable for a length Do not straighten the cable to more than 2 meters (7 feet). correct a bend that is too tight. Minimize the hanging weight Leave the cable bend as is. with intermediate retention points.

  • Page 22: Infiniband Cable Types

    Bends in the cable path increases the required length of the cable. Rarely does a ■ cable travel in a straight line from connector to connector. Bends in the cable path are necessary, and each bend increases the total length. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 23: Infiniband Cable Bundling

    Bundling increases the required length of the cables. Bundling causes one or more ■ cables to follow a common path. However, the bend radius is different in different parts of the bundle. If the bundle is large and unorganized, and there are many bends, one cable might experience only the inner radius of bends, while another cable might experience the outer radius of bends.

  • Page 24: Floor And Underfloor Delivery Of Infiniband Cables

    InfiniBand cable to the weight of the cable for the distance between the supports and the switch. Related Information “InfiniBand Cable Guidelines” on page 11 ■ “Floor and Underfloor Delivery of InfiniBand Cables” on page 14 ■ “Connecting InfiniBand Cables” on page 108 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 25: Installing The Switch

    Installing the Switch The following topics describe the installation of the switch. The topics are listed in the order of completion. “Understanding the Installation” on page 15 ■ “Installing the InfiniBand Software Stack” on page 23 ■ “Unpacking the Chassis” on page 25 ■...

  • Page 26: Supported Configurations

    2 CMCs ■ The fabric card and line card distribution determine how you configure a switch that is not fully populated. Related Information “Installing the Components” on page 63 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 27: Fabric Card Distribution

    Fabric Card Distribution In a less than fully populated configuration, you must install fabric cards in a particular order, to maintain thermal management. The following sequence is the placement order for fabric cards: 0, 1, 8, 7, 6, 2, 3, 5, 4 For example, if you are installing 5 fabric cards, install them in slots 0, 1, 8, 7, and 6.

  • Page 28: Line Card Handling Guidelines

    Line card fillers are empty shells without active components. Holes in the face of a filler permit restrictive air flow. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 29: Installation Preparation

    Fabric card fillers have IPMB controllers and fans, but no switch chips. This design enables the filler to assist with removing heat from the chassis. The fabric card filler is enabled and activated like a fabric card. Related Information “Install the Fabric Cards or Fabric Card Fillers” on page 67 ■...

  • Page 30: Antistatic Precautions For Installation

    “Install the HPC Software Suite” on page 23 “Remove the Shipping Carton Contents” on page 25 “Unsecure the Chassis From the Pallet” on page 28 “Install the Rail Assemblies in the Rack” on page 29 Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 31 Step Link “Install the Cable Trees” on page 32 “Install the Cable Curves” on page 38 “Install the Chassis in the Rack” on page 39 “Identify the Midplane Fabric Card Connectors” on page 46 “Inspect the Pins on the Rear Side of the Midplane” on page 47 “Identify the Midplane Line Card Connectors”...
  • Page 32 “Display the InfiniBand Cable FRU ID Information” on page 118 “Discover the InfiniBand Fabric Topology” on page 120 “Perform Diagnostics on the InfiniBand Fabric” on page 121 “Validate the InfiniBand Fabric and Report Errors” on page 123 Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 33: Installing The Infiniband Software Stack

    Installing the InfiniBand Software Stack The OFED InfiniBand software stack is available in the HPC software suite, created for Red Hat Enterprise Linux 5.2 systems. The HPC software is delivered as an .iso image. You can burn this image to a CD or DVD, or extract the files from within the image.
  • Page 34 During startup, the InfiniBand fabric becomes active. 14. Begin the installation of the switch. “Remove the Shipping Carton Contents” on page Related Information “Create the guid.txt File for Installation” on page 93 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 35: Unpacking The Chassis

    Unpacking the Chassis The first part of installing the switch chassis is to unpack it from its shipping container. Perform the tasks in the following order: “Remove the Shipping Carton Contents” on page 25 ■ “Unsecure the Chassis From the Pallet” on page 28 ■...
  • Page 36 6. Remove the outer carton. 7. Remove and unpack the shipping kit. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 37 8. Remove the internal saddle. 9. Lift off the switch bag. 10. Make a visual check to ensure that the switch chassis is undamaged. 11. Move the switch to the operating location using a computer lifting device. 12. Unsecure the switch chassis from the shipping pallet. “Unsecure the Chassis From the Pallet”...

  • Page 38: Unsecure The Chassis From The Pallet

    4. Prepare the switch chassis for component installation. “Install the Rail Assemblies in the Rack” on page Related Information “Suggested Tools for Installation” on page 19 ■ “Remove the Shipping Carton Contents” on page 25 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 39: Preparing The Chassis

    Preparing the Chassis After unpacking the switch chassis and components, mount the switch chassis and cable management hardware into the rack. Perform the tasks in the following order: Note – The illustrations in this topic describe the switch being installed at the middle position for better clarity.
  • Page 40 Caution – Do not install any rail-mounted equipment in the Sun Datacenter InfiniBand Switch 648 rack. The Sun Datacenter InfiniBand Switch 648 switches must be loaded into the rack from the bottom up. 1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.
  • Page 41 6. Insert the captive nuts for the rear bracket into cabinet holes t, u, and v. where t, u, and v are the respective cabinet holes: 2, 5, and 6 for bottom mounting ■ 36, 39, and 40 for middle mounting ■...

  • Page 42: Install The Cable Trees

    The outer branch installs onto the main trunk with the reinforcing ribs facing to ■ the rear of the rack. The center reinforcing rib has 4 openings above and 5 openings below. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 43 Note – For this procedure, the term cabinet holes refers to the holes in the posts of the cabinet. For the Sun Rack II cabinet, these holes are square. The holes are numbered from 1 starting at the very bottom. Hole 2, 3, 4, and so on, are the second, third, fourth, and so on, hole from the bottom.
  • Page 44 Using a No. 2 Phillips screwdriver, securely tighten all three screws. 5. Set the assembled cable tree at the left front corner of the rack. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 45 6. Insert the seven captive nuts for the left main trunk into cabinet holes a, b, c, d, e, f, and g. where a, b, c, d, e, f, and g are the respective cabinet holes: 6, 8, 13, 19, 24, 30, and 33 for bottom mounting ■...
  • Page 46 8. While holding the cable tree in position, have a helper install the two M6 mounting screws. a. Install the first screw for the top hole of the main trunk. Finger tighten the screw only. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 47 b. Install the second screw into the bottom hole of the main trunk. Finger tighten the screw only. 9. Press the cable tree outward from the rack centerline and using a No. 2 Phillips screwdriver, firmly tighten the top and bottom screws. 10.

  • Page 48: Install The Cable Curves

    3. Using a No. 2 Phillips screwdriver, attach the smaller of the cable curves to the bottom position of the right outer branch with two M4 screws. The cable curve points down and to the rear of the rack. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 49: Install The Chassis In The Rack

    4. Repeat Step 3 with the larger cable curve, attaching it to the middle position of the outer branch with two M4 screws. 5. Attach the hook-and-loop fastener straps to the ends of the cable curves. 6. Repeat Step 3 through Step 5 for the left outer branch.
  • Page 50 Caution – Do not install any rail-mounted equipment in the Sun Datacenter InfiniBand Switch 648 rack. Oracle’s Sun Datacenter InfiniBand Switch 648 switches must be loaded into the rack from the bottom up.
  • Page 51 Note – Lift the switch only high enough to remove the pallet. 5. Slowly move the switch to the rack. 6. Set the lift height so that the chassis is equal with the rails on the rack. 7. Carefully move the lifting device forward as possible, until the switch chassis is able to slide safely on the rails.
  • Page 52 8. Slide the switch into the rack and against the cable trees. 9. Pull the lift back from the rack so that it clears the cable trees. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 53 10. Lower the shipping cradle and move the lift out of the way. 11. Using a No. 2 Phillips screwdriver, install the 10 larger M6 screws on the front of the switch chassis to secure the switch to the rack posts. Note –...
  • Page 54 15. Inspect the switch components before installing them. “Inspecting the Components” on page Related Information “Physical Specifications” on page 2 ■ “Suggested Tools for Installation” on page 19 ■ “Install the Cable Curves” on page 38 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 55: Inspecting The Components

    Inspecting the Components After mounting the switch chassis into the rack, check the condition and suitability of the components for installation. Perform the tasks in the following order: Step Description Links Inspect the midplane. “Inspecting the Midplane for Installation” on page 45 Inspect the fabric cards.

  • Page 56: Identify The Midplane Fabric Card Connectors

    FC – Fabric card ■ slot is the fabric card slot, 0 through 8 (left to right) ■ position is the connector position, 0 through 8 (bottom to top) ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 57: Inspect The Pins On The Rear Side Of The Midplane

    4. Find the origin of the coordinate system. The origin of this coordinate system is the lower-left corner, at FC00. In the illustration, the connector identified is FC74. 5. Inspect the pins on the rear side of the midplane. “Inspect the Pins on the Rear Side of the Midplane” on page Related Information “Identify the Midplane Line Card Connectors”...

  • Page 58: Identify The Midplane Line Card Connectors

    “Installation Sequence” on page 2. Observe the front of the midplane. The front side of the midplane has an 9x9 matrix of XBOW connectors that interface with the line cards. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 59 3. Identify the connectur numbers. The unique connector number is located above each connector. For example, the lower-left connector number is LC08. The upper-right connector number is LC80. The connector number is decoded as LCslotposition. Where: LC – Line card ■...

  • Page 60: Inspect The Pins On The Front Side Of The Midplane

    5. Look for any contaminants that might be present. These contaminants might appear as light dots near the pins. 6. Move your head right to the next connector and inspect its pins, repeating Step 2 through Step Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 61: Inspecting The Fabric Cards For Installation

    7. Continue in this manner for all connectors in the row. 8. Repeat Step 2 Step 7 for line card slot 7 to line card slot 0. 9. Inspect the fabric cards. “Identify the Fabric Cards for Installation” on page Related Information “Inspecting the Line Cards for Installation”...

  • Page 62: Identify The Fabric Cards For Installation

    Fabric Card Features FIGURE: Figure Legend Upper retainer bolt Status LEDs Fans Lower retainer bolt XBOW connectors 3. Inspect the fabric cards. “Inspect the Fabric Card Chassis for Installation” on page Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 63: Inspect The Fabric Card Chassis For Installation

    Related Information “Identify the Line Cards for Installation” on page 56 ■ “Identify the InfiniBand Cable for Installation” on page 61 ■ ▼ Inspect the Fabric Card Chassis for Installation 1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.

  • Page 64: Inspect The Fans For Installation

    2. Place the fabric card on a work surface with the XBOW connectors facing you. 3. Using the flashlight positioned to the side and above you, look at the holes of the left-end connector. Look at the holes straight on. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 65: Inspecting The Line Cards For Installation

    Note – Do not look at the individual holes, rather look at all of the connector’s holes as a group. A damaged or contaminated hole will be apparent. 4. Look for any closed-over or contaminated holes. A closed-over or contaminated hole appears brighter than the surrounding holes. 5.

  • Page 66: Identify The Line Cards For Installation

    See the table in “Installation Sequence” on page 2. Use the following illustration to identify the various features of a line card. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 67 Line Card Features FIGURE: Figure Legend Left retainer bolt CXP connectors Status LEDs Right retainer bolt XBOW connectors 3. Inspect the line cards. “Inspect the Line Card Chassis for Installation” on page Related Information “Identify the Fabric Cards for Installation” on page 52 ■...

  • Page 68: Inspect The Line Card Chassis For Installation

    3. Inspect the line card XBOW connectors. “Inspect the Line Card XBOW Connectors for Installation” on page Related Information “Inspect the Fabric Card Retainer Bolts for Installation” on page 53 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 69: Inspect The Line Card Xbow Connectors For Installation

    ▼ Inspect the Line Card XBOW Connectors for Installation The XBOW connectors on line cards are the receptacles for the midplane connector pins. These connectors are checked as meticulously as the midplane connector pins. You must verify that no connectors are damaged and no holes are blocked. You need the following tools: Flashlight ■...

  • Page 70: Inspect The Line Card Cxp Connectors For Installation

    Sun Service representative. 3. Inspect the InfiniBand cables. “Inspecting the InfiniBand Cables for Installation” on page Related Information “Inspect the InfiniBand Cable Connectors for Installation” on page 63 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 71: Inspecting The Infiniband Cables For Installation

    Inspecting the InfiniBand Cables for Installation Before installing the InfiniBand cables, verify that they are suitable for installation. Perform the tasks in the following order: Step Description Links Identify the InfiniBand cables. “Identify the InfiniBand Cable for Installation” on page 61 Inspect the InfiniBand cable “Inspect the InfiniBand Cable Hardware for hardware.

  • Page 72: Inspect The Infiniband Cable Hardware For Installation

    3. Verify that the cable is not kinked or has a fold. 4. Verify that the cable is of the correct type from its label. 5. Inspect the cable connectors. “Inspect the InfiniBand Cable Connectors for Installation” on page Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 73: Inspect The Infiniband Cable Connectors For Installation

    Related Information “Inspect the Fabric Card Chassis for Installation” on page 53 ■ “Inspect the Line Card Chassis for Installation” on page 58 ■ ▼ Inspect the InfiniBand Cable Connectors for Installation 1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.

  • Page 74: Install The Midplane Stiffener

    2. Lift the midplane stiffener from the work surface and insert the stiffener into the chassis center at slot 4. Ensure that the stiffener is perpendicular to the surface of the midplane. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 75 3. Continue to slide the midplane stiffener in until it stops. 4. While pressing the midplane stiffener in with one hand, rotate the retainer bolts clockwise by hand so that they engage. When you feel resistance in the retainer bolts, stop. 5.
  • Page 76 Turning more than 1/4 turn damages the midplane stiffener and midplane XBOW connectors. Step 5 Step 6 7. Repeat until you feel resistance to turning. 8. Verify that the end of the midplane stiffener properly contacts the midplane crossbar. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 77: Install The Fabric Cards Or Fabric Card Fillers

    9. Give each retainer bolt a final torque of 2.0 ft-lb (0.27 kg-m) so that it is snug. 10. Install the fabric cards. “Install the Fabric Cards or Fabric Card Fillers” on page Related Information “Remove the Midplane Stiffener” on page 70 ■...
  • Page 78 Note – Once the fabric card is about halfway into the slot, it is no longer necessary to bear the weight of the fabric card. 4. Continue to slide the fabric card in, maintaining a vertical orientation. When you feel resistance, stop. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 79 5. While pressing the fabric card in with one hand, rotate the retainer bolts clockwise by hand so that they engage. When you feel resistance in the retainer bolts, stop. 6. Using a 10 mm socket and ratchet, turn the lower retainer bolt 1/4 turn clockwise.

  • Page 80: Remove The Midplane Stiffener

    See the table in “Installation Sequence” on page 2. Using a 20 mm socket and ratchet, turn the left retainer bolt 1/4 turn counter-clockwise. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 81 3. Turn the right retainer bolt 1/4 turn counter-clockwise. Caution – Do not turn the retainer bolts more than 1/4 revolution each time. Turning more than 1/4 turn damages the midplane stiffener and midplane XBOW connectors. Step 2 Step 3 4.

  • Page 82: Install The Line Cards Or Line Card Fillers

    1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch. See the table in “Installation Sequence” on page Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 83 2. Lift the line card from the work surface and orient it horizontally, with the link LEDs across the bottom. 3. Insert the line card into the chassis at slot 0. Ensure that the line card is square to the chassis and not skewed. Note –...
  • Page 84 Install the cards into the following slots in the following sequence: 0, 1, 2, 3, 4, 5, 6, 7, 8 15. Install the line card fillers into the vacant slots. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 85: Powering On The Cmcs

    a. Grasp the handles at each end of the line card filler and lift the filler to the vacant slot. b. Slide the filler into the chassis until it stops. c. While pressing the filler in with one hand, rotate the retainer bolts clockwise by hand so that they engage.

  • Page 86: Attach The Management Cables

    See the table in “Installation Sequence” on page 2. Connect the serial management cables from the management console to the connectors labeled SER MGT. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 87 3. (Optional) Connect the network management cables from the management console to the connectors labeled NET MGT. 4. Ready the management console for communication with the CMCs. Installing the Switch...

  • Page 88: Attach The Power Cords

    “Installation Sequence” on page 2. Ensure that the circuit breakers for the power supplies are switched off. 3. Plug a power cord into each power receptacle at the rear of the chassis. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 89 4. Route each power cord to its respective facility power receptacle. Use cable ties or hook and loop fastener straps to bundle and secure the cord. 5. Plug each power cord into the receptacle. 6. Energize the circuit breakers so that the power receptacles are live. 7.

  • Page 90: Check The Power Supply Status Leds

    1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch. See the table in “Installation Sequence” on page 2. Visually inspect the CMC status LEDs. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 91: Accessing The Cmc

    3. Compare what you see to the following table: Glyph Name Color State and Meaning Green On – CMC is operating normally. Off – No power is being supplied to the CMC. Flashing – CMC is inactive status as the backup CMC. Attention Amber On –...

  • Page 92: Access The Cmc From The Serial Management Port

    If you do not see the text, still progress to Step 4 4. Type root for the login name followed by the root password. shmm1500 login: root Password: password The # prompt is displayed. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 93: Set The Initial Uboot Parameters

    Note – As shipped, the root password is not set and therefore not required. Press the Enter or Return key when prompted for the password. Refer to the Sun Datacenter InfiniBand Switch 648 Administration Guide for instructions on how to set the root password.

  • Page 94: Access The Cmc From The Network Management Port

    See the table in “Installation Sequence” on page 2. Set the IP address for the CMC: # clia setlanconfig 1 3 IP_address where IP_address is in the form of xxx.xxx.xxx.xxx. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 95 3. Set the netmask for the CMC: # clia setlanconfig 1 6 netmask where netmask is the netmask. Typically, the netmask is 255.255.255.0. However, your network environment subnet might require a different netmask. Use a netmask number most appropriate to your environment. 4.

  • Page 96: Verify The Cmc Status

    I2C test of AT64C24 at bus 2 addr 57 -PASSED M9 CMC I2C access test PASSED If any test failed, use the switchover command to activate the other CMC and cable it for the rest of the installation process. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 97: Identify The Cmc Addresses

    4. Verify that all installed components are recognized as installed: # showpresent If an installed component is not recognized, investigate why. 5. Consider your next step: (Optional) Display and record the CMC FRU ID information. ■ “Identify the CMC Addresses” on page Verify the power supply status.

  • Page 98: Display The Cmc Fru Id Information

    = Sun Datacenter Switch DCS 648 - CMC board Product Part / Model# = 541-3807-02 Product Version = Rev 50 Product Serial Number = 0110SJC-0905CM0003 Asset Tag = none FRU Programmer File ID = M9_CMC_FRU_541-3807-02_0110SJC-0911CM0001.txt Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 99: Verify The Power Supply Status

    Multi Record Area: PICMG Board Point-to-Point Connectivity Record (ID=0x14) Version = 0 Record Type = 0xc0 OEM Defined Record Version = 2 PPS IPMB Topology Record (ID=0x05) Version = 0 3. Repeat Step 2 for the other CMC. 4. Verify the power supply status. “Verify the Power Supply Status”...

  • Page 100: Identify The Power Supply Addresses

    “Identify the CMC Addresses” on page 87 ■ “Identify the Fabric Card Addresses” on page 99 ■ “Identify the Line Card Addresses” on page 106 ■ “Identify the CXP Connector Addresses” on page 117 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 101: Display The Power Supply Fru Id Information

    ▼ Display the Power Supply FRU ID Information 1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch. See the table in “Installation Sequence” on page 2. On the CMC, type: # clia fruinfo 20 FRU_number where FRU_number is from “Identify the Power Supply Addresses”...

  • Page 102: Powering On The Fabric Cards

    Information” on page 100 Related Information “Understanding the Installation” on page 15 ■ “Powering On the CMCs” on page 75 ■ “Powering On the Line Cards” on page 101 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 103: Enable The Fabric Cards For Installation

    ▼ Enable the Fabric Cards for Installation 1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch. See the table in “Installation Sequence” on page 2. Enable standby power and the IPMB buses for the fabric cards on the CMC: # enablestby fc slot where slot is the slot number of the installed fabric cards (0–8).
  • Page 104 0xb2 ■ Note – For GUIDs that end with two zeros (00), you are effectively replacing the zeros with a2 and b2. For example, 0x00bad0fc0009f000 becomes: ■ 0x00bad0fc0009f0a2 ■ 0x00bad0fc0009f0b2 Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 105 6. Repopulate your list with the modified GUIDs. For example: 0x00bad0fc0009f0a2 0x00bad0fc0009f0b2 0x00bad0fc0033f1a2 0x00bad0fc0033f1b2 0x00bad00017fcf2a2 0x00bad00017fcf2b2 0x00bad00005fcf3a2 0x00bad00005fcf3b2 0x00bad00003fcf4a2 0x00bad00003fcf4b2 0x00bad00007fcf5a2 0x00bad00007fcf5b2 0x00bad0fc0011f6a2 0x00bad0fc0011f6b2 0x00bad0b00002f7a2 0x00bad0b00002f7b2 0x00bad0b00001f8a2 0x00bad0b00001f8b2 7. Save the list on the Linux InfiniBand host as the /etc/opensm/guid.txt file. 8.

  • Page 106: Power On The Fabric Cards For Installation

    1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch. See the table in “Installation Sequence” on page 2. Visually inspect the fabric card status LEDs. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 107: Verify The Fabric Card Status

    3. Compare what you see to the following table. Glyph Location Name Color State and Meaning Left Green On – Normal operation at full power. Ready-to- Off – No power at all applied. Remove Flashing – Fabric card has been disabled and is ready to be removed, standby voltage available.
  • Page 108 I4 FW image version : 7.2.00 8. Repeat Step 7 and record the information for each fabric card and compare to identify any discrepancies. All fabric cards should have the same firmware versions. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 109: Identify The Fabric Card Addresses

    9. Consider your next step: (Optional) Display and record the fabric cards FRU ID information. ■ “Identify the Fabric Card Addresses” on page Power on the line cards. ■ “Enable the Line Cards for Installation” on page 102. Related Information Switch Command Reference, showpresent command ■...

  • Page 110: Display The Fabric Card Fru Id Information

    = Sun Microsystems, Inc. Board Product Name = Sun Datacenter Switch DCS 648 - Fabric Card Board Serial Number = 0110SJC-0918FC0231 Board Part Number = 541-3811-04 FRU Programmer File ID = m9_fc_fru-info-0110SJC-0918FC0231.inf Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 111: Powering On The Line Cards

    Product Info Area: Version Language Code = 25 Manufacturer Name = Sun Microsystems, Inc. Product Name = Sun Datacenter Switch DCS 648 - Fabric Card Product Part / Model# = 541-3811-04 Product Version = Rev 1 Product Serial Number = 0110SJC-0918FC0231 Asset Tag = none FRU Programmer File ID...

  • Page 112: Enable The Line Cards For Installation

    5. Power on the line cards. “Power On the Line Cards for Installation” on page 103. Related Information Switch Command Reference, enablestby command ■ “Enable the Fabric Cards for Installation” on page 93 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 113: Power On The Line Cards For Installation

    ▼ Power On the Line Cards for Installation Caution – You must fully power on all fabric cards and fabric card fillers before powering on the line cards. See “Power On the Fabric Cards for Installation” on page 1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.

  • Page 114: Verify The Line Card Status

    “Installation Sequence” on page 2. On the CMC, verify that the installed line cards are active: # showpresent | grep LC If an installed line card is shown to be INACTIVE, investigate why. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 115 3. Check for line card power faults: # checkpwrfault | grep LC If an installed line card is shown to FAIL, investigate why. 4. Check line card internal voltages: # checkvoltages If an installed line card is shown to FAIL, investigate why. 5.

  • Page 116: Identify The Line Card Addresses

    3. Display and record the line cards FRU ID information. “Display the Line Card FRU ID Information” on page 107. Related Information Switch Administration, component addressing ■ “Identify the Power Supply Addresses” on page 90 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 117: Display The Line Card Fru Id Information

    “Identify the CMC Addresses” on page 87 ■ “Identify the Fabric Card Addresses” on page 99 ■ “Identify the CXP Connector Addresses” on page 117 ■ ▼ Display the Line Card FRU ID Information 1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.

  • Page 118: Connecting Infiniband Cables

    “Display the InfiniBand Cable FRU ID information. Information” on page 118 Related Information “Understanding InfiniBand Cabling” on page 9 ■ “Understanding the Installation” on page 15 ■ “Verifying the InfiniBand Fabric” on page 119 ■ Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 119: Attach Cables To The Cxp Connectors

    ▼ Attach Cables to the CXP Connectors Note – This procedure describes cabling the switch in a center split, left-right symmetrical configuration. 1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch. See the table in “Installation Sequence”...
  • Page 120 4. Plug the first cable into the CXP connector at 11B. Note – For each connector group, A is the upper connector, B is the lower connector. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 121 a. Remove the protective cap from the connector and visually inspect the cable connector. The shell should not be bent and should be parallel to the inner boards. If the connector is bent or damaged, use a different cable. b. Ensure that the retraction strap is forward. c.
  • Page 122 Continue to push the connector in until the hooks catch onto the top of the receptacle. 5. Curve the cable to the right through the branches of the cable tree. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...
  • Page 123 Caution – Do not allow any copper core InfiniBand cable to bend tighter than a 5 inch (127 mm) radius. Do not allow any optical InfiniBand cable to bend tighter than a 3.4 inch (85 mm) radius. Tight bends can damage the cable internally. 6.
  • Page 124 Step 4 for the fourth cable at CXP connector 1B. 14. Curve the cable to the left through the branches of the cable tree, next to the cable already there. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 125: Check The Cxp Connector Status Leds

    15. Place the cable over the cable curve and move the cable against the cable already there. Step 4 16. Repeat for cables connecting to CXP connectors 9B, 2B, and so on, to and including 6B and 5B. Place the cables next to the cables that are already there. 17.

  • Page 126: Check Link Status

    -e option of the checklinks command to attempt to enable a downed link. Note – The cable connection of the line card where the link is down is identified. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 127: Identify The Cxp Connector Addresses

    3. Consider your next step: (Optional) Display and record the InfiniBand cable FRU ID information. ■ “Identify the CXP Connector Addresses” on page 117. Interface with the InfiniBand software stack. ■ “Discover the InfiniBand Fabric Topology” on page 120. Related Information Switch Administration, checking status LEDs ■...

  • Page 128: Display The Infiniband Cable Fru Id Information

    Common Header: Format Version = 1 Board Info Area: Version Language Code = 25 Mfg Date/Time = Mar 13 10:34:00 2009 (6941434 minutes since 1996) Board Manufacturer Board Product Name Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 129: Verifying The Infiniband Fabric

    Board Serial Number Board Part Number FRU Programmer File ID Product Info Area: Version Language Code = 25 Manufacturer Name = MergeOptics GmbH Product Name = 10m, 84 Circuit iPass Stacked Active Optical Cable Product Part / Model# Product Version = Rev.

  • Page 130: Discover The Infiniband Fabric Topology

    "S-00bad0fc0100f8b2"[36] # "Sun DCS 648 shmm1500 FC slot 8 switch 1" lid 57 4xQDR "S-0021283a77bdf7a2"[19] # "Sun DCS 648 shmm1500 FC slot 7 switch 0" lid 54 4xQDR vendid=0x2c9 devid=0x673c sysimgguid=0x3ba000100c70b Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 131: Perform Diagnostics On The Infiniband Fabric

    caguid=0x3ba000100c708 2 "H-0003ba000100c708" # "nsn34-39 HCA-1" [2](3ba000100c70a) "S-00bad0cc010016a2"[22] # lid 58 lmc 0 "Sun DCS 648 shmm1500 LC slot 6 switch 0" lid 1 4xQDR Note – The output in the example is just a very small portion of the full output. 3.
  • Page 132 Partitions Check IPoIB Subnets Check Please see /tmp/ibdiagnet.log for complete log ---------------------------------------------------------------- -I- Done. Run time was 42 seconds. Note – The Performance Manager counters are available in the ibdiagnet.pm file. Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 133: Validate The Infiniband Fabric And Report Errors

    3. Validate and check errors for the InfiniBand fabric. “Validate the InfiniBand Fabric and Report Errors” on page 123. Related Information Switch Command Reference, ibdiagnet command ■ “Discover the InfiniBand Fabric Topology” on page 120 ■ “Installing the InfiniBand Software Stack” on page 23 ■...
  • Page 134 Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

  • Page 135: Index

    Index network management, 84 serial management, 82 accessing addressing, 87 CMC, 81 FRU ID, 88 network management, 84 LEDs, 80 serial management, 82 powering on, 75 acoustic noise emissions, 3 status, 86 activate command, 96, 103 command addressing activate, 96, 103 CMC, 87 boot, 83 fabric card, 99...
  • Page 136 CXP, 5 midplane inspection, 47 InfiniBand cable, 5 powering on, 92, 96 network management, 4 status, 97 serial management, 5 fabric card filler creating addressing, 99 guid.txt file, 93 enabling, 93 features, 52 CXP connector, 5 FRU ID, 100 handling guidelines, 17 inspecting, 51 diagnosing chassis, 53...
  • Page 137 ibnetdiscover command, 120 midplane, 45 InfiniBand cable installation addressing, 117 preparation, 19 attaching, 109 responsibilities, 20 bundling, 13 sequence, 20 cautions, 10 understanding, 15 connector, 5 installing delivery cable curve, 38 floor and underfloor, 14 cable tree, 32 overhead, 14 chassis, 39 features, 61 components, 63...
  • Page 138 reboot command, 83 removing management cables midplane stiffener, 70 attaching, 76 shipping carton contents, 25 requirements, 9 root node GUID file, 93 midplane connectors routing fabric card, 46 InfiniBand cable, 9 line card, 48 service cables, 7 inspecting, 45 fabric card, 47 line card, 50 saveenv command, 83 stiffener...
  • Page 139 InfiniBand cable, 9 installation, 15 switch specifications, 1 unpacking the chassis, 25 unsecuring the chassis, 28 validating the InfiniBand fabric, 123 verifying the InfiniBand fabric, 119 Index...
  • Page 140 Sun Datacenter InfiniBand Switch 648 Installation Guide • May 2010...

Table of Contents