HP Cluster Platform Interconnects v2010 Manual

Myrinet system interconnect guide.
Hide thumbs
HP Cluster Platform
Myrinet System Interconnect Guide
Part Number: A-CPMYI-1A
May 2005
Revision/Update Information:
Version 1.0
Product Version:
HP Clusters
This manual describes the Myricom Myrinet system interconnects used in HP Cluster Platform
solutions.
Hewlett-Packard Company
Palo Alto, California

Advertising

   Related Manuals for HP Cluster Platform Interconnects v2010

   Summary of Contents for HP Cluster Platform Interconnects v2010

  • Page 1

    HP Cluster Platform Myrinet System Interconnect Guide Part Number: A-CPMYI-1A May 2005 Revision/Update Information: Version 1.0 Product Version: HP Clusters This manual describes the Myricom Myrinet system interconnects used in HP Cluster Platform solutions. Hewlett-Packard Company Palo Alto, California...

  • Page 2

    © Copyright 2005 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty.

  • Page 3: Table Of Contents

    Contents About This Manual 1 Myrinet interconnect overview Interconnect characteristics ..........Backplane and chassis types .

  • Page 4: Table Of Contents

    128-Port Myrinet System Interconnect ........Fully Populated 128-Port Myrinet System Interconnect .

  • Page 5: About This Manual, Additional Documentation

    About This Manual This manual describes the Myricom Myrinet system interconnects used in some HP Cluster Platform solutions. This manual does not describe the procedures and tools that are required to install and configure the system hardware or software. It does contain references for cluster components that have their own documentation.

  • Page 6: Symbols In Text

    Symbols in text zk-2071 WARNING: Text set off in this manner indicates that failure to follow directions in the warning could result in bodily harm or loss of life. CAUTION: Text set off in this manner indicates that failure to follow zk-2071 directions in the caution could result in damage to equipment or loss of information.

  • Page 7: Myrinet Interconnect Overview, Pci Cards

    Myrinet interconnect overview The high-speed Myrinet system interconnect is used in some HP Cluster Platform solutions. The chassis for the interconnect is 5U or 9U, depending on the cluster configuration. A PCI adapter card connects each application node in the cluster to the interconnect.

  • Page 8

    1.2 Backplane and chassis types The Myrinet interconnect used in an HP Cluster Platform can be a 5U (64-port) or 9U (128-port) chassis. The backplane (Figure 1-2) provides connectivity for the 8-port switch cards. Figure 1-2: Myrinet System Interconnect Backplane For configurations greater than 128 total nodes, you must build the network out to increase the number of available ports and to maintain the cross-sectional bandwidth.

  • Page 9

    The 128-port, node-level switch in a federated configuration provides 64 downlinks to nodes and dedicates the other 64 ports to uplinks that provide connectivity to deeper parts of the network. A top-level switch, another Myrinet system interconnect, consolidates the uplinks from two node-level switches. Each switch supports up to 128 links, 64 links from each node-level switch;...

  • Page 10: Port Myrinet System Interconnect, Chassis Types

    Figure 1-5: 128-Port Myrinet System Interconnect xc-013B-IG Table 1-1 explains the chassis types, and the types and location of line cards within the interconnects. Table 1-1: Chassis types Chassis Type Component Quantity Location Bounded chassis M3-E128 17 slot chassis Monitoring Line Card Slot M 8 Port Switch Line Card 1 -- 16...

  • Page 11: Fully Populated 128-port Myrinet System Interconnect

    Figure 1-6: Fully Populated 128-Port Myrinet System Interconnect Application nodes start here, move left, and then down to the next card. Switch cards are numbered 0-15, beginning with the If the control node is card below the connected to the monitoring line card (1).

  • Page 13: Pci Identification Label

    PCI cards Every node in a cluster that uses a Myricom system interconnect has a PCI host bus adapter (HBA), and each host interface has a connection to the interconnect. This chapter presents the following information relating to the three versions of the PCI card that might be used in an HP Cluster Platform solution: •...

  • Page 14: Single-port Xp Pci Rev. D Card

    2.2 Single-port XP PCI card The single-port XP PCI card (Figure 2-2) is the Myricom Rev. D card. It supports one X-port interface and one PCI-X interface. Figure 2-2: Single-port XP PCI Rev. D card PCI cards...

  • Page 15

    Table 2-1 shows the single-port XP PCI card’s technical specifications. Table 2-1: XP PCI Host Interface Card Specifications Feature Specifications Notes XP PCI card 64-bit PCI-X PCI low-profile PCI short card Bus speed 75-133 MHz PCI-X slots 66 MHz (3.3V) PCI slots Memory 2 MB Memory bandwidth...

  • Page 16: Myricom 2xp Pci Rev. E Card

    Figure 2-3: Myricom 2XP PCI Rev. E card PCI cards...

  • Page 17

    Table 2-3 describes the technical specifications for the 2XP PCI card. Table 2-3: 2XP PCI Card Specifications Feature Specifications Notes 2XP PCI card 64-bit PCI-X and PCI Can be used in any 3.3V PCI protocols slot. Bus speed 66-133 MHz Capable of peak PCI date rates at the limits of the PCI-X or PCI buses...

  • Page 18: Myricom 2xp Pci Rev. F Card

    Figure 2-4: Myricom 2XP PCI Rev. F card HPTC-0105 PCI cards...

  • Page 19: Switch, Spine And Monitoring Line Cards, Myrinet System Interconnect 8-port Switch Card, Port Switch Card

    Switch, spine and monitoring line cards The Myrinet system interconnect uses three types of cards. This chapter presents the following information: • A description of the switch card (Section 3.1). • A description of the spine card (Section 3.2). • A description of the monitoring line card (Section 3.3).

  • Page 20: Myrinet System Interconnect 8-port Gige Switch Card

    Item Description Port identification number Type LC fiber port Figure 3-3 identifies the external features of the Myrinet system interconnect 8-port GigE switch card. Each GigE port has two green LEDs in the RJ45 connector. The right LED indicates link activity, and the left LED indicates traffic. Figure 3-3: Myrinet system interconnect 8-Port GigE switch card Item Description...

  • Page 21: Block Diagram Of The Gigabit Ethernet Switch Card

    Figure 3-4: Block Diagram of the gigabit ethernet switch card Myrinet Backplane Interface 7 6 5 4 3 2 1 0 dual + 12V Serial link Power & hot-swap circuits Scan path Sense & XBar16 Clocks Control Lanai-XM-based Myrinet-GbE conversion for each of these 8 ports GbE Front-panel Ports HPTC-0017 XBar16...

  • Page 22: Spine Card Block Diagram

    Figure 3-5: Spine Card Block Diagram Backplane Interface dual +12V µC Serial link Power & hot-swap circuits Sense & Clocks Control Physical-Level Conversion Circuits Front-Panel Ports 3.3 Monitoring line card All system interconnects must have a monitoring line card installed. The monitoring line card is powered from the dual +12V sources in the same way as the port line cards, and likewise includes a small microcontroller to monitor internal functions, voltages, and temperatures.

  • Page 23: Monitoring Line Card, External Features Of The Monitoring Line Card

    Figure 3-6: Monitoring Line Card The Ethernet ports are dual redundant and have the same MAC address. The Ethernet MAC address is shown on a label on the front panel of the line card. The dual-redundant, automatic-failover feature is implemented in the firmware. The Ethernet ports also require a DHCP server on the network to give the card’s microcontroller an IP address and subnet mask.

  • Page 25: Led Indications And Diagnostics

    LED indications and diagnostics If you encounter any operational problems with your Myrinet system interconnect, you can use the LEDs on each switch port and PCI interface to diagnose and correct the problem. The monitoring line card installed on the system interconnect also can be used for diagnostics.

  • Page 26

    Verify that the host booted correctly by making a connection to the host through its management console. If it did not boot correctly, check the startup procedures in the software documentation and see the HP ROM-Based Setup Utility User Guide, which is provided in the server’s documentation set.

  • Page 27

    • Enable or disable detection traps To configure network access to the monitoring line card, use the following procedure: Obtain the MAC address from the label on the front panel of the monitoring line card. Log in to the control node as root and edit the /etc/dhcpd.conf file to add a fixed IP address for the Myrinet line card MAC address.

  • Page 29: Replacing Components

    Replacing components Several components can be replaced in the Myrinet system interconnect: This chapter presents the following instructions relating to the replaceable components: • Installing a monitoring line card (Section 5.1). • Removing and installing an 8-port line card (Section 5.2). •...

  • Page 30: Unlocking Switch Card Handles

    5.1.1 Rebooting the monitoring line card The easiest way to reboot a Myrinet interconnect is with a Web browser. In the following steps, uc denotes microcontroller: Get the main page (http://switchname). Click on the link "Slot 56 (big uc)." Select "reboot" from the menu "commandForBigUc." Click on Apply.

  • Page 31: Ejecting A Switch Card, Sliding A Switch Card, Backplane Power Connector

    Figure 5-2: Ejecting a Switch Card It is normal that the ejection is a little stiff while the line card’s front panel is in contact with the front panel or bezels above and below. The bottom of each front panel and the bottom of the top bezel have spring gasketing as an EMI seal.

  • Page 32: Removing A Fan Tray

    _________________________ Note _________________________ Do not operate a Myrinet interconnect for extended periods with missing line cards. Use blank panels to fill in any empty line card slots. The front of the enclosure needs to be closed both for the efficiency of the fan cooling and to avoid electromagnetic interference (EMI).

  • Page 33: Interconnect Fan Tray

    Figure 5-6: Interconnect Fan Tray 5.4 Replacing an interconnect chassis The communications ports on the interconnect face toward the rear of the rack. The location of the interconnect in the rack depends on the specific configuration of your cluster. Up to four 14U interconnects might be installed in a single rack, and their rack positions might vary depending on the configuration rules for your system.

  • Page 34: Removing The Retainer Brackets

    Disconnect and remove each fiber-optic cable until all ports are unplugged. Secure the cables out of the interconnect’s removal path. _______________________ Note _______________________ Observe the maximum bend radius for fiber-optic cables. Bending the cables too much can cause the glass core to break. A good rule of thumb is to never bend more than a 2-inch radius.

  • Page 35: Sliding Interconnect Chassis From The Rack, Blanking Plate

    Figure 5-8: Sliding Interconnect Chassis from the Rack Rear of Rack Move the interconnect to a suitable work surface. Retain all screws for reuse. Use only the 6-32 x .375 in Phillips pan-head screws to attach the retainer brackets to the chassis. Use of incorrect screws might damage components. Procedure for 9U Interconnect To remove a 9U system interconnect from the rack, use the following procedure: Bring the cluster to an appropriate state for switch component removal and...

  • Page 36: Hold-down Bracket, Left Bracket Retainer, Right Bracket Retainer

    At the rear of the rack, disconnect the cables after first ensuring that the cables are properly labeled for their appropriate ports. This is important if you are planning to replace a defective interconnect. Remove the cables from the cable management straps, and route them carefully out of the way of the interconnect’s removal path.

  • Page 37: Removing The Existing Brackets

    5.4.2 Swapping the line cards When replacing an interconnect chassis, the monitoring line card and all switch and spine cards must be removed from the current interconnect and moved into the new chassis. Before starting the removal process, you must prepare the new interconnect for the line cards.

  • Page 38

    5.4.3 Replacing the interconnect in the rack After removing the interconnect and swapping the line cards, you are ready to replace the interconnect in the rack. You insert the interconnect chassis from the rear of the rack, with its front panel (the connection ports) facing the rear of the rack.

  • Page 39: Powering Up Procedure

    Powering up procedure To power up a Myrinet system interconnect, plug the power cord into the standard C14 power jack labeled 100–127/200–240 V-. The power cord is rated 10A for the 5U interconnect, and 15A for the 9U chassis. There is no power switch on the enclosure, which prevents an accidental network shutdown.

  • Page 40

    Characteristic 5U Chassis 9U Chassis Operating 15% to 80% @ 104° F Nonoperating 90% @ 149° F Powering up procedure...

  • Page 41

    Index Numbers and Special Characters 2XP PCI , 2-1 , 2-3 Rev. E host bus adapter , 2-5 ( See HBA ) Rev. F , 1-1, 3-5 HTTP , 1-1 application node identification label , 2-1 PCI card interconnect , 1-2, 3-2 backplane backplane , 1-1...

  • Page 42

    , 3-3 , 3-3 spine card spine card , 3-4, 4-2 , 3-3 monitoring line card numbering , 5-1 switch installing , 1-1 , 4-2 interconnect network access , 1-2, 3-1 , 5-2 switch card rebooting , 5-3 , 4-3 installing reseating , 3-2...

Comments to this Manuals

Symbols: 0
Latest comments: