HP Cluster Platform Interconnects v2010 Manual
Myrinet system interconnect guide
Hide thumbs
Also See for Cluster Platform Interconnects v2010:
- User manual (140 pages) ,
- Overview (130 pages) ,
- Manual (76 pages)
Table of Contents
Advertisement
Quick Links
HP Cluster Platform
Myrinet System Interconnect Guide
Part Number: A-CPMYI-1A
May 2005
Revision/Update Information:
Version 1.0
Product Version:
HP Clusters
This manual describes the Myricom Myrinet system interconnects used in HP Cluster Platform
solutions.
Hewlett-Packard Company
Palo Alto, California
Advertisement
Table of Contents
Related Manuals for HP Cluster Platform Interconnects v2010
Summary of Contents for HP Cluster Platform Interconnects v2010
- Page 1 HP Cluster Platform Myrinet System Interconnect Guide Part Number: A-CPMYI-1A May 2005 Revision/Update Information: Version 1.0 Product Version: HP Clusters This manual describes the Myricom Myrinet system interconnects used in HP Cluster Platform solutions. Hewlett-Packard Company Palo Alto, California...
- Page 2 © Copyright 2005 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty.
-
Page 3: Table Of Contents
Contents About This Manual 1 Myrinet interconnect overview Interconnect characteristics ..........Backplane and chassis types . - Page 4 128-Port Myrinet System Interconnect ........Fully Populated 128-Port Myrinet System Interconnect .
-
Page 5: About This Manual
About This Manual This manual describes the Myricom Myrinet system interconnects used in some HP Cluster Platform solutions. This manual does not describe the procedures and tools that are required to install and configure the system hardware or software. It does contain references for cluster components that have their own documentation. -
Page 6: Symbols In Text
Symbols in text zk-2071 WARNING: Text set off in this manner indicates that failure to follow directions in the warning could result in bodily harm or loss of life. CAUTION: Text set off in this manner indicates that failure to follow zk-2071 directions in the caution could result in damage to equipment or loss of information. -
Page 7: Myrinet Interconnect Overview
Myrinet interconnect overview The high-speed Myrinet system interconnect is used in some HP Cluster Platform solutions. The chassis for the interconnect is 5U or 9U, depending on the cluster configuration. A PCI adapter card connects each application node in the cluster to the interconnect. -
Page 8: Backplane And Chassis Types
1.2 Backplane and chassis types The Myrinet interconnect used in an HP Cluster Platform can be a 5U (64-port) or 9U (128-port) chassis. The backplane (Figure 1-2) provides connectivity for the 8-port switch cards. Figure 1-2: Myrinet System Interconnect Backplane For configurations greater than 128 total nodes, you must build the network out to increase the number of available ports and to maintain the cross-sectional bandwidth. -
Page 9: Nonfederated Configuration Switch
The 128-port, node-level switch in a federated configuration provides 64 downlinks to nodes and dedicates the other 64 ports to uplinks that provide connectivity to deeper parts of the network. A top-level switch, another Myrinet system interconnect, consolidates the uplinks from two node-level switches. Each switch supports up to 128 links, 64 links from each node-level switch;... -
Page 10: Wiring Rules
Figure 1-5: 128-Port Myrinet System Interconnect xc-013B-IG Table 1-1 explains the chassis types, and the types and location of line cards within the interconnects. Table 1-1: Chassis types Chassis Type Component Quantity Location Bounded chassis M3-E128 17 slot chassis Monitoring Line Card Slot M 8 Port Switch Line Card 1 -- 16... -
Page 11: Fully Populated 128-Port Myrinet System Interconnect
Figure 1-6: Fully Populated 128-Port Myrinet System Interconnect Application nodes start here, move left, and then down to the next card. Switch cards are numbered 0-15, beginning with the If the control node is card below the connected to the monitoring line card (1). -
Page 13: Pci Cards
PCI cards Every node in a cluster that uses a Myricom system interconnect has a PCI host bus adapter (HBA), and each host interface has a connection to the interconnect. This chapter presents the following information relating to the three versions of the PCI card that might be used in an HP Cluster Platform solution: •... -
Page 14: Single-Port Xp Pci Card
2.2 Single-port XP PCI card The single-port XP PCI card (Figure 2-2) is the Myricom Rev. D card. It supports one X-port interface and one PCI-X interface. Figure 2-2: Single-port XP PCI Rev. D card PCI cards... -
Page 15: Dual-Port 2Xp Pci
Table 2-1 shows the single-port XP PCI card’s technical specifications. Table 2-1: XP PCI Host Interface Card Specifications Feature Specifications Notes XP PCI card 64-bit PCI-X PCI low-profile PCI short card Bus speed 75-133 MHz PCI-X slots 66 MHz (3.3V) PCI slots Memory 2 MB Memory bandwidth... -
Page 16: Myricom 2Xp Pci Rev. E Card
Figure 2-3: Myricom 2XP PCI Rev. E card PCI cards... -
Page 17: Single-Port 2Xp Pci
Table 2-3 describes the technical specifications for the 2XP PCI card. Table 2-3: 2XP PCI Card Specifications Feature Specifications Notes 2XP PCI card 64-bit PCI-X and PCI Can be used in any 3.3V PCI protocols slot. Bus speed 66-133 MHz Capable of peak PCI date rates at the limits of the PCI-X or PCI buses... -
Page 18: Myricom 2Xp Pci Rev. F Card
Figure 2-4: Myricom 2XP PCI Rev. F card HPTC-0105 PCI cards... -
Page 19: Switch, Spine And Monitoring Line Cards
Switch, spine and monitoring line cards The Myrinet system interconnect uses three types of cards. This chapter presents the following information: • A description of the switch card (Section 3.1). • A description of the spine card (Section 3.2). • A description of the monitoring line card (Section 3.3). -
Page 20: Myrinet System Interconnect 8-Port Gige Switch Card
Item Description Port identification number Type LC fiber port Figure 3-3 identifies the external features of the Myrinet system interconnect 8-port GigE switch card. Each GigE port has two green LEDs in the RJ45 connector. The right LED indicates link activity, and the left LED indicates traffic. Figure 3-3: Myrinet system interconnect 8-Port GigE switch card Item Description... -
Page 21: Spine Card
Figure 3-4: Block Diagram of the gigabit ethernet switch card Myrinet Backplane Interface 7 6 5 4 3 2 1 0 dual + 12V Serial link Power & hot-swap circuits Scan path Sense & XBar16 Clocks Control Lanai-XM-based Myrinet-GbE conversion for each of these 8 ports GbE Front-panel Ports HPTC-0017 XBar16... -
Page 22: Monitoring Line Card
Figure 3-5: Spine Card Block Diagram Backplane Interface dual +12V µC Serial link Power & hot-swap circuits Sense & Clocks Control Physical-Level Conversion Circuits Front-Panel Ports 3.3 Monitoring line card All system interconnects must have a monitoring line card installed. The monitoring line card is powered from the dual +12V sources in the same way as the port line cards, and likewise includes a small microcontroller to monitor internal functions, voltages, and temperatures. -
Page 23: Monitoring Line Card
Figure 3-6: Monitoring Line Card The Ethernet ports are dual redundant and have the same MAC address. The Ethernet MAC address is shown on a label on the front panel of the line card. The dual-redundant, automatic-failover feature is implemented in the firmware. The Ethernet ports also require a DHCP server on the network to give the card’s microcontroller an IP address and subnet mask. -
Page 25: Led Indications And Diagnostics
LED indications and diagnostics If you encounter any operational problems with your Myrinet system interconnect, you can use the LEDs on each switch port and PCI interface to diagnose and correct the problem. The monitoring line card installed on the system interconnect also can be used for diagnostics. -
Page 26: Using The Monitoring Line Card For Diagnostics
Verify that the host booted correctly by making a connection to the host through its management console. If it did not boot correctly, check the startup procedures in the software documentation and see the HP ROM-Based Setup Utility User Guide, which is provided in the server’s documentation set. - Page 27 • Enable or disable detection traps To configure network access to the monitoring line card, use the following procedure: Obtain the MAC address from the label on the front panel of the monitoring line card. Log in to the control node as root and edit the /etc/dhcpd.conf file to add a fixed IP address for the Myrinet line card MAC address.
-
Page 29: Replacing Components
Replacing components Several components can be replaced in the Myrinet system interconnect: This chapter presents the following instructions relating to the replaceable components: • Installing a monitoring line card (Section 5.1). • Removing and installing an 8-port line card (Section 5.2). •... -
Page 30: Rebooting The Monitoring Line Card
5.1.1 Rebooting the monitoring line card The easiest way to reboot a Myrinet interconnect is with a Web browser. In the following steps, uc denotes microcontroller: Get the main page (http://switchname). Click on the link "Slot 56 (big uc)." Select "reboot" from the menu "commandForBigUc." Click on Apply. -
Page 31: Ejecting A Switch Card
Figure 5-2: Ejecting a Switch Card It is normal that the ejection is a little stiff while the line card’s front panel is in contact with the front panel or bezels above and below. The bottom of each front panel and the bottom of the top bezel have spring gasketing as an EMI seal. -
Page 32: Replacing A Fan Tray
_________________________ Note _________________________ Do not operate a Myrinet interconnect for extended periods with missing line cards. Use blank panels to fill in any empty line card slots. The front of the enclosure needs to be closed both for the efficiency of the fan cooling and to avoid electromagnetic interference (EMI). -
Page 33: Replacing An Interconnect Chassis
Figure 5-6: Interconnect Fan Tray 5.4 Replacing an interconnect chassis The communications ports on the interconnect face toward the rear of the rack. The location of the interconnect in the rack depends on the specific configuration of your cluster. Up to four 14U interconnects might be installed in a single rack, and their rack positions might vary depending on the configuration rules for your system. -
Page 34: Removing The Retainer Brackets
Disconnect and remove each fiber-optic cable until all ports are unplugged. Secure the cables out of the interconnect’s removal path. _______________________ Note _______________________ Observe the maximum bend radius for fiber-optic cables. Bending the cables too much can cause the glass core to break. A good rule of thumb is to never bend more than a 2-inch radius. -
Page 35: Sliding Interconnect Chassis From The Rack
Figure 5-8: Sliding Interconnect Chassis from the Rack Rear of Rack Move the interconnect to a suitable work surface. Retain all screws for reuse. Use only the 6-32 x .375 in Phillips pan-head screws to attach the retainer brackets to the chassis. Use of incorrect screws might damage components. Procedure for 9U Interconnect To remove a 9U system interconnect from the rack, use the following procedure: Bring the cluster to an appropriate state for switch component removal and... -
Page 36: Hold-Down Bracket
At the rear of the rack, disconnect the cables after first ensuring that the cables are properly labeled for their appropriate ports. This is important if you are planning to replace a defective interconnect. Remove the cables from the cable management straps, and route them carefully out of the way of the interconnect’s removal path. -
Page 37: Swapping The Line Cards
5.4.2 Swapping the line cards When replacing an interconnect chassis, the monitoring line card and all switch and spine cards must be removed from the current interconnect and moved into the new chassis. Before starting the removal process, you must prepare the new interconnect for the line cards. -
Page 38: Replacing The Interconnect In The Rack
5.4.3 Replacing the interconnect in the rack After removing the interconnect and swapping the line cards, you are ready to replace the interconnect in the rack. You insert the interconnect chassis from the rear of the rack, with its front panel (the connection ports) facing the rear of the rack. -
Page 39: Powering Up Procedure
Powering up procedure To power up a Myrinet system interconnect, plug the power cord into the standard C14 power jack labeled 100–127/200–240 V-. The power cord is rated 10A for the 5U interconnect, and 15A for the 9U chassis. There is no power switch on the enclosure, which prevents an accidental network shutdown. - Page 40 Characteristic 5U Chassis 9U Chassis Operating 15% to 80% @ 104° F Nonoperating 90% @ 149° F Powering up procedure...
-
Page 41: Index
Index Numbers and Special Characters 2XP PCI , 2-1 , 2-3 Rev. E host bus adapter , 2-5 ( See HBA ) Rev. F , 1-1, 3-5 HTTP , 1-1 application node identification label , 2-1 PCI card interconnect , 1-2, 3-2 backplane backplane , 1-1... - Page 42 , 3-3 , 3-3 spine card spine card , 3-4, 4-2 , 3-3 monitoring line card numbering , 5-1 switch installing , 1-1 , 4-2 interconnect network access , 1-2, 3-1 , 5-2 switch card rebooting , 5-3 , 4-3 installing reseating , 3-2...