Myricom Myrinet-2000 Manual

Switches and switch networks

Hide thumbs

Table Of Contents

Table of Contents

Quick Links

Download this manual

Switches and Switch Networks

Safety, Installation, and Use Precautions

Provide power to the switch enclosures only with a 3-wire, IEC line cord from an outlet that

includes a safety ground. The input power is 100–127 V~ or 200–240 V~, 50Hz or 60Hz. All

switch enclosures are shipped with a power cord appropriate to the unit and to the shipping

destination. The power cord should be rated for 10A except for 100–127V~ power cords for the

M3-E128, which are rated for 15A.

An M3-E128 switch enclosure fully populated with line cards weighs approximately 31 kg (68

pounds). Two people are required to mount this unit in a rack. One person should not attempt to

install an M3-E128 switch unit in a rack. The rack mounting holes conform to the EIA-310

standard for 19-inch racks.

Line cards and the fan tray may be "hot swapped" (inserted or removed with the power on).

However,

• Insert and remove line cards gently, according to the instructions that start on page 4.

• Do not operate a switch for extended periods with missing line cards. Use M3-BLANK

panels to fill in any empty line-card slots. The front of the enclosure needs to be closed

both for the efficiency of the fan cooling and to avoid electromagnetic interference

(EMI).

• Each line card includes thermal sensors. Operating the switch without a fan tray for more

than one minute may result in line cards turning themselves off.

Configurations with optical-fiber Myrinet ports are Class I Laser Products. Optical-fiber

components with this classification pose no threat of biological damage.

The most recent revision of this document can be downloaded from

and supplemental information is linked from this same web page.

Guide to Myrinet-2000

Myricom, Inc.

Revision: 27 August 2001

http://www.myri.com/myrinet/m3switch/guide/

-- 1 --

Revision: 27 August 2001

Table of Contents

Need help?

Do you have a question about the Myrinet-2000 and is the answer not in the manual?

Questions and answers

Summary of Contents for Myricom Myrinet-2000

Page 1 Guide to Myrinet-2000 Switches and Switch Networks Myricom, Inc. Revision: 27 August 2001 Safety, Installation, and Use Precautions Provide power to the switch enclosures only with a 3-wire, IEC line cord from an outlet that includes a safety ground. The input power is 100–127 V~ or 200–240 V~, 50Hz or 60Hz. All switch enclosures are shipped with a power cord appropriate to the unit and to the shipping destination.
Page 2 Safety. Myricom is proud to employ TUV Rheinland of North America, Inc., to perform the independent safety certifications for these Myrinet-2000 switch products. The M3-E16, M3- E32, M3-E64, and M3-E128 were certified in accordance with the IEC System for Conformity Testing and Certification of Electrical Equipment (IECEE), CB Scheme.
Page 3 Myrinet-Fiber links carry packet data at a 2+2 Gb/s data rate on industry-standard 50/125 multimode-fiber-pair cables with “LC” connectorization. Myricom ships Fiber components with a dust plug in each port. It is recommended that the dust plug be left in place in unused switch ports.
Page 4 Class A limits in the 3–6 GHz range. These tests show that this EMI is due to characteristics of the HSSDC connectors. Myricom is continuing engineering efforts to reduce the EMI emissions, including using a different type of HSSDC connector in the most recent production products.
Page 5 Hot swapping line cards and the fan tray Each Myrinet-2000 switch line card has a pair of “handles” that lock the card in place, and that serve as a lever for inserting or removing the line card. Let’s start with a line card that is inserted in a card slot of an enclosure.
Page 6 As the line card becomes fully inserted, the Status LED should show green within ~0.5s. The green Status LED indicates that the line card has passed all of its self tests, and is operating. © 2001, Myricom, Inc. -- 6 -- Revision: 27 August 2001...
Page 7 When a fan try is removed, it should be replaced within approximately one minute, or else line cards may power themselves off in response to an over-temperature condition. © 2001, Myricom, Inc. -- 7 -- Revision: 27 August 2001...
Page 8: Table Of Contents
5. Monitoring Line Card ....................... 17 6. Configurations up to 128 Hosts ....................19 7. Topology Concepts ........................21 8. Clos Networks .......................... 22 9. Clos Networks for more than 128 Hosts .................. 23 © 2001, Myricom, Inc. -- 8 -- Revision: 27 August 2001...
Page 9: Introduction
Guide to Myrinet-2000 Switches and Switch Networks Principles of Operation 1. Introduction The basic building block of this family of switch products is a 16-port Myrinet crossbar switch, which is implemented on a single chip designated as the XBar16. The XBar16 is pictured in the block diagrams below as a circle, and the Myrinet links as lines.
Page 10 (red boxes). These port line cards have up to 8 Myrinet ports through their front panel, and 8 Myrinet-SAN ports at connectors to the backplane. Myricom supplies different port line cards in which the front-panel ports are Myrinet Fiber, Serial, SAN (switchable between SAN-2000 and SAN-1280), or (legacy) LAN.
Page 11: The Family Of Enclosures
2. The Family of Enclosures Myricom offers a series of enclosures, so that you can choose the one that best fits your needs. Enclosure product code M3-E16 M3-E32 M3-E64 M3-E128 Maximum # of port line cards Maximum # of front-panel ports Physical height (1U = 1.75 in)
Page 12: Other Features Of The Enclosures And Line Cards
3. Other Features of the Enclosures and Line Cards The Myrinet-2000 enclosures and line cards are designed so that the line cards or the fan tray can be “hot-swapped,” i.e., inserted or removed from an enclosure that is powered. See the hot-swap instructions starting on page 5.
Page 13: Port Line Cards
(each). Also, these fans have tachometer outputs. The fan µC converts the tachometer outputs to fan RPMs, which can be read by the monitoring line card. The modularity of the Myrinet-2000 switches and switch networks provides configuration flexibility, manufacturing economies, and, most importantly: •...
Page 14 (HA) features to ‘on’ or ‘off.’ The line card must be ejected from an enclosure slot to change the © 2001, Myricom, Inc. -- 14 --...
Page 15: Monitoring Line Card
This information includes: • The type of line card and its serial number, • All information from the XBar16 scan path (port state, packet counts, CRC errors), © 2001, Myricom, Inc. -- 15 -- Revision: 27 August 2001...
Page 16 The front-panel ports are labeled 0...7, corresponding to the ports of the backplane interface. The indicator LEDs are the same, and the µC information is the same except that there is no XBar16 scan path. © 2001, Myricom, Inc. -- 16 -- Revision: 27 August 2001...
Page 17 The standard firmware supports two software interfaces over the Ethernet connections, SNMP and a web server. Current tools, documentation, the SNMP MIB, and the firmware is linked © 2001, Myricom, Inc. -- 17 -- Revision: 27 August 2001...
Page 18 Home page of an M3-E32 Beginning of page for first port line card End of page for first port line card A part of the fan-monitor page © 2001, Myricom, Inc. -- 18 -- Revision: 27 August 2001...
Page 19: Configurations Up To 128 Hosts
8 Fiber + (none) M3-E64 + M3-M + 6*M3-BLANK + 8 SAN M3-SW16-8F + M3-SW16-8M For each 8 hosts added for a particular PHY, just add another line card. © 2001, Myricom, Inc. -- 19 -- Revision: 27 August 2001...
Page 20 All configurations larger than 16 hosts and up to 128 hosts escape the (page 12) ‘special case’ optimization of the M3-E16 with an M3-SPINE line card, and use all M3-SW16 line cards. Indeed, the configuration of Myrinet-2000 switch networks for 16 < N ≤ 128, where N is the number of hosts, is particularly simple.
Page 22: Clos Networks
As a first step toward understanding these constructions, here is the Clos network of 16-port switches that connects 128 hosts, a network that we’ll refer to as the Clos128: © 2001, Myricom, Inc. -- 22 -- Revision: 27 August 2001...
Page 23: Clos Networks For More Than 128 Hosts
We can apply the principle illustrated above for the leaf switches of the 128-host Clos network. Let us use the standard construction for a 64-host network, which requires a spine of 8 8-port © 2001, Myricom, Inc. -- 23 --...
Page 24 16 of the Clos64+64 components to a spine composed of 64 16-port switches: 64 16-port Switches (spine) Spreader Network (1,024 Links) 16 Clos64+64 components for 1,024 hosts (A better diagram is linked from http://www.myri.com/myrinet/m3switch/guide/.) © 2001, Myricom, Inc. -- 24 -- Revision: 27 August 2001...
Page 25 128 < N ≤ 1024 range, and the optimization techniques are the same. For additional information about Myrinet Clos networks for 128 < N ≤ 8192, please see the supplemental information at http://www.myri.com/myrinet/m3switch/guide/ © 2001, Myricom, Inc. -- 25 -- Revision: 27 August 2001...