Download
Share
Print this page
Mellanox Technologies Grid Director 4200 Installation Manual
  1. Manuals
  2. Brands
  3. Mellanox Technologies Manuals
  4. Switch
  5. Grid Director 4200
  6. Installation manual
Mellanox Technologies Grid Director 4200 Installation Manual

Mellanox Technologies Grid Director 4200 Installation Manual

Hide thumbs

Advertisement

Quick Links

Download this manual
Grid Director 4200
Installation Manual
www.mellanox.com

Advertisement

loading
Need help?

Need help?

Do you have a question about the Grid Director 4200 and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Mellanox Technologies Grid Director 4200

Summary of Contents for Mellanox Technologies Grid Director 4200

  • Page 1 Grid Director 4200 Installation Manual www.mellanox.com...
  • Page 2 Names and logos identifying products of Mellanox in this document are registered trademarks or trademarks of Mellanox. Voltaire is a registered trademark of Mellanox Technologies, Ltd. All other trademarks mentioned in this document are the property of their respective owners.
  • Page 3 Power Supplies (sPSU-S) ..................15 2.3.8 Reset Buttons ......................15 2.3.9 Management Reset ....................15 Unpacking and Packing the Grid Director 4200 ................. 16 Package Contents ........................ 16 Packing the Grid Director 4200 .................... 19 System Configurations ......................... 21 Configuration Summary ......................21 4.1.1...
  • Page 4 Contents Racking Rules - Clearance Requirements ................26 5.10 Power Data ........................... 26 5.10.1 Type of Power Cord on the Power Supply ............. 26 5.10.2 Power Requirements ....................26 5.10.3 Power Characteristics ..................... 26 5.10.4 Power Data per Module ..................27 5.11 Calculating Heat Output in BTUs ..................
  • Page 5 6.3.2 Rail Kit Assembly Overview ..................46 6.3.3 Rail Kit Assembling Procedure in Detail ..............48 Mounting the Grid Director 4200 in a Rack ................50 Installing System Boards ......................53 Chassis Board Configuration ....................53 7.1.1 Grid Director 4200 ....................53 Inserting/Extracting Boards ....................
  • Page 6 Figure 6: Grid Director 4200 Packed ..................... 20 Figure 7: sFU-42H Horizontal Fan Unit ....................24 Figure 8: Grid Director 4200 Safety and EMC Label (Example) ............36 Figure 9: Center of Gravity Data......................38 Figure 10: Grid Director 4200 Grounding Connection ................42 Figure 11: Grounding Holes for the 4200 Grid Director ................
  • Page 7 Table 29: Vibration ..........................37 Table 30: Free Fall ..........................38 Table 31: Grid Director 4200 Center of Gravity Data ................38 Table 32: Material and Tools Required ....................39 Table 33: Grid Director 4200 QDR N:1/N:N Redundancy ..............44...
  • Page 8 For further information and assistance, go to http://www.mellanox.com/content/pages.php?pg=support_index. Audience This manual is for system administrators responsible for installing the Grid Director 4200. We assume that you are familiar with the concepts and terminology of InfiniBand, Ethernet, and local area networking.
  • Page 9 Grid Director 4200 Installation Manual CAUTION: Alerts you to the risk of personal injury, system damage, or loss of data. WARNING: Warns you that failure to take or avoid a specific action might result in personal injury or a malfunction of the hardware or software. Be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents before you work on any equipment.
  • Page 10 Introduction Mellanox Grid Director 4200 Overview The Mellanox Grid Director 4200 is a high performance, ultra low latency InfiniBand switch for high performance clusters. With configurations of up to 162 ports of 40 Gbps per port InfiniBand connectivity, the 4200 delivers 11.52 Tbps of bandwidth with between 100 and 300 nanoseconds of port-to-port latency.
  • Page 11  InfiniBand specification 1.2 compliant. Product Architecture The Grid Director 4200 is a modular, rack-mountable chassis which can be configured for a wide range of applications. Highlights of the Grid Director 4200 architecture:  Up to nine (9) Line Boards (sLB-4018) with 18 QSFP InfiniBand QSFP external interfaces...
  • Page 12 Director 4200. This section details the front panel of the Grid Director 4200. Figure 3: Grid Director4200 Front Panel Description The following table details the Grid Director 4200 front panel components (optimal configuration). Table 1: Grid Director 4200 Front Panel Components Component No.
  • Page 13 Grid Director 4200 Installation Manual Grid Director 4200 Rear Panel Description This section details the rear panel of a Grid Director 4200 (optimal configuration). Figure 4: Grid Director 4200 Rear Panel Description The following table details Grid Director 4200 rear panel components.
  • Page 14 2.3.2 Line Boards The rear side of the Grid Director 4200 can host up to up to 9 Line Boards (sLB-4018). Each line board has 18 external QSFP interfaces, and 16 internal 4X InfiniBand interfaces for connectivity between the Line cards and the Switch fabric. Line boards are installed at the rear of the chassis in a horizontal position.
  • Page 15 2.3.9 Management Reset To reset the Grid Director 4200, push any of the Reset buttons using a thick wire or the tip of a pen. Pressing the Reset button for one to six seconds resets the sMB-CM Management Board only;...
  • Page 16 Unpacking and Packing the Grid Director 4200 Unpacking and Packing the Grid Director 4200 Package Contents The Grid Director 4200 is supplied in a heavy duty tri-wall cardboard box secured with packing straps. Refer to System Configurations (on page 21) for information about the various 4200 configurations and items shipped with the chassis.
  • Page 17 Grid Director 4200 Installation Manual Figure 5: Unpacking the Grid Director 4200 (Schematic) 1 - Grid Director 4200 2 - Top Foam 3 - Cable Management 4 - Mounting Rails (Left and Right) 5 - Foam Plate 6 - Carton Plate – placed at the top of the rear of the chassis to protect the opening...
  • Page 18 1. Cut the packing straps and open the box. 2. Unpack the Grid Director 4200 chassis, as follows: Remove the box from around the chassis by sliding it upwards. The Grid Director 4200 chassis in its partially unpacked state is shown in the following figure.
  • Page 19 Grid Director 4200 Installation Manual 4. Lift the Grid Director 4200 from the pallet and place it on a flat surface. Due to its heavy weight, two people are needed to lift the Grid Director 4200. NOTE: Use the four handles at bottom of chassis to slide the unit into its place.
  • Page 20 Unpacking and Packing the Grid Director 4200 Figure 6: Grid Director 4200 Packed Chapter 4...
  • Page 21 4.1.1 Module Configuration per Chassis The following table provides a comprehensive list of modules as well as the maximum number of modules that can be installed in a Grid Director 4200 system. Table 3: Module Configuration per Chassis Module Name...
  • Page 22 Power Supplies (sPSU-S) Management Boards (sMB-CM) Embedded Chassis Management (CLI) Grid Director 4200 Configuration and Part Numbers The following table details the basic configuration of your Grid Director 4200. Table 5: Grid Director 4200 Basic Configuration Product Description Product/Module Mellanox P/N...
  • Page 23 Grid Director 4200 Installation Manual Technical Specifications and Certifications Grid Director 4200 Technical Specifications The following details the Grid Director 4200 technical specifications. Architecture  4 slots for Fabric Boards in the front of the chassis  9 slots for Line Boards in the rear panel ...
  • Page 24 Technical Specifications and Certifications  On board high performance, low power CPU for low-level chassis management purposes  On-board temperature monitoring reported to Management system  Thermal shutdown for over temperature sensing. The following table provides additional sLB-4018 specifications. Table 7: 4200 Line Board (sLB-4018) Specifications Item Specification Data throughput QDR...
  • Page 25 Fabric Board to chassis 2 (or use manual screwdriver) Module Configuration Rules The Grid Director 4200 supports up to two Management Boards that are used for redundancy/failover implementation. Use slot one (left) when only one Management Board is installed. For more information, see...
  • Page 26 Type of Power Cord on the Power Supply With each Power Supply, Mellanox provides a 15Amp cord (2 meter jumper), with a universal plug for a PDU (Power Distribution Unit). This cord is detachable from the Grid Director 4200 Power Supply. Cable wiring: 3x14AWG.
  • Page 27 Grid Director 4200 Installation Manual Table 12: Single Power Supply Characteristics Attribute Specification 90 – 254 volts, 47 – 63 Hz, auto-sensing Power Entry Power Rating 1400 Watts @ 220V per module 1200 Watts @ 110V per module Nameplate Power Rating (VA) Input: 100V-240Vac @ 16A;...
  • Page 28 110.0 sMB-CM 45.0 Line Board sLB-4018 110.0 (*) Add 1.5 Watt for each connected QSFP optical module Table 15: Spare Parts for Grid Director 4200 Description Module Name DC Power Consumption [Watt] sFU-42H, Horizontal Fan Unit sFU-42H 300.0 sFU-42V, Vertical Fan Unit sFU-42V 150.0...
  • Page 29 Minimum rack depth required for the rail kit Maximum rack depth required for the rail kit 5.15 Weights and Dimensions The following table details the Grid Director 4200 and modules weights and dimensions. Table 18: Table -. Grid Director 4700 Weights & Dimensions Metric (2) English (1)
  • Page 30 6.17 29.5 8.15 1 in = 2.54 cm 1 kg = 2.205 lbs 5.16 LED Indications The following tables detail the Grid Director 4200 modules LED Indications. 5.16.1 sLB-4018 Line Board LEDs Table 19: sLB-4018 LEDs Indication LED Description Remarks...
  • Page 31 Grid Director 4200 Installation Manual Indication LED Description Remarks OFF – indicates a problem in power supply to the board Info (Blue) General purpose LED for management use OFF – No attention required User only Blinking – Chassis attention required OFF –...
  • Page 32 Technical Specifications and Certifications 5.16.3 Management Board (sMB-CM) LEDs Table 21: sMB-CM LEDs Indication LED Description Remarks Fans (Green) Off- All fan drawer are OK Two Fans drawers On - Vertical and Horizontal fan units fault indication. Indicates that at least one of the 11 fans has failed. Temp (Orange) Indicates an over-temperature fault on the chassis: Temp (Orange)
  • Page 33 ON – Indicates that the AC power source is present. AC ON (Green) 5.18 Environmental Specifications The following table details the environment specifications for the Grid Director 4200. Table 24: Grid Director 4200 Environmental Specifications Ambient Operating Temperature 32° to 113°F (0° to 45°C)
  • Page 34 Technical Specifications and Certifications 5.20.1 4200 system configuration for MTBF calculation The system configuration is a fully populated chassis with 9 line cards, 4 fabric cards, 4 power supplies, 1 Management Board and 2 fan modules(Horizontal and Vertical drawers). Table 25: 4200 System MTBF 4200 QDR Director FPMH MTBF (Hours)
  • Page 35 Grid Director 4200 Installation Manual EN 61000-4-2: 1995 + A1: 98 + A2: 2001 EN 61000-4-3: 2006 EN 61000-4-4: 2004 EN 61000-4-5: 2006 EN 61000-4-6: 2007 EN 61000-4-8: 1993 + A1(01) EN 61000-4-11: 2004 Japan VCCI Technical Requirements, V-3/2008.04, Class A.
  • Page 36 Technical Specifications and Certifications 5.23 Label(s) The Grid Director 4200 label (example) below shows safety and EMC text and logos. Figure 8: Grid Director 4200 Safety and EMC Label (Example) 5.24 Gost-R Certification Mellanox products comply with the Gost-R Russian safety regulations.
  • Page 37 Rail Kit Assembly Overview. The Grid Director 4200 is susceptible to shock and vibration if the rail kit installation instructions are not followed precisely. The following tables detail the Grid Director 4200 compliance to shock, vibration and free fall tests for operation and transportation modes.
  • Page 38 Center of gravity for minimum, typical, and maximum configurations are detailed in the following table. The measurement origin starts at the front lower left corner. Table 31: Grid Director 4200 Center of Gravity Data Grid Director 4200 Width (X) - mm...
  • Page 39 Read the following sections prior to performing the installation. 6.1.1 Tools Required for Installation The materials and tools required to mount the Grid Director 4200 chassis into a rack are listed in the table below. Table 32: Material and Tools Required...
  • Page 40 6.1.3 Rack Requirements Mount the 11U Grid Director 4200 in a standard 19 rack with the Fabric Boards facing the front of rack and the Line Board InfiniBand ports and sMB-CM facing the rear of the rack. The rack should support the mechanical and environmental characteristics of a fully populated Grid Director 4200, as listed in Site Environment Specifications.
  • Page 41 Contact the appropriate electrical inspection authority or an electrician if you are uncertain that suitable grounding is available. NOTE: The Grid Director 4200 is delivered with a grounding stud kit. The system must be permanently grounded using the grounding stud. The grounding kit includes the following items: ...
  • Page 42 For convenience, grounding can be performed from any of these locations. Figure 11: Grounding Holes for the 4200 Grid Director NOTE: Grounding Stud installation is performed before mounting the Grid Director 4200 onto the rack.  To install the Grounding Stud: 1.
  • Page 43 Power Requirements Power requirements are useful for planning the power distribution system needed to support the Grid Director 4200. Heat dissipation is an important consideration for sizing the air-conditioning requirements for an installation. Verify the available power source at the site for the type of device you are installing.
  • Page 44 16A for Europe or 20A for North America. The amount of required power supplies per system is set according to system configuration. When the Grid Director 4200 is fully configured, it supports N:1 redundancy, as shown in the example below.
  • Page 45 2 sMB-CM Mounting Process Summary The Grid Director 4200 is mounted in a standard 19" rack with the Fabric Boards facing the front of the rack. The Grid Director 4200 is provided with an adjustable rail kit to accommodate racks of differing depth. The Grid Director 4200 rail kit is adjustable to accommodate a distance of depths of 24"...
  • Page 46 Hardware Installation 6.3.1 Rail Kit and Mounting Components The Grid Director 4200 package includes:  Rail kit  Mounting Ears (or 19" Brackets) that are factory installed at the front of the chassis The supplied Rail Kit is adjustable to allow for mounting the chassis in racks of varying depths.
  • Page 47 Grid Director 4200 Installation Manual The following image shows the two rails completely assembled and how the left rail looks when it is attached to the rack. Figure 13: Overview of Assembled Rail Kit NOTES:  Use all 8 screws. Note the order of the nuts and washers.
  • Page 48 Hardware Installation 7. Install the rail kit by positioning the rail brackets in alignment with the adequate mark on the Rack. 8. Attach the front side of the Rail to the rack’s front vertical side rail with 2 screws for each side and the rear side of the Rail to the rack’s rear vertical side rail with 2 screws for each side.
  • Page 49 Grid Director 4200 Installation Manual 3. Select the round holes in the bracket and assemble the Rail with flat washers, spring washers, and nuts. Do not tighten the screws at this time. 4. Align the Rail mounting holes with the mounting holes in the equipment rack.
  • Page 50 1U marking on the rack rail. Mounting the Grid Director 4200 in a Rack This section provides instructions on how to mount the Grid Director 4200 in a rack. NOTE: Screws and Clip nuts to connect the Chassis to the Rack and Rail kit to Rack are not supplied by Mellanox.
  • Page 51 19" bracket attached to the chassis  To install the Grid Director 4200: 1. Lift the chassis using a hydraulic chassis lift (or another adequate tool) and place it on the Rails, with the front of the chassis towards you, as shown in the figure below.
  • Page 52 Hardware Installation 5. To stabilize the chassis for transport, secure the 4200 chassis onto the Rail by using three 8-32 screws, 3/8" tooth washers, and flat washers, insert them into the oval holes and the threaded holes in the chassis side walls. Once the rack is installed on the premises, these screws should be taken away to allow chassis removal in case the sides of the rack are not accessible.
  • Page 53 Grid Director 4200. The Grid Director 4200 is factory pre-assembled with fan units. Additional boards and modules are shipped according to the system configuration and are added during installation.
  • Page 54 Installing System Boards The following figures illustrate the module insertion procedure and the ejectors action. This procedure is applicable to modules with ejectors, namely Line Boards (sLB), Fabric Boards (sFB), and Management Boards (sMB). 1. Slide the module into the slot. 2.
  • Page 55 Grid Director 4200 Installation Manual 4. Use a screwdriver to secure module in place. NOTE: The ejector mechanism on the Management Board (sMB-CM) is identical to the one used on the Line and Fabric boards.
  • Page 56 When using an electric screwdriver to fasten the board screws in place, do not use more than Torque 3 force as this could damage the screws. Installing Fabric Boards The Grid Director 4200 supports up to four (4) swappable Fabric Boards (Fabric) that determine the total bandwidth of the switching CLOS topology within the chassis.
  • Page 57 Grid Director 4200 Installation Manual You can install up to nine Fabric Boards at the front of the Grid Director 4200 chassis to achieve a fully non-blocking system. Figure 14: sFB-4200 Refer to LED Indications for a comprehensive list of LEDs, LED description, and functionality.
  • Page 58 4X QDR InfiniBand ports for connectivity between the outside world and the Switch fabrics. You can install up to 9 Line Boards (sLB-4018) at the rear of the Grid Director 4200. The sLB-2018 hosts 18 external QSFP ports and 18 internal ports on each sLB-4018.
  • Page 59 You can either install one or two sMB in the back of the Grid Director 4200. The front panel of the Management Board is shown in the following figure.
  • Page 60 3. Pull the ejector outwards and slowly slide the board out. Installing the Power Supply Unit (sPSU-S) Power Supply modules are installed in the rear of the Grid Director 4200. Ideally, you should connect each Power Supply module line cord to a different circuit for maximum fault tolerance in the event of power problems.
  • Page 61 Fan Units (sFUs) The Grid Director 4200 is pre-assembled at the factory with the Fan Units installed at the front of the chassis. The Fan Units are hot swappable modules.
  • Page 62 Installing System Boards The Grid Director 4200 hosts a Vertical Fan Unit (sFU-42V) and a Horizontal Fan Unit (sFU-42H). When replacing one of the 4200 Fan Units, the other takes over and changes to turbo mode, to ensure proper system ventilation.
  • Page 63 Grid Director 4200 Installation Manual includes eight fans with fan speed control; fan speed is dynamically controlled as a function of temperature. LED indications provide an onsite report on the operating conditions of the air cooling system modules, and provide an alarm indicator in the event of a high temperature condition.
  • Page 64 The rack mounted Cabling Guide Bracket should be installed with its bend facing out, otherwise it will prevent Line Boards from being inserted or removed. Make sure that the rack rails that overlap with the 11U of the Grid Director 4200 are free of PDU and power inlets.
  • Page 65 Grid Director 4200. It is not important from the Cable Management perspective where the Grid Director 4200 is installed in the rack. 2. Install the Cabling Guide Brackets in parallel with the Grid Director 4200 so that each Line Board lines up with its corresponding hanger on the Bracket.
  • Page 66 Tie wraps and zip ties can damage the cables. Velcro tie is the recommended option. Do not bend the InfiniBand cables too sharply. The recommended minimum bend radius is 4" (10 cm). The following figure shows the Grid Director 4200 cable dressing: Figure 21: Grid Director 4200 Cable Dressing InfiniBand Cabling Guidelines It is recommended to dress the cables connected to the InfiniBand connectors of each sLB into two groups, left and right, and fasten them down gently after connectivity has been verified.
  • Page 67 4. Replace the cable connections. Connecting to the Management Ports The Grid Director 4200 has management ports located at top back of the chassis. This includes two interfaces: a serial interface for local console access and a 10/100/1G Ethernet interface for network access.
  • Page 68 HyperTerminal or ProComm Plus. 3. Make sure that your PC/console terminal emulation software is configured to communicate with the Grid Director 4200 via hardware flow control. Configure the baud rate and character format of the PC terminal emulation program to match these RS-232 management port default characteristics: ...
  • Page 69 Operation Operation Overview Once the Grid Director 4200 and the boards are installed is secure within the rack, set up the electrical cables from the power distribution source in preparation for connection. Power cords are supplied by Voltaire according to the system configuration, one per power supply unit.
  • Page 70 10/100/1G Ethernet management. Where to Go Next Once you have verified that the Grid Director 4200 hardware is properly installed, it is ready for optional system configuration. To configure the system software, refer to Grid Director 4000 Family User Manual.
  • Page 71 2. Check the Power Supply LEDs on the rear panel. The Power LEDs illuminate immediately upon the connection of power to the Grid Director 4200. If the LEDs are not on, the power supplies may need to be replaced.
  • Page 72 Before you call, have the following information ready to help your service representative assist you as quickly as possible:  The date that you received the Grid Director 4200 and modules  The chassis serial number ...
  • Page 73 Grid Director 4200 Installation Manual Cabling Information and Specifications This appendix provides the QSFP cable specifications and the 1 GbE port pinouts. 11.1 QSFP Cable Key Features  Small size, high density, hot-pluggable connector with four transmit and four receive channels ...
  • Page 74 Cabling Information and Specifications 11.2 QSFP Cable Overview Every 4200 port on the switch is equipped with transmit and receive circuits that can adjust to fit different types of link characteristics and either optical or copper cables. The QSFP cables transmit and receive up to 10 Gbps per channel (total: up to 40 Gbps per port) over a single active/passive optical or copper cable, Passive copper cables do not consume any additional power while active copper and optic cables consume up to 2.5 Watt.