Mellanox Technologies MSX6512-NR Hardware Installation Manual
216 port infiniband fdr switch platform
Hide thumbs
Also See for MSX6512-NR:
- Hardware installation manual (118 pages) ,
- Hardware user manual (129 pages) ,
- Dismantling manual (14 pages)
Related Manuals for Mellanox Technologies MSX6512-NR
Summary of Contents for Mellanox Technologies MSX6512-NR
- Page 1 216 Port InfiniBand FDR Switch Platform Hardware Installation Guide PN: MSX6512-NR, MSX6512-4R Rev 1.0 www.mellanox.com...
- Page 2 THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT.
-
Page 3: Table Of Contents
10.3.1 Yellow Status LED (for the Chassis) on the Management Module is Lit 10.3.2 Yellow LED for the Leaf Fan on the Management Module is Lit 10.3.3 Yellow LED for the Spine Fan on the Management Module is Lit 10.4 Spine Board 10.5 MLNX-OS Software Mellanox Technologies... - Page 4 Figure 25: Top and Bottom Ports Figure 26: Spine Side Panel Display Status Indications Figure 27: Management Module Status Indications for Normal Operation Figure 28: Connecting the Harness to the Console Figure 29: Web UI Login Page Figure 30: Reset Button Mellanox Technologies...
- Page 5 Reference Documents and Web Sites Table 3: Serial Terminal Program Configuration Table 4: Configuration Wizard Session - IP Configuration by DHCP Table 5: Configuration Wizard Session - IP Zeroconf Configuration Table 6: Configuration Wizard Session - Static IP Configuration Mellanox Technologies...
-
Page 6: Table 1: Revision History Of This User's Manual
Rev 1.0 Revision History Table 1 - Revision History of this User’s Manual Revision Date Details July2010 Initial release Mellanox Technologies... -
Page 7: Table 2: Reference Documents And Web Sites
Mellanox MLNX-OS™ SwitchX® Software User This document contains information regarding configuring and Manual managing Mellanox Technologies' SwitchX Switch Platforms. MLNX-OS™ Software Command Reference Guide Command Reference Guide for MLNX-OS listing all of the commands available through MLNX-OS with explanations and examples. - Page 8 This symbol indicates information that is helpful to the user. This symbol indicates a situation that can potentially cause damage to hardware or software. WARNING! This symbol indicates a situation that can potentially cause personal injury and / or damage to hardware or software. Mellanox Technologies...
-
Page 9: Chapter 1 Chassis Installation
It is highly recommended that the installation site building be equipped with a light- ning rod. It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system. Mellanox Technologies... -
Page 10: Chassis Package Contents
Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box. The customer will receive as many boxes as needed to fill the order. Mellanox Technologies... -
Page 11: Spine Package Contents
Put an ESD prevention wrist strap on your wrist, and make sure there is good contact between your body and the strap. Step 2. Plug the other end of the wrist strap to a valid ground. Make sure that this is a tight fit. Mellanox Technologies... -
Page 12: Installation Procedure
The container has shock and tilt stickers applied. These will turn red if the container has been mis- handled or roughly handled. Upon receipt of the container look for and inspect the shock and tilt stickers to confirm that they have not tripped. If one or more are red notify the shipper and Mella- Mellanox Technologies... -
Page 13: Figure 1: Shock And Tilt Stickers
This on its own does not indicate damage to the contents. But, be sure to carefully inspect the contents if any of the shock and tilt stickers have tripped. Figure 1: Shock and Tilt Stickers WA CH Stickers unaffected by shipping HANDLE WITH CARE HANDLE WITH CARE WA CH Stickers showing excessive shock or tilt. HANDLE WITH CARE HANDLE WITH CARE Mellanox Technologies... -
Page 14: Figure 2: Shelf Installation Kit Parts
Rev 1.0 Chassis Installation Figure 2: Shelf Installation Kit Parts Shelf Angled Rail Bracket Rail Slide M6 X4 Caged nut M6 X4 bolt Mellanox Technologies... - Page 15 If you are not using a mechanical lift to install the chassis, reduce the weight of the chassis by removing all of the spines, power supply units, fan units, and management modules, and put aside for reinstallation after the chassis is installed in the rack. Mellanox Technologies...
-
Page 16: Figure 3: Inserting The Caged Nuts For The Shelf
Figure 4: Use These Holes for the Rail Slides Step 4. Using two of the bolts for each rail slide, install the rail slides onto the rack. Check that both sides of the switch, left and right, are the same level in the rack. Mellanox Technologies... -
Page 17: Figure 5: Connect Rail Slide To Rack Vertical Support
Step 9. Insert 8 caged nuts for the faceplate in the exact locations shown in Figure 6,“Inserting the Caged Nuts for the Faceplate”. Figure 6: Inserting the Caged Nuts for the Faceplate Place the caged nuts here. Shelf Mellanox Technologies... -
Page 18: Installing The Chassis
Figure 7: Chassis Rails and Rail Slides Rail Slide Step 1. Screw the handles onto the chassis. Use the 8 allen head screws. Two go on the face- plate and two go on the sides of the chassis. Mellanox Technologies... -
Page 19: Figure 8: Screw The Handles Onto The Chassis
Figure 9: The Rails are Already Connected Onto the Chassis When lifting manually use your legs, bend your knees, and not your back. <40 lbs 70 - 121 lbs >121 lbs 40 - 70 lbs <18 kgs 32 - 55 kgs >55 kgs 18 - 32 kgs Mellanox Technologies... -
Page 20: Figure 10: Chassis On The Shelf
Remove the two handles from the sides of the chassis. Step 4. Push the chassis into the rack until the faceplate is ~ 20cm (~ 8”) from the vertical sup- port. Figure 10: Chassis on the Shelf Step 5. Put the rail slides onto the rails. Mellanox Technologies... -
Page 21: Figure 11: Put On The Rail Slide
Holes in the faceplate for mounting on the rack. Mellanox ® Fan #1 Fan #2 Fan #3 Fan #4 SX6512 Step 8. Screw the 8 screws through the faceplate and into the caged nuts. Step 9. Slide the rail slides to the vertical rails on the leaf side. Mellanox Technologies... -
Page 22: Inserting The Leafs
It can take up to 5 minutes to boot up, during which time the status LED may indicate flashing green. Step 16. You can start connecting all of the cables to the switch. Inserting the Leafs The leaf boards can be installed in any of the available leaf slots. Mellanox Technologies... -
Page 23: Inserting A Management Module
Step 3. Press the ejectors inward until the locks snap. This indicates that the board is fully seated. Figure 15: Ejector Latch Management Module /Spine Management Module /Spine Push the latch release. Mellanox Technologies... -
Page 24: Inserting A Spine Board
Now press both latches inward simultaneously. You should feel the latches are hooked into the front panel. Step 6. Press both latches applying some force to close them completely. The locking latch will engage. Figure 17: Ejector Latch Management Module /Spine Management Module /Spine Push the latch release. Mellanox Technologies... -
Page 25: Chapter 2 Installing The Cable Holder
12 X M6 bolt for caged nut Left side cable holder Step 1. Place the Cable holder next to the rack, on the connector side of the chassis, and identify the holes where the caged nuts were placed. Mellanox Technologies... -
Page 26: Figure 19: Inserting The Caged Nuts For The Cable Holder
The cable holders should go to the outside of the vertical supports. Step 2. Set the bottom of the cable holder at the level of the shelf. If the cable holder is not set properly the upper bracket will not line up with the cable holder. Mellanox Technologies... -
Page 27: Figure 20: Cable Holder Placement
Notch for upper bracket washer Set this even with the top of the shelf Step 3. Screw the cable holder onto the rack using the screws provided. Step 4. Repeat steps 1- 3 for the second cable holder. Mellanox Technologies... -
Page 28: Chapter 3 Ground Connections
Test the ground using an Ohm meter. Figure 21: Ground Connection Mellanox ® Fan #2 Fan #3 Fan #4 Fan #1 SX6512 ESD Post Ground Post Mellanox Technologies... -
Page 29: Chapter 4 Power Connections
With N+N optional PSU grid redundancy the chassis can run on 1/2 of the full comple- ment of PSUs. N+N allows the chassis to run on 2 PSUs supplied from one power grid while 2 are connected to a second power grid. Mellanox Technologies... -
Page 30: Figure 22: Multiple Power Inlets - Electric Caution Notification
It can take up to 5 minutes to boot up the system. Turn off the system if any LEDs remain red for more than 5 minutes. Step 5. Check that the leaf status LEDs in the spines are lit. Figure 23: Spine Module Spine Leaf IB Connection Status Spine Status Fan Status Mellanox Technologies... -
Page 31: Chapter 5 Infiniband Qsfp Cable Installation
The ports on each leaf board are placed in two rows, 9 ports to a row. The ports are labelled as shown in Figure 24. The bottom row ports are flipped from the top row. See Figure 25. Mellanox Technologies... -
Page 32: Supported Approved Cables
Chassis and switches need to be able to dissipate the heat generated by high power I/O cables and modules. The Mellanox SX65xx chassis are rated for cables up to class 2 as per the SFF committee classification (SFF-8436.PDF). See http://www.mellanox.com/content/pages.php?pg=cables the cable class rating of Mellanox cables. Mellanox Technologies... -
Page 33: Chapter 6 Chassis Power Up
5. Tighten all leaf and spine mounting screws. 6. Connect the power cords to the PSUs. 7. Connect the power cords to grounded electrical outlets. With N+1 1000 W PSU redundancy, do not power up the chassis with less than all PSUs installed. Mellanox Technologies... -
Page 34: Power Supply, Management Board, And Spine Board Indicator Status At Power On
MNG 2 STATUS FANS FANS STATUS STATUS Step 5. Check the Spine LEDs and make sure they coincide with Figure 27. Figure 27: Management Module Status Indications for Normal Operation S.FANS L.FANS MASTER CONSOLE MGT MAC: STATUS 00:02:C9:11:34:8A Mellanox Technologies... -
Page 35: Chapter 7 Infiniband Fabric Initialization And Management
L.FANS MASTER CONSOLE STATUS No remote IP connection is available at this stage. Step 2. Configure a serial terminal program (for example, HyperTerminal, Minicom, or Tera Term) on your host PC with the settings described in Table 3. Mellanox Technologies... -
Page 36: Table 3: Serial Terminal Program Configuration
Step 3: Enable IPv6? [yes] yes The management interface will be able to use IPv6 addresses. Step 4: Enable IPv6 autoconfig (SLAAC) on mgmt0 This turns on autoconfiguration of the IPv6 addresses. interface? [no] no This is unsuitable for DHCPv6. Mellanox Technologies... - Page 37 Config mode. Otherwise hit <enter> to save changes and exit. Choice: <Enter> Configuration changes saved. To return to the wizard from the CLI, enter the “con- figuration jump-start” command from configuration mode. Launching CLI... <switch name> [> Mellanox Technologies...
-
Page 38: Table 5: Configuration Wizard Session - Ip Zeroconf Configuration
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI... switch-1 > Mellanox Technologies... -
Page 39: Table 6: Configuration Wizard Session - Static Ip Configuration
Before attempting a remote (for example, SSH) connection to the switch, check the eth0 interface configuration. Specifically, verify the existence of an IP address. To check the current eth0 configuration, enter the following com- mands: sx-43 [standalone: master] > enable sx-43 [standalone: master] # show interfaces mgmt0 Mellanox Technologies... -
Page 40: Rerunning The Wizard
Availability according to the steps in the MLNX-OS Software UM section “Box IP Centralized Location”. Rerunning the Wizard If you want to rerun the wizard run the following commands: sx-43 [standalone: master] > enable sx-43 [standalone: master]# configure terminal sx-43 [standalone: master]# configuration jump-start Mellanox Technologies... -
Page 41: Chapter 8 Mlnx-Os Management Software
(“the remote machine” henceforth) using a standard RJ-45 connec- tor. Step 2. Start a Web browse – Google Chrome, Internet Explorer 7.0 or Mozilla Firefox 3.0. Make sure the screen resolution is set to 1024*768 or higher. Mellanox Technologies... -
Page 42: Figure 29: Web Ui Login Page
<switch_IP_address> is the IP address of the switch or its DNS name. Step 4. You will receive the login window for remote management of the switch. Figure 29: Web UI Login Page Step 5. Use admin for both the login (Account) and the password. Mellanox Technologies... -
Page 43: Chapter 9 Resetting The Switch - Rst
When the button is held down for 15 seconds the management module is reset and the password is deleted. You will then be able to enter without a password and make a new password for the user admin. Mellanox Technologies... -
Page 44: Chapter 10 Troubleshooting
Make sure that the latest FW version is installed on both the HCA card and the switch. Step 4. If media adapters are used check that the all connections are good, tight, and secure. Step 5. Replace the cable. The Activity indication does not come on: Mellanox Technologies... -
Page 45: Management Module
Remove and reinstall the problematic fan unit. Make sure the mating connec- tor of the new unit is free of any dirt and/or obstacles. See Section 5.4 on page 117. Step 6. Replace the Leaf fan module. Replace defective leaf fan modules as soon as they are identified. Mellanox Technologies... -
Page 46: Yellow Led For The Spine Fan On The Management Module Is Lit
Reburn the FW and remove and reinstall the spine. Step 7. Replace the spine board. 10.5 MLNX-OS Software For a instructions concerning software updates see the Mellanox MLNX-OS™ SwitchX® Soft- ware WebUI User’s Manual or the Mellanox MLNX-OS™ SwitchX® Software User Manual. Mellanox Technologies... - Page 47 216 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0 For more information and documentation on the SX6512 Series go to: the Mellanox Website http://www.mellanox.com Mellanox Technologies...
-
Page 48: Mellanox Technologies
Rev 0.1 Mellanox Technologies Inc. 350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085 Tel: (408) 970-3400 Fax: (408) 970-3403 www.mellanox.com E-mail: info@mellanox.com Map and Directions Tech Support: 408-916-0055 (8:00AM to 5:00 PM - Pacfic) 216 Port InfiniBand FDR Switch Platform Installation Guide Mellanox Technologies, Ltd.