Mellanox Technologies MSX6536-10R Hardware Installation Manual
648 port infiniband fdr switch platform
Hide thumbs
Also See for MSX6536-10R:
- Hardware installation manual (126 pages) ,
- Dismantling manual (14 pages)
Related Manuals for Mellanox Technologies MSX6536-10R
Summary of Contents for Mellanox Technologies MSX6536-10R
- Page 1 648 Port InfiniBand FDR Switch Platform Hardware Installation Guide PN: MSX6536-10R, MSX6536-NR Rev 1.4 www.mellanox.com...
- Page 2 KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.
-
Page 3: Table Of Contents
7.5 MLNX-OS Software ........51 Mellanox Technologies... - Page 4 Figure 38: Reset Button ........... 47 Mellanox Technologies...
- Page 5 Configuration Wizard Session - IP Zeroconf Configuration ....42 Table 6: Configuration Wizard Session - Static IP Configuration ....43 Mellanox Technologies...
-
Page 6: Table 1: Revision History Of This Installation Guide
This manual is intended for users and system administrators responsible for installing and setting up the switch platform. ® The manual assumes familiarity with the InfiniBand architecture specification. Related Documentation The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the following: Mellanox Technologies... -
Page 7: Table 2: Reference Documents And Web Sites
Mellanox MLNX-OS™ SwitchX® Software User This document contains information regarding configuring and Manual managing Mellanox Technologies' SwitchX® Switch Platforms. MLNX-OS™ Software Command Reference Guide Command Reference Guide for MLNX-OS listing all of the commands available through MLNX-OS with explanations and examples. -
Page 8: Chapter 1 Installation
It is highly recommended that the installation site building be equipped with a light- ning rod. It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system. Mellanox Technologies... -
Page 9: Chassis Package Contents
1.3 Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box. The customer will receive as many boxes as needed to fill the order. Mellanox Technologies... -
Page 10: Spine Package Contents
Make sure that a fully populated rack including cables will have sufficient air flow for cooling. Choose a rack which is able to support the mechanical and environmental characteris- tics of a fully populated switch chassis. Mellanox Technologies... -
Page 11: Starting With The Rack
6 cm from the inside of the door. This is required so that the shelf will not hit the door. If you move the vertical supports make sure that you reinstall them vertically plumb. 1.6.2 ESD Connection Before starting any procedure on the SX6536 switch system: Mellanox Technologies... -
Page 12: Installation Procedure
- 12 lock washers for the upper brackets to vertical support upper bracket offsets - 12 hex head bolts – 8 for the upper - 34 caged nuts 20 for the shelf and 14 for the brackets and 4 for the shelf extension chassis faceplate Mellanox Technologies... -
Page 13: Figure 2: Installation Kit Parts
The container has shock and tilt stickers applied. These will turn red if the container has been mishandled or roughly handled. Upon receipt of the container look for and inspect the shock and tilt stickers to confirm that they have not tripped. If one or more are red Mellanox Technologies... -
Page 14: Figure 3: Shock And Tilt Stickers
Stickers unaffected by shipping HANDLE WITH CARE HANDLE WITH CARE WA CH Stickers showing excessive shock or tilt. HANDLE WITH CARE HANDLE WITH CARE 1.6.3.4 Opening the Container Step 1. Before starting the procedure, put the ESD strap on and connect it to a valid ground. Mellanox Technologies... -
Page 15: Figure 4: Container Screws
• 10 PSUs are installed for the chassis • all 8 fans are installed Step 4. Remove all protective plastic film from all sides and top of the chassis. The leafs, spines, and management modules are shipped separately. Mellanox Technologies... - Page 16 The side of the chassis with the spine units will sit flush with the vertical supports of the rack. The side of the chassis with the QSFP connectors will sit at the edge of the shelf closer to the center of the rack. Mellanox Technologies...
-
Page 17: Figure 5: Vertical Rack Supports Dimension
These two brackets will need to be adjusted. They may even need to be removed and repositioned. The holes in the bracket are off- set to one side. The holes must be closer to the top of the shelf. Mellanox Technologies... -
Page 18: Figure 7: Placing The Shelf Bracket
Do not tighten the hex head bolts. You may need to remove the brackets and reinstall them. Figure 7: Placing the Shelf Bracket Adjusting slots Figure 8: Caged Nut Locations for the Shelf Brackets Mellanox Technologies... -
Page 19: Figure 9: Shelf Brackets
Install and tighten the 20 M6 bolts into the caged nuts. Figure 9: Shelf Brackets The spine side of the shelf has two slots for the extension Adjustable shelf brackets Step 9. Tighten the 16 hex head bolts on the adjustable brackets. Mellanox Technologies... -
Page 20: Figure 10: U Numbering Above Shelf
The chassis is on ball bearings and can roll easily. Be aware that the chassis can roll off of the pallet. Should the chassis start to fall, move out of the way. The chassis can cause grave bodily harm should it fall on or near you. Step 1. Put the container on the fork lift. Mellanox Technologies... -
Page 21: Figure 11: Putting The Chassis In The Rack Using A Fork Lift
648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4 Figure 11: Putting the Chassis in the Rack Using a Fork Lift The chassis is screwed down to the pallet on both the spine side and the leaf side. Mellanox Technologies... -
Page 22: Figure 12: Putting The Chassis In The Rack Using A Mechanical Lift
Installation Figure 12: Putting the Chassis in the Rack Using a Mechanical Lift The chassis is screwed down to the pallet on both the spine side and the leaf side. Mellanox Technologies... -
Page 23: Figure 13: Place The Upper Bracket Offset
Step 2. Screw on the two Upper Bracket offsets onto the chassis using six flat head screws for each offset. Figure 14: Attach the Shelf Extension Step 3. Screw on the shelf extension using four bolts with flat washers. Mellanox Technologies... -
Page 24: Figure 15: Place Caged Nuts
Figure 16: Location for the Lock-down Bar Bottom of the chassis Step 5. Unscrew both lock-down bars. Step 6. Decide which side of the rack you want the data connectors. Mellanox Technologies... -
Page 25: Figure 17: Placement Of Chassis In Rack
Step 7. Place the chassis in front of the rack and as close as possible to the rack. Figure 18: Fork Lift Container floor Hinged extension Mellanox Technologies... -
Page 26: Figure 19: Mechanical Lift
Lower the chassis until the hinged extension is sitting on the shelf extension. Step 11. Push the chassis from the container to the shelf. Save the container pallet to remove this chassis. It will be necessary for the removal and disposal procedures. Mellanox Technologies... -
Page 27: Figure 20: Face Plate Mounting Bolt Locations
Do not move the forklift away from the rack just yet. Step 13. Confirm that the caged nuts for the faceplate are in the correct location. Mellanox Technologies... -
Page 28: Figure 21: Upper Bracket Installation
Slide the chassis onto the shelf until the face plate of the chassis touches the vertical rack support. Step 17. Make sure the upper brackets are installed in their correct location. Step 18. Screw the 14 bolts through the faceplate into the caged nuts, DO NOT TIGHTEN. Mellanox Technologies... -
Page 29: Figure 23: Offset To Upper Bracket Bolts
DO NOT TIGHTEN. Figure 24: Location for the Lock-down Bar Bottom of the chassis Figure 25: Lock-Down Bar These holes are to screw the bar into the Chassis These holes are to screw the bar into the shelf Mellanox Technologies... -
Page 30: Installing The Cable Holder
Tighten the lock down bars. See Figure 26. Step 27. Go to Section 3, “Chassis Power Up,” on page 37 to power up the switch system. 1.6.4 Installing the Cable Holder Now is the time to install the cable holder. Mellanox Technologies... -
Page 31: Figure 27: Cable Holder
If the cable holder is not set properly the upper bracket will not line up with the cable holder. Step 3. Screw the cable holder onto the rack using the screws provided. Step 4. Repeat steps 1- 3 for the second cable holder. Mellanox Technologies... -
Page 32: Ground Connections
Test the ground using an Ohm meter. Figure 29: Ground Connection Grounding post Mellanox Technologies... -
Page 33: Power Connections
With N+N optional PSU grid redundancy the chassis can run on 1/2 of the full comple- ment of PSUs. N+N allows the chassis to run on 5 PSUs supplied from one power grid while 5 are connected to a second power grid. Mellanox Technologies... -
Page 34: Figure 30: Multiple Power Inlets - Electric Caution Notification
It can take up to 5 minutes to boot up the system. Turn off the system if any LEDs remain red for more than 5 minutes. Step 5. Check that the leaf status LEDs in the spines are lit. Figure 31: Spine Module Spine Leaf IB Connection Status Spine Status Fan Status Mellanox Technologies... -
Page 35: Chapter 2 Infiniband Qsfp Cable Installation
The ports on each leaf board are placed in two rows, 9 ports to a row. The ports are labelled as shown in Figure 32. The bottom row ports are flipped from the top row. See Figure 33. Mellanox Technologies... -
Page 36: Supported Approved Cables
Chassis and switches need to be able to dissipate the heat generated by high power I/O cables and modules. The Mellanox SX65xx chassis are rated for cables up to class 2 as per the SFF committee classification (SFF-8436.PDF). http://www.mellanox.com/content/pages.php?pg=cables for the cable class rating of Mella- nox cables. Mellanox Technologies... -
Page 37: Chapter 3 Chassis Power Up
Tighten all leaf and spine mounting screws. Step 6. Connect the power cords to the PSUs. Step 7. Connect the power cords to grounded electrical outlets. With N+1 1000 W PSU redundancy, do not power up the chassis with less than all PSUs installed. Mellanox Technologies... -
Page 38: Power Supply, Management Board, And Spine Board Indicator Status At Power On
MNG 2 STATUS FANS FANS STATUS STATUS Step 5. Check the Spine LEDs and make sure they coincide with Figure 35. Figure 35: Management Module Status Indications for Normal Operation S.FANS L.FANS MASTER CONSOLE MGT MAC: STATUS 00:02:C9:11:34:8A Mellanox Technologies... -
Page 39: Chapter 4 Infiniband Fabric Initialization And Management
L.FANS MASTER CONSOLE STATUS No remote IP connection is available at this stage. Step 2. Configure a serial terminal program (for example, HyperTerminal, Minicom, or Tera Term) on your host PC with the settings described in Table 3. Mellanox Technologies... -
Page 40: Table 3: Serial Terminal Program Configuration
Step 3: Enable IPv6? [yes] yes The management interface will be able to use IPv6 addresses. Step 4: Enable IPv6 autoconfig (SLAAC) on mgmt0 This turns on autoconfiguration of the IPv6 addresses. interface? [no] no This is unsuitable for DHCPv6. Mellanox Technologies... - Page 41 Config mode. Otherwise hit <enter> to save changes and exit. Choice: <Enter> Configuration changes saved. To return to the wizard from the CLI, enter the “con- figuration jump-start” command from configuration mode. Launching CLI... <switch name> [> Mellanox Technologies...
-
Page 42: Table 5: Configuration Wizard Session - Ip Zeroconf Configuration
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI... switch-1 > Mellanox Technologies... -
Page 43: Table 6: Configuration Wizard Session - Static Ip Configuration
Before attempting a remote (for example, SSH) connection to the switch, check the eth0 interface configuration. Specifically, verify the existence of an IP address. To check the current eth0 configuration, enter the following com- mands: sx-43 [standalone: master] > enable sx-43 [standalone: master] # show interfaces mgmt0 Mellanox Technologies... -
Page 44: Rerunning The Wizard
Availability according to the steps in the MLNX-OS Software UM section “Box IP Centralized Location”. Rerunning the Wizard If you want to rerun the wizard run the following commands: sx-43 [standalone: master] > enable sx-43 [standalone: master]# configure terminal sx-43 [standalone: master]# configuration jump-start Mellanox Technologies... -
Page 45: Chapter 5 Mlnx-Os Management Software
(“the remote machine” henceforth) using a standard RJ-45 connec- tor. Step 2. Start a Web browse – Google Chrome, Internet Explorer 7.0 or Mozilla Firefox 3.0. Make sure the screen resolution is set to 1024*768 or higher. Mellanox Technologies... -
Page 46: Figure 37: Web Ui Login Page
<switch_IP_address> is the IP address of the switch or its DNS name. Step 4. You will receive the login window for remote management of the switch. Figure 37: Web UI Login Page Step 5. Use admin for both the login (Account) and the password. Mellanox Technologies... -
Page 47: Chapter 6 Resetting The Switch - Rst
When the button is held down for 15 seconds the management module is reset and the password is deleted. You will then be able to enter without a password and make a new password for the user admin. Mellanox Technologies... -
Page 48: Chapter 7 Troubleshooting
Make sure that the latest FW version is installed on both the HCA card and the switch. Step 4. If media adapters are used check that the all connections are good, tight, and secure. Step 5. Replace the cable. The Activity indication does not come on: Mellanox Technologies... -
Page 49: Management Module
See Section 5.4 on page Step 6. Replace the Leaf fan module. There are two non-interchangeable types of leaf fan modules. If the new leaf fan does not go into the chassis confirm that it is the correct fan module. Mellanox Technologies... -
Page 50: Yellow Led For The Spine Fan On The Management Module Is Lit
See Section 5.3 on page 69. Step 5. Make sure that the spine and the Leafs both have the same version of FW. Step 6. Reburn the FW and remove and reinstall the spine. Step 7. Replace the spine board. Mellanox Technologies... -
Page 51: Mlnx-Os Software
For a instructions concerning software updates see the Mellanox MLNX-OS™ SwitchX® Software WebUI User’s Manual or the Mellanox MLNX-OS™ SwitchX® Software User Manual. For more information and documentation on the SX6536 Series go to: the Mellanox Website http://www.mellanox.com Mellanox Technologies... - Page 52 Rev 1.4 Mellanox Technologies Inc. 350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085 Tel: (408) 970-3400 Fax: (408) 970-3403 www.mellanox.com E-mail: info@mellanox.com Map and Directions Tech Support: 408-916-0055 (8:00AM to 5:00 PM - Pacfic) 648 Port InfiniBand FDR Switch Platform Installation Guide Mellanox Technologies, Ltd.