Related Manuals for Mellanox Technologies MSX6518-6R
Summary of Contents for Mellanox Technologies MSX6518-6R
- Page 1 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide PN: MSX6518-6R, MSX6518-NR Rev 1.0 www.mellanox.com...
- Page 2 THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
-
Page 3: Table Of Contents
10.3.1 Yellow Status LED (for the Chassis) on the Management Module is Lit 10.3.2 Yellow LED for the Leaf Fan on the Management Module is Lit 10.3.3 Yellow LED for the Spine Fan on the Management Module is Lit 10.4 Spine Board 10.5 MLNX-OS Software Mellanox Technologies... - Page 4 Figure 32: Spine Module Figure 33: Port Numbering Figure 34: Top and Bottom Ports Figure 35: Spine Side Panel Display Status Indications Figure 36: Management Module Status Indications for Normal Operation Figure 37: Connecting the Harness to the Console Mellanox Technologies...
- Page 5 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0 Figure 38: Web UI Login Page Figure 39: Reset Button Mellanox Technologies...
- Page 6 Reference Documents and Web Sites Table 3: Serial Terminal Program Configuration Table 4: Configuration Wizard Session - IP Configuration by DHCP Table 5: Configuration Wizard Session - IP Zeroconf Configuration Table 6: Configuration Wizard Session - Static IP Configuration Mellanox Technologies...
-
Page 7: Table 1: Revision History Of This Installation Guide
324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0 Revision History Table 1 - Revision History of this Installation Guide Revision Date Details March 2012 Initial release Mellanox Technologies... -
Page 8: Table 2: Reference Documents And Web Sites
Mellanox MLNX-OS™ SwitchX® Software User This document contains information regarding configuring and Manual managing Mellanox Technologies' SwitchX Switch Platforms. MLNX-OS™ Software Command Reference Guide Command Reference Guide for MLNX-OS listing all of the commands available through MLNX-OS with explanations and examples. - Page 9 This symbol indicates information that is helpful to the user. This symbol indicates a situation that can potentially cause damage to hardware or software. WARNING! This symbol indicates a situation that can potentially cause personal injury and / or damage to hardware or software. Mellanox Technologies...
-
Page 10: Chapter 1 Chassis Installation
It is highly recommended that the installation site building be equipped with a light- ning rod. It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system. Mellanox Technologies... -
Page 11: Chassis Package Contents
8cm (3”) of space to the front and rear of the switch. This will ensure proper air flow through the chassis. This is crucial for maintaining good airflow at ambient temperature. In particular, route cables such that they do not impede the air into or out of the chassis. Mellanox Technologies... -
Page 12: Leaf Package Contents
Choose a rack which is able to support the mechanical and environmental characteris- tics of a fully populated switch chassis. 1.6.1 Starting with the Rack The rack may need to be modified to accommodate the SX6518 chassis. The distance between the vertical supports must be between 65 and 80 cm. Mellanox Technologies... -
Page 13: Esd (Electrostatic Discharge) Connection
Plug the other end of the wrist strap to a valid ground. Make sure that this is a tight fit. 1.6.3 Installation Procedure Installing the chassis will be a lot easier if you can remove the sides of the rack. Mellanox Technologies... - Page 14 - 12 lock washers for the upper brackets to vertical support upper bracket offsets - hex head bolts – 8 for the upper - 20 caged nuts 12 for the shelf and 8 for the brackets and for the shelf extension chassis faceplate Mellanox Technologies...
-
Page 15: Figure 2: Installation Kit Parts
- 10 M6 bolts Figure 2: Installation Kit Parts Lock-down bar Shelf Shelf extension Upper bracket Upper bracket washer Upper bracket offset Flat washers Phillips pan head hex head bolt Flat head screw Caged nuts Lock washers screw Mellanox Technologies... -
Page 16: Figure 3: Shock And Tilt Stickers
Check the shock and tilt stickers on the container to see if it was subjected to excessive shock or tilting. If any of these stickers are tripped contact your Mellanox representa- tive. Step 2. Before starting the procedure, put the ESD strap on and connect it to a valid ground. Mellanox Technologies... -
Page 17: Figure 4: Container Screws
Do not remove any of the blocks or either of the lock-down bars at this time. 1.6.3.5 Installing the Shelf Step 1. Select the location in the rack. Try to keep the chassis as low as possible to keep the center of gravity low. Mellanox Technologies... -
Page 18: Figure 5: Vertical Rack Supports Dimension
The side of the chassis with the spine units will sit flush with the vertical supports of the rack. The side of the chassis with the QSFP connectors will sit at the edge of the shelf closer to the center of the rack. Figure 5: Vertical Rack Supports Dimension Vertical Supports Mellanox Technologies... -
Page 19: Figure 6: Adjustable Shelf Brackets
Top of shelf Bottom of shelf is even with this side of the bracket, the greater distance between the holes and the edge of the bracket show that this is the bottom side. Mellanox Technologies... -
Page 20: Figure 8: Placing The Shelf Bracket
Put the Shelf into the rack and twist the shelf into place. Make sure the spine side of the shelf is closest to the rack doors. See Figure 10 to identify the spine side of the shelf. Step 7. Install and tighten the 12 M6 bolts into the caged nuts. Mellanox Technologies... -
Page 21: Figure 10: Shelf Brackets
The holes in the faceplate start at 2U above the shelf and are spaced every 4U centered as you go up the vertical rack support. There are 4 holes on each side. Mellanox Technologies... -
Page 22: Figure 12: Putting The Chassis In The Rack
Step 1. Put the container on the fork lift. Figure 12: Putting the Chassis in the Rack The chassis is screwed down to the pallet on both the spine side and the leaf side. Mellanox Technologies... -
Page 23: Figure 13: Place The Upper Bracket Offset
Step 3. Screw on the shelf extension using two bolts with flat washers. Step 4. If you are installing the cable management system, place 10 caged nuts on the other side of the rack in the vertical supports. Mellanox Technologies... -
Page 24: Figure 15: Location For The Lock-Down Bar
Unscrew the lock-down bars. Step 6. Place caged nuts in all of the locations specified in Figure 16. The black holes show where to place caged nuts; the red holes show the location of the upper brackets on the leaf side. Mellanox Technologies... -
Page 25: Figure 16: Caged Nut Placement By U Numbers
Figure 16: Caged Nut Placement by U Numbers These are for the cable holder. Place 5 caged nuts here. Spine side Leaf Side Step 7. Place the chassis in front of the rack and as close as possible to the rack. Mellanox Technologies... -
Page 26: Figure 17: Fork Lift
Figure 18: Face Plate Mounting Bolt Locations SX6518 STATUS Holes in the faceplate for mount- SPINE FANS LEAF FANS MNG1 MASTER MNG2 MASTER Mellanox ing on the rack. ® SX6518 SX6518 The holes are separated by 4U Mellanox Mellanox units. Mellanox Technologies... -
Page 27: Figure 19: Upper Bracket Installation
Confirm that the caged nuts for the faceplate are in the correct location. Step 14. Slide the chassis back out, far enough to install the upper brackets. Figure 19: Upper Bracket Installation This hole is 60cm up from the top of the shelf. Mellanox Technologies... -
Page 28: Figure 20: Upper Brackets And Offset Location
Make sure the upper brackets are installed in their correct location. Step 18. Screw the 8 bolts through the faceplate into the caged nuts, DO NOT TIGHTEN. Figure 21: Offset to Upper bracket Bolts Step 19. Now you can remove the fork lift and pallet. Mellanox Technologies... -
Page 29: Figure 22: Location For The Lock-Down Bar
10 hex head screws and washers for each lock down bar. Step 22. Catch the screws for the second lock-down bar but do not tighten them yet. You may have to wiggle the chassis to catch the screws in the threaded holes. Mellanox Technologies... -
Page 30: Installing The Leafs
The leaf boards can be installed in any of the available leaf slots. Figure 25: Ejector Latch Leaf Leaf Push the latch release. Step 1. Start with the latch fully open, that is 45 to the front panel of the leaf. Mellanox Technologies... -
Page 31: Inserting A Management Module
Push the board in until the hooks on the latches are touching the vertical bar. Step 5. Now press both latches inward simultaneously. You should feel the latches are hooked into the front panel. Step 6. Press both latches applying some force to close them completely. The locking latch will engage. Mellanox Technologies... -
Page 32: Figure 28: Ejector Latch
Rev 1.0 Chassis Installation Figure 28: Ejector Latch Management Module /Spine Management Module /Spine Push the latch release. Mellanox Technologies... -
Page 33: Chapter 2 Installing The Cable Holder
Step 2. Set the bottom of the cable holder at the level of the shelf. Step 3. Screw the cable holder onto the rack using the screws provided. Step 4. Repeat steps 1- 3 for the second cable holder. Mellanox Technologies... -
Page 34: Chapter 3 Ground Connections
Test the ground using an Ohm meter. Figure 30: Ground Connection Mellanox Mellanox Ground Post ESD Post Mellanox Technologies... -
Page 35: Chapter 4 Power Connections
With N+N optional PSU grid redundancy the chassis can run on 1/2 of the full comple- ment of PSUs. N+N allows the chassis to run on 3 PSUs supplied from one power grid while 3 are connected to a second power grid. Mellanox Technologies... -
Page 36: Figure 31: Multiple Power Inlets - Electric Caution Notification
It can take up to 5 minutes to boot up the system. Turn off the system if any LEDs remain red for more than 5 minutes. Step 5. Check that the leaf status LEDs in the spines are lit. Figure 32: Spine Module Spine Leaf IB Connection Status Spine Status Fan Status Mellanox Technologies... -
Page 37: Chapter 5 Infiniband Qsfp Cable Installation
The ports on each leaf board are placed in two rows, 9 ports to a row. The ports are labelled as shown in Figure 33. The bottom row ports are flipped from the top row. See Figure 34. Mellanox Technologies... -
Page 38: Supported Approved Cables
Chassis and switches need to be able to dissipate the heat generated by high power I/O cables and modules. The Mellanox SX65xx chassis are rated for cables up to class 2 as per the SFF committee classification (SFF-8436.PDF). See http://www.mellanox.com/content/pages.php?pg=cables the cable class rating of Mellanox cables. Mellanox Technologies... -
Page 39: Chapter 6 Chassis Power Up
5. Tighten all leaf and spine mounting screws. 6. Connect the power cords to the PSUs. 7. Connect the power cords to grounded electrical outlets. With N+1 1000 W PSU redundancy, do not power up the chassis with less than all PSUs installed. Mellanox Technologies... -
Page 40: Power Supply, Management Board, And Spine Board Indicator Status At Power On
MNG 2 STATUS FANS FANS STATUS STATUS Step 5. Check the Spine LEDs and make sure they coincide with Figure 36. Figure 36: Management Module Status Indications for Normal Operation S.FANS L.FANS MASTER CONSOLE MGT MAC: STATUS 00:02:C9:11:34:8A Mellanox Technologies... -
Page 41: Chapter 7 Infiniband Fabric Initialization And Management
L.FANS MASTER CONSOLE STATUS No remote IP connection is available at this stage. Step 2. Configure a serial terminal program (for example, HyperTerminal, Minicom, or Tera Term) on your host PC with the settings described in Table 3. Mellanox Technologies... -
Page 42: Table 3: Serial Terminal Program Configuration
Step 3: Enable IPv6? [yes] yes The management interface will be able to use IPv6 addresses. Step 4: Enable IPv6 autoconfig (SLAAC) on mgmt0 This turns on autoconfiguration of the IPv6 addresses. interface? [no] no This is unsuitable for DHCPv6. Mellanox Technologies... - Page 43 Config mode. Otherwise hit <enter> to save changes and exit. Choice: <Enter> Configuration changes saved. To return to the wizard from the CLI, enter the “con- figuration jump-start” command from configuration mode. Launching CLI... <switch name> [> Mellanox Technologies...
-
Page 44: Table 5: Configuration Wizard Session - Ip Zeroconf Configuration
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI... switch-1 > Mellanox Technologies... -
Page 45: Table 6: Configuration Wizard Session - Static Ip Configuration
Before attempting a remote (for example, SSH) connection to the switch, check the eth0 interface configuration. Specifically, verify the existence of an IP address. To check the current eth0 configuration, enter the following com- mands: sx-43 [standalone: master] > enable sx-43 [standalone: master] # show interfaces mgmt0 Mellanox Technologies... -
Page 46: Rerunning The Wizard
Availability according to the steps in the MLNX-OS Software UM section “Box IP Centralized Location”. Rerunning the Wizard If you want to rerun the wizard run the following commands: sx-43 [standalone: master] > enable sx-43 [standalone: master]# configure terminal sx-43 [standalone: master]# configuration jump-start Mellanox Technologies... -
Page 47: Chapter 8 Mlnx-Os Management Software
(“the remote machine” henceforth) using a standard RJ-45 connec- tor. Step 2. Start a Web browse – Google Chrome, Internet Explorer 7.0 or Mozilla Firefox 3.0. Make sure the screen resolution is set to 1024*768 or higher. Mellanox Technologies... - Page 48 <switch_IP_address> is the IP address of the switch or its DNS name. Step 4. You will receive the login window for remote management of the switch. Figure 38: Web UI Login Page Step 5. Use admin for both the login (Account) and the password. Mellanox Technologies...
-
Page 49: Chapter 9 Resetting The Switch - Rst
When the button is held down for 15 seconds the management module is reset and the password is deleted. You will then be able to enter without a password and make a new password for the user admin. Mellanox Technologies... -
Page 50: Chapter 10 Troubleshooting
Make sure that the latest FW version is installed on both the HCA card and the switch. Step 4. If media adapters are used check that the all connections are good, tight, and secure. Step 5. Replace the cable. The Activity indication does not come on: Mellanox Technologies... -
Page 51: Management Module
See Section 5.4 on page Step 6. Replace the Leaf fan module. There are two non-interchangeable types of leaf fan modules. If the new leaf fan does not go into the chassis confirm that it is the correct fan module. Mellanox Technologies... -
Page 52: Yellow Led For The Spine Fan On The Management Module Is Lit
See Section 5.3 on page 68. Step 5. Make sure that the spine and the Leafs both have the same version of FW. Step 6. Reburn the FW and remove and reinstall the spine. Step 7. Replace the spine board. Mellanox Technologies... -
Page 53: Mlnx-Os Software
For a instructions concerning software updates see the Mellanox MLNX-OS™ SwitchX® Software WebUI User’s Manual or the Mellanox MLNX-OS™ SwitchX® Software User Manual. For more information and documentation on the SX6518 Series go to: the Mellanox Website http://www.mellanox.com Mellanox Technologies... - Page 54 Rev 1.0 Mellanox Technologies Inc. 350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085 Tel: (408) 970-3400 Fax: (408) 970-3403 www.mellanox.com E-mail: info@mellanox.com Map and Directions Tech Support: 408-916-0055 (8:00AM to 5:00 PM - Pacfic) 324 Port InfiniBand FDR Switch Platform Installation Guide Mellanox Technologies, Ltd.