Related Manuals for Mellanox Technologies MIS5600Q-10DNC
Summary of Contents for Mellanox Technologies MIS5600Q-10DNC
-
Page 1: Installation Guide
648 Port InfiniBand QDR Switch Platform Installation Guide PN: MIS5600Q-10DNC Rev 2.1 www.mellanox.com... - Page 2 THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
-
Page 3: Table Of Contents
8.2 Installing Licenses 8.3 Getting and Inserting the License Key 8.4 FabricIT Management and Inspection License 8.5 Starting an SSH Connection to the Switch 8.6 Starting a WebUI Connection to the Switch Chapter 9 Resetting the Switch – RST Mellanox Technologies... - Page 4 Figure 32: Spine Module Figure 33: Top and Bottom Ports Figure 34: Spine Side Panel Display Status Indications Figure 35: Management Module Status Indications for Normal Operation Figure 36: Connecting the Harness to the Console Figure 37: License Key Entitlement Number Mellanox Technologies...
-
Page 5: Mellanox Technologies
648 Port InfiniBand QDR Switch Platform User Manual Rev 2.1 Figure 38: License Key Generation Form Figure 39: FabricIT Status/Licensing Figure 40: Web UI Login Page Figure 41: Reset Button Mellanox Technologies... -
Page 6: List Of Tables
Reference Documents and Web Sites Table 3: Serial Terminal Program Configuration Table 4: Configuration Wizard Session - IP Configuration by DHCP Table 5: Configuration Wizard Session - IP Zeroconf Configuration Table 6: Configuration Wizard Session - Static IP Configuration Mellanox Technologies... -
Page 7: Table 1: Revision History Of This Installation Guide
Added notes for Hotswap Added note for EFM upgrade Turned the file into a book with FP, TOC... Added getting the license section April 2010 Reworked the Installation kit April 2010 Power numbers Jan. 2010 Formatting Dec. 2009 Initial release Mellanox Technologies... -
Page 8: Table 2: Reference Documents And Web Sites
Switch Firmware and Firmware Update Tools http://www.mellanox.com/content/ pages.php?pg=management_tools&menu_section=34 Note that the Switch System described in this manual is based on ® Mellanox Technologies’ InfiniScale IV switch device. Talk to your Mellanox representative for information regarding FabricIT Enterprise Fabric Management licensing and implementation of the FabricIT Enterprise Fabric (EFM) Software User’s Manual... - Page 9 648 Port InfiniBand QDR Switch Platform User Manual Rev 2.1 This symbol indicates a situation that can potentially cause damage to hardware or software. BEWARE! This symbol indicates a situation that can potentially cause personal injury or damage to hardware or software. Mellanox Technologies...
-
Page 10: Chapter 1 Chassis Installation
It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference. It is highly recommended that the installation site building be equipped with a light- ning rod. Mellanox Technologies... -
Page 11: Chassis Package Contents
Check the parts for visible damages that may have occurred during shipping. The leafs, spines, and management modules are shipped separately. If anything is damaged or missing, contact your customer representative immediately. Mellanox Technologies... -
Page 12: Leaf Package Contents
Make sure that a fully populated rack including cables will have sufficient air flow for cooling. Choose a rack which is able to support the mechanical and environmental characteris- tics of a fully populated switch chassis. Mellanox Technologies... -
Page 13: 1.6.1 Starting With The Rack
The rack may need to be modified to accommodate the 5600 chassis. The distance between the vertical supports must be between 65 and 80 cm. Figure 1: Distance Between the Vertical Supports This edge of the shelf must be as close as possible to the inside of the door when closed. Mellanox Technologies... -
Page 14: 1.6.2 Esd Connection
• ground wire of sufficient length and gauge to properly ground the chassis • One 9/16 box wrench The installation will be much easier with a power screwdriver. It is recommended to use AWG6 or 4mm diameter wire for grounding purposes. Mellanox Technologies... - Page 15 - 4 flat washers for the shelf extension • Parts included in the cable management installation kit: - 1 cable management rack RH - 1 cable management rack LH - 16 caged nuts M6 - 16 M6 bolts Mellanox Technologies...
-
Page 16: Figure 2: Installation Kit Parts
Upper bracket offset 1.6.3.2 Opening the Container Before starting the procedure, put the ESD strap on and connect it to a valid ground. Remove the four sides of the container by unscrewing the phillips head screws holding the sides on. Mellanox Technologies... -
Page 17: Figure 3: Container Screws
Remove all protective plastic film from all sides and top of the chassis. The leafs, spines, and management modules are shipped separately. Do not remove any of the blocks or either of the lock-down bars at this time. Mellanox Technologies... - Page 18 Measure the distance between the vertical supports and adjust the shelf bracket according to this distance. This chassis can be installed a rack with a distance between the vertical sup- ports of from 65cm(25.6in.) to 80cm(31.5in.). Mellanox Technologies...
-
Page 19: Figure 4: Vertical Rack Supports Dimension
The holes must be closer to the top of the shelf. Move the shelf brackets on the shelf. Place the shelf bracket so that it measures slightly more than the measured distance between the vertical rack supports. Make Mellanox Technologies... -
Page 20: Figure 6: Placing The Shelf Bracket
Insert five caged nuts into the rack for each of the four shelf brackets (the 2 brackets that are adjustable and the 2 that are fixed). Make sure that the caged nuts are all at the same height on the vertical supports. Do not use the top hole in the bracket. Mellanox Technologies... -
Page 21: Figure 7: Caged Nut Locations For The Shelf Brackets
Put the Shelf into the rack and twist the shelf into place. Make sure the spine side of the shelf is closest to the rack doors. See Figure 8 to identify the spine side of the shelf. Install and tighten the 20 M6 bolts into the caged nuts. Mellanox Technologies... -
Page 22: Figure 8: Shelf Brackets
Place 14 caged nuts into the vertical rack support at the locations corresponding to the holes in the chassis faceplate. The holes in the faceplate start at 2U above the shelf and are spaced every 4U centered as you go up the vertical rack support. There are 7 holes on each side. Mellanox Technologies... -
Page 23: Figure 10: U Numbering Above Shelf
Figure 10: U Numbering Above Shelf The first cage nut on the spine side goes here. The next six caged nuts go every 4 U units up from here. Top of Shelf 1.6.3.4 Inserting the Chassis Put the container on the fork lift. Mellanox Technologies... -
Page 24: Figure 11: Putting The Chassis In The Rack
The chassis is screwed down to the pallet on both the spine side and the leaf side. Figure 12: Location for the Lock-down Bar Bottom of the chassis Screw on the two Upper Bracket offsets onto the chassis using six flat head screws for each offset. Mellanox Technologies... -
Page 25: Figure 13: Place The Upper Bracket Offset
Screw on the shelf extension using two bolts with flat washers. Figure 14: Attach the Shelf Extension If you are installing the cable management system, place 14 caged nuts on the other side of the rack in the vertical supports. Mellanox Technologies... -
Page 26: Figure 15: Place Caged Nuts
Raise the chassis so that it is ~5cm above the shelf in the rack. Raise the hinged extension on the container floor. Lower the chassis until the hinged extension is sitting on the shelf extension. Push the chassis from the container to the shelf. Mellanox Technologies... -
Page 27: Figure 16: Fork Lift
648 Port InfiniBand QDR Switch Platform User Manual Rev 2.1 Figure 16: Fork Lift Container floor Hinged extension Save the container pallet until you want to remove this chassis. It will be necessary for the removal and disposal procedures. Mellanox Technologies... -
Page 28: Figure 17: Face Plate Mounting Bolt Locations
Slide the chassis onto the shelf and determine the location of the upper brackets, on the verti- cal supports on the connector side of the chassis. Slide the chassis out just far enough to install the upper brackets. Mellanox Technologies... -
Page 29: Figure 18: Upper Brackets And Offset Location
Figure 19: Upper Bracket Installation This hole is 60cm up from the top of the shelf Screw the upper brackets onto the vertical supports and tighten in place. You will need the socket wrench with both extensions to tighten these bolts. Mellanox Technologies... -
Page 30: Figure 20: Offset To Upper Bracket Bolts
Place the front lock-down bar at the edge of the chassis and align the holes in the lock-down bar with the holes in the shelf. Screw down the lock-down bar. Figure 21: Lock-Down Bar These holes are to screw the bar into the Chassis These holes are to screw the bar into the shelf Mellanox Technologies... -
Page 31: Figure 22: Bolt And Washer Order
Remove the shelf extension and save for removal of the chassis. Connect a valid ground, See Section 3, “Ground Connections,” on page 38, for detailed instructions. Go to Section 6, “Chassis Power Up,” on page 43 to power up the switch system. Mellanox Technologies... -
Page 32: 1.6.4 Installing The Cable Holder
4 The cable holders should go to the outside of the vertical supports. Set the bottom of the cable holder at the level of the shelf. Screw the cable holder onto the rack using the screws provided. Mellanox Technologies... -
Page 33: Installing The Leafs
Carefully seat the management module into the slot and slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge. Press the ejectors inward until the locks snap. This indicates that the board is fully seated. Mellanox Technologies... -
Page 34: Inserting A Spine Board
Push the board in until the hooks on the latches are touching the vertical bar. Now press both latches inward simultaneously. You should feel the latches are hooked into the front panel. Press both latches applying some force to close them completely. The locking latch will engage. Mellanox Technologies... -
Page 35: Figure 27: Ejector Latch
Pull the module out of the chassis Spine Spine Push the latch release Figure 28: Spine Board Insertion Open the Ejector Latches Status 9 11 1 3 5 19 21 23 Fan Status 2 4 6 8 10 12 14 16 18 Mellanox Technologies... -
Page 36: Chapter 2 Installing The Cable Holder
Now is the time to install the cable holder. Figure 29: Cable Holders Place the cable holder next to the rack, on the connector side of the chassis, and identify the holes where the caged nutswere placed in step 4. Mellanox Technologies... - Page 37 The cable holders should go to the outside of the vertical supports. Set the bottom of the cable holder at the level of the shelf. Screw the cable holder onto the rack using the screws provided. Repeat steps 1- 3 for the second cable holder. Mellanox Technologies...
-
Page 38: Chapter 3 Ground Connections
If you choose to not use the ground screw, make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack. Test the ground using an Ohm meter. Figure 30: Ground Connection Grounding post Mellanox Technologies... -
Page 39: Chapter 4 Power Connections
Should one of the PSUs fail, the total current consumption will then be divided by the remaining working PSUs. When the failed PSU is hot swapped the new PSU will ramp up and pass its share of current, so that the total current is always divided by the number of working PSUs. Mellanox Technologies... -
Page 40: Figure 31: Multiple Power Inlets - Electric Caution Notification
Check that the leaf status LEDs in the spines are lit. Figure 32: Spine Module Fan status LEDs Status 1 3 5 9 11 19 21 23 Fan Status 2 4 6 8 10 12 14 16 18 Status LED Status LED Leaf connection LEDs Mellanox Technologies... -
Page 41: Chapter 5 Infiniband Qsfp Cable Installation
Cables in the bottom row of the leafs should be inserted up side down in relation to the how the cables are inserted in the top row of the leafs. Figure 33: Top and Bottom Ports Mellanox Technologies... -
Page 42: Cable Power Classes
Chassis and switches need to be able to dissipate the heat generated by high power I/O cables and modules. The Mellanox IS5X00 series chassis are rated for cables up to class 2 as per the SFF committee classification (SFF-8436.PDF). http://www.mellanox.com/content/ pages.php?pg=cables for the cable class rating of Mellanox cables. Mellanox Technologies... -
Page 43: Chapter 6 Chassis Power Up
A fully loaded IS5600 series switch system can draw 8 kW of power. Make sure that the outlets and circuits will not be overloaded. Spread out the load over at least two or three circuits, or a 3 phase supply. Mellanox Technologies... -
Page 44: Power Supply, Management Board, And Spine Board Indicator Status At Power On
LEAF MGT 1 MGT 2 STATUS FANS FANS STATUS STATUS Check the Spine LEDs and make sure they coincide with Table 35. Figure 35: Management Module Status Indications for Normal Operation S.FANS L.FANS MASTER CONSOLE MGT MAC: 00:02:C9:11:34:8A Mellanox Technologies... -
Page 45: Chapter 7 Infiniband Fabric Initialization And Management
Initial configuration must be done on all of the management modules. Connect the host PC RS232 connector to the CONSOLE port of the switch using the supplied cable. Figure 36: Connecting the Harness to the Console Connect the host PC to here S.FANS L.FANS MASTER CONSOLE 00:0 Mellanox Technologies... -
Page 46: Table 3: Serial Terminal Program Configuration
If you enter ‘yes’ (yes Zeroconf), the session will continue as shown in Table 5. If you enter ‘no’ (no Zeroconf), then you need to enter a static IP, and the session will continue as shown in Table 6. Mellanox Technologies... - Page 47 To run the command “configuration jump-start” you Choice: <Enter> must be in Config mode. Configuration changes saved. To return to the wizard from the CLI, enter the “con- figuration jump-start” command from configuration mode. Launching CLI... <switch name> [> Mellanox Technologies...
-
Page 48: Table 5: Configuration Wizard Session - Ip Zeroconf Configuration
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI... switch-1 > Mellanox Technologies... -
Page 49: Table 6: Configuration Wizard Session - Static Ip Configuration
Launching CLI... switch-1 > switch-112428> enable // Enter ‘enable’ mode of switch-112428# show interface eth0 // # indicates ‘enable’ mode The following is an example of the output: Interface eth0 state Admin up: Link up: Mellanox Technologies... - Page 50 TX collisions: switch-112428# Before attempting a remote (for example, SSH) connection to the switch, check the eth0 interface configuration. Specifically, verify the existence of an IP address. To check the cur- rent eth0 configuration, enter the following commands: Mellanox Technologies...
- Page 51 TX queue len: 1000 switch-1 > enable // Enter ‘enable’ mode of switch-1 # show interface eth0 // # indicates ‘enable’ mode The following is an example of the output: Interface eth0 state Admin up: Link up: IP address: 192.168.10.6 Mellanox Technologies...
-
Page 52: Rerunning The Wizard
On IS5X00 systems, configure the Box IP (BIP) centralized location for chassis High Avail- ability according to the steps in the FabricIT Software UM section “Box IP Centralized Location”. Rerunning the Wizard If you want to rerun the wizard run the following commands: sx-43 [standalone: master] > enable Mellanox Technologies... - Page 53 648 Port InfiniBand QDR Switch Platform User Manual Rev 2.1 sx-43 [standalone: master]# configure terminal sx-43 [standalone: master]# configuration jump-start Mellanox Technologies...
-
Page 54: Chapter 8 Fabric Management Software
For the serial number location on the chassis see Section 1.1.1, “Serial Number and Product Version Information,” on page 11. Go to http://license.mellanox.com. Provide the Entitlement Number, provided on purchase of FabricIT Enterprise Fabric Man- ager. Mellanox Technologies... -
Page 55: Figure 37: License Key Entitlement Number
Provide the purchased Switch Serial Number (Switch S/N). Figure 38: License Key Generation Form Enter the purchased Switch Serial Number and generate the License Key. Print the generated license Key (PDF document is generated). Go to the FabricIT page and click on Setup =>Licensing. Mellanox Technologies... -
Page 56: Figure 39: Fabricit Status/Licensing
Enter this Key in the FabricIT GUI or using FabricIT CLI. Click “Add Licenses”. EFM is enabled. On purchase of FabricIT EFM, every switch is shipped with an “Entitlement Certificate”. License Key can be obtained using “CLI” or Web UI”. Print up the certificate with all the information. Mellanox Technologies... -
Page 57: Fabricit Management And Inspection License
Starting a WebUI Connection to the Switch Set up an Ethernet connection between the switch and a local network machine (“the remote machine” henceforth) using a standard RJ-45 connector. Start a Web browse – Google Chrome, Internet Explorer 7.0 or Mozilla Firefox 3.0. Mellanox Technologies... -
Page 58: Figure 40: Web Ui Login Page
<switch_IP_address> is the IP address of the switch or its DNS name. You will receive the login window for remote management of the switch. Figure 40: Web UI Login Page Use admin for both the login (Account) and the password. Mellanox Technologies... -
Page 59: Chapter 9 Resetting The Switch - Rst
When the button is held down for 15 seconds the management module is reset and the password is deleted. You will then be able to enter without a password and make a new password for the user admin. For more information and documentation on the IS5600 Series go to: http://www.mellanox.comhttp://www.mellanox.com/content/ pages.php?pg=products_dyn&product_family=59&menu_section=49 Mellanox Technologies... - Page 60 Rev 2.1 Mellanox Technologies Inc. 350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085 Tel: (408) 970-3400 Fax: (408) 970-3403 www.mellanox.com E-mail: info@mellanox.com Map and Directions Tech Support: 408-916-0055 (8:00AM to 5:00 PM - Pacfic) 648 Port InfiniBand QDR Switch Platform Installation Guide Mellanox Technologies, Ltd.