Subscribe to Our Youtube Channel
Related Manuals for Sun Oracle Sun Datacenter InfiniBand Switch 36
Summary of Contents for Sun Oracle Sun Datacenter InfiniBand Switch 36
- Page 1 Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 Part No.: E36268-01 March 2013, Revision A...
- Page 2 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means.
-
Page 3: Table Of Contents
Contents Using This Documentation vii Understanding the Switch 1 Switch Overview 1 Physical Specifications 2 Environmental Requirements 2 Acoustic Noise Emissions 3 Electrical Specifications 3 NET MGT Connector and Pins 4 USB Management Connector and Pins 5 InfiniBand QSFP Connector and Pins 5 Understanding Cabling 9 Routing Service Cables 9 Power Cord Requirements 9... - Page 4 Check Link Status 49 Verifying the InfiniBand Fabric 49 InfiniBand Node Description 50 ▼ Discover the InfiniBand Fabric Topology 50 ▼ Perform Diagnostics on the InfiniBand Fabric 51 Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
- Page 5 Index 53 Contents...
- Page 6 Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
-
Page 7: Using This Documentation
Using This Documentation This installation guide provides detailed procedures that describe the preparation and installation of the Sun Datacenter InfiniBand Switch 36 from Oracle. This document is written for technicians, system administrators, and users who have advanced experience installing and administering InfiniBand fabric hardware. - Page 8 Access to Oracle Support Oracle customers have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id= visit info http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired. viii Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
-
Page 9: Understanding The Switch
“Installing the Switch” on page 25 ■ Switch Overview The Sun Datacenter InfiniBand Switch 36 provides InfiniBand fabric switching and management capabilities. The switch utilizes the OFED InfiniBand software stack through an internally configured management controller. The switch has 36 physical InfiniBand QDR ports. -
Page 10: Physical Specifications
41˚F to 89.6˚F (5˚C to 32˚C) Relative humidity 5% to 85% noncondensing, 80˚F (27˚C) maximum wet bulb Elevation (Sun requirement) Maximum 9840 feet (3000 meters) at 104˚F (40˚C) Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 11: Acoustic Noise Emissions
Acoustic Noise Emissions This table lists the permitted noise emissions of the switch. Parameter Operating Idling Acoustic power LWAd (1B=10dB) 7.1 B 7.2 B Acoustic pressure LpAm 58.9 dBA 59.0 dBA Electrical Specifications This table lists the electrical requirements of the switch. Parameter AC Version Requirement Voltage... -
Page 12: Net Mgt Connector And Pins
“Management Cable Requirements” on page 10 ■ “Attach the Management Cables” on page 31 ■ “Access the Management Controller From the NET MGT Port” on page 36 ■ Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 13: Usb Management Connector And Pins
USB Management Connector and Pins This table lists the pinout of the USB management connector. Signal +5 VDC - Data + Data Related Information “Management Cable Requirements” on page 10 ■ “Attach the Management Cables” on page 31 ■ “Access the Management Controller From the USB Management Port” on page 38 ■... - Page 14 Module select on low - enables reception of I C commands ResetL Reset on low LPMode Low power mode ModPrsL Module presence on low - identifies existence of QSFP connector Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
- Page 15 Signal Description IntL Interrupt on low - enables fault indication Related Information “InfiniBand Cable Types” on page 14 ■ “Connecting InfiniBand cables” on page 44 ■ Understanding the Switch...
- Page 16 Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
-
Page 17: Understanding Cabling
Understanding Cabling These topics provide information that helps you understand the cabling requirements of the switch. “Routing Service Cables” on page 9 ■ “Understanding InfiniBand Cabling” on page 11 ■ Related Information “Understanding the Switch” on page 1 ■ “Preparing to Install the Switch” on page 17 ■... -
Page 18: Management Cable Requirements
IP address to the management controller upon boot. If a DHCP server is not available, then the USB connection is used. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 19: Understanding Infiniband Cabling
The advantage of the NET connection over the USB connection is that administration of the switch can happen from anywhere on the network. There is no cable length constraint for the network management route because of the re-amplification, filtering, and processing that happens at each hub or switch within the Ethernet network. -
Page 20: Infiniband Cable Cautions
“InfiniBand Cable Guidelines” on page 13 ■ “InfiniBand Cable Types” on page 14 ■ “InfiniBand Cable Path Lengths” on page 14 ■ “InfiniBand Cable Bundling” on page 15 ■ Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 21: Infiniband Cable Guidelines
“Floor and Underfloor Delivery of InfiniBand Cables” on page 16 ■ “Overhead Delivery of InfiniBand Cables” on page 16 ■ InfiniBand Cable Guidelines Proper InfiniBand cable installation requires the following: 1. Plan the cable routes and cable length needs. Identify problematic cable route bends, minimizing the length of continuous vertical runs to no more than 2 meters (7 feet), and specify hardware to support cable routing. -
Page 22: Infiniband Cable Types
In this situation, the differences of the required lengths of the cables is quite substantial. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 23: Infiniband Cable Bundling
If you are routing the InfiniBand cable under the floor, consider the height of the ■ raised floor when calculating cable path length. Related Information “InfiniBand Cable Cautions” on page 12 ■ “InfiniBand Cable Guidelines” on page 13 ■ “InfiniBand Cable Types” on page 14 ■... -
Page 24: Floor And Underfloor Delivery Of Infiniband Cables
“InfiniBand Cable Path Lengths” on page 14 ■ “InfiniBand Cable Bundling” on page 15 ■ “Floor and Underfloor Delivery of InfiniBand Cables” on page 16 ■ Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 25: Preparing To Install The Switch
Preparing to Install the Switch These topics provide you with information that you need to know to prepare you for the installation process. “Installation Preparation” on page 17 ■ “Suggested Tools for Installation” on page 18 ■ “Antistatic Precautions for Installation” on page 19 ■... -
Page 26: Suggested Tools For Installation
“Verify Shipping Carton Contents” on page 21 ■ “Assemble the Optical Fiber InfiniBand Cables” on page 22 ■ “Route the InfiniBand Cables” on page 24 ■ Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 27: Antistatic Precautions For Installation
Antistatic Precautions for Installation When installing the switch chassis, take care to follow antistatic precautions: Use an antistatic mat as a work surface. ■ Wear an antistatic wrist strap that is attached to either the mat or a metal portion ■... -
Page 28: Installation Sequence
“Antistatic Precautions for Installation” on page 19 ■ “Installation Responsibilities” on page 19 ■ “Verify Shipping Carton Contents” on page 21 ■ “Assemble the Optical Fiber InfiniBand Cables” on page 22 ■ Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 29: Verify Shipping Carton Contents
“Route the InfiniBand Cables” on page 24 ■ ▼ Verify Shipping Carton Contents 1. Open the switch shipping carton and any additional cartons. Power cords and InfiniBand cables are shipped separately. 2. Compare the contents to this figure. Item Description Switch Front mounting brackets, long Front mounting brackets, short... -
Page 30: Assemble The Optical Fiber Infiniband Cables
2. Remove the QSFP optical transceivers, SFP+ optical transceivers, and fiber optic cables from their packaging. 3. Sort the components into groupings ready for assembly. You will need two QSFP optical transceivers and a fiber optic cable. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... - Page 31 4. Remove the plug from the QSFP optical transceiver. 5. Remove the cap from the MTP connector of the fiber optic cable. 6. Holding the shaft of the MTP connector, insert the MTP connector into the receptacle of the QSFP optical transceiver. Note –...
-
Page 32: Route The Infiniband Cables
“Installation Responsibilities” on page 19 ■ “Installation Sequence” on page 20 ■ “Verify Shipping Carton Contents” on page 21 ■ “Assemble the Optical Fiber InfiniBand Cables” on page 22 ■ Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 33: Installing The Switch
Installing the Switch These topics provide procedures that instruct you how to install the switch. “Install the Switch in the Rack” on page 25 ■ “Powering On the Switch” on page 31 ■ “Connecting InfiniBand cables” on page 44 ■ “Verifying the InfiniBand Fabric”... - Page 34 Place the long rail to the mounting location on the post of the rack. b. Butt the flange of the extender to the flange of the long rail. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
- Page 35 c. Secure the assembly to the post with two captive nuts and two screws. d. Repeat from Step a for the other cable management extender and long rail. 5. Attach the long front brackets (with cutouts) to the front of the switch with four screws on each side.
- Page 36 9. Mount the front of the switch chassis to the front rack posts with two captive nuts and two screws at each side. Tighten the screws securely. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
- Page 37 10. Install the cable management bracket to the tabs of the attachment brackets at the rear of the switch, tightening the thumbscrews on each side of the cable management bracket. 11. Install the cable management bracket cover. Installing the Switch...
- Page 38 12. Tighten the thumbscrews on each side of the cover. 13. Attach the management cables. “Attach the Management Cables” on page Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
-
Page 39: Powering On The Switch
Related Information “Powering On the Switch” on page 31 ■ “Connecting InfiniBand cables” on page 44 ■ “Verifying the InfiniBand Fabric” on page 49 ■ Powering On the Switch After installing the components, enable powering on of the switch by performing these tasks. - Page 40 This connection provides alternative communication with the management controller. The terminal device must be configured for 115200 baud, 8 bit, no parity, 1 stop bit. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
- Page 41 4. (Optional) Connect the serial management cables from the management console to the USB-to-serial adapter and from the adapter to the connector labeled with the USB symbol. 5. Route the management cables so that they do not interfere with other cables, with servicing the switch, or with other systems.
-
Page 42: Attach The Power Cords
4. Route the end of each power cord to its respective facility power receptacle. Use cable ties or hook and loop fastener straps to bundle and secure the cord. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... - Page 43 5. Plug each power cord into the receptacle. Note – To provide redundancy, connect each power cord to a separate power source. The switch can operate with only one power source, but there is no redundancy in that case. 6. Energize the circuit breakers so that the power receptacles are live. 7.
-
Page 44: Accessing The Management Controller
2. If you have not already done so, configure your DHCP server. Use the MAC address of the management controller to provide a host name and IP address for the switch. “Attach the Management Cables” on page Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... - Page 45 3. Open a SSH session and connect to the management controller by specifying the controller’s host name as configured with the DHCP server. For example: % ssh -l ilom-admin switch_name ilom-admin@switch_name’s password: password Last login: Wed Dec 12 03:48:06 2012 from 10.195.1.11 ############################################### # This user never changed the default password.
-
Page 46: Access The Management Controller From The Usb Management Port
Kernel 2.6.27.13-nm2 on an i686 switch_name login: where switch_name is the host name of the management controller. Even if you do not see the text, go to Step Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 47: Verify The Switch Status
5. Type ilom-admin for the login name followed by the password of ilom-admin. switch_name login: ilom-admin Password: ilom-admin -> The -> prompt is displayed. Note – You can change the password at a later time. Refer to Switch Administration, changing the administrator password. If you do not see this output or prompt, there is a problem with the serial configuration, the USB-to-serial adapter, or the management controller. - Page 48 If not present is in the output of the getfanspeed command, yet a fan is ■ installed at that particular slot, there is a problem with that fan. For either condition, check the fan. Refer to Switch Service, servicing fans. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
- Page 49 5. Check the status of the switch chip: FabMan@switch_name->checkboot Switch OK All Internal ibdevices OK FabMan@switch_name-> If in the output of the checkboot command a component is not OK, there is a problem with that component. Try resetting the component. Refer to Switch Administration, resetting the switch chip.
-
Page 50: Start The Subnet Manager
InfiniBand fabric, and there are no Subnet Managers other than the one within the management controller, you do not need to set the Subnet Manager priority. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... - Page 51 1. Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure. “Installation Sequence” on page 2. Enable the Subnet Manager: FabMan@switch_name-> enablesm Starting IB Subnet Manager. Starting partitiond daemon. FabMan@switch_name-> 3. (Optional) Configure the priority of the Subnet Manager within the management controller.
-
Page 52: Connecting Infiniband Cables
1. Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure. “Installation Sequence” on page 2. Loosen the two captive thumbscrews that secure the cover to the cable management bracket. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... - Page 53 3. Lift the cover off. Installing the Switch...
- Page 54 Refer to Switch Service, identifying the InfiniBand cable. 7. Slowly move the connector in. As you slide the connector in, the shell should be in the center of the QSFP receptacle. Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
- Page 55 If the connector stops or binds after about 1/4 in. (5 mm) travel, back out and ■ repeat from Step If the connector stops or binds with about 1/8 in. (2 mm) still to go, back out ■ and repeat Step 8.
- Page 56 “Understanding InfiniBand Cabling” on page 11 ■ “Attach the Management Cables” on page 31 ■ “Attach the Power Cords” on page 34 ■ “Check Link Status” on page 49 ■ Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
-
Page 57: Check Link Status
“Attach the InfiniBand Cables” on page 44 ■ Verifying the InfiniBand Fabric Use the ibnetdiscover and ibdiagnet commands to initially determine the operational status of your Sun Datacenter InfiniBand Switch 36 from Oracle in the InfiniBand fabric. “InfiniBand Node Description” on page 50 ■... -
Page 58: Infiniband Node Description
Switch 36 "S-0021283a8389a0a0" # "Sun DCS 36 QDR switch localhost" enhanced port 0 lid 15 lmc 0 [23] "H-0003ba000100e388"[2](3ba000100e38a) # "mnm33-43 HCA-1" lid 14 4xQDR Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013... -
Page 59: Perform Diagnostics On The Infiniband Fabric
vendid=0x2c9 devid=0x673c sysimgguid=0x3ba000100e38b caguid=0x3ba000100e388 2 "H-0003ba000100e388" # "mnm33-43 HCA-1" [2](3ba000100e38a) "S-0021283a8389a0a0"[23] # lid 14 lmc 0 "Sun DCS 36 QDR switch localhost" lid 15 4xQDR FabMan@switch_name-> Note – The output for your InfiniBand fabric will differ from that in the example. 3. - Page 60 Refer to Switch Remote Management, backing up the configuration. Related Information Switch Reference, ibdiagnet command ■ “InfiniBand Node Description” on page 50 ■ “Discover the InfiniBand Fabric Topology” on page 50 ■ Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
- Page 61 Index connector InfiniBand cable, 5 accessing NET MGT, 4 management controller, 36 QSFP, 5 network management, 36 USB management, 5 USB management, 38 current, 3 acoustic noise emissions, 3 idling, 3 operating, 3 diagnosing the InfiniBand fabric, 51 antistatic precautions, 19 discovering the InfiniBand fabric, 50 assembling InfiniBand cables, 22...
- Page 62 InfiniBand cable attaching, 44 physical specifications, 2 bundling, 15 power cord cautions, 12 attaching, 34 connector, 5 requirements, 9 delivery power specifications, 3 floor and underfloor, 16 powering on overhead, 16 switch, 31 handling guidelines, 13 preparation, 17 length, 14 path lengths, 14 type, 14 understanding, 11...
- Page 63 understanding cabling, 9 InfiniBand cable, 11 installation, 17 switch specifications, 1 USB management cable requirements, 10 connector, 5 verifying InfiniBand fabric, 49 switch status, 39 voltage, 3 Index...
- Page 64 Sun Datacenter InfiniBand Switch 36 Installation Guide for Firmware Version 2.1 • March 2013...
Need help?
Do you have a question about the Sun Datacenter InfiniBand Switch 36 and is the answer not in the manual?
Questions and answers