1. Manuals
  2. Brands
  3. Sun Oracle Manuals
  4. Gateway
  5. Sun Network QDR InfiniBand Gateway Switch
  6. Installation manual

Sun Oracle Sun Network QDR InfiniBand Installation Manual

Switch, firmware version 2.1
Hide thumbs Also See for Sun Network QDR InfiniBand:
Table of Contents

Advertisement

Quick Links

Download this manual
Sun Network QDR InfiniBand Gateway Switch
Installation Guide for Firmware Version 2.1
Part No.: E36259-01
March 2013, Revision A

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Sun Network QDR InfiniBand and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Sun Oracle Sun Network QDR InfiniBand

Summary of Contents for Sun Oracle Sun Network QDR InfiniBand

  • Page 1 Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 Part No.: E36259-01 March 2013, Revision A...
  • Page 2 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means.

  • Page 3: Table Of Contents

    Contents Using This Documentation vii Understanding the Gateway 1 Gateway Overview 1 Physical Specifications 2 Environmental Requirements 2 Acoustic Noise Emissions 3 Electrical Specifications 3 NET MGT Connector and Pins 4 USB Management Connector and Pins 5 Data QSFP Connector and Pins 5 Understanding Cabling 9 Routing Service Cables 9 Power Cord Requirements 9...
  • Page 4 Check Link Status 53 Verifying the InfiniBand Fabric 53 InfiniBand Node Description 54 ▼ Discover the InfiniBand Fabric Topology 54 ▼ Perform Diagnostics on the InfiniBand Fabric 55 Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 5 Index 57 Contents...
  • Page 6 Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 7: Using This Documentation

    Using This Documentation This installation guide provides detailed procedures that describe the preparation and installation of the Sun Network QDR InfiniBand Gateway Switch from Oracle. This document is written for technicians, system administrators, and users who have advanced experience installing and administering InfiniBand fabric hardware.
  • Page 8 Access to Oracle Support Oracle customers have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id= visit info http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired. viii Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 9: Understanding The Gateway

    “Installing the Gateway” on page 27 ■ Gateway Overview The Sun Network QDR InfiniBand Gateway Switch provides both InfiniBand fabric switching and management capabilities and in-band access to 10GbE networks. The gateway utilizes the OFED InfiniBand software stack and the BXOFED BridgeX software stack through an internally configured management controller.

  • Page 10: Physical Specifications

    41˚F to 89.6˚F (5˚C to 32˚C) Relative humidity 5% to 85% noncondensing, 80˚F (27˚C) maximum wet bulb Elevation (Sun requirement) Maximum 9840 feet (3000 meters) at 104˚F (40˚C) Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 11: Acoustic Noise Emissions

    Acoustic Noise Emissions This table lists the permitted noise emissions of the gateway. Parameter Operating Idling Acoustic power LWAd (1B=10dB) 7.1 B 7.2 B Acoustic pressure LpAm 58.9 dBA 59.0 dBA Electrical Specifications This table lists the electrical requirements of the gateway. Parameter AC Version Requirement Voltage...

  • Page 12: Net Mgt Connector And Pins

    “Management Cable Requirements” on page 10 ■ “Attach the Management Cables” on page 33 ■ “Access the Management Controller From the NET MGT Port” on page 38 ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 13: Usb Management Connector And Pins

    USB Management Connector and Pins This table lists the pinout of the USB management connector. Signal +5 VDC - Data + Data Related Information “Management Cable Requirements” on page 10 ■ “Attach the Management Cables” on page 33 ■ “Access the Management Controller From the USB Management Port” on page 40 ■...
  • Page 14 Module select on low - enables reception of I C commands ResetL Reset on low LPMode Low power mode ModPrsL Module presence on low - identifies existence of QSFP connector Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 15 Signal Description IntL Interrupt on low - enables fault indication Related Information “Data Cable Types” on page 14 ■ “Connecting Data cables” on page 47 ■ Understanding the Gateway...
  • Page 16 Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 17: Understanding Cabling

    Understanding Cabling These topics provide information that helps you understand the cabling requirements of the gateway. “Routing Service Cables” on page 9 ■ “Understanding Data Cabling” on page 11 ■ Related Information “Understanding the Gateway” on page 1 ■ “Preparing to Install the Gateway” on page 17 ■...

  • Page 18: Management Cable Requirements

    IP address to the management controller upon boot. If a DHCP server is not available, then the USB connection is used. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 19: Understanding Data Cabling

    The advantage of the NET connection over the USB connection is that administration of the gateway can happen from anywhere on the network. There is no cable length constraint for the network management route because of the re-amplification, filtering, and processing that happens at each hub or switch within the Ethernet network.

  • Page 20: Data Cable Cautions

    “Data Cable Guidelines” on page 13 ■ “Data Cable Types” on page 14 ■ “Data Cable Path Lengths” on page 14 ■ “Data Cable Bundling” on page 15 ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 21: Data Cable Guidelines

    “Floor and Underfloor Delivery of Data Cables” on page 16 ■ “Overhead Delivery of Data Cables” on page 16 ■ Data Cable Guidelines Proper data cable installation requires the following: 1. Plan the cable routes and cable length needs. Identify problematic cable route bends, minimizing the length of continuous vertical runs to no more than 2 meters (7 feet), and specify hardware to support cable routing.

  • Page 22: Data Cable Types

    However, the bend radius is different in different parts of the bundle. If the bundle is large and unorganized, and there are many Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 23: Data Cable Bundling

    bends, one cable might experience only the inner radius of bends, while another cable might experience the outer radius of bends. In this situation, the differences of the required lengths of the cables is quite substantial. If you are routing the data cable under the floor, consider the height of the raised ■...

  • Page 24: Floor And Underfloor Delivery Of Data Cables

    “Data Cable Path Lengths” on page 14 ■ “Data Cable Bundling” on page 15 ■ “Floor and Underfloor Delivery of Data Cables” on page 16 ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 25: Preparing To Install The Gateway

    Preparing to Install the Gateway These topics provide you with information that you need to know to prepare you for the installation process. “Installation Preparation” on page 17 ■ “Suggested Tools for Installation” on page 18 ■ “Antistatic Precautions for Installation” on page 19 ■...

  • Page 26: Suggested Tools For Installation

    “Verify Shipping Carton Contents” on page 21 ■ “Assemble the Optical Fiber Data Cables” on page 22 ■ “Route the Data Cables” on page 25 ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 27: Antistatic Precautions For Installation

    Antistatic Precautions for Installation When installing the gateway chassis, take care to follow antistatic precautions: Use an antistatic mat as a work surface. ■ Wear an antistatic wrist strap that is attached to either the mat or a metal portion ■...

  • Page 28: Installation Sequence

    ■ “Installation Responsibilities” on page 19 ■ “Verify Shipping Carton Contents” on page 21 ■ “Assemble the Optical Fiber Data Cables” on page 22 ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 29: Verify Shipping Carton Contents

    “Route the Data Cables” on page 25 ■ ▼ Verify Shipping Carton Contents 1. Open the gateway shipping carton and any additional cartons. Power cords and data cables are shipped separately. 2. Compare the contents to this figure. Item Description Gateway Front mounting brackets, long Front mounting brackets, short...

  • Page 30: Assemble The Optical Fiber Data Cables

    If you are to assemble an Infiniband cable or Ethernet pass-through cable, you ■ will need two QSFP optical transceivers and a fiber optic cable. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 31 If you are to assemble an Ethernet splitter cable, you will need one QSFP ■ optical transceiver, four SFP+ optical transceivers, and an optical splitter cable. 4. Remove the plug from the QSFP optical transceiver. 5. Remove the cap from the MTP connector of the fiber optic cable. 6.
  • Page 32 “Installation Preparation” on page 17 ■ “Suggested Tools for Installation” on page 18 ■ “Antistatic Precautions for Installation” on page 19 ■ “Installation Responsibilities” on page 19 ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 33: Route The Data Cables

    “Installation Sequence” on page 20 ■ “Verify Shipping Carton Contents” on page 21 ■ “Route the Data Cables” on page 25 ■ ▼ Route the Data Cables 1. Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure. “Installation Sequence”...
  • Page 34 Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 35: Installing The Gateway

    Installing the Gateway These topics provide procedures that instruct you how to install the gateway. “Install the Gateway in the Rack” on page 27 ■ “Powering On the Gateway” on page 33 ■ “Connecting Data cables” on page 47 ■ “Verifying the InfiniBand Fabric”...
  • Page 36 Place the long rail to the mounting location on the post of the rack. b. Butt the flange of the extender to the flange of the long rail. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 37 c. Secure the assembly to the post with two captive nuts and two screws. d. Repeat from Step a for the other cable management extender and long rail. 5. Attach the long front brackets (with cutouts) to the front of the gateway with four screws on each side.
  • Page 38 9. Mount the front of the gateway chassis to the front rack posts with two captive nuts and two screws at each side. Tighten the screws securely. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 39 10. Install the cable management bracket to the tabs of the attachment brackets at the rear of the gateway, tightening the thumbscrews on each side of the cable management bracket. 11. Install the cable management bracket cover. Installing the Gateway...
  • Page 40 12. Tighten the thumbscrews on each side of the cover. 13. Attach the management cables. “Attach the Management Cables” on page Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 41: Powering On The Gateway

    Related Information “Powering On the Gateway” on page 33 ■ “Connecting Data cables” on page 47 ■ “Verifying the InfiniBand Fabric” on page 53 ■ Powering On the Gateway After installing the components, enable powering on of the gateway by performing these tasks.
  • Page 42 This connection provides alternative communication with the management controller. The terminal device must be configured for 115200 baud, 8 bit, no parity, 1 stop bit. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 43 4. (Optional) Connect the serial management cables from the management console to the USB-to-serial adapter and from the adapter to the connector labeled with the USB symbol. 5. Route the management cables so that they do not interfere with other cables, with servicing the gateway, or with other systems.

  • Page 44: Attach The Power Cords

    4. Route the end of each power cord to its respective facility power receptacle. Use cable ties or hook and loop fastener straps to bundle and secure the cord. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 45 5. Plug each power cord into the receptacle. Note – To provide redundancy, connect each power cord to a separate power source. The gateway can operate with only one power source, but there is no redundancy in that case. 6. Energize the circuit breakers so that the power receptacles are live. 7.

  • Page 46: Accessing The Management Controller

    Use the MAC address of the management controller to provide a host name and IP address for the gateway. “Attach the Management Cables” on page Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 47 3. Open a SSH session and connect to the management controller by specifying the controller’s host name as configured with the DHCP server. For example: % ssh -l ilom-admin gateway_name ilom-admin@gateway_name’s password: password Last login: Thu Dec 13 03:58:59 2012 from sr1-sfbay-01.us.oracle.com Oracle(R) Integrated Lights Out Manager Version ILOM 3.0 r47111 Copyright (c) 2012, Oracle and/or its affiliates.

  • Page 48: Access The Management Controller From The Usb Management Port

    Kernel 2.6.27.13-nm2 on an i686 gateway_name login: where gateway_name is the host name of the management controller. Even if you do not see the text, go to Step Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 49: Verify The Gateway Status

    5. Type ilom-admin for the login name followed by the password of ilom-admin. gateway_name login: ilom-admin Password: ilom-admin -> The -> prompt is displayed. Note – You can change the password at a later time. Refer to Gateway Administration, changing the administrator password. If you do not see this output or prompt, there is a problem with the serial configuration, the USB-to-serial adapter, or the management controller.
  • Page 50 ■ installed at that particular slot, there is a problem with that fan. For either condition, check the fan. Refer to Gateway Service, servicing fans. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 51 5. Check the status of the switch chip and BridgeX chips: FabMan@gateway_name->checkboot Switch OK Bridge-0 OK Bridge-1 OK All Internal ibdevices OK FabMan@gateway_name-> If in the output of the checkboot command a component is not OK, there is a problem with that component. Try resetting the component. Refer to Gateway Administration, resetting the switch chip and Gateway Administration, resetting the BridgeX chips.
  • Page 52 FabMan@gateway_name-> setgwinstance --list No BXM system name set, using 6 last bits of the ip-address, value: 34 FabMan@gateway_name-> The instance number for this example is 34. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 53: Start The Subnet Manager

    b. If the instance number is a duplicate, change it: FabMan@gateway_name-> setgwinstance number where number is the instance number, from 0 to 63. For example: FabMan@gateway_name-> setgwinstance 3 Stopping Bridge Manager.. Starting Bridge Manager. FabMan@gateway_name-> c. Repeat from Step a for all remaining gateways in the InfiniBand subnet.
  • Page 54 Starting IB Subnet Manager. Starting partitiond daemon. FabMan@gateway_name-> 4. Attach the data cables. “Attach the Data Cables” on page Related Information Gateway Reference, setsmpriority command ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 55: Connecting Data Cables

    Gateway Reference, enablesm command ■ “Attach the Management Cables” on page 33 ■ “Attach the Power Cords” on page 36 ■ “Accessing the Management Controller” on page 38 ■ “Verify the Gateway Status” on page 41 ■ Connecting Data cables After verifying the gateway operational status, you can begin attaching the data cables.
  • Page 56 “Installation Sequence” on page 2. Loosen the two captive thumbscrews that secure the cover to the cable management bracket. 3. Lift the cover off. Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 57 4. Remove the protective cap from the connector or transceiver and visually inspect the cable connector. The shell should not be bent and should be parallel to the inner boards. If the connector is bent or damaged, use a different cable. 5.
  • Page 58 As you slide the connector in, the shell should be in the center of the QSFP receptacle. If the connector stops or binds after about 1/4 in. (5 mm) travel, back out and ■ repeat from Step Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 59 If the connector stops or binds with about 1/8 in. (2 mm) still to go, back out ■ and repeat Step 8. Continue to push the connector in until you feel a detent. 9. Place the cable into the open slot on the cable management bracket. 10.
  • Page 60 “Understanding Data Cabling” on page 11 ■ “Attach the Management Cables” on page 33 ■ “Attach the Power Cords” on page 36 ■ “Check Link Status” on page 53 ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 61: Check Link Status

    “Attach the Data Cables” on page 47 ■ Verifying the InfiniBand Fabric Use the ibnetdiscover and ibdiagnet commands to initially determine the operational status of your Sun Network QDR InfiniBand Gateway Switch from Oracle in the InfiniBand fabric. “InfiniBand Node Description” on page 54 ■...

  • Page 62: Infiniband Node Description

    Switch 36 "S-0021283a8389a0a0" # "Sun DCS 36 QDR switch localhost" enhanced port 0 lid 15 lmc 0 [23] "H-0003ba000100e388"[2](3ba000100e38a) # "mnm33-43 HCA-1" lid 14 4xQDR Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

  • Page 63: Perform Diagnostics On The Infiniband Fabric

    vendid=0x2c9 devid=0x673c sysimgguid=0x3ba000100e38b caguid=0x3ba000100e388 2 "H-0003ba000100e388" # "mnm33-43 HCA-1" [2](3ba000100e38a) "S-0021283a8389a0a0"[23] # lid 14 lmc 0 "Sun DCS 36 QDR switch localhost" lid 15 4xQDR FabMan@gateway_name-> Note – The output for your InfiniBand fabric will differ from that in the example. 3.
  • Page 64 Refer to Gateway Remote Management, backing up the configuration. Related Information Gateway Reference, ibdiagnet command ■ “InfiniBand Node Description” on page 54 ■ “Discover the InfiniBand Fabric Topology” on page 54 ■ Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...
  • Page 65 Index ssh, 38 connecting data cables, 47 accessing management controller, 38 connector network management, 38 data cable, 5 USB management, 40 NET MGT, 4 QSFP, 5 acoustic noise USB management, 5 emissions, 3 idling, 3 current, 3 operating, 3 antistatic precautions, 19 assembling data cable data cables, 22...
  • Page 66 Ethernet assembling cables, 22 NET MGT connector, 4 routing cables, 25 network management cable requirements, 10 connector, 4 gateway node description, 54 installing into rack, 27 overview, 1 powering on, 33 operating noise, 3 specifications, 1 overview verifying status, 41 gateway, 1 getfanspeed command, 41 physical specifications, 2...
  • Page 67 link, 53 Subnet Manager enabling, 45 starting, 45 tools, 18 understanding cabling, 9 data cable, 11 gateway specifications, 1 installation, 17 USB management cable requirements, 10 connector, 5 verifying gateway status, 41 InfiniBand fabric, 53 voltage, 3 Index...
  • Page 68 Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013...

Table of Contents