1. Manuals
  2. Brands
  3. Oracle Manuals
  4. Server
  5. Exalogic Elastic Cloud
  6. Owner's manual

Oracle Exalogic Elastic Cloud Owner's Manual

Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
®
Oracle
Exalogic Elastic Cloud
Machine Owner's Guide
Release 12.2.1.0.0
E18478-21
April 2017
This document describes how to manage and configure an
Exalogic machine.
Release 12.2.1.0.0 is for Oracle Exalogic Elastic Cloud X6-2,
X5-2, X4-2, X3-2, and X2-2.

Advertisement

Table of Contents
loading

Related Manuals for Oracle Exalogic Elastic Cloud

Summary of Contents for Oracle Exalogic Elastic Cloud

  • Page 1 ® Oracle Exalogic Elastic Cloud Machine Owner's Guide Release 12.2.1.0.0 E18478-21 April 2017 This document describes how to manage and configure an Exalogic machine. Release 12.2.1.0.0 is for Oracle Exalogic Elastic Cloud X6-2, X5-2, X4-2, X3-2, and X2-2.
  • Page 2 Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

  • Page 3: Table Of Contents

    Documentation Accessibility ........................xx Related Documents............................. xx Conventions..............................xx 1 Learn About the Exalogic Machine 1.1 About Oracle Exalogic ........................1-1 1.1.1 Features of Exalogic ......................1-1 1.2 Hardware Components of Exalogic....................1-2 1.2.1 PDUs in Exalogic ........................1-3 1.2.2 Spares Kit for Exalogic......................1-3 1.2.3 Pre-cabling for Exalogic X4-2 and newer Machines............
  • Page 4 6 Understand Network Requirements and Configuration 6.1 Overview of Network Requirements..................... 6-1 6.2 Naming Conventions ........................6-5 6.3 Important Notes for Oracle Solaris Users ..................6-5 6.4 Network Connection and IP Address Requirements ..............6-5 6.4.1 Network Connection Requirements ................... 6-6 6.4.2 IP Address Requirements.....................
  • Page 5 6.6.4 IPMP Overview for Oracle Solaris Users................. 6-12 6.6.5 Connectivity Between Exalogic Compute Nodes............6-13 6.6.6 Connectivity Between Exalogic Machine and External LAN Through Sun Network QDR InfiniBand Gateway Switch................... 6-13 6.6.7 Additional InfiniBand Network Requirements and Specifications ......6-15 6.7 Prepare to Reconfigure the Networking of Exalogic Machine..........
  • Page 6 10.1 Introduction to Virtual NICs (VNICs) ..................10-1 10.1.1 VNIC Resource Limit......................10-1 10.2 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Linux ............10-2 10.3 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris............. 10-7 10.3.1 Set Up Ethernet Over InfiniBand on Oracle Solaris 11.1 ..........10-7 10.3.2 Set Up Ethernet Over InfiniBand on Oracle Solaris 11.2 ...........
  • Page 7 12.1.6 Verify the InfiniBand Fabric .................... 12-5 12.1.7 Monitor a Gateway Switch Using Web Interface ............12-7 12.2 What Next? ............................ 12-8 13 Manage the InfiniBand Network Using Subnet Manager 13.1 Understand Administrative Commands................... 13-1 13.1.1 Hardware Command Overview ..................13-1 13.1.2 InfiniBand Command Overview..................
  • Page 8 16.8 Perform the Post-Configuration Steps..................16-15 16.9 Important Notes for Combined Exalogic-Exadata Fabric Users.......... 16-15 16.10 Partitioning Limitations......................16-16 17 Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17.1 Overview............................17-2 17.2 Key Features ..........................17-2 17.3 Prerequisites ..........................17-3 17.4 Accessing Oracle Enterprise Manager Ops Center Documentation ........
  • Page 9 17.15.2 Adding a Monitoring Rule from the Asset View............17-30 17.16 Using Reports in Oracle Enterprise Manager Ops Center..........17-31 17.17 Using Oracle Services in Oracle Enterprise Manager Ops Center ........17-32 17.17.1 Prerequisites for Using Oracle Services in Oracle Enterprise Manager Ops Center ..............................
  • Page 10 D.5 Parts for ZS3-ES Storage Appliance for Exalogic X4-2............... D-5 D.6 Parts for Sun ZFS Storage 7320 Appliance................... D-6 D.7 Parts for Oracle Server X5-2 Compute Nodes ................D-7 D.8 Parts for Sun Server X4-2 Compute Nodes.................. D-8 D.9 Parts for Sun Server X3-2 Compute Nodes.................. D-9 D.10 Parts for X4170 M2 Compute Nodes..................
  • Page 11 E.2.3 Exalogic Machine Quarter Rack..................E-18 E.2.4 Exalogic Machine Eighth Rack..................E-20 E.3 Power Distribution Unit Cabling Tables..................E-22 E.3.1 Single Phase PDU Power Cabling Full Rack..............E-23 E.3.2 Single Phase PDU Power Cabling Half Rack..............E-24 E.3.3 Single Phase PDU Power Cabling Quarter Rack............E-25 E.3.4 Single Phase PDU Power Cabling Eighth Rack..............
  • Page 13 Create Filesystem........................8-18 NIS Service............................ 9-2 NFS Service........................... 9-3 NIS Service............................ 9-4 Oracle Enterprise Manager Home Page................. 17-4 17-1 Management of Exalogic Machine Hardware Using Oracle Enterprise Manager Ops 17-2 Center............................ 17-5 Custom Asset Discovery Screen....................17-7 17-3 Configure Group........................17-10 17-4 Group Summary Screen......................
  • Page 14 Phone Home Tab on the List of Services................18-5 18-1 Phone Home Settings Page...................... 18-6 18-2 ASR Activation - Asset Screen in My Oracle Support............18-7 18-3 Exalogic Eighth Rack Network Connectivity................. C-2 Exalogic Quarter Rack Network Connectivity............... C-3 Exalogic Half Rack Network Connectivity................C-3...
  • Page 15 Replacement Parts for Oracle ZS3-ES Appliance..............D-5 Replacement Parts for Sun ZFS Storage 7320 appliance............D-6 Replacement Parts for Oracle Server X5-2 Compute Nodes..........D-7 Replacement Parts for Sun Server X4-2 Compute Nodes............. D-8 Replacement Parts for Sun Server X3-2 Compute Nodes............. D-9 Replacement Parts for X4170 M2 Compute Nodes..............
  • Page 16 Replacement Parts for Gari DE2-24C Disk Enclosure ............D-15 D-19 Replacement Parts for Gari DE2-24C Disk Enclosure ............D-16 D-20 Gigabit Ethernet Cabling for Exalogic Machine Full Rack........... E-2 Gigabit Ethernet Cabling for Exalogic Machine Half Rack........... E-4 Gigabit Ethernet Cabling for Exalogic Machine Quarter Rack..........E-5 Gigabit Ethernet Cabling for Exalogic Machine Eighth Rack..........

  • Page 17: Preface

    Note: All hardware-related specifications in this guide are based on information for a typical deployment provided by Oracle at the time this guide was written. Oracle is not responsible for hardware problems that may result from following the average specifications in this document. For detailed information about preparing your site for Oracle Exalogic machine deployment, consult your hardware specification.
  • Page 18 Next?: Added note with a link to MOS document for changing passwords. – Cabling Diagrams : Removed sections about multirack cabling and added a reference to the Exalogic Elastic Cloud Multirack Cabling Guide. • E18478-16: December 2013 – Hardware Components of Exalogic: Added generic link that points to X4-2 data sheet.
  • Page 19 E18478-09: September 2012 – Configure Ethernet Over InfiniBand: Updated the procedure to set up EoIB on Oracle Linux and Oracle Solaris, to correct certain errors and inconsistencies. Added information about the recommended naming convention for VNICs on Oracle Linux. –...

  • Page 20: Documentation Accessibility

    For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup? ctx=acc&id=docacc. Access to Oracle Support Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/ or visit topic/lookup?ctx=acc&id=info http://www.oracle.com/pls/...

  • Page 21: Learn About The Exalogic Machine

    Operational Procedures for Exalogic Machines 1.1 About Oracle Exalogic Oracle Exalogic is an integrated hardware and software system designed to provide a complete platform for a wide range of application types and widely varied workloads. Exalogic is intended for large-scale, performance-sensitive, mission-critical application deployments.

  • Page 22: Hardware Components Of Exalogic

    PDU options. You can connect up to eight Exalogic machines, or a combination of Exalogic and Exadata on the same InfiniBand fabric, without the need for any external switches. 1-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 23: Pdus In Exalogic

    When Oracle Services personnel visit your site for repairing and replacing hardware, you should make the spares kit available to them. Note that any parts used by Oracle Service personnel from the spares kit will be replenished by Oracle.

  • Page 24: Operational Procedures For Exalogic Machines

    U02 to U05 Compute Node Compute Node Vented one-unit NM2-36P InfiniBand filler Switch 1.4 Operational Procedures for Exalogic Machines This section includes the following topics: • Non-emergency Power Procedure • Emergency Power-off Considerations 1-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 25: Non-Emergency Power Procedure

    Compute nodes and controllers can also be powered off by using operating system commands. For example, you can use the following command on the Oracle Linux operating system: shutdown -h -y now On the Oracle Solaris operating system, you can use the following command:...

  • Page 26: Emergency Power-Off Considerations

    To perform an emergency power-off procedure for the Exalogic machine, turn off power at the circuit breaker or pull the emergency power-off switch in the computer room. After the emergency, contact Oracle Support Services to restore power to the machine.

  • Page 27: Site Requirements

    The environmental requirements for an Exalogic machine depend on the purchased hardware configuration. The environmental requirements for an Exalogic rack can be found in the data sheets at the following location: http://www.oracle.com/technetwork/server-storage/engineered- systems/exalogic/index.html 2.2 Space Requirements All Exalogic machines use the same rack, and have the same space requirements. The space requirements are as follows: •...

  • Page 28: Receiving And Unpacking Requirements

    The maintenance area must be large enough for the Exalogic machine, and have the required access space. For example, the required space to remove the side panels is 675. 64 mm (26.6 inches). Table 2-2 lists the maintenance access requirements for the Exalogic machine. 2-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 29: Flooring Requirements

    914 mm (36 inches) 2.3 Flooring Requirements Oracle recommends that the Exalogic machine be installed on raised flooring. The site floor and the raised flooring must be able to support the total weight of the Exalogic machine as specified in Environmental Requirements.

  • Page 30: Pdu Requirements For Low Voltage

    96A (3x32A) 109A (6 × 18.1A) Outlets 42 C13, 6 C19 42 C13, 6 C19 42 C13, 6 C19 Number of Inputs 1x25A, 3 ph 3 x 32A 1 ph 2x25A, 3 ph 2-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 31: Facility Power Requirements

    Electrical Power Requirements Table 2-5 (Cont.) PDU Requirements for High Voltage Specification 15 kVA, 3 ph 22 kVA, 1 ph 24 kVA, 3 ph Current 24A max. per input 32 A per input 18 A max. per input Data Center IEC 309-4P5W-IP44 IEC 309-2P3W-IP44 (32A, IEC 309-4P5W-IP44 (32A,...

  • Page 32: Temperature And Humidity Requirements

    The ambient relative humidity range of 45 to 50 percent is suitable for safe data processing operations. Most computer equipment can operate in a wide range (20 to 80 percent), but the range of 45 to 50 percent is recommended for the following reasons: 2-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 33: Ventilation And Cooling Requirements

    Ventilation and Cooling Requirements • Optimal range helps protect computer systems from corrosion problems associated with high humidity levels. • Optimal range provides the greatest operating time buffer in the event of air conditioner control failure. • This range helps avoid failures or temporary malfunctions caused by intermittent interference from static discharges that may occur when relative humidity is too low.

  • Page 34: Typical Data Center Configuration For Perforated Floor Tiles

    Figure 2-1 shows a typical installation of the floor tiles for an Exalogic machine full rack in a typical data center. Figure 2-1 Typical Data Center Configuration for Perforated Floor Tiles 2-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 35: Install Planning And Procedure

    Install Planning and Procedure This chapter explains how to prepare for the installation of Exalogic machines, to install an Exalogic machine at the site, and to power on the rack and its components. It contains the following topics: • Installation Overview •...

  • Page 36: Prepare To Install An Exalogic Machine

    Install the Exalogic machine on a flat, level surface. • Use a power outlet that uses proper grounding. When using shared grounding, the grounding resistance must not be greater than 10 ohms. Ensure that your 3-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 37: Unpack The Exalogic Machine Rack

    Prepare to Install an Exalogic Machine facility administrator or a qualified electrical engineer verifies the grounding method for the building, and performs the grounding work. • Be sure that each grounding wire used for the Exalogic machine is used exclusively for the Exalogic machine. Also be sure to observe the precautions, warnings, and notes about handling that appear on labels on the equipment.

  • Page 38: Tools For Installation

    Figure 3-1 Unpacking Exalogic Machine Rack 3.2.4 Tools for Installation The following tools are required for installation: • Screwdriver handle (magnetic) • T-30 Torx wrench key • T-25 Torx wrench key 3-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 39: Prepare The Installation Site

    Prepare to Install an Exalogic Machine • 6-mm hexagon Allen wrench key • SW 12-mm single-headed wrench • Side panel removal tool • Keys to the front door, rear door, and side panel locks • Cage nut mounting tool • 32 M6 cage nuts •...
  • Page 40 Locate the four leveling feet located at the bottom corners of the Exalogic machine. Figure 3-2 shows the location of the leveling feet on the bottom of the Exalogic machine. 3-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 41: Location Of Leveling Feet On Exalogic Machine

    Prepare to Install an Exalogic Machine Figure 3-2 Location of Leveling Feet on Exalogic Machine • 1: Distance from the edge of the mounting feet to the side of the rack is 33.75 mm (1.33 inches) • 2: Width from the outside edges of the leveling feet is 532.5 mm (20.96 inches) •...

  • Page 42: Securing Exalogic Machine Using The Leveling Feet

    Prepare the installation site by drilling four holes into the floor. Before permanently stabilizing the Exalogic machine with the mounting brackets, pre-drill the mounting holes. Figure 3-4 shows the location and dimensions of the mounting brackets. 3-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 43: Location Of Mounting Brackets On Rack

    Obtain four bolts and washers to mount the Exalogic machine to the floor. The bolt holes in the mounting brackets have a 10.0 mm diameter. Oracle does not provide mounting bolts because different floors require different bolt types and strengths.

  • Page 44: Base Position To Measure

    Figure 3-6 Base Position to Measure Caution: Do not create a hole where the rack casters or leveling feet brackets will be placed. Open the front and rear Exalogic machine doors. 3-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 45: Securing Exalogic Machine Using Mounting Brackets

    Prepare to Install an Exalogic Machine Install the mounting brackets to the rack as shown in Figure 3-7 using a 6 mm hexagon Allen wrench key. Figure 3-7 Securing Exalogic Machine Using Mounting Brackets Using bolts and washers that are appropriate for your installation site, permanently mount your system to the floor using the four mounting brackets as shown in Figure 3-7.

  • Page 46: Power On The System The First Time

    3.3.1 Inspect the Machine After It Is in Place The following procedure describes how to visually examine the Exalogic machine physical system after it is in place, but before power is supplied: Check the rack for damage. 3-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 47: Connect The Power Cords

    Power on the System the First Time Check the rack for loose or missing screws. Check your Exalogic machine for the ordered configuration. Refer to the Customer Information Sheet (CIS) on the side of the packaging. Check that all cable connections are secure and firmly in place as follows: Check the power cables.

  • Page 48: Power On The Exalogic Machine

    The 24 kVA PDU has 4 plugs, and the 15 kVA PDU has 2 plugs. In either case, you can use the instructions described in this section to power on your Exalogic machine. 3-14 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 49: Pdu Switch Locations

    Power on the System the First Time Figure 3-11 PDU Switch Locations Wait 3 to 5 minutes for all Oracle Integrated Lights Out Manager (ILOM) service processors to boot. Open the front cabinet door. Verify that server standby power is on for compute nodes, InfiniBand gateway switches, and storage appliance in the Exalogic machine.

  • Page 50: Exalogic Compute Node Led Lights

    Soft Switches on the Server Heads in the Sun ZFS Storage 7320 appliance Figure 3-14 shows the soft switch and LED lights on each of the server heads in the storage appliance. 3-16 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 51: Soft Switch And Led Lights On Server Heads

    Power on the System the First Time Figure 3-14 Soft Switch and LED Lights on Server Heads Wait 3-5 minutes for the storage appliance to initiate NFS services, daemons, and basic services. Tip: You can ping the IP address assigned to the storage appliance to verify whether the system is up and running.

  • Page 52: Connect A Laptop To The Exalogic Machine

    Exalogic machine quarter rack. This spine switch is used in multirack configuration scenarios (an Exalogic machine to another Exalogic machine, and an Exalogic machine to an Oracle Exadata Database Machine) only. 3.4 Connect a Laptop to the Exalogic Machine You can connect a laptop to the Exalogic machine as follows: Ensure that you have a laptop with functional USB and network ports.
  • Page 53 Cisco Ethernet switch resides. You can use the default NET0 IP addresses of compute nodes assigned at the time of manufacturing or the custom IP address that you reconfigured using the Oracle Exalogic Configuration Utility. For the list of default NET0 IP addresses, see Default IP Addresses and Ports.

  • Page 54: What Next

    IP addresses for the Exalogic machine, you can use a set of default IP addresses. If you have already run the Oracle Exalogic Configuration Utility set of tools and scripts, you can use the network IP address that you provided as input to the Exalogic Configurator spreadsheet.

  • Page 55: Default Ip Addresses And Ports

    Default IP Addresses and Ports This chapter lists the default ILOM, NET0, InfiniBand Bonded IP addresses, and ports assigned to Exalogic machine components during manufacturing. This chapter contains the following sections: • Exalogic Machine Full Rack • Exalogic Machine Half Rack •...
  • Page 56 192.168.1.12 192.168.10.12 Compute Node 192.168.1.111 192.168.1.11 192.168.10.11 Compute Node 192.168.1.110 192.168.1.10 192.168.10.10 Compute Node 192.168.1.109 192.168.1.9 192.168.10.9 Compute Node 192.168.1.108 192.168.1.8 192.168.10.8 Compute Node 192.168.1.107 192.168.1.7 192.168.10.7 Compute Node 192.168.1.106 192.168.1.6 192.168.10.6 4-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 57 The following table shows the hardware that is included in different Exalogic versions. Compute Cisco Storage Appliance Node’s Management Server Switch Oracle Cisco 4948E Oracle ZS5-ES –2 Server X6-2 Oracle Cisco 4948E Oracle ZS3-ES –2 Server X5-2 Cisco 4948E- Oracle ZS3-ES –2...

  • Page 58: Exalogic Machine Half Rack

    2U Solid Filler applicable applicable applicable applicable applicable applicable Compute Node 192.168.1.118 192.168.1.18 192.168.10.18 Compute Node 192.168.1.117 192.168.1.17 192.168.10.17 1U Solid Filler applicable applicable applicable 4-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 59 Exalogic Machine Half Rack Table 4-2 (Cont.) Default ILOM, NET0, and IB Bonded IP for Exalogic Half Rack Unit Rack Component (Front View) ILOM IP NET0 IP IB Bonded IP Address Address Address Cisco Management Switch 192.168.1.200 applicable applicable 1U Solid Filler applicable applicable applicable...

  • Page 60: Exalogic Machine Quarter Rack

    4.3 Exalogic Machine Quarter Rack Table 4-3 lists the default ILOM, NET0, and InfiniBand Bonded IP addresses assigned during manufacturing to Exalogic compute nodes and other hardware components in an Exalogic machine quarter rack. 4-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 61: Default Ilom, Net0, And Ib Bonded Ip For Exalogic Quarter Rack

    Exalogic Machine Quarter Rack Table 4-3 Default ILOM, NET0, and IB Bonded IP for Exalogic Quarter Rack Unit Rack Component (Front View) ILOM IP NET0 IP InfiniBand Address Address Bonded IP Address 4U Solid Filler applicable applicable applicable applicable applicable applicable applicable applicable...
  • Page 62 192.168.1.8 192.168.10.8 Compute Node 192.168.1.107 192.168.1.7 192.168.10.7 Compute Node 192.168.1.106 192.168.1.6 192.168.10.6 Compute Node 192.168.1.105 192.168.1.5 192.168.10.5 Compute Node 192.168.1.104 192.168.1.4 192.168.10.4 Compute Node 192.168.1.103 192.168.1.3 192.168.10.3 Compute Node 192.168.1.102 192.168.1.2 192.168.10.2 4-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 63: Exalogic Machine Eighth Rack

    The following table shows the hardware that is included in different Exalogic versions. Compute Cisco Storage Appliance Node’s Management Server Switch Oracle Cisco 4948E Oracle ZS5-ES –2 Server X6-2 Oracle Cisco 4948E Oracle ZS3-ES –2 Server X5-2 Cisco 4948E- Oracle ZS3-ES –2...

  • Page 64: Default Ilom, Net0, And Ib Bonded Ip For Exalogic Eighth Rack

    2U Solid Filler applicable applicable applicable applicable applicable applicable 2U Solid Filler applicable applicable applicable applicable applicable applicable 1U Solid Filler applicable applicable applicable Cisco Management Switch 192.168.1.200 applicable applicable 4-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 65 Exalogic Machine Eighth Rack Table 4-4 (Cont.) Default ILOM, NET0, and IB Bonded IP for Exalogic Eighth Rack Unit Rack Component (Front View) ILOM IP NET0 IP InfiniBand Address Address Bonded IP Address 1U Solid Filler applicable applicable applicable Sun Network QDR InfiniBand 192.168.1.202 Gateway Switch applicable...

  • Page 66: Default Port Assignments

    Sun Fire Cisco 4948 Sun ZFS 7320 –2 X4170 4.5 Default Port Assignments Table 4-5 lists the default ports assigned to Exalogic compute nodes and other hardware components in an Exalogic machine. 4-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 67: Default Ports

    Default Port Assignments Table 4-5 Default Ports Source Target Protocol Port Application Compute nodes, SSH over TCP storage appliance heads, and InfiniBand ILOMs Compute nodes, HTTP over TCP Web (user storage appliance configurable) heads, and InfiniBand ILOMs Compute nodes, SNMP over UDP SNMP (Simple storage appliance Network...
  • Page 68 (Dynamic Host heads, and Configuration InfiniBand ILOMs Protocol) Compute nodes, RADIUS over UDP 1812 Outgoing RADIUS storage appliance (Remote heads, and Authentication InfiniBand ILOMs Dial In User Service) (user configurable) 4-14 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 69 Default Port Assignments Table 4-5 (Cont.) Default Ports Source Target Protocol Port Application Compute nodes, and ASR Manager SNMP Telemetry storage appliance messages sent to heads ILOMs ASR Manager Storage appliance HTTP over TCP Browser interface Storage NTP over UDP Outgoing NTP management HTTP over TCP...
  • Page 70 Default Port Assignments 4-16 Machine Owner's Guide...

  • Page 71: Add The Exalogic Machine To Your Network

    Add the Exalogic Machine to Your Network This chapter discusses the factory configuration and initial network configuration for the Exalogic machine. This chapter contains the following topics: • Default State of the Exalogic Machine Network Configuration • Verify the Factory Configuration •...
  • Page 72 5-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 73 Verify the Factory Configuration cnode9_SPNET0_IPADDR=192.168.1.109 cnode10_host_name=el01cn10 cnode10_private_host_name=el01cn10-priv cnode10_eth0_IPADDR=192.168.1.10 cnode10_bond0_IPADDR=192.168.10.10 cnode10_SPNET0_IPADDR=192.168.1.110 cnode11_host_name=el01cn11 cnode11_private_host_name=el01cn11-priv cnode11_eth0_IPADDR=192.168.1.11 cnode11_bond0_IPADDR=192.168.10.11 cnode11_SPNET0_IPADDR=192.168.1.111 cnode12_host_name=el01cn12 cnode12_private_host_name=el01cn12-priv cnode12_eth0_IPADDR=192.168.1.12 cnode12_bond0_IPADDR=192.168.10.12 cnode12_SPNET0_IPADDR=192.168.1.112 cnode13_host_name=el01cn13 cnode13_private_host_name=el01cn13-priv cnode13_eth0_IPADDR=192.168.1.13 cnode13_bond0_IPADDR=192.168.10.13 cnode13_SPNET0_IPADDR=192.168.1.113 cnode14_host_name=el01cn14 cnode14_private_host_name=el01cn14-priv cnode14_eth0_IPADDR=192.168.1.14 cnode14_bond0_IPADDR=192.168.10.14 cnode14_SPNET0_IPADDR=192.168.1.114 cnode15_host_name=el01cn15 cnode15_private_host_name=el01cn15-priv cnode15_eth0_IPADDR=192.168.1.17 cnode15_bond0_IPADDR=192.168.10.17 cnode15_SPNET0_IPADDR=192.168.1.117 cnode16_host_name=el01cn16 cnode16_private_host_name=el01cn16-priv cnode16_eth0_IPADDR=192.168.1.18 cnode16_bond0_IPADDR=192.168.10.18 cnode16_SPNET0_IPADDR=192.168.1.118 cnode17_host_name=el01cn17 cnode17_private_host_name=el01cn17-priv cnode17_eth0_IPADDR=192.168.1.19...
  • Page 74 5-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 75: Prerequisites For Connecting Exalogic Machine To External Ethernet Network

    Exalogic machine to another Exalogic machine rack or to an Oracle Exadata Database Machine rack Note: You can use the Oracle Exalogic Configuration Utility set of tools and scripts to configure the compute nodes, storage heads, and Sun Network QDR InfiniBand Gateway Switches. For more information about Exalogic...
  • Page 76 Press RETURN to get started. Change to enable mode using the following command. The Cisco switch prompts for the default password. For the default password, contact Oracle Support. Switch> enable Configure the network for a single VLAN. The following is an example of the...
  • Page 77 Initial Network Configuration of Exalogic Machine Note: If IP routing is required on the switch, then leave the IP routing setting as the default, and configure the default gateway as follows: Switch# configure terminal Enter configuration commands,one per line.End with CNTL/Z. Switch(config)# ip route 0.0.0.0 0.0.0.0 10.7.7.1 Switch(config)# end *Sep 15 14:13:26.013:%SYS-5-CONFIG_I:Configured from console by console...
  • Page 78 US Eastern time: Switch# configure terminal Enter configuration commands,one per line.End with CNTL/Z. Switch(config)# clock timezone EST -5 Switch(config)# clock summer-time EDT recurring Switch(config)# end Switch# clock set 21:00:00 December 09 2009 5-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 79 Initial Network Configuration of Exalogic Machine Switch#write memory Building configuration... Compressed configuration from 3784 bytes to 1465 bytes [OK ] Switch# show clock 21:00:06.643 EST Wed Dec 9 2009 Configure up to two NTP servers. The following is an example: Switch# configure terminal Enter configuration commands,one per line.End with CNTL/Z.
  • Page 80 10.7.7.1 ip http server control-plane line con 0 stopbits 1 line vty 0 4 password Your_Password login line vty 5 15 password Your_Password login ntp server 10.7.7.32 prefer ntp server 172.16.9.1 5-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 81 Initial Network Configuration of Exalogic Machine If any setting is incorrect, then repeat the appropriate step. To erase a setting, enter no in front of the same command. For example, to erase the default gateway, the following commands would be entered: no ip default-gateway 10.7.7.1 write memory Save the current configuration using the following command:...
  • Page 82 Initial Network Configuration of Exalogic Machine 5-12 Machine Owner's Guide...

  • Page 83: Understand Network Requirements And Configuration

    It contains the following topics: • Overview of Network Requirements • Naming Conventions • Important Notes for Oracle Solaris Users • Network Connection and IP Address Requirements • Default InfiniBand Bonded IP Addresses • Introduction to Oracle Exalogic Network Configuration •...
  • Page 84 To deploy the Exalogic machine, verify that you meet the minimum network requirements. There are up to five networks for an Exalogic machine. Each network must be on a distinct and separate subnet from the others. The network descriptions are as follows: 6-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 85 Sun Network QDR InfiniBand Gateway Switch for high availability (HA) and each host EoIB vNIC is associated with a different HCA IB port (On Oracle Linux, vNIC0 -> ib0, vNIC1 -> ib1; on Oracle Solaris, vNIC0 -> ibp0, vNIC1 -> ibp1).

  • Page 86: Network Diagram For Exalogic Machine

    Sun Network QDR InfiniBand Gateway Switches for Note that each compute node requires a bond for each external network (physical network or VLAN). Figure 6-1 shows the network diagram for the Exalogic machine with Oracle Linux operating system. Figure 6-1 Network Diagram for Exalogic Machine...

  • Page 87: Naming Conventions

    Naming Conventions Note: If you are using Oracle Solaris, you can assign the logical names of IPMP groups to be ipmp0 or BOND0, and ipmp1 or BOND1 and have the name of the datalink corresponding to the NET0 Ethernet port to be displayed as igp0 or net0 in the Solaris administration commands.

  • Page 88: Network Connection Requirements

    How many 10 Gb ports per LAN are required (your network throughput requirements for client access network) At least two 10 Gb Ethernet ports per compute node - one active and another passive. 6-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 89: Ip Address Requirements

    Exalogic rack is assigned a separate Local Identifier (LID) by the IB Subnet Manager. Oracle Exalogic supports a single IP over IB (IPoIB) link over a logical Exalogic deployment over the IB subnet. Multiple IP subnets can be layered on top of the default IPoIB link.

  • Page 90: Ip Address Requirements For Exalogic Machine

    IP address, such as W_floatingIP:Port. This request is forwarded through the Client. For any new requests, Client accesses the application Server through T3:W_floatingIP:Port. 6-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 91 Network Connection and IP Address Requirements Table 6-2 (Cont.) IP Address Requirements for Exalogic Machine Type of Management Client Access Network InfiniBand Private Exalogic Network Requirements Network (IPoIB) Machine Requirements Requirements half rack Minimum of 38 IP Same as above. Minimum of one addresses IP address per...

  • Page 92: Default Infiniband Bonded Ip Addresses

    IPMP Overview for Oracle Solaris Users • Connectivity Between Exalogic Compute Nodes • Connectivity Between Exalogic Machine and External LAN Through Sun Network QDR InfiniBand Gateway Switch • Additional InfiniBand Network Requirements and Specifications 6-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 93: Infiniband Fabric

    Sun Datacenter InfiniBand Switch 36 (part number NM2-36P) is used in multirack configuration scenarios (an Exalogic machine to another Exalogic machine, and an Exalogic machine to an Oracle Exadata Database Machine) only. This switch is not connected or used in an Exalogic machine single rack.

  • Page 94: Ipmp Overview For Oracle Solaris Users

    NICs of different speeds within the same IPMP group, as long as the NICs are of the same type. IPMP groups on Oracle Solaris provide the same functionality as Bonded Interfaces on Oracle Linux in the Exalogic environment. For...

  • Page 95: Connectivity Between Exalogic Compute Nodes

    Introduction to Oracle Exalogic Network Configuration Note: For information about administering and configuring IPMP groups on the Oracle Solaris operating system installed on Exalogic compute nodes, see Oracle Solaris 11.1 documentation. 6.6.5 Connectivity Between Exalogic Compute Nodes Compute nodes in the Exalogic machine are connected to one another through dual- ported InfiniBand quad data rate (QDR) host channel adapters (HCAs).

  • Page 96: Connectivity Between Nm2-Gw And External 10 Gb Ethernet Switch

    LANs, as required. 6.6.6.3 Transceiver and Cable Requirements Table 6-3 lists the transceiver and cable requirements that you must complete to connect your Exalogic machine to your data center's 10 Gb Ethernet switch. 6-14 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 97: Additional Infiniband Network Requirements And Specifications

    Introduction to Oracle Exalogic Network Configuration Table 6-3 Transceivers and Cables Optical Module on Cable Needed Ethernet Switch Transceiver Needed Exalogic's Sun Vendor Network QDR InfiniBand Gateway Switch QSFP module QSFP MTP to 4 LC A Sun Oracle switch For Sun Oracle...

  • Page 98: Prepare To Reconfigure The Networking Of Exalogic Machine

    Identify your current client access network. Determine if you will connect multiple Exalogic machines or a combination of Exalogic machines and Oracle Exadata Database Machines. If you plan do to either of these, then contact your Oracle representative. Provide input to the Exalogic Configurator spreadsheet, which will be used to generate configuration scripts required for the initial network configuration.

  • Page 99: General Network Configuration Worksheet (Required)

    Table 6-5 is the general network configuration worksheet. Fill out this worksheet and provide it to the Oracle representative, or use as a reference to complete the initial configuration of your Exalogic machine. Information entered in this worksheet is used as input to Oracle Exalogic Configuration Utility.

  • Page 100: Client Access Network Configuration Worksheet (Required)

    Table 6-6 is the management network configuration worksheet. Fill out this worksheet and provide it to the Oracle representative, or use as a reference to complete the initial configuration of your Exalogic machine. Information entered in this worksheet is used as input to Oracle Exalogic Configuration Utility.

  • Page 101: Private Infiniband Network Configuration Worksheet (Required)

    Table 6-8 is the client access network configuration worksheet. Fill out this worksheet and provide it to the Oracle representative, or use as a reference to complete the initial configuration of your Exalogic machine. Information entered in this worksheet is used as input to Oracle Exalogic Configuration Utility.
  • Page 102 Network Configuration Worksheets 6-20 Machine Owner's Guide...

  • Page 103: Set Up Ilom On The Compute Nodes

    Set Up ILOM on the Compute Nodes This chapter describes how to set up and access Oracle Integrated Lights Out Manager (ILOM) for Exalogic compute nodes. It contains the following topics: • ILOM Overview • Important Notes Before You Begin •...

  • Page 104: Ilom Interfaces

    (CLI). The ILOM management interface is also integrated with Oracle Enterprise Manager Ops Center. Oracle Enterprise Manager Ops Center can discover new and existing systems on your network, update firmware and BIOS configurations, provision the operating environment with off-the-shelf distributions, manage updates and configuration changes, and remotely control key aspects of the service processor such as boot control, power status, and indicator lights.

  • Page 105: Ilom Ip Addresses For Exalogic Machine Components

    For a list of default ILOM IP addresses, see Default IP Addresses and Ports. However, you can reconfigure the IP addresses by using the Oracle Exalogic Configuration Utility. The Oracle Exalogic Configuration Utility and its associated scripts reconfigure ILOM IP addresses based on the input you provide to the Exalogic Configurator spreadsheet.

  • Page 106: Connect To The Cli

    The network must be connected to the NET0 port (with sideband) of each of the Exalogic compute nodes or hardware components. Oracle recommends that you use this connection method to access ILOM for Exalogic compute nodes. This section discusses the following topics: •...

  • Page 107: Connect To Ilom Via A Serial Connection

    Connect to ILOM via a Serial Connection Click the Remote Control tab, and click Launch Remote Console. The remote console is displayed. 7.6 Connect to ILOM via a Serial Connection You can connect to ILOM via a serial connection if you are unable to access ILOM via the network due to any of the following problems: •...

  • Page 108: Connect To The Ilom Of A Sun Network Qdr Infiniband Gateway Switch

    This step is required only if your network access must be reconfigured. You can reconfigure network access after connecting to the ILOM using a serial connection or an Ethernet connection. Note: Oracle recommends that you use an Ethernet connection if reconfiguration of network access becomes necessary. This section contains the following topics: •...

  • Page 109: Reconfigure The Network Access Using A Serial Connection

    Reconfigure the Network Access 7.7.1 Reconfigure the Network Access Using a Serial Connection Connect to the ILOM using a serial connection, as described in Connect to ILOM via a Serial Connection. Run the following commands at the ILOM command prompt: set pendingipdiscovery=static set pendingipaddress=<ip_address>...

  • Page 110: Configure Ilom Ip Addresses Manually

    Reconfigure the Network Access 7.7.4 Configure ILOM IP Addresses Manually If ILOM IP addresses get misconfigured due to Oracle Exalogic Configuration Utility failures, you can configure ILOM IP addresses manually. Under normal circumstances, you do not need to configure ILOM IP addresses manually.

  • Page 111: What Next

    What Next? set pendingipdiscovery=static set pendingipaddress=<ip_address> set pendingipnetmask=<ip_netmask> set pendingipgateway=<ip_gateway> set pendingmanagementport=/SYS/MB/NET0 set commitpending=true 7.8 What Next? After configuring ILOM for your Exalogic machine, verify the various configurations and proceed to complete the initial configuration of the storage appliance, which is included in the Exalogic machine.
  • Page 112 What Next? 7-10 Machine Owner's Guide...

  • Page 113: Configure The Storage Appliance

    Password for configuring an Ethernet interface on the storage controllers • Running the Oracle Exalogic Configuration Utility to reconfigure IP addresses and other network parameters for the storage appliance 8.2 Getting Started You can access the storage appliance over Ethernet via the Cisco Ethernet Management Switch.
  • Page 114 Getting Started Tip: Refer to the Cluster documentation in the Oracle ZFS Storage Appliance Administration Guide located on http://docs.oracle.com/cd/ for more information. E27998_01/html/E48433/index.html Complete the following steps: Verify that the storage appliance is powered up and on the network. Connect an Ethernet cable from your network to the NET0 port on the back panel of the controller (storage server head).

  • Page 115: Storage Appliance Overview

    For example, for a compute node with the host name abc, the default project abc_1 is created. You can create all shares within this default project. However, Oracle recommends that you create additional projects for organizational purposes. 8.3.2 Introduction to Shares Shares are file systems and LUNs that are exported over supported data protocols to clients of the appliance.

  • Page 116: Connect Storage Heads To The Management Network And Accessing The Web Interface

    When you run the Oracle Exalogic Configuration Utility set of tools and scripts, the initial configuration for the storage appliance is completed. For more information, see the Shares and Configuration sections of the Oracle ZFS Storage Appliance Administration Guide (http://docs.oracle.com/cd/ E27998_01/html/E48433/toc.html).

  • Page 117: Network Configuration Screen

    Configuration Overview https://hostname:215 The login screen appears. Type root into the Username field and the administrative password that you entered into the appliance shell kit interface and press the Enter key. The Welcome screen is displayed. Click the Configuration tab, and click NETWORK. The default networking configuration is displayed, as shown in Figure 8-2.

  • Page 118: Cluster Network Configuration

    8.4.3 Cluster Network Configuration The cluster is set up in an active-passive configuration. All resources, data interface links, and storage pool are owned by the active storage head. When the active node 8-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 119: Network Configuration Options

    Note: For more information about network configuration for the storage appliance, see the topic in the Oracle ZFS Storage Appliance Administration Guide. Network 8.4.4 Network Configuration Options You can choose any of the following network configuration options for the storage appliance, based on your specific requirements: •...
  • Page 120 The bonded interface is a new interface, such as dr-repl-interface, with igb2 and igb3 configured as an IPMP group. For example, the network settings of the dr- repl-interface is shown in Figure 8-4. 8-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 121: Igb2 And Igb3 In An Ipmp Group

    Configuration Overview Figure 8-4 igb2 and igb3 in an IPMP Group Note: The interface names and IP addresses shown on the screens in this chapter are examples only. You must verify the interface names in your environment and use them accordingly. In the Properties section, if you select the Allow Administration option, management is enabled on the interface.

  • Page 122: Creating A New Ipmp Group Interface

    Provides Exalogic Configuration Utility, which is used to reconfigure the Exalogic machine based on your specific requirements, with ports igb0 and igb1 • Provides single management URL for both storage heads, using ports igb2 and igb3 8-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 123 Configuration Overview Note: This option does not separate the management path from the disaster recovery path. To configure this option, complete the following steps: Ensure that the physical connections are correct, as shown in Figure 8-1. Ensure that the free hanging cable from the igb3 port is connected to your data center network switch.

  • Page 124: Ipmp Network Interface Settings

    8.4.4.3 Configure Option 3: ASR Support and No Disaster Recovery, But with Single Management URL In this custom configuration, the igb0 port on your active storage head (head 1) is used, and the management option is enabled. The igb0 port on your stand-by storage 8-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 125: Default Storage Configuration

    Configuration Overview head (head 2) is not used. The igb1 port on your stand-by storage head (head 2) is used, and the management option is disabled. The igb2 port uses a virtual IP, and the management option is enabled. The igb3 port is not used. This configuration option offers the following benefits: •...

  • Page 126: Custom Configuration

    This table represents the default configuration of the storage appliance before the Exalogic machine rack configuration is modified at the customer's site. Oracle Exalogic Configuration Utility does not alter this configuration. 8.4.6 Custom Configuration You can create and configure a number of projects and shares on the storage appliance to meet your specific storage requirements in the enterprise.

  • Page 127: Create Custom Projects

    Create Custom Projects Note: For information about the recommended directory structure and shares, see the Oracle Fusion Middleware Exalogic Enterprise Deployment Guide. 8.5 Create Custom Projects Shares are grouped together as Projects. For example, you can create a project for Dept_1.
  • Page 128 Restrict ownership change - By default, this check box is selected and the ownership of files can only be changed by a root user. This property can be removed on a per-filesystem or per-project basis by 8-16 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 129: Create Custom Shares

    Create Custom Shares Table 8-2 (Cont.) Project Settings Section and Description Setting deselecting this check box. When deselected, file ownership can be changed by the owner of the file or directory. Default Custom settings for file systems, to be used as default, include the Settings following: •...

  • Page 130: Create Filesystem

    You can use this feature to control access to the file system, based on the access types (users and groups) in Dept_1. You can either inherit a mountpoint by selecting the Inherit mountpoint option or set a mountpoint. 8-18 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 131: Case Sensitivity Values

    Create Custom Shares Note: The mount point must be under /export. The mount point for one share cannot conflict with another share. In addition, it cannot conflict with another share on cluster peer to allow for proper failover. When inheriting the mountpoint property, the current dataset name is appended to the project's mountpoint setting, joined with a slash ('/').

  • Page 132: Use The Phone Home Service To Manage The Storage Appliance

    Heartbeats - daily heartbeat messages are sent to Oracle to indicate that the system is up and running. Oracle support may notify the technical contact for an account when one of the activated systems fails to send a heartbeat for too long.

  • Page 133: Register Your Storage Appliance

    Use the Phone Home Service to Manage the Storage Appliance 8.7.1 Register Your Storage Appliance To register the appliance for the first time, you must provide a Oracle Single Sign-On account and specify one of that account's inventory teams into which to register the appliance.
  • Page 134 Use the Phone Home Service to Manage the Storage Appliance 8-22 Machine Owner's Guide...

  • Page 135: Configure Nfs Version 4 On Exalogic

    Create NFSv4 Mount Points on Oracle Linux 9.1 Overview Oracle Exalogic is a multihomed system where multiple IP addresses are configured on the InfiniBand network interface. NFSv4 addresses file locking problems on multihomed systems, such as Exalogic. The file locking problem occurs when NFSv3 is used.

  • Page 136: Verify The Nis Setting On Exalogic

    Figure 9-1 NIS Service 9.3 Configure the Storage Appliance You must configure two services in the storage appliance for using NFSv4 on Exalogic: • Configure the NFS Service • Configure the NIS Service 9-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 137: Configure The Nfs Service

    Configure the Storage Appliance 9.3.1 Configure the NFS Service Log in to the browser user interface (BUI) of the storage appliance, and configure the NFS service as follows: Click Configuration on the home page. Expand Services on the left navigation menu. Click NFS.

  • Page 138: Configure The Nis Service

    Configure an Exalogic Linux compute node as follows: Log in to the compute node as root. Edit the /etc/idmapd.conf configuration file: vi /etc/idmapd.conf Set the domain value, as in the following example: Domain = us.myexample.com 9-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 139 # service ypbind restart Check the yp service by running this command: # ypwhich Verify if you can access Oracle user accounts: # ypcat passwd Add ypbind to your boot sequence, so that it starts automatically after rebooting. # chkconfig ypbind on...

  • Page 140: Create Nfsv4 Mount Points On Oracle Linux

    Create NFSv4 Mount Points on Oracle Linux 9.5 Create NFSv4 Mount Points on Oracle Linux Create NFSv4 mount points on Oracle Linux as follows: Create a new share on the storage appliance. In this example, it is assumed that the new share is accessible as el01sn01:/export/common/patches. This mount point is mounted on the compute node as /u01/common/patches.

  • Page 141: Configure Ethernet Over Infiniband

    This chapter contains the following topics: • Introduction to Virtual NICs (VNICs) • Set Up Ethernet Over InfiniBand (EoIB) on Oracle Linux • Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris 10.1 Introduction to Virtual NICs (VNICs) A virtual NIC or VNIC maps an Ethernet connector on the Sun Network QDR InfiniBand Gateway Switch to a network interface within the compute node.

  • Page 142: Set Up Ethernet Over Infiniband (Eoib) On Oracle Linux

    Oracle Linux, complete the following steps: Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch. Oracle recommends that you log in as the root user. For example, log in to el01gw04 as root.
  • Page 143 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Linux Node GUID: 0x0021280001a0a364 System image GUID: 0x0021280001a0a367 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 120 LMC: 0 SM lid: 6 Capability mask: 0x02510868 Port GUID: 0x0021280001a0a365 Link layer: IB...
  • Page 144 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Linux Switch : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0 lid 65 corresponds to gateway switch el01gw04 with GUID 0x00212856d0a2c0a0. Define a dummy MAC address in the following format:...
  • Page 145 DEVICE directive in the interface file must not be based on the kernel- assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE directive in the...
  • Page 146 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Linux Device name: ethA_B A is the EPORT_ID, and B is the number after the colon (:) in the IOA_PORT value. Example: Interface file name: ifcfg-eth331_1 Device name: eth331_1 In this example, 331 is the EPORT_ID, and 1 is the value derived from the IOA_PORT.

  • Page 147: Set Up Ethernet Over Infiniband (Eoib) On Oracle Solaris

    Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris 10.3 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris This section includes the following procedures to set up EoIB on Oracle Solaris: • Set Up Ethernet Over InfiniBand on Oracle Solaris 11.1 •...
  • Page 148 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris el01cn02# dladm show-ib LINK HCAGUID PORTGUID PORT STATE PKEYS ibp0 21280001A0A694 21280001A0A695 FFFF ibp1 21280001A0A694 21280001A0A696 FFFF In the output, information about two ports is displayed. From this output, you must determine which port GUID to use. This example procedure uses the port GUID 21280001A0A695 (port 1).
  • Page 149 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris GUID of switch el01gw05: 00:21:28:56:d1:62:c0:a0 Last three octets of the switch GUID: 62:c0:a0 Administrative IP address of compute node: 192.168.1.5 Last three octets of the compute node's IP address: 168.1.5 Last three octets in hexadecimal notation: a8:01:05.
  • Page 150 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris On the compute node, run the following command to display the list of VNICs available on the compute node: el01cn02# dladm show-phys | grep eoib This command displays the name of the new interface, as seen on the compute node, such as eoib0.
  • Page 151 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris To configure eoib0 and eoib1 in an IPMP group for high availability purposes, do the following: Identify the data links associated with the VNICs you created on the InfiniBand switch by running the following command:...
  • Page 152 To set up Ethernet over InfiniBand connectivity for Exalogic compute nodes running Oracle Solaris 11.2 Base Image of EECS 2.0.6.2.0 perform the following procedure: Use an SSH client, such as PuTTY, to log in to a compute node as root. For this example log in to el01cn16 as root.
  • Page 153 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris net4 Infiniband 32000 unknown ibp0 net5 Infiniband 32000 unknown ibp1 Open a second terminal and log in as root to the switch net5 is connected to (el01sw-ib02 for this example). Run the showvlan command to verify that the VLAN 0 is associated with the default partition and that the VLAN ID is created on the correct IB partition.
  • Page 154 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris 10.128.21.186 leaf:1" enhanced port 0 lid 14 lmc 0 . . . Switch : 0x0010e00b6d80c0a0 ports 36 "SUN IB QDR GW switch el01sw-ib03 10.128.21.187 leaf:2" enhanced port 0 lid 15 lmc 0...
  • Page 155 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris From the compute node session create a host-based VNIC with a VLAN tag. Run the following commands: root@el01cn16:~# dladm create-vnic -l eoib0 -v 3066 vnic3066_0 root@el01cn16:~# dladm show-vnic The following is an example of the show-vnic command output:...
  • Page 156 Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris root@el01cn16:~# ipadm set-ifprop -p standby=on -m ip vnic3066_1 root@el01cn16:~# ipmpstat -i INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE vnic3066_1 no bond1 is----- disabled ok vnic3066_0 yes bond1 --mbM-- disabled ok bond0_1...

  • Page 157: Set Up Virtual Lans

    Example Scenario • Tag the Ethernet Connectors With a VLAN Identifier • Oracle Linux: Creating VNICs and Associating Them with VLANs • Oracle Solaris: Creating VNICs and Associating Them with VLANs 11.1 Introduction to VLAN The Ethernet standard has a provision to combine multiple broadcast domains, and thus IP subnets, onto a single Ethernet cable using a Virtual LAN (IEEE 802.1Q VLAN)

  • Page 158: Tag The Ethernet Connectors With A Vlan Identifier

    Log in to the gateway switch interface as root, and run the following commands: # createvlan 1A-ETH-3 -VLAN 10 -PKEY default Where 1A-ETH-3 is the Ethernet connector on the gateway switch, 10 is the VLAN identifier, and default is the partition key used in Exalogic. 11-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 159: Oracle Linux: Creating Vnics And Associating Them With Vlans

    Where 1A-ETH-3 is the Ethernet connector, 11 is the VLAN identifier, and default is the partition key used in Exalogic. If you are using Oracle Solaris compute nodes, you should also enable the connector for untagged traffic, by running the following command on the gateway...

  • Page 160: Oracle Solaris: Creating Vnics And Associating Them With Vlans

    VNIC configuration to a file. For information about doing this on Oracle Linux, seeSet Up Ethernet Over InfiniBand (EoIB) on Oracle Linuxfor more information. Be sure to create a bonded interface comprising two VNICs on Oracle Linux, for high availability purposes. 11.5 Oracle Solaris: Creating VNICs and Associating Them with VLANs If you wish to associate a VNIC with a VLAN, you should provide a VLAN identifier when mapping the MAC address.
  • Page 161 The output of this command shows the MAC address, which is the MAC address you defined for the VNIC. If you want to configure the VNIC interfaces in an IPMP group on Oracle Solaris, for high availability purposes, perform on of the following procedures depending on the OS version: •...
  • Page 162 Oracle Solaris: Creating VNICs and Associating Them with VLANs 11-6 Machine Owner's Guide...

  • Page 163: Use The Infiniband Gateway Switches

    Use the InfiniBand Gateway Switches This chapter describes how to use the Sun Network QDR InfiniBand Gateway switches in your Exalogic machine. The number of gateway switches depends on your purchased Exalogic machine rack configuration. It also describes how to manage the InfiniBand network using Subnet Manager. This chapter contains the following sections: •...

  • Page 164: Access The Command-Line Interface (Cli) Of A Gateway Switch

    For each gateway switch, you can check the status of the CLI, power supplies, fans, and switch chip. Verify that the voltage and temperature values of the gateway switch are within specification: 12-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 165 Using Sun Network QDR InfiniBand Gateway Switches # showunhealthy # env_test An unfavorable output from these commands indicates a hardware fault with that particular component. A voltage or temperature deviating more than 10% from the provided specification means a problem with the respective component. For example, on the CLI of one of the gateway switches, enter the following command to check its status: # env_test...

  • Page 166: Start The Subnet Manager Manually

    After starting the SM, you can verify that the Link LEDs for cabled links are green. If the Link LED is dark, the link is down. If the Link LED flashes, there are symbol errors. To check the link status of the cables: 12-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 167: Verify The Infiniband Fabric

    Using Sun Network QDR InfiniBand Gateway Switches # listlinkup If the link for a connector is reported as not present, the link at either end of the cable is down. If a port is down, use the enableswitchport 0 portnumber command to bring the port up.
  • Page 168 -W- Topology file is not specified. Reports regarding cluster links will use direct routes. Loading IBDM from: /usr/lib/ibdm1.2 -I- Using port 0 as the local port. -I- Discovering ... 2 nodes (1 Switches & 1 CA-s) discovered. 12-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 169: Monitor A Gateway Switch Using Web Interface

    Using Sun Network QDR InfiniBand Gateway Switches -I- Links With links width != 4x (as set by -lw option) -I--------------------------------------------------- -I- No unmatched Links (with width != 4x) were found -I--------------------------------------------------- -I- Links With links speed != 10 (as set by -ls option) -I--------------------------------------------------- -I- No unmatched Links (with speed != 10) were found -I- Stages Status Report:...

  • Page 170: What Next

    (vnic0 and vnic1) configured on separate Sun Network QDR InfiniBand Gateway Switches for the Ethernet over InfiniBand (EoIB) BOND1 interface. Optionally, you can create VLANs and vNICs using the InfiniBand gateway switches. 12-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 171: Manage The Infiniband Network Using Subnet Manager

    Manage the InfiniBand Network Using Subnet Manager This chapter describes how to manage the InfiniBand Network Using Subnet Manager in your Exalogic machine. This chapter contains the following sections: • Understand Administrative Commands • Manage InfiniBand Network Using Subnet Manager •...

  • Page 172: Manage Infiniband Network Using Subnet Manager

    Single Exalogic machine All leaf switches All leaf switches: 5 Two half- or full-rack Exalogic machines Spine switches Spine switch: 8 Two quarter-rack Exalogic machines All leaf switches All leaf switches: 5 13-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 173 Manage InfiniBand Network Using Subnet Manager Table 13-1 (Cont.) Running the Subnet Manager in Different Rack Configurations Rack Configuration SM Should Run SM Priority On... Three or more Exalogic machines Spine switches Spine switch: 8 Half- or full-rack Exalogic machine connected to a Spine switches Spine switch: 8 half- or full-rack Exadata machine.

  • Page 174: Monitor The Subnet Manager

    13.2.4.1 Identify the Location of Master Subnet Manager From any InfiniBand switch in the network (leaf switch or spine switch), log in as root and run the getmaster command to obtain the location of the master SM as follows: 13-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 175: Work With The Default Rack-Level Infiniband Partition

    InfiniBand Gateway Switch) to the spine switch (Sun Datacenter InfiniBand Switch 36) when you are connecting more than one Exalogic machine. This step is also necessary when you are connecting an Exalogic machine to an Oracle Exadata Database Machine. Relocating the master SM does not affect the availability of the InfiniBand network.

  • Page 176: Partition In Exalogic Machine

    Exalogic compute nodes and the storage appliance are full members of this default partition. Note: Oracle recommends that you create IP subnets over the default IP over InfiniBand (IPoIB) link to implement isolate application deployments in the Exalogic environments. Each IP subnet will have a single multicast domain.

  • Page 177: Use The Sun Datacenter Infiniband Switch 36 In Multirack Configurations

    This chapter describes how to set up and configure Sun Datacenter InfiniBand Switch 36, which is used as the spine switch in multirack configurations (an Exalogic machine to another Exalogic machine, or an Exalogic machine to an Oracle Exadata Database Machine) only. This spine switch is not included in Exalogic machine quarter-rack configuration.

  • Page 178: Access The Cli Of A Sun Datacenter Infiniband Switch 36

    For the Sun Datacenter InfiniBand Switch 36, you can check the status of the command-line interface (CLI), power supplies, fans, and switch chip. Verify that the voltage and temperature values of the switch are within specification: # showunhealthy # env_test 14-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 179 Verify the Switch Status An unfavorable output from these commands indicates a hardware fault with that particular component. A voltage or temperature deviating more than 10% from the provided specification means a problem with the respective component. For example, on the CLI of the switch, enter the following command to check its status: # env_test This command performs a set of checks and displays the overall status of switch, as in...

  • Page 180: Start The Subnet Manager In Multirack Configuration Scenarios

    After assigning the SM priority on the spine switch correctly, on the CLI of the gateway switches (Sun Network QDR InfiniBand Gateway Switches referred to as leaf switches in this guide), run the following command to disable Subnet Manager individually on the gateway switches: # disablesm 14-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 181: Check Link Status

    Check Link Status 14.5 Check Link Status After starting the Subnet Manager, you can verify that the Link LEDs for cabled links are green. If the Link LED is dark, the link is down. If the Link LED flashes, there are symbol errors.

  • Page 182: Discover The Infiniband Network Topology In A Multirack Configuration

    (CLI) of the spine switch: # ibdiagnet In the following example, the ibdiagnet command is minimized to determine which links are underperforming: # ibdiagnet -lw 4x -ls 10 -skip all Loading IBDIAGNET from: /usr/lib/ibdiagnet1.2 14-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 183: Check For Errors In The Infiniband Fabric In A Multirack Configuration

    Verify the InfiniBand Fabric in a Multirack Configuration -W- Topology file is not specified. Reports regarding cluster links will use direct routes. Loading IBDM from: /usr/lib/ibdm1.2 -I- Using port 0 as the local port. -I- Discovering ... 2 nodes (1 Switches & 1 CA-s) discovered. -I- Links With links width != 4x (as set by -lw option) -I--------------------------------------------------- -I- No unmatched Links (with width != 4x) were found...

  • Page 184: Monitor The Spine Switch Using Web Interface

    The Fabric Monitor is displayed. 14.8 What Next? After setting up the Sun Datacenter InfiniBand Switch 36 in a multirack configuration scenario, you can proceed to monitor and control the InfiniBand fabric. 14-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 185: Monitor And Control The Infiniband Fabric

    Monitor and Control the InfiniBand Fabric This chapter describes how to monitor and control the InfiniBand fabric. It contains the following topics: • Monitor the InfiniBand Fabric • Control the InfiniBand Fabric • For More Information 15.1 Monitor the InfiniBand Fabric This section contains the following topics: •...

  • Page 186: Display The Infiniband Fabric Topology

    On the command-line interface (CLI), enter the following command: # ibnetdiscover The output is displayed, as in the following example: # Topology file: generated on Sat Apr 13 22:28:55 2002 # Max of 1 hops discovered 15-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 187: Display A Route Through The Fabric

    Monitor the InfiniBand Fabric # Initiated from node 0021283a8389a0a0 port 0021283a8389a0a0 vendid=0x2c9 devid=0xbd36 sysimgguid=0x21283a8389a0a3 switchguid=0x21283a8389a0a0(21283a8389a0a0) Switch 36 "S-0021283a8389a0a0" # "Sun DCS 36 QDR switch localhost" enhanced port 0 lid 15 lmc 0 [23] "H-0003ba000100e388"[2](3ba000100e38a) # "nsn33-43 HCA-1" lid 14 4xQDR vendid=0x2c9 devid=0x673c sysimgguid=0x3ba000100e38b...

  • Page 188: Display The Link Status Of A Node

    LID of the node in the fabric, and port is the port of the node. Note: If a port value of 255 is specified for a switch node, the counters are the total for all switch ports. 15-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 189: Display Data Counters For A Node

    Monitor the InfiniBand Fabric For example: # perfquery 15 23 # Port counters: Lid 15 port 23 PortSelect:......23 CounterSelect:....0x1b01 SymbolErrors:....0 VL15Dropped:.....0 XmtData:......20232 RcvData:......20232 XmtPkts:......281 RcvPkts:......281 Note: The output in the example is just a portion of the full output. 15.1.7 Display Data Counters for a Node To list the data counters for a node in the fabric, use the ibdatacounts command.

  • Page 190: Display Low-Level Detailed Information For A Port

    # smpquery portinfo 15 23 Mkey:......0x0000000000000000 GidPrefix:.......0x0000000000000000 Lid:......0x0000 SMLid:......0x0000 CapMask:......0x0 DiagCode:......0x0000 MkeyLeasePeriod:....0 LocalPort:.......0 LinkWidthEnabled:....1X or 4X LinkWidthSupported:....1X or 4X LinkWidthActive:....4X LinkSpeedSupported:....2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkState:.......Active PhysLinkState:....LinkUp LinkDownDefState:....Polling ProtectBits:.....0 LMC:......0 SubnetTimeout:....0 15-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 191: Map Lids To Guids

    Monitor the InfiniBand Fabric RespTimeVal:.....0 LocalPhysErr:....8 OverrunErr:......8 MaxCreditHint:....85 RoundTrip:.......16777215 Note: The actual output for your InfiniBand fabric will differ from that in the example, and it is just a portion of the full output. 15.1.10 Map LIDs to GUIDs In the InfiniBand fabric in Exalogic machines, as a Subnet Manager and Subnet administrator, you may want to assign subnet-specific LIDs to nodes in the fabric.

  • Page 192: Perform Comprehensive Diagnostics For The Entire Fabric

    Command Line Arguments Done with args Flow = Create Inventory Aug 13 19:44:53 601222 [B7D466C0] 0x7f -> Setting log level to: 0x03 Aug 13 19:44:53 601969 [B7D466C0] 0x02 -> osm_vendor_init: 1000 pending umadsspecified 15-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 193 Monitor the InfiniBand Fabric using default guid 0x21283a8620b0f0 Aug 13 19:44:53 612312 [B7D466C0] 0x02 -> osm_vendor_bind: Binding to port0x21283a8620b0f0 Aug 13 19:44:53 636876 [B7D466C0] 0x02 -> osmtest_validate_sa_class_port_info: ----------------------------- SA Class Port Info: base_ver:1 class_ver:2 cap_mask:0x2602 cap_mask2:0x0 resp_time_val:0x10 ----------------------------- OSMTEST: TEST "Create Inventory" PASS After an event, compare the present topology to that saved in the inventory file, as in the following example: # osmtest -f v...

  • Page 194: Determine Which Links Are Experiencing Significant Errors

    # Checking Ca: nodeguid 0x0003ba000100e388 Node check lid 14: OK Port check lid 14 port 2: OK ## Summary: 5 nodes checked, 0 bad nodes found ## 10 ports checked, 0 ports with bad state found 15-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 195: Control The Infiniband Fabric

    Control the InfiniBand Fabric Note: The ibcheckstate command requires time to complete, depending upon the size of your InfiniBand fabric. Without the -v option, the output contains only failed ports. The output in the example is only a small portion of the actual output.

  • Page 196: Set Port Speed

    # ibportstate 15 23 speed 7 Initial PortInfo: # Port info: Lid 15 port 23 LinkSpeedEnabled:....2.5 Gbps After PortInfo set: # Port info: Lid 15 port 23 LinkSpeedEnabled:....2.5 Gbps or 5.0 Gbps or 10.0 Gbps 15-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 197: Disable A Port

    Control the InfiniBand Fabric 15.2.5 Disable a Port If a port is found to be problematic due to a bad cable connection or a physical damage to the connectors, you can disable the port. On the command-line interface (CLI), run the following command: # disableswitchport [--reason=reason] connector|ibdev port where reason is the reason for disabling the port, Blacklist or Partition.

  • Page 198: Enable A Port

    # Port info: DR path slid 65535; dlid 65535; 0 port 7 LinkState:.......Down PhysLinkState:....Polling 15.3 For More Information For more information about Sun Network QDR InfiniBand Gateway Switches, see the product documentation at the following URL: http://download.oracle.com/docs/cd/E19671-01/index.html 15-14 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 199: Use Infiniband Partitions In Exalogic Physical Environments

    Exalogic's InfiniBand fabric in the Exalogic physical environment. Note: If you are connecting your Exalogic machine to Oracle Exadata Database Machine on the same InfiniBand fabric, you must use the default partition for data traffic between Exalogic machine and Oracle Exadata Database Machine.

  • Page 200: Understand Partition Keys

    For more information about hybrid racks, see the Exalogic Elastic Cloud Release Notes. On a hybrid Exalogic rack, Exalogic Control ensures that a unique pkey is assigned to each partition in the virtual environment.

  • Page 201: Before You Begin

    Before You Begin Select a list that is preferably near the upper end of the 0x0001–0x7ffe range. For example, if you identify 0x7000 to 0x7ffe as you range, you can create up to 4096 partitions. We recommend the upper end, because Exalogic Control assigns pkeys starting from lower end—that is, 0x0001.
  • Page 202 Bridge-0 1 Bridge-0-1 4 0x002128548062c001 0x0015 Active Up Bridge-0 2 Bridge-0-2 3 0x002128548062c002 0x000d Active Up Bridge-1 1 Bridge-1-1 2 0x002128548062c041 0x000f Active Up Bridge-1 2 Bridge-1-2 1 0x002128548062c042 0x0010 Active Up 16-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 203: Identify All Infiniband Switches In The Fabric

    Move from a Default Partition to a Custom Partition Tip: In an Exalogic machine full rack, compute nodes 1 to 15 (start from the bottom of the rack) connect their InfiniBand port 1 to gateway switch1 and their InfiniBand port2 to gateway switch 2. Similarly, compute nodes 16 to 30 are connected to gateway switches 3 and 4.

  • Page 204: Create An Ipoib Partition And Adding Ports

    Create an IPoIB Partition and Adding Ports Note: Do not complete this step if your Exalogic machine is connected to Oracle Exadata Database Machine on the same InfiniBand fabric. See the following example: Run the following command to start the process:...
  • Page 205 Create an IPoIB Partition and Adding Ports In this example, BridgeXPort1, BridgeXPort2, BridgeXPort3, and BridgeXPort4 are the BridgeX ports that you want to add to the partition. Follow these steps to add the ibp0 and the ibp1 network device port GUIDs to the partition: SSH to the storage appliance and run the following commands to determine the port GUID of the ibp0 network device:...
  • Page 206 Mount the share from the remote device. Use the IP specified in the configuration step. In our example we used 192.168.33.15. # mount 192.168.33.15:/export/share_name /mnt/share_name Repeat the previous steps for all compute nodes that are members of the partition. 16-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 207: Delete A Partition

    Delete a Partition 16.6 Delete a Partition You can delete a non-default partition by running the following command: # smpartition delete -n myIPoIB This command deletes the myIPoIB partition. Note: Do not attempt to delete the default partition. 16.7 Create a Partition for EoIB and Associating the pkey with a VNIC and VLAN You can create a partition for EoIB (both inbound and outbound) and associate the partition's pkey with a VLAN and VNIC on the edge network.
  • Page 208 You can also specify the bonded IPoIB address of the compute node. Example: el01cn01# iblinkinfo.pl -R | grep el01cn01 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 2[ ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps) 16-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 209 Create a Partition for EoIB and Associating the pkey with a VNIC and VLAN 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 1[ ] "el01cn01 EL-C 192.168.10.29 HCA-1" (Could be 5.0 Gbps) From the output of the iblinkinfo command, note the switch lid value (65, in first column) associated with the Base lid of the compute node port that you noted earlier (121, in the first line): Determine the gateway switch that corresponds to the switch lid 65 by running...
  • Page 210 GUID, a2:c0:a0:a8:01:01 is the dummy MAC address defined in Step 4, 10 is the VLAN identifier, and 0x005 is the partition key that you created earlier. This example creates a VNIC, such as eth4 (on Oracle Linux) or eoib0 (on Oracle Solaris) associated with VLAN 10 associated with a partition with 0x005 as the pkey.
  • Page 211 MAC address on the compute node. For example, to verify the new interface and its MAC address, run the following command on the Oracle Linux compute node for which the VNIC was created: # ifconfig -a eth4 The output of this command shows the HWADDR, which is the MAC address you defined for the VNIC in Step 5.
  • Page 212 MAC address for the HWADDR directive. After creating the interface files, create the ifcfg-bond1 file. If the file already exists, verify its contents. Example: # more /etc/sysconfig/network-scripts/ifcfg-bond1 DEVICE=bond1 IPADDR=192.168.48.128 NETMASK=255.255.255.0 16-14 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 213: Perform The Post-Configuration Steps

    16.9 Important Notes for Combined Exalogic-Exadata Fabric Users Read the following notes if you are using partitions in a scenario where your Exalogic machine is connected to an Oracle Exadata Database Machine on the same InfiniBand fabric: •...

  • Page 214: Partitioning Limitations

    Exalogic nodes as limited members of the default partition will not be able to communicate with any other Exalogic node in the default partition. However, client access to Oracle Exadata Database Machine is provided via IPoIB in the default partition.

  • Page 215: Monitoring The Exalogic Machine Using Oracle Enterprise Manager Ops Center

    Exalogic. The information in this chapter is intended to be a reference model, to help you understand how an external, standalone installation of an Oracle product like Enterprise Manger Ops Center 11g or Enterprise Manager Cloud Control can be used to manage the assets in an Exalogic machine in a physical configuration.

  • Page 216: Overview

    Using Reports in Oracle Enterprise Manager Ops Center • Using Oracle Services in Oracle Enterprise Manager Ops Center 17.1 Overview Oracle Enterprise Manager 11g Ops Center can optionally be used to monitor the following components in the Exalogic machine infrastructure: • Exalogic compute nodes •...

  • Page 217: Prerequisites

    You can access Oracle Enterprise Manager Ops Center documentation at the following URL: http://docs.oracle.com/cd/E11857_01/nav/management.htm 17.5 Launching Oracle Enterprise Manager Ops Center When you launch Oracle Enterprise Manager Ops Center in a web browser, the login page is displayed. After successful login, the home page is displayed, as in Figure 17-1.

  • Page 218: Understanding The Workflow

    Understanding the Workflow Figure 17-1 Oracle Enterprise Manager Home Page 17.6 Understanding the Workflow Figure 17-2 illustrates the workflow for the management of Exalogic machine hardware components using Oracle Enterprise Manager Ops Center. 17-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 219: Management Of Exalogic Machine Hardware Using Oracle Enterprise Manager Ops Center

    Understanding the Workflow Figure 17-2 Management of Exalogic Machine Hardware Using Oracle Enterprise Manager Ops Center Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-5...

  • Page 220: Managing Users And Roles

    Managing Users and Roles 17.7 Managing Users and Roles You can add users to Oracle Enterprise Manager Ops Center from the local authentication subsystem of the Enterprise Controller's operating system. Each user can be given a different role which grants or denies access to the different functions of Oracle Enterprise Manager Ops Center.

  • Page 221: Custom Asset Discovery Screen

    Sun Network QDR InfiniBand Gateway Switches • Sun Datacenter InfiniBand Switch 36 Note: This switch is used in multirack configuration scenarios only. It is not connected in a Exalogic machine single rack configuration. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-7...
  • Page 222 Managed Assets icon. The selected asset is now listed under the Managed Assets tab on the home page. Note: In addition to discovering hardware assets, you may discover operating systems running on the host compute nodes. 17-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 223: Grouping Exalogic Machine Hardware Assets

    In the Configure Group screen, in the Group Name field, enter a name for the new Exalogic group. For example, enter Exalogic. Enter a description in the Description field. Select the Top Level (Root) option as Location. See Figure 17-4. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-9...

  • Page 224: Configure Group

    Exalogic rack. For example, you can enter Exalogic el01 as a group name. After entering the group information and selecting the location, click Next. The Summary screen is displayed, as in Figure 17-5. 17-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 225: Creating A Sub-Group For Exalogic Compute Nodes

    Exalogic group. For example, enter ComputeNodes. Enter a description in the Description field. Select the Inside a user-defined Group/Subgroup option as Location. In the Group Selection Panel, select the Exalogic top-level group that Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-11...

  • Page 226: Creating A Sub-Group For The Storage Appliance

    After creating a top-level group for the Exalogic machine, you can create a sub-group for the storage appliance, which is included in the Exalogic machine. You can create a sub-group for this storage appliance as follows: 17-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 227: Configure Sub-Group For Storage Appliance

    Figure 17-7. Figure 17-7 Configure Sub-Group for Storage Appliance After entering the sub-group information and selecting the group location, click Next. The Summary screen is displayed. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-13...

  • Page 228: Creating A Sub-Group For Infiniband Switches

    Description field. Select the Inside a user-defined Group/Subgroup option as Location. In the Group Selection Panel, select the Exalogic top-level group that you created in Creating the Exalogic Top-Level Group. The Configure Group screen should be similar to Figure 17-8. 17-14 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 229: Adding Assets To A Group

    To add an Exalogic-specific managed asset to a group, do the following: On the All Assets page, click the Managed Assets tab. Exalogic-specific managed assets are listed under All Managed Assets. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-15...

  • Page 230: Viewing Exalogic Compute Nodes

    On the left navigation pane, under Assets, from the drop-down list, select the All User Defined Groups option. Select the Exalogic top-level group. The Exalogic group page is displayed, as in Figure 17-9. 17-16 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 231: Exalogic Group View

    Click the Hardware tab to view detailed information about that compute node. This information includes CPU, memory, network adapter, disk, power supply, and fan tray (fan). For example, see Figure 17-10. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-17...

  • Page 232: Compute Node Hardware Information

    Console tab. If your console connection is not enabled, enable it by clicking the Enable the Console Connection icon. Click the Connect to the Console icon. The SP console for the selected compute node is launched, as shown in Figure 17-11. 17-18 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 233: Viewing Infiniband Switches

    On the left navigation pane, select a particular switch under the IBswitches sub-group to view information about that switch. You should see a page similar to the compute node page shown in Figure 17-12. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-19...

  • Page 234: Switch View

    Click the Hardware tab to view detailed information about that switch. This information includes ports, network adapters connected to the switch, and so on. The hardware information about the switch is displayed, as in Figure 17-13. 17-20 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 235: Switch Hardware Information

    On the right navigation pane named Actions, click the Launch LOM Controller option to verify that the Sun Integrated Lights Out Manager (ILOM) web interface for the switch can be launched. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-21...

  • Page 236: Viewing The Storage Appliance

    Figure 17-14. Figure 17-14 Storage Appliance View Click the Hardware tab to view detailed information about the storage appliance. This information includes CPU, memory, network adapters, disks, and so on. 17-22 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 237: Storage Appliance Hardware View

    Exalogic machine. Click the Storage Shares tab to view the shares (exported file systems) configured on the storage appliance, as shown in Figure 17-16. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-23...

  • Page 238: Viewing The Infiniband Fabric And Its Nodes

    Figure 17-16 Viewing Shares Configured on the Storage Appliance If you wish to launch the administration console for the storage appliance from within the Oracle Enterprise Manager Ops Center interface, click the Terminal tab. Note: You can also launch the storage appliance UI, detailed dashboard, analytics,...

  • Page 239: Fabric View

    Similarly, on the left navigation pane, under the fabric drop-down list, click a gateway switch. Information about the switch as a fabric node is shown, as shown Figure 17-18. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-25...

  • Page 240: Viewing A Switch In The Fabric

    Viewing the InfiniBand Fabric and Its Nodes Figure 17-18 Viewing a Switch in the Fabric To monitor switch ports, click the Port Monitoring tab. Monitoring information about the switch ports is displayed, as shown in Figure 17-19. 17-26 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 241: Switch Port Monitoring

    Similarly, on the left navigation pane, under the fabric drop-down list, click the storage appliance. Information about the storage appliance as a fabric node is shown, as shown in Figure 17-20. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-27...

  • Page 242: About Problem Management

    Viewing the Storage Appliance in the Fabric If you wish to launch the administration console for any of the fabric nodes (compute nodes, storage appliance, or gateway switches) from within the Oracle Enterprise Manager Ops Center interface, click the Terminal tab in the respective fabric dashboard pages.

  • Page 243: Using Monitoring Profiles And Rules

    Enterprise Manager Ops Center 11g documentation. 17.15.1 Creating a Monitoring Profile You can create a new monitoring profile and edit the profile to add rules. To create a monitoring profile, do the following: Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-29...

  • Page 244: Adding A Monitoring Rule From The Asset View

    Complete the Alert parameters for the different severity levels. Use the Immediate Action field to define what action should take place when a problem is detected. Click Apply to save the rule. The new rule will appear in the profile. 17-30 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 245: Using Reports In Oracle Enterprise Manager Ops Center

    Oracle Enterprise Manager Ops Center 11g documentation. This topic describes the following tasks: – Creating charts – Creating a system information report – Updating reports – Creating problem reports Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-31...

  • Page 246: Using Oracle Services In Oracle Enterprise Manager Ops Center

    Oracle.com account, go to https:// myprofile.oracle.com/EndUser/faces/profile/createUser.jspx? tid=minimal&nextURL=https://support.oracle.com:443/CSP/ui/flash.html?login. To determine if you are running in Connected Mode and have access to My Oracle Support, view the following connection icons in the upper right corner of the UI: • World icon indicates Internet connection •...

  • Page 247: Viewing Service Requests

    Opened by Others to display a list of requests. To view details of a particular service request, highlight a row, then click the View Service Request icon. The following graphic shows an example service request. Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17-33...
  • Page 248 Using Oracle Services in Oracle Enterprise Manager Ops Center 17-34 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 249: Filing A Service Request

    Requests tab in the content pane. 17.17.3 Filing a Service Request When your assets are associated with a contract and registered in the Oracle database, you can create and file a service request in Ops Center from a problem or from an asset.
  • Page 250 Using Oracle Services in Oracle Enterprise Manager Ops Center 17-36 Machine Owner's Guide...

  • Page 251: Install The Asr Software

    Oracle Support Services that provides auto-case generation when common hardware component faults occur. ASR is designed to enable faster problem resolution by eliminating the need to initiate contact with Oracle Support Services for common hardware component failures, reducing both the number of phone calls needed and overall phone time required.

  • Page 252: Recommended Configuration

    Exalogic compute nodes, on a system outside of Exalogic. This machine can also be a virtual machine that runs Oracle Solaris or Oracle Linux. Although it is possible to install ASR Manager on one of the Exalogic compute nodes, it is recommended that you install it outside of Exalogic.

  • Page 253: Install Sasm Package

    To install the SASM package, complete the following steps: Run the following command, as a root user, to verify you have version 1.2.1 or later: • On Oracle Linux: # rpm -q SUNWsasm • On Oracle Solaris: # pkginfo -l SUNWsasm If required, download SASM 1.2.1 from the following URL:...

  • Page 254: Register The Asr Manager

    To activate the storage appliance included in your Exalogic machine, complete the following steps: Ensure that you have a valid My Oracle Support Account. Ensure that you can connect to the Internet from your storage appliance using either a direct connection or a web proxy.

  • Page 255: Phone Home Tab On The List Of Services

    Activate the Storage Appliance Figure 18-1 Phone Home Tab on the List of Services When you click Phone Home, the Phone Home page is displayed, as shown in Figure 18-2. Install the ASR Software 18-5...

  • Page 256: Approve And Verify Asr Activation For Exalogic Machine Assets

    Phone Home Settings Page In the My Oracle Support Username field, enter your username. In the Password field, enter the password for your My Oracle Support account. If you are using a web proxy to connect to the Internet from the storage appliance, select the Use web proxy option, and enter the following information: •...

  • Page 257: Approve Exalogic Machine Assets In My Oracle Support

    ASR Activation - Asset Screen in My Oracle Support Click Approve to complete activation. Note: An Exalogic machine asset must be in an active ASR state in My Oracle Support in order for Service Request autocreate to work. 18.9.2 View and Verify ASR Assets...
  • Page 258 This command should list ASR assets in your Exalogic machine, including compute nodes and storage server heads. Log in to My Oracle Support. You must see the same set of ASR assets with their correct serial numbers. 18-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 259: A Configure The Exalogic Machine Using Ecu

    Configure the Exalogic Machine Using ECU This appendix provides an overview of the Exalogic Configuration Utility (ECU). It contains the following topics: • Overview • Important Notes Before You use the ECU • Configuration Tasks A.1 Overview The ECU enables you to perform the initial configuration of an Exalogic machine and to connect the machine to your existing network.

  • Page 260: Configuration Tasks

    Configure Ethernet Over InfiniBand. A.3 Configuration Tasks Oracle recommends strongly that the configuration tasks using ECU be performed by fully trained, qualified Oracle personnel or by formally accredited Oracle partners. For more information, contact Oracle Advanced Customer Support (ACS): Email: acsdirect_us@oracle.com...

  • Page 261: B Site Checklists

    Site Checklists Complete the checklists in this appendix to ensure that the site is prepared for installing and deploying Exalogic machines. This appendix contains the following topics: • System Components Checklist • Data Center Room Checklist • Data Center Environment Checklist •...

  • Page 262: Data Center Room Checklist

    Have cabinet stabilization measures been considered? Does the raised floor satisfy the weight requirements for the new hardware? Can floor tiles be removed without permission to accommodate service? Are there cable routing channels or cutouts? B-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 263: Data Center Environment Checklist

    Data Center Environment Checklist Data Center Room Checklist Items Comment Are you providing any additional hardware? Is the hardware you are providing fully compatible with the Exalogic machine? Will the new hardware location require any non-standard cable lengths? Is the floor to ceiling height a minimum of 2914 mm or 2.9 m (9.6 feet)? Is the depth of the raised floor a minimum of 46 cm (18 inches)?
  • Page 264 • The pallet jack supports the device weight? • The pallet jack tines are compatible with the shipping pallet? If there are stairs, then is a loading elevator accessible for the equipment? B-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 265: Facility Power Checklist

    Facility Power Checklist Access Route Considerations Comment If an elevator is to be used, then have you confirmed the following: • The elevator car is wide enough for the device to be carried into it? • The elevator car is high enough for the device to be carried into it? •...

  • Page 266: Power Checklist

    (kW) /kilovolt (kVA) to express power load. B.6 Power Checklist Complete the following checklist to ensure that the power requirements are met for Sun Oracle Database Machine. For information about power requirements, see Electrical Power Requirements. Power Checklist Considerations...

  • Page 267: Logistics Checklist

    Logistics Checklist Safety Checklist Considerations Comment Is there a fire protection system in the data center room? Is the computer room adequately equipped to extinguish a fire? Is antistatic flooring installed? Is the floor below the raised floor free of obstacles and blockages? B.8 Logistics Checklist Complete the following checklist to ensure that the logistics requirements are met.

  • Page 268: Network Specification Checklist

    B.9 Network Specification Checklist Complete the following checklist to ensure that the network specification requirements are met. For information about IP addresses, see Understand Network Requirements and Configuration. B-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 269 Network Specification Checklist Note: • If you are deploying multiple Exalogic machines that will be connected to form a single system, then contact your Oracle representative for instructions. • By default, there is one default InfiniBand partition at the Exalogic machine level.

  • Page 270: Reracking Checklists

    Exalogic full racks cannot be reracked due to potential air flow and cooling issues. Note: • Customer must purchase both the Oracle Reracking Service and Oracle Installation Service. • Oracle does not provide support for customer-supplied equipment.
  • Page 271 If the rack is less than 600 mm wide, then it must have additional depth to accommodate mounting behind the server CMAs. Does the target rack support Oracle cable management arms (CMAs)? Does the target rack support installation of Oracle vented and...

  • Page 272: Rack Recycling

    If Oracle is recycling the rack, then has the customer completed the appropriate application at the Oracle Electronic Waste and Product Return website? The website is at http://www.oracle.com/us/ products/applications/green/ waste-and-product- returns-185031.html B-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 273: C Cabling Diagrams

    Exalogic Full Rack NOT_SUPPORTED: For information about connecting an Exalogic machine with one or more Exalogic or Exadata machines, see the Exalogic Elastic Cloud Multirack Cabling Guide. C.1 Exalogic Eighth Rack Figure C-1 illustrates the cabling for a single Exalogic eighth rack.

  • Page 274: Exalogic Quarter Rack

    This connection is for high availability and load distribution purposes. Exalogic machine eighth racks do not contain the Sun Datacenter InfiniBand Switch C.2 Exalogic Quarter Rack Figure C-2 illustrates the cabling for a single Exalogic quarter rack. C-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 275: Exalogic Half Rack

    Exalogic Half Rack Figure C-2 Exalogic Quarter Rack Network Connectivity Figure C-2, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches). Servers refers to compute nodes in the Exalogic machine. Half of the compute nodes are connected with their active ports to the first gateway switch and their passive ports to the second gateway switch.

  • Page 276: Exalogic Full Rack

    C-4, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches). Servers refers to compute nodes in the Exalogic machine. The Sun Datacenter InfiniBand Switch 36, referred to as spine switch, is not connected. C-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 277: Rack-Level Frus For Exalogic X4-2

    When Oracle Services personnel visit your site for repairing and replacing hardware, you should make the spares kit available to them. Note that any parts used by Oracle Service personnel from the spares kit will be replenished by Oracle.

  • Page 278: Rack-Level Replacement Parts For El X5-2

    All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty. Table D-2 Rack-Level Replacement Parts for EL X4-2 Part Number Description Coverage under Basic Warranty 7079238 Base Rack D-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 279: Rack-Level Frus For Exalogic X2-2 And X3-2

    Rack-Level FRUs for Exalogic X2-2 and X3-2 Table D-2 (Cont.) Rack-Level Replacement Parts for EL X4-2 Part Number Description Coverage under Basic Warranty 7015400 JUMPER CABLE, 2 M, C13-C14 530-4445-01 3M, 10GE QSFP Passive Copp, CBL 530-4502-01 CBL, SHIELD RJ45 CAT5E, 1M 530-4436-01 CBL, 7FT ETHERNET(CAT5/CAT5E) BLACK 530-4526-01...

  • Page 280: Parts For Zs3-Es Storage Appliance For Exalogic X5-2

    ASSY, SSD, 1.6TB, 2.5, SAS2,Readzilla 7066874 FRU, DR, 900GB, SAS2, 10K Marlin 541-2732-03 ASSY, DVD FILLER W/USB, BZL, TRAY 7046442 DUAL PORT CX3 4XQDR PCI-E, LP 7048710 ASSY, RISER, X8, 1U, PCI-E D-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 281: Parts For Zs3-Es Storage Appliance For Exalogic X4-2

    Parts for ZS3-ES Storage Appliance for Exalogic X4-2 Table D-4 (Cont.) Replacement Parts for Oracle ZS3-ES Appliance Part Number Description Coverage under Basic Warranty 7048707 ASSY, RISER, X16, 1U, PCI-E 7027478 ASSY, 2.5 DISK BP 7013526 ASSY, FAN MODULE, CR, 40X56...

  • Page 282: Parts For Sun Zfs Storage 7320 Appliance

    1 x Intel Westmere E5620 F541-2545 Heatsink, CPU, X4170, ATO F371-4898 4GB DDR3-1333 Registered ECC Memory (1x4GB) F371-4743 4GB,SLC,USB-2,SBC (USB Thumb Drive) F542-0371 500GB 7.2K 2.5 inch SATA HDD (Silver Marlin Bracket, OS drive) D-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 283: Parts For Oracle Server X5-2 Compute Nodes

    Sun ZFS Storage 7x20 Appliance Customer Service Manual. D.7 Parts for Oracle Server X5-2 Compute Nodes Table D-7 lists the replaceable parts for the Oracle Server X5-2 compute nodes used in the Exalogic machine. Note: All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty.

  • Page 284: Parts For Sun Server X4-2 Compute Nodes

    Parts for Sun Server X4-2 Compute Nodes Table D-7 (Cont.) Replacement Parts for Oracle Server X5-2 Compute Nodes Part Number Description Coverage with Basic Warranty 7098505 MOTHER BOARD ASSEMBLY 7050794 ASSY, BBU08 BATTERY MODULE, 55 C 7057184 ASSY, MOUNT-BATTERY, REMOTE...

  • Page 285: Parts For Sun Server X3-2 Compute Nodes

    Parts for Sun Server X3-2 Compute Nodes Table D-8 (Cont.) Replacement Parts for Sun Server X4-2 Compute Nodes Part Number Description Coverage with Basic Warranty 7027478 ASSY, 2.5 DISK BP 7018701 DIMM, 16GB, DDR3, 1600, 2Rx4, 1.35V 7038488 FLASH DRIVE, TLC, OSA ID, USB, 4GB 7047503 ASY, 6G SAS RAID HBA, INT, B4, 8P 7057184...

  • Page 286: Parts For X4170 M2 Compute Nodes

    Replacement Parts for X4170 M2 Compute Nodes Part Description Coverage Number with Basic Warranty F371-4889 1 x Intel Westmere X5670 F541-2545 Heatsink, CPU, X4170, ATO F371-4966 8 GB DDR3-1333 Low Voltage Registered ECC Memory (1 x 8GB) D-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 287: Parts For Sun Network Qdr Infiniband Gateway Switch For Exalogic X5-2

    Parts for Sun Network QDR InfiniBand Gateway Switch for Exalogic X5-2 Table D-10 (Cont.) Replacement Parts for X4170 M2 Compute Nodes Part Description Coverage Number with Basic Warranty F375-3696 Dual port 4X QDR InfiniBand PCI-E HCA (CX2) F375-3701 6 GB/s SAS RAID HBA,Internal F371-4982 Battery for 6 GB/s SAS RAID HBA,Internal (BBU08) F540-7841...

  • Page 288: Parts For Sun Network Qdr Infiniband Gateway Switch For Exalogic X4-2

    D.13 Parts for Sun Network QDR InfiniBand Gateway Switch for Exalogic X2-2 and X3-2 Table D-13 lists the replaceable parts for the Sun Network QDR InfiniBand Gateway Switches used in Exalogic X2-2 and X3-2 machines. D-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 289: Replacement Parts For Sun Network Qdr Infiniband Gateway Switch For Exalogic X2-2 And X3-2 Machines

    Parts for Sun Datacenter InfiniBand Switch 36 for Exalogic X5-2 Note: All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty. Table D-13 Replacement Parts for Sun Network QDR InfiniBand Gateway Switch for Exalogic X2-2 and X3-2 Machines Part Number Description...

  • Page 290: Replacement Parts For The Cisco Catalyst 4948E-F-S Switch

    Cisco Catalyst 4948E-F-S switch. Table D-17 Replacement Parts for the Cisco Catalyst 4948E-F-S Switch Part Description Repair Coverage Number Categor with Basic Warranty 7023685 Cisco C4948E-F-S switch without the power supply and fan D-14 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 291: Parts For The Cisco Catalyst 4948E-F-S Switch

    Parts for the Cisco Catalyst 4948 Switch Table D-17 (Cont.) Replacement Parts for the Cisco Catalyst 4948E-F-S Switch Part Description Repair Coverage Number Categor with Basic Warranty 7024423 Power supply for Cisco 4948E-F-S switch 7024424 Cooling fan for Cisco 4948E-F-S switch For repair information, see the Cisco Catalyst 4948E Switch Installation Guide.
  • Page 292 ASSY, 2M, 4X MINI SAS CBL, SHLD 530-3886 ASSY, 0.5M, 4X MINI SAS CBL, SHLD 7044619 ASSY, DE2-24P UNIVERSAL RAIL KIT 7044396 ASSY DE2-24P SAS 73GB SSD, LogZ 7066831 Assy, DRIVE, 4T 3.5", SAS/7200 rpm, DE2-24C D-16 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 293: E Cabling Tables

    Cabling Tables The tables in this appendix show the cable layouts for Exalogic machines. This appendix contains the following topics: • Administrative Gigabit Ethernet Network-Cabling Tables • InfiniBand Network Cabling Tables • Power Distribution Unit Cabling Tables Note: The eighth-, quarter-, and half-rack X4-2 and newer machines are pre-cabled with a complete set (equivalent in number to a full rack) of InfiniBand (92), Ethernet (42), and power cables.

  • Page 294: Gigabit Ethernet Cabling For Exalogic Machine Full Rack

    InfiniBand Gateway Switch Sun Network QDR 7' Black Cat 5 NET0 InfiniBand Gateway Switch Storage appliance server head 7' Blue Cat 5 NET0 Storage appliance server head 7' Blue Cat 5 NET1 E-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 295 Administrative Gigabit Ethernet Network-Cabling Tables Table E-1 (Cont.) Gigabit Ethernet Cabling for Exalogic Machine Full Rack From Type of Equipment From Description Rack Slot Equipment Cisco Port Switch Port Storage appliance server head 7' Blue Cat 5 NET2 Storage appliance server head 7' Blue Cat 5 NET0 Storage appliance server head...

  • Page 296: Gigabit Ethernet Cabling For Exalogic Machine Half Rack

    10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5 NET0 E-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 297: Gigabit Ethernet Cabling For Exalogic Machine Quarter Rack

    Administrative Gigabit Ethernet Network-Cabling Tables Table E-2 (Cont.) Gigabit Ethernet Cabling for Exalogic Machine Half Rack From Type of Equipment From Description Rack Slot Equipment Cisco Port Switch Port Compute node 10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5...

  • Page 298: Gigabit Ethernet Cabling For Exalogic Machine Eighth Rack

    7' Black Cat 5 NET0 Gateway Switch Compute node 10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5 NET0 Compute node 10' Blue Cat 5 NET0 E-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 299: Exalogic Machine Full Rack

    InfiniBand Network Cabling Tables E.2 InfiniBand Network Cabling Tables This section contains the tables for the InfiniBand network cabling. The Sun Network QDR InfiniBand Gateway Switch is located in located in rack units 26, 24, 23, or 16 (depending on the rack configuration). This section contains the following topics: •...
  • Page 300 InfiniBand cable Compute node PCIe 2, P2 2 meter QDR InfiniBand cable Compute node PCIe 2, P2 2 meter QDR InfiniBand cable Compute node PCIe 2, P1 2 meter QDR InfiniBand cable E-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 301 InfiniBand Network Cabling Tables Table E-5 (Cont.) InfiniBand Network Cabling for Exalogic Machine Full Rack From Port To Rack Type of Port Description Rack Unit Unit Equipment (Sun Network InfiniBand Gateway Switch) Compute node PCIe 2, P1 2 meter QDR InfiniBand cable Compute node PCIe 2, P1...
  • Page 302 Storage appliance PCIe 0, P1 3 meter QDR server head InfiniBand cable Compute node PCIe 2, P2 2 meter QDR InfiniBand cable Compute node PCIe 2, P2 2 meter QDR InfiniBand cable E-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 303 InfiniBand Network Cabling Tables Table E-5 (Cont.) InfiniBand Network Cabling for Exalogic Machine Full Rack From Port To Rack Type of Port Description Rack Unit Unit Equipment (Sun Network InfiniBand Gateway Switch) Compute node PCIe 2, P2 2 meter QDR InfiniBand cable Compute node PCIe 2, P2...
  • Page 304 Sun Network 1 meter QDR QDR InfiniBand InfiniBand cable Gateway Switch Sun Network 1 meter QDR QDR InfiniBand InfiniBand cable Gateway Switch Sun Network 1 meter QDR QDR InfiniBand InfiniBand cable Gateway Switch E-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 305: Infiniband Gateway Switch Cross-Connections For Exalogic Machine Full Rack

    InfiniBand Network Cabling Tables Table E-7 lists the InfiniBand gateway switch cross-connections for an Exalogic machine full rack. Table E-7 InfiniBand Gateway Switch Cross-Connections for Exalogic Machine Full Rack From Port To Rack Type of Port Description Rack Unit Unit Equipment (Sun Network...

  • Page 306: Exalogic Machine Half Rack

    InfiniBand Network Cabling for Exalogic Machine Half Rack From Port To Rack Type of Port Description Rack Unit Unit Equipment (Sun Network InfiniBand Gateway Switch) Compute node PCIe 2, P1 2 meter QDR InfiniBand cable E-14 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 307 InfiniBand Network Cabling Tables Table E-8 (Cont.) InfiniBand Network Cabling for Exalogic Machine Half Rack From Port To Rack Type of Port Description Rack Unit Unit Equipment (Sun Network InfiniBand Gateway Switch) Compute node PCIe 2, P1 2 meter QDR InfiniBand cable Storage appliance PCIe 0, P2...
  • Page 308 InfiniBand cable Compute node PCIe 2, P1 2 meter QDR InfiniBand cable Compute node PCIe 2, P1 2 meter QDR InfiniBand cable Compute node PCIe 2, P1 3 meter QDR InfiniBand cable E-16 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 309: Infiniband Gateway Switch Inter-Connections For Exalogic Machine Half Rack

    InfiniBand Network Cabling Tables Table E-8 (Cont.) InfiniBand Network Cabling for Exalogic Machine Half Rack From Port To Rack Type of Port Description Rack Unit Unit Equipment (Sun Network InfiniBand Gateway Switch) Compute node PCIe 2, P1 3 meter QDR InfiniBand cable Compute node PCIe 2, P1...

  • Page 310: Exalogic Machine Quarter Rack

    InfiniBand cable Compute node PCIe 2, P2 3 meter QDR InfiniBand cable Compute node PCIe 2, P2 3 meter QDR InfiniBand cable Compute node PCIe 2, P2 3 meter QDR InfiniBand cable E-18 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 311 InfiniBand Network Cabling Tables Table E-10 (Cont.) InfiniBand Network Cabling for Exalogic Machine Quarter Rack From Port To Rack Type of Port Description Rack Unit Unit Equipment (Sun Network InfiniBand Gateway Switch) Compute node PCIe 2, P2 3 meter QDR InfiniBand cable Compute node PCIe 2, P2...

  • Page 312: Exalogic Machine Eighth Rack

    E.2.4 Exalogic Machine Eighth Rack This section contains the following tables: • Table E-12 • Table E-13 Table E-12 lists the location, ports and cables for the InfiniBand connections for an Exalogic machine eighth rack. E-20 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 313: Infiniband Network Cabling For Exalogic Machine Eighth Rack

    InfiniBand Network Cabling Tables Table E-12 InfiniBand Network Cabling for Exalogic Machine Eighth Rack From Port To Rack Type of Port Description Rack Unit Unit Equipment (Sun Network InfiniBand Gateway Switch) Storage appliance PCIe 0, P2 2 meter QDR server head InfiniBand cable Storage appliance PCIe 0, P2...

  • Page 314: Power Distribution Unit Cabling Tables

    • Single Phase PDU Power Cabling Full Rack • Single Phase PDU Power Cabling Half Rack • Single Phase PDU Power Cabling Quarter Rack • Single Phase PDU Power Cabling Eighth Rack E-22 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 315: Single Phase Pdu Power Cabling Full Rack

    Power Distribution Unit Cabling Tables This section contains the following topics for Three Phase PDUs: • Three Phase PDU Power Cabling Full Rack • Three Phase PDU Power Cabling Half Rack • Three Phase PDU Power Cabling Quarter Rack • Three Phase PDU Power Cabling Eighth Rack E.3.1 Single Phase PDU Power Cabling Full Rack This section contains the following tables:...

  • Page 316: Single Phase Pdu Power Cabling Half Rack

    E.3.2 Single Phase PDU Power Cabling Half Rack This section contains the following tables: • Table E-15 Table E-15 lists the location, ports and cables for the Single Phase PDU connections for an Exalogic machine half rack. E-24 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 317: Single Phase Pdu Power Cabling Quarter Rack

    Power Distribution Unit Cabling Tables Table E-15 Single Phase PDU Power Cabling for Exalogic Machine Half Rack From Rack Type of Equipment PDU-A PDU-B Jumper Length Unit Compute node G3-3 G2-3 Compute node G3-2 G2-4 Cisco Switch G3-1 G2-5 NM2-GW-SW G2-6 G3-0 ZS3-ES HN...

  • Page 318: Single Phase Pdu Power Cabling Eighth Rack

    From Rack Type of Equipment PDU-A PDU-B Jumper Length Unit Cisco Switch G3-1 G2-5 NM2-GW-SW G2-6 G3-0 ZS3-ES HN G2-5 G3-1 ZS3-ES HN G2-4 G3-2 DE2-24C DS G2-3 G3-3 NM2-GW-SW G1-6 G4-0 E-26 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 319: Three Phase Pdu Power Cabling Full Rack

    Power Distribution Unit Cabling Tables Table E-17 (Cont.) Single Phase PDU Power Cabling for Exalogic Machine Eighth Rack From Rack Type of Equipment PDU-A PDU-B Jumper Length Unit Compute node G0-4 G5-2 Compute node G0-3 G5-3 Compute node G0-2 G5-4 Compute node G0-1 G5-5...

  • Page 320: Three Phase Pdu Power Cabling Half Rack

    E.3.6 Three Phase PDU Power Cabling Half Rack This section contains the following tables: • Table E-19 Table E-19 lists the location, ports and cables for the Three Phase PDU connections for an Exalogic machine half rack. E-28 Oracle Exalogic Elastic Cloud Machine Owner's Guide...

  • Page 321: Three Phase Pdu Power Cabling Quarter Rack

    Power Distribution Unit Cabling Tables Table E-19 Three Phase PDU Power Cabling for Exalogic Machine Half Rack From Rack Type of Equipment PDU-A PDU-B Jumper Length Unit Compute node G3-4 G0-2 Compute node G3-3 G0-3 Cisco Switch G3-1 G0-5 NM2-GW-SW G2-5 G5-1 ZS3-ES HN...

  • Page 322: Three Phase Pdu Power Cabling Eighth Rack

    From Rack Type of Equipment PDU-A PDU-B Jumper Length Unit Cisco Switch G3-1 G0-5 NM2-GW-SW G2-5 G5-1 ZS3-ES HN G3-0 G0-6 ZS3-ES HN G2-4 G5-2 DE2-24C DS G2-3 G5-3 NM2-GW-SW G1-6 G4-0 E-30 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 323 Power Distribution Unit Cabling Tables Table E-21 (Cont.) Three Phase PDU Power Cabling for Exalogic Machine Eighth Rack From Rack Type of Equipment PDU-A PDU-B Jumper Length Unit Compute node G0-4 G3-2 Compute node G0-3 G3-3 Compute node G0-2 G3-4 Compute node G0-1 G3-5...
  • Page 324 Power Distribution Unit Cabling Tables E-32 Machine Owner's Guide...

  • Page 325: F Manage Solaris Zones On Exalogic

    Manage Solaris Zones on Exalogic Oracle Solaris zones are an integral part of the Oracle Solaris operating system. Zones isolate software applications and services using flexible software-defined boundaries. This appendix describes how to manage Solaris zones on Exalogic. This appendix contains the following sections: •...

  • Page 326: Create A Solaris Zone

    In the Alias field, enter a name for your iSCSI target. For the Initiator authentication mode, select the authentication mode you are using for communication between the compute node and storage appliance. By default, no authentication is used. F-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 327 For more information on setting up CHAP authentication between the compute node and storage, see the "Setting Up CHAP Authentication" topic in the following document: http://www.oracle.com/technetwork/ server-storage/sun-unified-storage/documentation/iscsi- quickstart-v1-2-051512-1641594.pdf From the Network interfaces list, select the interface that corresponds to your InfiniBand partition.
  • Page 328 In the Target Group field, select the target group you used in Create an iSCSI Target. In the Initiator Group field, select the initiator group you used in Create an iSCSI Initiator. F-4 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 329 Create a Solaris Zone Click Apply. Note the GUID of the LUN you created in the list of LUNs. For example, g600144f09c96cca900005190bfc4000a. Note: After creating the LUN, ensure that the Write cache enabled check box is deselected. You can find this check box in the Protocols tab of the LUN. F.3.1.4 Disable the Write Cache You must disable the write cache on the LUN permanently by doing the following: Log in to the compute node for which you identified the initiator node name as...
  • Page 330 Label the LUN by doing the following: Enter 6 to label the LUN. The format prompt appears. Enter label to label the disk as follows: format> label The list of label types appears. F-6 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 331 The zone you want to create should be given access to an exclusive network. You should create the necessary VNICs for the zone by doing the following: Create a VLAN and VNIC. Follow the procedure in Oracle Solaris: Creating VNICs and Associating Them with VLANs..

  • Page 332: Set Up A Solaris Zone

    Create the zone by running the following commands: zonecfg:zone04> set zonepath=/zones/zone04 zonecfg:zone04> add rootzpool zonecfg:zone04:rootzpool> add storage iscsi:// IPoIB_Address_of_the_storage_Appliance/luname.naa.LUNGUID zonecfg:zone04:rootzpool> end zonecfg:zone04> remove anet zonecfg:zone04> add net zonecfg:zone04:net> set physical=vnic1_name zonecfg:zone04:net> end F-8 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 333 Create a Solaris Zone zonecfg:zone04> add net zonecfg:zone04:net> set physical=vnic2_name zonecfg:zone04:net> end zonecfg:zone04> verify zonecfg:zone04> commit In this example: • IPoIB_address_of_the_storage_appliance is the IP address of the storage appliance on the IPoIB network. • LUNGUID is the GUID of the LUN you created in Create the Project and LUN.
  • Page 334 20130513T104657Z.zone04.install Boot up the zone by running the following command: # zoneadm -z zone04 boot Once the zone has booted up, log in to the zone using the zlogin command as follows: F-10 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 335 # zlogin zone04 [Connected to zone 'zone04' pts/7] Bond the VNICs you created as described in step 5 of Oracle Solaris: Creating VNICs and Associating Them with VLANs.. Run the following command to display the bond you created in the previous step:...

  • Page 336: Migrate A Zone To A New Host

    Log in to the compute node you want to migrate the zone to as the root user. Import the zone from the configuration file you created in the previous step by running the following command: # zonecfg -z name_of_zone -f directory/name_of_zone.cfg F-12 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 337 Migrate a Zone to a New Host Example: # zonecfg -z zone04 -f /common/general/zone04/zone04.cfg Attach the zone by running the following command: # zoneadm -z name_of_zone attach Example: # zoneadm -z zone04 attach Configured zone storage resource(s) from: iscsi://192.168.14.133/luname.naa.600144f09c96cca900005190bfc4000a Imported zone zpool: zone04_rpool Progress being logged to /var/log/zones/zoneadm.20130513T135704Z.zone04.attach Installing: Using existing zone boot environment Zone BE root dataset: zone04_rpool/rpool/ROOT/solaris...
  • Page 338 Migrate a Zone to a New Host F-14 Machine Owner's Guide...

  • Page 339: G Customize Linux On The Compute Nodes

    Customize Linux on the Compute Nodes This appendix describes how to customize the Linux operating system on compute nodes in an Exalogic machine, to suit your business needs, by installing additional RPMs, and updating or removing RPMs that are installed by default. It applies to Exalogic in a physical configuration, with the Linux operating system installed on bare metal.

  • Page 340: Prepare The Compute Nodes For Yum Updates

    To update a specific RPM, run the following command: # yum update rpm_name • To update all the RPMs, run the following command: # yum update • To remove an RPM, run the following command: G-2 Oracle Exalogic Elastic Cloud Machine Owner's Guide...
  • Page 341 Install, Update, and Remove RPMs Using Yum # yum remove rpm_name For the yum update or yum remove command, if you specify an RPM that is in the exclusion list defined earlier, the command will fail. Customize Linux on the Compute Nodes G-3...
  • Page 342 Install, Update, and Remove RPMs Using Yum G-4 Machine Owner's Guide...

Table of Contents