IBM TotalStorage DS4100 Installation, User's, And Maintenance Manual
  1. Manuals
  2. Brands
  3. IBM Manuals
  4. Network Storage Server
  5. TotalStorage DS4100
  6. Installation, user's, and maintenance manual

IBM TotalStorage DS4100 Installation, User's, And Maintenance Manual

Ibm switch users and maintenance guide
Hide thumbs Also See for TotalStorage DS4100:
Table of Contents

Advertisement

Quick Links

Download this manual
IBM TotalStorage DS4100 Storage Server
Installation, User's, and Maintenance Guide
GC26-7712-00

Advertisement

Table of Contents
loading

Related Manuals for IBM TotalStorage DS4100

Summary of Contents for IBM TotalStorage DS4100

  • Page 1 IBM TotalStorage DS4100 Storage Server Installation, User’s, and Maintenance Guide GC26-7712-00...
  • Page 3 IBM TotalStorage DS4100 Storage Server Installation, User’s, and Maintenance Guide GC26-7712-00...
  • Page 4 Note: Before using this information and the product it supports, be sure to read the general information in “Notices” on page 127. First Edition (November 2004) © Copyright International Business Machines Corporation 2004. All rights reserved. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
  • Page 5 An attention notice is placed just before the instruction or situation in which damage could occur. Before installing this product, read the following danger and caution notices. © Copyright IBM Corp. 2004 ® Safety Information document that is provided with your IBM ® DS4100 storage server. Each caution and danger statement is...
  • Page 6 2. First, attach all cables to devices. 3. Attach signal cables to connectors. 4. Attach power cords to outlet. 5. Turn device ON. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide To Disconnect: 1. Turn everything OFF. 2. First, remove power cords from outlet.
  • Page 7 Statement 3: CAUTION: When laser products (such as CD-ROMs, DVD drives, fiber optic devices, or transmitters) are installed, note the following: v Do not remove the covers. Removing the covers of the laser product could result in exposure to hazardous laser radiation. There are no serviceable parts inside the device.
  • Page 8 The device also might have more than one power cord. To remove all electrical current from the device, ensure that all power cords are disconnected from the power source. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...
  • Page 9 Statement 8: CAUTION: Never remove the cover on a power supply or any part that has the following label attached. Hazardous voltage, current, and energy levels are present inside any component that has this label attached. There are no serviceable parts inside these components.
  • Page 10 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 11: Table Of Contents

    Storage server operating environment ....13 DS4100 specifications ..... 14 Heat output, airflow, and cooling .
  • Page 12 Preparing the rack cabinet ....19 Installing the DS4100 ..... . 20 Rack mounting template .
  • Page 13 Installing SFPs and fiber-optic cables ....109 Adding a DS4000 EXP100 to an existing DS4100 configuration (base storage server only) ......109 Chapter 6.
  • Page 14 Index ......141 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 15: Figures

    ......28 17. Installing the DS4100 ......29 18.
  • Page 16 78. Cabling the DS4100 to a redundant loop ..... . 110 79. DS4100 parts list ......116 80.
  • Page 17 8. Features at a glance ......4 9. IBM TotalStorage DS4100 specifications ..... . 14 10.
  • Page 18 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 19: About This Document

    About this document This document provides instructions for installing and customizing the configuration of your IBM TotalStorage DS4100 storage server. It also provides hardware maintenance procedures and troubleshooting information. FAStT product renaming IBM is in the process of renaming some FAStT family products. Table 1 identifies each new DS4000 product name with its corresponding FAStT product name.

  • Page 20: How This Document Is Organized

    Appendix B, “Rack mounting template,” on page 121 provides the rack mounting templates for installation of the DS4100. If you want to tear out the templates from the document for use during installation, use these copies of the templates.

  • Page 21: Figures Used In This Document

    The figures used in this document are for illustrative purposes only. In some cases, the actual device might look different from the figure. Note: This applies particularly in cases where the DS4100 single-controller storage server differs from the DS4100 base storage server.

  • Page 22: Ds4000 Storage Server Publications

    DS4000 Storage Server publications The following tables present an overview of the DS4500, DS4400, DS4300 Fibre Channel, and DS4100 SATA Storage Server product libraries, as well as other related documents. Each table lists documents that are included in the libraries and what common tasks they address.
  • Page 23 Table 2. TotalStorage DS4500 storage server document titles by user tasks (continued) Title Planning Hardware Installation IBM FAStT Management Suite Java User’s Guide, 32P0081 IBM TotalStorage DS4000 Hardware Maintenance Manual, GC26-7702 IBM TotalStorage DS4000 Problem Determination Guide, GC26-7703 User Tasks Software Configuration Operation and...

  • Page 24: Ds4400 Storage Server Library

    Java User’s Guide, 32P0081 IBM TotalStorage DS4000 Hardware Maintenance Manual, GC26-7702 IBM TotalStorage DS4000 Problem Determination Guide, GC26-7703 xxii IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide User Tasks Software Configuration Installation Operation and Diagnosis and Administration Maintenance...

  • Page 25: Ds4300 Storage Server Library

    IBM TotalStorage FAST600 Fibre Channel Cabling Instructions, 24P8126 DS4100 storage server library Table 5 on page xxiv associates each document in the DS4100 (previously FAStT100) storage server library with its related common user tasks. User Tasks Software Configuration Installation Operation and...

  • Page 26: Totalstorage Ds4100 Storage Server Document Titles By User Tasks

    Table 5. TotalStorage DS4100 storage server document titles by user tasks Title Planning Hardware Installation IBM TotalStorage DS4100 Installation, User’s, and Maintenance Guide, GC26-7712 IBM TotalStorage DS4100 Fibre Channel Cabling Instructions, 25R0325 IBM TotalStorage FAStT FC2-133 Dual Port Host Bus Adapter Installation and User’s Guide,...

  • Page 27: Ds4000-Related Hardware Publications

    DS4000-related hardware publications Table 6 associates each of the following documents related to DS4000 (previously FAStT) operations with its related common user tasks. Table 6. TotalStorage DS4000-related document titles by user tasks Title Planning Hardware Installation IBM Safety Information, P48P9741 IBM TotalStorage FAStT Quick Start Guide, GC26-7662...

  • Page 28: Ds4000 Storage Manager Version 9 Publications

    Manager Version 9 Copy Services User’s Guide, GC26-7707 IBM TotalStorage FAStT Storage Manager Version 9 Concepts Guide, GC26-7661 xxvi IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide User Tasks Software Configuration Installation User Tasks Software Configuration Installation Operation and...

  • Page 29: Getting Information, Help, And Service

    Getting information, help, and service If you need help, service, or technical assistance or just want more information about IBM products, you will find a wide variety of sources available from IBM to assist you. This section contains information about where to go for additional information about IBM and IBM products, what to do if you experience a problem with your IBM Eserver xSeries service, if it is necessary.

  • Page 30: Software Service And Support

    Because IBM does not test any equipment for compatibility with fire suppression systems, IBM does not make compatibility claims of any kind nor does IBM provide recommendations on fire suppression systems. xxviii IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 31: How To Send Your Comments

    Table 9 on page 14 lists the environmental specifications for the DS4100. How to send your comments Your feedback is important in helping us to provide the most accurate and high-quality information. If you have comments or suggestions for improving this publication, you can send us comments electronically by using these addresses: v Internet: starpubs@us.ibm.com...
  • Page 32 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 33: Chapter 1. Introduction

    Note: SATA hard drive CRUs cannot be interchanged with the fibre-channel hard drive CRUs that are supported in other DS4000 storage products. The DS4100 can be used in storage area networks to satisfy the needs of various fixed content, data-reference applications that require large amounts of storage capacity but do not have the high utilization and access characteristics satisfied by fibre channel disk drive storage.

  • Page 34: About The Ds4100 Single-Controller Storage Server

    DS4000 EXP100 storage expansion units. Currently, the only DS4000 storage expansion unit model that can be attached to the DS4100 base storage server is the DS4000 EXP100. Do not connect other DS4000 storage expansion units (such as the DS4000 EXP700 or DS4000 EXP710) to the DS4100 base storage server.

  • Page 35: Fibre Channel Defined

    1. For the DS4100, each drive is considered to be a device in a fibre-channel loop, even though the DS4100 drive interface is SATA (not fibre channel). The DS4000 EXP100 ESM and the DS4100 controller and SATA hard drive interposer convert SATA interface protocol to fibre channel protocol.

  • Page 36: Features At A Glance

    (and 1710 if you are connecting DS4000 EXP100 expansion units to the DS4100). Check the boxes for any other DS4000 products for which you would like to receive information. Scroll to the bottom of the page and select Save and Return.

  • Page 37: Clustering Support

    Clustering requires software specific to your operating system. For more information about clustering, go to the following Web site: www.pc.ibm.com/us/compat/nos/matrix.shtml Inventory checklist After you unpack the DS4100 storage server, verify that you have the following items: v Hardware – Up to 14 hard disk drives (shipped installed in the DS4100) Note: The DS4100 base storage server is shipped with no SATA hard disk –...

  • Page 38: Best Practices Guidelines

    If you have not already done so, record your storage server serial number in the table in Appendix A, “Records,” on page 119. The serial number is located on a label at the rear of the DS4100. The machine and model number are found on the product information label above the serial number.

  • Page 39: Storage Server Components

    DS4000 Storage Manager guide for your operating system. Save the profile in a location other than the logical drives created for the DS4100 storage server. v Ensure that your system is in an optimal state before you shut it down. Never turn the power off if any fault light is lit;...

  • Page 40: Hot-Swap Drive Bays

    Hot-swap drive bays Figure 2. DS4100 hot-swap drive bays The DS4100 supports up to 14 DS4000 1.5 Gbps SATA hard disk drives. These drives are preinstalled in drive trays. The unit consisting of the drive and the drive carrier assembly is called a drive CRU. The SATA drive CRU includes the drive tray, the SATA drive, and the hard disk drive (HDD) interposer card.

  • Page 41: Back View

    Tray handle: Use this multipurpose handle to insert and remove a drive CRU in the bay. Back view Figure 4 on page 10 shows the components at the back of the DS4100 base storage server. Fault LED Latch...

  • Page 42: Back View Of Ds4100 Base Storage Server

    Figure 4. Back view of DS4100 base storage server Figure 5 shows the components at the back of the DS4100 single-controller storage server. The DS4100 single-controller storage server has a controller blank instead of a second RAID controller and a power supply blank instead of a second power supply.

  • Page 43: Interface Ports And Switches

    Figure 6. Interface ports and switches for DS4100 base storage server Figure 7 on page 12 shows the ports and switches on the back of the DS4100 single-controller storage server. Note: The DS4100 single-controller storage server has a controller blank instead of a second RAID controller and a power supply blank instead of a second power supply.

  • Page 44: Interface Ports And Switches For Ds4100 Single-Controller Storage Server

    Expansion port The expansion port (labelled Expansion) is used to connect additional expansion units to the RAID controllers on the DS4100 base storage server. You first insert an SFP into the port and then connect a fibre channel cable. Note: In the DS4100 single-controller storage server, the expansion port on...

  • Page 45: Storage Server Operating Environment

    Attention: speed setting. Do not adjust the storage server speed setting. Storage server operating environment This section provides general operating information about the DS4100. All components plug directly into the backplane. When connecting the DS4100 to DS4000 EXP100 storage Do not remove the metal plate that covers the storage server...

  • Page 46: Ds4100 Specifications

    The storage server can run without interruption within the limits shown in Table 10 on page 15. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Heat dissipation v Fully configured storage server (14 1.5 Gbps SATA hard disk drives) –...

  • Page 47: Heat Output, Airflow, And Cooling

    Typical voltage: 230 V AC, 60 Hz at 77% power-supply efficiency and 0.96 power factor Heat output, airflow, and cooling The maximum heat output of the DS4100 is 390 watts (1235 BTU/hr). Each DS4100 rack-mounted unit requires an airflow of 2.5 m input air temperature to all rack cabinets should be in the range 10°...

  • Page 48: Example Of Cold Aisle/Hot Aisle Rack Cabinet Configuration

    1220 mm cold aisle width Figure 8. Example of cold aisle/hot aisle rack cabinet configuration IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide back T42 racks front Cold aisle front T42 racks back Hot aisle back T42 racks...

  • Page 49: Chapter 2. Installing The Storage Server

    If desired, you can open the sides of the shipping box and remove the CRUs from the DS4100 before you lift it from the shipping box in order to reduce the weight of the storage server. See “Removing the CRUs” on page 25.

  • Page 50: Handling Static-Sensitive Devices

    DS4000 Storage Manager 9.10 host software. 11. Turn on the DS4100. See “Turning the storage server on and off” on page 65. 12. Use the DS4000 Storage Manager 9.10 host software to verify the configuration.

  • Page 51: Preparing The Site

    Weight: The DS4100 total weight depends on the number of DS4000 1.5 Gbps SATA hard disk drives that are installed. A fully configured DS4100 with two fans, two power supplies, two RAID controllers, and up to 14 SATA hard disk drives installed weighs 40.8 kg (85.7 lbs).

  • Page 52: Installing The Ds4100

    DS4100 to a rack cabinet. The locations for the M6 screws are highlighted in the templates. The DS4100 is 3 U high. Align the template with the rack at a U boundary. U boundaries are shown as horizontal dashed lines in the rack mounting templates.

  • Page 53: Front Rack Mounting Template

    DS4100 FRONT Rack Mounting Template Bezel Front Left Rail assembly Front Left Figure 9. Front rack mounting template Bezel Front Right Rail assembly Front Right Chapter 2. Installing the storage server...

  • Page 54: Installing The Support Rails

    Figure 10. Rear rack mounting template Installing the support rails Before installing the DS4100 in a rack cabinet, you must install the rails and rack-mounting hardware that come with your storage server. The DS4100 requires an Electronic Industries Association (EIA) 310-D Type A 19-inch rack cabinet. The distance between EIA rails, from the front to the rear of the rack cabinet, is 60.96...

  • Page 55: Alignment Pins With And Without Spacers

    Note the following considerations before you choose where to install the rails: v If you are installing the DS4100 into an empty rack, install it into the lowest available position so that the rack does not become unstable.

  • Page 56: Installing The Support Rails

    10. Tighten the rail adjustment screws 2 on both rails. 11. Continue with “Removing the CRUs” on page 25. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide can be correctly mounted on only one side of the rack cabinet. The rails should be mounted with the alignment pins at the front of the rack cabinet.

  • Page 57: Removing The Crus

    The DS4100 comes with a rack-mounting hardware kit for installing it in a rack cabinet. It is easier to lift the DS4100 and install it in a rack cabinet if you remove all the CRUs first. You will need an antistatic wrist strap and a cart or level surface (to hold the CRUs).

  • Page 58: Removing A Power Supply

    2. Grasp the pull-rings; then pull on the levers and remove the RAID controller. 3. Place the RAID controller on a level surface. 4. For the DS4100 base storage server, repeat step 1 through step 3 for the second RAID controller.

  • Page 59: Removing A Fan

    Figure 14. Removing a power supply 3. Place the power supply on a level surface. 4. For the DS4100 base storage server, repeat step 1 through step 3 for the second power supply. Note: In the DS4100 single-controller storage server, you do not need to remove the power supply blank.

  • Page 60: Removing A Hard Disk Drive

    Complete the following steps to install the DS4100 in a rack cabinet. 1. Place the DS4100 in the rack cabinet. Note: If you have removed the CRUs from the DS4100, you should be able to a. Move the DS4100 to the front of the rack-mount cabinet.

  • Page 61: Replacing The Ds4100 Crus

    Figure 17. Installing the DS4100 2. The bezel is secured to the front of the DS4100 either by plastic wingnuts or by adhesive tape. Remove and discard the wingnuts or the adhesive tape. The bezel should remain attached to the DS4100 chassis.

  • Page 62: Replacing A Raid Controller

    2. Close both levers until the latch locks into place. Attention: chassis. 3. For the DS4100 base storage server, repeat step 1 and step 2 to replace the second RAID controller. Replacing a power supply Complete the following steps to replace the power supplies that you removed before the installation: 1.

  • Page 63: Replacing A Fan

    Then gently push the front of the power supply to ensure that it is fully seated. 3. For the DS4100 base storage server, repeat step 1 and step 2 to replace the second power supply.

  • Page 64: Replacing A Hard Disk Drive

    3. Repeat step 1 and step 2 to replace additional drives. Setting the interface options The DS4100 has two server ID switches that identify the DS4100 on a fibre-channel loop and that assign physical addresses to the drives. The server ID switches are located on the back of the storage server, as shown in Figure 22 on page 33.

  • Page 65: Fibre Channel Loop And Id Settings

    The DS4100 comes with a server ID switch (also referred to as an enclosure ID switch) that is used to identify the DS4100 on a fibre channel loop. The server ID switch is located on the back of the unit, as shown in Figure 22.

  • Page 66: Storage Server Speed Settings

    The amber Conflict LED (located on the right-side of the switch) will be lit if another unit in the drive loop has the same ID setting as the server ID. Attention: units, DO NOT use the tens digit (x10) setting. Use only the ones digit (x1) setting to set unique server IDs or enclosure IDs.

  • Page 67: Direct (Out-Of-Band) Management Method

    Figure 24 on page 36 shows the direct (out-of-band) management method. Note: Do not connect the DS4100 storage server Ethernet ports to a public network or public subnet. It is strongly recommended that you create a private network between the DS4100 storage server and the management station.

  • Page 68: Fibre Channel Connections

    (HDDs). The redundant drive fibre-channel loop consists of fibre channel HDDs and optional storage expansion units, fibre-channel cables, and SFPs. Exception: The DS4100 single-controller storage server does not support drive loops or expansion units. The DS4100 base storage server supports redundant disk-drive-loop configurations.

  • Page 69: Redundant Host And Drive Loops

    Figure 25 on page 38 v Single SAN fabric configuration, as shown in Figure 26 on page 39 v Dual SAN fabric configuration, with an additional DS4100 and a DS4000 EXP100 attached, as shown in Figure 27 on page 39...

  • Page 70: Examples Of Redundant Host And Drive Fibre Channel Loop Configurations (Base Storage Server)

    In Figure 26 on page 39, the fibre channel switches are connected together via the inter-switch link (ISL) to form a single SAN fabric. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Host system Host system...

  • Page 71: Example Of A Single San Fabric Configuration (Base Storage Server)

    Switch Interswitch link DS4100 Figure 26. Example of a single SAN fabric configuration (base storage server) In Figure 27, the fibre channel switches are not connected together. Each switch forms its own SAN fabric. Host system with two host adapters...

  • Page 72: Example Of A Two-Cluster Configuration (Base Storage Server)

    Figure 28. Example of a two-cluster configuration (base storage server) DS4100 single-controller storage server configurations: This section provides the following examples of DS4100 single-controller storage server host fibre channel configurations. v Single server configuration with one adapter, as shown in Figure 29...

  • Page 73: Example Of A Single Server Configuration With Two Adapters (Single-Controller Storage Server)

    Host Server DS4100 Single Controller Figure 30. Example of a single server configuration with two adapters (single-controller storage server) Host Host Server Server DS4100 Single Controller Figure 31. Example of a dual server configuration with one adapter on each server (single-controller storage server) Chapter 2.

  • Page 74: Installing The Storage Subsystem Configuration

    Using the information gathered in the previous sections, install the host systems and host adapters. Notes: 1. See the documentation provided with your host adapters for installation requirements and procedures. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Host Server DS4100 Single Controller...
  • Page 75 2. Use the correct host adapter driver. For the latest supported host adapters and drivers, go to the following Web site: www.ibm.com/servers/storage/disk/ Attach fiber-optic interface cables to each host adapter. You will connect the other end of the cables to the controller later in the installation process. For more information about handling fiber-optic cables, see “Handling fiber-optic cables”...
  • Page 76 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 77: Chapter 3. Cabling The Storage Server

    Chapter 3. Cabling the storage server This chapter provides fibre channel and power cabling information for the storage server. After you attach the storage server power cables, use the instructions that are provided in “Turning the storage server on and off” on page 65 for the initial startup of the storage server.

  • Page 78: Installing Sfp Modules

    Installing SFP modules The DS4100 host ports require SFP modules. SFP modules are used to convert electrical signals to optical signals that are required for fibre channel transmission to and from RAID controllers. The DS4100 supports two types of SFP modules: one with a plastic latch and one with a wire latch.

  • Page 79: Small Form-Factor Pluggable (Sfp) Module

    DANGER Some laser products contain an embedded Class 3A or Class 3B laser diode. Note the following. Laser radiation when open. Do not stare into the beam, do not view directly with optical instruments, and avoid direct exposure to the beam. Attention: When you handle static-sensitive devices, take precautions to avoid damage from static electricity.

  • Page 80: Removing Sfp Modules

    6. Replace the protective cap on the port. Installing fiber-optic cables To connect the fiber-optic cables, perform the following steps. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide To avoid damage to the cable or to the SFP module, make sure you Plastic tab...

  • Page 81: Using Lc-Lc Fibre-Channel Cables

    The LC-LC fibre-channel cable is a fiber-optic cable that you use to connect into one of the following devices: v An SFP module installed in an IBM DS4100 v An SFP module installed in an IBM DS4000 EXP100 expansion unit See Figure 40 on page 50 for an illustration of the LC-LC fibre-channel cable.

  • Page 82: Connecting An Lc-Lc Cable To An Sfp Module

    LC-LC fibre-channel cable. Figure 40. LC-LC fibre-channel cable Note: If you are connecting the DS4100 to a 1 Gbps device (such as a 1 Gbps fibre channel switch), you must also use an LC-SC fibre-channel cable adapter. For more information about using an LC-SC fibre-channel cable adapter, see “Using LC-SC fibre-channel cable adapters”...

  • Page 83: Removing An Lc-Lc Fibre-Channel Cable

    4. Carefully insert this end of the LC-LC cable into an SFP module that is installed in the DS4100. The cable connector is keyed to ensure it is inserted into the SFP module correctly. Holding the connector, push in the cable until it clicks into place, as shown in Figure 42.

  • Page 84: Using Lc-Sc Fibre-Channel Cable Adapters

    1-Gbps fibre channel switch v Fibre channel host bus adapter For more information about connecting to these devices, see the documentation that comes with the LC-SC fibre-channel cable adapter. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Lever Latches...

  • Page 85: Connecting An Lc-Sc Cable Adapter To A Device

    1. Read the information in “Handling fiber-optic cables” on page 45. 2. Connect one end of an LC-LC cable to an SFP module in the DS4100. For instructions, see “Using LC-LC fibre-channel cables” on page 49.

  • Page 86: Removing An Lc-Lc Cable From An Lc-Sc Cable Adapter

    Figure 47. Connecting an LC-LC cable into the LC-SC cable adapter 5. If you are connecting the DS4100 to a 1-Gbps fibre channel switch or fibre channel host bus adapter, connect the SC connector end of the LC-SC cable adapter to a Gigabit Interface Converter (GBIC) that is installed in the 1-Gbps fibre channel switch or fibre channel host bus adapter.

  • Page 87: Connecting Hosts To The Raid Controllers

    “Connecting hosts to a DS4100 single-controller storage server” on page 56 Connecting hosts to a DS4100 base storage server To connect a host adapter to the DS4100 base storage server RAID controllers, perform the following steps: 1. Install an SFP in a host port on the RAID controller.

  • Page 88: Connecting Hosts To A Ds4100 Single-Controller Storage Server

    “Fibre channel loop configurations” on page 36. Connecting hosts to a DS4100 single-controller storage server To connect a host adapter to the DS4100 single-controller storage server RAID controllers, perform the following steps: 1. Install an SFP in a host port on the RAID controller.

  • Page 89: Connecting Secondary Interface Cables To A Ds4100 Base Storage Server

    2. Do not connect the DS4100 in a public LAN or public subnet. Use a local private network for the DS4100 storage server and the management station Ethernet ports.
  • Page 90 DS4100 storage server, refer to the IBM TotalStorage DS4000 Hard Drive and Storage Expansion Enclosure Installation and Migration Guide.

  • Page 91: Ds4000 Exp100 Loop Redundancy (Ds4100 Base Storage Server Only)

    B provides redundancy, which is an alternate path to your disk drives. Figure 54 on page 60 shows a redundant loop that you might create with a DS4100 and three DS4000 EXP100s. The numbers provided in Figure 54 on page 60 show the order in which cables should be attached.

  • Page 92: Ds4000 Exp100 Redundant Loop Configuration

    Each ESM on the DS4000 EXP100 has an SFP module input port and an SFP module output port. As you cable DS4000 EXP100s together, connecting input ports to output ports can improve diagnostic capability. Refer to the DS4100 Storage Server Fibre Channel Cabling Instructions for cabling connections between input and output ports.

  • Page 93: Power Cabling

    The DS4100 storage server does not ship with power cords. Refer to Appendix C, “Power cords,” on page 125 for the list of IBM-approved power cords for your region.

  • Page 94: Installing The Ds4000 Storage Manager Client

    See Table 10 on page 15 for the DS4100 AC power requirements. See Figure 56 for an example of redundant power cabling.
  • Page 95 DS4000 Storage Manager guide for your operating system. Save the profile in a location other than the logical drives created for the DS4100 storage server. Important: Controller firmware version 05.42.xx.xx is required for the DS4100.
  • Page 96 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 97: Chapter 4. Operating The Storage Server

    DS4100. This ensures that the DS4100 and all its attached storage expansion units will have power if only one power circuit is available. In addition, having all the right or all the left power...

  • Page 98: Turning On The Storage Server

    All hard disk drives are locked securely in place. For more information, see “Installing hot-swap hard disk drives” on page 80. Note: At least one drive must be present before you start the DS4100. Attention: least 10 seconds before you turn on the power by using the power-on sequence in step 2.

  • Page 99: Turning Off The Storage Server

    green on the front of the storage server. If they are not, use the DS4000 Storage Manager client to diagnose the problem (see “Monitoring status through software” on page 70). Note: The green drive active LED and amber drive fault LED below the drive Turning off the storage server Attention: Fault LEDs are on.

  • Page 100: Restoring Power After An Unexpected Shutdown

    (listed in “Turning off the storage server” on page 67) because of an emergency situation. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide To turn off all power to the storage server, you must turn off both storage server.

  • Page 101: Performing An Emergency Shutdown

    Attention: the system after an unexpected shutdown. If the storage server shuts down unexpectedly, go to “Restoring power after an over-temperature shutdown.” Otherwise, go to “Restoring power after an emergency shutdown.” Note: Unattended restoration of power is supported; however, if power restoration is attended, you should follow the best practices guidelines and the power restoration sequence in “Turning on the storage server”...

  • Page 102: Monitoring Status Through Software

    Attention To ensure proper operation when connecting DS4000 EXP100 expansion units to the DS4100, you must verify that the DS4000 EXP100 ESM firmware level is at level 9554 or later. If the ESM firmware level is lower than 9554, disconnect the DS4000 EXP100 and contact IBM support, your IBM reseller, or your IBM representative for assistance.

  • Page 103: Checking The Leds

    v Locate the failed component v Determine the recovery procedures to repair the failure Although the storage server has fault LEDs, these lights do not necessarily indicate which component has failed or needs to be replaced, or which type of recovery procedure you must perform.

  • Page 104: Storage Server Leds (Front)

    Always use the DS4000 Storage Manager client to identify the failure. Not all component failures turn on this LED. For more information, see “Monitoring status through software” on page 70. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Fault LED Latch...

  • Page 105: Raid Controller Leds

    2Gbps Host 1 Host 2 10BT 100BT 2Gb/s 2Gb/s Host 1 Host 2 10BT 100BT indicator indicator Figure 58. RAID controller LEDs Table 13. RAID controller LEDs Icon Color Fault Amber Host loop Green Cache active Green Battery Green Expansion link indicator Expansion 10101 Expansion...

  • Page 106: Fan And Power Supply Leds

    Always use the DS4000 Storage Manager client to identify the failure. Fan fault LED Figure 59. Fan and power supply LEDs Table 14. Fan LED Fault IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Color Operating states Expansion Amber v Off - Normal operation.

  • Page 107: Cache Memory And Raid Controller Battery

    Always use the DS4000 Storage Manager client to identify the failure. Cache memory and RAID controller battery Each RAID controller in the DS4100 contains 128 MB of cache data. Each controller also contains a rechargeable battery that maintains data in the cache in the event of a power failure.

  • Page 108: Raid Controller Cache Battery

    Data caching starts after the battery completes the startup tests. Figure 61 on page 77 shows the location of the Battery LED on the front of the RAID controller. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Cache active...

  • Page 109: Battery Led

    Battery Figure 61. Battery LED Chapter 4. Operating the storage server...
  • Page 110 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 111: Chapter 5. Installing And Replacing Components

    (CRUs), such as hot-swap drives, fans, RAID controllers, and power supplies. This chapter also provides instructions to help you add new DS4000 EXP100 expansion units to existing DS4100 configuration. For more CRU and Option part number information, see the following Web site: www-1.ibm.com/servers/storage/support/disk/...

  • Page 112: Installing Hot-Swap Hard Disk Drives

    Note: For proper cooling and EMC protection, do not leave a drive slot open without either a blank tray or a hard disk drive CRU inserted. v Drive CRUs: The DS4100 supports up to fourteen IBM 1.5 Gbps SATA hard disk drives.

  • Page 113: Hot-Swap Hard Disk Drive Leds

    1. Read the documentation that comes with the hard disk drive. 2. Check for Fault LEDs shown in Figure 62. If any amber LEDs are lit, see Chapter 6, “Hardware maintenance,” on page 111. Fault LED Activity LED Tray handle Latch Figure 62.

  • Page 114: Replacing Hot-Swap Hard Disk Drives

    Storage Management client before attempting to remove it from the drive slot. 2. Remove the drive CRU: IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Tray handle Failure to replace the hard disk drives in their correct bays might result...

  • Page 115: Upgrading Drives

    a. Release the latch on the drive CRU by pinching together the blue line on the tray latch and the finger hole on the tray handle, as shown by the arrow in Figure 63 on page 82. b. Pull the tray handle out into the open position. c.

  • Page 116: Replacing All Drives At The Same Time

    Compare the kit instructions with this procedure to determine if you need to modify this procedure. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...
  • Page 117 2. Use the DS4000 Storage Manager client to check the status of the storage server. Correct any problems that are reported. 3. Perform a complete backup of the drives that you are replacing. You need the backup to restore data on the drives later in this procedure. Attention: When you handle static-sensitive devices, take precautions to avoid damage from static electricity.

  • Page 118: Replacing The Drives One At A Time

    2. Use the DS4000 Storage Manager client to check the status of the unit. Correct any problems that are reported. 3. Back up the data on the drives that you are replacing. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...
  • Page 119 Attention: When you handle static-sensitive devices, take precautions to avoid damage from static electricity. For details about handling static-sensitive devices, see “Handling static-sensitive devices” on page 79. 4. Unpack the new drives. Set the drives on a dry, level surface away from magnetic fields. Save the packing material and documentation in case you need to return the drives.

  • Page 120: Working With Hot-Swap Cooling Fans

    Attention: because it might cause damage to the internal components and circuitry. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide finished if there is I/O activity to that drive. In that case, use the host software to determine if the data reconstruction is finished.

  • Page 121: Working With Power Supplies

    Working with power supplies The power supply CRUs provide power to the internal components by converting incoming ac voltage to dc voltage. The DS4100 base storage server power system operates with two power supply CRUs that can be hot-swapped. In the DS4100 base storage server, one power supply CRU can maintain electrical power to the unit if the other power supply is turned off or malfunctions.

  • Page 122: Power Supply Controls

    “Turning off the storage server” on page 67 and follow the procedure described there before replacing the power supply on a DS4100 single-controller storage server. Each power supply CRU has a built-in sensor that detects the following conditions: v Over-voltage...

  • Page 123: Removing A Power Supply

    67 and follow the procedure described there before proceeding with the following steps. If you are replacing a failed power supply on a DS4100 single-controller storage server, you do not need to power down the storage server before replacing the failed power supply.

  • Page 124: Installing A Power Supply

    Figure 68. Installing a hot-swap power supply To install a hot-swap power supply, perform the following steps: IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Lever AC power Strain-relief...

  • Page 125: Connecting The Power Cord To The Ac Power Connector

    6. Plug the power cord into a properly grounded electrical outlet. 7. Turn on the power-supply switch. 8. DS4100 single-controller storage server only: If you have replaced a working (non-failed) power supply on a DS4100 single-controller storage server, turn on...

  • Page 126: Working With Raid Controllers

    The DS4100 single-controller storage server has only one RAID controller. If you are replacing a failed RAID controller in the DS4100 single-controller storage server, you do not need to power down the DS4100 before replacing the RAID controller. If you are replacing a functioning (non-failed) RAID controller in the DS4100...

  • Page 127: Raid Controller Levers And Labels

    Levers Figure 70. RAID controller levers and labels In a DS4100 base storage server, you can hot swap one controller while the other is still active. The figures in this procedure show the base storage server. If your storage server is the single-controller storage server, controller slot B will have a blank controller rather than the second controller as shown in these figures.

  • Page 128: Replacing A Raid Controller

    The system will automatically update the firmware for the new controller to match the configuration database. If the controller in a DS4100 single-controller storage server fails, you may need to manually update the firmware level to match the configuration database. If the firmware level of the new controller matches the configuration database, the storage server will boot normally after you install the new controller.

  • Page 129: Location Of Sfps And Fiber-Optic Cables

    The DS4100 single-controller storage server has only one RAID controller. If you are replacing a failed RAID controller in the DS4100 single-controller storage server, you do not need to power down the DS4100 before replacing the RAID controller. If you are replacing a functioning (non-failed) RAID controller in the DS4100...

  • Page 130: Unlocking The Sfp Module Latch - Plastic Variety

    Push down on the latch (centered above the RAID controller). Pull both levers at the same time out of the locked position. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide by pulling the plastic tab outward 10°, as shown in Figure 72.

  • Page 131: Pull-Rings For Removing A Controller

    Figure 74. Pull-rings for removing a controller b. Grasp the pull-rings; then pull on the levers and remove the RAID controller. 7. To access the cache battery, place the RAID controller CRU bottom up on a flat surface. Using a #2 Phillips head screwdriver, remove the six screws (three on each side of the RAID controller) that hold the RAID controller bottom panel to the sides of the RAID controller, as shown in Figure 75.

  • Page 132: Removing The Controller Battery

    DS4000 storage server. 18. Write the battery replacement information on the label on the battery pack. v Installed date – Write today’s date IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Battery Harness Battery Two-Pin...
  • Page 133 27. Follow the appropriate procedure for the your DS4100 storage server model: v For the DS4100 base storage server: When you replace the failed RAID controller, the DS4000 Storage Manager client automatically updates the firmware for the new controller to match the configuration database. After you replace a controller, always use the DS4000 Storage Manager client to verify the firmware levels and NVSRAM files.
  • Page 134 LEDs in the front of the unit for at least four minutes. - If the drive LEDs flash one or more times, then the new DS4100 controller does have the same firmware as the replaced DS4100 controller, but another problem exists.

  • Page 135: Manually Upgrading The Firmware For The Ds4100 Single-Controller Storage Server

    DS4100 single-controller storage server to running (optimal) state after you replace the RAID controller. v Reboot the host server that has mapped logical drives from the DS4100 single-controller storage server. Rebooting the host server enables the multipath fail-over to recognize the logical drives under the new WWNN.
  • Page 136 RAID controller is determined to be incompatible with the configuration database during the boot process, the DS4100 single-controller storage server will enter a Stop State and will not allow any DS4000 Storage Manager operations to be performed. In this case, you must manually upgrade or downgrade the replacement RAID controller firmware to match the version required by the configuration database.

  • Page 137: Replacing The Battery In The Raid Controller

    Each RAID controller contains a rechargeable battery that maintains the data in the cache memory for three days without power to the unit. Important: If you are replacing the battery in a DS4100 single-controller storage Use the following procedure if the DS4000 Storage Manager client instructs you to replace the battery because the current battery has failed or is nearing its expiration date.
  • Page 138 RAID controller that contains the battery that needs to be replaced. This is done by either placing the controller offline before removing it from the DS4100 server or turning off caching in the DS4100 server, using the DS4000 Storage Manager client. Follow the steps given in the software procedure before you continue with this procedure.
  • Page 139 v Installed date – Write today’s date v Replacement Date – Write the date three years from today’s date Note: The preferred method of checking for the battery operating age is through the storage server management software. CONTAINS SEALED LEAD BATTERY.

  • Page 140: Installing A New Raid Controller

    24. If the operation fails, repeat the command. If the operation fails again, reseat the controller and repeat this step. Note: Note that on all DS4100 models, it might take 15 minutes to a few 25. Use the DS4000 Storage Manager client to monitor the status of the storage server and to reset the battery age.

  • Page 141: Installing Sfps And Fiber-Optic Cables

    Adding a DS4000 EXP100 to an existing DS4100 configuration (base storage server only) You can add up to seven DS4000 EXP100 storage expansion units to a DS4100 storage server. However, you do not need to configure the DS4100 with the maximum number of DS4000 EXP100s at the time of initial installation.

  • Page 142: Cabling The Ds4100 To A Redundant Loop

    Storage server Figure 78. Cabling the DS4100 to a redundant loop In Figure 78, the existing DS4100 configuration already includes two DS4000 EXP100s. The dashed lines show you how to add a third DS4000 EXP100 to the configuration. To add the new DS4000 EXP100 shown in Figure 78, you would perform the following steps: 1.

  • Page 143: Chapter 6. Hardware Maintenance

    Using the diagnostic hardware The DS4100 comes with an LC wrap-plug adapter and LC-LC coupler. The wrap-plug adapter and LC-LC coupler are used to perform the loopback test and to identify fibre path problems. See the IBM TotalStorage DS4000 Problem Determination Guide for more information.

  • Page 144: Symptom-To-Fru Index

    Problem indicator Amber LED on Amber LED is off and green LED off Amber and green LEDs on IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Component Possible cause Drive CRU Drive failure Fan CRU Fan failure RAID controller...
  • Page 145 Use the DS4000 Storage battery has failed Manager client to check and replace the battery. Host or switch is Check if host or switch is on. off or has failed Replace attached devices that have failed. Fibre-channel Ensure that the fibre-channel...
  • Page 146 (continued) Intermittent or sporadic power loss to the storage server Unable to access drives Random errors IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Component Possible cause Expansion loop Drives are improperly installed or not installed RAID controller...

  • Page 147: Parts Listing

    Note: If you cannot find the problem in the Symptom-to-FRU index, test the entire system. Parts listing Figure 79 on page 116 and the following table provide a parts listing for the DS4100. Component Possible cause Controller No drives are present in the storage server.

  • Page 148: Ds4100 Parts List

    Figure 79. DS4100 parts list Table 18. Parts listing (TotalStorage DS4100 storage server) Index IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide DS4100 storage server Rail kit Fan ASM CRU Power supply CRU, 400W DS4100 SATA RAID controller...
  • Page 149 Table 18. Parts listing (TotalStorage DS4100 storage server) (continued) Index DS4100 storage server Cable, FRU-1M Cable, FRU-5M Cable, FRU-25M Short wave SFP Long wave SFP Power cord, 2.8M Power cord, jumper Battery Wrap plug (loopback adapter) LC-SC fibre channel cable adapter...
  • Page 150 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 151: Appendix A. Records

    Model number: Serial number: Controller firmware version: The serial number is located on a label at the rear of the DS4100. The machine and model number are found on the product information label above the serial number. Installed device records Use the following table to record the number of the hard disk drives that you replaced and the corresponding bay number.
  • Page 152 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 153: Appendix B. Rack Mounting Template

    DS4100 in a rack cabinet. The locations for the M6 screws are highlighted in the templates. The DS4100 is 3 U high. Align the template with the rack at a U boundary. U boundaries are shown as horizontal dashed lines in the rack mounting templates.

  • Page 154: Front Rack Mounting Template

    Figure 80. Front rack mounting template IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide DS4100 FRONT Rack Mounting Template Bezel Front Left Rail assembly Front Left Bezel Front Right Rail assembly Front Right...

  • Page 155: Rear Rack Mounting Template

    Rack Mounting Rail assembly Rear Left Figure 81. Rear rack mounting template DS4100 REAR Template Rail assembly Rear Right ® Printed in the U.S.A. Appendix B. Rack mounting template...
  • Page 156 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 157: Appendix C. Power Cords

    Appendix C. Power cords For your safety, IBM provides a power cord with a grounded attachment plug to use with this IBM product. To avoid electrical shock, always use the power cord and plug with a properly grounded outlet. IBM power cords used in the United States and Canada are listed by Underwriter’s Laboratories (UL) and certified by the Canadian Standards Association (CSA).
  • Page 158 14F0069 14F0087 1838574 36L8880 02K0546 6952300 6952301 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide Cord specifics Used in these countries or regions 250V/10A Bangladesh, Lesotho, Maceo, Maldives, Namibia, Nepal, 2.8M Pakistan, Samoa, South Africa, Sri Lanka, Swaziland,...

  • Page 159: Notices

    Notices This publication was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used.

  • Page 160: Important Notes

    This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 161: Chinese Class A Compliance Statement

    communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits.

  • Page 162: Taiwan Class A Warning Statement

    Taiwan Class A warning statement Japanese Voluntary Control Council for Interference (VCCI) statement IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide This is a Class A product. In a domestic environment this product may...

  • Page 163: Glossary

    Glossary This glossary provides definitions for the terminology and abbreviations used in IBM TotalStorage DS4000 publications. If you do not find the term you are looking for, see the IBM Glossary of Computing Terms located at the following Web site: www.ibm.com/ibm/terminology This glossary also includes terms and definitions from:...
  • Page 164 CRC. See cyclic redundancy check. CRT. See cathode ray tube. CRU. See customer replaceable unit. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide customer replaceable unit (CRU). An assembly or part that a customer can replace in its entirety when any of its components fail.
  • Page 165 N_ports using address information in the frame header. A fabric can be as simple as a point-to-point channel between two N-ports, or as complex as a frame-routing switch that provides multiple and redundant internal pathways within the fabric between F_ports.
  • Page 166 (HBA). An interface between the fibre-channel network and a workstation or server. host computer. See host. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide host group. An entity in the storage partition topology that defines a logical collection of host computers that require shared access to one or more logical drives.
  • Page 167 Internet Protocol (IP). A protocol that routes data through a network or interconnected networks. IP acts as an intermediary between the higher protocol layers and the physical network. Internet Protocol (IP) address. The unique 32-bit address that specifies the location of each device or workstation on the Internet.
  • Page 168 ODM. See Object Data Manager. out-of-band. Transmission of management protocols outside of the fibre-channel network, typically over Ethernet. IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide partitioning. See storage partition. PCI local bus. See peripheral component interconnect local bus.
  • Page 169 random-access memory (RAM). A temporary storage location in which the central processing unit (CPU) stores and executes its processes. Contrast with DASD. RDAC. See redundant disk array controller. read-only memory (ROM). Memory in which stored data cannot be changed by the user except under special conditions.
  • Page 170 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide storage partition topology. In the DS4000 Storage Manager client, the Topology view of the Mappings window displays the default host group, the defined host group, the host computer, and host-port nodes.
  • Page 171 Vendor property might be discovered, but will always remain editable. worldwide name (WWN). A unique identifier for a switch on local and global networks. WORM. See write-once read-many. write-once read many (WORM). Any type of storage medium to which data can be written only a single time, but can be read from any number of times.
  • Page 172 IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide...

  • Page 173: Index

    42 configure storage hardware xix configure storage subsystems on host xix connect power xix controller firmware manually upgrading for the DS4100 single-controller storage server 103 © Copyright IBM Corp. 2004 controller firmware (continued) requirements for DS4100 63...
  • Page 174 LC-SC using the 52 fibre-channel loop adding the DS4000 EXP100 109 fibre-channel loop IDs 80 fire suppression xxviii IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide firmware manually upgrading for the DS4100 single-controller storage server 103 requirements for DS4100 63...
  • Page 175 EIA 310-D Type A 19 inch 22 preparatory procedures 20 rail installation 22 site preparation 18 rack mounting template for DS4100 20, 121 RAID controller cache battery 76 connecting Ethernet cable to 56 connecting host to 55 connecting RS-232 (serial) cable 57...
  • Page 176 13 start server xix static-sensitive devices, handling 79 static-sensitive precautions 79 storage expansion unit installation guide xix IBM TotalStorage DS4100 Storage Server: Installation, User’s, and Maintenance Guide storage server components back view 9 hard disk drives 79 installing power supply 92...
  • Page 177 Readers’ Comments — We’d Like to Hear from You IBM TotalStorage DS4100 Storage Server Installation, User’s, and Maintenance Guide Publication No. GC26-7712-00 Overall, how satisfied are you with the information in this book? Very Satisfied Overall satisfaction How satisfied are you that the information in this book is:...
  • Page 178 Readers’ Comments — We’d Like to Hear from You GC26-7712-00 Fold and Tape _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ BUSINESS REPLY MAIL FIRST-CLASS MAIL PERMIT NO.
  • Page 180 Part Number: 25R0314 Printed in USA GC26-7712-00...

Table of Contents