Download
Share
Print this page
IBM 6000 Hardware Planning And Installation Manual
  1. Manuals
  2. Brands
  3. IBM Manuals
  4. Storage
  5. 6000
  6. Hardware planning and installation manual
IBM 6000 Hardware Planning And Installation Manual

IBM 6000 Hardware Planning And Installation Manual

Storage scale system
Hide thumbs Also See for 6000:

Advertisement

Quick Links

Download this manual
IBM Storage Scale System 6000
6.2.0
Hardware Planning and Installation
Guide
IBM
SC28-3485-02

Advertisement

loading
Need help?

Need help?

Do you have a question about the 6000 and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for IBM 6000

Summary of Contents for IBM 6000

  • Page 1 IBM Storage Scale System 6000 6.2.0 Hardware Planning and Installation Guide SC28-3485-02...
  • Page 2 IBM welcomes your comments; see the topic “How to submit your comments” on page xi. When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.

  • Page 3: Table Of Contents

    Conventions used in this information......................x How to submit your comments........................xi Chapter 1. Notices....................1 Trademarks..............................2 Caution, danger, and warning notices for IBM Storage Scale System 6000..........3 Environmental notices..........................5 Electromagnetic compatibility notices......................6 Canada Notice............................6 European Community and Morocco Notice................... 6 Germany Notice............................
  • Page 4 Installing the management server in a rack..................41 Unpacking IBM Storage Scale System 6000 (IBM SSR task)..............42 Installing support rails for the IBM Storage Scale System 6000 (IBM SSR task)......42 Installing enclosures (IBM SSR task)....................46 Installing the cable management arm (IBM SSR task)............... 49 Chapter 5. Monitoring the system using LEDs............53...

  • Page 5: Figures

    6. Performance configuration with 24 drives and 24 fillers installed............17 7. Performance configuration with 48 drives installed.................. 17 8. Front view of anIBM Storage Scale System 6000 with a 48-drive configuration........18 9. Power supply units............................18 10. SSR access port............................19 11.
  • Page 6 36. IBM Storage Scale System 6000 server canister LEDs................54 37. Close view of server canister LEDs......................55 38. Close view of IBM Storage Scale System 6000 Ethernet and BMC port LEDs........55 39. Close view of theIBM Storage Scale System 6000 PSU status LEDs............58 40.
  • Page 7 15. Random vibration PSD profile breakpoints (nonoperational)..............29 16. Power cord feature codes......................... 34 17. Power planning reference information for IBM Storage Scale System Utility Node ......35 18. Ports on IBM Storage Scale System 6000....................37 19. Summary of supported host interface adapters..................38 20.
  • Page 8 23. Description of the system status LEDs..................... 54 24. IBM Storage Scale System 6000 Ethernet and BMC port status LEDs........... 55 25. Description of the canister status LEDs....................56 26. IBM Storage Scale System 6000 PSU status LEDs.................. 58 27. IBM Storage Scale System 6000 drive status LEDs.................59 28.

  • Page 9: About This Information

    About this information Who should read this information This information is intended for administrators of IBM Storage Scale System 6000 that includes IBM Storage Scale RAID. IBM Storage Scale System 6000 information units Information unit Type of information Intended users...

  • Page 10: Conventions Used In This Information

    • IBM POWER9 servers, see https://www.ibm.com/docs/en/ess/6.1.9_ent?topic=guide-5105-22e- reference-information. For the latest support information about IBM Storage Scale RAID, see the IBM Storage Scale RAID FAQ in IBM Documentation. Conventions used in this information Table 1 on page x describes the typographic conventions used in this information. UNIX file name conventions are used throughout this information.

  • Page 11: How To Submit Your Comments

    In the left margin of the document, vertical lines indicate technical changes to the information. How to submit your comments To contact the IBM Storage Scale development organization, send your comments to the following email address: scale@us.ibm.com About this information  xi...
  • Page 12 Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 13: Chapter 1. Notices

    This information was developed for products and services offered in the US. This material might be available from IBM in other languages. However, you may be required to own a copy of the product or product version in that language in order to access it.

  • Page 14: Trademarks

    IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.

  • Page 15: Caution, Danger, And Warning Notices For Ibm Storage Scale System 6000

    Caution, danger, and warning notices for IBM Storage Scale System 6000 Ensure that you read and understand all notices for IBM Storage Scale System 6000. Attention: Only a trained person must install, remove, or replace the field replaceable units (FRU).

  • Page 16: Multiple Power Sources

    Protective earthing DANGER: The equipment must be grounded properly to earth. Failure to use properly grounded outlets may result in electrical shocks. 4  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 17: Environmental Notices

    Environmental notices This information contains all the required environmental notices for IBM Storage Scale System 6000. The IBM Systems Environmental Notices includes statements on limitations, product information, product recycling and disposal, battery information, flat panel display, refrigeration and water-cooling systems, external power supplies, and safety data sheets.

  • Page 18: Electromagnetic Compatibility Notices

    IBM Storage Scale System 6000 (MTM 5149-F48) meets all regulatory EMC compliance requirements as offered, including auxiliary equipment and cables that may be ordered from IBM to be used in conjunction with IBM Storage Scale System 6000 (MTM 5149-F48).

  • Page 19: Japan Electronics And Information Technology Industries Association (Jeita) Notice

    Tel: 914-499-1900 Der verantwortliche Ansprechpartner des Herstellers in der EU ist: IBM Deutschland GmbH Technical Relations Europe, Abteilung M456 IBM-Allee 1, 71139 Ehningen, Germany Tel: +49 800 225 5426 e-mail: Halloibm@de.ibm.com Generelle Informationen: Das Gerät erfüllt die Schutzanforderungen nach EN 55024 und EN 55032 Klasse A.

  • Page 20: Japan Voluntary Control Council For Interference (Vcci) Notice

    Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. 8  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 21: Homologation Statement

    Properly shielded, grounded, and approved cables and connectors must be used in order to meet FCC emission limits. IBM is not responsible for any radio or television interference caused by using other than approved cables and connectors, or by unauthorized changes or modifications to this equipment.
  • Page 22 10  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 23: Chapter 2. Technical Overview

    SAS-attached flash drives. IBM Storage Scale System 6000 can contain up to 48 NVMe-attached drives. You can either use 24 drives in a half-populated configuration or 48 drives in a fully populated configuration. These drives are accessible from the front of IBM Storage Scale System 6000, as shown in the following figure.
  • Page 24 Figure 2. Rear view of IBM Storage Scale System 6000 The IBM Storage Scale System 6000 uses two AMD EPYC 9454 SP5 per server canister. The following table provides an overview of IBM Storage Scale System 6000. Table 2. Overview of the IBM Storage Scale System 6000 system...
  • Page 25 – PN: 01LL984 - CX-7 VPI, dual port, 200 Gb. – PN: 01LL983 - CX-7 InfiniBand, single port, 400 Gb. The ports and LEDs details of IBM Storage Scale System 6000 are shown in the following figure. Chapter 2. Technical overview  13...

  • Page 26: Ibm Storage Scale System 6000 Ports And Leds

    Dual CXL (not used) USB A Figure 3. IBM Storage Scale System 6000 ports and LEDs Networking details The networking details of IBM Storage Scale System 6000 are shown in the following figure. 14  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 27: Ibm Storage Scale System 6000 Networking

    • FC: AJQQ - CX-7 InfiniBand, single port, 400 Gb NVMe transport protocol in IBM Storage Scale System 6000 The IBM Storage Scale System 6000 systems use the Non-Volatile Memory express (NVMe) drive transport protocol. • NVMe is designed specifically for flash technologies. It is a faster and less complicated storage drive transport protocol than SAS.

  • Page 28: System Topology

    You need a minimum of one EMS node as a part of your IBM Storage Scale System cluster. The EMS node can be ordered as a part of IBM Storage Scale System 6000.

  • Page 29: Ibm Storage Scale System 6000 Configurations

    For more information, see Table 2 on page 12 and Table 3 on page 12. Drives IBM Storage Scale System 6000 supports industry-standard 2.5-inch NVMe-attached drive options. The following storage capacities are supported: • 3.84 TB, 7.68 TB, 15.36 TB, and 30.72 TB for NVMe-attached drives Drives are hot swappable from the front of the enclosure.

  • Page 30: Drive Slots

    IBM Storage Scale System 6000 supports 24 and 48 drives configurations. All drives must be identical in an IBM Storage Scale System 6000 enclosure. In a 24-drives configuration, drives must be plugged only into slots with even numbers (2-24) in the top;...

  • Page 31: Ssr Access Port

    Note: An IBM intranet connection is required to access the guide. In each of the IBM Storage Scale System 6000 server canisters, the RJ45 10 GbE port is designated as the SSR access port, as shown in the following figure.
  • Page 32 20  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 33: Chapter 3. Planning For Ibm Storage Scale System 6000 Hardware

    Ensure that the system's physical configuration records are easily accessible to be used as a reference as needed. The customers must work with the IBM technical team to plan for installation and deployment of an IBM Storage Scale System 6000.
  • Page 34 • Multiple power cords. The product might be equipped with multiple AC power cords or multiple DC power cables. To remove all hazardous voltages, disconnect all power cords and power cables. (L003) 22  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...
  • Page 35 If there is less airflow, additional mechanisms are required to cool the enclosure. An appropriate IBM rack configuration would be the 7965-S42 IBM Rack Model S42, with standard rear door and feature code 6069 Front Door For 2.0 Meter Rack (High Perforation).

  • Page 36: Environmental Requirements

    *Note: This value represents the absolute maximum power draw of the power supplies. Actual power consumption will vary relative to workload/system utilization. For example, for a fully populated IBM Storage Scale System 6000 (48 NVMe drives) at a theoretical 100% system utilization, the estimated power consumption is ~3.2kW.
  • Page 37 The weight of this part or unit is more than 46 kg (102 lb). It takes specially trained persons, a lifting device, or both to safely lift this part or unit. (C011) The following table shows the rack space requirements for IBM Storage Scale System 6000 in tabular form.

  • Page 38: Service Envelope Illustration (Dimensions In Millimeters)

    Service envelope illustration The following figure provides a close-up view of the IBM Storage Scale System 6000 mounting envelope. Figure 11. Service envelope illustration (dimensions in millimeters) Service clearance for IBM Storage Scale System 6000 The service clearance area is the area around the rack that is needed for the authorized service representatives to service the enclosure as shown in the sample illustration in the following figure.

  • Page 39: Service Clearance Requirements

    Figure 12. Sample illustration to show space around the rack For IBM Storage Scale System 6000, use the recommended measurements that are given in the following table. Table 9. Service clearance requirements Front Back 915 mm (36 in.) 915 mm (36 in.) Storage racks require larger service clearances in the front of the rack.

  • Page 40: Supported Drives

    Supported drives The following table provides drive specifications for your IBM Storage Scale System 6000 system. Table 11. Drive specifications Model and type 2.5 in. drives 4U IBM Storage Scale System 6000, with 48 drive 12 or 24 dual-port NVMe drives slots of 2.5 in.

  • Page 41: Powering On Ibm Storage Scale System 6000

    Powering on IBM Storage Scale System 6000 Powering on an IBM Storage Scale System 6000 by connecting cables The following devices are needed to power on an IBM Storage Scale System 6000: • IBM Storage Scale System 6000 • 4 power cables with C19 connectors •...

  • Page 42: Ibm Storage Scale System 6000 Ports

    • A USB keyboard Complete the following steps to power on an IBM Storage Scale System 6000. 1. Keep the IBM Storage Scale System 6000 on a bench. Figure 14. IBM Storage Scale System 6000 2. Identify all the ports that are needed to power on the system.

  • Page 43: Planning For Site Preparation

    Planning for site preparation This information is intended to help you prepare your physical site for the installation of an IBM Storage Scale System 6000. Marketing and installation planning representatives are also available to help you Chapter 3. Planning for IBM Storage Scale System 6000 hardware  31...

  • Page 44: Site Preparation And Physical Planning

    IBM Storage Scale System 6000 contains two server canisters. Each IBM Storage Scale System 6000 cluster consists of the following components: • One or more IBM Storage Scale System 6000 systems – in a rack, each system requires a space of 4U (standard rack units).

  • Page 45: Ibm Storage Scale System 6000 Model

    The following figure shows the CPUs and the location of the DIMM slots. It shows one of the two server canisters for an IBM Storage Scale System 6000 (MTM 5149-F48), which is open and oriented for DIMM removal and replacement.

  • Page 46: Planning For Power For Server Canister

    Attention: The power cords are the main power disconnect. Ensure that the socket outlets are located near the equipment and are easily accessible. The following figure shows the rear view of an IBM Storage Scale System 6000. The power modules are located on the sides of the IBM Storage Scale System 6000.

  • Page 47: Physical Installation Planning

    “Environmental requirements” on page 24. • Do you have adequate rack space for your hardware? The system requires two Electronic Industries Alliance (EIA) units for each IBM Storage Scale System 6000. For more information, see “Dimensions and weight requirements for rack installation” on page •...

  • Page 48: Operating Environment

    As you plan your installation, you must consider IP address requirements and service access for the system. IBM Storage Scale System 6000 uses 100 GbE / 200 GbE / 100 Gb EDR / 200 Gb HDR network for cluster communication and data transport. For IP port usage requirements, see IBM Storage Scale Documentation.

  • Page 49: Planning For Cables

    Planning for cables Connections for IBM Storage Scale System 6000 Care must be taken to note the orientation of each IBM Storage Scale System 6000 so that the interconnect cables are properly connected. The following figure shows the ports on IBM Storage Scale System 6000.

  • Page 50: Summary Of Supported Host Interface Adapters

    Used with switch/ adapter Single port NDR OSPF 30 m AJQU InfiniBand CX-7, single port OSFP. transceiver, 400 Gbps, flat-top single MPO/ APC, 8 Watts. For more information, see NVIDIA documentation. 38  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 51: Planning For Adapters

    OSFP: octal small form factor pluggable. QSFP: quad small form factor pluggable. Planning for adapters Each IBM Storage Scale System 6000 server has two PCIe interface slots to support extra adapters. The following two adapters are supported for use in the IBM Storage Scale System 6000: •...
  • Page 52 40  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 53: Chapter 4. Installing

    Note: An IBM intranet connection is required to access this content. Note: Much of the information in this section is intended only for IBM authorized service providers. Customers need to consult the terms of their warranty to determine the extent to which they must attempt any IBM Storage Scale System 6000 maintenance.

  • Page 54: Unpacking Ibm Storage Scale System 6000 (Ibm Ssr Task)

    5. Remove the four corner reinforcement pieces from the carton. 6. Using the box knife, carefully cut the four corners of the carton from top to bottom. 7. Fold the sides and back of the carton down to uncover the rear of the IBM Storage Scale System 6000 system.
  • Page 55 Figure 22. Pulling the inner rail b) Slide and hold the release tab in the direction shown in  a  in the next figure. c) Remove the inner sliding rail from the outer mounting rail. See  b  in the following figure. Figure 23.
  • Page 56 Retract and hold the front retention bracket. See  a  in the following figure. b) Insert the pegs on the mounting flange into the rack holes. See  b  in the following figure. 44  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...
  • Page 57 c) Release the front retention bracket. See  c  in the following figure. Figure 26. Mounting the front left rail to the rack posts 4. Release the front retention bracket. a) Retract and hold the rear retention bracket. See  a  in the following figure. b) Insert the pegs on the mounting flange into the rack holes.

  • Page 58: Installing Enclosures (Ibm Ssr Task)

    Following your enclosure location plan, install the IBM Storage Scale System 6000 system. CAUTION: • To lift an assembled IBM Storage Scale System 6000 system, you must use a mechanical lift or involve four people. Do NOT attempt to reduce the enclosure weight by temporarily removing server components from the enclosure.
  • Page 59 Figure 29. Lifting the system by using a mechanical lift 2. On both sides, engage the inner rail with the outer rail. 3. Slide the system until it stops. 4. Pull the release tab on the inner rail to release the lock. Then, push the system further into the rack. Chapter 4.
  • Page 60 Figure 30. Release tab of the inner rail 5. Use the slam latch of each ear to lock the system assembly on the rack. See the following figure for reference. 48  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 61: Installing The Cable Management Arm (Ibm Ssr Task)

    Installing the cable management arm (IBM SSR task) Each canister of the IBM Storage Scale System 6000 requires a cable management arm (CMA); a total of two CMAs is required per enclosure. When working on replacing hot-swappable components, use the CMAs to smoothly pull and push the enclosure.
  • Page 62 - Outer rail connector  E  - CMA body connector Figure 32. CMA parts and orientation 2. Install the CMA in the bottom canister: Figure 33. Installing the CMA in the bottom canister 50  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...
  • Page 63 a) Insert the CMA body connector ( E ) into the outer rail connector ( D ). b) Insert the inner CMA connector ( A ) into the inner rail connector ( B ). c) Insert the outer CMA connector ( C ) into the outer rail connector ( D ). 3. Install the CMA in the top canister: Figure 34.
  • Page 64 52  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 65: Chapter 5. Monitoring The System Using Leds

    • PCIe slots System status LEDs System status LEDs provide information about the status of the IBM Storage Scale System 6000. In the front of the system, three status LEDs are available per enclosure in the left ear, as shown in the following figure.

  • Page 66: Server Canister Status Leds

    Identify. Server canister status LEDs These LEDs are used to monitor the status of server canisters and Ethernet ports of the IBM Storage Scale System 6000. The IBM Storage Scale System 6000 server has three status LEDs on the operator panel of each server canister.

  • Page 67: Ethernet And Bmc Ports Status Leds

    Ethernet and BMC ports status LEDs These LEDs are used to monitor Ethernet and BMC ports of the IBM Storage Scale System 6000. The Ethernet ports and the BMC ports on the rear of server canister are for standard RJ45-style connectors.

  • Page 68: Server Canister Status Leds

    Server canister status LEDs These LEDs are used to monitor server canisters of the IBM Storage Scale System 6000. The table describe the server canister LEDs included in the IBM Storage Scale System 6000. Table 25. Description of the canister status LEDs...

  • Page 69: Psu Status Leds

    PSU status LEDs These LEDs are used to monitor power supply units (PSUs) of the IBM Storage Scale System 6000. The PSU status LEDs are visible from the rear of the enclosure. The following figure provides a close view of the power module status LED.

  • Page 70: Ibm Storage Scale System 6000 Psu Status Leds

    Figure 39. Close view of theIBM Storage Scale System 6000 PSU status LEDs The following table describes bi-color PSU status LEDs. Table 26. IBM Storage Scale System 6000 PSU status LEDs Green color Amber color Description DC GOOD and FAULT...

  • Page 71: Drive Status Leds

    These LEDs are used to monitor drives of the IBM Storage Scale System 6000. Each drive carrier assembly of IBM Storage Scale System 6000 includes a set of LEDs that are visible from the top of the drive carrier. On system startup, the green LED illuminates automatically.
  • Page 72 Table 27. IBM Storage Scale System 6000 drive status LEDs (continued) Color States Description Fault Amber No isolated FRU failures on the drive. Blink The drive is being identified by using an identify command. Solid The associated drive has been isolated and is requiring service or replacement.

  • Page 73: Appendix A. Ibm Storage Scale System 6000 - Slot Placement Summary

    Appendix A. IBM Storage Scale System 6000 - Slot placement summary Each slot number and its usage, as well as VM usage when setup, are explained in the slot placement summary. The following table and illustration represent each slot and its usage: Figure 41.
  • Page 74 62  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 75: Appendix B. Pcie 5 X16 Dual-Port Ndr 200 Gb/S Infiniband And Ethernet Connectx-7 Vpi Adapter (Fc: Ajqs)

    ConnectX-7 adapter is a PCI Express (PCIe) generation 5 (Gen 5) x16 adapter. This adapter can be used in a Gen 5 x16 PCIe slot in the IBM Storage Scale System 6000 system. The adapter enables higher HPC performance with new Message Passing Interface (MPI) offloads, such as MPI Tag Matching and MPI AlltoAll operations, advanced dynamic routing, and new capabilities to perform various data algorithms.
  • Page 76 Thermal requirement If you have a IBM Storage Scale System Utility Node, you might be required to set the thermal mode of the system to a setting other than the default setting, depending on your system, adapter, and cable type.

  • Page 77: Accessibility Features For The System

    IBM Documentation, and its related publications, are accessibility-enabled. Keyboard navigation This product uses standard Microsoft Windows navigation keys. IBM and accessibility See the IBM Human Ability and Accessibility Center (www.ibm.com/able) for more information about the commitment that IBM has to accessibility. © Copyright IBM Corp. 2023, 2024...
  • Page 78 66  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 79: Glossary

    Glossary This glossary provides terms and definitions for the IBM Storage Scale System 6000 solution. The following cross-references are used in this glossary: • See refers you from a non-preferred term to the preferred term or from an abbreviation to the spelled- out form.
  • Page 80 A data structure containing key information about a file system. This information includes the disks assigned to the file system (stripe group), the current state of the file system, and pointers to key files such as quota files and log files. 68  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...
  • Page 81 Hardware Management Console (HMC) Standard interface for configuring and operating partitioned (LPAR) and SMP systems. See Hardware Management Console (HMC). IBM Security Key Lifecycle Manager (ISKLM) For GPFS encryption, the ISKLM is used as an RKM server to store MEKs. independent fileset A fileset that has its own inode space.
  • Page 82 From a system management perspective, it is the central coordinator of the cluster. It also serves as a client node in an ESS building block. master encryption key (MEK) A key that is used to encrypt other keys. See also encryption key. 70  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...
  • Page 83 maximum transmission unit (MTU) The largest packet or frame, specified in octets (eight-bit bytes), that can be sent in a packet- or frame-based network, such as the Internet. The TCP uses the MTU to determine the maximum size of each packet in any transmission. See master encryption key (MEK).
  • Page 84 A network protocol that ensures a loop-free topology for any bridged Ethernet local-area network. The basic function of STP is to prevent bridge loops and the broadcast radiation that results from them. 72  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...
  • Page 85 See secure shell (SSH). See Spanning Tree Protocol (STP). symmetric multiprocessing (SMP) A computer architecture that provides fast performance by making multiple processors available to complete individual processes simultaneously. See Transmission Control Protocol (TCP). Transmission Control Protocol (TCP) A core protocol of the Internet Protocol Suite that provides reliable, ordered, and error-checked delivery of a stream of octets between applications running on hosts communicating over an IP network.
  • Page 86 74  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...

  • Page 87: Index

    5 shock specifications 29 site planning 32 site preparation 32 homologation statement 9 space requirements IBM Storage Scale System 6000 27 specifications shock 29 IBM Storage Scale System 6000 vibration 29 acoustic output of drives 29 submitting xi...
  • Page 88 76  IBM Storage Scale System 6000: Hardware Planning and Installation Guide...
  • Page 90 IBM® Product Number: 5765-DME   5765-DAE SC28-3485-02...