1. Manuals
  2. Brands
  3. IBM Manuals
  4. Server
  5. RS/6000 SP
  6. Handbook

IBM RS/6000 SP Handbook

Clustered ibm eserver pseries systems
Hide thumbs Also See for RS/6000 SP:
1
2
3
4
5
6
Table Of Contents
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
Table of Contents

Advertisement

Quick Links

Download this manual See also: Service Manual

Front cover

RS/6000 SP and Clustered IBM
^ pSeries Systems
Handbook
Overview of the RS/6000 SP and
pSeries servers configurations
Information on available
machine types, models, and
feature codes
In-depth discussion of
hardware architectures
Dino Qintero
Christian Courty
Corneliu Holban
Willis C. White III
ibm.com/redbooks

Advertisement

Table of Contents
loading

Related Manuals for IBM RS/6000 SP

Summary of Contents for IBM RS/6000 SP

  • Page 1: Front Cover

    Front cover RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook Overview of the RS/6000 SP and pSeries servers configurations Information on available machine types, models, and feature codes In-depth discussion of hardware architectures Dino Qintero Christian Courty Corneliu Holban Willis C.
  • Page 3 International Technical Support Organization RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook September 2001 SG24-5596-02...
  • Page 4 “Special notices” on page 441. Third Edition (September 2001) This edition applies to Version 3, Release 2 of the IBM Parallel Support Programs for AIX (PSSP) Licensed Program (product number 5765-D51), and AIX Version 4 Release 3 Licensed Program (product number 5765-C34).

  • Page 5: Summary Of Changes

    Clustering configurations are discussed in Chapter 10, “Clustering configurations” on page 285. Changed information Overview of the RS/6000 SP and Clustered ^ pSeries is discussed in Chapter 1, “Introduction” on page 1. SP cluster components are discussed in Chapter 2, “SP internal processor nodes”...
  • Page 6 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 7: Table Of Contents

    1.4.5 Control workstations ........17 1.5 RS/6000 SP and AIX clustering ......18 Chapter 2.
  • Page 8 2.8 332 MHz SMP nodes ........61 2.8.1 332 MHz SMP node system architecture ..... 61 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 9 2.8.2 SMP system bus ........62 2.8.3 Processor/L2 cache controller.
  • Page 10 5.3 SP Switch, 16-port (F/C 4011) ......166 viii RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 11 8.2 Descriptions of adapter supported on the SP ....231 8.2.1 IBM Network Terminal Accelerator adapter (F/C 2403) ..231 8.2.2 SCSI-2 Differential Fast/Wide adapter/A (F/C 2416) .
  • Page 12 8.2.4 IBM Short Wave Serial HIPPI PCI adapter (FC 2732) ..232 8.2.5 Long-Wave Serial HIPPI adapter (F/C 2733) ....233 8.2.6 IBM Network Terminal Accelerator adapter (F/C 2402 &...
  • Page 13 9.1.4 IBM Virtual Shared Disk (VSD) ......264 9.1.5 Security ..........264 9.1.6 Performance Toolbox Parallel Extensions for AIX (PTPE) .
  • Page 14 Supported levels..........414 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 15 IBM Redbooks collections ........
  • Page 16 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 17: Figures

    Figures RS/6000 SP system sample configuration..... . . 10 RS/6000 SP system with SP-attached servers sample configuration . . 11 Clustered Enterprise Server System sample configuration .
  • Page 18 B-10 Boot server frame approach ....... . 403 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 19 Cable information for 2-port multi protocol adapter ....242 PSSP 3.2 software requirements ......267 xvii © Copyright IBM Corp.
  • Page 20 Node upgrade features ........410 IBM LPPs per supported PSSP and AIX release....414 AIX and PSSP support for different node types .

  • Page 21: Preface

    Preface This redbook is an update of the RS/6000 SP Systems Handbook published in 2000. This publication includes Blue Hammer and Blue Hammer Jr. clustering solution configurations. This publication includes the following: Information on M80, H80, 6H1 SP-attached support as well as clustering solutions capabilities with PSSP 3.2.
  • Page 22 Paris. He moved to Montpellier in 1994 to support the SP2 for EMEA in the SP support group. Corneliu Holban is an IBM Advisory IT specialist in New York. He has 8 years of experience in system engineering, technical support, and sales on RS/6000 and RS/6000 SP systems.

  • Page 23: Special Notice

    Comments welcome Your comments are important to us! We want our IBM Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at:...
  • Page 24 Send your comments in an Internet note to: redbook@us.ibm.com Mail your comments to the address on page ii. xxii RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 25: Chapter 1. Introduction

    The RS/6000 SP system is IBM’s family of parallel computing solutions. It provides a state-of-the-art parallel computing system and industry-leading application enablers.

  • Page 26: Overview On Rs/6000 Sp And Clustered ^ Pseries

    1.1 Overview on RS/6000 SP and Clustered ^ pSeries The RS/6000 SP system simultaneously brings dozens to hundreds of RISC processor nodes to a computing problem. Its parallel processing capability enhances computing performance and throughput many times in comparison to serial computing.
  • Page 27 If you outgrow your existing system, you can readily add increments of computing power. The RS/6000 SP system, as a cluster of SP processor nodes, managed by the AIX operating system and PSSP management software from the CWS, is the first step to universal clustering.

  • Page 28: Features And Benefits

    The IBM software offerings for the SP/CES system provide an interlocking set of licensed programs designed to address a wide range of system and application needs.
  • Page 29 The RS/6000 SP system takes the advantages of the industrial leading RS/6000 workstation technology, the latest RS/6000 processors are repackaged for use as SP nodes, and the SP nodes are interconnected by a high-performance, multistage, packet-switched network for inter processor communication in order to perform the parallel functions.
  • Page 30 UNIX technical and commercial computing market. This broad-based success is attributable to the highly flexible and general-purpose nature of the system. RS/6000 SP system can be used for doing a variety of tasks including parallel, serial, batch, and interactive jobs.

  • Page 31: Business Solutions

    Scalability Makes upgrading and expansion easier and allows for transparent application growth. AIX operating system Provides a wealth of multiuser communications and systems management technologies and complies with major industry standards. Additionally, it provides AIX binary compatibility, where most AIX 4 applications already running on other RS/6000 systems can run unmodified.
  • Page 32 Business intelligence Provides scalable database capacity with support of leading parallel databases including IBM DB2 UDB EEE, Oracle Parallel Server, and Informix Dynamic Server AD/XP. This also offers proven scalable performance with leadership TPC-H results and delivers mainframe inter operability for optimal data movement.

  • Page 33: Hardware Components

    SP switches Control Workstation (CWS) These components connect to your existing computer network through a local area network (LAN) making the RS/6000 SP system accessible from any network-attached workstation. The Machine Type (M/T) for the RS/6000 SP system is 9076.

  • Page 34: Rs/6000 Sp System Sample Configuration

    The SP nodes and the SP Switch Router are connected to the SP Ethernet interface of the CWS. The frame with the SP nodes and the SP Switch are connected to the RS-232 interface of the CWS. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 35: Rs/6000 Sp System With Sp-Attached Servers Sample Configuration

    SP Switch Connection RS232C Connection Figure 1-2 RS/6000 SP system with SP-attached servers sample configuration Figure 1-2 shows an RS/6000 SP system with SP-attached servers sample configuration: The SP node and the SP Switch are mounted in a tall frame.

  • Page 36: Processor Nodes

    SP Ethernet interface and to the RS-232 interface of the CWS. 1.4.1 Processor nodes The IBM RS/6000 SP system is scalable from 1 to 128 processor nodes, which can be contained in multiple SP frames. Up to 16 processor nodes (SP nodes) can be mounted in a tall frame, while a short frame will hold up to eight processor nodes (SP nodes).
  • Page 37 There are four types of RS/6000 SP processor nodes: High Nodes Wide Nodes Thin Nodes SP-attached servers and Clustered Enterprise Servers All nodes are available in an SMP configuration and they use PCI architecture. High nodes The high nodes occupy two full drawer locations. These nodes require the 1.93 m tall frame, thus four nodes can be housed on this frame.
  • Page 38 SP-attached servers and Clustered Enterprise Servers The SP-attached server is an ^ pSeries or an RS/6000 Enterprise Server, supported to be configured to operate with an RS/6000 SP system. The Table 1-1 shows the list with the ^ pSeries and RS/6000 Enterprise Servers supported as an SP-attached server.

  • Page 39: Frames

    77. 1.4.2 Frames The IBM RS/6000 SP system frames contain and provide power for processor nodes, switches, hard disk drives and other hardware. A frame feature code provides an empty frame with its integral power subsystem and AC power cable.

  • Page 40: Extension Nodes

    9077-16S models). It is a licensed version of the Lucent GRF switched IP router that has been enhanced for direct connection to the SP Switch. The SP Switch router uses networking cards (Ethernet, FDDI, ATM, Sonet, HIPPI, HSSI) that fit RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 41: Switches

    into slots in the SP Switch Router. In the same way that the SP Switch Router Adapter connects the SP Switch Router directly to the SP Switch, these networking cards enable the SP Switch Router to directly connect to an external network.

  • Page 42: Rs/6000 Sp And Aix Clustering

    These diverse cluster solutions must deliver the needed attributes of QoS, CoD, security, scalability, and manageability. On the IBM AIX operating system environment, we may consider the following different cluster computing models and their main characteristics:...

  • Page 43: Load Balancing Cluster

    WebSphere WebSphere WebSphere Application Server Application Server Application Server Instance 1 Instance 2 Instance 3 Load Balancer Figure 1-4 Load balancing cluster – High Availability Cluster • Increases application or data availability • Two or more nodes (primary and backup) •...

  • Page 44: High Performance Technical Computing Cluster

    • Extreme database scalability • Shared nothing or shared disk • Commercial “Parallel” databases (DB2 EEE, Informix XPS, Oracle Parallel Server) Figure 1-7 on page 21 illustrates a high performance commercial computing cluster. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 45: High Performance Commercial Computing Cluster

    Partition Partition Partition Partition Partition Partition Partition Partition High Performance Interconnect Figure 1-7 High performance commercial computing cluster Server Consolidation (applications are consolidated to reduce costs) – Common Management Domain • Consolidation of different workloads • Heterogeneous building blocks • Central management point-of control •...

  • Page 46: Common Management Domain

    Easier to adopt newer technology Manage all systems from one console The RS/6000 SP system, as an SP universal cluster, can implement any type of the mentioned cluster computing models separately or together. Referring the RS/6000 SP systems and the AIX clusters evolution, we mention...
  • Page 47 Solution Advantage, enabling customers to manage their infrastructures efficiently and effectively. The SP system is an IBM success story, not only because of Deep Blue, but because the SP, as a universal cluster system, fits any customer need. Chapter 1. Introduction...
  • Page 48 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 49: Chapter 2. Sp Internal Processor Nodes

    – The POWER3 SMP High Node (F/C 2054) or Nighthawk node. In 332 MHz SMP architecture: – 332 MHz SMP Wide Node (F/C 2051) or Silver Wide Node. – 332 MHz SMP Thin Node (F/C 2050) or Silver Thin Node. © Copyright IBM Corp.

  • Page 50: Sp Internal Node

    2.1 SP internal node The basic RS/6000 SP building block is the servers or processor node. Each node has a processor or processors, memory, disk drives, and PCI expansion slots for I/O and connectivity such as any other server. There are three types of internal SP nodes: Thin, wide, and high.

  • Page 51: 375 Mhz Power3 Smp Nodes

    The previous nodes are referenced in Appendix B, “Hardware” on page 365. 2.2 375 MHz POWER3 SMP nodes SP 375 MHz POWER3 SMP node system design is based on the IBM PowerPC Architecture and the RS/6000 Platform Architecture. The node is designed as a...

  • Page 52: 375 Mhz Power3 Processor

    3-cycle latency, 1-cycle throughput, and double/single precision Multiply-Add execution rate. Hence, the 375 MHz POWER3 microprocessor is capable of executing four floating-point operations per clock cycle, which results in a peak throughput of 1500 MFLOPS. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 53 Switched memory subsystem I/O hub Switch complex II Switch complex I SP Switch2 adapter Memory cards, 1GB to 64GB ECC memory available RIO-MX MX-PCI MX-PCI High node unit I1 with 32-bit @ 33MHz PCI slots: I2 to I5 with 64-bit @ 33MHz SP Expansion I/O unit MX-PCI Bus features:...

  • Page 54: System Memory

    The I/O hub can also drive up to six SP Expansion I/O units (RIO). Each RIO have eight PCI slots. Figure 2-3 on page 31 shows the rear view of POWER3 High Node and five SP Expansion I/O Units connections. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 55: Service Processor

    Figure 2-3 POWER3 High Node and five SP Expansion I/O Units connections 2.2.4 Service processor The service processor function is integrated on the I/O planar board in the 375 MHz POWER3 SMP node. Service processor function is for initialization, system error recovery, and diagnostics.

  • Page 56: System Firmware And Rtas

    Node type 375 MHz POWER3 SMP High Node Feature code 2058 Processor 64-bit POWER3-II 4-way, 8-way, 12-way, or 16-way Clock 375 MHz Data/instruction L1 cache 64 KB/32 KB L2 cache/processor 8 MB RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 57 Node type 375 MHz POWER3 SMP High Node Standard RAM 1 GB Maximum RAM 64 GB Disk/media bays 2/26 with SP Expansion I/O Unit AIX storage Internal/external boot disks Internal disks installed in pairs to support mirroring Maximum internal storage 946.4 GB with SP Expansion I/O Unit PCI bus speed 33 MHz...
  • Page 58 Figure 2-4 375 MHz POWER3 High Node packaging RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 59: 375 Mhz Power3 Smp Thin And Wide Nodes

    2.2.8 375 MHz POWER3 SMP Thin and Wide Nodes Table 2-2 outlines the 375 MHz POWER3 SMP Thin and Wide Nodes. Figure 2-5 on page 36 shows the 375 MHz POWER3 SMP Wide Node packaging. Table 2-2 375 MHz POWER3 SMP Thin and Wide Nodes Node type 375 MHz POWER3 SMP Thin 375 MHz POWER3 SMP Wide...
  • Page 60 Figure 2-5 375 Mhz POWER3 SMP Wide Node packaging RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 61: 375 Mhz Power3 Smp High Node (F/C 2058)

    Nighthawk-II node. The 375 MHz POWER3 SMP High Nodes (F/C 2058) use PCI bus architecture and have four, eight, twelve, or sixteen 375 MHz 630FP 64-bit processors per node. Your IBM RS/6000 SP system must be operating at PSSP 3.1 APAR IY11093 (or later) to use these nodes.

  • Page 62: Processor Requirements And Options

    128 MB and 256 and 512 MB DIMMs which must be mounted in banks of eight DIMMs. Different capacity DIMMs (in banks of eight) can be mixed on the memory cards. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 63 For the best memory-access bandwidth, memory DIMMs should be distributed evenly across four memory cards. As an example, you can realize better bandwidth by using four bank of 128 MB DIMMs (4 GB total) distributed evenly over four memory cards rather than by using one bank of 512 MB DIMMs (4 GB total) on one memory card.

  • Page 64: Disk Requirements And Options

    2.3.7 Switch and communication adapter requirements and options If you are planning to use the switch in your SP system, you need switch adapters to connect each RS/6000 SP node to the switch subsystem. SP switch adapters have several characteristics and restrictions, including the following: 375 MHz POWER3 SMP High Nodes are not supported with the SP Switch-8.

  • Page 65: I/O Adapters

    Important: For SP Switch systems, the nodes require the SP Switch MX2 Adapter (F/C 4023). For SP Switch2 systems, the nodes require the SP Switch2 Adapter (F/C 4025). 2.3.8 I/O adapters The 375 MHz POWER3 SMP High Node has five PCI (Peripheral Component Interconnect) adapter slots.

  • Page 66: Hard Disk Drive Options

    2.01 m frame or in the 1.25 m frame. Figure 2-3 on page 31 shows a POWER3 High Node rear view and SP Expansion I/O Units connections SP Expansion I/O Unit Placement: IBM suggests that SP Expansion I/O Units be mounted in separate frames, so as not to interfere with switch port utilization.

  • Page 67: 375 Mhz Power3 Smp Wide Node (F/C 2057)

    375 MHz 64-bit processors per node. These nodes are functionally equivalent to an IBM RS/6000 7044-270 workstation. Your IBM RS/6000 SP system must be operating at PSSP 3.1.1 (or later) to use these nodes. The node occupies one full drawer, thus eight nodes can be housed in a tall frame.

  • Page 68: Options

    CPUs. Table 2-5 shows the processor options for the 375 MHz POWER3 SMP Wide Nodes. Table 2-5 375 MHz POWER3 SMP Wide Nodes processor option Feature Code Description Minimum per Maximum per node node 4444 One processor card with two CPUs RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 69: Memory Requirements And Options

    2.4.5 Memory requirements and options 375 MHz POWER3 SMP Wide Nodes require two memory cards and a minimum of 256 MB of memory. These nodes support a maximum of 16 GB of memory; each card has an 8 GB capacity. Memory is supplied by 128, 256, and 512 MB DIMMs which must be mounted in pairs.

  • Page 70: Hard Disk Drive Requirements And Options

    The F/C 1241 option requires a PCI type SCSI adapter F/C 6206. External storage devices can be accessed through an optional Ultra SCSI adapter (F/C 6207), SCSI-2 adapter (F/C 6209), and SSA adapter (F/C 6230). RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 71: Switch And Communication Adapter Requirements

    2.4.7 Switch and communication adapter requirements If you are planning to use a switch in your SP system, you need switch adapters to connect each RS/6000 SP node to the switch subsystem. SP switch adapters have several characteristics and restrictions, including the following:...

  • Page 72: 375 Mhz Power3 Smp Thin Node (F/C 2056)

    375 MHz 64-bit processors per node. These nodes are functionally equivalent to an IBM RS/6000 7044-270 workstation. Your IBM RS/6000 SP system must be operating at PSSP 3.1.1 (or later) to use these nodes. The node occupies half of a drawer (one slot). Up to 16 of these nodes can be housed in a tall frame.

  • Page 73: Options

    Two processors (mounted in one slot). 256 MB of memory. 4.5 GB of mirrored Hard disk drive (with internal booting). An upgraded power system on early-style frames. 2.5.3 Options Available options include the following: Four processors in two slots. Two memory slots supporting up to 16 GB of memory. Two (32-bit) PCI slots for communication adapters.

  • Page 74: Memory Requirements And Options

    4 GB – (8 x 512 MB DIMMs) 4422 8 GB – (16 x 512 MB DIMMs) 4423 12 GB – (24 x 512 MB DIMMs) 4424 16 GB – (32 x 512 MB DIMMs) RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 75: Hard Disk Drive Requirements And Options

    2.5.7 Switch and communication adapter requirements If you are planning to use a switch in your SP system, you need switch adapters to connect each RS/6000 SP node to the switch subsystem. SP switch adapters have several characteristics and restrictions, including the following:...

  • Page 76: Power3 Smp Nodes

    64-bit PowerPC implementation. Since the node conforms to the RS/6000 Platform Architecture, compatibility is maintained for existing device drivers, other subsystems, and applications. The POWER3 SMP node supports AIX operating systems beginning with version 4.3.2. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 77: Power3 Smp Node System Architecture

    2.6.1 POWER3 SMP node system architecture POWER3 SMP node system design is based on the IBM PowerPC Architecture and the RS/6000 Platform Architecture. The node is designed as a bus-based symmetrical multiprocessor (SMP) system using a 64-bit address and a 128-bit data system bus running at a 2:1 processor clock ratio.

  • Page 78: 6Xx Bus

    “I/O subsystem” on page 30. The System firmware and RTAS are similar to 375 MHz POWER3 describe in Section 2.2.5, “System firmware and RTAS” on page 32. Service Processor microcode and System firmware microcode are available in the following Web site: http://www.austin.ibm.com/support/micro/download.html RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 79: 222 Mhz Power3 Smp High Node (F/C 2054)

    Nighthawk Node. The 222 MHz POWER3 SMP High Nodes (F/C 2054) use PCI bus architecture and have either two, four, six, or eight 222 MHz 64-bit processors per node. Your IBM RS/6000 SP system must be operating at PSSP 3.1.1 (or later) to use these nodes.

  • Page 80: Options

    The S3 port is supported only for HACMP serial heartbeat; the S2 port is not supported for this use. A 9- to 25-pin converter cable is included with the node for this connector. – Node-to-node HACMP cable (F/C 3124). – Frame-to-frame HACMP cable (F/C 3125). RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 81: Processor Requirements And Options

    2.7.4 Processor requirements and options 222 MHz POWER3 SMP High Nodes require a minimum of two 222 MHz, PowerPC processors mounted on one card. You can order up to three additional two-processor cards (F/C 4849) to configure the node with a total of sixteen CPUs.

  • Page 82: Disk Requirements And Options

    2.7.7 Switch and communication adapter requirements and options If you are planning to use a switch in your SP system, you need switch adapters to connect each RS/6000 SP node to the switch subsystem. SP switch adapters have several characteristics and restrictions, including the following.

  • Page 83: I/O Adapters

    Important: For SP Switch systems, these nodes require the SP Switch MX2 Adapter (F/C 4023). For SP Switch2 systems, these nodes require the SP Switch2 Adapter (F/C 4025). 2.7.8 I/O adapters The POWER3 SMP High Node has five PCI (Peripheral Component Interconnect) adapter slots.

  • Page 84: Hard Disk Drive Options

    2.01 m frame or in the 1.25 m frame. Figure 2-3 on page 31 shows a POWER3 High Node rear view and SP Expansion I/O Units connections. SP Expansion I/O Unit Placement: IBM suggests that SP Expansion I/O Units be mounted in separate frames, so as not to interfere with switch port utilization.

  • Page 85: 332 Mhz Smp Nodes

    2.8 332 MHz SMP nodes On April 21, 1998, IBM first introduced the PCI-based nodes, the RS/6000 SP 332 MHz SMP thin nodes and wide nodes. They provide two or four way symmetric multiprocessing (SMP) utilizing PowerPC technology and extend the RS/6000 PCI I/O technology to the SP system.

  • Page 86: Smp System Bus

    There is no sequential ordering requirement, and each operation is tagged with an 8-bit tag, which allows a maximum of up to 256 transactions to be in progress in the RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 87: Processor/L2 Cache Controller

    system at any one time. The address bus includes status and coherency response buses for returning flow control, error reports, or coherency information for each request. It can support a new request every other bus cycle at a sustained rate of over 40 million/sec. The coherency protocol used is an enhanced Modified Exclusive Shared and Invalid (MESI) protocol that allows for cache-to-cache transfers (intervention) on both modified and shared data found in the L2 or L1 caches.

  • Page 88: Memory-I/O Controller

    Writes will assume priority if the write queue is full or there is a new read address matching a write in the write queue. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 89: I/O Bridge Bus

    The SDRAM DIMM itself contains multiple internal banks. The 32 MB DIMM has two internal banks, and the 128 MB DIMM has four internal banks. A different non-busy bank can be accessed to keep the memory fully utilized while previously accessed banks are precharging. In general, more banks of memory are better and allow the controller to schedule non-busy banks.

  • Page 90: Pci I/O Bridges

    The combination of high performing, independent I/O bridges and the node split bus architecture yields outstanding I/O performance and scalability. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 91: 332 Mhz Smp Wide Nodes (F/C 2051)

    With few exceptions, the ten PCI slots in the 332 MHz SMP Wide Node can be used for any valid RS/6000 SP PCI system adapter. While most PCI adapters will function in any 332 MHz SMP Wide Node slot, the following adapters cannot...

  • Page 92: Requirements

    332 MHz SMP Wide Nodes occupy one full node drawer. These nodes are symmetrically configured for memory, disk, and adapters. Up to eight 332 MHz SMP Wide Nodes may be installed in one tall frame and up to four in a short frame. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 93: Options

    The mandatory requirements are: PSSP 2.4 (or later) on the control workstation, backup nodes, and processor node Two processors (mounted in one slot) 256 MB of memory 4.5 GB of hard disk drive (with internal booting) An upgraded power system on older frames 2.9.4 Options Each 332 MHz SMP wide node is functionally equivalent to an RS/6000 7025-F50 and has the following options:...

  • Page 94: Memory Requirements And Options

    4.5 GB Ultra SCSI disk drive pair (F/C 2904) 9.1 GB Ultra SCSI disk drive (F/C 2908) 9.1 GB Ultra SCSI disk drive pair (F/C 2909) 18.2 GB Ultra SCSI disk drive pair (F/C 2918) RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 95: Switch Adapter Requirements And Options

    Silver Thin Node. The 332 MHz SMP thin nodes (F/C 2050) have PCI bus architecture and use either two or four 332 MHz PowerPC processors per node. These nodes are functionally equivalent to an IBM RS/6000 7025-F50 workstation. The SP system must be operating at PSSP 2.4 (or later) to use these nodes.

  • Page 96: Bus Description

    Therefore, the single SMP thin node in these frames is the last node in the configuration. A single POWER3 SMP thin node and a single 332 MHz SMP thin node each occupy one half of a node drawer. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 97: Requirements

    Single POWER3 SMP thin nodes and single 332 MHz SMP thin nodes may be mixed in a thin node drawer. If a frame has more than six single SMP thin nodes installed, that frame will have an uneven weight distribution. You must be careful when moving these frames.

  • Page 98: Processor Requirements And Options

    SCSI-2 network. The 332 MHz SMP thin node can have either no internal disk (with external booting) or from 4.5 GB to a maximum of 36.4 GB of internal disk storage (18.2 mirrored). RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 99: Switch Adapter Requirements And Options

    External storage devices can be accessed through an optional Ultra SCSI Adapter (F/C 6207) or SCSI-2 Adapter (F/C 6209) and SSA adapter (F/C 6230). Optional direct access storage devices are available as follows: 4.5 GB Ultra SCSI disk drive (F/C 2900) 4.5 GB Ultra SCSI disk drive pair (F/C 2904) 9.1 GB Ultra SCSI disk drive (F/C 2908) 9.1 GB Ultra SCSI disk drive pair (F/C 2909)
  • Page 100 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 101: Chapter 3. Sp-Attached Servers And Clustered Enterprise Servers

    Chapter 3. Clustered Enterprise Servers The RS/6000 SP system, as a cluster of SP nodes, managed by the AIX operating system and PSSP management software from the Control Workstation (CWS), is the path to universal clustering. Managed by the AIX and PSSP from the CWS, ^ pSeries and RS/6000 Enterprise Servers, can function as SP-attached servers on an SP system.

  • Page 102: Overview

    3.1 Overview The RS/6000 SP system, as a cluster of SP nodes and managed by the AIX operating system and PSSP management software from the Control Workstation, is the path to universal clustering. We may consider SP systems with SP-attached servers. We may consider also Clustered Enterprise Server systems.

  • Page 103: Sp-Attached Servers / Ces - M/T 7017 Supported Models

    3.2.1 ^ pSeries 680 Model S85 (7017-S85) The 7017-S85 server is the latest member of the popular S-family of 64-bit enterprise servers from IBM. The 7017-S85 server blends copper silicon-on-insulator technology and larger memory capacity with a proven system architecture to give you reliability you can count on. Scalability is enhanced by increasing the amount of memory to 96 GB by leveraging the performance enhancements of AIX Version 4.3.3 and the options of server clustering or SP...
  • Page 104 IBM. The customer is responsible for placing an order for the additionally activated processors with their IBM or IBM Business Partner sales representative. The 7017-S85 server is packaged as a Central Electronics Complex (CEC) and an I/O rack.
  • Page 105 Commercial performance in 4-, 6-, 12-, 18-, or 24-way enterprise servers using the RS64 IV microprocessor technology. Capacity Upgrade on Demand which allows additional processors to be easily added to the pSeries 680 as your business expands. Reliability, availability, and serviceability (RAS) providing: –...
  • Page 106 RAID support SSA. The operating system provides RAID 0 and 1. The optional SSA RAID adapter provides RAID level 0, 1, and 5. Figure 3-1 illustrates the system architecture for 7017-S85 server. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 107: System Architecture For 7017-S85 Server

    Switched memory subsystem I/O hub Switch complex II Switch complex I Memory quads, 4GB to 96GB ECC memory available CEC unit Primary I/O drawer unit Bus features: L2 cache bus: 256-bit @ 300 MHz System bus: 128-bit @ 150MHz I/O hub bus: 128-bit @ 150MHz Memory bus: 512-bit @ 75MHz RIO bus: 8-bit @ 250MHz DUPLEX I/O controller bus: 64-bit @ 66MHz...

  • Page 108: Publications For The 7017-S85 Server

    Note: The IBM pSeries 680 Model S85 (7017-S85) is now available with 450 MHz RS64 III processors as well. Model S85 configurations equipped with 450 MHz processors are available in 6-, 12-, 18-, and 24-way configurations. If additional processing power is needed, customers can replace their 450 MHz processors with 600 MHz processors.

  • Page 109: Rs/6000 Enterprise Server 7017 Model S80 (7017-S80)

    3.2.2 RS/6000 Enterprise Server 7017 Model S80 (7017-S80) The 7017-S80 server is the member of the popular S-family of 64-bit enterprise servers from IBM. The 7017-S80 blends new copper technology and enhanced backplane speeds with a proven system architecture to provide reliability you can count on.
  • Page 110 – Each I/O drawer supports 14 33 MHz PCI bus slots (five 64-bit and nine 32-bit) and 12 disks that can be hot plugged Support for Ultra SCSI, SSA, and FCAL interfaces Support for 32-bit and 64-bit applications AIX license included, can be preinstalled RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 111 Description The 7017-S80 provides performance and supports the evolution of your enterprise to 64-bit computing. Its outstanding benefits include: Commercial performance in 6-, 12-, 18-, or 24-way enterprise servers using the RS64 III microprocessor technology. Reliability, availability, and serviceability (RAS) providing: –...

  • Page 112: Publications For The 7017-S80 Server

    Table 3-3 Publications for the 7017-S80 server Order Number Title SA38-0558 Enterprise Server S80 pSeries 680 Model S85 Installation Guide SA23-0557 Enterprise Server S80 pSeries 680 Model S85 User’s Guide SA38-0538 PCI Adapter Placement Reference RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 113: Rs/6000 Enterprise Server 7017 Model S7A (7017-S7A)

    The 7017-S7A (S70 Advanced) server is a member of the S- family - the generation of 64-bit, 4-way, 8-way, or 12-way, symmetric multiprocessing (SMP) enterprise servers from IBM. The 7017-S7A provides the power, capacity, reliability, and expandability to help move you into the mission-critical commercial computing.
  • Page 114 Commercial performance in 4-, 8-, or 12-way enterprise servers, using the RS64 II microprocessor technology. Reliability, availability, and serviceability (RAS) providing: – Error checking and correcting (ECC) system memory and cache – I/O link failure recovery RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 115 – Hot-swappable disk drives – Environmental sensing – N+1 Power – Service processor for integrated system monitoring – Concurrent diagnostics Preconfigured tailored solutions that take advantage of the rack packaging to custom tailor your solution. Choose from multiple SCSI and SSA: –...

  • Page 116: Rs/6000 Enterprise Server 7017 Model S70 (7017-S70)

    The 7017-S70 server is the first member of the S-family, which is the generation of 64-bit, 4-way to 12-way symmetric multiprocessing (SMP) enterprise servers from IBM. The 7017-S70 provides the power, capacity, reliability, and expandability to help move you into the mission critical commercial computing.
  • Page 117 Highlights This system is the first 64-bit-ready member of the Enterprise Server Family. It delivers a cost-efficient growth path to the future through these attributes: 4-way to 12-way, 64-bit, 125 MHz system scalability – Packaged on three processor cards (four RS64 processors per card) –...
  • Page 118 The 7017-S70 is the natural growth path for the Models R30, R40, and R50. It offers more performance, memory, and storage, and the added advantage of running your current 32-bit applications alongside new 64-bit solutions. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 119: Summary Features Of The Sp-Attached Servers/Ces - M/T 7017

    SA38-0509 Diagnostic Information for Multiple bus systems The publications are available for download from the Internet: http://www.ibm.com/servers/eserver/pseries/library 3.2.5 Summary features of the SP-attached servers/CES - M/T 7017 A summary of the features of the SP-attached servers/Clustered Enterprise Servers - M/T 7017 is shown in Table 3-6.
  • Page 120 ROLTP performance 736.0 736.0 136.7 62.2 Note: Memory standard/maximum is shared memory. L1 cache and L2 cache are per processor. At least three PCI slots are already occupied by mandatory PCI adapters. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 121: Sp-Attached Servers/Ces - M/T 7026 Supported Models

    3.3.1 RS/6000 Enterprise Server 7026 Model M80 (7026-M80) The 7026-M80 server is a mid-range member of the 64-bit family of symmetric multiprocessing (SMP) enterprise servers from IBM. Positioned between the Model H80 and the powerful Model S80, the 7026-M80 server provides the power, capacity, and expandability required for e-business mission-critical computing.
  • Page 122 32 GB of system memory 56 available hot-plug PCI slots 7 available media bays The 7026-M80 CEC and I/O drawers are mounted in the IBM 7014 rack. Additional space in the 7014 rack may be utilized for various storage and communications subsystems.
  • Page 123 The 7026-M80 server is shipped and delivered with all internal adapters and devices already installed and configured. AIX Version 4.3.3 software is included with every 7026-M80 and can be preinstalled. The 7026-M80 provides performance and supports the evolution of your enterprise to Web integration and 64-bit computing.
  • Page 124 RAID 0 and 1. The optional SCSI and SSA RAID adapters provide RAID level 0, 1, and 5. Environmentally-induced (AC power) failure. Figure 3-2 illustrates the system architecture for the 7017-M80 server. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 125: System Architecture For 7026-M80 Server

    CEC unit Memory octals, 1 to 64 GB ECC memory 8-Word memory bus 512-bit @ 62.5 MHz 4MB L2 RS64 III 4MB L2 SRAM RS64 III 500 MHz SRAM 250 MHz 500 MHz 250 MHz 4MB L2 RS64 III 4MB L2 SRAM RS64 III 500 MHz...
  • Page 126 In addition, an HACMP packaged solution with dual 7026-M80s is offered to provide industry-leading high availability.The high availability solutions provided with the 7026-M80, can be ordered in a high-availability configuration with a choice of components. The high-availability configurations combined with RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 127: Pseries 660 Model 6H1 (7026-6H1)

    3.3.2 ^ pSeries 660 Model 6H1 (7026-6H1) The 7026-6H1 server is a member of the 64-bit family of symmetric multiprocessing (SMP) servers from IBM. It offers a design that makes it a solution for mission-critical ERP/SCM, Data Warehouse/Data Mart, OLTP, and e-business applications.
  • Page 128 – 14 available hot-plug PCI bus slots – Two media bays Support for SCSI, SSA, and Fibre Channel attached storage systems Support for 32-bit and 64-bit applications AIX Version 4.3.3 or 5.1 license included RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 129 Two I/O drawers incorporating: – 28 available hot-plug PCI slots – Three available media bays The 7026-6H1 CEC and I/O drawers are mounted in the IBM 7014 rack. Additional space in the 7014 rack may be utilized for various storage and communications subsystems.

  • Page 130: System Architecture For 7026-6H1 Server

    The incorporation of hot-plug PCI I/O slots brings new and increased levels of availability for this system. The need to power down and reboot when adding or replacing most PCI adapters is RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 131 eliminated, thus reducing time delays while keeping network and application processing up and running. The 7026-6H1 can accommodate one processor module in its smaller configuration and grow to two, four, or six processors as required, offering scalability as your business needs dictate. The 7026-6H1 server is targeted at the key commercial processing segments of e-business, ERP, SCM, and Business Intelligence.

  • Page 132: Pseries 660 Model 6H0 (7026-6H0)

    3.3.3 ^ pSeries 660 Model 6H0 (7026-6H0) The 7026-6H0 server is a member of the 64-bit family of symmetric multiprocessing (SMP) servers from IBM. It delivers a cost-efficient growth path to the future through such attributes as 64-bit system scalability with the following processor options: –...

  • Page 133: Rs/6000 Enterprise Server 7026 Model H80 (7026-H80)

    3.3.4 RS/6000 Enterprise Server 7026 Model H80 (7026-H80) The 7026-H80 server is a member of the 64-bit family of symmetric multiprocessing (SMP) enterprise servers from IBM. A follow-on to the popular Model H70, the 7026-H80 provides the power, capacity, and expandability required for e-business, mission-critical computing.
  • Page 134 16 GB of system memory 14 available hot-plug PCI slots One available media bay The 7026-H80 CEC and I/O drawers are mounted in the IBM 7014 rack. Additional space in the 7014 rack may be utilized for various storage and communications subsystems.
  • Page 135 Commercial performance in 1-, 2-, 4-, or 6-way enterprise servers using the RS64 III microprocessor technology. Reliability, availability, and serviceability (RAS) providing: – Error checking and correcting (ECC) system memory and cache – I/O link failure recovery – Environmental sensing –...
  • Page 136 In e-business, the 7026-H80 can serve as a business- to-business Web server, going to other systems for business data or hosting the data storage itself. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 137 The high-availability configurations combined with ClusterProven software give you a superior base for your high-availability requirements. The IBM high-availability solutions also provide a complete and simple process for deciding which components are required and what software is best suited to a high- availability environment.

  • Page 138: Summary Features Of The Sp-Attached Servers/Ces - M/T 7026

    2,4,6,8-way 1,2,4,6-way 1,2,4,6-way Clock rate MHz 450 (1,2,4-way) (1,2,4-way) 500 (6-way) 600 (1,2,4-way) (1,2,4-way) 668 (6-way) 668 (6-way) Memory standard/maximum 1GB/32GB 512MB/32GB 512MB/32GB Memory ECC Data/Instruction (L1) cache 128KB/128KB 128KB/128KB 128KB/128KB RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 139 Machine type and Model 7026-M80 7026-6H1 7026-H80 L2 cache 2MB (1-way) 2MB (1-way) 4MB (2,4-way) 4MB (2,4,6-way) 8MB (6-way) L2 ECC Service Processor Hot-swappable disks External only External only External only Dynamic CPU Deallocation Hot-plug slots Redundant hot-plug power Optionally Optionally Redundant hot/plug cooling PCI slots available...

  • Page 140: Cable And Pci Card For Sp Cws Attachment (F/C 3154)

    (S1TERM), as well as power, are provided through the ribbon cable. The device driver support is in the server base microcode. F/C 3154 provides S1TERM, Hardmon, Sysmon and PSSP Perspectives capabilities. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 141: Sp-Attached Server Environment

    The SP-attached servers use RS/6000 feature codes, not RS/6000 SP feature codes. The RS/6000 SP feature codes associated with the SP-attached server refers only to the system connections that attach the server to the RS/6000 SP system, they do not refer to hardware components. Since the SP-attached...
  • Page 142 2. Each SP-attached server requires one valid, unused node slot in the SP system for switch port assignment. An assigned switch port is required in both switch-configured and switchless SP systems. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 143: Sp-Attached Server Installation Requirements

    3. In some cases, the number of SP-attached servers you plan to install can exceed the number of available node slots in an SP frame. In this case, you can take advantage of any valid, unused node slots (and the associated switch ports) that exist in other SP frames in your system.
  • Page 144 SP Switch adapter in that server. The adapter you use to connect the SP-attached server to the SP Switch is the RS/6000 SP System Attachment adapter (F/C 8396). Because the RS/6000 SP System Attachment adapter is not an SP system adapter, you must order it using an RS/6000 feature code.
  • Page 145 Note: Consult the appropriate documentation for specific installation procedures. When you order an RS/6000 SP System Attachment adapter as server feature code, you must also order the SP Switch cable that connects the adapter to a valid switch port on the SP Switch. The SP Switch cable must be ordered with the RS/6000 SP system.

  • Page 146: Planning The Sp-Attached Server Interfaces

    Chapter 8, “PCI communication adapters” on page 227. With the exception of the RS/6000 SP System Attachment adapter (which is supported), if an adapter does not appear in the list, it is not supported for the SP-attached server. If you...
  • Page 147 Each SP-attached server M/T 7017 must be attached to the control workstation via the following two serial cable features: A custom, IBM-supplied, 15 m (49 ft.) RS-232 cable connection from the control workstation to the SP-attached server SAMI port on the left side of the front operator panel A custom, IBM-supplied 15m (49 ft.) RS-232 cable connection from the...
  • Page 148 There are some SP-attached server placement limitations. Placement of the SP-attached server is limited by the length of its following supplied cables: 1. The 10 m (33 ft.) frame-to-frame ground cable and RS/6000 SP System Attachment adapter cable 2. The 15 m (49 ft.) RS-232 cables 3.
  • Page 149 Connections needed for SP-attached servers - M/T 7026 Each SP-attached server M/T 7026 must be attached to the control workstation via a serial cable feature. It is a custom, IBM-supplied, 15 m (49 ft.) RS-232 cable connection from the control workstation to the SP-attached Server internal Attachment Adapter (F/C 3154) in slot #7 of the primary I/O drawer.

  • Page 150: Sp-Attached Server Configurator Rules

    There are some SP-attached server placement limitations. Placement of the SP-attached server is limited by the length of its following supplied cables: 1. The 10 m (33 ft.) frame-to-frame ground cable and RS/6000 SP System Attachment adapter cable. 2. The 15 m (49 ft.) RS-232 cables.
  • Page 151 15 Meter RS-232 cables for hardware control and connectivity between the CWS and the SP-attached server, and a 10 meter ground cable. It also traps some data so that the RS/6000 SP system configurator session can keep track of how many nodes (real and logical) are in the system.

  • Page 152: Sp-Attached Servers Connectivity Diagram

    Figure 3-4 SP-attached servers connectivity diagram 3.4.5 Configuring Service Director on SP-attached servers Service Director, as set of IBM software applications supplied with the Enterprise Servers M/T 7017 and M/T 7026, monitors the “health” of the system. In a typical Enterprise Server installation, Service Director transmits reports through a modem supplied with that server.

  • Page 153: Clustered Enterprise Server Environment

    Service Director and Electronic Service Agent. 3.5 Clustered Enterprise Server environment The Clustered Enterprise Servers use RS/6000 feature codes, not RS/6000 SP feature codes. The RS/6000 feature codes associated with the Clustered Enterprise Server refer to the cable connections that attach the servers to the control workstation.
  • Page 154 Switch, you must install an SP Switch adapter in that server. The adapter you use to connect the Clustered Enterprise Server to the SP Switch is the RS/6000 SP System Attachment adapter (F/C 8396). Because the RS/6000 SP System Attachment adapter is not an SP system adapter, you must order it using an RS/6000 feature code.
  • Page 155 For example, if you attach an existing 7017 Enterprise Server to a CES system, you might find a SCSI adapter installed in slot #9 of the server. However, the SCSI adapter in slot 9 is typically connected to a boot device and requires special attention before the SCSI adapter is removed.

  • Page 156: Planning The Clustered Enterprise Server Interfaces

    Chapter 8, “PCI communication adapters” on page 227. With the exception of the RS/6000 SP System Attachment adapter (which is supported), if an adapter does not appear in the list, it is not supported for the CES. If you plan to use an...
  • Page 157 The interconnection between servers in the CES system, can be accomplished utilizing the IBM RS/6000 SP System Attachment Adapter (F/C 8396) for the SP Switch connection. If used, the adapter has to be ordered as a server feature code and it must be installed in slot #10 of the Clustered Enterprise Server primary I/O drawer;...
  • Page 158 There are some Clustered Enterprise Server placement limitations. Placement of the Clustered Enterprise Server is limited by the length of its following supplied cables: 1. The 10 m (33 ft.) frame-to-frame ground cable and RS/6000 SP System Attachment adapter cable 2. The 15 m (49 ft.) RS-232 cables Approximately 3 m (10 ft.) of cable is needed for the vertical portion of the cable...
  • Page 159 Note: The customer must supply all required Ethernet cables. The interconnection between servers in the CES system, can be accomplished utilizing the IBM RS/6000 SP System Attachment Adapter (F/C 8396) for the SP Switch connection. If used, the adapter has to be ordered as a server feature code and it must be installed in slot #6 of the server primary I/O drawer.

  • Page 160: Clustered Enterprise Servers Connectivity Diagram

    There are some Clustered Enterprise Server placement limitations. Placement of the Clustered Enterprise Server is limited by the length of its supplied cables: 1. The 10 m (33 ft.) frame-to-frame ground cable and RS/6000 SP System Attachment adapter cable 2. The 15 m (49 ft.) RS-232 cables Approximately 3 m (10 ft.) of cable is needed for the vertical portion of the cable...

  • Page 161: Configuring Service Director On Ces Systems

    Figure 3-5 Clustered Enterprise Servers connectivity diagram 3.5.4 Configuring Service Director on CES systems Service Director, a set of IBM software applications supplied with the Enterprise Servers M/T 7017 and M/T 7026, monitors the “health” of the system. In a typical Enterprise Server installation, Service Director transmits reports through a modem supplied with that server.
  • Page 162 Clustered Enterprise Server. The Service Director modem for the CES system is attached to the control workstation. IBM intents to replace Service Director with Electronic Service Agent. Note: Refer Appendix C, “SP system upgrades” on page 407 for more details about Service Director and Electronic Service Agent.

  • Page 163: Chapter 4. Frames

    Tall frames – Tall model frames (Model 550, 555) – Tall expansion frames (F/C 1550) – SP Switch frames (F/C 2031, 2032) Short frames: – Short model frames (Model 500) – Short expansion frames (F/C 1500) © Copyright IBM Corp.

  • Page 164: Frames

    This led to an increasing number of possible prepackaged configurations when more nodes became available. Figure 4-1 on page 141 shows the tall model frame (Model 550). RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 165 Figure 4-1 SP frame model 550 The product structure of the RS/6000 SP system was simplified with the introduction, on April 21, 1998, of a new frame. The new SP system was simplified with only two models (Model 500 and Model 550) and two expansion frame features (F/C 1500 and F/C 1550).

  • Page 166: Sp Frame Dimensions

    With the new offering, IBM simplified the SP frame options by decoupling the imbedded node from the frame offering. Therefore, when you order a frame, all you receive is a frame with the power supply units and a power cord.

  • Page 167: Frame Electrical Power Requirements

    4.1.2 Frame electrical power requirements The electrical power specifications for the old and new power subsystems are summarized in Table 4-2. Table 4-2 SP frame electrical power requirements Power Requirements Tall frame Short frame Legacy Legacy Model Legacy Model SEPBU SP Frame Power output (kW) 10.5...

  • Page 168: Sp Frame Redundant Power Supply (F/C 3885, 3886)

    (F/C 3886). Figure 4-2 shows the rear of the redundant power supply feature. Dual Linecord Bump Rear view of new SEPBU Figure 4-2 Rear of the redundant power supply feature RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 169: Tall Frames

    4.2 Tall frames Tall frames are available in three variations: Tall model frames (Model 550, 555) Tall expansion frames (F/C 1550) SP switch frames (F/C 2031, 2032) Figure 4-3 shows a tall frame’s front and rear views. Processor Node LED and Control Breaker Processor Node (Slot 8) Processor Node...

  • Page 170: Model 550 Frames

    Some of these configurations use switches and some do not. 4.3.1 Model 550 non-switched configuration This configuration consists of 1-64 processor nodes mounted in one required Model 550 frame and in additional tall, non-switched expansion frames (F/C 1550). RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 171: Model 550 Sp Switch-8 Configuration

    4.3.2 Model 550 SP Switch-8 configuration This configuration consists of one to eight processor nodes mounted in one required Model 550 frame equipped with an eight-port SP Switch-8 (F/C 4008). A non-switched expansion frame (F/C 1550) is supported in this configuration only if the model frame is filled before the total node count of eight is reached.

  • Page 172: Model 555 Standalone Switch For Clustered Nodes

    SP Switch. The SP Switch ports in these frames connect to the SP System Attachment adapters (RS/6000 F/C 8396) in the CES servers. Figure 4-4 on page 149 shows a 9076-555 model 555 with two SP-attached servers and a CWS. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 173: Expansion Frames (F/C 1550)

    9076 mod 555 ground cable ground cable 10m/20m Switch Cables SP Ethernet Frame Supervisor Cable Control Connections (sami) ground cable Figure 4-4 9076 mod 555 with two SP-attached servers and a CWS 4.5 Expansion frames (F/C 1550) Used in Model 550, 3BX, 20X, 30X, 40X, and 55H systems, F/C 1550 is a tall frame with eight empty node drawers and a SEPBU power supply.
  • Page 174 (containing the switch) and one or more additional frames, which do not contain a switch. The additional frames share the switch of the first frame in the set. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 175: Using F/C 1550 Non-Switched Expansion Frames

    Uniprocessor and SMP Thin Node Pair Alert: Frames that have uniprocessor thin nodes cannot be used as non-switched expansion frames. Similarly, if a frame has uniprocessor thin nodes and a switch with unused switch ports, it cannot have a non-switched expansion frame attached to the unused switch ports.

  • Page 176: Using F/C 1550 Switched Expansion Frames

    80 nodes. If your SP system uses two-stage switching, you can scale your SP system even larger. In these systems, 128 nodes (or more) are supported. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 177: Sp Switch Frames (F/C 2031, 2032)

    4.6 SP Switch frames (F/C 2031, 2032) Switch frames are base, tall frames with integral SEPBU power supply equipped with switches but no processor nodes. A switch frame is required for systems having more than five switches; it interconnects the switches in the model and expansion frames.

  • Page 178: Short Frame

    However, the short frame Scalable Electric Power Base Unit (SEPBU) power subsystem has been redesigned to accommodate the latest SP processor nodes. This upgrade is required before you can use SMP wide nodes and SMP thin nodes in older frames. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 179: Model 500 Frames

    Note: In order to maintain your entire SP system at the same electrical potential, you must attach a frame-to-frame ground between all frames in your SP system using IBM cables (P/N 46G5695). 4.8 Model 500 frames model frame is always the first frame in an SP system and it designates the type or model class of your SP system.

  • Page 180: Model 500 Switched Configuration

    F/C 1500 non-switched expansion frames. The model frame can be either of the following: Configured with processor nodes and a switch Configured with processor nodes only RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 181: Configurations With Processor Nodes And A Switch

    Expansion Frame Terminology: Non-switched expansion frames were sometimes previously referred to as “logical switch expansion frames.” Both of these terms indicate a shared-switch frame set. The frames of a shared-switch frame set include the first frame (containing the switch) and one or more additional frames which do not contain a switch.
  • Page 182 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 183: Chapter 5. The Switch

    This chapter describe the two available switch series: The SP Switch or SPS (F/C 4011 and F/C 4008). The SP Switch2 or SPS2 (F/C 4012). This chapter does not describe the previous High Performance Switch (HiPS) series. © Copyright IBM Corp.

  • Page 184: Sp Switch

    An SP Switch link connects two network devices. It contains two channels carrying packets in opposite directions. Each channel includes 8 bits Data, 1 bit Data Valid, and 1 bit Token signal. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 185: Sp Switch Ports

    5.2.2 SP Switch ports An SP Switch port is part of a network device and is connected to other SP Switch ports through the SP Switch link. The SP Switch port includes two ports (input and output ports) for full duplex communication. The following diagram shows the relationship between the SP Switch link and the SP Switch port.

  • Page 186: Sp Switch Chip

    SPSwitch port 8 x 8 Crossbar Output port Output port Input port Input port SPSwitch port SPSwitch port Output port Output port Input port Input port Figure 5-3 SP Switch chip RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 187: Sp Switch Board

    5.2.4 SP Switch board An SP Switch board contains eight SP Switch chips that provide connection points for each of the nodes to the SP Switch network as well as for each of the SP Switch boards to the other SP Switch boards. There are 32 SP Switch ports in total.

  • Page 188: Sp Switch System

    Sixteen SP Switch adapters are connected to 16 of 32 SP Switch ports in the SP Switch board. Another 16 SP Switch ports are available for other SP Switch boards; see Section 5.7.2, “Switch-to-switch cabling” on page 171 for the possible Switch to Switch connections. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 189: Sp Switch System

    SP switch Adapter SP Node SP Node SP switch Adapter Frame SP switch Adapter SP Node SP Node SP switch Adapter SP switch Adapter SP Node SP switch Adapter SP Node SP switch Adapter SP Node SP switch Adapter SP Node SP switch Adapter SP switch Adapter SP Node SP Node...

  • Page 190: Sp Switch, 16-Port (F/C 4011)

    TCP/IP, file transfers, remote procedure calls, and relational database functions. The required SP Switch Adapter (F/C 4020), SP Switch MX Adapter (F/C 4022), or SP Switch MX2 Adapter (F/C 4023) connects RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 191: Sp Switch-8 (F/C 4008)

    SP Switch Frame. Note: An SP Switch Router adapter is needed to connect the RS/6000 SP Switch Router to the SP Switch. An RS/6000 SP System Attachment adapter is needed to connect the SP-attached server and Clustered Enterprise Servers to the SP Switch.

  • Page 192: Sp Switch Adapter (F/C 4020, 4022, And 4023)

    If you plan to use a switch in your SP system, you will need a switch adapter to connect each RS/6000 SP node to the switch subsystem. SP Switches use either the SP Switch adapter (F/C 4020) for MCA-type nodes or the SP Switch MX Adapter (F/C 4022) or SP Switch MX2 Adapter (F/C 4023) for PCI-type nodes.

  • Page 193: Special Sp Switch Adapter (F/C 4021 And F/C 8396)

    Adapter Feature Description Code 4023 SP Switch MX2 adapter (SP switch adapter for POWER3 SMP nodes and 332 MHz SMP nodes) for installation as follows: Optional Order one adapter per PCI node 5.6 Special SP Switch adapter (F/C 4021 and F/C 8396) Some optional SP system equipment requires special adapters in order to connect to the SP Switch network.

  • Page 194: Switch-To-Node Cabling

    The cables for the Clustered Enterprise Servers may be ordered when you configure the Clustered Enterprise Server as a RS/6000 Server attached frame (F/C 9123) and as a RS/6000 Server attached node (F/C 9122) on the 9076-555 Standalone Switch for Clustered nodes. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 195: Switch-To-Switch Cabling

    5.7.2 Switch-to-switch cabling The issues that you must consider when planning switch cable connections between switch-equipped frames in multi-switch SP systems include: 1. Determining the quantity of cables required 2. Determining the length of required cables 3. Minimizing cable lengths to reduce latency 4.

  • Page 196: Single-Stage Switch Cable Paths

    The numbered squares represent switch-equipped frames. The arrows are the cable sets with the quantity of individual cables in each set indicated. Switch-to-node cabling and non-switched expansion frames are not shown. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 197: Two-Stage Switch Cable Paths

    An SP Switch frame (F/C 2031 or F/C 2032) is required for systems using more than five switches (known as two-stage switching). Each switch in a two-stage system is cable-connected only to the SP Switch frame. Figure 5-8 illustrates an example of a two-stage switch configuration that has eight switch-equipped frames.

  • Page 198: Sp Switch2

    SP Switch2 provide a one-way bandwidth of up to 500 MB per second between nodes (1 GB bidirectional) for interconnecting POWER3 SMP high nodes. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 199 Because the SP Switch2 design improvements are evolutionary steps on the SP and High Performance switches, the SP Switch2 is fully compatible with applications written for the older switches. The SP Switch2 software support continues to emphasize usage of MPI (Message Passing Interface), LAPI (Low level Application Interface), and IP (Internet Protocol), but the new switch subsystem enhances message handling so that the switch can keep pace with the increased performance of the POWER3 SMP High Node.

  • Page 200: Sp Switch2 (F/C 4012)

    Like the SP Switch, the SP Switch2 has 32 ports; 16 ports for switch-to-node connections and 16 ports for switch-to-switch connections. Management of the SP Switch2 network is the same as SP Switch, using service packets over the switch plane. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 201 The N+1 feature of these switches allows for concurrent replacement of any failed power supplies or cooling fans. The supervisor can also be replaced while the switch is operating. PSSP level 3.2 software is required to use these switches. This section describes some of the most important hardware characteristics of the SP Switch2 hardware.

  • Page 202: Sp Switch2 Adapter (F/C 4025)

    1000 MB/s uni-direction buses. Figure 5-9 on page 179 shows the SP Switch2 adapter hardware structure. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 203: Sp Switch2 Adapter Hardware Structure

    RDRAM Port-0 6XX BUS Node Fabric Memory Interface Interface Controller Switch2 TBIC3 60X BUS Port-1 Node Controller SRAM (2M) 225MHz System Interface POWER3 SMP high node Figure 5-9 SP Switch2 adapter hardware structure Node Bus Adapter (NBA) The NBA chip moves data between the adapter and POWER3 SMP node’s 6xx bus.
  • Page 204 The 740 microprocessor is responsible for passing control information between the adapter and the software on the POWER3 SMP node, for encoding or formatting packet headers, for building packet routing information, and for handling error conditions. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 205: Chapter 6. Sp Switch Routers 9077

    SP Switch Routers 9077 Chapter 6. This chapter presents the IBM 9077 SP Switch Router within an SP environment only and with the SP Switch only. © Copyright IBM Corp.

  • Page 206: The 9077 Sp Switch Routers

    The IBM 9077 SP Switch Router is a licensed version of the Ascend GRF switched IP router that has been enhanced for direct connection to the SP Switch. IBM remarkets models of the GRF that connect to the SP Switch as the SP Switch Router: IBM 9077 SP Switch Router model 04S (9077-04S) is based on Ascend GRF 400.

  • Page 207: Overview

    If the SP Switch Router contains more than one adapter, it can route data between SP systems. For the RS/6000 SP Switch Router, this card is called a Switch Router adapter (F/C 4021). Data transmission is accomplished by linking the dependent node adapters in the switch router with the logical dependent nodes located in different SP systems or system partitions.

  • Page 208: Sp Switch Router Configuration

    With independent routing available from each media card, the SP Switch Router gives your SP system excellent scalability characteristics. The SP Switch Router has dynamic network configuration to bypass failed network paths using standard IP protocols. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 209: Rear View Of The Sp Switch Router Model 04S

    Each SP Switch Router has redundant (N+1) hot swappable power supplies. Two versions of the RS/6000 SP Switch Router can be used with the SP Switch: The Model 04S offers four media card slots. The Model 16S offers sixteen media card slots.

  • Page 210: Installation Requirements

    Figure 6-4 Rear view of SP Switch Router model 16S 6.3 Installation requirements There are several requirements for hardware and software that must be met before you can place the RS/6000 SP Switch Router into service with your SP system. These requirements are in the following categories: System requirements...

  • Page 211: System Requirements

    Model 04S has the capacity for a total of four SP Switch Router adapters and network media cards in any combination suiting the needs of your SP system. The RS/6000 SP Switch Router Model 16S has the capacity for a total of 16 SP Switch Router adapters and network media cards in any combination.

  • Page 212: Network Media Card Requirements

    6.3.3 Network media card requirements Each network media card requires one media card slot in the RS/6000 SP Switch Router. Remember, the network media cards use the same slots as the SP Switch Router adapters. The following network media cards shown in Table 6-1 are available as options for the SP Switch Router.

  • Page 213: Software Requirements

    6.4 Network interface The RS/6000 SP Switch Router (Machine Type 9077) requires a minimum of three connections with your SP system in order to establish a functional and safe network. These connections are: 1.

  • Page 214: Connecting To The Control Workstation

    All SP administrative connections are made to the SP Switch Router using the port on the Router’s control board. From the SP Switch Router’s control board, the connection to the control workstation is made using one of the following methods: RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 215: Connecting To Multiple Sp Systems

    Ethernet hub (switch). All coax and twisted pair Ethernet cables must be supplied by the customer. 3. The RS/6000 SP Switch Router may also be connected to an additional 10Base-T adapter (such as F/C 2992) that has been installed directly in the control workstation for this purpose.

  • Page 216: Connecting To The Sp Switch

    Switch to improve throughput of data coming into and going out of the RS/6000 SP system. Each SP Switch Router adapter in the RS/6000 SP Switch Router will require a valid unused switch port in the SP system. A valid unused switch port is a switch port that meets the rules for configuring frames and switches.
  • Page 217 Another example is a two-frame system with one switch. The first frame is fully populated with eight wide nodes. The second frame has three wide nodes in system node positions 17, 19, and 21. The only valid switch ports in this configuration would be those switch ports associated with system node numbers 23, 25, 27, 29, and 31 in the second frame.
  • Page 218 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 219: Chapter 7. Control Workstations

    Environment (DCE) can be used with Kerberos V4, or by itself, or nothing (AIX standard security). A high availability solution may be implemented for the Control Workstation. The solution is named the High Availability Control Workstation. © Copyright IBM Corp.

  • Page 220: Overview

    M/T 7017 or Clustered Enterprise Server M/T 7026 requires one RS-232 connection as well as the SP Ethernet connection. Fifteen meter (49 ft.) cables are supplied with each SP-attached server for both RS-232 and BNC Ethernet RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 221 only. You have to order the RS-232 cables for each Clustered Enterprise Server. No BNC Ethernet cables supplied for the Clustered Enterprise servers. On any of these cases, it is the customer’s responsibility to supply the cables for twisted-pair SP Ethernet network (if it exists). Note that most PCI control workstations provide either a 10BaseT or AUI connection for the SP Ethernet LAN.

  • Page 222: Installation Requirements

    Make sure you have ordered them in time so they will arrive when the rest of your SP does. To coordinate delivery of the SP and control workstation, your IBM representative should link the SP and control workstation orders with a System Order Number. The installation requirements...

  • Page 223: System Requirements

    Notes: 1. The native RS-232 ports on the system planar can not be used as tty ports for the hardware controller interface. The 8-port asynchronous adapter EIA-232/ RS-422, PCI bus (F/C 2943) or the 128-port Asynchronous Controller (F/C 2944) are the only RS-232 adapters that are supported. These adapters require AIX 4.2.1 or AIX 4.3 on the control workstation.
  • Page 224 – An appropriate network adapter for your external communication network. The adapter does not have to be on the control workstation. If it is not on the control workstation, the SP Ethernet must extend to another host that RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 225: Interface Adapter Requirements

    is not part of the SP system. A backup control workstation does not satisfy this requirement. This additional connection is used to access the control workstation from the network when the SP nodes are down. – SP Ethernet adapters for connection to the SP Ethernet The number of Ethernet adapters required depends completely on the Ethernet topology you use on your SP system.

  • Page 226: Serial Port Adapters For Pci / Mca Control Workstations

    2.4 MB/sec enhanced remote asynchronous node (RAN) 16-port EIA-232 8138 2.4 MB/sec enhanced remote asynchronous node (RAN) 16-port RS-422 8-port MCA 2930 8-port asynchronous adapter 2995 multiport interface cable 16-port MCA RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 227: Software Requirements

    For additional PCI/MCA Ethernet ports, select the equipment you need from the feature codes shown in Table 7-3. Table 7-3 Ethernet adapters for PCI/MCA control workstations Description Currently Available 2968 IBM 10/100 Mbps Ethernet PCI adapter 2985 PCI Ethernet BNC/RJ-45 adapter 2987 PCI Ethernet AUI/RJ-45 adapter 4224...
  • Page 228 An AIX license is required for the CWS. The PSSP ordered for the SP/CES system is entitled for use the entire SP/CES complex, which includes the CWS (and the backup CWS, if used). Compilers are necessary for IBM service of PSSP. Also, without the compiler, dump diagnosis tools like crash cannot work effectively.

  • Page 229: Sp Control Workstation Offerings

    – Event Perspective – Performance Monitor – Virtual Shared Disk Perspective – System Partitioning Aid – SP TaskGuides – SP Resource Center – SP Perspectives Launch Pad – Perspectives online help – Netfinity Services Manager) 7.2.5 SP control workstation offerings The latest CWS models are the RS/6000 7044-170 and 7025-F80, and the pSeries 7025-6F1 and 7026-6H1.
  • Page 230 7025-F80 control workstation small configuration This configuration is recommended for SP systems that include no more than 30 nodes. Example 7-2 on page 207 shows a 7025-F80 control workstation small configuration. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 231 Example 7-2 7025-F80 control workstation small configuration *** HARDWARE *** Product Description 7025-F80 RS/6000 Enterprise Server Model 1.44MB 3.5-in Diskette Drive Integrated Ultra2 SCSI Adapter Integrated Ethernet Adapter 2624 32x Speed CD-ROM 2830 POWER GXT130P Graphics Adapter (PCI) 2943 8-Port Async Adapter, EIA-232/422 (PCI) 2968 10/100 Mbps Ethernet PCI Adapter...
  • Page 232 3109 SCSI External Port to SCSI Internal 6 Pack Cable Assembly 3628 P260 Color Monitor, Stealth Black 4075 Memory Board, 16-position 4110 256 MB (2x128MB) SDRAM DIMMs 4223 Ethernet 10Base2 Transceiver 5005 Preinstall RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 233 5202 2-Way RS64 III 450 MHz Processor Card, 4MB L2 Cache 6159 12GB/24GB 4mm Tape Drive 6553 SCSI Hot Swap 6-Pack 8700 Quiet Touch Keyboard, Stealth Black - English (US) 8741 3-Button Mouse - Stealth Black 9300 Language - English (US) 9800 Power Cord - US/Canada (125V, 15A) *** SOFTWARE ***...
  • Page 234 9800 Power Cord - US/Canada (125V, 15A) *** SOFTWARE *** Product Description 5765-C34 AIX Version 4 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 235 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0903 CWS Diagnostic Pre-Load 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM 5005 Preinstall 5924 AIX 4.3 PII - English 9001 Asset Registration 7026-6H1 control workstation small configuration This configuration is recommended for SP systems that include no more than 30 nodes.
  • Page 236 5765-C34 AIX Version 4 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 237 Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0903 CWS Diagnostic Pre-Load 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM 5005 Preinstall 5924 AIX 4.3 PII - English 9001 Asset Registration 7026-6H1 control workstation medium configuration This configuration is recommended for SP systems that include more than 30 nodes and for SP/CES systems that include no more than 16 enterprise servers.
  • Page 238 8031 Raven Black Drawer Cover 8518 One 10K/18.2GB Advanced Disk Drive Module 8801 1m Advanced SSA Cable 9300 Language - English (US) 9910-A30 APC 5000VA Smart-UPS Rack-Mount SmartSlot 4x4 10BaseT SNMP Adapter (Standard) RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 239 *** SOFTWARE *** Product Description 5765-C34 AIX Version 4 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export.
  • Page 240 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0903 CWS Diagnostic Pre-Load 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM 5005 Preinstall RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 241 5924 AIX 4.3 PII - English 9001 Asset Registration 7025-6F1 control workstation medium configuration This configuration is recommended for SP systems that include more than 30 nodes and for SP/CES systems that include no more than 16 enterprise servers. Example 7-8 shows a 7025-6F1 control workstation medium configuration. Example 7-8 7025-6F1 control workstation medium configuration *** HARDWARE *** Product...
  • Page 242 This configuration is recommended for SP systems with a large number of nodes and for SP/CES systems that includes more than 16 enterprise servers. Example 7-9 on page 219 shows a 7025-6F1 control workstation large configuration. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 243 Example 7-9 7025-6F1 control workstation large configuration *** HARDWARE *** Product Description 7025-6F1 pSeries 620 1.44MB 3.5-in Diskette Drive Integrated SCSI-2 F/W Adapter Integrated Ultra2 SCSI Adapter Integrated Ethernet Adapter 2624 32x Speed CD-ROM 2830 POWER GXT130P Graphics Adapter (PCI) 2944 128-Port Async Controller (PCI) 2968 10/100 Mbps Ethernet PCI Adapter...

  • Page 244: High Availability Control Workstation

    SP system. There are already redundant power supplies and replaceable nodes. However, there are also many elements of hardware and software that could fail on a RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 245: Overview

    control workstation. With a HACWS, your SP system will have the added security of a backup control workstation. Also, HACWS allows your control workstation to be powered down for maintenance or updating without affecting the entire SP system. The design of the HACWS is modeled on the High Availability Cluster Multi-Processing for the AIX (HACMP) licensed program product.

  • Page 246: High Availability Control Workstation

    This means that there is only one active control workstation at any time. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 247: Installation Requirements

    The primary and backup control workstations are also connected on a private point-to-point network and a serial TTY link or target mode SCSI. The backup control workstation assumes the IP address, IP aliases, and hardware address of the primary control workstation. This lets client applications run without changes. The client application, however, must initiate reconnects when a network connection fails.

  • Page 248: Limits And Restrictions

    Two licenses for IBM C for AIX Version 3.6.6 or later or the batch C and C++ Version 3.6.6 or later compilers and runtime libraries of VisualAge C++ Professional Version 4.0 for AIX or later. If the compiler’s license server is on the control workstation, the backup control workstation should also have a license server with at least one license.

  • Page 249: Service Director On The Cws

    Service Director to send an automated E-mail message containing the fault information to your system administrator (requires mail to be active on each node). Upon receiving the fault notification, IBM will automatically dispatch a service engineer (with parts if needed) to correct the problem.
  • Page 250 Note: Refer to Appendix C, “SP system upgrades” on page 407 for more details about the Service Director and the Electronic Service Agent. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 251: Chapter 8. Pci Communication Adapters

    PCI communication Chapter 8. adapters This chapter provides information for communication adapters and communication adapter extensions supported on the SP. © Copyright IBM Corp.

  • Page 252: Pci Communication Adapters Supported On The Sp

    ESCON 2920 4/16 Auto Short 4.2.1* 2.4* streamer 2943 8-Port Short 4.2.1* 2.4* Async EIA 232/RS-422 2944 128 Port Short 4.2.1* 2.4* Async 2946 Turboways Short 4.2.1/4.3.2 2.4* 622Mbps PCI MMF RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 253 Feature code Adapter Description Memory Card AIX Level PSSP Available Address size supported level Space supported 2947 Long 4.2.1/4.3.2 2.4* ARTIC960H x 4-Port Selectable 2962 2-Port Short 4.2.1* 2.4* Multiprotoco l X.25 2963 TURBOWA Short 4.2.1* 2.4* YS 155 UTP 2968 10/100 Short...

  • Page 254: Token Ring (F/C 4959)

    2.4* Differential Ultra/Wide 6208 SCSI-2 Short 4.2.1* 2.4* Fast/Wide 6209 SCSI-2 Short 4.2.1* 2.4* Differential Fast/Wide 6215 SSA RAID 5 Long 4.2.1* 2.4* 6225 Advance Long 4.2.1/4.3.1/ 2.4* Serial RAID 4.3.3* RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 255: Descriptions Of Adapter Supported On The Sp

    159 and in Chapter 6, “SP Switch Routers 9077” on page 181. 8.2.1 IBM Network Terminal Accelerator adapter (F/C 2403) The IBM Network Terminal Accelerator adapter (F/C 2403) is an Ethernet adapter that accelerates network performance by off-loading the telnet and rlogin daemons, TCP/IP protocol stack, and virtual terminal I/O management from the RS/6000 system.

  • Page 256: Scsi-2 Differential Fast/Wide Adapter/A (F/C 2416)

    (62.3 feet). 8.2.4 IBM Short Wave Serial HIPPI PCI adapter (FC 2732) The IBM Short-Wave Serial HIPPI PCI adapter (F/C 2732) provides high-speed connectivity via the ANSI serial HIPPI channel. It provides (via short-wave optics) the capability to participate in supercomputer environments, attach to disk-array subsystems, other SP systems, HIPPI switches, other vendor computers, and tape subsystems.

  • Page 257: Long-Wave Serial Hippi Adapter (F/C 2733)

    Feature components This feature order provides the following: Adapter card Diagnostic wrap plugs Installation instructions Customer components You must supply the following components for this feature: Short-wave cabling Dual SC connector 50/125 micron multi-mode fiber 8.2.5 Long-Wave Serial HIPPI adapter (F/C 2733) The Long-Wave Serial HIPPI adapter (F/C 2733) provides high-speed connectivity via the ANSI serial HIPPI channel.

  • Page 258: Ibm Network Terminal Accelerator Adapter (F/C 2402 & 2403)

    8.2.6 IBM Network Terminal Accelerator adapter (F/C 2402 & 2403) The IBM Network Terminal Accelerator adapter (F/C 2402) is an Ethernet adapter that accelerates network performance by off-loading the telnet and rlogin daemons, TCP/IP protocol stack and virtual terminal I/O management from the RS/6000 system.
  • Page 259 Customer supplied components You must supply the following components for this feature: An FDDI concentrator, such as the IBM 8240 (or equivalent) concentrator, to connect to your FDDI local area network. Vice driver and FDDI common code (provided with adapter).

  • Page 260: S/390 Escon Channel Pci Adapter (F/C 2751)

    8.2.11 S/390 ESCON Channel PCI adapter (F/C 2751) The PCI S/390 ESCON Channel adapter (F/C 2751) provides the SP system with an attachment to IBM Enterprise Systems Connection (ESCON) channels on System/390 mainframes. This direct ESCON channel connection provides a fiber optic link that can take advantage of ESCON Directors (fiber optic switches) permitting multiple channel connections.

  • Page 261: Pci Auto Lanstreamer Token Ring Adapter (F/C 2920)

    8.2.13 EIA 232/RS-422 8-Port Asynchronous adapter (F/C 2943) The 8-port Async adapter (F/C 2943) provides the RS/6000 SP with up to eight EIA 232 or RS-422 asynchronous serial lines from a single PCI bus slot. This adapter adheres to the PCI Revision 2.1 standards for EIA 232 and RS-422. It features a low cost, high performance 32-bit card, 33 MHz bus speed, and a PCI bus transfer rate of 132 MBps.

  • Page 262: Wan Rs232 128-Port Adapter (F/C 2944)

    PCI bus slot. This adapter adheres to the PCI standard. It features a low cost, high performance 32-bit card, 33 MHz bus speed, and a PCI bus transfer rate of 132 MBps. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 263 RS-422 maximum distance 1200 meter dependent on baud rate Customer supplied components F/C 2944 utilizes the following optional RANs and device cables, which are available from IBM: 1.2 MBps RANs and cables: – F/C 8130: 1.2 MBps Remote Asynchronous Node, 16-port, EIA-232 (US) –...

  • Page 264: Turboways 622 Mbps Pci Mmf Atm Adapter (F/C 2646)

    8.2.15 Turboways 622 Mbps PCI MMF ATM adapter (F/C 2646) The IBM Turboways 622 Mbps PCI MMF ATM adapter (F/C 2646) is a 64-bit, Universal PCI Adapter. This adapter provides direct access to the ATM network at a dedicated 622 Mbps full-duplex connection. The Turboways 622 Mbps PCI...

  • Page 265: 2-Port Multiprotocol X.25 Adapter (F/C 2962)

    Featured components This feature order provides the following: One ARTIC960Hx adapter (F/C 2947) A connecting cable (required). The following are available from IBM: – F/C 2861: ARTIC960Hx 4-port EIA-232 cable – F/C 2862: ARTIC960Hx 4-port RS-449 cable – F/C 2863: ARTIC960Hx 4-port X.21 cable –...

  • Page 266: Atm 155 Turboways Utp Adapter (F/C 2963)

    Customer supplied components If you plan to operate this adapter using X.25 protocols, then you must separately order the IBM AIXLINK/X.25 LPP (5696-926). This package provides a V.24, V.35, or X.21 port connection to X.25 packet switched networks. The system interface is determined by the cable connected to this adapter. See Table 8-2 for a list of available cables and the interface supported by each cable.

  • Page 267: Ethernet 10/100 Mb Adapter (F/C 2968)

    If you plan to use multipoint connections, you must provide an ATM switch. 8.2.19 Ethernet 10/100 MB adapter (F/C 2968) The IBM 10/100 Ethernet TX PCI adapter (F/C 2968) is a 10/100 PCI Ethernet adapter that is compatible with IEEE 802.3 and 802.3u specifications. The adapter has one RJ-45 connection that supports connections to 100BaseTX and 10BaseT networks.

  • Page 268: Gigabit Ethernet - Sx Adapter (F/C 2969)

    Network equipment such as a switch or router is required to attach to 1000BASESX networks. All Ethernet cables.The maximum operating distances for the fiber cables are: – 260 meters with 62.5 micron multimode fiber – 440 meters with 50 micron multimode fiber RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 269: 10/100/1000 Base-T Ethernet Pci Adapter (Fc 2975)

    8.2.21 10/100/1000 Base-T Ethernet PCI adapter (FC 2975) 10/100/1000 Base-T Ethernet PCI adapter (F/C 2975) is a Full Duplex Gigabit Ethernet adapter designed with highly integrated components to optimize cost and performance. The adapter interfaces to the system via the PCI bus and connects to the network using a 4-pair CAT-5 Unshielded Twisted Pair (UTP) cable for distances of up to 100m.

  • Page 270: Atm 155 Mmf Adapter (F/C 2988)

    Provides signaling channel setup Provides virtual connection set up and tear down Supports point-to-point and point-to-multipoint switching Supports virtual circuits (maximum 1024) Supports classical IP and ATRP over ATM (RFC 1577) RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 271: Four-Port 10/100 Base-Tx Ethernet Pci Adapter (F/C 4951)

    An ATM switch 8.2.25 Four-Port 10/100 BASE-TX Ethernet PCI adapter (F/C 4951) The IBM Four-Port 10/100 BASE-TX Ethernet PCI adapter (F/C 4951) makes available four Ethernet ports using a single PCI slot. It is a 32/64-bit, long PCI adapter, supporting four industry-standard Ethernet 10BASE-T or 100BASE-TX interfaces supporting 10 or 100 Mbps data rates, either half or full duplex on each of four separate ports.

  • Page 272: Crytographic Coprocessor Pci Adapter (F/C 4958)

    32 KB of battery-backed RAM Supports standard interrupts, DMA, controls, real-time clock PCI 32-bit Bus Master Featured components This feature provides the following: Adapter card CD-ROM with adapter device driver Installation instructions RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 273: High-Speed Token-Ring Pci Adapter (F/C 4959)

    Limitations: The PCI Cryptographic Coprocessor Adapter is a field-installed only device in order to meet restrictive shipping requirements. 8.2.27 High-Speed Token-Ring PCI adapter (F/C 4959) The High-Speed Token-Ring PCI adapter (F/C 4959) is a PCI 16/4 Token Ring Adapter that is compatible with IEEE 802.5 specifications. The adapter has two external connections: RJ-45 to attach to UTP cabling and a 9-pin D-Shell to attach to STP cabling.

  • Page 274: Scsi-2 Ultra/Wide De Pci Adapter (F/C 6204)

    LVD devices or as an alternative, one channel can be used to support mixed performance single-ended devices. Industry standard VHDCI connectors are available for external connection to each channel. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 275: Scsi-2 Ultra/Wide Single Ended Pci Adapter (F/C 6206)

    Installation instructions Customer components You must supply the following component for this feature: Cabling (available through IBM) 8.2.30 SCSI-2 Ultra/Wide Single Ended PCI adapter (F/C 6206) The PCI SCSI-2 Ultra/Wide Single Ended adapter (F/C 6206) provides a single ended SCSI-2 Ultra/Wide interface that can burst data between devices on the SCSI bus at 40 MBps (twice the fast/wide rate) using block sizes greater than 64 K.

  • Page 276: Ultra Scsi Differential Adapter (F/C 6207)

    This adapter conforms to SCSI-2 standards and the Fast-20 (Ultra) documentation. Industry standard SCSI P (68-pin) connectors are incorporated on the adapter. Feature characteristics This feature has the following characteristics: 32-bit Bus Master adapter. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 277: Scsi-2 F/W Single-Ended Adapter (F/C 6208)

    68 pin SCSI-3 standard connector. Customer components You must supply the following component for this feature: Cabling (available through IBM) Adapter limitations: Data transfer rates with F/C 6207 are limited to the speed of the slowest device on the SCSI bus.

  • Page 278: Scsi-2 F/W Differential Adapter (F/C 6209)

    RAID 5 arrays from (2+P) up to (15+P). Up to 6 (15+P) or 32 (2+P) RAID 5 Array Groups per adapter. All members of a RAID 5 array must be on the same SSA loop. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 279: Ssa Fast-Write Cache Module (F/C 6222)

    The Advanced SerialRAID Adapter also supports connectivity to external disk enclosures and internal RS/6000 SSA configurations. Optional SSA Fiber-Optic Extender is also supported (Refer to IBM 7133 Sales literature for additional information). Any supported RS/6000 system can be set up to boot from an Advanced SerialRAID Adapter (F/C 6225), provided a non-RAID SSA disk is included as part of the configuration.

  • Page 280: Advanced Serialraid Plus Pci Adapter (F/C 6230)

    6225), which has been withdrawn from marketing, has a data transfer rate of up to 160 MBps per loop. This high-performance multi-initiator Serial Storage Architecture (SSA) adapter provides eight-initiator non-RAID capability, two-initiator RAID-5 capability, and one-initiator RAID-0 capability. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 281 SSA HA data protection, disk mirroring for the best performance, or multi-initiator RAID for the lowest total system cost. SSA EL adapter architecture enables RS/6000 SP PCI systems to share SSA storage in a multi-host SSA environment (cluster/SP).

  • Page 282: Artic960Rxd Quad Digital Trunk Adapter (F/C 6310)

    Feature components This feature order provides the following: One ARTIC960RxD adapter (F/C 6310). A connecting cable (required). The following are available from IBM: – F/C 2709: ARTIC960Hx 4-port T1 RJ45 cable – F/C 2710: ARTIC960Hx 4-port E1 RJ45 cable – F/C 2871: ARTIC960RxD Quad DTA, T1, 100 ohm, 3 m 4-port cable –...

  • Page 283: Artic960Rxf Digital Trunk Resource Adapter (F/C 6311)

    One ARTIC960RxF adapter (F/C 6311) ARTIC RTE CD kit Diagnostics disk A connecting cable (required); the following cables are available from IBM: – (F/C 2877): ARTIC960RxD Quad DTA, H.100, 4-drop cable – (F/C 4353): ARTIC960RxD Quad DTA, H.100, 8-drop cable...
  • Page 284 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 285: Chapter 9. Software Support

    This chapter describes the software available for SP systems. It covers the following software: Parallel System Support Programs (PSSP) General Parallel File System (GPFS) LoadLeveler Scientific Subroutine Library (ESSL, PESSL & OSL) Parallel Environment (PE) IBM High Availability Cluster Multi-Processing for AIX (HACMP) © Copyright IBM Corp.

  • Page 286: Parallel System Support Programs (Pssp)

    PSSP contains the System Data Repository (SDR) for storing management data that can be retrieved across the control workstation, file servers, and SP nodes. It includes File collections for managing duplicated files and directories on multiple nodes. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 287: Rs/6000 Cluster Technology (Rsct)

    PSSP comes with login control for blocking unauthorized user or group access to a specific SP node or a set of nodes. It offers consolidated accounting for centralizing records at the node level (for tracking use by wall clock time rather than processor time) and for gathering statistics on parallel jobs.

  • Page 288: System Monitoring And Control

    9.1.4 IBM Virtual Shared Disk (VSD) The IBM Virtual Shared Disk (IBM VSD) allows multiple nodes to access a disk as if the disk were attached locally to each node. This feature can enhance the performance of applications that provide concurrent control for data integrity, such as Oracle databases or the GPFS file system.

  • Page 289: Performance Toolbox Parallel Extensions For Aix (Ptpe)

    (PTX) or AIX by adding monitoring functions and statistics specific to the SP. PTPE provides SP performance statistics from software, such as LoadLeveler and IBM Virtual Shared Disk (VSD), and hardware components. These statistics are available in live (current) and archived (historical) forms for use in supporting both active monitoring and comparative analysis over time.

  • Page 290: Other Convenient Services

    PSSP supports a multi-threaded, standards-compliant Message Passing Interface (MPI) through an IBM Parallel Environment for AIX (PE) as well as maintaining its single-threaded MPI support. In addition, PSSP includes a Low-level Application Programming Interface (LAPI) with a flexible, active message style, communications programming model on the SP Switch.

  • Page 291: Software Requirements

    9.2 General Parallel File System (GPFS) Program number - 5765 -B95 IBM General Parallel File System (GPFS) is a standards-based, parallel file system that delivers high performance, high availability, and high scalability while preserving the application interfaces used in standard file systems. GPFS allows...

  • Page 292: Administration And Operation

    GPFS will transparently failover locked servers and other GPFS central services. Through its use of IBM RVSD, GPFS continues to operate in the event of disk connection failures. GPFS allows data replication to further reduce the chances of losing data if storage media fail.

  • Page 293: Migration

    Table 9-2 shows the software requirements for GPFS 1.4. Table 9-2 Software requirements for GPFS 1.4 AIX version PSSP version other software 4.3.3 or higher 3.2 or higher IBM Virtual Shared Disk image Recoverable Virtual Shared Disk image Chapter 9. Software support...

  • Page 294: Loadleveler

    9.3 LoadLeveler Program number - 5765-D61 IBM LoadLeveler is a job management system that allows users to run more jobs in less time by matching the jobs’ processing needs with the available resources. LoadLeveler schedules jobs, and provides functions for building, submitting, and processing jobs quickly and efficiently in a dynamic environment.

  • Page 295: Capabilities

    9.3.2 Capabilities Parallel processing For parallel jobs, LoadLeveler interfaces with parallel programming software to obtain the multiple SP nodes required for the job's parallel tasks. Individual control At the node level, users can specify to LoadLeveler when their processing nodes are available and how they are to be used. For example, some users might let their workstations accept any job during the night, but only certain jobs during the day when they most need their resources.

  • Page 296: New In Loadleveler 2.2

    LoadLeveler to support DCE security services: fileset ssp.clients 3.2 is required. 9.4 Mathematical Subroutine Libraries (PESSL) There are three types of Mathematical Subroutine Libraries; they include: Program number for ESSL - 5765-C42 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 297 Program number for PESSL - 5765-C41 Program number for OSLp - 5765-392 The Engineering and Scientific Subroutine Library (ESSL) family of products is a state-of-the-art collection of mathematical subroutines that provide a wide range of high-performance mathematical functions for many different scientific and engineering applications.

  • Page 298: Operations

    Examples of applications that use these types of mathematical subroutines are: Structural analysis Time series analysis Computational chemistry Computational techniques Fluid dynamics analysis Mathematical analysis Seismic analysis Dynamic systems simulation Reservoir modeling RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 299: New In Essl 3.2

    Nuclear engineering Quantitative analysis Electronic circuit design For more detailed information on ESSL, PESSL and OSLp refer to RS/6000 Scalable POWER parallel System: Scientific and Technical Computing Overview, SG24-4541-01. 9.4.2 New in ESSL 3.2 The ESSL Libraries are tuned for the RS/6000 POWER3-II processors. New enhancements have been made to ESSL's Dense Linear Algebraic Equations Subroutines, which now include: Symmetric Indefinite Matrix Factorization and Multiple Right-Hand Side Solve...

  • Page 300: New In Oslp 1.3

    IBM VisualAge C++Professional for AIX Version 5.0 C for AIX, Version 5.0.1 PL/1 Set for AIX, Version 1.1 or higher C libraries (Included in the AIX Version 4 Application Development Toolkit.) RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 301: Parallel Environment (Pe)

    9.5 Parallel Environment (PE) Program number 5765-D93 IBM Parallel Environment for AIX is a high-function development and execution environment for parallel applications using the SP system. It is a complete solution for enterprises that need to develop, debug, analyze, tune, and execute parallel programs on the AIX platform.

  • Page 302: Operation

    It supports Low-level Application Programming Interface (LAPI) programs. PE makes for easier parallel application development. PE gives enhanced XProfiler graphical performance tool. It enables easy application portability to networked RS/6000 or RS/6000 SP systems. 9.5.1 Operation PE application development solution consists of: 1.
  • Page 303 Support for MPI shared memory message passing on symmetric multiprocessor (SMP) nodes, for the Internet Protocol (IP) library and for the User Space (US) library. Instrumentation probes that the analysis tool defines and inserts into the target application to collect data. An asynchronous communication callback facility that connects the class library with the daemons.

  • Page 304: Software Requirements

    C for AIX, V4.4, or higher; or C for AIX V5.1, or higher. VisualAge C++ Professional for AIX V4.1 VisualAge C++ Professional for AIX V5.0, or higher. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 305: Ibm High Availability Cluster Multi-Processing For Aix (Hacmp)

    9.6 IBM High Availability Cluster Multi-Processing for AIX (HACMP) Program number - 5765-E54 HACMP aims to eliminate any single point of failure, be it disks, network adapters, power supply, processors, or other components. HACMP provides a highly-available environment by identifying the set of cluster resources that are essential to processing and defining the takeover relationships among the cluster nodes that access these resources.

  • Page 306: Features

    With HACMP, two to thirty two servers may be configured for high availability. HACMP offers built- in growth and helps protect investments in cluster hardware, software, services, and training. Easy upgrade/migration RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 307: New In Hacmp 4.4

    The cluster configuration can be expanded or modified without service disruption through Dynamic Reconfiguration options. Version Compatibility and Dynamic Reconfiguration eliminate the most common causes of scheduled cluster down time. Concurrent processing With the HACMP Concurrent Resource Manager (CRM) feature option, data can be processed concurrently when using Oracle Parallel Server (OPS) Version 7.3 or 8.

  • Page 308: Software Requirements

    4.3.3.10 or later 3.2 or later HAView facility requires the installation of Tivoli NetView for AIX Data Path Optimizer (DPO) Version 1.1 is only supported in the CRM and ESCRM RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 309: Chapter 10. Clustering Configurations

    Section 10.3, “PSSP environment for SP/CES systems” on page 337 shows the PSSP enhancements made to the SDR; commands and sample monitoring definitions are provided. Section 10.4, “Building SP/CES systems scenarios” on page 344 provides different scenarios for building your SP/CES system. © Copyright IBM Corp.

  • Page 310: Configurations Rules

    Non-switched configuration SP-Switch (eight ports) configuration SP Switch (16 ports) and SP Switch2 (16 ports) configuration SP Switch (16 ports) and SP Switch2 (16 ports) with Intermediate Switch Board (ISB) configuration RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 311: Placement Rules For Nodes, Frames, And Switches

    Any SP-attached server is considered a node (F/C 9122) and a frame (F/C 9123) in an SP system. Any Clustered Enterprise Server is considered a node and a frame in a CES system. Any SP-attached server and any Clustered Enterprise Server can be developed into non-switched and SP Switch (F/C 4011) configurations only.
  • Page 312 Thin node in the right side of the same drawer. Additionally, for Short frames, a single PCI thin node (if it exists) must be the last node in the frame. Figure 10-1 on page 289 shows examples on this node configuration rule. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 313 Invalid Configurations MCA thin nodes must be in the same type in a drawer. Nodes must be placed in sequence in short frames MCA Thin PCI Thin (Empty) (Empty) PCI Thin PCI Thin PCI Thin PCI Thin MCA Thin MCA Thin MCA Thin MCA Thin PCI Thin PCI Thin...
  • Page 314 W ide No de High No de (M CA) Wide Node Valid Configuration Thin Thin PCI Thin Thin Thin Wide Node High Node (MCA) Wide Node Figure 10-2 Example on frame configuration RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 315 Frame configuration rule 3: A Short frame supports one SP Switch-8 board only. The Short frame supports only the SP Switch-8 (F/C 4008) and only one Switch can be configured in an SP system composed of Short frames. This configuration mounts one to eight nodes through a single SP eight port Switch and the nodes can be hosted in a single Short frame, or with non-switched Short expansion frames after the first Short frame is fully populated.
  • Page 316 When a Tall frame is installed with an eight port Switch, a maximum of eight nodes and one frame is supported. No expansion frame can be added. No imbedded empty drawer is allowed. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 317 When using the SP Switch-8 (F/C 4008), the tall frame can only mount one to eight nodes. A minimum configuration is formed by one Tall model frame equipped with an SP Switch-8 and a single PCI thin node, one wide node, one MCA high node (a PCI high node is not supported), or one pair of MCA thin nodes.
  • Page 318 W ide N ode H igh N ode W ide N ode S P S wi tch Fram e 1 Fram e 2 Figure 10-5 Examples on single SP Switch-16 configurations RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 319 Wide Node High Node Wide Node PCI Thin Wide Node High Node Wide Node Wide Node Wide Node High Node Wide Node Wide Node Wide Node High Node High Node Wide Node SP Switch Frame 2 Frame 1 Frame 3 High Node High Node Wide Node...
  • Page 320 Figure 10-7 Example on multiple SP-Switch-16 configurations Frame configuration rule 8: For a more than five SP Switches configuration, an intermediate switch frame is required, which can expand up to 128 nodes. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 321 An SP intermediate switch frame is a base, tall frame equipped with four SP Switches but no processor nodes. The switch frame is required for systems using more than five SP Switches to interconnect all the switches in the system. A switch frame supports systems with 65 to 128 nodes.
  • Page 322 9076-555 Standalone Switch for Clustered Nodes with one up to two F/C 4011. Switch configuration rule 5: SP Switch2 is compatible with PCI high nodes only (requires PSSP Version 3.2). RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 323: Numbering Rules For Frames, Slots, Nodes, And Switch Ports

    SP Switch2 (F/C 4012) provides a higher throughput and performance than SP Switch (F/C 4011) and it supports PCI high nodes and model 550 Tall frames only. Table 10-1 shows the current compatibility of the switches and nodes; note that the switches in an SP/CES system cannot be mixed.
  • Page 324 SP/CES system. Each node in the system, are assigned a node number, regardless of node or frame type, by the following formula (where slot_number is the lowest slot number occupied by the node): node_number =((frame_number - 1) x 16) + slot_number RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 325 Each type (size) of node occupies one slot or a consecutive sequence of slots. For each node, there is an integer n such that a thin node (or Expansion I/O unit) occupies slot n, a wide node occupies slots n and n+1, and a high node occupies n, n+1, n+2 and n+3.
  • Page 326 These ranges are also in sequential order with their switch board number. For example, switch board 1 contains switch port numbers 0 through 15. Figure 10-10 on page 303 shows some possible examples of the switch port number assignment. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 327: Switch Port Numbering On An Sp Switch

    (1 ) S P S w itch F ra m e n (2 ) S P S w itc h F ra m e n + 1 F r a m e n SP Switch Frame n+1 Frame n Frame n+2 SP Switch Frame n+1 Frame n+3...

  • Page 328: Sp Universal Clustering Capabilities

    = ((frame_number -1) x 16) + slot_number - 1 10.2 SP universal clustering capabilities This section shows different SP universal clustering capabilities: SP system in non-switched and switched environments SP system with SP-attached servers in switched environment RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 329: Sp System In Non-Switched And Switched Environments

    CES system in non-switched and switched environments 10.2.1 SP system in non-switched and switched environments We may consider an SP system in non-switched or switched environments. SP system in non-switched environment Figure 10-12 on page 305 shows an example of a non-switched SP system. The SP system has 16 thin nodes in an SP tall frame and a 7044-170 RS/6000 system as CWS.
  • Page 330 1 2056 thin thin ---------------------------------------------------------------------- thin: 375MHz Thin Node (2056) Quantity placed: 16 Product Description 2909 9.1 GB Ultra-SCSI Disk Drive Pair 4098 Memory Expansion Feature 4110 256 MB (2x128MB) SDRAM DIMMs RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 331 5765-C34 AIX Version 4 4061 AIX 4.3, per Node 9001 Asset Registration 5765-D51 Parallel System Support Program V3.2 4011 SP Nodes 5812 RS/6000 SP Resource Center (CD-ROM) 5813 CD-ROM 9001 Asset Registration 5765-D61 LoadLeveler V2.2 2005 SP Nodes 5000 CD-ROM...
  • Page 332 *** 170 CWS Default Entry SOFTWARE *** Product Description 5765-C34 AIX Version 4 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 333: Switched Sp Environment

    0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0903 CWS Diagnostic Pre-Load 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM 5005 Preinstall 5924 AIX 4.3 PII - English 9001 Asset Registration SP system in switched environment Figure 10-13 shows an example of a switched SP system using the SP Switch.
  • Page 334 Below is a description of your proposed system. This includes the nodes used, placement, and their devices. ---------------------------------------------------------------------- Frame: 1 Feature: 9076-550 Drawer Left Right 8 2057 wide 7 2057 wide 6 2057 wide 5 2057 wide RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 335 4 2057 wide 3 2057 wide 2 2057 wide 1 2057 wide ---------------------------------------------------------------------- Frame: 2 Feature: Drawer Left Right 8 2057 wide 7 2057 wide 6 2057 wide 5 2057 wide 4 2057 wide 3 2057 wide 2 2057 wide 1 2057 wide ----------------------------------------------------------------------...
  • Page 336 0922 Performance Toolbox V3.0 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM 9001 Asset Registration F80 CWS Default Small *** F80 CWS Default Small HARDWARE *** Product Description 7025-F80 RS/6000 Enterprise Server Model RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 337 1.44MB 3.5-in Diskette Drive Integrated Ultra2 SCSI Adapter Integrated Ethernet Adapter 2624 32x Speed CD-ROM 2830 POWER GXT130P Graphics Adapter (PCI) 2943 8-Port Async Adapter, EIA-232/422 (PCI) 2968 10/100 Mbps Ethernet PCI Adapter 2985 Ethernet BNC/RJ-45 Adapter, PCI 3109 SCSI External Port to SCSI Internal 6 Pack Cable Assembly 3153 18.2 GB 10K RPM 1"...

  • Page 338: Sp System With Sp-Attached Servers

    Each SP node and each SP-attached server is connected to the SP Switch, which is hosted on the SP standard tall frame. For a Twisted Pair network, a hub must be provided for twisted pair connections. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 339: Switched Sp Environment With Sp-Attached Servers

    SP-attached servers SP frame M/T 7017 Enterprise Server M/T 7026 Enterprise Server with 8 wide nodes Supported Models Supported Models SP switch SP switch network RS232 Administrative ethernet 7025-F80 as Figure 10-14 Switched SP environment with SP-attached servers Example 10-3 shows a sample feature codes configuration for the switched SP environment with SP-attached servers considered.
  • Page 340 8 2057 wide 7 2057 wide 6 2057 wide 5 2057 wide 4 2057 wide 3 2057 wide 2 2057 wide 1 2057 wide ---------------------------------------------------------------------- Frame: 2 Feature: 9123 Drawer Left Right RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 341 2 9122 1 Node RS/6000 Server Attac ---------------------------------------------------------------------- Frame: 3 Feature: 9123 Drawer Left Right 2 9122 1 Node RS/6000 Server Attac ---------------------------------------------------------------------- Frame: 4 Feature: 9123 Drawer Left Right 2 9122 1 Node RS/6000 Server Attac ---------------------------------------------------------------------- Chapter 10. Clustering configurations...
  • Page 342 RS/6000 Server Attac ---------------------------------------------------------------------- RS/6000 Server Attached Node: RS/6000 Server Attached Node Quantity placed: 5 ---------------------------------------------------------------------- wide: 375MHz Wide Node (2057) Quantity placed: 8 Product Description 2909 9.1 GB Ultra-SCSI Disk Drive RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 343 5765-C34 AIX Version 4 4061 AIX 4.3, per Node 9001 Asset Registration 5765-D51 Parallel System Support Program V3.2 4011 SP Nodes 5812 RS/6000 SP Resource Center (CD-ROM) 5813 CD-ROM 9001 Asset Registration 5765-D61 LoadLeveler V2.2 2005 SP Nodes 5000 CD-ROM...
  • Page 344 7036 36U I/O Rack 8396 SP System Attachment Adapter 9171 Power Distribution Unit, Side-Mount , 1 Phase 9300 Language - English (US) 9800 Power Cord - US/Canada *** M/T 7017 SOFTWARE *** Product Description RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 345 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration 5765-D51 Parallel System Support Program V3.2 4012 External Attached Servers (Sxx models) 5812 RS/6000 SP Resource Center (CD-ROM) 5813 CD-ROM 9001 Asset Registration 5765-E68 Performance AIDE V3.0 0001 Installs 9001 Asset Registration...
  • Page 346 Product Description 5765-C34 AIX Version 4 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration 5765-D51 Parallel System Support Program V3.2 4014 H80 Servers 5812 RS/6000 SP Resource Center (CD-ROM) 5813 CD-ROM RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 347 9001 Asset Registration 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0917 Performance AIDE V3.0 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM 5005 Preinstall...

  • Page 348: Ces System In Non-Switched And Switched Environment

    We may consider a CES system in non-switched or switched environments. CES system in non-switched environment Figure 10-15 on page 325 shows an example of a CES system in non-switched environment (pure CES system). RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 349 It is composed of seven Enterprise Servers M/T 7017 and 12 Enterprise Servers M/T 7026. The CWS is an 7025-F80 RS/6000 system with one 128-port Asynchronous adapter and a chain of two necessary RANs. Each 7017 server has two serial line connection and each 7026 server has one serial connection to the CWS.
  • Page 350 Side-Mount , 1 Phase 9300 Language - English (US) 9800 Power Cord - US/Canada *** M/T 7017 SOFTWARE *** Product Description 5765-C34 AIX Version 4 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 351 5765-D51 Parallel System Support Program V3.2 4012 External Attached Servers (Sxx models) 5812 RS/6000 SP Resource Center (CD-ROM) 5813 CD-ROM 9001 Asset Registration 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non...
  • Page 352 5765-C34 AIX Version 4 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration 5765-D51 Parallel System Support Program V3.2 4013 M80 Servers 5812 RS/6000 SP Resource Center (CD-ROM) 5813 CD-ROM 9001 Asset Registration RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 353 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0917 Performance AIDE V3.0 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM 5005 Preinstall 5924 AIX 4.3 PII - English 9001 Asset Registration...
  • Page 354 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0903 CWS Diagnostic Pre-Load 0917 Performance AIDE V3.0 0922 Performance Toolbox V3.0 1004 CD-ROM Process Charge 2924 English RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 355: Ces System In Switched Environment

    3410 CD-ROM 5005 Preinstall 5924 AIX 4.3 PII - English 9001 Asset Registration CES system in switched environment Figure 10-16 on page 331 shows an example of a CES system in switched environment. It consists of two Enterprise Servers M/T 7017 and three Enterprise Servers M/T 7026.
  • Page 356 7036 36U I/O Rack 8396 SP System Attachment Adapter 9171 Power Distribution Unit, Side-Mount , 1 Phase 9300 Language - English (US) 9800 Power Cord - US/Canada *** M/T 7017 SOFTWARE *** RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 357 5005 Preinstall 9001 Asset Registration 5765-D51 Parallel System Support Program V3.2 4012 External Attached Servers (Sxx models) 5812 RS/6000 SP Resource Center (CD-ROM) 5813 CD-ROM 9001 Asset Registration 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD 0857 AIX 4.3...
  • Page 358 5765-C34 AIX Version 4 4061 AIX 4.3 5005 Preinstall 9001 Asset Registration 5765-D51 Parallel System Support Program V3.2 4014 H80 Servers 5812 RS/6000 SP Resource Center (CD-ROM) 5813 CD-ROM 9001 Asset Registration RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 359 5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0917 Performance AIDE V3.0 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM 5005 Preinstall 5924 AIX 4.3 PII - English 9001 Asset Registration...
  • Page 360 0016 Base License 3483 Single Pallet Delivery 5809 CD-ROM 9001 Asset Registration 5765-E68 Performance AIDE V3.0 0001 Installs 9001 Asset Registration 5765-E74 Performance Toolbox V3.0 0001 Server Installs 9001 Asset Registration RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 361: Pssp Environment For Sp/Ces Systems

    5692-AIX System Software 0598 AIX Welcome Center 0838 AIX 4.3 Update CD 0857 AIX 4.3 0859 Preinstall AIX 4.3 Server Bundle 0860 AIX 4.3 Bonus Pack - Non Export. 0903 CWS Diagnostic Pre-Load 0917 Performance AIDE V3.0 0922 Performance Toolbox V3.0 1004 CD-ROM Process Charge 2924 English 3410 CD-ROM...
  • Page 362 For details on these and other commands, see IBM Parallel System Support Programs for AIX: Command and Technical Reference, SA22-7351. The spframe command The spframe command is used to enter configuration data for a frame or a series of frames and, optionally, to set up the initial System Data Repository (SDR).
  • Page 363 FLAGS -r no | yes Indicates wether you want to initialize the SDR. If this is the last or only time you are invoking this command during installation, specify -r yes. If -r yes is specified, the /spdata/sys1/spmon/hmacls file has the default entries created.
  • Page 364 The VSD Perspective The System Partitioning Aid Perspective The Performance Monitor Perspective SP Perspectives are enabled to perform system management tasks for your CES through direct manipulation of system objects represented by icons. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 365 You simply select one or more Enterprise Server system objects (managed CES system resources, such as a frames or nodes) by clicking on them with a mouse and then selecting an action to perform on those system objects from the menu or tool bar.
  • Page 366 One of the file systems is over 90 percent full. frameControllerNotRespo The frame controller is not responding. nding framePowerOff The power to the frame has been turned off. The node is not responding hostResponds RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 367 EVENT DEFINITION The key mode switch on a node was taken out of keyNotNormal the Normal position nodeEnvProblem The environment indicator LED on the node is illuminated. A hardware problem has been detected. nodeNotReachable Group Services has found no way to communicate with the node.

  • Page 368: Building Sp/Ces Systems Scenarios

    SP system with SP-attached servers. You want use all of your SP frames for other SP systems. Therefore, you remove all of your SP frames from your SP system and manage it as a CES system. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 369: Standard Sp System

    If you want to develop an SP system using the standard thin, wide and/or high SP nodes, use the steps described in Chapter 2, “Installing and Configuring a New RS/6000 SP System” in IBM Parallel System Support Programs for AIX: Installation and Migration Guide, GA22-7347 to install this type of system.
  • Page 370 SP-attached servers from scratch. Your Enterprise Servers are either newly obtained or you do not care about the software previously installed on them. Use the steps described in Chapter 2, “Installing and Configuring a New RS/6000 SP System” in IBM Parallel System Support Programs for AIX: Installation and Migration Guide, GA22-7347 to install this type of system.
  • Page 371 In order to initiate “Step 29: Enter Frame Information and Reinitialize the SDR,” described in Chapter 2, “Installing and Configuring a New RS/6000 SP System” of the IBM Parallel System Support Programs for AIX: Installation and Migration...
  • Page 372 /spdata/sys1/install/pssplpp directory on the boot/install server from the Enterprise Server. For example, issue: umount /mnt – Step 18.9: Run pssp_script: Run the by issuing: pssp_script /usr/lpp/ssp/install/bin/pssp_script – Step 18.10: Reboot: Perform a reboot. For example: shutdown -Fr RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 373: Ces System

    1, respectively SAMI port: /dev/tty2 and S1 port: /dev/tty3 on frame 2). In order to initate “Step 29: Enter Frame Information and Reinitialize the SDR” described in Chapter 2, “Installing and Configuring a New RS/6000 SP System” of the IBM Parallel System Support Programs for AIX: Installation and Migration...

  • Page 374: Migration An Existing Sp System To A Ces System

    Performing the steps described in Chapter 6, “Reconfiguring the RS/6000 SP system” in IBM Parallel System Support Programs for AIX: Installation and Migration Guide, GA22-7347, you can migrate your Enterprise Servers to a CES system without reinstalling their software.

  • Page 375: Scaling Up An Existing Ces System To An Sp System

    An important point to know here is that you have to remove all the SP frames at the same time. You are not allowed to remove them one by one. To remove all the SP frames from your SP system at the same time, use the spdelfram command with the flag as follows:...
  • Page 376 Table 10-5. Note that the IP addresses assigned won’t match those given in Table 10-5. Table 10-5 Node number and IP address Node number IP address Netmask 129.33.34.11 255.255.255.0 129.33.34.12 255.255.255.0 33 (high node) 129.33.34.13 255.255.255.0 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 377 In this case, the lowest IP address is node 1. However, you cannot assign IP addresses to SP Switch adapters as follows: spadaptrs 3 1 3 css0 129.33.34.13 255.255.255.0 Instead, use the IP address of the node with the lowest node number as follows: spadaptrs 1 1 3 css0 129.33.34.11 255.255.255.0 The potential problem of changing from a CES system to an SP-attached system is that the frame numbers configured in the CES system might duplicate those...
  • Page 378 In this case, it may be better to skip the frame number 2 also when assigning frame numbers to the CES from the beginning. Sometimes, if SP nodes are installed in the SP frame, the starting frame number for CES may be larger than 2. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 379 2. If the CES system has less than 32 enterprise servers and add SP frames, leave space for flexible SP node addition. For example when a CES with 15 enterprise servers is redefined to be SP-attached servers in the SP system, the servers occupy at least 15 switch port numbers.
  • Page 380 Another expansion plan could be considered when you plan to upscale from a CES system to a switched SP system Both expansion plans take in consideration numbers such as: Size of CES Frame/Node Type Frame Number Node Number RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 381: Pseries 680 Cluster Solutions

    AIX will be pre-loaded on each server in the cluster ready solution at the factory. IBM PSSP for AIX version 3.2 with APAR IY16350 or later is required for each server in the cluster. PSSP will be shipped separately and installed by the customer.
  • Page 382 – One - Clustered Server Serial Port to CWS cable (F/C 3150) – One - Clustered Server Control Panel to CWS cable (F/C 3151) – One - IBM 10/100 Mbps Ethernet PCI Adapter (F/C 2968) – One - PSSP Version 3.2 or later license –...

  • Page 383: Pseries 680 Advanced Cluster Server - S85/M80

    – One - Clustered Server Serial Port to CWS cable (F/C 3150) – One - Clustered Server Control Panel to CWS cable (F/C 3151) – One - IBM 10/100 Mbps Ethernet PCI Adapter (F/C 2968) – One - PSSP Version 3.2 or later license –...

  • Page 384: Pseries 680 Advanced Cluster Server

    – One - Clustered Server Control Panel to CWS cable (F/C 3151) – One - Cable and PCI Card for SP CWS Attachment (F/C 3154) – One - IBM 10/100 Mbps Ethernet PCI Adapter (F/C 2968) – One - PSSP Version 3.2 or later license –...

  • Page 385: Appendix A. Naming Convention

    Naming convention Appendix A. The naming convention appendix provides naming conventions used in the contents of the redbook. This appendix is formatted in the following manner: Overview – Naming convention © Copyright IBM Corp.

  • Page 386: Definitions Overview And Naming Convention

    Definitions overview and naming convention IBM announced changes in product names with the arrival of the new IBM ^ pSeries Servers. It is important to bear in mind that this redbook uses the new naming convention. Naming convention Table A-1 shows the different combinations of SP/CES names.
  • Page 387 Node name Machine type Nickname used (M/T) here IBM RS/6000 7017 Model S70 7017-S70 Appendix A. Naming convention...
  • Page 388 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 389: Appendix B. Hardware

    3. Legacy SP Node 4. Legacy Models by Frame and Switch Types 5. Legacy Model by Frame and Nodes Types 6. SP LAN Topologies 7. Boot/install Server requirements 8. Disk space required for PSSP file sets © Copyright IBM Corp.

  • Page 390: Sp Communication Adapter Requirements

    I2 and I3 (I/O side). The CPU side is on the left and the I/O side is on the right, seen as you face the side of the node having the slot labels. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 391: Pci Bus Description For Smp Nodes

    Descriptions of PCI bus group numbering, bus type, slot width and slot speed for POWER3 and 332 MHz SMP nodes are shown in Table B-1. Table B-1 PCI bus description for SMP nodes Node type Bus type Slot Slot Slot group numbers width...

  • Page 392: Maximum Quantities Of Pci Adapters For Smp Thin And Wide Nodes

    ARTIC960Hx 4-port selectable 2962 2-port Multiprotocol Adapter 2963 ATM Turboways 155 2968 Ethernet 10/100 Mb 2969 Gigabit Ethernet-SX 2985 Ethernet 10 Mb BNC 2987 Ethernet 10 Mb AUI 2988 ATM 155 MMF RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 393 PCI Adapter Name Number of Max. Number Notes PCI slots Allowed Per Required Node Wide/Thin 4951 Four-port 10/100 4958 Crypto 4959 Token Ring 6204 Ultra SCSI DE 6205 Dual Channel Ultra3 SCSI 6206 Ultra SCSI SE 6207 Ultra SCSI DE 6208 SCSI-2 F/W SE 6209...

  • Page 394: Pci Adapters For High Nodes And Sp Expansion I/O Units

    Token Ring Auto Lanstreamer 2943 RS-422 8-port Async 2944 WAN RS-232 128-port 2947 ARTIC960Hx 4-port selectable 2962 2-port Multiprotocol Adapter 2963 ATM Turboways 155 2968 Ethernet 10/100 Mb 2969 Gigabit Ethernet-SX RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 395 PCI Adapter Name Max. Max. Combi Combi Quantity Quantity per High per SP ation ation node Expansi (Note (Note on I/O Unit 2985 Ethernet 10 Mb BNC 2987 Ethernet 10 Mb AUI 2988 ATM 155 MMF 4951 Four-port 10/100 LAN 4958 Crypto 4959...
  • Page 396 F/C 2732, 2733, 2969, 6205, 6225, 6227, 6228, 6230 cannot occupy the same PCI bus. The combined quantity of F/C 6225 and 6230 cannot exceed two per PCI bus. F/C 6205 cannot be placed in slot I5 to I8. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 397 If two F/C 2732 are installed, one must be placed in slot I2 or I3 (CPU side) and the other must be placed in slot I1 to I4 (I/O side). If two F/C 2733 are installed, one must be placed in slot I2 or I3 (CPU side) and the other must be placed in slot I1 to I4 (I/O side).
  • Page 398 If two F/C 2733 are installed, one must be placed in slot I2 or I3 (CPU side) and the other must be placed in slot I1 to I4 (I/O side). F/C 4958 cannot be placed in slot 1 or 2 (CPU side). RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 399 332 MHz Wide Node: No more than four (in any combination) of F/C 2741, 2742, 2743, 2963, 2968, 2988 can be installed. If one F/C 2969 is installed, no more than two (in any combination) of F/C 2741, 2742, 2743, 2963, 2968, 2988 can be installed. If two F/C 2969 are installed, none of F/C 2741, 2742, 2743, 2963, 2968, 2988 can be installed.

  • Page 400: Pci Adapters Weighting Factors

    FDDI SK-NET LP Comm 2742 FDDI SK-NET LP Comm 2743 FDDI SK-NET UP Comm 2751 ESCON CU Comm Medium Emulation 2920 Token Ring Auto Comm Lanstreamer 2943 RS-422 8-port Comm Async RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 401 Description Type Weighting Notes Factor 2944 WAN RS-232 Comm 128-port 2947 ARTIC960Hx 4-port Comm selectable 2962 2-port Multiprotocol Comm Adapter 2963 ATM Turboways Comm Medium 155 UTP 2968 Ethernet 10/100 Mb Comm Low (10) Medium (100)* 2969 Gigabit Comm High Ethernet-SX 2985 Ethernet 10 Mb...
  • Page 402 6. Medium* for RAID single loop operation, High for non-RAID 7. Low for no ports at 100 Mb, Medium with 1 or 2 ports at 100 Mb, High for 3 or 4 ports at 100 Mb RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 403 Following are lists of weighting factor guidelines and suggested quantities of adapters (by node type, PCI bus, and weighting factor) to help you achieve optimum operation on PCI buses: Adapter guidelines for using weighting factors 1. Distribute adapters across the PCI buses to equalize the weighting factor on each bus (application usage might require a different distribution).
  • Page 404 1 Medium (place on Bus 2) and 4 Low (3 should be on Bus 2 with precedence given to comm-type adapters, and 1 on Bus 3) 6 Low (4 on Bus 2, with precedence given to comm-type adapters, and 2 should be on Bus 3) RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 405: Mca Bus Adapter Requirements

    Allowed Per Required Node Wide/Thin/High Optics Daughter Card 2/1/NA Fibre Channel 1 GB 2/2/NA Fibre Channel 266 2/2/NA IBM Network Terminal 7/4/4 Accelerator 256 IBM Network Terminal 7/4/4 Accelerator 2048 SCSI-2 External I/O Controller 7/4/NA Enhanced SCSI-2 Differential 7/4/14 Fast/Wide Adapter/A...
  • Page 406 ATM TURBOWAYS 155 4/2/4 Ethernet/FDX 10 Mbps TP/AUI 7/3/12 MC Adapter Ethernet/FDX 10 Mbps BNC MC 7/3/12 Adapter 10/100 Ethernet Twisted Pair MC 2/2/4 Adapter Ethernet 10BASE-T Transceiver 8/4/15 9333 HPSA 6/2/8 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 407 MCA Adapter Name Number of Max. Number Notes MCA slots Allowed Per Required Node Wide/Thin/High SSA 4-Port Adapter 4/2/8 SSA 4-Port Adapter 4/2/8 SSA 4-Port RAID Adapter 4/2/8 Micro Channel SSA 4/2/8 Multi-Initiator/RAID EL Adapter SSA Fast-Write Cache Option Mounts on 6219 Digital Trunk Dual 6/3/2 Realtime Interface Co-Processor...

  • Page 408: Evaluating Sp System Power Requirement

    8. Will not operate in SSA loops containing F/C 6214 or F/C 6216. Withdrawn from production. Evaluating SP System power requirement The total power requirement of your RS/6000 SP system is the sum of the power required by all of the following individual components: Each base frame with integral power subsystem.

  • Page 409: Sp System Frame Power Computation Worksheet

    Make one copy of Table B-6 for each frame in your SP system. Fill in the worksheet copies using your quantities of all applicable features. Complete the calculations, using the power factors given for each feature, to determine the total power requirement of each frame. After calculating the power requirements of each frame, you can use the results to calculate the thermal output of your system.
  • Page 410 9.1 GB Ultra SCSI 10K RPM disk pair (F/C (___) x 41.7 3804) 18.2 GB Ultra SCSI 10K RPM disk pair (F/C (___) x 51.3 3810) 36.4 GB Ultra SCSI 10K RPM disk pair (F/C (___) x 74.1 3820) RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 411 Component Feature Qty x Watts Factor Serial HIPPI SW (F/C 2732) (___) x 39.7 adapters Serial HIPPI LW (F/C 2733) (___) x 39.7 FDDI SK-NET LP SAS (F/C 2741) (___) x 9.9 FDDI SK-NET LP DAS (F/C 2742) (___) x 15.2 FDDI SK-NET UP DAS (F/C 2743) (___) x 9.9 ESCON Channel (F/C 2751)
  • Page 412 3. The PDU power supply does not have a power factor of 1, thus: 4. For frames with a PDU, multiply W x 1.16 for VA 5. For a PDU and High-voltage transformer, multiply W x 1.19 for VA RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 413: Legacy Sp Nodes

    Legacy SP Nodes Table B-7 shows Legacy SP Nodes. Table B-7 Legacy SP nodes Feature Name clock Architecture Code 2001 THIN 62Mhz Power1 2002 THIN 66Mhz Power2 2003 WIDE 66Mhz Power2 2004 THIN 66Mhz Power2 2005 WIDE 77Mhz Power2 2006 HIGH 112Mhz PowerPC604...

  • Page 414: Sp Legacy Models By Frame And Switch Types

    Tall - 200.7 cm (79 in) SPS - SP Switch SPS-8 - SP Switch 8-port model Switch Frame - Not available for < 64 nodes – Optional for 65-80 nodes – Required for > 81 nodes RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 415: Sp Legacy Models By Frame And Nodes Types

    SP Legacy Models by Frame and Nodes Types Table B-9 shows the third digits defining the Frame Models in Legacy frames by Node Types. Table B-9 Legacy models by frames and node types (table 2 of 2) Model Form Processor I/O Bus Available Type...

  • Page 416: Sp Lan Topologies

    Note: IBM supplies an Ethernet LAN and one 15 meter (50 foot) Ethernet cable for each RS/6000 SP frame and each SP-attached server, as well as the required node-to-node cables. The Ethernet cable provided are BNC-type cables. IBM does not supply twisted-pair Ethernet cables or hubs. The customer must provide all other cables, network hubs, LAN switches, and routers.

  • Page 417: Shared Or Switched 100Base-Tx Network

    However, switches with the required switching capabilities are available. An illustration of a 100BASE-TX network is shown in Figure B-1. 100 Mbps Ethernet switch 100 Mbps Figure B-1 Shared or switched 100BASE-TX network Heterogeneous 10/100 Mbps network SP systems can be upgraded with new nodes having fast Ethernet connections, while older or more lightly loaded nodes can continue to run with 10 Mbps SP LAN connections.

  • Page 418: Heterogeneous 10/100 Mbps Network

    Shared 10BASE2 network In small SP systems such as a single frame, the control workstation and the nodes typically share a single thin-wire Ethernet as illustrated in Figure B-3 on page 395. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 419: Shared 10Base2 Network

    Ethernet / 1 10 Mbps Figure B-3 Shared 10BASE2 network This network has the following properties: No routing is required since the CWS and all nodes share one subnet. The entire SP LAN is a single broadcast domain as well as a single collision domain.

  • Page 420: Segmented 10Base2 Network

    Ethernet networks is Ethernet switching. Even with an added router, this setup is usually preferable to a segmented network with boot/install servers, from the viewpoints of performance, management, and complexity. This network is illustrated in Figure B-5 on page 397. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 421: Switched 10Base2 Network

    BNC to UTP 10 Mbps Coll.Domain/2 Coll.Domain/1 media converters Ethernet switch 100 Mbps Figure B-5 Switched 10BASE2 network This network setup has the following properties: No routing is required. All Ethernet segments are transparently combined into one large LAN by the Ethernet switch. Node-to-node connections within a single Ethernet segment still have to share that 10BASE2 medium in half-duplex mode.

  • Page 422: Sp Boot/Install Server Requirements

    When installing the nodes, you should limit yourself to installing eight nodes at a time because this is the limit of acceptable throughput on the Ethernet. Figure B-6 on page 399 shows an Ethernet topology for a single-frame system. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 423: Ethernet Topology With One Adapter For A Single-Frame Sp System

    Eth erne t 1 C W S Figure B-6 Ethernet topology with one adapter for a single-frame SP system An alternate way to configure your system is to install a second Ethernet adapter in your control workstation, if you have an available I/O slot, and use two Ethernet segments to the SP nodes.

  • Page 424: Method 1 - Ethernet Topology For Multiple-Frame Sp System

    Running two installs will install up to 64 nodes. Figure B-7 shows an Ethernet topology for this multi-frame system. Ethernet 1 Ethernet 2 Ethernet 3 Ethernet 4 Figure B-7 Method 1 - Ethernet topology for multiple-frame SP system RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 425: Method 2 - Ethernet Topology For Multi-Frame Sp System

    You will have to set up your SP Ethernet routing so nodes on one Ethernet can communicate with nodes on another. You also need to set up your network mask so that each SP Ethernet is its own subnet within a larger network address.

  • Page 426: Method 3 - Ethernet Topology For Multi-Frame Sp System

    SP Ethernet from the control workstation is connected to up to eight network install nodes in a frame. These network install nodes are, in turn, connected to additional frames. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 427 Ethernet 4 Ethernet 3 Ethernet 2 Ethernet 1 Figure B-10 Boot server frame approach The advantage of this is that when you add an additional frame to your SP configuration, all you need to do is connect the new frame to one of the network install nodes, and reconfigure the system.

  • Page 428: Disk Space Required For Pssp File Sets

    533KB rsct.clients.sp 27KB PSSP image:rsct.core 900KB rsct.core.utils 1.4MB PSSP image: ssp 150MB ssp.authent 660KB ssp.basic 6.8MB ssp.cediag 433KB ssp.clients 10.5MB ssp.css 79.2MB ssp.docs 30MB ssp.gui 31.2MB ssp.ha_topsvcs.compat ssp.jm 550KB ssp.perlpkg 1.6MB RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 429 Image or Fileset Name Image Size File Size ssp.pman 1.2MB ssp.public 12MB ssp.spmgr 634KB ssp.st 765KB ssp.sysctl 1.5MB ssp.sysman 1.3MB ssp.tecad 203KB ssp.tguides 1.1MB ssp.top 1.4MB ssp.top.gui 1.2MB ssp.ucode 907KB PSSP image: ssp.hacws 160KB ssp.hacws 141KB PSSP image: ptpe ptpe.docs 3.1MB ptpe.program 6.2MB...
  • Page 430 Image or Fileset Name Image Size File Size PSSP image ssp.resctr for the Resource Center ssp.resctr.rte Note: The total storage can cross multiple file systems. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 431: Appendix C. Sp System Upgrades

    MCA-type nodes to PCI nodes. If you are planning an upgrade for your SP system, you might need the following identifying information for the ordering process: Machine Type (M/T 9076) Model number SP System serial number © Copyright IBM Corp.

  • Page 432: Adding Power3, And 332 Mhz Smp Thin And Wide Nodes

    You can add a tall expansion frame (F/C 1550) to a Model 550, 20X, 30X, or 40X system allowing additional nodes to be installed. You can also add a short expansion frame (F/C 1500) to a Model 500 (eight nodes maximum) system. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 433: Adding Switches

    Adding switches SP switches can be added in the following configurations: 1. Switchless to SP Switch (F/C 4011), SP Switch-8 (F/C 4008), or SP Switch2 (F/C 4012) 2. SP Switch (F/C 4011) to SP Switch2 (F/C 4012) 3. High Performance Switch (F/C 4010) to SP Switch (F/C 4011) 4.

  • Page 434: Upgrading Smp-Type Nodes

    Replacing withdrawn processor nodes Replacement features are available to replace withdrawn early processor nodes with 332 MHz, POWER3, and 375 MHz POWER3 SMP Thin, Wide, and High nodes. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 435: Relocating Processor Nodes

    Relocating processor nodes If you plan to relocate any nodes or switches to another frame, you must order one of the following no-charge features: Relocating a Thin node (F/C 9250) Relocating a Wide node (F/C 9251) Relocating a High node (F/C 9252) Relocating an SP Switch (F/C 9253) If you are relocating thin or wide nodes to a frame that does not have the 48 V dc power and Ethernet cables already installed, you can order the following features...
  • Page 436 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 437: Appendix D. Software Appendix

    3. Virtual shared disk and Recoverable virtual shared disk (VSD and (RVSD) 4. General Parallel File System for AIX (GPFS) 5. Parallel Environment (PE) 6. Engineering and Scientific Subroutine Library (ESSL) and Parallel ESSL (PESSL) 7. LoadLeveler © Copyright IBM Corp.

  • Page 438: Pssp And Aix

    RVSD 2.1.1 HACMP/ES and HACMP 4.2 Note: Before PSSP 3.1, IBM Recoverable Virtual Shared Disk was a separate LPP. The High Availability Control WorkStation and the Performance Toolbox Parallel Extensions components were priced features. They are now optional components which you receive with PSSP.

  • Page 439: Migration

    Table D-2 shows AIX and PSSP support for different node types: Table D-2 AIX and PSSP support for different node types AIX/PSSP POWER3 POWER3 POWER3 Thin Wide High Thin Wide High Thin Wide AIX 4.2.1 PSSP 2.4 AIX 4.3.1 PSSP 2.4 AIX 4.3.2 PSSP 3.1 AIX 4.3.3...

  • Page 440: Coexistence

    Table D-4 Levels of PSSP and AIX supported in a mixed system partition Product AIX 4.1.5 AIX 4.2.1 AIX 4.3.3 PSSP 3.2 PSSP 3.1.1 PSSP 2.4 PSSP 2.3 PSSP 2.2 S = Supported N = Not supported RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 441: High Availability Cluster Multi-Processing (Hacmp)

    High Availability Cluster Multi-Processing (HACMP) This section discussed HACMP supported levels, migration paths and coexistence options. Supported levels While PSSP has no direct requirement for HACMP, if you install hardware that requires PSSP 3.2, regardless of system partitioning, and you already use or are planning to use HACMP, you must also install and use HACMP 4.3.

  • Page 442: Migration Considerations

    HACMP 4.2 or HACMP 4.1, this function is intended as a migration aid only. Once the migration is completed, each processor in the HACMP cluster must be at the same AIX and HACMP release levels, including all PTFs. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 443: Vsd And Rvsd

    VSD and RVSD In order to exploit the new functions available in the Recoverable Virtual Shared Disk (RVSD) option of PSSP 3.2, all nodes in the system must be running PSSP 3.2 with the RVSD component. RVSD requires the Virtual Shared Disk (VSD) optional components to be running also.

  • Page 444: Migration Considerations

    1. When migrating from IBM Recoverable Virtual Shared Disk 1.2 and PSSP 2.2 you can migrate one node at a time but you must migrate both IBM Recoverable Virtual Shared Disk and PSSP at the same time. 2. You can migrate later levels of PSSP one node at a time without having to migrate IBM Recoverable Virtual Shared Disk at the same time.

  • Page 445: General Parallel File System For Aix (Gpfs)

    General Parallel File System for AIX (GPFS) A file system managed by the GPFS LPP can only be accessed from within the system partition. GPFS 1.3 changes the locking semantics which control access to data and as a result requires that all nodes be at the same level. The task of migrating one node at a time is not supported.

  • Page 446: Parallel Environment (Pe)

    2. Message Passing Libraries (MPI, MPL) 3. Parallel Utilities which facilitate file manipulations (MPI sample programs) IBM Parallel Environment for AIX, is not supported in a mixed system partition. See Table D-8 for AIX and PSSP support levels. Table D-8 Supported PE LPP levels PSSP 2.4...

  • Page 447: Essl And Parallel Essl

    From AIX 4.3.2, PSSP 2.4, PE 2.3 + PTF AIX 4.3.2, PSSP 3.1.1, PE 2.4 AIX 4.3.2, PSSP 2.4, PE 2.3 + PTF AIX 4.3.3, PSSP 3.2, PE 2.4 AIX 4.3.2, PSSP 3.1.1, PE 2.4 AIX 4.3.3, PSSP 3.2, PE 3.1 ESSL and Parallel ESSL Engineering and Scientific Subroutine Library (ESSL) is not directly dependent on a level of PSSP.

  • Page 448: Migration Considerations

    LoadLeveler 2.2 with Parallel Environment 3.1 LoadLeveler 2.2 with Parallel Environment 2.4 LoadLeveler 2.1 with Parallel Environment 2.4 LoadLeveler 1.3 with Parallel Environment 2.3 LoadLeveler 1.3 with Parallel Environment 2.2 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 449: Appendix E. Sp/Ces Tools

    The tools are: Network Installation Manager (for installation and migration) Service Director Electronic Service Agent (to alert user and IBM about a problem) Inventory Scout (to check if the last available microcode is needed and gather the VPD) © Copyright IBM Corp.

  • Page 450: Network Installation Manager

    SP environment. Differences with SP/CES versus R/S6000 NIM The main differences to be considered are: 1. setup_server 2. SDR 3. Installation overview 4. Boot/install server requirements 5. Netbooting 6. Node customization RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 451 7. Wrappers setup_server setup_server is a crucial function since it is called on every node and BIS (Boot/Install Server) on bootup and is central to configuring and maintaining the SP NIM environment. For more detailed information on boot/install servers, refer to PSSP: Administration Guide, SA22-7348.
  • Page 452 NIM resources; for detailed information on the complete installation process, refer to the following guides: • IBM RS/6000 SP: Planning Volume 2, Control Workstation and Software Environment, GA22-7281 • PSSP: Installation and Migration Guide, GA22-7347 The first part of the installation procedure deals with setting up the CWS and making sure it has the necessary prerequisites to actually act in this capacity.
  • Page 453 At this step the installation of the CWS is complete, we can go ahead and enter our configuration details for the frame(s) and nodes into the SDR. – Enter Site Environment Details – Enter Frame Information and Reinitialize the SDR. –...
  • Page 454 In an SP environment, we never configure NIM manually. Instead, the PSSP software package has scripts called wrappers for configuring the NIM environment. Wrappers are functions written in Perl and each one is an independent script. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 455 These wrappers are called by the setup_server command for configuring the NIM environment in the CWS and Boot/Install servers (BIS). These scripts can also be called from the command line if you know the sequence in which they have to be executed. These scripts use the information from the SDR to initiate the appropriate NIM commands with the required options.The NIM wrappers are part of the ssp.basic filesets.

  • Page 456: Service Director

    Service Director for RS/6000 is offered as a part of the IBM Warranty or IBM Maintenance Service package for no additional charge. Due to entitlement checking at the IBM server, machines not on IBM Warranty or Service cannot be entitled to this service.
  • Page 457 IBM support center for remote analysis and action. If necessary, an IBM Service Representative is dispatched to the customer site with the parts needed to correct the problem reported.

  • Page 458: Electronic Service Agent

    It allows IBM Service to respond faster and with the correct repair parts. In some cases, IBM may notify a customer of an error and provide a resolution prior to an outage.

  • Page 459: What Does Service Agent Do

    During the registration process, an electronic key is created that becomes part of your resident. Service Agent program. This key is used each time Service Agent places a call for service. The IBM Service Agent Server checks the current customer service status from the IBM entitlement database; if this reveals that you are not on Warranty or MA, then the service call is refused and posted back via e-mail notification.

  • Page 460: Electronic Service Agent In Sp/Ces Environment

    (in the United States) is required. Please refer to local procedures in your country to see what the modem requirements are for Service Agent. – IBM ships the following modem types for use with Service Agent on some products: 7852 Model 400, 7857-017 or 7858-336 On your Gateway server, ensure Point-to-Point Protocol (PPP) is installed and configured.

  • Page 461: Obtaining Electronic Service Agent

    README of the latest level of microcode. 2. Gathers Vital Product Data (VPD) from the user's machine and when used with web-based VPD Capture Service, uploads it to IBM's MRPD database. This VPD will be helpful in determining the correct components to ship when a Miscellaneous Equipment Specification (MES) upgrade is ordered.
  • Page 462 VPD Capture Service Http://techsupport.services.ibm.com/rs6k/vcs.html RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 463: Related Publications

    The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook. IBM Redbooks For information on ordering these publications, see “How to get IBM Redbooks” on page 440. IBM RS/6000 Clustered Enterprise Servers Systems Handbook, SG24-5120 RS/6000 Systems Handbook 2000 Edition, SG24-5120 Exploiting HACMP 4.4: Enhancing the Capabilities of Mutil-Processing,...

  • Page 464: How To Get Ibm Redbooks

    Redbooks are also available on CD-ROMs. Click the CD-ROMs button on the Redbooks Web site for information about all the CD-ROMs offered, as well as updates and formats. RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 465: Special Notices

    While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.
  • Page 466 Anywhere.,TME, NetView, Cross-Site, Tivoli Ready, Tivoli Certified, Planet Tivoli, and Tivoli Enterprise are trademarks or registered trademarks of Tivoli Systems Inc., an IBM company, in the United States, other countries, or both. In Denmark, Tivoli is a trademark licensed from Kjøbenhavns Sommer - Tivoli A/S.

  • Page 467: Abbreviations And Acronyms

    First Failure Data Capture Permanent Virtual Circuits GPFS General Parallel File System Quadratic Programming Graphical User Interface Remote Async Nodes HACMP IBM High Availability Cluster Multi-Processing for AIX RSCT RS/6000 Cluster Technology HACWS High Availability Control RVSD Recoverable Virtual Shared...
  • Page 468 Tool command Language User Space Unshielded Twisted Pair Virtual Shared Disk Wide Area Network RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...

  • Page 469: Index

    72 ARTIC960Hx 4-port RS-449 cable 241 disk option 74 ARTIC960Hx 4-port T1 RJ45 cable 258 disk requirement 74 ARTIC960Hx 4-port V.35 (DTE) cable 241 memory option 74 ARTIC960Hx 4-port X.21 cable 241 memory requirement 74 © Copyright IBM Corp.
  • Page 470 ERP (Enterprise Resource Planning) 3, 8 rvsdrestrict 419 ESSL (Scientific Subroutine Library) 261 setup_server 431 Event Perspective 340 spadaptrs 352 expansion frame 155 spdelfram 337 expansion frames 146 spframe 337, 338, 347, 349 spmon 429 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 471 2962 2-port Multiprotocol X.25 Adapter 241 2058 375 MHz POWER3 SMP High Node 37 2963 2-port Multiprotocol X.25 Adapter 242 2402 IBM Network Terminal Accelerator - 256 2968 Ethernet 10/100 MB 243 Session 234 2969 Gigabit Ethernet - SX 244...
  • Page 472 3821 HACMP 221 36.4 GB Ultra SCSI 10K RPM disk pair 42 HACMP (High Availability Cluster Multi-Processing ) feature code 3822 36.4 GB SSA 10K RPM disk pair 42 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 473 13 router.log 176 scan-out.log 176 scan-save.log 176 topology.data 176 IBM 9077 SP Switch Router model 04S 182 loose clusters 18 IBM 9077 SP Switch Router model 16S 182 Low-level Application Programming Interface 266 IBM DB2 UDB EEE 8...
  • Page 474 SP frames 143 SCSI-2 Differential External I/O Controller 232, 234 power supply phases for SP frames 143 SCSI-2 Differential Fast/Wide Adapter/A 232 POWER3 2 SCSI-2 Fast/Wide Adapter/A 234 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 475 SDR 262 connection to the SP Switch 192 SEPBU 140, 154 network interface 189 Serial port to serial port cable for drawer-to-drawer network media card requirement 188 connections 239 software requirement 189 Serial port to serial port cable for rack-to-rack con- system requirement 187 nections 239 SP Switch Router Adapter 169...
  • Page 476 VPD (Vital Product Data) 6 VSD (Virtual Shared Disk) 265 VSD (Virtual Shared Disks) 5 VSD Perspective 340 Wide Nodes 13 wide nodes 13 WLM (AIX WorkLoad Manager) 5 X.21 cable 242 XL FORTRAN 273 RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook...
  • Page 480 ® RS/6000 SP and Clustered IBM ^ pSeries Systems Handbook Overview of the This redbook is an update of the RS/6000 SP Systems INTERNATIONAL Handbook published in 2000. This publication includes Blue RS/6000 SP and TECHNICAL Hammer and Blue Hammer Jr. clustering solutions...

Table of Contents