1. DL manuals
  2. IBM
  3. Switch
  4. RS/6000 SP
  5. Handbook

IBM RS/6000 SP Handbook - Sp Internal Processor Nodes

Other manuals for RS/6000 SP: Service Manual
Manual is about: Clustered IBM eServer pSeries Systems

Summary of RS/6000 SP

  • Page 1

    Ibm.Com/redbooks dino qintero christian courty corneliu holban willis c. White iii rs/6000 sp and clustered ibm ^ pseries systems handbook overview of the rs/6000 sp and pseries servers configurations information on available machine types, models, and feature codes in-depth discussion of hardware a...

  • Page 3

    Rs/6000 sp and clustered ibm ^ pseries systems handbook september 2001 international technical support organization sg24-5596-02.

  • Page 4

    © copyright international business machines corporation 1999 - 2001. All rights reserved. Note to u.S government users – documentation related to restricted rights – use, duplication or disclosure is subject to restrictions set forth in gsa adp schedule contract with ibm corp. Third edition (septemb...

  • Page 5: Summary of Changes

    © copyright ibm corp. Iii summary of changes this section describes the technical changes made in this edition of the book and in previous editions. This edition may also include minor corrections and editorial changes that are not identified. Third edition, september 2001 this revision reflects the...

  • Page 6

    Iv rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 7: Contents

    © copyright ibm corp. V contents summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Iii third edition, september 2001. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Iii figures . . . . . . . . . . . . . . . . . . . ....

  • Page 8

    Vi rs/6000 sp and clustered ibm ^ pseries systems handbook 2.3.3 options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.3.4 processor requirements and options . . . . . . . . . . . . . . . . . . . . . . . . 38 2.3.5 memory requirements and options ...

  • Page 9

    Contents vii 2.8.2 smp system bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 2.8.3 processor/l2 cache controller. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 2.8.4 memory-i/o controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....

  • Page 10

    Viii rs/6000 sp and clustered ibm ^ pseries systems handbook 3.4.4 sp-attached servers connectivity diagram . . . . . . . . . . . . . . . . . . . 128 3.4.5 configuring service director on sp-attached servers. . . . . . . . . . . 128 3.5 clustered enterprise server environment . . . . . . . . . . . ....

  • Page 11

    Contents ix 5.4 sp switch-8 (f/c 4008) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 5.5 sp switch adapter (f/c 4020, 4022, and 4023) . . . . . . . . . . . . . . . . 168 5.6 special sp switch adapter (f/c 4021 and f/c 8396) . . . . . . . . . . . . 169 5.6.1 sp switch router...

  • Page 12

    X rs/6000 sp and clustered ibm ^ pseries systems handbook 8.2.4 ibm short wave serial hippi pci adapter (fc 2732) . . . . . . . . . . 232 8.2.5 long-wave serial hippi adapter (f/c 2733) . . . . . . . . . . . . . . . . . 233 8.2.6 ibm network terminal accelerator adapter (f/c 2402 & 2403) . . . 234 8...

  • Page 13

    Contents xi 9.1.4 ibm virtual shared disk (vsd) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 9.1.5 security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 9.1.6 performance toolbox parallel extensions for aix (ptpe) . . . . . . . 265 9....

  • Page 14

    Xii rs/6000 sp and clustered ibm ^ pseries systems handbook 10.2.2 sp system with sp-attached servers . . . . . . . . . . . . . . . . . . . . . . 314 10.2.3 ces system in non-switched and switched environment . . . . . . . 324 10.3 pssp environment for sp/ces systems . . . . . . . . . . . . . . . . ...

  • Page 15

    Contents xiii migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 coexistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416 high availability cluster multi-processing (hacmp) ...

  • Page 16

    Xiv rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 17: Figures

    © copyright ibm corp. Xv figures 1-1 rs/6000 sp system sample configuration. . . . . . . . . . . . . . . . . . . . . . . 10 1-2 rs/6000 sp system with sp-attached servers sample configuration . . 11 1-3 clustered enterprise server system sample configuration . . . . . . . . . . 12 1-4 load balancing...

  • Page 18

    Xvi rs/6000 sp and clustered ibm ^ pseries systems handbook 7-1 control workstation interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 7-2 high availability control workstation . . . . . . . . . . . . . . . . . . . . . . . . . . 222 10-1 examples on node configuration . . ...

  • Page 19: Tables

    © copyright ibm corp. Xvii tables 1-1 servers supported as sp-attached servers . . . . . . . . . . . . . . . . . . . . . 14 2-1 375 mhz power3 smp high node . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2-2 375 mhz power3 smp thin and wide nodes . . . . . . . . . . . . . . . . . . 35 2-3 ...

  • Page 20

    Xviii rs/6000 sp and clustered ibm ^ pseries systems handbook 9-2 software requirements for gpfs 1.4 . . . . . . . . . . . . . . . . . . . . . . . . . . 269 9-3 software requirements for loadleveler 2.2 . . . . . . . . . . . . . . . . . . . . . 272 9-4 requirements for essl 3.2, parallel essl 2.2 an...

  • Page 21: Preface

    © copyright ibm corp. Xix preface this redbook is an update of the rs/6000 sp systems handbook published in 2000. This publication includes blue hammer and blue hammer jr. Clustering solution configurations. This publication includes the following: information on m80, h80, 6h1 sp-attached support as...

  • Page 22

    Xx rs/6000 sp and clustered ibm ^ pseries systems handbook since 1996. His areas of expertise include enterprise backup and recovery, disaster recovery planning and implementation, and rs/6000. He is also a microsoft certified systems engineer. Currently, he focuses on rs/6000 cluster technology by ...

  • Page 23: Special Notice

    Preface xxi special notice this publication is intended to help ibm customers, ibm business partners, ibm sales professionals, ibm i/t specialists, and the ibm technical support community when proposing rs/6000 cluster base solutions. The information in this publication is not intended as the specif...

  • Page 24

    Xxii rs/6000 sp and clustered ibm ^ pseries systems handbook ibm.Com/redbooks send your comments in an internet note to: redbook@us.Ibm.Com mail your comments to the address on page ii..

  • Page 25: Introduction

    © copyright ibm corp. 1 chapter 1. Introduction the ibm rs/6000 sp high-performance system uses the power of parallel processing to expand your applications. Designed and extended as a clustered ^ pseries system for performance, scalability, reliability, availability, serviceability, and management,...

  • Page 26: Pseries

    2 rs/6000 sp and clustered ibm ^ pseries systems handbook 1.1 overview on rs/6000 sp and clustered ^ pseries the rs/6000 sp system simultaneously brings dozens to hundreds of risc processor nodes to a computing problem. Its parallel processing capability enhances computing performance and throughput...

  • Page 27

    Chapter 1. Introduction 3 effective parallel computing requires high-bandwidth, low-latency internode communications. The sp switch, recommended for high-performance commercial computing, provides a one-way bandwidth of up to 150 mb per second between nodes (300 mb bidirectional) for interconnecting...

  • Page 28: 1.2  Features And Benefits

    4 rs/6000 sp and clustered ibm ^ pseries systems handbook the sp universal cluster system (sp/ces system) delivers balanced performance with processor, memory, switch, and i/o scalability. Over time, the sp system has demonstrated leadership in standard industry benchmarks. An sp e-business server i...

  • Page 29

    Chapter 1. Introduction 5 each processor node contains its own copy of the standard aix operating system and other standard rs/6000 system software, providing access to thousands of available aix applications. A set of new software products designed specifically for the sp (pssp with virtual shared ...

  • Page 30

    6 rs/6000 sp and clustered ibm ^ pseries systems handbook the sp/ces processor nodes incorporate a dynamic cpu de-allocation function, which enables the system to detect failing processors and take them offline without re-booting the system. The system service processor can record this action and no...

  • Page 31: 1.3  Business Solutions

    Chapter 1. Introduction 7 scalability makes upgrading and expansion easier and allows for transparent application growth. Aix operating system provides a wealth of multiuser communications and systems management technologies and complies with major industry standards. Additionally, it provides aix b...

  • Page 32

    8 rs/6000 sp and clustered ibm ^ pseries systems handbook commercial computing users implement the sp/ces mission-critical commercial computing solutions to address business intelligence applications, server consolidation, and collaborative computing comprised of lotus notes, lotus domino server, in...

  • Page 33: 1.4  Hardware Components

    Chapter 1. Introduction 9 helps reduce the complexities and costs of systems management, lowering the total cost of ownership and allowing simplification of application service level management. Additionally, this solution leverages investment in hardware and software, allowing better sharing of res...

  • Page 34

    10 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 1-1 rs/6000 sp system sample configuration as shown in figure 1-1: the thin, wide, and high sp nodes, and the sp switch are mounted in a tall frame. The sp nodes and sp switch router are connected to the sp switch. The sp nodes and th...

  • Page 35

    Chapter 1. Introduction 11 figure 1-2 rs/6000 sp system with sp-attached servers sample configuration figure 1-2 shows an rs/6000 sp system with sp-attached servers sample configuration: the sp node and the sp switch are mounted in a tall frame. The sp-attached servers (m/t 7017 and m/t 7026) are ho...

  • Page 36

    12 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 1-3 clustered enterprise server system sample configuration figure 1-3 shows a pseries ces system sample configuration: the clustered enterprise servers (m/t 7017 and m/t 7026) are housed in their own frames. The sp switch is mounted ...

  • Page 37

    Chapter 1. Introduction 13 there are four types of rs/6000 sp processor nodes: high nodes wide nodes thin nodes sp-attached servers and clustered enterprise servers all nodes are available in an smp configuration and they use pci architecture. High nodes the high nodes occupy two full drawer locatio...

  • Page 38

    14 rs/6000 sp and clustered ibm ^ pseries systems handbook thin nodes the thin nodes occupy half of a drawer (one slot). Up to sixteen of these nodes can be housed in a tall frame. When installed singly within a drawer, these nodes must be placed in an odd-numbered node slot. The wide nodes are comp...

  • Page 39

    Chapter 1. Introduction 15 the clustered enterprise server (ces) is an ^ pseries or an rs/6000 enterprise server, supported to be configured to operate with a clustered ^ pseries system (ces system). All the possible sp-attached servers to an sp system, can be defined as clustered enterprise servers...

  • Page 40

    16 rs/6000 sp and clustered ibm ^ pseries systems handbook frames have locations known as drawers into which the processor nodes are mounted. Tall frames have eight drawers and short frames have four. Each drawer location is further divided into two slots. A slot has the capacity of one thin node or...

  • Page 41

    Chapter 1. Introduction 17 into slots in the sp switch router. In the same way that the sp switch router adapter connects the sp switch router directly to the sp switch, these networking cards enable the sp switch router to directly connect to an external network. Network connections through sp swit...

  • Page 42

    18 rs/6000 sp and clustered ibm ^ pseries systems handbook rs/6000 44p model 170 (7044-170) a high availability control workstation (hacws) option is also available on configurations with sp standard frames only. For more information, refer to chapter 7, “control workstations” on page 195. 1.5 rs/60...

  • Page 43

    Chapter 1. Introduction 19 figure 1-4 load balancing cluster – high availability cluster • increases application or data availability • two or more nodes (primary and backup) • active/passive and active/active configurations • “fail-over” of resources to backup node (nodes) when primary node fails f...

  • Page 44

    20 rs/6000 sp and clustered ibm ^ pseries systems handbook parallel applications (applications are segmented to run in parallel and scale) – high performance technical computing (message passing cluster) • extreme vertical scalability for a single large problem • mostly based on mpi (message passing...

  • Page 45

    Chapter 1. Introduction 21 figure 1-7 high performance commercial computing cluster server consolidation (applications are consolidated to reduce costs) – common management domain • consolidation of different workloads • heterogeneous building blocks • central management point-of control • consolida...

  • Page 46

    22 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 1-8 common management domain the following list shows benefits of aix clustering: physical isolation between partitions can be used as fail-over for each other easy accounting for user departments economic solution for startup and gro...

  • Page 47

    Chapter 1. Introduction 23 to meet the demands of e-business, the sp/ces systems, as clustered computing-based infrastructure solutions, deliver primary computing characteristics as: capacity on demand: the infrastructure may allocate and deallocate computing resources (servers, storage, processors)...

  • Page 48

    24 rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 49: Sp Internal Processor Nodes

    © copyright ibm corp. 25 chapter 2. Sp internal processor nodes sp processor nodes are independent rs/6000 systems mounted in sp frames. Each node has its own processors, memory, disks, i/o slots, and aix operating system. They are the basic building blocks of the sp systems. This chapter presents t...

  • Page 50: 2.1  Sp Internal Node

    26 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.1 sp internal node the basic rs/6000 sp building block is the servers or processor node. Each node has a processor or processors, memory, disk drives, and pci expansion slots for i/o and connectivity such as any other server. There are thr...

  • Page 51

    Chapter 2. Sp internal processor nodes 27 375 mhz power3 smp thin node (f/c 2056) current supported nodes (cannot be ordered anymore): power3 smp high node (f/c 2054) 332 mhz smp wide node (f/c 2051) 332 mhz smp thin node (f/c 2050) the previous nodes are referenced in appendix b, “hardware” on page...

  • Page 52

    28 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.2.1 375 mhz power3 processor the 375 mhz power3 design contains a superscalar core, which is comprised of eight execution units, supported by a high bandwidth memory interface capable of performing four floating-point operations per clock ...

  • Page 53

    Chapter 2. Sp internal processor nodes 29 figure 2-2 375 mhz power3 smp architecture (high node) memory cards, 1gb to 64gb ecc memory available i/o hub switch complex i switch complex ii switched memory subsystem i2 i4 i3 i1 i6 i8 i7 i5 mx-pci mx-pci mx-pci mx-pci rio-mx sp expansion i/o unit high n...

  • Page 54

    30 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.2.2 system memory the sp 375 mhz power3 smp system supports 256 mb to 64 gb of 10 ns sdram. System memory is controlled by the memory-i/o chip set through the memory bus. The memory bus consists of a 512-bit data bus and operates at 62.5 m...

  • Page 55

    Chapter 2. Sp internal processor nodes 31 figure 2-3 power3 high node and five sp expansion i/o units connections 2.2.4 service processor the service processor function is integrated on the i/o planar board in the 375 mhz power3 smp node. Service processor function is for initialization, system erro...

  • Page 56

    32 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.2.5 system firmware and rtas the 375 mhz power3 smp node system firmware flash memory is located on the i/o planar. System firmware contains code that is executed by the 375 mhz power3 microprocessor during the initial program load (ipl) p...

  • Page 57

    Chapter 2. Sp internal processor nodes 33 standard ram 1 gb maximum ram 64 gb disk/media bays 2/26 with sp expansion i/o unit aix storage internal/external boot disks internal disks installed in pairs to support mirroring maximum internal storage 946.4 gb with sp expansion i/o unit pci bus speed 33 ...

  • Page 58

    34 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 2-4 375 mhz power3 high node packaging.

  • Page 59

    Chapter 2. Sp internal processor nodes 35 2.2.8 375 mhz power3 smp thin and wide nodes table 2-2 outlines the 375 mhz power3 smp thin and wide nodes. Figure 2-5 on page 36 shows the 375 mhz power3 smp wide node packaging. Table 2-2 375 mhz power3 smp thin and wide nodes node type 375 mhz power3 smp ...

  • Page 60

    36 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 2-5 375 mhz power3 smp wide node packaging.

  • Page 61

    Chapter 2. Sp internal processor nodes 37 2.3 375 mhz power3 smp high node (f/c 2058) this section describes the 375 mhz power3 smp high nodes also called nighthawk-ii node. The 375 mhz power3 smp high nodes (f/c 2058) use pci bus architecture and have four, eight, twelve, or sixteen 375 mhz 630fp 6...

  • Page 62

    38 rs/6000 sp and clustered ibm ^ pseries systems handbook four memory slots supporting up to 64 gb of memory five pci slots (four 64-bit and one 32-bit) for communication adapters a dedicated mezzanine bus (mx) slot for an optional switch adapter integrated ethernet with bnc and rj45 ports (only on...

  • Page 63

    Chapter 2. Sp internal processor nodes 39 for the best memory-access bandwidth, memory dimms should be distributed evenly across four memory cards. As an example, you can realize better bandwidth by using four bank of 128 mb dimms (4 gb total) distributed evenly over four memory cards rather than by...

  • Page 64

    40 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.3.6 disk requirements and options 375 mhz power3 smp high nodes can have one pair of internal hard disk drive attached through an integrated ultra scsi network. The node can have either no internal hard disk drive (with external booting) o...

  • Page 65

    Chapter 2. Sp internal processor nodes 41 2.3.8 i/o adapters the 375 mhz power3 smp high node has five pci (peripheral component interconnect) adapter slots. A full line of pci adapters is offered for these nodes including: scsi-2 ethernet token ring fddi atm async wide area network (wan) ssa raid5 ...

  • Page 66

    42 rs/6000 sp and clustered ibm ^ pseries systems handbook expansion units can be mounted in the same frame as the node, using 2 m cables (f/c 3126), or in separate frames using 15 m cables (f/c 3127). These units require a tall, deep frame (model 550 or f/c 1550); they are not supported in the with...

  • Page 67

    Chapter 2. Sp internal processor nodes 43 2.4 375 mhz power3 smp wide node (f/c 2057) this section describes the 375 mhz power3 smp wide nodes, also called the winterhawk-ii wide node. The 375 mhz power3 smp wide nodes (f/c 2057) have pci bus architecture and either two or four 375 mhz 64-bit proces...

  • Page 68

    44 rs/6000 sp and clustered ibm ^ pseries systems handbook an upgraded power system on early-style frames. 2.4.3 options available options include the following: two processor slots allowing a maximum of four processors per node. Two memory slots supporting up to 8 gb of memory. Ten pci slots (two 3...

  • Page 69

    Chapter 2. Sp internal processor nodes 45 2.4.5 memory requirements and options 375 mhz power3 smp wide nodes require two memory cards and a minimum of 256 mb of memory. These nodes support a maximum of 16 gb of memory; each card has an 8 gb capacity. Memory is supplied by 128, 256, and 512 mb dimms...

  • Page 70

    46 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.4.6 hard disk drive requirements and options 375 mhz power3 smp wide nodes can have up to two pairs of internal hard disk drives attached through an integrated ultra scsi network. This node can have either no internal hard disk drive (with...

  • Page 71

    Chapter 2. Sp internal processor nodes 47 2.4.7 switch and communication adapter requirements if you are planning to use a switch in your sp system, you need switch adapters to connect each rs/6000 sp node to the switch subsystem. Sp switch adapters have several characteristics and restrictions, inc...

  • Page 72

    48 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.5 375 mhz power3 smp thin node (f/c 2056) this section describes the 375 mhz power3 smp thin nodes, also called the winterhawk-ii thin node. 375 mhz power3 smp thin nodes (f/c 2056) have pci bus architecture and either two or four 375 mhz ...

  • Page 73

    Chapter 2. Sp internal processor nodes 49 two processors (mounted in one slot). 256 mb of memory. 4.5 gb of mirrored hard disk drive (with internal booting). An upgraded power system on early-style frames. 2.5.3 options available options include the following: four processors in two slots. Two memor...

  • Page 74

    50 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.5.5 memory requirements and options 375 mhz power3 thin nodes require two memory cards and a minimum of 256 mb of memory. These nodes support a maximum of 16 gb of memory; each card has an 8 gb capacity. Memory is supplied by 128, 256, and...

  • Page 75

    Chapter 2. Sp internal processor nodes 51 2.5.6 hard disk drive requirements and options 375 mhz power3 smp thin nodes can have one pair of internal hard disk drive attached through an integrated ultra scsi network. The node can have either no internal hard disk drive (with external booting) or from...

  • Page 76: 2.6  Power3 Smp Nodes

    52 rs/6000 sp and clustered ibm ^ pseries systems handbook switch adapters for these nodes are not interchangeable with either the switch adapters used on uniprocessor wide nodes or with the sp switch mx adapter (f/c 4022) used on 332 mhz smp nodes. I/o adapters the 375 mhz power3 smp thin nodes hav...

  • Page 77

    Chapter 2. Sp internal processor nodes 53 2.6.1 power3 smp node system architecture power3 smp node system design is based on the ibm powerpc architecture and the rs/6000 platform architecture. The node is designed as a bus-based symmetrical multiprocessor (smp) system using a 64-bit address and a 1...

  • Page 78

    54 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.6.3 6xx bus the 6xx bus or system bus, connects up to four processor cards to the memory-i/o controller chip set. This bus is optimized for high performance and multiprocessing applications. It provides 40 bits of real address and a separa...

  • Page 79

    Chapter 2. Sp internal processor nodes 55 2.7 222 mhz power3 smp high node (f/c 2054) this section describes the 222 mhz power3 smp high node, also called nighthawk node. The 222 mhz power3 smp high nodes (f/c 2054) use pci bus architecture and have either two, four, six, or eight 222 mhz 64-bit pro...

  • Page 80

    56 rs/6000 sp and clustered ibm ^ pseries systems handbook two processors (on one card, mounted in one slot). 1 gb of memory. 9.1 gb mirrored hard disk drive (with internal booting). 2.7.3 options available options include the following: four processors slots allowing a maximum of eight processors p...

  • Page 81

    Chapter 2. Sp internal processor nodes 57 2.7.4 processor requirements and options 222 mhz power3 smp high nodes require a minimum of two 222 mhz, powerpc processors mounted on one card. You can order up to three additional two-processor cards (f/c 4849) to configure the node with a total of sixteen...

  • Page 82

    58 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.7.6 disk requirements and options 222 mhz power3 smp high nodes can have one pair of internal hard disk drive attached through an integrated ultra scsi network. The node can have either no internal hard disk drive (with external booting) o...

  • Page 83

    Chapter 2. Sp internal processor nodes 59 2.7.8 i/o adapters the power3 smp high node has five pci (peripheral component interconnect) adapter slots. A full line of pci adapters is offered for these nodes including: scsi-2 ethernet token ring fddi atm async wide area network (wan) ssa raid5 s/390 es...

  • Page 84

    60 rs/6000 sp and clustered ibm ^ pseries systems handbook each expansion unit (or pair of units) requires a mounting shelf (f/c 9935). This shelf occupies the space of one drawer in a frame. If only a single expansion unit is mounted in the shelf, a filler plate (f/c 9936) is required for the other...

  • Page 85: 2.8  332 Mhz Smp Nodes

    Chapter 2. Sp internal processor nodes 61 2.8 332 mhz smp nodes on april 21, 1998, ibm first introduced the pci-based nodes, the rs/6000 sp 332 mhz smp thin nodes and wide nodes. They provide two or four way symmetric multiprocessing (smp) utilizing powerpc technology and extend the rs/6000 pci i/o ...

  • Page 86

    62 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 2-6 332 mhz smp node system architecture block diagram 2.8.2 smp system bus the system bus of the 332 mhz smp node is optimized for high performance and multiprocessing applications. It has a separate 64-bit + parity address bus and a...

  • Page 87

    Chapter 2. Sp internal processor nodes 63 system at any one time. The address bus includes status and coherency response buses for returning flow control, error reports, or coherency information for each request. It can support a new request every other bus cycle at a sustained rate of over 40 milli...

  • Page 88

    64 rs/6000 sp and clustered ibm ^ pseries systems handbook since the l2 cache maintains inclusion with the processor's l1 cache, it filters the majority of other processor snoop transactions, thus, reducing the contention for the l1 cache and increasing processor performance. 332 mhz smp node x5 cac...

  • Page 89

    Chapter 2. Sp internal processor nodes 65 the sdram dimm itself contains multiple internal banks. The 32 mb dimm has two internal banks, and the 128 mb dimm has four internal banks. A different non-busy bank can be accessed to keep the memory fully utilized while previously accessed banks are precha...

  • Page 90

    66 rs/6000 sp and clustered ibm ^ pseries systems handbook is 400 mbps, and it is also a split-transaction, fully tagged bus, like the smp system bus. Multiple requests from each i/o bridge and the processors may be active at any one time, therefore, subject only to the amount of queuing and tags av...

  • Page 91

    Chapter 2. Sp internal processor nodes 67 2.9 332 mhz smp wide nodes (f/c 2051) this section describes the 332 mhz power3 smp high nodes also called silver wide node. The 332 mhz smp wide nodes (f/c 2051) have pci bus architecture and use either two or four 332 mhz powerpc processors per node. These...

  • Page 92

    68 rs/6000 sp and clustered ibm ^ pseries systems handbook artic960hx 4-port selectable (f/c 2947) 2-port multiprotocol (f/c 2962) atm 155 utp (f/c 2963) gigabit ethernet - sx (f/c 2969) atm 155 mmf (f/c 2988) ultra scsi se (f/c 6206) ultra scsi de (f/c 6207) ssa raid5 (f/c 6215) artic960rxd quad di...

  • Page 93

    Chapter 2. Sp internal processor nodes 69 the mandatory requirements are: pssp 2.4 (or later) on the control workstation, backup nodes, and processor node two processors (mounted in one slot) 256 mb of memory 4.5 gb of hard disk drive (with internal booting) an upgraded power system on older frames ...

  • Page 94

    70 rs/6000 sp and clustered ibm ^ pseries systems handbook 2.9.6 memory requirements and options 332 mhz smp wide nodes have two memory cards and require a minimum of 256 mb of memory. These nodes support a maximum of 3 gb of memory. Memory is supplied by 128 mb dimms that must be mounted in pairs (...

  • Page 95

    Chapter 2. Sp internal processor nodes 71 this node does not require special cables or adapters to mount internal disks. However, the 332 mhz smp wide node has an option (f/c 1241) that provides an independent scsi hookup. It accomplishes the following: eliminates the hard disk drive controller as a...

  • Page 96

    72 rs/6000 sp and clustered ibm ^ pseries systems handbook the 332 mhz smp thin node occupies one half of a drawer and may be installed singly with systems operating at pssp 3.1 or later. Therefore, up to 16 smp thin nodes can be housed in a tall frame. When installed singly, 332 mhz smp thin nodes ...

  • Page 97

    Chapter 2. Sp internal processor nodes 73 single power3 smp thin nodes and single 332 mhz smp thin nodes may be mixed in a thin node drawer. If a frame has more than six single smp thin nodes installed, that frame will have an uneven weight distribution. You must be careful when moving these frames....

  • Page 98

    74 rs/6000 sp and clustered ibm ^ pseries systems handbook – frame-to-frame hacmp cable ( f/c 3125) 2.10.5 processor requirements and options smp thin nodes require a minimum of two 332 mhz powerpc processors mounted on one card. However, you can order an additional processor card (f/c 4320) to conf...

  • Page 99

    Chapter 2. Sp internal processor nodes 75 external storage devices can be accessed through an optional ultra scsi adapter (f/c 6207) or scsi-2 adapter (f/c 6209) and ssa adapter (f/c 6230). Optional direct access storage devices are available as follows: 4.5 gb ultra scsi disk drive (f/c 2900) 4.5 g...

  • Page 100

    76 rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 101: Sp-Attached Servers And

    © copyright ibm corp. 77 chapter 3. Sp-attached servers and clustered enterprise servers the rs/6000 sp system, as a cluster of sp nodes, managed by the aix operating system and pssp management software from the control workstation (cws), is the path to universal clustering. Managed by the aix and p...

  • Page 102: 3.1  Overview

    78 rs/6000 sp and clustered ibm ^ pseries systems handbook 3.1 overview the rs/6000 sp system, as a cluster of sp nodes and managed by the aix operating system and pssp management software from the control workstation, is the path to universal clustering. We may consider sp systems with sp-attached ...

  • Page 103: Models

    Chapter 3. Sp-attached servers and clustered enterprise servers 79 up to 16 of the supported m/t 7017 models and up to 32 of the supported m/t 7026 models, up to 32 mixed servers in total, can be integrated on a ces system as clustered enterprise servers. Like an sp node, the sp-attached server and ...

  • Page 104

    80 rs/6000 sp and clustered ibm ^ pseries systems handbook processors, determines that the number of processors has been increased and notifies ibm. The customer is responsible for placing an order for the additionally activated processors with their ibm or ibm business partner sales representative....

  • Page 105

    Chapter 3. Sp-attached servers and clustered enterprise servers 81 commercial performance in 4-, 6-, 12-, 18-, or 24-way enterprise servers using the rs64 iv microprocessor technology. Capacity upgrade on demand which allows additional processors to be easily added to the pseries 680 as your busines...

  • Page 106

    82 rs/6000 sp and clustered ibm ^ pseries systems handbook industry-standard pci subsystem supporting 32-bit and 64-bit adapters. Greater application throughput is achieved by the four independent pci buses per rawer, providing up to 500 mb per second aggregate bandwidth per drawer. A high-performan...

  • Page 107

    Chapter 3. Sp-attached servers and clustered enterprise servers 83 figure 3-1 system architecture for 7017-s85 server memory quads, 4gb to 96gb ecc memory available i/o hub switch complex i switch complex ii switched memory subsystem primary i/o drawer unit i2 i4 i3 i1 i6 i8 i7 i5 pci c nt rl pc i c...

  • Page 108

    84 rs/6000 sp and clustered ibm ^ pseries systems handbook product positioning the 7017-s85 server has design and application performance that makes it a solution for mission-critical erp/scm, data warehouse/datamart, oltp, and e-business applications. It offers more performance, memory, and storage...

  • Page 109

    Chapter 3. Sp-attached servers and clustered enterprise servers 85 the publications are available for download from the internet: http://www.Ibm.Com/servers/eserver/pseries/library/ 3.2.2 rs/6000 enterprise server 7017 model s80 (7017-s80) the 7017-s80 server is the member of the popular s-family of...

  • Page 110

    86 rs/6000 sp and clustered ibm ^ pseries systems handbook up to three additional 7017-s80 i/o drawers can be added. Additional i/o racks can also be ordered with the7017-s80. Existing rs/6000 7015 model r00 racks or the 7014 model s00 rack can also be used for additional storage and communication d...

  • Page 111

    Chapter 3. Sp-attached servers and clustered enterprise servers 87 description the 7017-s80 provides performance and supports the evolution of your enterprise to 64-bit computing. Its outstanding benefits include: commercial performance in 6-, 12-, 18-, or 24-way enterprise servers using the rs64 ii...

  • Page 112

    88 rs/6000 sp and clustered ibm ^ pseries systems handbook industry-standard pci subsystem supporting 32-bit and 64-bit adapters. Greater application throughput is achieved by the four independent pci buses per drawer, providing up to 500 mb per second aggregate bandwidth per drawer. A high-performa...

  • Page 113

    Chapter 3. Sp-attached servers and clustered enterprise servers 89 the publications are available for download from the internet: http://www-1.Ibm.Com/servers/eserver/pseries/library/ 3.2.3 rs/6000 enterprise server 7017 model s7a (7017-s7a) the 7017-s7a (s70 advanced) server is a member of the s- f...

  • Page 114

    90 rs/6000 sp and clustered ibm ^ pseries systems handbook description the 7017-s7a is packaged as a central electronics complex and an i/o rack. The 7017-s7a central electronics complex entry configuration starts with a 4-way scalable smp system that utilizes the 64-bit, 262 mhz, rs64 ii processor ...

  • Page 115

    Chapter 3. Sp-attached servers and clustered enterprise servers 91 – hot-swappable disk drives – environmental sensing – n+1 power – service processor for integrated system monitoring – concurrent diagnostics preconfigured tailored solutions that take advantage of the rack packaging to custom tailor...

  • Page 116

    92 rs/6000 sp and clustered ibm ^ pseries systems handbook choice of power from 4, 8, or 12 processors extensive memory options up to 32 gb up to 55 available expansion slots up to 14 tb of disk storage the 7017-s7a server is designed from the ground up for mission-critical environments with a servi...

  • Page 117

    Chapter 3. Sp-attached servers and clustered enterprise servers 93 highlights this system is the first 64-bit-ready member of the enterprise server family. It delivers a cost-efficient growth path to the future through these attributes: 4-way to 12-way, 64-bit, 125 mhz system scalability – packaged ...

  • Page 118

    94 rs/6000 sp and clustered ibm ^ pseries systems handbook the 7017-s70 provides outstanding performance and supports the evolution of your enterprise to 64-bit computing. Its benefits include: commercial performance in 4-way to 12-way enterprise servers that use the new rs64 microprocessor technolo...

  • Page 119

    Chapter 3. Sp-attached servers and clustered enterprise servers 95 it has the capacity to meet your current and future needs with a choice of power from four to 12 processors, extensive memory options to 32 gb, up to 54 expansion slots and 14 tb of disk storage. The 7017-s70 is designed from the gro...

  • Page 120

    96 rs/6000 sp and clustered ibm ^ pseries systems handbook data/instruction (l1) cache 128kb/128kb 128kb/128kb 64kb/64kb 64kb/64kb l2 cache 16mb 8mb/16mb 8mb 4mb l2 ecc yes yes yes yes service processor yes yes yes yes hot-swappable disks yes yes yes yes dynamic cpu deallocation yes yes yes - hot-pl...

  • Page 121: Models

    Chapter 3. Sp-attached servers and clustered enterprise servers 97 3.3 sp-attached servers/ces - m/t 7026 supported models this section describes the m/t 7026 models which are supported as sp-attached servers and clustered enterprise servers. 3.3.1 rs/6000 enterprise server 7026 model m80 (7026-m80)...

  • Page 122

    98 rs/6000 sp and clustered ibm ^ pseries systems handbook description the 7026-m80 delivers a cost-efficient growth path to the future. It provides 64-bit scalability via the 64-bit rs64 iii processor packaged as 2-way and 4-way cards. With its two processor positions, the 7026-m80 can be configure...

  • Page 123

    Chapter 3. Sp-attached servers and clustered enterprise servers 99 the 7026-m80 server is shipped and delivered with all internal adapters and devices already installed and configured. Aix version 4.3.3 software is included with every 7026-m80 and can be preinstalled. The 7026-m80 provides performan...

  • Page 124

    100 rs/6000 sp and clustered ibm ^ pseries systems handbook internal i/o system scalability and expandability that supports: – up to four system i/o drawers with up to 56 pci slots – up to eight media bays for optical storage devices (a cd-rom readable device) and tape devices primary i/o drawer fea...

  • Page 125

    Chapter 3. Sp-attached servers and clustered enterprise servers 101 figure 3-2 system architecture for 7026-m80 server product positioning the 7026-m80 consists of a rack-mounted central electronics complex (cec) drawer containing the processors and memory connected to another rack-mounted rawer con...

  • Page 126

    102 rs/6000 sp and clustered ibm ^ pseries systems handbook the 7026-m80 is targeted at the key commercial processing segments of e-business, erp, scm, and business intelligence. In each segment, the 7026-m80 can meet critical requirements. For example: in erp, the 7026-m80 is an application server ...

  • Page 127

    Chapter 3. Sp-attached servers and clustered enterprise servers 103 clusterproven software give you a superior base for your high-availability requirements. The ibm high-availability solutions also offer a simple, but complete process for deciding which components are required and what software is b...

  • Page 128

    104 rs/6000 sp and clustered ibm ^ pseries systems handbook combined with an array of industry software and services, the 7026-6h1 is a cost-efficient growth path to the future. It provides 64-bit scalability via the 64-bit rs64 iii and iv processors packaged in 1-, 2-, 4-, and 6-way processor confi...

  • Page 129

    Chapter 3. Sp-attached servers and clustered enterprise servers 105 description the 7026-6h1 is packaged as a rack-mounted central electronics complex (cec) drawer, cable attached to rack-mounted remote i/o (rio) drawers. The cec and i/o drawers offer redundant power options and redundant cooling. T...

  • Page 130

    106 rs/6000 sp and clustered ibm ^ pseries systems handbook chipkill dimms for the 7026-6h1 provide the self-contained capability to correct real-time, multi-bit dram errors, including complete dram failures. This "raid for memory" technology provides enhanced multi-bit error detection and correctio...

  • Page 131

    Chapter 3. Sp-attached servers and clustered enterprise servers 107 eliminated, thus reducing time delays while keeping network and application processing up and running. The 7026-6h1 can accommodate one processor module in its smaller configuration and grow to two, four, or six processors as requir...

  • Page 132

    108 rs/6000 sp and clustered ibm ^ pseries systems handbook in addition, an hacmp packaged solution with dual 7026-6h1 systems is offered to provide industry-leading high availability. The high-availability solutions provided with the 7026-6h1, can be ordered in a high-availability configuration wit...

  • Page 133

    Chapter 3. Sp-attached servers and clustered enterprise servers 109 support for 32-bit and 64-bit applications aix version 4.3.3 or 5.1 license included 3.3.4 rs/6000 enterprise server 7026 model h80 (7026-h80) the 7026-h80 server is a member of the 64-bit family of symmetric multiprocessing (smp) e...

  • Page 134

    110 rs/6000 sp and clustered ibm ^ pseries systems handbook support for 32-bit and 64-bit applications aix version 4.3.3 license included description the 7026-h80 is packaged as a rack-mounted central electronics complex (cec) drawer, cable attached to rack-mounted remote i/o (rio) drawers. The cec ...

  • Page 135

    Chapter 3. Sp-attached servers and clustered enterprise servers 111 commercial performance in 1-, 2-, 4-, or 6-way enterprise servers using the rs64 iii microprocessor technology. Reliability, availability, and serviceability (ras) providing: – error checking and correcting (ecc) system memory and c...

  • Page 136

    112 rs/6000 sp and clustered ibm ^ pseries systems handbook – keyboard and mouse ports these are standard on the system and are made available without utilizing any of the expansion slots provided. Industry-standard pci subsystem supporting 32-bit and 64-bit adapters. Greater application throughput ...

  • Page 137

    Chapter 3. Sp-attached servers and clustered enterprise servers 113 the 7026-h80's rack-mounted drawer configuration offers flexibility regarding the number of cec and i/o drawers that can be mounted in the rack, providing more compute and i/o power per square foot of valuable floor space. The 7026-...

  • Page 138

    114 rs/6000 sp and clustered ibm ^ pseries systems handbook a serial number protected model conversion to the 7026-m80 system. This model conversion replaces the 7026-h80 cec drawer with an 7026-m80 cec while allowing the 7026-h80 memory, i/o drawers, and adapters to be utilized with the new 7026-m8...

  • Page 139

    Chapter 3. Sp-attached servers and clustered enterprise servers 115 l2 cache 4mb 2mb (1-way) 4mb (2,4-way) 8mb (6-way) 2mb (1-way) 4mb (2,4,6-way) l2 ecc yes yes yes service processor yes yes yes hot-swappable disks external only external only external only dynamic cpu deallocation yes yes yes hot-p...

  • Page 140

    116 rs/6000 sp and clustered ibm ^ pseries systems handbook 3.3.6 cable and pci card for sp cws attachment (f/c 3154) the sp cws attachment (f/c 3154) provides a cable and a pci card necessary to attach an internal connector on the primary i/o drawer planar to the rear bulkhead of the i/o drawer. Th...

  • Page 141

    Chapter 3. Sp-attached servers and clustered enterprise servers 117 table 3-10 shows f/c 3154 characteristics. Table 3-10 f/c 3154 characteristics 3.4 sp-attached server environment the sp-attached servers use rs/6000 feature codes, not rs/6000 sp feature codes. The rs/6000 sp feature codes associat...

  • Page 142

    118 rs/6000 sp and clustered ibm ^ pseries systems handbook because the sp-attached server has both frame and node characteristics, it must have both a frame number and a node number. However, since the sp-attached server does not have full sp frame characteristics, it cannot be considered a standar...

  • Page 143

    Chapter 3. Sp-attached servers and clustered enterprise servers 119 3. In some cases, the number of sp-attached servers you plan to install can exceed the number of available node slots in an sp frame. In this case, you can take advantage of any valid, unused node slots (and the associated switch po...

  • Page 144

    120 rs/6000 sp and clustered ibm ^ pseries systems handbook 3. You must make several cable connections. Since the sp-attached servers may require multiple rs-232 connections, you must use a multi-port asynchronous adapter inside the cws. Do not use the integrated ethernet for your sp lan. 4. Since s...

  • Page 145

    Chapter 3. Sp-attached servers and clustered enterprise servers 121 when you order an rs/6000 sp system attachment adapter as server feature code, you must also order the sp switch cable that connects the adapter to a valid switch port on the sp switch. The sp switch cable must be ordered with the r...

  • Page 146

    122 rs/6000 sp and clustered ibm ^ pseries systems handbook for a complete listing of currently supported sp system pci adapters, see chapter 8, “pci communication adapters” on page 227. With the exception of the rs/6000 sp system attachment adapter (which is supported), if an adapter does not appea...

  • Page 147

    Chapter 3. Sp-attached servers and clustered enterprise servers 123 connections needed for sp-attached servers - m/t 7017 each sp-attached server m/t 7017 must be attached to the control workstation via the following two serial cable features: a custom, ibm-supplied, 15 m (49 ft.) rs-232 cable conne...

  • Page 148

    124 rs/6000 sp and clustered ibm ^ pseries systems handbook you must install a frame-to-frame, 10 m (33 ft.) electrical ground cable which is supplied with the sp system when you order f/c 9122 and f/c 9123. This cable is required in addition to the sp-attached server electrical ground. The frame-to...

  • Page 149

    Chapter 3. Sp-attached servers and clustered enterprise servers 125 connections needed for sp-attached servers - m/t 7026 each sp-attached server m/t 7026 must be attached to the control workstation via a serial cable feature. It is a custom, ibm-supplied, 15 m (49 ft.) rs-232 cable connection from ...

  • Page 150

    126 rs/6000 sp and clustered ibm ^ pseries systems handbook if used, the adapter must be ordered as server feature code and it must be installed in slot #6 of the server primary i/o drawer. Slot #7 must only be used for the internal attachment adapter (f/c 3154). The sp attachment adapter uses an sp...

  • Page 151

    Chapter 3. Sp-attached servers and clustered enterprise servers 127 the rs/6000 sp system configurator session for all variations of sp-attached will include the following, when adding a sp node attachment: f/c 9122 node attachment feature - an order for this feature will generate an order for two 1...

  • Page 152

    128 rs/6000 sp and clustered ibm ^ pseries systems handbook 3.4.4 sp-attached servers connectivity diagram figure 3-4 on page 128 shows sp-attached servers connectivity diagram. Figure 3-4 sp-attached servers connectivity diagram 3.4.5 configuring service director on sp-attached servers service dire...

  • Page 153

    Chapter 3. Sp-attached servers and clustered enterprise servers 129 to configure service director for the sp-attached server you must perform the following: 1. Configure the sp-attached server as a machine type 7017 or 7026 in service director. You must do this manually. 2. Configure service directo...

  • Page 154

    130 rs/6000 sp and clustered ibm ^ pseries systems handbook 2. You must make several cable connections. Since the ces may require multiple rs-232 connections, you must use a multi-port asynchronous adapter inside the cws. Do not use the integrated ethernet for your sp lan. 3. Some of the cables used...

  • Page 155

    Chapter 3. Sp-attached servers and clustered enterprise servers 131 for example, if you attach an existing 7017 enterprise server to a ces system, you might find a scsi adapter installed in slot #9 of the server. However, the scsi adapter in slot 9 is typically connected to a boot device and require...

  • Page 156

    132 rs/6000 sp and clustered ibm ^ pseries systems handbook if you plan to attach an existing 7017 or 7026 server to your system, you must place an sp lan ethernet adapter in the slot en0 position inside the server. Due to the fact that the ethernet adapter in this slot must be configured for pssp c...

  • Page 157

    Chapter 3. Sp-attached servers and clustered enterprise servers 133 connections needed for the ces - m/t 7017 each clustered enterprise server m/t 7017 must be attached to the control workstation via the following two serial cable features: clustered server serial port to control workstation cable (...

  • Page 158

    134 rs/6000 sp and clustered ibm ^ pseries systems handbook the sp attachment adapter uses an sp switch cable to connect to an available switch port on a 9076-555 stand-alone sp switch. There are some clustered enterprise server placement limitations. Placement of the clustered enterprise server is ...

  • Page 159

    Chapter 3. Sp-attached servers and clustered enterprise servers 135 connections needed for the ces - m/t 7026 each clustered enterprise server m/t 7026 must be attached to the control workstation via a serial cable feature. It is a custom 15 m (49 ft.) rs-232 cable – a clustered server control panel...

  • Page 160

    136 rs/6000 sp and clustered ibm ^ pseries systems handbook the sp attachment adapter uses an sp switch cable to connect to an available switch port on a 9076-555 stand-alone sp switch. There are some clustered enterprise server placement limitations. Placement of the clustered enterprise server is ...

  • Page 161

    Chapter 3. Sp-attached servers and clustered enterprise servers 137 figure 3-5 clustered enterprise servers connectivity diagram 3.5.4 configuring service director on ces systems service director, a set of ibm software applications supplied with the enterprise servers m/t 7017 and m/t 7026, monitors...

  • Page 162

    138 rs/6000 sp and clustered ibm ^ pseries systems handbook 2. Configure service director on each clustered enterprise server to forward messages to the local service director server. The modem supplied with the server is not used. 3. Configure service director on the local service director server t...

  • Page 163: Frames

    © copyright ibm corp. 139 chapter 4. Frames this chapter describe the different models of frames used in sp system and in ces. The frames are used to hold nodes and switches in the sp system and to provide nodes interconnectivity in clustered enterprise servers (ces) configuration. There are two fra...

  • Page 164: 4.1  Frames

    140 rs/6000 sp and clustered ibm ^ pseries systems handbook 4.1 frames the sp processor nodes can be mounted in either a tall or short sp frame. The frame spaces that nodes fit into are called drawers. A tall frame has eight drawers, while a short frame has four drawers. Each drawer is further divid...

  • Page 165

    Chapter 4. Frames 141 figure 4-1 sp frame model 550 the product structure of the rs/6000 sp system was simplified with the introduction, on april 21, 1998, of a new frame. The new sp system was simplified with only two models (model 500 and model 550) and two expansion frame features (f/c 1500 and f...

  • Page 166

    142 rs/6000 sp and clustered ibm ^ pseries systems handbook all new designs are completely compatible with all valid sp configurations using older equipment. Also, all new nodes can be installed in any existing sp frame provided that the required power supply upgrades have been implemented in that f...

  • Page 167

    Chapter 4. Frames 143 4.1.2 frame electrical power requirements the electrical power specifications for the old and new power subsystems are summarized in table 4-2. Table 4-2 sp frame electrical power requirements power requirements tall frame short frame legacy pdu legacy sepbu model 550 555 legac...

  • Page 168

    144 rs/6000 sp and clustered ibm ^ pseries systems handbook 4.1.3 sp frame redundant power supply (f/c 3885, 3886) the redundant power supply feature permits feeding ac power to sp system frames from two separate sources to prevent system outages caused by the failure of a single ac power source. Wi...

  • Page 169: 4.2  Tall Frames

    Chapter 4. Frames 145 4.2 tall frames tall frames are available in three variations: tall model frames (model 550, 555) tall expansion frames (f/c 1550) sp switch frames (f/c 2031, 2032) figure 4-3 shows a tall frame’s front and rear views. Figure 4-3 tall frame from front and rear views tall frames...

  • Page 170: 4.3  Model 550 Frames

    146 rs/6000 sp and clustered ibm ^ pseries systems handbook tall frames contain a sepbu power subsystem to accommodate all sp processor nodes. The sepbu is equipped with redundant (n+1) power supply modules; if one module fails, another takes over. The self-regulating sepbus are designed for concurr...

  • Page 171

    Chapter 4. Frames 147 4.3.2 model 550 sp switch-8 configuration this configuration consists of one to eight processor nodes mounted in one required model 550 frame equipped with an eight-port sp switch-8 (f/c 4008). A non-switched expansion frame (f/c 1550) is supported in this configuration only if...

  • Page 172

    148 rs/6000 sp and clustered ibm ^ pseries systems handbook 4.4 model 555 standalone switch for clustered nodes the model 555 is a tall frame with an integral sepbu power supply, one sp switch (f/c 4011), but no processor nodes. This frame provides node interconnectivity for clustered enterprise ser...

  • Page 173

    Chapter 4. Frames 149 figure 4-4 9076 mod 555 with two sp-attached servers and a cws 4.5 expansion frames (f/c 1550) used in model 550, 3bx, 20x, 30x, 40x, and 55h systems, f/c 1550 is a tall frame with eight empty node drawers and a sepbu power supply. All processor nodes and optional switches must...

  • Page 174

    150 rs/6000 sp and clustered ibm ^ pseries systems handbook f/c 1550 is the base offering for the 1.93 m (75.8 inch) sp expansion frame. These frames are equipped with a 10.5 kw three-phase sepbu self regulating power subsystem. All 1.93 m frames have eight empty node drawers for separately purchase...

  • Page 175

    Chapter 4. Frames 151 4.5.1 using f/c 1550 non-switched expansion frames a non-switched expansion frame is defined as a base offering expansion frame equipped with processor nodes only. Some model 550 sp system configurations can be scaled into larger systems using these frames. These sp system conf...

  • Page 176

    152 rs/6000 sp and clustered ibm ^ pseries systems handbook frames containing processor nodes only non-switched expansion frames (configured with processor nodes only) can be added to the model 550 sp frame to take advantage of unused node slots. In these cases, node-to-node data transfers are compl...

  • Page 177

    Chapter 4. Frames 153 4.6 sp switch frames (f/c 2031, 2032) switch frames are base, tall frames with integral sepbu power supply equipped with switches but no processor nodes. A switch frame is required for systems having more than five switches; it interconnects the switches in the model and expans...

  • Page 178: 4.7  Short Frame

    154 rs/6000 sp and clustered ibm ^ pseries systems handbook 4.7 short frame short frames are available in two variations: 1. Short model frames (model 500) 2. Short expansion frames (f/c 1500) figure 4-3 illustrates a short frame from a front view. Table 4-3 short frame front view all short frames (...

  • Page 179: 4.8  Model 500 Frames

    Chapter 4. Frames 155 short frames have four drawers and can house up to eight thin nodes or four wide nodes. All node types can be directly installed as long as configuration rules are not violated. However, note that power3 smp high nodes cannot be installed in these frames due to a depth limitati...

  • Page 180

    156 rs/6000 sp and clustered ibm ^ pseries systems handbook 4.8.2 model 500 switched configuration this configuration consists of one to eight processor nodes connected through a single eight-port sp switch-8 (f/c 4008). These nodes are mounted in one required model 500 frame containing the sp switc...

  • Page 181

    Chapter 4. Frames 157 4.8.5 configurations with processor nodes and a switch non-switched expansion frames can be added to the model 500 sp frame configured with processor nodes and an sps-8 switch to take advantage of unused switch ports. One example of an under-utilized switch is a model 500 frame...

  • Page 182

    158 rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 183: The Switch

    © copyright ibm corp. 159 chapter 5. The switch the switch network is the most efficient network available in sp system and clustered enterprise servers. This chapter describe the two available switch series: the sp switch or sps (f/c 4011 and f/c 4008). The sp switch2 or sps2 (f/c 4012). This chapt...

  • Page 184: 5.1  Sp Switch

    160 rs/6000 sp and clustered ibm ^ pseries systems handbook 5.1 sp switch the design objectives for the sp switch network is low-latency, scalable high-bandwidth, and fault-tolerant communication to support the demands of parallel and client-server applications. The sp switch provides the message pa...

  • Page 185

    Chapter 5. The switch 161 5.2.2 sp switch ports an sp switch port is part of a network device and is connected to other sp switch ports through the sp switch link. The sp switch port includes two ports (input and output ports) for full duplex communication. The following diagram shows the relationsh...

  • Page 186

    162 rs/6000 sp and clustered ibm ^ pseries systems handbook crossbar requests on a least recently served basis. A transmitter will honor no crossbar request if it is already transmitting a packet or if it has packet chunks stored in the central queue. Minimum latency is achieved for packets that use...

  • Page 187

    Chapter 5. The switch 163 5.2.4 sp switch board an sp switch board contains eight sp switch chips that provide connection points for each of the nodes to the sp switch network as well as for each of the sp switch boards to the other sp switch boards. There are 32 sp switch ports in total. Of these, ...

  • Page 188

    164 rs/6000 sp and clustered ibm ^ pseries systems handbook an sp switch adapter is designed for nodes that have both a pci bus and an mx or mx2 bus. Figure 5-5 on page 164 provides a logical diagram of an sp switch adapter. Figure 5-5 logical diagram of an sp switch adapter 5.2.6 sp switch system f...

  • Page 189

    Chapter 5. The switch 165 figure 5-6 sp switch system switch software considerations switch planning involves many issues; node placement, node addressing, and system partitioning are a few that you must consider when planning your switch layout. Make certain that you consult the planning volume 2, ...

  • Page 190

    166 rs/6000 sp and clustered ibm ^ pseries systems handbook table 5-1 shows the aix, pssp, and the switches software level matrix relationship. Table 5-1 aix/pssp/switches level matrix * additional ptfs associated with apar ix56314 are required for this configuration. Later modification levels of ai...

  • Page 191

    Chapter 5. The switch 167 each sp node to the sp switch subsystem. One adapter of the required type must be ordered for each node in a switch-configured sp system. If you are using switch expansion frames, the sp switch subsystem will allow you to scale your sp system up to 128 nodes. When you order...

  • Page 192

    168 rs/6000 sp and clustered ibm ^ pseries systems handbook for upgrades to greater than eight node support, the sp switch-8 is replaced by the 16 port sp switch (f/c 4011). The sp switch uses a similar network topology, proprietary protocol and communication physical layer as the sp switch-8. 5.5 s...

  • Page 193: 5.7  Switch Cabling

    Chapter 5. The switch 169 5.6 special sp switch adapter (f/c 4021 and f/c 8396) some optional sp system equipment requires special adapters in order to connect to the sp switch network. These devices are: sp switch router (machine type 9077) clustered enterprise servers 5.6.1 sp switch router (machi...

  • Page 194

    170 rs/6000 sp and clustered ibm ^ pseries systems handbook for systems that require only single-stage switching, you might want to initially install an sp switch frame if you plan to scale up to a two-stage configuration at a later date. Switch cabling is much simpler in a two-stage switch configur...

  • Page 195

    Chapter 5. The switch 171 5.7.2 switch-to-switch cabling the issues that you must consider when planning switch cable connections between switch-equipped frames in multi-switch sp systems include: 1. Determining the quantity of cables required 2. Determining the length of required cables 3. Minimizi...

  • Page 196

    172 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 5-7 illustrates single-stage configurations from two to five switches. Figure 5-7 single-stage switch cable paths the numbered squares represent switch-equipped frames. The arrows are the cable sets with the quantity of individual ca...

  • Page 197

    Chapter 5. The switch 173 an sp switch frame (f/c 2031 or f/c 2032) is required for systems using more than five switches (known as two-stage switching). Each switch in a two-stage system is cable-connected only to the sp switch frame. Figure 5-8 illustrates an example of a two-stage switch configur...

  • Page 198: 5.8  Sp Switch2

    174 rs/6000 sp and clustered ibm ^ pseries systems handbook use the switch cables listed here for switch-to-switch connections between switch-equipped frames. You can choose from the following lengths: f/c 9305: 5 meter (16 ft.) f/c 9310:10 meter (33 ft.) f/c 9315:15 meter (49 ft.) f/c 9320: 20 mete...

  • Page 199

    Chapter 5. The switch 175 because the sp switch2 design improvements are evolutionary steps on the sp and high performance switches, the sp switch2 is fully compatible with applications written for the older switches. The sp switch2 software support continues to emphasize usage of mpi (message passi...

  • Page 200

    176 rs/6000 sp and clustered ibm ^ pseries systems handbook – this is a new page listing for ethernet, token ring, fddi and css (switch information) information. – css attributes for sp switch2 include: adapter, adapter port, switch port number, network address, netmask, subnet, switch chip, switch ...

  • Page 201

    Chapter 5. The switch 177 the n+1 feature of these switches allows for concurrent replacement of any failed power supplies or cooling fans. The supervisor can also be replaced while the switch is operating. Pssp level 3.2 software is required to use these switches. This section describes some of the...

  • Page 202

    178 rs/6000 sp and clustered ibm ^ pseries systems handbook 5.8.2 sp switch2 adapter (f/c 4025) like the sp switch adapter, the sp switch2 adapter is controlled by node software and on-card microprocessor microcode. The adapter has two ports. Only one port (the upper port) is fully functional, the s...

  • Page 203

    Chapter 5. The switch 179 figure 5-9 sp switch2 adapter hardware structure node bus adapter (nba) the nba chip moves data between the adapter and power3 smp node’s 6xx bus. The nba chip’s function is similar to that provided by the mba, which was supplied on the tb3mx. The primary change is that it ...

  • Page 204

    180 rs/6000 sp and clustered ibm ^ pseries systems handbook the 740 microprocessor the 740 microprocessor is responsible for passing control information between the adapter and the software on the power3 smp node, for encoding or formatting packet headers, for building packet routing information, an...

  • Page 205: Sp Switch Routers 9077

    © copyright ibm corp. 181 chapter 6. Sp switch routers 9077 this chapter presents the ibm 9077 sp switch router within an sp environment only and with the sp switch only. 6.

  • Page 206

    182 rs/6000 sp and clustered ibm ^ pseries systems handbook 6.1 the 9077 sp switch routers the ibm 9077 sp switch router is a licensed version of the ascend grf switched ip router that has been enhanced for direct connection to the sp switch. Ibm remarkets models of the grf that connect to the sp sw...

  • Page 207: 6.2  Overview

    Chapter 6. Sp switch routers 9077 183 6.2 overview a physical dependent node, such as an rs/6000 sp switch router (machine type 9077), may have multiple logical dependent nodes, one for each dependent node adapter it contains. If the sp switch router contains more than one adapter, it can route data...

  • Page 208

    184 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 6-2 sp switch router configuration although you can equip an sp node with a variety of network adapters and use the node to make your network connections, the sp switch router with the switch router adapter and optional network media...

  • Page 209

    Chapter 6. Sp switch routers 9077 185 using multiple switch router adapters in the same sp switch router, you can provide high performance connections between system partitions in a single sp system or between multiple sp systems. A single sp system can also have more than one sp switch router attac...

  • Page 210

    186 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 6-4 rear view of sp switch router model 16s 6.3 installation requirements there are several requirements for hardware and software that must be met before you can place the rs/6000 sp switch router into service with your sp system. T...

  • Page 211

    Chapter 6. Sp switch routers 9077 187 6.3.1 system requirements in addition to the sp switch router, the following requirements must be met before you can place the router into service: you must have at least one sp switch router adapter. You will need a vt100 compatible terminal with an rs-232 cabl...

  • Page 212

    188 rs/6000 sp and clustered ibm ^ pseries systems handbook 6.3.3 network media card requirements each network media card requires one media card slot in the rs/6000 sp switch router. Remember, the network media cards use the same slots as the sp switch router adapters. The following network media c...

  • Page 213: 6.4  Network Interface

    Chapter 6. Sp switch routers 9077 189 6.3.4 software requirements the sp switch router requires an sp system operating with pssp 2.3 (or later) with the appropriate apar level and aix 4.2.1 (or later) on the primary and backup nodes for the sp switch and on the control workstation. If the sp switch ...

  • Page 214

    190 rs/6000 sp and clustered ibm ^ pseries systems handbook both the sp switch cable and the grounding cable are shipped with each sp switch router adapter. The recommended cable for connecting the sp switch router adapter to the sp switch is 10 meters long (f/c 9310). An optional 20 meter cable (f/...

  • Page 215

    Chapter 6. Sp switch routers 9077 191 1. If the control workstation is connected to the sp system through a 10base-2 (thin coax) network, the sp switch router may be connected to the network through a customer supplied 10base-t to 10base-2 hub (or bridge), such as the ibm 8222 ethernet workgroup hub...

  • Page 216

    192 rs/6000 sp and clustered ibm ^ pseries systems handbook – atm – fddi frame to frame grounds are required. 6.4.3 connecting to the sp switch in addition to the control workstation ethernet connection, the rs/6000 sp switch router requires a connection between the sp switch and the sp switch route...

  • Page 217

    Chapter 6. Sp switch routers 9077 193 another example is a two-frame system with one switch. The first frame is fully populated with eight wide nodes. The second frame has three wide nodes in system node positions 17, 19, and 21. The only valid switch ports in this configuration would be those switc...

  • Page 218

    194 rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 219: Control Workstations

    © copyright ibm corp. 195 chapter 7. Control workstations the sp/ces systems require a customer-supplied pseries or rs/6000 system known as a control workstation. The control workstation serves as a point of control for managing, monitoring, and maintaining the sp/ces system frames and individual pr...

  • Page 220: 7.1  Overview

    196 rs/6000 sp and clustered ibm ^ pseries systems handbook 7.1 overview you may consider the control workstation as a server to the sp/ces system applications. The subsystems running on cws are the sp/ces server applications for the sp/ces system nodes. The nodes are clients of the cws server appli...

  • Page 221

    Chapter 7. Control workstations 197 only. You have to order the rs-232 cables for each clustered enterprise server. No bnc ethernet cables supplied for the clustered enterprise servers. On any of these cases, it is the customer’s responsibility to supply the cables for twisted-pair sp ethernet netwo...

  • Page 222

    198 rs/6000 sp and clustered ibm ^ pseries systems handbook 7.2 installation requirements the control workstation and some of its software are not part of the sp package and must be ordered separately. Make sure you have ordered them in time so they will arrive when the rest of your sp does. To coor...

  • Page 223

    Chapter 7. Control workstations 199 we strongly recommend using the 6h1, 6f1, h80, f80, and f50 models only, as cws on sp systems with sp-attached servers and ces systems. The other cws are considered underpowered due to the sp-attachment daemons put on the cws. 7.2.2 system requirements the minimum...

  • Page 224

    200 rs/6000 sp and clustered ibm ^ pseries systems handbook at least 9 gb of disk storage. If the sp is going to use an hacws configuration, you can configure 9 gb of disk storage in the rootvg volume group and 9 gb for the spdata in an external volume group. Because the control workstation is used ...

  • Page 225

    Chapter 7. Control workstations 201 is not part of the sp system. A backup control workstation does not satisfy this requirement. This additional connection is used to access the control workstation from the network when the sp nodes are down. – sp ethernet adapters for connection to the sp ethernet...

  • Page 226

    202 rs/6000 sp and clustered ibm ^ pseries systems handbook table 7-2 serial port adapters for pci / mca control workstations f/c description currently available 8-port pci 2931 8-port asynchronous adapter pci bus eia-232 / rs-422 no 2932 8-port asynchronous adapter pci bus eia-232 / rs-422a no 2943...

  • Page 227

    Chapter 7. Control workstations 203 ethernet adapters for additional pci/mca ethernet ports, select the equipment you need from the feature codes shown in table 7-3. Table 7-3 ethernet adapters for pci/mca control workstations 7.2.4 software requirements the control workstation and some of its softw...

  • Page 228

    204 rs/6000 sp and clustered ibm ^ pseries systems handbook pssp 3.2 (5765-d51) (or later). C for aix 3.6.6 (or later) or visualage c++ professional 4.0 for aix (or later). An aix license is required for the cws. The pssp ordered for the sp/ces system is entitled for use the entire sp/ces complex, w...

  • Page 229

    Chapter 7. Control workstations 205 – event perspective – performance monitor – virtual shared disk perspective – system partitioning aid – sp taskguides – sp resource center – sp perspectives launch pad – perspectives online help – netfinity services manager) 7.2.5 sp control workstation offerings ...

  • Page 230

    206 rs/6000 sp and clustered ibm ^ pseries systems handbook eia-232/422 (pci) 2968 10/100 mbps ethernet adapter 1 3628 p260 color monitor, stealth 1 black 4110 256 mb (2x128mb) sdram dimms 1 4223 ethernet 10base2 transceiver 1 4349 333mhz power3-ii processor card 1 5005 preinstall 1 6159 12gb/24gb 4...

  • Page 231

    Chapter 7. Control workstations 207 example 7-2 7025-f80 control workstation small configuration *** hardware *** product description qty 7025-f80 rs/6000 enterprise server model 1 f80 1.44mb 3.5-in diskette drive 1 integrated ultra2 scsi adapter 1 integrated ethernet adapter 1 2624 32x speed cd-rom...

  • Page 232

    208 rs/6000 sp and clustered ibm ^ pseries systems handbook 0838 aix 4.3 update cd 1 0857 aix 4.3 1 0859 preinstall aix 4.3 server 1 bundle 0860 aix 4.3 bonus pack - non 1 export. 0903 cws diagnostic pre-load 1 1004 cd-rom process charge 1 2924 english 1 3410 cd-rom 1 5005 preinstall 1 5924 aix 4.3 ...

  • Page 233

    Chapter 7. Control workstations 209 5202 2-way rs64 iii 450 mhz 1 processor card, 4mb l2 cache 6159 12gb/24gb 4mm tape drive 1 6553 scsi hot swap 6-pack 1 8700 quiet touch keyboard, stealth 1 black - english (us) 8741 3-button mouse - stealth black 1 9300 language - english (us) 1 9800 power cord - ...

  • Page 234

    210 rs/6000 sp and clustered ibm ^ pseries systems handbook 1.44mb 3.5-in diskette drive 1 integrated ultra2 scsi adapter 1 integrated ethernet adapter 1 2432 point to point cable, 1.1m 1 2624 32x speed cd-rom 1 2830 power gxt130p graphics adapter 1 (pci) 2944 128-port async controller (pci) 1 2968 ...

  • Page 235

    Chapter 7. Control workstations 211 0857 aix 4.3 1 0859 preinstall aix 4.3 server 1 bundle 0860 aix 4.3 bonus pack - non 1 export. 0903 cws diagnostic pre-load 1 1004 cd-rom process charge 1 2924 english 1 3410 cd-rom 1 5005 preinstall 1 5924 aix 4.3 pii - english 1 9001 asset registration 1 7026-6h...

  • Page 236

    212 rs/6000 sp and clustered ibm ^ pseries systems handbook 6324 primary i/o drawer, 5 eia 1 6540 ipl disk mounting hardware, 1 cables, terminator 8700 quiet touch keyboard, stealth 1 black - english (us) 8741 3-button mouse - stealth black 1 9172 ac power specify 1 9300 language - english (us) 1 98...

  • Page 237

    Chapter 7. Control workstations 213 bundle 0860 aix 4.3 bonus pack - non 1 export. 0903 cws diagnostic pre-load 1 1004 cd-rom process charge 1 2924 english 1 3410 cd-rom 1 5005 preinstall 1 5924 aix 4.3 pii - english 1 9001 asset registration 1 7026-6h1 control workstation medium configuration this ...

  • Page 238

    214 rs/6000 sp and clustered ibm ^ pseries systems handbook adapter 6324 primary i/o drawer, 5 eia 1 6540 ipl disk mounting hardware, 1 cables, terminator 8700 quiet touch keyboard, stealth 1 black - english (us) 8741 3-button mouse - stealth black 1 9172 ac power specify 1 9300 language - english (...

  • Page 239

    Chapter 7. Control workstations 215 *** software *** product description qty 5765-c34 aix version 4 1 4061 aix 4.3 1 5005 preinstall 1 9001 asset registration 1 5692-aix system software 1 0598 aix welcome center 1 0838 aix 4.3 update cd 1 0857 aix 4.3 1 0859 preinstall aix 4.3 server 1 bundle 0860 a...

  • Page 240

    216 rs/6000 sp and clustered ibm ^ pseries systems handbook 3153 18.2 gb 10k rpm 1" ultra3 scsi 2 16-bit disk 3628 p260 color monitor, stealth 1 black 3752 service package 1 4110 256 mb (2x128mb) sdram dimms 1 5005 preinstall 1 5211 1-way rs64 iv 600 mhz processor 1 card, 2mb l2 cache 6159 12gb/24gb...

  • Page 241

    Chapter 7. Control workstations 217 5924 aix 4.3 pii - english 1 9001 asset registration 1 7025-6f1 control workstation medium configuration this configuration is recommended for sp systems that include more than 30 nodes and for sp/ces systems that include no more than 16 enterprise servers. Exampl...

  • Page 242

    218 rs/6000 sp and clustered ibm ^ pseries systems handbook *** software *** product description qty 5765-c34 aix version 4 1 4061 aix 4.3 1 5005 preinstall 1 9001 asset registration 1 5765-e26 visualage c++ professional v5 1 0006 otc 1 3483 single pallet delivery 1 5819 cd-rom - english 1 9001 asse...

  • Page 243

    Chapter 7. Control workstations 219 example 7-9 7025-6f1 control workstation large configuration *** hardware *** product description qty 7025-6f1 pseries 620 1 1.44mb 3.5-in diskette drive 1 integrated scsi-2 f/w adapter 1 integrated ultra2 scsi adapter 1 integrated ethernet adapter 1 2624 32x spee...

  • Page 244

    220 rs/6000 sp and clustered ibm ^ pseries systems handbook 5765-c34 aix version 4 1 4061 aix 4.3 1 5005 preinstall 1 9001 asset registration 1 5765-e26 visualage c++ professional v5 1 0006 otc 1 3483 single pallet delivery 1 5819 cd-rom - english 1 9001 asset registration 1 5692-aix system software...

  • Page 245

    Chapter 7. Control workstations 221 control workstation. With a hacws, your sp system will have the added security of a backup control workstation. Also, hacws allows your control workstation to be powered down for maintenance or updating without affecting the entire sp system. The design of the hac...

  • Page 246

    222 rs/6000 sp and clustered ibm ^ pseries systems handbook . Figure 7-2 high availability control workstation the frame supervisor tty network is modified to add a standby link. The second control workstation is the backup figure 7-2 shows a logical view of a high availability control workstation. ...

  • Page 247

    Chapter 7. Control workstations 223 the primary and backup control workstations are also connected on a private point-to-point network and a serial tty link or target mode scsi. The backup control workstation assumes the ip address, ip aliases, and hardware address of the primary control workstation...

  • Page 248

    224 rs/6000 sp and clustered ibm ^ pseries systems handbook two licenses for ibm c for aix version 3.6.6 or later or the batch c and c++ version 3.6.6 or later compilers and runtime libraries of visualage c++ professional version 4.0 for aix or later. If the compiler’s license server is on the contr...

  • Page 249

    Chapter 7. Control workstations 225 7.4 service director on the cws the service director software is provided with the sp/ces system. If you plan to use it, consider whether you want to install it on the cws or somewhere else - any workstation connected to the lan. Service director monitors the heal...

  • Page 250

    226 rs/6000 sp and clustered ibm ^ pseries systems handbook note: refer to appendix c, “sp system upgrades” on page 407 for more details about the service director and the electronic service agent..

  • Page 251: Pci Communication

    © copyright ibm corp. 227 chapter 8. Pci communication adapters this chapter provides information for communication adapters and communication adapter extensions supported on the sp. 8.

  • Page 252

    228 rs/6000 sp and clustered ibm ^ pseries systems handbook 8.1 pci communication adapters supported on the sp the following is a list of all supported communication adapters and most adapter extensions supported on the sp. Entries are ordered by feature code (f/c). Table 8-1 shows all currently sup...

  • Page 253

    Chapter 8. Pci communication adapters 229 2947 9-r ibm artic960h x 4-port selectable pci 8m long 4.2.1/4.3.2 * 2.4* yes 2962 9-l 2-port multiprotoco l x.25 0 short 4.2.1* 2.4* yes 2963 9-j turbowa ys 155 utp atm 8m short 4.2.1* 2.4* yes 2968 9-p 10/100 ethernet tx pci 16m short 4.2.1* 2.4* yes 2969 ...

  • Page 254

    230 rs/6000 sp and clustered ibm ^ pseries systems handbook 4951 9-z ibm 4-port 10/100 base-tx ethernet pci 16m short no 4958 6-h ibm pci cryptograph ic coprocesso r 128m long no 4959 9-y 4/16 auto lan streamer full duplex token ring 8m short 4.2.1/4.3.3 * 2.4* yes 6204 4-u ibm pci universal differe...

  • Page 255

    Chapter 8. Pci communication adapters 231 * this version or later 8.2 descriptions of adapter supported on the sp the following is a brief description of all pci adapters supported on the sp. Mca (micro channel) and switch adapters are not discussed in this chapter. A list of mca adapters supported ...

  • Page 256

    232 rs/6000 sp and clustered ibm ^ pseries systems handbook network adapter software provides a pass-through capability for other ethernet protocols, which can eliminate the need for a separate ethernet adapter. The network adapter supports on board snmp for network management. 8.2.2 scsi-2 differen...

  • Page 257

    Chapter 8. Pci communication adapters 233 feature components this feature order provides the following: adapter card diagnostic wrap plugs installation instructions customer components you must supply the following components for this feature: short-wave cabling dual sc connector 50/125 micron multi...

  • Page 258

    234 rs/6000 sp and clustered ibm ^ pseries systems handbook 8.2.6 ibm network terminal accelerator adapter (f/c 2402 & 2403) the ibm network terminal accelerator adapter (f/c 2402) is an ethernet adapter that accelerates network performance by off-loading the telnet and rlogin daemons, tcp/ip protoc...

  • Page 259

    Chapter 8. Pci communication adapters 235 they are all pci fiber optic fddi adapters that are compatible with the fddi-ansi x3t12 specifications and fddi standard series. Ibm no longer sells fddi adapters, but it does support them. The following are some common characteristics of the adapters. Featu...

  • Page 260

    236 rs/6000 sp and clustered ibm ^ pseries systems handbook unique features of fddi-up das the adapter provides single attachment to a fddi concentrator (or point to point) using category 5 unshielded twisted pair (utp) cabling (not supplied with the adapter). Customer supplied components you must s...

  • Page 261

    Chapter 8. Pci communication adapters 237 escon cabling requires 62.5/125 multimode fiber cable with escon duplex connectors on both ends. Aix program feature, escon control unit lpp. 8.2.12 pci auto lanstreamer token ring adapter (f/c 2920) the pci auto lanstreamer token ring adapter (f/c 2920) is ...

  • Page 262

    238 rs/6000 sp and clustered ibm ^ pseries systems handbook this adapter provides a single db-78 output that connects directly to the 8-port db-25 connector box. All eight ports are software programmable to support either protocol at baud rates up to 230 k. The full set of modem control lines for as...

  • Page 263

    Chapter 8. Pci communication adapters 239 two 2.4 mbps synchronous channels link the adapter to a maximum of eight 16-port remote async nodes (rans). Each synchronous channel uses an hd-15 female connector to link up to four rans. Each ran supports either eia-232 or rs-422 connections (16 per ran), ...

  • Page 264

    240 rs/6000 sp and clustered ibm ^ pseries systems handbook 8.2.15 turboways 622 mbps pci mmf atm adapter (f/c 2646) the ibm turboways 622 mbps pci mmf atm adapter (f/c 2646) is a 64-bit, universal pci adapter. This adapter provides direct access to the atm network at a dedicated 622 mbps full-duple...

  • Page 265

    Chapter 8. Pci communication adapters 241 – eia-530 – rs-449 – x.21 – v.35 support for sdlc and x.25 full-duplex, synchronous protocols featured components this feature order provides the following: one artic960hx adapter (f/c 2947) a connecting cable (required). The following are available from ibm...

  • Page 266

    242 rs/6000 sp and clustered ibm ^ pseries systems handbook feature characteristics this feature has the following characteristics: 32-bit bus master pci 2.1 adapter provides two, 36-pin high density (male) ports provides four interface types, eia 232d/v.24, v.35, v.36/eia 449, and x.21 simultaneous...

  • Page 267

    Chapter 8. Pci communication adapters 243 32-bit bus master pci 2.1 adapter external rj45 connector provides signaling channel setup provides virtual connection setup and tear down supports point-to-point and point-to-multipoint switching supports virtual circuits (maximum 1024) supports classical i...

  • Page 268

    244 rs/6000 sp and clustered ibm ^ pseries systems handbook installation instructions customer supplied components you must supply the following components for this feature: network equipment, such as a hub or switch, required to attach to 10baset ethernet lans. All ethernet cables. For 100base-tx c...

  • Page 269

    Chapter 8. Pci communication adapters 245 8.2.21 10/100/1000 base-t ethernet pci adapter (fc 2975) 10/100/1000 base-t ethernet pci adapter (f/c 2975) is a full duplex gigabit ethernet adapter designed with highly integrated components to optimize cost and performance. The adapter interfaces to the s...

  • Page 270

    246 rs/6000 sp and clustered ibm ^ pseries systems handbook supports half duplex operations over 10base2 networks via the bnc connector supports both full and half duplex operation over 10base-t networks via the rj-45 connector feature components this feature order provides the following: adapter ca...

  • Page 271

    Chapter 8. Pci communication adapters 247 supports ethernet lan emulation and token ring supports atm snmp customer supplied components you must supply the following components with this feature: plenum rated 62.5/125 multimode fiber cables terminated with an sc connector an atm switch 8.2.25 four-p...

  • Page 272

    248 rs/6000 sp and clustered ibm ^ pseries systems handbook diskette with adapter device driver installation instructions. Customer supplied components you must supply the following components for this feature: network equipment such as a hub or switch. All ethernet cables. For 100base-tx connection...

  • Page 273

    Chapter 8. Pci communication adapters 249 8.2.27 high-speed token-ring pci adapter (f/c 4959) the high-speed token-ring pci adapter (f/c 4959) is a pci 16/4 token ring adapter that is compatible with ieee 802.5 specifications. The adapter has two external connections: rj-45 to attach to utp cabling ...

  • Page 274

    250 rs/6000 sp and clustered ibm ^ pseries systems handbook network equipment such as a mau and/or switching hub to connect the token-ring network utp or stp cable to attach to the token-ring network 8.2.28 scsi-2 ultra/wide de pci adapter (f/c 6204) the pci scsi-2 ultra/wide differential adapter (f...

  • Page 275

    Chapter 8. Pci communication adapters 251 feature characteristics this feature has the following characteristics: two ultra2/lvd scsi buses pci bus specification 2.1 – fits in pci full-size slots – supports both 5.0 and 3.3 volt signaling – supports pci data streaming – two independent dma channels ...

  • Page 276

    252 rs/6000 sp and clustered ibm ^ pseries systems handbook feature characteristics this feature has the following characteristics: 32-bit bus master pci 2.1 adapter supports attachment of internal and external single ended 8-bit and 16-bit scsi or ultra scsi devices – external connections on j2 wit...

  • Page 277

    Chapter 8. Pci communication adapters 253 supports attachment of external 8-bit or 16-bit scsi devices on the j2 port using a 68 pin scsi-3 standard connector. Customer components you must supply the following component for this feature: cabling (available through ibm) 8.2.32 scsi-2 f/w single-ended...

  • Page 278

    254 rs/6000 sp and clustered ibm ^ pseries systems handbook 8.2.33 scsi-2 f/w differential adapter (f/c 6209) the pci scsi-2 fast/wide differential adapter (f/c 6209) provides a differential scsi-2 fast/wide interface that can burst data between devices on the scsi bus at 20 mbps. It conforms to scs...

  • Page 279

    Chapter 8. Pci communication adapters 255 8.2.35 ssa fast-write cache module (f/c 6222) the ssa fast-write cache module (f/c 6222) is an optional 4 mb fast-write module that plugs into the pci ssa raid 5 adapter (f/c 6215). The f/c 6222 cache option uses non-volatile ram, which has over seven years ...

  • Page 280

    256 rs/6000 sp and clustered ibm ^ pseries systems handbook single speed supported is 1.0625 gbps (wire-speed) which corresponds to approximately 100 mbps. The supported media includes 50 and 62.5 micron multimode fiber. The supported transmitter is short-wave laser with open fiber control. Feature ...

  • Page 281

    Chapter 8. Pci communication adapters 257 the adapter utilizes the ssa enhanced loop (el) architecture, which offers a choice for ssa ha data protection, disk mirroring for the best performance, or multi-initiator raid for the lowest total system cost. Ssa el adapter architecture enables rs/6000 sp ...

  • Page 282

    258 rs/6000 sp and clustered ibm ^ pseries systems handbook feature components this feature order provides the following: adapter card cd-rom with adapter device driver installation instructions customer supplied components no customer components required. 8.2.39 artic960rxd quad digital trunk adapt...

  • Page 283

    Chapter 8. Pci communication adapters 259 – f/c 2876: artic960rxd quad dta, e1, 75 ohm unbalanced-ungrounded, 1.8 m 4-port cable – f/c 2877: artic960rxd quad dta, h.100, 4-drop cable 8.2.40 artic960rxf digital trunk resource adapter (f/c 6311) the artic960rxf digital trunk resource adapter provides ...

  • Page 284

    260 rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 285: Software Support

    © copyright ibm corp. 261 chapter 9. Software support this chapter describes the software available for sp systems. It covers the following software: parallel system support programs (pssp) general parallel file system (gpfs) loadleveler scientific subroutine library (essl, pessl & osl) parallel env...

  • Page 286

    262 rs/6000 sp and clustered ibm ^ pseries systems handbook 9.1 parallel system support programs (pssp) program number- 5765 - d51 the pssp software is a comprehensive suite of applications to manage an sp systems as a full-function parallel processing system. It provides a single point of control f...

  • Page 287

    Chapter 9. Software support 263 pssp comes with login control for blocking unauthorized user or group access to a specific sp node or a set of nodes. It offers consolidated accounting for centralizing records at the node level (for tracking use by wall clock time rather than processor time) and for ...

  • Page 288

    264 rs/6000 sp and clustered ibm ^ pseries systems handbook 9.1.3 system monitoring and control pssp provides system management tools that enable the system administrator to manage the sp system. Pssp allows authorized users to monitor and manipulate sp hardware variables at node or frame levels. It...

  • Page 289

    Chapter 9. Software support 265 9.1.6 performance toolbox parallel extensions for aix (ptpe) as its name implies, ptpe extends the capabilities of performance toolbox (ptx) or aix by adding monitoring functions and statistics specific to the sp. Ptpe provides sp performance statistics from software,...

  • Page 290

    266 rs/6000 sp and clustered ibm ^ pseries systems handbook this isolation is useful for correcting an error condition or installing new hardware and software without impacting production. High availability control workstation (hacws) connects two rs/6000 workstations (with hacmp installed) to an sp...

  • Page 291

    Chapter 9. Software support 267 9.1.12 software requirements table 9-1 shows the software requirements for pssp 3.2. Table 9-1 pssp 3.2 software requirements 9.2 general parallel file system (gpfs) program number - 5765 -b95 ibm general parallel file system (gpfs) is a standards-based, parallel file...

  • Page 292

    268 rs/6000 sp and clustered ibm ^ pseries systems handbook 9.2.1 administration and operation gpfs provides functions that simplify multinode administration and can be performed from any node in the sp configuration. These functions are based on, and are in addition to, the aix administrative comma...

  • Page 293

    Chapter 9. Software support 269 9.2.4 migration upgrading to a new release of gpfs can be tested on a system currently running gpfs. This eases migration by allowing the testing of a new level of code without inhibiting the production gpfs application. 9.2.5 new in gpfs 1.4 gpfs 1.4 was announced to...

  • Page 294: 9.3  Loadleveler

    270 rs/6000 sp and clustered ibm ^ pseries systems handbook 9.3 loadleveler program number - 5765-d61 ibm loadleveler is a job management system that allows users to run more jobs in less time by matching the jobs’ processing needs with the available resources. Loadleveler schedules jobs, and provid...

  • Page 295

    Chapter 9. Software support 271 9.3.2 capabilities parallel processing for parallel jobs, loadleveler interfaces with parallel programming software to obtain the multiple sp nodes required for the job's parallel tasks. Individual control at the node level, users can specify to loadleveler when their...

  • Page 296

    272 rs/6000 sp and clustered ibm ^ pseries systems handbook 9.3.3 new in loadleveler 2.2 loadleveler 2.2 was announced to the public on july 18, 2000 and includes the following enhancements: improved security: by exploiting the pssp security services within the distributed computing environment (dce...

  • Page 297

    Chapter 9. Software support 273 program number for pessl - 5765-c41 program number for oslp - 5765-392 the engineering and scientific subroutine library (essl) family of products is a state-of-the-art collection of mathematical subroutines that provide a wide range of high-performance mathematical f...

  • Page 298

    274 rs/6000 sp and clustered ibm ^ pseries systems handbook 9.4.1 operations essl’s mathematical functions: basic linear algebra subroutines (blas) linear algebraic equations eigensystem analysis fourier transforms the essl products are compatible with public domain subroutine libraries, such as bas...

  • Page 299

    Chapter 9. Software support 275 nuclear engineering quantitative analysis electronic circuit design for more detailed information on essl, pessl and oslp refer to rs/6000 scalable power parallel system: scientific and technical computing overview, sg24-4541-01. 9.4.2 new in essl 3.2 the essl librari...

  • Page 300

    276 rs/6000 sp and clustered ibm ^ pseries systems handbook reduce a real symmetric positive definite generalized eigenproblem to standard form 9.4.4 new in oslp 1.3 the following are new modules and routines found in oslp 1.3: a pair of new routines provided for transforming a general integer probl...

  • Page 301

    Chapter 9. Software support 277 9.5 parallel environment (pe) program number 5765-d93 ibm parallel environment for aix is a high-function development and execution environment for parallel applications using the sp system. It is a complete solution for enterprises that need to develop, debug, analyz...

  • Page 302

    278 rs/6000 sp and clustered ibm ^ pseries systems handbook the advantages of parallel environment include: it exploits threads and thread-safe mpi message passing on all nodes, including symmetric multiprocessors (smps). It supports low-level application programming interface (lapi) programs. Pe ma...

  • Page 303

    Chapter 9. Software support 279 support for mpi shared memory message passing on symmetric multiprocessor (smp) nodes, for the internet protocol (ip) library and for the user space (us) library. Instrumentation probes that the analysis tool defines and inserts into the target application to collect ...

  • Page 304

    280 rs/6000 sp and clustered ibm ^ pseries systems handbook 9.5.3 software requirements table 9-5 shows the software requirements of pe 3.1 table 9-5 software requirements for pe 3.1 aix version pssp version other software 4.3.3.10 or higher 3.2 or higher xl fortran, v7, or higher, if fortran progra...

  • Page 305: Aix (Hacmp)

    Chapter 9. Software support 281 9.6 ibm high availability cluster multi-processing for aix (hacmp) program number - 5765-e54 hacmp aims to eliminate any single point of failure, be it disks, network adapters, power supply, processors, or other components. Hacmp provides a highly-available environmen...

  • Page 306

    282 rs/6000 sp and clustered ibm ^ pseries systems handbook network addresses application servers the takeover relationships among cluster nodes determine which cluster nodes control a resource group and which cluster nodes take over control of the resource group when the original node relinquishes ...

  • Page 307

    Chapter 9. Software support 283 the cluster configuration can be expanded or modified without service disruption through dynamic reconfiguration options. Version compatibility and dynamic reconfiguration eliminate the most common causes of scheduled cluster down time. Concurrent processing with the ...

  • Page 308

    284 rs/6000 sp and clustered ibm ^ pseries systems handbook has and crm support clusters of up to eight systems. Es supports clusters of up to 32 sp nodes or rs/6000 systems. Escrm supports clusters of up to 8 sp nodes of rs/6000 systems. 9.6.4 software requirements table 9-6 on page 284shows the so...

  • Page 309: Clustering Configurations

    © copyright ibm corp. 285 chapter 10. Clustering configurations in this chapter you find information on how to build your complex cluster. Section 10.1, “configurations rules” on page 286 shows all the rules you need to know before you design your cluster system; frame, switch, and node numbering ru...

  • Page 310: 10.1  Configurations Rules

    286 rs/6000 sp and clustered ibm ^ pseries systems handbook 10.1 configurations rules the basic components that comprise an sp/ces system are nodes/servers, frames, and switches. The sp architecture can generally support up to 128 nodes (512 with special request). Within these 128 nodes, you can hav...

  • Page 311

    Chapter 10. Clustering configurations 287 any sp-attached server is considered a node (f/c 9122) and a frame (f/c 9123) in an sp system. Any clustered enterprise server is considered a node and a frame in a ces system. Any sp-attached server and any clustered enterprise server can be developed into ...

  • Page 312

    288 rs/6000 sp and clustered ibm ^ pseries systems handbook the pci high nodes are supported on model 550 tall frames and expansion frames (f/c 1550) only. In addition, they are supported in switchless, sp-switch (f/c 4011) or sp-switch2 (f/c 4012) configurations only. The expansion i/o units for pc...

  • Page 313

    Chapter 10. Clustering configurations 289 figure 10-1 examples on node configuration note that for mca nodes, a pair of nodes is still required for each sp node drawer. Node configuration rule 4: sp-attached servers are supported on sp model 550 only. Sp-attached servers are supported on model 550 o...

  • Page 314

    290 rs/6000 sp and clustered ibm ^ pseries systems handbook placement rules for frames frame configuration rule 1: all the frames in an sp system must be either tall frames or short frames; a mixture of both is not supported. The sp switch frame is classified as a tall frame and therefore can be use...

  • Page 315

    Chapter 10. Clustering configurations 291 frame configuration rule 3: the short frame supports only the sp switch-8 (f/c 4008) and only one switch can be configured in an sp system composed of short frames. This configuration mounts one to eight nodes through a single sp eight port switch and the no...

  • Page 316

    292 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 10-3 maximum short frame configurations with sp eight port switch frame configuration - rule 4: when a tall frame is installed with an eight port switch, a maximum of eight nodes and one frame is supported. No expansion frame can be ...

  • Page 317

    Chapter 10. Clustering configurations 293 when using the sp switch-8 (f/c 4008), the tall frame can only mount one to eight nodes. A minimum configuration is formed by one tall model frame equipped with an sp switch-8 and a single pci thin node, one wide node, one mca high node (a pci high node is n...

  • Page 318

    294 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 10-5 examples on single sp switch-16 configurations s p s w itch fra m e 1 w ide n ode w ide n ode w ide no de w ide no de t hin thin t hin thin thin thin t hin t hin s p s wi tc h fram e 1 fram e 2 w id e n o d e w ide n o d e w id ...

  • Page 319

    Chapter 10. Clustering configurations 295 figure 10-6 other examples on single sp switch-16 configurations frame configuration - rule 6: in configuration (a) of figure 10-5 on page 294, four wide nodes and eight thin nodes are mounted in a tall model frame equipped with an sp switch. There are four ...

  • Page 320

    296 rs/6000 sp and clustered ibm ^ pseries systems handbook in configuration (c) of figure 10-6 on page 295, there are eight wide nodes mounted in a tall model frame equipped with an sp switch and four high nodes mounted in a non-switched expansion frame (frame 2). The second non-switched expansion ...

  • Page 321

    Chapter 10. Clustering configurations 297 an sp intermediate switch frame is a base, tall frame equipped with four sp switches but no processor nodes. The switch frame is required for systems using more than five sp switches to interconnect all the switches in the system. A switch frame supports sys...

  • Page 322

    298 rs/6000 sp and clustered ibm ^ pseries systems handbook placement rules for switches switch configuration rule 1: all eight port switches do not support pci high nodes, i/o expansion units, or sp-attached servers. As for the limited scalability of the eight port switches, they support up to a ma...

  • Page 323

    Chapter 10. Clustering configurations 299 sp switch2 (f/c 4012) provides a higher throughput and performance than sp switch (f/c 4011) and it supports pci high nodes and model 550 tall frames only. Table 10-1 shows the current compatibility of the switches and nodes; note that the switches in an sp/...

  • Page 324

    300 rs/6000 sp and clustered ibm ^ pseries systems handbook each sp-attached server and clustered enterprise server occupies one frame number. If using pssp version 3.1.1 or before, the sp-attached server cannot be the first frame. It also cannot be inserted between a switched frame and any non-swit...

  • Page 325

    Chapter 10. Clustering configurations 301 each type (size) of node occupies one slot or a consecutive sequence of slots. For each node, there is an integer n such that a thin node (or expansion i/o unit) occupies slot n, a wide node occupies slots n and n+1, and a high node occupies n, n+1, n+2 and ...

  • Page 326

    302 rs/6000 sp and clustered ibm ^ pseries systems handbook switch boards are numbered sequentially starting with 1 from the frame with the lowest frame number to that with the highest frame number. Each full switch board contains a range of 16 port numbers that can be assigned. These ranges are als...

  • Page 327

    Chapter 10. Clustering configurations 303 figure 10-10 switch port numbering on an sp switch 0 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 s p s w itch f ra m e n 1 (1 ) 0 2 4 6 8 1 0 1 2 1 4 1 3 5 7 1 1 1 5 s p s w itc h f r a m e n f ra m e n + 1 9 1 3 (2 ) 0 2 4 6 8 10 12 14 1 5 sp switch frame n fra...

  • Page 328

    304 rs/6000 sp and clustered ibm ^ pseries systems handbook sp-attached servers can connect to any available switch port of a switched sp frame, regardless of the frame number assigned to the sp-attached server. For example, in figure 10-11 on page 304, an sp-attached server can be assigned with swi...

  • Page 329

    Chapter 10. Clustering configurations 305 ces system in non-switched and switched environments 10.2.1 sp system in non-switched and switched environments we may consider an sp system in non-switched or switched environments. Sp system in non-switched environment figure 10-12 on page 305 shows an exa...

  • Page 330

    306 rs/6000 sp and clustered ibm ^ pseries systems handbook product description qty 9076-550 sp tall frame model 1 9123 rs/6000 server frame attachment 1 2056 375mhz thin node 16 2909 9.1 gb ultra-scsi disk drive 16 pair 4098 memory expansion feature 32 4110 256 mb (2x128mb) sdram dimms 16 4444 375 ...

  • Page 331

    Chapter 10. Clustering configurations 307 4444 375 mhz power3-ii processor, 1 2-way 9432 aix 4.3 with pssp v3.2 1 *** sp software *** product description qty 5765-c34 aix version 4 1 4061 aix 4.3, per node 16 9001 asset registration 1 5765-d51 parallel system support program 1 v3.2 4011 sp nodes 16 ...

  • Page 332

    308 rs/6000 sp and clustered ibm ^ pseries systems handbook 170 cws default entry *** 170 cws default entry hardware *** product description qty 7044-170 rs/6000 44p model 170 1 1.44mb 3.5-in diskette drive 1 integrated ultra scsi adapter 1 integrated external ultra2 scsi 1 port integrated ethernet ...

  • Page 333

    Chapter 10. Clustering configurations 309 0857 aix 4.3 1 0859 preinstall aix 4.3 server 1 bundle 0860 aix 4.3 bonus pack - non 1 export. 0903 cws diagnostic pre-load 1 1004 cd-rom process charge 1 2924 english 1 3410 cd-rom 1 5005 preinstall 1 5924 aix 4.3 pii - english 1 9001 asset registration 1 s...

  • Page 334

    310 rs/6000 sp and clustered ibm ^ pseries systems handbook example 10-2 shows a sample feature codes configuration for the switched sp environment considered. Example 10-2 sample configuration for switched sp environment sp *** sp hardware *** product description qty 9076-550 sp tall frame model 1 ...

  • Page 335

    Chapter 10. Clustering configurations 311 4 2057 wide 3 2057 wide 2 2057 wide 1 2057 wide ---------------------------------------------------------------------- frame: 2 feature: drawer left right 8 2057 wide 7 2057 wide 6 2057 wide 5 2057 wide 4 2057 wide 3 2057 wide 2 2057 wide 1 2057 wide -------...

  • Page 336

    312 rs/6000 sp and clustered ibm ^ pseries systems handbook 5765-d51 parallel system support program 1 v3.2 4011 sp nodes 16 5812 rs/6000 sp resource center 1 (cd-rom) 5813 cd-rom 1 9001 asset registration 1 5765-d61 loadleveler v2.2 1 2005 sp nodes 16 5000 cd-rom 1 9001 asset registration 1 5765-e3...

  • Page 337

    Chapter 10. Clustering configurations 313 1.44mb 3.5-in diskette drive 1 integrated ultra2 scsi adapter 1 integrated ethernet adapter 1 2624 32x speed cd-rom 1 2830 power gxt130p graphics adapter 1 (pci) 2943 8-port async adapter, 1 eia-232/422 (pci) 2968 10/100 mbps ethernet pci 1 adapter 2985 ethe...

  • Page 338

    314 rs/6000 sp and clustered ibm ^ pseries systems handbook 2924 english 1 3410 cd-rom 1 5005 preinstall 1 5924 aix 4.3 pii - english 1 9001 asset registration 1 10.2.2 sp system with sp-attached servers figure 10-14 shows an example of an sp system with sp-attached servers. It is a switched sp envi...

  • Page 339

    Chapter 10. Clustering configurations 315 figure 10-14 switched sp environment with sp-attached servers example 10-3 shows a sample feature codes configuration for the switched sp environment with sp-attached servers considered. Example 10-3 sample configuration for switched sp environment with sp-a...

  • Page 340

    316 rs/6000 sp and clustered ibm ^ pseries systems handbook 2-way 9223 node attachment - t/p ethernet 5 9302 2 meter sp switch to node cable 8 9310 10 meter sp switch/switch or 5 switch/node cable 9432 aix 4.3 with pssp v3.2 8 9920 low voltage power (200-240v, 1 50/60hz) 9930 pci node enclosure 8 be...

  • Page 341

    Chapter 10. Clustering configurations 317 3 2 9122 1 node rs/6000 server attac ---------------------------------------------------------------------- frame: 3 feature: 9123 drawer left right 8 7 6 5 4 3 2 9122 1 node rs/6000 server attac --------------------------------------------------------------...

  • Page 342

    318 rs/6000 sp and clustered ibm ^ pseries systems handbook frame: 5 feature: 9123 drawer left right 8 7 6 5 4 3 2 9122 1 node rs/6000 server attac ---------------------------------------------------------------------- frame: 6 feature: 9123 drawer left right 8 7 6 5 4 3 2 9122 1 node rs/6000 server...

  • Page 343

    Chapter 10. Clustering configurations 319 pair 4023 sp switch mx2 adapter 1 4098 memory expansion feature 2 4110 256 mb (2x128mb) sdram dimms 1 4444 375 mhz power3-ii processor, 1 2-way 9432 aix 4.3 with pssp v3.2 1 *** sp software *** product description qty 5765-c34 aix version 4 1 4061 aix 4.3, p...

  • Page 344

    320 rs/6000 sp and clustered ibm ^ pseries systems handbook m/t 7017 (quantity: 2) *** m/t 7017 hardware *** product description qty 7017-s85 pseries 680 1 1.44mb 3.5-in diskette drive 1 this machine will function as 1 an sp node 2447 16-bit pci scsi se adapter to 6 1 scsi bays cable 2624 32x speed ...

  • Page 345

    Chapter 10. Clustering configurations 321 5765-c34 aix version 4 1 4061 aix 4.3 1 5005 preinstall 1 9001 asset registration 1 5765-d51 parallel system support program 1 v3.2 4012 external attached servers (sxx 1 models) 5812 rs/6000 sp resource center 1 (cd-rom) 5813 cd-rom 1 9001 asset registration...

  • Page 346

    322 rs/6000 sp and clustered ibm ^ pseries systems handbook 2624 32x speed cd-rom 1 2968 10/100 mbps ethernet pci 1 adapter 3027 9.1 gb 10k rpm ultra scsi disk 1 drive 3142 remote i/o cable - 3m 2 3154 cable and pci card for sp 1 control workstation attachment 4110 256 mb (2x128mb) sdram dimms 1 500...

  • Page 347

    Chapter 10. Clustering configurations 323 9001 asset registration 1 5692-aix system software 1 0598 aix welcome center 1 0838 aix 4.3 update cd 1 0857 aix 4.3 1 0859 preinstall aix 4.3 server 1 bundle 0860 aix 4.3 bonus pack - non 1 export. 0917 performance aide v3.0 1 1004 cd-rom process charge 1 2...

  • Page 348

    324 rs/6000 sp and clustered ibm ^ pseries systems handbook 6159 12gb/24gb 4mm tape drive 1 6553 scsi hot swap 6-pack 1 8131 4.5m controller cable 1 8137 enhanced async node 16-port 1 eia-232 8700 quiet touch keyboard, stealth 1 black - english (us) 8741 3-button mouse - stealth black 1 9300 languag...

  • Page 349

    Chapter 10. Clustering configurations 325 it is composed of seven enterprise servers m/t 7017 and 12 enterprise servers m/t 7026. The cws is an 7025-f80 rs/6000 system with one 128-port asynchronous adapter and a chain of two necessary rans. Each 7017 server has two serial line connection and each 7...

  • Page 350

    326 rs/6000 sp and clustered ibm ^ pseries systems handbook *** m/t 7017 hardware *** product description qty 7017-s80 rs/6000 enterprise server s80 1 1.44mb 3.5-in diskette drive 1 this machine will be part of a 1 cluster 2447 16-bit pci scsi se adapter to 6 1 scsi bays cable 2624 32x speed cd-rom ...

  • Page 351

    Chapter 10. Clustering configurations 327 5765-d51 parallel system support program 1 v3.2 4012 external attached servers (sxx 1 models) 5812 rs/6000 sp resource center 1 (cd-rom) 5813 cd-rom 1 9001 asset registration 1 5692-aix system software 1 0598 aix welcome center 1 0838 aix 4.3 update cd 1 085...

  • Page 352

    328 rs/6000 sp and clustered ibm ^ pseries systems handbook control workstation attachment 4073 memory board, 32-position 1 4133 1024 mb memory (8x128mb dimms) 1 5005 preinstall 1 5200 2-way rs64 iii 500mhz processor 1 card, 4mb l2 cache 5992 system control and 1 initialization cable 6132 cec to pri...

  • Page 353

    Chapter 10. Clustering configurations 329 5692-aix system software 1 0598 aix welcome center 1 0838 aix 4.3 update cd 1 0857 aix 4.3 1 0859 preinstall aix 4.3 server 1 bundle 0860 aix 4.3 bonus pack - non 1 export. 0917 performance aide v3.0 1 1004 cd-rom process charge 1 2924 english 1 3410 cd-rom ...

  • Page 354

    330 rs/6000 sp and clustered ibm ^ pseries systems handbook 6553 scsi hot swap 6-pack 1 8131 4.5m controller cable 2 8133 rj-45 to db-25 converter cables 1 (4 cables per set) 8137 enhanced async node 16-port 2 eia-232 8700 quiet touch keyboard, stealth 1 black - english (us) 8741 3-button mouse - st...

  • Page 355

    Chapter 10. Clustering configurations 331 3410 cd-rom 1 5005 preinstall 1 5924 aix 4.3 pii - english 1 9001 asset registration 1 ces system in switched environment figure 10-16 on page 331 shows an example of a ces system in switched environment. It consists of two enterprise servers m/t 7017 and th...

  • Page 356

    332 rs/6000 sp and clustered ibm ^ pseries systems handbook example 10-5 shows a sample feature codes configuration for the ces system in switched environment considered. Example 10-5 ces system in switched environment m/t 7017 (quantity: 2) *** m/t 7017 hardware *** product description qty 7017-s85...

  • Page 357

    Chapter 10. Clustering configurations 333 product description qty 5765-c34 aix version 4 1 4061 aix 4.3 1 5005 preinstall 1 9001 asset registration 1 5765-d51 parallel system support program 1 v3.2 4012 external attached servers (sxx 1 models) 5812 rs/6000 sp resource center 1 (cd-rom) 5813 cd-rom 1...

  • Page 358

    334 rs/6000 sp and clustered ibm ^ pseries systems handbook 3027 9.1 gb 10k rpm ultra scsi disk 1 drive 3142 remote i/o cable - 3m 2 3154 cable and pci card for sp 1 control workstation attachment 4110 256 mb (2x128mb) sdram dimms 1 5005 preinstall 1 5211 1-way rs64 iv 600 mhz processor 1 card, 2mb ...

  • Page 359

    Chapter 10. Clustering configurations 335 5692-aix system software 1 0598 aix welcome center 1 0838 aix 4.3 update cd 1 0857 aix 4.3 1 0859 preinstall aix 4.3 server 1 bundle 0860 aix 4.3 bonus pack - non 1 export. 0917 performance aide v3.0 1 1004 cd-rom process charge 1 2924 english 1 3410 cd-rom ...

  • Page 360

    336 rs/6000 sp and clustered ibm ^ pseries systems handbook 2968 10/100 mbps ethernet pci 3 adapter 2985 ethernet bnc/rj-45 adapter, pci 1 3109 scsi external port to scsi 1 internal 6 pack cable assembly 3153 18.2 gb 10k rpm 1" ultra3 scsi 2 16-bit disk 3628 p260 color monitor, stealth 1 black 4075 ...

  • Page 361

    Chapter 10. Clustering configurations 337 5692-aix system software 1 0598 aix welcome center 1 0838 aix 4.3 update cd 1 0857 aix 4.3 1 0859 preinstall aix 4.3 server 1 bundle 0860 aix 4.3 bonus pack - non 1 export. 0903 cws diagnostic pre-load 1 0917 performance aide v3.0 1 0922 performance toolbox ...

  • Page 362

    338 rs/6000 sp and clustered ibm ^ pseries systems handbook for details on these and other commands, see ibm parallel system support programs for aix: command and technical reference, sa22-7351. The spframe command the spframe command is used to enter configuration data for a frame or a series of fr...

  • Page 364

    340 rs/6000 sp and clustered ibm ^ pseries systems handbook the command has been enhanced to provide a new -l flag to specify a list of frame numbers to be deleted instead of using the start_frame and frame_count values. This list option can be used to delete all sp frames in a single operation to r...

  • Page 365

    Chapter 10. Clustering configurations 341 you simply select one or more enterprise server system objects (managed ces system resources, such as a frames or nodes) by clicking on them with a mouse and then selecting an action to perform on those system objects from the menu or tool bar. You can use t...

  • Page 366

    342 rs/6000 sp and clustered ibm ^ pseries systems handbook in general, the “view or modify properties frame” notebook provides information on relationships between the enterprise server and the cws. On the other hand, the “view or modify properties node” notebook provides information on the enterpr...

  • Page 367

    Chapter 10. Clustering configurations 343 because of hardware limitations of ces systems, the following default definitions are not useful: • framepoweroff • keynotnormal • nodeenvproblem • nodepowerled • switchnotreachable • switchpowerled • switchresponds0 keynotnormal the key mode switch on a nod...

  • Page 368

    344 rs/6000 sp and clustered ibm ^ pseries systems handbook if you are interested in some of these event definitions, you can use them by registering them. For example, if you are interested in checking that all the enterprise servers in your ces system are up and running, you can use the hostrespon...

  • Page 369

    Chapter 10. Clustering configurations 345 scaling up an existing ces system to an sp system: this is the case when you have a ces system and you want to make it a larger scale sp system. Therefore your enterprise servers will become sp-attached servers and they will beneficiate of the advantages of ...

  • Page 370

    346 rs/6000 sp and clustered ibm ^ pseries systems handbook if you want to add one sp-attached server (as frame c) to the sp system, there are three possible positions that you can place the sp-attached server in. These positions are the followings: before frame a (1 = between frame a+2 and frame b ...

  • Page 371

    Chapter 10. Clustering configurations 347 tty port: /dev/tty8 switch port number:16 in order to initiate “step 29: enter frame information and reinitialize the sdr,” described in chapter 2, “installing and configuring a new rs/6000 sp system” of the ibm parallel system support programs for aix: inst...

  • Page 372

    348 rs/6000 sp and clustered ibm ^ pseries systems handbook • the cws host name. • the name of the boot/install server's interface that is attached to the enterprise server's en0 interface (sp administrative lan interface). – step 18.3: set up routing to the cws host name: if you have a default rout...

  • Page 373

    Chapter 10. Clustering configurations 349 10.4.3 ces system if you want to manage enterprise servers from a single point of control, you have to integrate these servers into a ces system, adding a cws to your servers and install pssp on all of them. Within the ces system, the enterprise servers can ...

  • Page 374

    350 rs/6000 sp and clustered ibm ^ pseries systems handbook ces system with existing enterprise servers in this case, you have to migrate these existing servers to ces. So, you have to preserve the current content of these servers that are currently used for your business. Performing the steps descr...

  • Page 375

    Chapter 10. Clustering configurations 351 an important point to know here is that you have to remove all the sp frames at the same time. You are not allowed to remove them one by one. To remove all the sp frames from your sp system at the same time, use the spdelfram command with the -l flag as foll...

  • Page 376

    352 rs/6000 sp and clustered ibm ^ pseries systems handbook adding sp frames to a ces system this section discusses upgrading an existing ces system to an sp system. There are similar conditions when you add sp frames to a ces system. Remember, when you add sp frames to a ces system, the system is n...

  • Page 377

    Chapter 10. Clustering configurations 353 in this case, the lowest ip address is node 1. However, you cannot assign ip addresses to sp switch adapters as follows: spadaptrs 3 1 3 css0 129.33.34.13 255.255.255.0 instead, use the ip address of the node with the lowest node number as follows: spadaptrs...

  • Page 378

    354 rs/6000 sp and clustered ibm ^ pseries systems handbook frame can be assigned the frame number 1. In this case, the enterprise servers can be sp-attached to the sp with switch port number 0 and 1, as long as slot 0 and 1 are available. However, in this way, reconfiguration of the ces numbering i...

  • Page 379

    Chapter 10. Clustering configurations 355 2. If the ces system has less than 32 enterprise servers and add sp frames, leave space for flexible sp node addition. For example when a ces with 15 enterprise servers is redefined to be sp-attached servers in the sp system, the servers occupy at least 15 s...

  • Page 380

    356 rs/6000 sp and clustered ibm ^ pseries systems handbook figure 10-20 changing a 31-node ces system to an sp system recommended expansion plan if you are thinking of scaling up your ces system to an sp system in the future, you have to plan out appropriate frame numbers and switch port numbers fo...

  • Page 381

    Chapter 10. Clustering configurations 357 switch port number for details on recommended expansion plans on ces systems to sp systems, see ibm rs/6000 clustered enterprise servers systems handbook, sg24-5978. 10.5 pseries 680 cluster solutions entry and advanced cluster ready solution packages are av...

  • Page 382

    358 rs/6000 sp and clustered ibm ^ pseries systems handbook one - complete pseries 680 server (7017-s85), incorporating the following: – one - software preinstall (f/c 5005) – one - entry cluster solution indicator (f/c 0701) – two - scsi boot disks – one - asynch adapter for control workstation att...

  • Page 383

    Chapter 10. Clustering configurations 359 figure 10-21 pseries 680 entry cluster server 10.5.2 pseries 680 advanced cluster server - s85/m80 each pseries 680 advanced cluster server solution order must include a minimum of the following hardware and software that must be ordered together as specifie...

  • Page 384

    360 rs/6000 sp and clustered ibm ^ pseries systems handbook – one - software preinstall (f/c 5005) two - complete 7026-m80 servers, each incorporating the following: – one - advanced cluster solution indicator (f/c 0702) – two - scsi boot disks – one - clustered server control panel to cws cable (f/...

  • Page 385: Naming Convention

    © copyright ibm corp. 361 appendix a. Naming convention the naming convention appendix provides naming conventions used in the contents of the redbook. This appendix is formatted in the following manner: overview – naming convention a.

  • Page 386: Naming Convention

    362 rs/6000 sp and clustered ibm ^ pseries systems handbook definitions overview and naming convention ibm announced changes in product names with the arrival of the new ibm ^ pseries servers. It is important to bear in mind that this redbook uses the new naming convention. Naming convention table a...

  • Page 387

    Appendix a. Naming convention 363 ibm rs/6000 7017 model s70 7017-s70 s70 node name machine type (m/t) nickname used here.

  • Page 388

    364 rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 389: Hardware

    © copyright ibm corp. 365 appendix b. Hardware this appendix includes information regarding: 1. Sp communication adapter requirements 2. Evaluating sp system power requirement 3. Legacy sp node 4. Legacy models by frame and switch types 5. Legacy model by frame and nodes types 6. Sp lan topologies 7...

  • Page 390

    366 rs/6000 sp and clustered ibm ^ pseries systems handbook sp communication adapter requirements the sp communication adapter requirements annex contains information on the quantities of adapters of each type that can be installed in specific sp node types, suggested quantities to use, and adapter ...

  • Page 391

    Appendix b. Hardware 367 descriptions of pci bus group numbering, bus type, slot width and slot speed for power3 and 332 mhz smp nodes are shown in table b-1. Table b-1 pci bus description for smp nodes node type bus group bus type slot numbers slot width slot speed 375 mhz power3 and power3 high 0 ...

  • Page 392

    368 rs/6000 sp and clustered ibm ^ pseries systems handbook pci adapter maximum quantities the maximum quantities of adapters for 375 mhz power3, power3, and 332 mhz smp thin and wide nodes are shown in table b-2. Table b-2 maximum quantities of pci adapters for smp thin and wide nodes f/c pci adapt...

  • Page 393

    Appendix b. Hardware 369 4951 four-port 10/100 lan 1 3/1 2 4958 crypto 1 4/0 4959 token ring 1 6/2 6204 ultra scsi de 1 6/2 3 6205 dual channel ultra3 scsi 1 3/1 2 6206 ultra scsi se 1 6/2 6207 ultra scsi de 1 6/2 6208 scsi-2 f/w se 1 8/2 6209 scsi-2 f/w de 1 8/2 6215 ssa raid5 1 6/2 6222 f/w cache ...

  • Page 394

    370 rs/6000 sp and clustered ibm ^ pseries systems handbook the maximum quantities for 375 mhz power3 high nodes, power3 high nodes, sp expansion i/o units, and node/expansion unit combinations are shown in table b-3. Table b-3 pci adapters for high nodes and sp expansion i/o units notes: 1. Group r...

  • Page 395

    Appendix b. Hardware 371 2985 ethernet 10 mb bnc 5 8 24 24 2987 ethernet 10 mb aui 5 8 24 24 2988 atm 155 mmf 3 8 24 24 4951 four-port 10/100 lan 1 3 6 12 4958 crypto 2 4 4 4 4959 token ring 3 8 24 24 6204 ultra scsi de 3 8 24 24 6205 dual channel ultra3 scsi 1 2 6 8 6206 ultra scsi se 3 8 24 24 620...

  • Page 396

    372 rs/6000 sp and clustered ibm ^ pseries systems handbook pci adapter plugging rules and restrictions this section lists specific pci adapter installation rules and restrictions by node type. 375 mhz power3 smp wide node: if one f/c 2969 is installed, no more than four (in any combination) of f/c ...

  • Page 397

    Appendix b. Hardware 373 if two f/c 2732 are installed, one must be placed in slot i2 or i3 (cpu side) and the other must be placed in slot i1 to i4 (i/o side). If two f/c 2733 are installed, one must be placed in slot i2 or i3 (cpu side) and the other must be placed in slot i1 to i4 (i/o side). Sum...

  • Page 398

    374 rs/6000 sp and clustered ibm ^ pseries systems handbook f/c 6215 must be placed in slot 1, 3, 5, or 7. F/c 2969, 6205, 6225, 6230 must be placed in slot 1 or 3. F/c 2963 and 2988 cannot be placed in the adjacent slot to the right of f/c 2751 f/c 2751 must be placed in slot 1, 3, 5, or 7 in unit ...

  • Page 399

    Appendix b. Hardware 375 332 mhz wide node: no more than four (in any combination) of f/c 2741, 2742, 2743, 2963, 2968, 2988 can be installed. If one f/c 2969 is installed, no more than two (in any combination) of f/c 2741, 2742, 2743, 2963, 2968, 2988 can be installed. If two f/c 2969 are installed...

  • Page 400

    376 rs/6000 sp and clustered ibm ^ pseries systems handbook maximum of one f/c 2732, 2733, 2969, 6205 per thin node. The combined quantity of f/c 6225 and 6230 cannot exceed two per pci bus. Suggested pci adapter quantities for optimum operation the guidelines in this section help you configure your...

  • Page 401

    Appendix b. Hardware 377 2944 wan rs-232 128-port comm low 2947 artic960hx 4-port selectable comm low 2962 2-port multiprotocol adapter comm low 2963 atm turboways 155 utp comm medium 2968 ethernet 10/100 mb comm low (10) medium (100)* 2 2969 gigabit ethernet-sx comm high 2985 ethernet 10 mb bnc com...

  • Page 402

    378 rs/6000 sp and clustered ibm ^ pseries systems handbook 6207 ultra scsi de storage low (f) medium (f/w) high (ultra)* 4 6208 scsi-2 f/w se storage medium 6209 scsi-2 f/w de storage medium 6215 ssa raid5 storage medium (raid) high (dual loop) 6225 ssa raid el storage high 6 6227 fc-al storage hig...

  • Page 403

    Appendix b. Hardware 379 following are lists of weighting factor guidelines and suggested quantities of adapters (by node type, pci bus, and weighting factor) to help you achieve optimum operation on pci buses: power3 high node select one of the following options for pci bus 0: 1 high 2 medium 1 med...

  • Page 404

    380 rs/6000 sp and clustered ibm ^ pseries systems handbook 2 low power3 thin and wide node select one of the following options for pci bus 1: 1 high 2 medium 1 medium and 1 low 2 low power3 wide node select one of the following options for both pci bus 2 and pci bus 3: 1 high 2 medium and 2 low 1 m...

  • Page 405

    Appendix b. Hardware 381 mca bus adapter requirements table b-5 contains reference information on mca communication adapters. Table b-5 micro channel adapter requirements f/c mca adapter name number of mca slots required max. Number allowed per node wide/thin/high notes 190 2 optics daughter card 0 ...

  • Page 406

    382 rs/6000 sp and clustered ibm ^ pseries systems handbook 275 4 s/390 escon channel emulator adapter 2 2/1/4 275 5 bmca 1 2/2/2 5,6 275 6 escon control unit adapter 2 2/1/4 293 0 rs-232 8-port adapter 1 7/4/14 294 0 8-port async adapter 1 7/4/14 296 0 x.25 interface co-processor/2 1 7/4/8 297 0 to...

  • Page 407

    Appendix b. Hardware 383 621 4 ssa 4-port adapter 1 4/2/8 621 6 ssa 4-port adapter 1 4/2/8 621 7 ssa 4-port raid adapter 1 4/2/8 621 9 micro channel ssa multi-initiator/raid el adapter 1 4/2/8 8 622 2 ssa fast-write cache option 0 mounts on 6219 630 5 digital trunk dual 1 6/3/2 700 6 realtime interf...

  • Page 408

    384 rs/6000 sp and clustered ibm ^ pseries systems handbook evaluating sp system power requirement the total power requirement of your rs/6000 sp system is the sum of the power required by all of the following individual components: each base frame with integral power subsystem. Each processor node....

  • Page 409

    Appendix b. Hardware 385 make one copy of table b-6 for each frame in your sp system. Fill in the worksheet copies using your quantities of all applicable features. Complete the calculations, using the power factors given for each feature, to determine the total power requirement of each frame. Afte...

  • Page 410

    386 rs/6000 sp and clustered ibm ^ pseries systems handbook memory power3 smp high node memory card ( f/c 4880) (___) x 66 375 mhz power3 smp thin and wide node memory card (f/c 4098) (___) x 65 power3 smp thin and wide node memory card ( f/c 4098) (___) x 65 332 mhz smp thin and wide node memory ca...

  • Page 411

    Appendix b. Hardware 387 pci adapters serial hippi sw (f/c 2732) (___) x 39.7 serial hippi lw (f/c 2733) (___) x 39.7 fddi sk-net lp sas (f/c 274 1) (___) x 9.9 fddi sk-net lp das (f/c 2742) (___) x 15.2 fddi sk-net up das (f/c 2743) (___) x 9.9 escon channel (f/c 2751) (___) x 22.8 token ring (f/c ...

  • Page 412

    388 rs/6000 sp and clustered ibm ^ pseries systems handbook fibre channel (f/c 6227) (___) x 26.6 ssa raid 5 (f/c 6230) (___) x 36.5 f/w cache module (f/c 6235) (___) x 5.9 artic960rxd quad digital trunk (f/c 6310) (___) x 30.4 mca adapters mca slots occupied in uniprocessor thin nodes (___) x 46 mc...

  • Page 413: Legacy Sp Nodes

    Appendix b. Hardware 389 legacy sp nodes table b-7 shows legacy sp nodes. Table b-7 legacy sp nodes feature code name clock architecture 2001 thin 62mhz power1 2002 thin 66mhz power2 2003 wide 66mhz power2 2004 thin 66mhz power2 2005 wide 77mhz power2 2006 high 112mhz powerpc604 2007 wide 135mhz p2s...

  • Page 414

    390 rs/6000 sp and clustered ibm ^ pseries systems handbook sp legacy models by frame and switch types table b-8 shows the first two digits defining the frame models in legacy frames by switch types. Table b-8 legacy models by frames and switch types (table 1 of 2). Model frame switch switch frame n...

  • Page 415

    Appendix b. Hardware 391 sp legacy models by frame and nodes types table b-9 shows the third digits defining the frame models in legacy frames by node types. Table b-9 legacy models by frames and node types (table 2 of 2) model mhz form processor type i/o bus available xx1 62 thin uni mca no xx2 66 ...

  • Page 416: Sp Lan Topologies

    392 rs/6000 sp and clustered ibm ^ pseries systems handbook sp lan topologies the network topology on sp/ces systems depends mainly on the size of the system. If the applications needed perform significant communication among nodes, additional network connectivity should be provided by sp switch, et...

  • Page 417

    Appendix b. Hardware 393 however, switches with the required switching capabilities are available. An illustration of a 100base-tx network is shown in figure b-1. Figure b-1 shared or switched 100base-tx network heterogeneous 10/100 mbps network sp systems can be upgraded with new nodes having fast ...

  • Page 418

    394 rs/6000 sp and clustered ibm ^ pseries systems handbook figure b-2 heterogeneous 10/100 mbps network in a similar case, you can use this same setup if a particular frame contains a mix of older nodes with 10 mbps bnc and newer nodes with 100 mbps utp connections. Shared 10base2 network in small ...

  • Page 419

    Appendix b. Hardware 395 figure b-3 shared 10base2 network this network has the following properties: no routing is required since the cws and all nodes share one subnet. The entire sp lan is a single broadcast domain as well as a single collision domain. Performance is limited to one 10 mbps hdx co...

  • Page 420

    396 rs/6000 sp and clustered ibm ^ pseries systems handbook figure b-4 segmented 10base2 network a network with n separate subnets and n ethernet cards in the cws has the following properties: nodes in one subnet need static routes to the (n-1) other subnets through the cws and routing or ip forward...

  • Page 421

    Appendix b. Hardware 397 figure b-5 switched 10base2 network this network setup has the following properties: no routing is required. All ethernet segments are transparently combined into one large lan by the ethernet switch. Node-to-node connections within a single ethernet segment still have to sh...

  • Page 422

    398 rs/6000 sp and clustered ibm ^ pseries systems handbook sp boot/install server requirements for the sp ethernet topology, you should consider your network install server requirements. The network install process uses the sp ethernet for transferring the install image from the install server to t...

  • Page 423

    Appendix b. Hardware 399 figure b-6 ethernet topology with one adapter for a single-frame sp system an alternate way to configure your system is to install a second ethernet adapter in your control workstation, if you have an available i/o slot, and use two ethernet segments to the sp nodes. Each ne...

  • Page 424

    400 rs/6000 sp and clustered ibm ^ pseries systems handbook you have to set up your sp ethernet routing so nodes on one ethernet can communicate to nodes on the other network. You also need to set up your network mask so that each sp ethernet is its own subnet within a larger network address. Multip...

  • Page 425

    Appendix b. Hardware 401 you will have to set up your sp ethernet routing so nodes on one ethernet can communicate with nodes on another. You also need to set up your network mask so that each sp ethernet is its own subnet within a larger network address. This method is applicable up to the number o...

  • Page 426

    402 rs/6000 sp and clustered ibm ^ pseries systems handbook figure b-9 method 3 - ethernet topology for multi-frame sp system an alternative to the router in this configuration is an ethernet switch, which could have a high-speed network connection to the control workstation. Future expansion consid...

  • Page 427

    Appendix b. Hardware 403 figure b-10 boot server frame approach the advantage of this is that when you add an additional frame to your sp configuration, all you need to do is connect the new frame to one of the network install nodes, and reconfigure the system. The network install procedure for this...

  • Page 428

    404 rs/6000 sp and clustered ibm ^ pseries systems handbook disk space required for pssp file sets table b-10 shows the approximate sizes of the install image and individual fileset shipped with the pssp software. Table b-10 disk space required for install images and to install pssp file sets image ...

  • Page 429

    Appendix b. Hardware 405 ssp.Pman 1.2mb ssp.Public 12mb ssp.Spmgr 634kb ssp.St 765kb ssp.Sysctl 1.5mb ssp.Sysman 1.3mb ssp.Tecad 203kb ssp.Tguides 1.1mb ssp.Top 1.4mb ssp.Top.Gui 1.2mb ssp.Ucode 907kb pssp image: ssp.Hacws 160kb ssp.Hacws 141kb pssp image: ptpe 7mb ptpe.Docs 3.1mb ptpe.Program 6.2mb...

  • Page 430

    406 rs/6000 sp and clustered ibm ^ pseries systems handbook pssp image ssp.Resctr for the resource center ssp.Resctr.Rte 4mb note: the total storage can cross multiple file systems. Image or fileset name image size file size.

  • Page 431: Sp System Upgrades

    © copyright ibm corp. 407 appendix c. Sp system upgrades this section contains information on available rs6000 sp upgrades, conversions, and feature additions. Examples of typical upgrades include adding frames to your sp system, installing new higher-performance nodes or switches, and converting mc...

  • Page 432: Nodes

    408 rs/6000 sp and clustered ibm ^ pseries systems handbook adding power3, and 332 mhz smp thin and wide nodes smp thin and wide nodes can be added to earlier sp systems. Adding these nodes to earlier withdrawn frames requires a power system upgrade. Both pdu and sepbu power subsystems must be upgra...

  • Page 433: Adding Switches

    Appendix c. Sp system upgrades 409 adding switches sp switches can be added in the following configurations: 1. Switchless to sp switch (f/c 4011), sp switch-8 (f/c 4008), or sp switch2 (f/c 4012) 2. Sp switch (f/c 4011) to sp switch2 (f/c 4012) 3. High performance switch (f/c 4010) to sp switch (f/...

  • Page 434

    410 rs/6000 sp and clustered ibm ^ pseries systems handbook upgrading smp-type nodes available node upgrade features are shown in table c-1. Table c-1 node upgrade features replacing withdrawn processor nodes replacement features are available to replace withdrawn early processor nodes with 332 mhz,...

  • Page 435

    Appendix c. Sp system upgrades 411 relocating processor nodes if you plan to relocate any nodes or switches to another frame, you must order one of the following no-charge features: relocating a thin node (f/c 9250) relocating a wide node (f/c 9251) relocating a high node (f/c 9252) relocating an sp...

  • Page 436

    412 rs/6000 sp and clustered ibm ^ pseries systems handbook.

  • Page 437: Software Appendix

    © copyright ibm corp. 413 appendix d. Software appendix this appendix contains detailed information about the relationships between supported software packages. In this appendix, the following software packages are described: 1. Pssp and aix 2. High availability cluster multi-processing (hacmp) 3. V...

  • Page 438: Pssp And Aix

    414 rs/6000 sp and clustered ibm ^ pseries systems handbook pssp and aix since pssp and aix must work together on the sp to allow other applications to run, they will be considered as one unit in this section. Supported levels each lpp is not supported in every level on pssp and aix. Table d-1 shows...

  • Page 439: Migration

    Appendix d. Software appendix 415 table d-2 shows aix and pssp support for different node types: table d-2 aix and pssp support for different node types a = available; factory and field orderable s = supported n = not supported migration only certain migration paths are possible. Table d-3 list them...

  • Page 440: Coexistence

    416 rs/6000 sp and clustered ibm ^ pseries systems handbook coexistence coexistence is supported in the same system partition or a single default system partition (the entire sp system) for nodes running any combination of aix and pssp as shown in table d-4. Table d-4 levels of pssp and aix supporte...

  • Page 441: Supported Levels

    Appendix d. Software appendix 417 high availability cluster multi-processing (hacmp) this section discussed hacmp supported levels, migration paths and coexistence options. Supported levels while pssp has no direct requirement for hacmp, if you install hardware that requires pssp 3.2, regardless of ...

  • Page 442: Migration Considerations

    418 rs/6000 sp and clustered ibm ^ pseries systems handbook migration considerations hacmp 4.3.1 on the sp requires aix 4.3.3 (or later) and pssp 3.2. You have the following migration options: migrating from hacmp/6000 1.2, 2.1, or 3.1 to hacmp 4.3.1 involves taking configuration snapshots, bringing...

  • Page 443: Vsd And Rvsd

    Appendix d. Software appendix 419 vsd and rvsd in order to exploit the new functions available in the recoverable virtual shared disk (rvsd) option of pssp 3.2, all nodes in the system must be running pssp 3.2 with the rvsd component. Rvsd requires the virtual shared disk (vsd) optional components t...

  • Page 444: Migration Considerations

    420 rs/6000 sp and clustered ibm ^ pseries systems handbook migration considerations the following is a list of points that should be considered before starting a migration. 1. When migrating from ibm recoverable virtual shared disk 1.2 and pssp 2.2 you can migrate one node at a time but you must mi...

  • Page 445: Supported Levels

    Appendix d. Software appendix 421 general parallel file system for aix (gpfs) a file system managed by the gpfs lpp can only be accessed from within the system partition. Gpfs 1.3 changes the locking semantics which control access to data and as a result requires that all nodes be at the same level....

  • Page 446: Parallel Environment (Pe)

    422 rs/6000 sp and clustered ibm ^ pseries systems handbook parallel environment (pe) parallel environment is comprised of: 1. Parallel operating environment (poe) 2. Message passing libraries (mpi, mpl) 3. Parallel utilities which facilitate file manipulations (mpi sample programs) ibm parallel env...

  • Page 447: Essl And Parallel Essl

    Appendix d. Software appendix 423 essl and parallel essl engineering and scientific subroutine library (essl) is not directly dependent on a level of pssp. Essl coexistence and migration is however, the same as for parallel environment because it is dependent on the parallel environment lpp. Paralle...

  • Page 448: Migration Considerations

    424 rs/6000 sp and clustered ibm ^ pseries systems handbook migration considerations loadleveler 2.2 supports node by node migration from loadleveler 2.1 (if a ptf is applied to 2.1). Loadleveler provides other mechanisms for migration from loadleveler 1.3 to 2.1, including the use of separate loadl...

  • Page 449: Sp/ces Tools

    © copyright ibm corp. 425 appendix e. Sp/ces tools this chapter presents several tools that are used in sp/ces systems management. The tools are: network installation manager (for installation and migration) service director electronic service agent (to alert user and ibm about a problem) inventory ...

  • Page 450

    426 rs/6000 sp and clustered ibm ^ pseries systems handbook network installation manager network installation manager, or nim, is an efficient tool to manage an installation or migration. This appendix will highlight some characteristics of the way nim operates in an sp environment including ces env...

  • Page 451

    Appendix e. Sp/ces tools 427 7. Wrappers setup_server setup_server is a crucial function since it is called on every node and bis (boot/install server) on bootup and is central to configuring and maintaining the sp nim environment. For more detailed information on boot/install servers, refer to pssp...

  • Page 452

    428 rs/6000 sp and clustered ibm ^ pseries systems handbook sdr is similar in concept to the aix object data manager (odm) and is a database containing classes, attributes, and objects. It is crucial to the installation procedure. In order to install our sp system, we must initialize and populate th...

  • Page 453

    Appendix e. Sp/ces tools 429 at this step the installation of the cws is complete, we can go ahead and enter our configuration details for the frame(s) and nodes into the sdr. – enter site environment details – enter frame information and reinitialize the sdr. – verify system monitor installation by...

  • Page 454

    430 rs/6000 sp and clustered ibm ^ pseries systems handbook setup_server defines this as a nim resource of the type script; therefore, it is run on the node before nim reboots it. Pssp_script is also run on bootup if a node is set to customize in the sdr. In order to detect this, the script, /etc/rc...

  • Page 455

    Appendix e. Sp/ces tools 431 these wrappers are called by the setup_server command for configuring the nim environment in the cws and boot/install servers (bis). These scripts can also be called from the command line if you know the sequence in which they have to be executed. These scripts use the i...

  • Page 456: Service Director

    432 rs/6000 sp and clustered ibm ^ pseries systems handbook service director service director overview service director for rs/6000 is an application program that operates on a customer's ibm rs/6000. It is invoked by the proper installation of the program and successful registration, set up, custom...

  • Page 457

    Appendix e. Sp/ces tools 433 service calls placed to ibm without operator intervention vital product data (vpd) reported to ibm common focal point for service interface problem-definable threshold levels for call placement reduced hardware error logs high-availability cluster multiprocessing (hacmp)...

  • Page 458: Electronic Service Agent

    434 rs/6000 sp and clustered ibm ^ pseries systems handbook sd may be obtain using internet from: ftp://ftp.Software.Ibm.Com/aix/servdir_client_code/ electronic service agent the electronic service agent (sa) is a software that resides on your system to monitor events and transmit data to ibm. Elect...

  • Page 459

    Appendix e. Sp/ces tools 435 what does service agent do here are some of the key things you can accomplish using electronic service agent for pseries and rs/6000: automatic problem analysis. Problem-definable threshold levels for error reporting. Automatic problem reporting; service calls placed to ...

  • Page 460

    436 rs/6000 sp and clustered ibm ^ pseries systems handbook ensure your pseries or rs/6000 is at version 4.1 or above with concurrent diagnostics installed. 9076 parallel system support program (pssp) code must be at release 1.2 or above. Java is required on all monitored machines. Java for aix 4.3....

  • Page 461: Inventory Scout (Invscout)

    Appendix e. Sp/ces tools 437 obtaining electronic service agent electronic service agent for pseries and rs/6000 user’s guide, sc38-7105 is available at: ftp://ftp.Software.Ibm.Com/aix/service_agent_code inventory scout (invscout) inventory scout is a tool which contains microcode discovery service ...

  • Page 462

    438 rs/6000 sp and clustered ibm ^ pseries systems handbook http://techsupport.Services.Ibm.Com/rs6k/mds.Html vpd capture service http://techsupport.Services.Ibm.Com/rs6k/vcs.Html.

  • Page 463: Related Publications

    © copyright ibm corp. 439 related publications the publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook. Ibm redbooks for information on ordering these publications, see “how to get ibm redbooks” on page 440. Ib...

  • Page 464: How To Get Ibm Redbooks

    440 rs/6000 sp and clustered ibm ^ pseries systems handbook 440 rs/6000 sp and clustered ibm ^ pseries systems handbook http://techsupport.Services.Ibm.Com/rs6k/vcs.Html vpd capture service http://www.Ibm.Com/servers/eserver/pseries/library/ pseries servers publications website http://www.Austin.Ibm...

  • Page 465: Special Notices

    © copyright ibm corp. 441 special notices references in this publication to ibm products, programs or services do not imply that ibm intends to make these available in all countries in which ibm operates. Any reference to an ibm product, program, or service is not intended to state or imply that onl...

  • Page 466

    442 rs/6000 sp and clustered ibm ^ pseries systems handbook the following terms are trademarks of other companies: tivoli, manage. Anything. Anywhere.,the power to manage., anything. Anywhere.,tme, netview, cross-site, tivoli ready, tivoli certified, planet tivoli, and tivoli enterprise are trademar...

  • Page 467: Abbreviations And Acronyms

    © copyright ibm corp. 443 api application programming interfaces atm asynchronous transfer mode blas basic linear algebra subroutines csp converged service processor csu channel service units crm concurrent resource manager cws control workstation das lp dual attach station das up dual attach statio...

  • Page 468

    444 rs/6000 sp and clustered ibm ^ pseries systems handbook tcl tool command language us user space utp unshielded twisted pair vsd virtual shared disk wan wide area network.

  • Page 469: Index

    © copyright ibm corp. 445 index symbols /etc/hosts 430 /etc/inittab 430 /etc/rc.Sp 430 /spdata/sys1/install/pssplpp 348 /spdata/sys1/sdr 427 /tftpboot 430 /tftpbooy/pssfb_script 430 /usr/lpp/ssp/install/bin 430 /vad/adm/splogs/css 176 /var/adm/splogs/css0 176 numerics 1.2 mb/sec rack mountable remot...

  • Page 470

    446 rs/6000 sp and clustered ibm ^ pseries systems handbook artic960rxd quad dta, e1, 120 ohm balanced, 3 m 4-port cable 258 artic960rxd quad dta, e1, 120 ohm balanced, 7.5 m extension cable 258 artic960rxd quad dta, e1, 75 ohm unbal- anced-grounded, 1.8 m 4-port cable 258 artic960rxd quad dta, e1, ...

  • Page 471

    Index 447 f feature code 1241 independent scsi hookup 71 1500 15 1550 15, 37 18.2 gb ultra scsi 10k rpm disk pair 40 2031 15 2032 15 2050 14 2050 332 mhz smp thin node 71 2051 13 2051 332 mhz smp wide node 67 2054 13 2054 power3 smp high node 55 2056 14 2056 375 mhz power3 smp thin node 48 2057 13 2...

  • Page 472

    448 rs/6000 sp and clustered ibm ^ pseries systems handbook 6209 scsi-2 f/w differential 254 6215 ssa raid 5 254 6222 ssa fast-write cache module 255 6310 artic960rxd quad digital trunk adapter 258 8130 1.2 mb/sec remote asynchronous node, 16-port, eia-232 (us) 239 8131 128-port asynchronous control...

  • Page 473

    Index 449 hacmp (high availability cluster multi-processing) 5 hacws 220 hacws (high availability control workstation) 18 hardware perspective 340 high availability 5 high availability cluster 19 high availability cluster multi-processing for aix 221 high availability control workstation 220 limitat...

  • Page 474

    450 rs/6000 sp and clustered ibm ^ pseries systems handbook reservoir modeling 274 seismic analysis 274 structural analysis 274 time series analysis 274 memory-i/o 27 message passing cluster 20 message passing interface 266 mksysb 200 model 500 15 model 550 15, 37 model 555 15 model frame 146 motif ...

  • Page 475

    Index 451 sdr 262 sepbu 140, 154 serial port to serial port cable for drawer-to-drawer connections 239 serial port to serial port cable for rack-to-rack con- nections 239 server consolidation 21 server consolidation 8 server consolidation 8 service director 425 setup_server 427, 431 shared memory pa...

  • Page 476

    452 rs/6000 sp and clustered ibm ^ pseries systems handbook tpc-d 8 turboways 100 atm adapter 245 twisted pair network 314 two-stage switch system 147 u utility loading for sp frames 143 v v.35 cable 242 v.36/eia 449 cable 242 vertical scaling 23 vmm (virtual memory manager) 5 vpd (vital product dat...

  • Page 477: /6000

    (0 .1 ”sp in e) 0. 1 ” > 0 .1 6 9 ” 5 3 ->8 9 pa ge s rs /6000 sp and clustere d i b m ^ pserie s sys te m s h a ndbook.

  • Page 480: Rs/6000 Sp and Clustered

    ® sg24-5596-02 isbn 0738422991 international technical support organization building technical information based on practical experience ibm redbooks are developed by the ibm international technical support organization. Experts from ibm, customers and partners from around the world create timely te...