1. DL manuals
  2. IBM
  3. Server
  4. BladeCenter HX5
  5. Implementation Manual

IBM BladeCenter HX5 Implementation Manual - Preface

Manual is about: Redbooks

Summary of BladeCenter HX5

  • Page 1

    Ibm.Com/redbooks front cover ibm ex5 implementation guide david watts aaron belisle duncan furniss scott haddow michael hurman jeneea jervay eric kern cynthia knight miroslav peic tom sorcic evans tanurdin covers the ibm system x3950 x5, x3850 x5, x3690 x5, and the ibm bladecenter hx5 details techni...

  • Page 3

    International technical support organization ibm ex5 implementation guide may 2011 sg24-7909-00

  • Page 4

    © copyright international business machines corporation 2011. All rights reserved. Note to u.S. Government users restricted rights -- use, duplication or disclosure restricted by gsa adp schedule contract with ibm corp. First edition (may 2011) this edition applies to the following servers: ibm syst...

  • Page 5: Contents

    © copyright ibm corp. 2011. All rights reserved. Iii contents notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xi trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...

  • Page 6

    Iv ibm ex5 implementation guide 2.7.2 system power settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.7.3 performance-related individual system settings . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.8 ibm exflash . . . . . . . . . . . ....

  • Page 7

    Contents v 3.13.4 uefi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.13.5 integrated trusted platform module (tpm). . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 3.13.6 light path diagnostics . . . . . . . . . . . ...

  • Page 8

    Vi ibm ex5 implementation guide 4.13 integrated virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 4.14 supported operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 4.15 rack mounting...

  • Page 9

    Contents vii 6.2.4 processor installation tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 6.3 local memory configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 6.3.1 testing the memory dimms . . . . . ....

  • Page 10

    Viii ibm ex5 implementation guide 7.4.1 before you attach the max5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 7.4.2 installing in a rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 7.4.3 max5 cables . . ...

  • Page 11

    Contents ix 8.5.4 hx5 2-node uefi settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 8.5.5 hx5 with max5 attached . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 8.5.6 operating system-specific settings in uefi . . ....

  • Page 12

    X ibm ex5 implementation guide 9.13 megaraid storage manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521 9.13.1 installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521 9.13.2 drive states ....

  • Page 13: Notices

    © copyright ibm corp. 2011. All rights reserved. Xi notices this information was developed for products and services offered in the u.S.A. Ibm may not offer the products, services, or features discussed in this document in other countries. Consult your local ibm representative for information on the...

  • Page 14: Trademarks

    Xii ibm ex5 implementation guide trademarks ibm, the ibm logo, and ibm.Com are trademarks or registered trademarks of international business machines corporation in the united states, other countries, or both. These and other ibm trademarked terms are marked on their first occurrence in this informa...

  • Page 15: Preface

    © copyright ibm corp. 2011. All rights reserved. Xiii preface high-end workloads drive ever-increasing and ever-changing constraints. In addition to requiring greater memory capacity, these workloads challenge you to do more with less and to find new ways to simplify deployment and ownership. And al...

  • Page 16

    Xiv ibm ex5 implementation guide rack, power and cooling. He is an ibm certified it specialist and member of the it specialist certification review board. Scott haddow is a presales technical support specialist for ibm in the uk. He has 12 years of experience working with servers and storage. He has...

  • Page 17

    Preface xv evans tanurdin is an it specialist at ibm global technology services in indonesia. He provides technical support and services on the ibm system x, bladecenter, and system storage product lines. His technology focus areas include the design, operation, and maintenance services of enterpris...

  • Page 18

    Xvi ibm ex5 implementation guide richard mancini tim martin kevin powell heather richardson david tareen from ibm development: justin bandholz ralph begun jon bitner charles clifton candice coletrane-pagan david drez royce espy dustin fredrickson larry grasso dan kelaher randy kolvick chris leblanc ...

  • Page 19: Comments Welcome

    Preface xvii comments welcome your comments are important to us! We want our books to be as helpful as possible. Send us your comments about this book or other ibm redbooks publications in one of the following ways: use the online contact us review redbooks form found at: ibm.Com/redbooks send your ...

  • Page 20

    Xviii ibm ex5 implementation guide.

  • Page 21: Introduction

    © copyright ibm corp. 2011. All rights reserved. 1 chapter 1. Introduction the ibm ex5 product portfolio represents the fifth generation of servers built upon enterprise x-architecture. Enterprise x-architecture is the culmination of bringing generations of ibm technology and innovation derived from...

  • Page 22: 1.1  Ex5 Systems

    2 ibm ex5 implementation guide 1.1 ex5 systems the four systems in the ex5 family are the x3850 x5, x3950 x5, x3690 x5, and the hx5 blade. The ex5 technology is primarily designed around three major workloads: database servers, server consolidation using virtualization services, and enterprise resou...

  • Page 23: 1.2  Model Summary

    Chapter 1. Introduction 3 1.2 model summary this section summarizes the models that are available for each of the ex5 systems. 1.2.1 ibm system x3850 x5 models table 1-1 lists the standard x3850 x5 models. Table 1-1 base models of the x3850 x5: four socket-scalable server 1.2.2 workload-optimized x3...

  • Page 24

    4 ibm ex5 implementation guide table 1-2 models of the x3950 x5: workload-optimized models 1.2.3 x3850 x5 models with max5 table 1-3 lists the models that are standard with the 1u max5 memory expansion unit. Table 1-3 models of the x3850 x5 with the max5 standard 1.2.4 base x3690 x5 models table 1-4...

  • Page 25

    Chapter 1. Introduction 5 table 1-4 x3690 x5 models 1.2.5 workload-optimized x3690 x5 models table 1-5 on page 6 lists the workload-optimized models. Model 3dx is designed for database applications and uses ssds for the best i/o performance. Backplane connections for sixteen 1.8-inch solid-state dri...

  • Page 26

    6 ibm ex5 implementation guide table 1-5 x3690 x5 workload-optimized models 1.2.6 bladecenter hx5 models table 1-6 shows the base models of the bladecenter hx5, with and without the max5 memory expansion blade. In the table, opt indicates optional and std indicates standard. Table 1-6 models of the ...

  • Page 27: 1.3  Positioning

    Chapter 1. Introduction 7 also available is a virtualization workload-optimized model of these hx5s. This is a pre-configured, pre-tested model targeted at large-scale consolidation. Table 1-7 shows the model. Table 1-7 workload-optimized models of the hx5 model 7872-68x is a virtualization-optimize...

  • Page 28

    8 ibm ex5 implementation guide 1.3.1 ibm system x3850 x5 and x3950 x5 the system x3850 x5 and the workload-optimized x3950 x5 are the logical successors to the x3850 m2 and x3950 m2. The x3850 x5 and x3950 x5 both support up to four processors and 1.024 tb (terabyte) of ram in a single-node environm...

  • Page 29

    Chapter 1. Introduction 9 1.3.2 ibm system x3690 x5 the x3690 x5, as shown on figure 1-3, is a 2-processor server that exceeds the capabilities of the current mid-tier server, the x3650 m3. You can configure the x3690 x5 with processors that have more cores and more cache than the x3650 m3. You can ...

  • Page 30: 1.4  Energy Efficiency

    10 ibm ex5 implementation guide figure 1-4 blade hx 5 dual scaled table 1-11 compares these blades. Table 1-11 hs22, hs22v, and hx5 compared 1.4 energy efficiency we put extensive engineering effort into keeping your energy bills low - from high-efficiency power supplies and fans to lower-draw proce...

  • Page 31: 1.5  Services Offerings

    Chapter 1. Introduction 11 dynamic fan speeds: in the event of a fan failure, the other fans run faster to compensate until the failing fan is replaced. Regular fans must run faster at all times, just in case, thereby wasting power. Although these systems provide incremental gains at the individual ...

  • Page 32

    12 ibm ex5 implementation guide.

  • Page 33: Product Overview

    © copyright ibm corp. 2011. All rights reserved. 13 part 1 product overview in this first part of the book, we provide detailed technical information about the servers in the ex5 portfolio. This information is most useful in designing, configuring, and planning to order a server solution. This part ...

  • Page 34

    14 ibm ex5 implementation guide.

  • Page 35: Ibm Ex5 Technology

    © copyright ibm corp. 2011. All rights reserved. 15 chapter 2. Ibm ex5 technology this chapter describes the technology that ibm brings to the ibm ex5 portfolio of servers. The chapter describes the fifth generation of ibm enterprise x-architecture (exa) chip sets, called ex5 . This chip set is the ...

  • Page 36: 2.1  Ex5 Chip Set

    16 ibm ex5 implementation guide 2.1 ex5 chip set the members of the ex5 server family are defined by their ability to use ibm fifth-generation chip sets for intel x86 server processors. Ibm engineering, under the banner of enterprise x-architecture (exa), brings advanced system features to the intel...

  • Page 37

    Chapter 2. Ibm ex5 technology 17 table 2-1 two-socket, 2-socket scalable, and 4-socket scalable intel processors minimum configuration for more information about processor options and the installation order of the processors, see the following links: ibm system x3850 x5: 3.7, “processor options” on ...

  • Page 38

    18 ibm ex5 implementation guide hyper-threading technology is designed to improve server performance by exploiting the multi-threading capability of operating systems and server applications in such a way as to increase the use of the on-chip execution resources available on these processors. Applic...

  • Page 39

    Chapter 2. Ibm ex5 technology 19 figure 2-1 shared front-side bus, in the ibm x360 and x440; with snoop filter in the x365 and x445 the front-side bus carries all reads and writes to the i/o devices, and all reads and writes to memory. Also, before a processor can use the contents of its own cache, ...

  • Page 40

    20 ibm ex5 implementation guide figure 2-3 independent processor buses, as in the x3850 m2 and x3950 m2 instead of a parallel bus connecting the processors to a core chip set, which functions as both a memory and i/o controller, the xeon 6500 and 7500 family processors implemented in ibm ex5 servers...

  • Page 41

    Chapter 2. Ibm ex5 technology 21 2.2.5 processor performance in a green world all ex5 servers from the factory are designed to use power in the most efficient means possible. Controlling how much power the server is going to use is managed by controlling the core frequency and power applied to the p...

  • Page 42: 2.3  Memory

    22 ibm ex5 implementation guide 2.3 memory in this section, we describe the major features of the memory subsystem in ex5 systems. We describe the following topics in this section: 2.3.1, “memory speed” on page 22 2.3.2, “memory dimm placement” on page 23 2.3.3, “memory ranking” on page 24 2.3.4, “n...

  • Page 43

    Chapter 2. Ibm ex5 technology 23 memory performance test on various memory speeds based on benchmarks using an ibm internal load generator run on an x3850 x5 system configured with four x7560 processors and 64x 4 gb quad-rank dimms, the following results were observed: peak throughput per processor ...

  • Page 44

    24 ibm ex5 implementation guide figure 2-6 relative memory performance based on dimm placement (one processor and two memory cards shown) 2.3.3 memory ranking the underlying speed of the memory as measured in mhz is not sensitive to memory population. (in intel xeon 5500 processor-based systems, suc...

  • Page 45

    Chapter 2. Ibm ex5 technology 25 extend the pages across ranks, thereby making the pages effectively larger and therefore creating more page-hit cycles. We used three types of memory dimms for this analysis: four gb 4rx8 (four ranks using x8 dram technology) two gb 2rx8 (two ranks) one gb 1rx8 (one ...

  • Page 46

    26 ibm ex5 implementation guide 2.3.4 nonuniform memory architecture (numa) nonuniform memory architecture (numa) is an important consideration when configuring memory, because a processor can access its own local memory faster than non-local memory. Not all configurations use 64 dimms spread across...

  • Page 47

    Chapter 2. Ibm ex5 technology 27 on each of the attached memory buffers), dimms must be installed in quantities of four per processor to enable hemisphere mode. In addition, because eight dimms per processor are required for using all memory channels, eight dimms per processor need to be installed a...

  • Page 48

    28 ibm ex5 implementation guide 2.3.6 reliability, availability, and serviceability (ras) features in addition to hemisphere mode, dimm balance and memory size, memory performance is also affected by the various memory reliability, availability, and serviceability (ras) features that can be enabled ...

  • Page 49

    Chapter 2. Ibm ex5 technology 29 memory sparing sparing provides a degree of redundancy in the memory subsystem, but not to the extent of mirroring. In contrast to mirroring, sparing leaves more memory for the operating system. In sparing mode, the trigger for failover is a preset threshold of corre...

  • Page 50

    30 ibm ex5 implementation guide rbs operates automatically without issuing a predictive failure analysis (pfa) or light path diagnostics alert to the administrator, although an event is logged to the service processor log. After the second dimm failure, pfa and light path diagnostics alerts occur on...

  • Page 51: 2.4  Max5

    Chapter 2. Ibm ex5 technology 31 for more information regarding each of the ex5 systems and the available i/o adapters, see the following sections: ibm system x3850 x5: 3.12, “i/o cards” on page 104. Ibm system x3690 x5: 4.10.4, “i/o adapters” on page 168. Ibm bladecenter hx5: 5.13, “i/o expansion c...

  • Page 52

    32 ibm ex5 implementation guide figure 2-10 shows the x3850 x5 with the max5 attached. Figure 2-10 ibm system x3850 x5 with max5 (the max5 is the 1u unit beneath the main system) figure 2-11 shows the max5 removed from the housing. Figure 2-11 ibm max5 for the x3850 x5 and x3690 x5 max5 connects to ...

  • Page 53: 2.5  Scalability

    Chapter 2. Ibm ex5 technology 33 table 2-2 through table 2-4 show the memory capacity and bandwidth increases that are possible with max5 for the hx5, x3690 x5, and x3850 x5. Table 2-2 hx5 compared to hx5 with max5 table 2-3 x3690 x5 compared to x3690 x5 with max5 table 2-4 x3850 x5 compared to x385...

  • Page 54: 2.6  Partitioning

    34 ibm ex5 implementation guide as shown in figure 2-12 on page 33, the following scalability is possible: memory scaling: a max5 unit can attach to an ex5 server through qpi link cables. This method provides the server with additional memory dimm slots. We refer to this combination as a memory-enha...

  • Page 55

    Chapter 2. Ibm ex5 technology 35 figure 2-14 shows option for putting a partition into stand-alone mode figure 2-15 hx5 partition in stand-alone mode the amm can be accessed remotely, so partitioning can be done without physically touching the systems. Partitioning can allow you to qualify two syste...

  • Page 56: 2.7  Uefi System Settings

    36 ibm ex5 implementation guide 2.7 uefi system settings the unified extensible firmware interface (uefi) is a pre-boot environment that provides an interface between server firmware and operating system. It replaces bios as the software that manages the interface between server firmware, operating ...

  • Page 57

    Chapter 2. Ibm ex5 technology 37 figure 2-17 uefi settings main panel choose system settings to access the system settings options that we will describe here, as shown in figure 2-18 on page 38..

  • Page 58

    38 ibm ex5 implementation guide figure 2-18 uefi system settings panel for more information about system-specific uefi options, see the following sections: ibm system x3850 x5: 6.9, “uefi settings” on page 259 ibm system x3690 x5: 7.8, “uefi settings” on page 337 ibm bladecenter hx5: 8.5, “uefi sett...

  • Page 59

    Chapter 2. Ibm ex5 technology 39 figure 2-19 operating modes in uefi acoustic mode figure 2-20 shows the acoustic mode predetermined values. They emphasize power-saving server operation for generating less heat and noise. In turn, the system is able to lower the fan speed of the power supplies and t...

  • Page 60

    40 ibm ex5 implementation guide figure 2-21 efficiency mode predetermined values performance mode figure 2-22 shows the performance mode predetermined values. The server is set to use the maximum performance limits within uefi. These values include turning off several power management features of th...

  • Page 61

    Chapter 2. Ibm ex5 technology 41 controller makes decisions to move virtual servers between physical hosts to cater to each guest os resource requirement and, when possible, shuts down unneeded physical hosts to save power. Aggressive power management at the hardware level can interfere with the os-...

  • Page 62

    42 ibm ex5 implementation guide additional settings in addition to the operating mode selection, the uefi settings under operating modes include these additional settings: quiet boot (default: enable ) this mode enables system booting with less information displayed. Halt on severe error (default: d...

  • Page 63

    Chapter 2. Ibm ex5 technology 43 figure 2-24 uefi power settings window 2.7.3 performance-related individual system settings the uefi default settings are configured to provide optimal performance with reasonable power consumption. Other operating modes are also available to meet various power and p...

  • Page 64

    44 ibm ex5 implementation guide figure 2-25 uefi processor system settings panel the following processor feature options are available: turbo mode (default: enable ) this mode enables the processor to increase its clock speed dynamically as long as the cpu does not exceed the thermal design power (t...

  • Page 65

    Chapter 2. Ibm ex5 technology 45 hyper-threading (default: enable ) this option enables logical multithreading in the processor, so that the operating system can execute two threads simultaneously for each physical core. Execute disable bit (default: enable ) this option enables the processor to dis...

  • Page 66

    46 ibm ex5 implementation guide figure 2-26 uefi memory system settings panel the following memory feature options are available: memory spare mode (default: disable ) this option enables memory sparing mode, as described in “memory sparing” on page 29. Memory mirror mode (default: non-mirrored ) th...

  • Page 67: 2.8  Ibm Exflash

    Chapter 2. Ibm ex5 technology 47 cke low power (default: disable ) this option enables the memory to enter a low-power state for power savings by reducing the signal frequency. Patrol scrub (default: disable ) this option enables scheduled background memory scrubbing before any error is reported, as...

  • Page 68

    48 ibm ex5 implementation guide each exflash can replace four 2.5-inch serial-attached scsis (sas) or (ssds). You can install the following number of exflash units: the x3850 x5 can have either of the following configurations: – up to four sas or serial advanced technology attachment (sata) drives, ...

  • Page 69

    Chapter 2. Ibm ex5 technology 49 in addition to using less power than rotating magnetic media, the ssds are more reliable, and they can service many more iops. These attributes make them well suited to i/o intensive applications, such as complex queries of databases. Figure 2-27 shows an exflash uni...

  • Page 70

    50 ibm ex5 implementation guide under similar optimized benchmarking conditions, eight of the 50 gb, 1.8-inch ssds are able to sustain 48,000 read iops and, in a separate benchmark, 16,000 write iops. The cost of usd12,000 for the ssds works out at approximately usd0.25 per iop and usd60 per gigabyt...

  • Page 71: Implementation

    Chapter 2. Ibm ex5 technology 51 rhev-h (kvm) supports 96 cpu threads (an 8-core processor with hyper-threading enabled has 16 threads) and can address 1 tb ram. Kvm includes the following features: advanced memory management support robust and scalable linux virtual memory manager support for large...

  • Page 72

    52 ibm ex5 implementation guide be assigned to ssd storage. When you must use memory swap files, assign the swap file space to conventional sas or sata hard drives. 2.10.2 ssd drives and battery backup cache on raid controllers when using conventional sas or sata hard drives on a serveraid controlle...

  • Page 73

    Chapter 2. Ibm ex5 technology 53 the original database lookup. This capability allows the database or web content server to off-load the processing time that is needed to maintain those client sessions. You can define the same physical servers as virtual servers with access to a collection of ssd dr...

  • Page 74

    54 ibm ex5 implementation guide.

  • Page 75

    © copyright ibm corp. 2011. All rights reserved. 55 chapter 3. Ibm system x3850 x5 and x3950 x5 in this chapter, we introduce the ibm system x3850 x5 and the ibm system x3950 x5. The x3850 x5 and x3950 x5 are the follow-on products to the ex4-based x3850 m2, and like their predecessor, are 4-socket ...

  • Page 76: 3.1  Product Features

    56 ibm ex5 implementation guide 3.1 product features the ibm system x3850 x5 and x3950 x5 servers address the following requirements that many ibm enterprise clients need: the ability to have increased performance on a smaller it budget the ability to increase database and virtualization performance...

  • Page 77

    Chapter 3. Ibm system x3850 x5 and x3950 x5 57 the x3850 x5 has the following physical specifications: width: 440 mm (17.3 in.) depth: 712 mm (28.0 in.) height: 173 mm (6.8 in.) or 4 rack units (4u) minimum configuration: 35.4 kg (78 lb.) maximum configuration: 49.9 kg (110 lb.) figure 3-1 shows the...

  • Page 78

    58 ibm ex5 implementation guide figure 3-2 x3850 x5 internals figure 3-3 shows the connectors at the back of the server. Figure 3-3 rear of the x3850 x5 two 1975w rear-access hot-swap, redundant power supplies four intel xeon cpus eight memory cards for 64 dimms total – eight 1066mhz ddr3 dimms per ...

  • Page 79

    Chapter 3. Ibm system x3850 x5 and x3950 x5 59 3.1.2 ibm system x3950 x5 product features for certain enterprise workloads, ibm offers preconfigured models under the product name x3950 x5 . These models do not differ from standard x3850 x5 models in terms of the machine type or the options used to c...

  • Page 80

    60 ibm ex5 implementation guide the max5 has the following specifications: ibm exa5 chip set. Intel memory controller with eight memory ports (four dimms on each port). Intel quickpath interconnect (qpi) architecture technology to connect the max5 to the x3850 x5. Four qpi links operate at up to 6.4...

  • Page 81

    Chapter 3. Ibm system x3850 x5 and x3950 x5 61 figure 3-5 max5 connectors and leds figure 3-6 shows the internals of the max5 including the ibm exa chip, which acts as the interface to the qpi links from the x3850 x5. Figure 3-6 max5 memory expansion unit internals for an in-depth look at the max5 o...

  • Page 82

    62 ibm ex5 implementation guide table 3-1 comparison of the x3850 m2 to the x3850 x5 subsystem x3850 x5 x3850 m2 cpu card no voltage regulator modules (vrms), 4 voltage regulator down (vrds) top access to cpus and cpu card no voltage regulator down (vrds), 4 voltage regulator modules (vrms) top acce...

  • Page 83: 3.2  Target Workloads

    Chapter 3. Ibm system x3850 x5 and x3950 x5 63 3.2 target workloads this solution includes the following target workloads: virtualization the following features address this workload: – integrated usb key: all x3850 x5 models support the addition of an internal usb key that is preloaded with vmware ...

  • Page 84: 3.3  Models

    64 ibm ex5 implementation guide 3.3 models this section lists the currently available models. The x3850 x5 and x3950 x5 (both models are machine type 7145) have a three-year warranty. For information about the recent models, consult tools, such as the configurations and options guide (cog) or standa...

  • Page 85

    Chapter 3. Ibm system x3850 x5 and x3950 x5 65 backplane connections for four 2.5-inch sas hdds are standard; however, you must order the sas hdds separately. A serveraid br10i sas controller is standard in this model. Table 3-3 models of the x3950 x5: workload-optimized models x3850 x5 models with ...

  • Page 86: 3.4  System Architecture

    66 ibm ex5 implementation guide 3.4 system architecture this section explains the system board architecture and the use of the qpi wrap card. 3.4.1 system board figure 3-7 shows the system board layout of a single-node 4-way system. Figure 3-7 block diagram for single-node x3850 x5 in figure 3-7, th...

  • Page 87

    Chapter 3. Ibm system x3850 x5 and x3950 x5 67 figure 3-8 shows the qpi wrap card. Figure 3-8 qpi wrap card for single-node systems with three or four processors installed, but without the max5 memory expansion unit connected, install two qpi wrap cards. Figure 3-9 shows a diagram of how the qpi wra...

  • Page 88: 3.5  Max5

    68 ibm ex5 implementation guide the qpi wrap cards are not included with standard server models and must be ordered separately. See table 3-5. Table 3-5 ordering information for the qpi wrap card the qpi wrap cards are installed in the qpi bays at the back of the server, as shown in figure 3-10. Qpi...

  • Page 89

    Chapter 3. Ibm system x3850 x5 and x3950 x5 69 page 64, also, you can order the max5 separately, as listed in table 3-6. When ordering a max5, remember to order the cable kit as well. For power supply fault redundancy, order the optional power supply. Table 3-6 ordering information for the ibm max5 ...

  • Page 90: 3.6  Scalability

    70 ibm ex5 implementation guide the max5 is connected to the x3850 x5 using four cables, connecting the qpi ports on the server to the four qpi ports on the max5. Figure 3-12 shows architecturally how a single-node x3850 x5 connects to a max5. Figure 3-12 the x3850 x5: connectivity of the system uni...

  • Page 91

    Chapter 3. Ibm system x3850 x5 and x3950 x5 71 two x3850 x5 servers connected to form a single image 8-socket server. This configuration is sometimes referred to as a 2-node server . 3.6.1 memory scalability with max5 the max5 memory expansion unit permits the x3850 x5 to scale to an additional 32 d...

  • Page 92

    72 ibm ex5 implementation guide figure 3-14 cabling diagram for two node x3850 x5 connecting the two x3850 x5 servers to form a 2-node system requires one ibm x3850 x5 and x3950 x5 qpi scalability kit, which consists of four qpi cables. See table 3-8. Table 3-8 ordering information for the ibm x3850...

  • Page 93

    Chapter 3. Ibm system x3850 x5 and x3950 x5 73 figure 3-15 qpi links for a 2-node x3850 x5 qpi-based scaling is managed primarily through the unified extensible firmware interface (uefi) firmware of the x3850 x5. For the 2-node x3850 x5 scaled through the qpi ports, when those cables are connected, ...

  • Page 94: 3.7  Processor Options

    74 ibm ex5 implementation guide 3.7 processor options the x3850 x5 is supported with two, three, or four processors. Table 3-9 shows the option part numbers for the supported processors. In a 2-node system, you must have eight processors, which must all be identical. For a list of the processor opti...

  • Page 95

    Chapter 3. Ibm system x3850 x5 and x3950 x5 75 use these population guidelines: each cpu requires a minimum of two dimms to operate. All processors must be identical. Only configurations of two, three, or four processors are supported. The number of installed processors dictates what memory cards ca...

  • Page 96: 3.8  Memory

    76 ibm ex5 implementation guide 3.8 memory memory is installed in the x3850 x5 in memory cards. Up to eight memory cards can be installed in the server, and each card holds eight dimms. Therefore, the x3850 x5 supports up to 64 dimms. This section includes the following topics: 3.8.1, “memory cards ...

  • Page 97

    Chapter 3. Ibm system x3850 x5 and x3950 x5 77 standard models contain two or more memory cards. You can configure additional cards, as listed in table 3-10. Table 3-10 ibm system x3850 x5 and x3950 x5 memory card the memory cards are installed in the server, as shown in figure 3-17. Each processor ...

  • Page 98

    78 ibm ex5 implementation guide dimms for the x3850 x5 table 3-11 shows the available dimms that are supported in the x3850 x5 server. The table also indicates the dimm options that are also supported in the max5. When used in the max5, the dimms have separate feature codes, which are shown as - fc ...

  • Page 99

    Chapter 3. Ibm system x3850 x5 and x3950 x5 79 max5 memory the max5 memory expansion unit has 32 dimm sockets and is designed to augment the installed memory in the attached x3850 x5 server. Table 3-12 shows the available memory options that are supported in the max5 memory expansion unit. These opt...

  • Page 100

    80 ibm ex5 implementation guide installing dimms in the x3850 x5 and max5 in the correct order is essential for system performance. See “mixed dimms and the effect on performance” on page 86 for performance effects when this guideline is not followed. X3850 x5 single-node and 2-node configurations t...

  • Page 101

    Chapter 3. Ibm system x3850 x5 and x3950 x5 81 table 3-14 numa-compliant dimm installation (three processors): x3850 x5 table 3-15 shows the numa-compliant memory installation sequence for four processors. Table 3-15 numa-compliant dimm installation (four processors): x3850 x5 nu m b e r of d imm s ...

  • Page 102

    82 ibm ex5 implementation guide max5 configurations the memory installed in the max5 operates at the same speed as the memory that is installed in the x3850 x5 server. As explained in 2.3.1, “memory speed” on page 22, the memory speed is derived from the qpi link speed of the installed processors, w...

  • Page 103

    Chapter 3. Ibm system x3850 x5 and x3950 x5 83 figure 3-18 dimm numbering on max5 table 3-16 shows the population order of the max5 dimm slots, ensuring that memory is balanced among the memory buffers. The colors in the table match the colors in figure 3-18. Table 3-16 dimm installation sequence in...

  • Page 104

    84 ibm ex5 implementation guide max5 memory as seen by the operating system max5 is capable of two modes of operation in terms of the way that memory is presented to the operating system: memory in max5 can be split and assigned between the cpus on the host system (partitioned mode). This mode is th...

  • Page 105

    Chapter 3. Ibm system x3850 x5 and x3950 x5 85 figure 3-19 shows eight possible memory configurations for the two memory cards and 16 dimms connected to one processor socket. Each configuration has a relative performance score. Note the key information from this chart: the best performance is achiev...

  • Page 106

    86 ibm ex5 implementation guide populate one dimm per channel on every channel on memory before populating a second dimm in any channel. Populate dimms at the end of a memory channel first before populating the dimm closer to the memory buffer. That is, install to sockets 1, 3, 6, and 8 first. If yo...

  • Page 107

    Chapter 3. Ibm system x3850 x5 and x3950 x5 87 figure 3-20 relative memory performance using mixed dimms as you can see, mixing dimm sizes can cause performance loss up to 18%, even if all channels are occupied and hemisphere mode is maintained. 3.8.4 memory mirroring memory mirroring is supported u...

  • Page 108

    88 ibm ex5 implementation guide table 3-17 x3850 x5 memory mirroring 4-processor 2-node table 3-18 shows the memory mirroring card pairs. Table 3-18 memory mirroring: card pairs max5 memory mirroring population order table 3-19 on page 89 shows the installation guide for max5 memory mirroring. Nu m ...

  • Page 109

    Chapter 3. Ibm system x3850 x5 and x3950 x5 89 table 3-19 max5 memory mirroring setup 3.8.5 memory sparing sparing provides a degree of redundancy in the memory subsystem, but not to the extent of mirroring. For more information regarding memory sparing, see “memory sparing” on page 29. Use these gu...

  • Page 110: 3.9  Storage

    90 ibm ex5 implementation guide figure 3-21 relative memory throughput by memory mode 3.9 storage in this section, we look at the internal storage and raid options for the x3850 x5, with suggestions where you can obtain the details about supported external storage arrays. This section includes the f...

  • Page 111

    Chapter 3. Ibm system x3850 x5 and x3950 x5 91 figure 3-22 front view of the x3850 x5 showing eight 2.5-inch sas drives 3.9.2 sas and ssd 2.5-inch disk support this section describes backplane, controller, and drive options for 2.5-inch disk drives and ssds. 2.5-inch sas disks and ssds use the same ...

  • Page 112

    92 ibm ex5 implementation guide table 3-22 lists the 2.5-inch sas 10k and 15k rpm disk drives and the 2.5-inch ssd that are supported in the x3850 x5. These drives are supported with the sas hard disk backplane, 59y6135. Table 3-22 supported 2.5-inch sas drives and 2.5-inch ssds table 3-23 lists the...

  • Page 113

    Chapter 3. Ibm system x3850 x5 and x3950 x5 93 3.9.3 ibm exflash and 1.8-inch ssd support database-optimized models of the x3950 x5 include one ibm exflash ssd backplane, supporting eight 1.8-inch solid-state drives, as listed in table 3-3 on page 65. Other models also support the addition of an exf...

  • Page 114

    94 ibm ex5 implementation guide table 3-25 controllers supported with the exflash ssd backplane option when ordering m5000 series controllers (m5014, m5015, or m5025) for use only with ssd drives, the cache battery must not be used for performance reasons. If using m5000 series controllers in a mixe...

  • Page 115

    Chapter 3. Ibm system x3850 x5 and x3950 x5 95 figure 3-24 disabling battery cache on controller in megaraid web bios serveraid m5000 series performance accelerator key serveraid m5000 series performance accelerator key for system x enables performance enhancements that are needed by emerging ssd te...

  • Page 116

    96 ibm ex5 implementation guide the failure rate of ssds is low because, in part, the drives have no moving parts. The 50 gb high iops ssd is a single level cell (slc) device with enterprise wear leveling. As a consequence of both of these technologies, the additional layer of protection that is pro...

  • Page 117

    Chapter 3. Ibm system x3850 x5 and x3950 x5 97 supports raid-0, raid-1, and raid-1e no battery and no onboard cache serveraid m5014 and m5015 controllers the serveraid m5014 and m5015 adapter cards have the following specifications: eight internal 6 gbps sas/sata ports two mini-sas internal connecto...

  • Page 118

    98 ibm ex5 implementation guide for more information, see serveraid m5015 and m5014 sas/sata controllers for ibm system x, tips0738, at the following website: http://www.Redbooks.Ibm.Com/abstracts/tips0738.Html?Open serveraid m5025 controller the key difference between the serveraid m5025 and m5015 ...

  • Page 119

    Chapter 3. Ibm system x3850 x5 and x3950 x5 99 for more information, see serveraid m5025 sas/sata controller for ibm system x, tips0739, at the following website: http://www.Redbooks.Ibm.Com/abstracts/tips0739.Html?Open serveraid m1015 controller the serveraid m1015 sas/sata controller has the follo...

  • Page 120

    100 ibm ex5 implementation guide for more information, see ibm 6gb ssd host bus adapter for ibm system x, tips0744, which is available at the following website: http://www.Redbooks.Ibm.Com/abstracts/tips0744.Html?Open serveraid b5015 ssd controller the serveraid b5015 is a high-performance raid cont...

  • Page 121

    Chapter 3. Ibm system x3850 x5 and x3950 x5 101 figure 3-25 serveraid m5015 sas/sata controller raid-6, raid-60, and encryption are a further optional upgrade for the m5015 through the serveraid m5000 series advance feature key or the performance accelerator key. 3.9.6 external storage connectivity ...

  • Page 122: 3.10  Optical Drives

    102 ibm ex5 implementation guide intelligent lithium polymer battery backup unit is standard with up to 48 hours of data retention. Support for raid levels 0, 1, 5, 10, and 50 (raid-6 and 60 support with either the optional m5000 advanced feature key or the optional m5000 performance key). Connectio...

  • Page 123: 3.11  Pcie Slots

    Chapter 3. Ibm system x3850 x5 and x3950 x5 103 3.11 pcie slots the x3850 x5 has a total of seven pci express (pcie) slots. Slot 7 holds the emulex 10gb ethernet adapter that is standard in most models (see 3.3, “models” on page 64). We describe the emulex 10gb ethernet adapter in 3.12.1, “standard ...

  • Page 124: 3.12  I/o Cards

    104 ibm ex5 implementation guide slots 5 - 7, the onboard broadcom-based ethernet dual-port chip and the custom slot for the raid controller are on the first pcie bridge and require that either cpu 1 or 2 is installed and operational. Slots 1 - 4 are on the second pcie bridge and require that either...

  • Page 125

    Chapter 3. Ibm system x3850 x5 and x3950 x5 105 figure 3-26 top view of slot 6 and 7 showing that slot 7 is slightly longer than slot 6 the emulex 10gb ethernet adapter in the x3850 x5 has been customized with a special type of connector called an extended edge connector . The card itself is colored...

  • Page 126

    106 ibm ex5 implementation guide table 3-32 emulex adapter part numbers general details about this card are in emulex 10gb virtual fabric adapter for ibm system x, tips0762, which is available at the following website: http://www.Redbooks.Ibm.Com/abstracts/tips0762.Html the emulex 10gb ethernet adap...

  • Page 127

    Chapter 3. Ibm system x3850 x5 and x3950 x5 107 deployment of this adapter and other emulex oneconnect-based adapters with onecommand manager interoperable with bnt 10gb top of rack (tor) switch for fcoe functions interoperable with cisco nexus 5000 and brocade 10gb ethernet switches for nic/fcoe sf...

  • Page 128

    108 ibm ex5 implementation guide this list is constantly updated and changed. To see the latest updates, see the following website: http://ibm.Com/systems/xbc/cog/x3850x5_7145/x3850x5_7145io.Html tools, such as the configurations and options guide (cog) or ssct, contain information about supported p...

  • Page 129

    Chapter 3. Ibm system x3850 x5 and x3950 x5 109 this page lists the third-party vendors that have performed their own testing of their options with our servers. This support information means that those vendors agree to support the combinations that are shown in those particular pages. Although ibm ...

  • Page 130

    110 ibm ex5 implementation guide 3.13.3 integrated management module (imm) the system x3850 x5 includes an imm that provides industry-standard intelligent platform management interface (ipmi) 2.0-compliant systems management. You access the imm through software that is compatible with ipmi 2.0 (xcat...

  • Page 131

    Chapter 3. Ibm system x3850 x5 and x3950 x5 111 using all of the features of uefi requires an uefi-aware operating system and adapters. Uefi is fully backward-compatible with bios. For more information about uefi, see the ibm white paper, introducing uefi-compliant firmware on ibm system x and blade...

  • Page 132

    112 ibm ex5 implementation guide figure 3-29 shows the light path diagnostics panel on the x3850 x5. Figure 3-29 light path diagnostics panel on the x3850 x5 full details about the functionality and operation of the light path diagnostics in this system are in the ibm system x3850 x5 and x3950 x5 pr...

  • Page 133

    Chapter 3. Ibm system x3850 x5 and x3950 x5 113 figure 3-30 cooling zones in the x3850 x5 six strategically located hot-swap/redundant fans, combined with efficient airflow paths, provide highly effective system cooling for the ex5 systems. This technology is known as ibm calibrated vectored cooling...

  • Page 134

    114 ibm ex5 implementation guide 3.15 integrated virtualization selected models of the x3950 x5 include an installed usb 2.0 flash key that is preloaded with either vmware esxi 4.0 or vmware esxi 4.1, as shown in figure 3-31. However, all models of x3850 x5 support several usb keys as options. For a...

  • Page 135: 3.17  Rack Considerations

    Chapter 3. Ibm system x3850 x5 and x3950 x5 115 because a short delay in qualification for several of these operating systems might exist, check the serverproven operating system support page for a current statement at the following website: http://www.Ibm.Com/servers/eserver/serverproven/compat/us/...

  • Page 136

    116 ibm ex5 implementation guide.

  • Page 137: Ibm System X3690 X5

    © copyright ibm corp. 2011. All rights reserved. 117 chapter 4. Ibm system x3690 x5 the x3690 x5 servers are powerful 2-socket rack-mount servers with 4-core, 6-core, and 8-core intel xeon ex processors. You can combine certain models of the x3690 x5 servers with the ibm max5 memory expansion for up...

  • Page 138: 4.1  Product Features

    118 ibm ex5 implementation guide 4.1 product features the x3690 x5 is a 2u, 2-socket, scalable system that offers up to four times the memory capacity of current 2-socket servers. It has the following features: up to two sockets for intel xeon 6500 or xeon 7500 processors. Depending on the processor...

  • Page 139

    Chapter 4. Ibm system x3690 x5 119 figure 4-1 ibm system x3690 x5 figure 4-1 shows the x3690 x5 server with 16 hot-swap 2.5-inch sas disk drives installed. The x3690 server has these physical specifications: height: 86 mm (3.5 inches, 2u) depth: 698 mm (27.4 inches) width: 429 mm (16.8 inches) maxim...

  • Page 140

    120 ibm ex5 implementation guide figure 4-3 rear view of x3690 x5 figure 4-4 shows the system with the top cover removed. Figure 4-4 the x3690 x5 internals pci slot 1 pci slot 2 pci slot 3 pci slot 4 pci slot 5 power supply 3 power connectors system management ethernet connector video serial connect...

  • Page 141

    Chapter 4. Ibm system x3690 x5 121 4.1.2 ibm max5 memory expansion unit the ibm max5 for system x (max5) memory expansion unit has 32 ddr3 dimm sockets, one or two 675-watt power supplies, and five 40 mm hot-swap speed-controlled fans. It provides added memory and multinode scaling support for host ...

  • Page 142

    122 ibm ex5 implementation guide five hot-swap 40 mm fans. Power supply: – hot-swap power supplies with built-in fans for redundancy support. – 675-watt (110 - 220 v ac auto-sensing). – one power supply standard; a second redundant power supply is optional. Light path diagnostics leds: – board led –...

  • Page 143: 4.2  Target Workloads

    Chapter 4. Ibm system x3690 x5 123 figure 4-7 max5 memory expansion unit internals for an in-depth look at the max5 offering, see 4.5, “max5” on page 127. 4.2 target workloads the x3690 x5 is an excellent choice for business applications that demand performance and memory. It provides maximum perfor...

  • Page 144: 4.3  Models

    124 ibm ex5 implementation guide 4.3 models in addition to the details in the tables in this chapter, each standard model has the following specifications: the servers have 16 dimm sockets on the system planar. The additional 16-dimm socket memory mezzanine (memory tray) is optional on most models a...

  • Page 145

    Chapter 4. Ibm system x3690 x5 125 workload-optimized x3690 x5 models table 4-2 lists the workload-optimized models. Model 3dx is designed for database applications and uses ssds for the best i/o performance. Backplane connections for sixteen 1.8-inch ssds are standard and there is space for an addi...

  • Page 146: 4.4  System Architecture

    126 ibm ex5 implementation guide 4.4 system architecture figure 4-8 shows the block diagram of the x3690 x5. Figure 4-8 x3690 x5 block diagram figure 4-9 shows the block diagram of the max5. The max5 is connected to the x3690 x5 using two cables connecting the two qpi ports on the server to two of t...

  • Page 147: 4.5  Max5

    Chapter 4. Ibm system x3690 x5 127 4.5 max5 as introduced in 4.1.2, “ibm max5 memory expansion unit” on page 121, the max5 memory expansion drawer is available for the x3690 x5. Certain standard models include the max5, as described in 4.3, “models” on page 124, and the max5 can also be ordered sepa...

  • Page 148: 4.6  Scalability

    128 ibm ex5 implementation guide the max5 is connected to the x3690 x5 using two cables, connecting the qpi ports on the server to two of the four qpi ports on the max5. The other two qpi ports of the max5 are unused. The exa ports are for future scaling capabilities. Figure 4-11 shows architectural...

  • Page 149

    Chapter 4. Ibm system x3690 x5 129 the x3690 x5 supports the following configurations: a single x3690 x5 server with two processor sockets. This configuration is sometimes referred to as a single-node server . A single x3690 x5 server with a single max5 memory expansion unit attached. This configura...

  • Page 150: 4.7  Processor Options

    130 ibm ex5 implementation guide 4.7 processor options several intel xeon 6500 and xeon 7500 processor options are available for the x3690 x5, as listed in table 4-5. Table 4-5 x3690 x5 processor options see 2.2, “intel xeon 6500 and 7500 family processors” on page 16 for an in-depth description of ...

  • Page 151: 4.8  Memory

    Chapter 4. Ibm system x3690 x5 131 the parallelization of workloads. When hyper-threading is engaged in the bios, for each processor core that is physically present, the operating system addresses two. For more information, see 2.2.2, “hyper-threading technology” on page 17. All cpu options include ...

  • Page 152

    132 ibm ex5 implementation guide the memory mezzanine is an optional component and orderable as listed in table 4-6. Table 4-6 x3690 x5 memory mezzanine option part number figure 4-13 shows the memory mezzanine and dimms. Figure 4-13 location of the memory dimms with the intel xeon 6500 and 7500 pro...

  • Page 153

    Chapter 4. Ibm system x3690 x5 133 the memory uses ddr3 technology and operates at memory speeds of 800, 978, and 1066 mhz. The memory speed is dictated by the memory speed of the processor (see table 4-5 on page 130). For more information about how this is calculated, see 2.3.1, “memory speed” on p...

  • Page 154

    134 ibm ex5 implementation guide figure 4-14 x3690 x5 planar showing memory dimm locations one or two processors without the memory mezzanine in this configuration, all of the system’s memory directly attaches to processor 1. If a second processor is installed, it accesses main memory through proces...

  • Page 155

    Chapter 4. Ibm system x3690 x5 135 table 4-8 one or two processor dimm installation when the memory mezzanine is not installed two processors with memory mezzanine installed with two processors installed in the system, memory is evenly distributed between both processors, which maximizes system perf...

  • Page 156

    136 ibm ex5 implementation guide figure 4-15 memory mezzanine tray table 4-9 numa-compliant dimm installation: two processors and the memory mezzanine installed 4.8.3 max5 memory the max5 memory expansion unit has 32 dimm sockets. It is designed to augment the memory that is installed in the attache...

  • Page 157

    Chapter 4. Ibm system x3690 x5 137 max5 memory options table 4-10 shows the available memory options that are supported in the max5 memory expansion unit. These options are a subset of the options that are supported in the x3690 x5 because the max5 requires that all dimms use identical dram technolo...

  • Page 158

    138 ibm ex5 implementation guide one important consideration when installing memory in max5 configurations is that the server must be fully populated before adding dimms to the max5. As we describe in 2.3.2, “memory dimm placement” on page 23, you get the best performance by using all memory buffers...

  • Page 159

    Chapter 4. Ibm system x3690 x5 139 max5 memory as seen by the operating system max5 is capable of two modes of operation in terms of the way that memory is presented to the operating system: memory in max5 can be split and assigned between the cpus on the host system ( non-pooled mode ). This mode i...

  • Page 160

    140 ibm ex5 implementation guide access only to remote or “far” memory. So, threads executing on processor 1 have a longer latency to access memory as compared to threads on processor 0. This situation is due to the latency penalty incurred to traverse the qpi links to access the data on the other p...

  • Page 161

    Chapter 4. Ibm system x3690 x5 141 memory channels, with configurations balanced across the memory buffers, so that hemisphere mode is maintained. Figure 4-18 relative memory performance using mixed dimms as you can see, mixing dimm sizes can cause performance loss up to 18%, even if all channels ar...

  • Page 162

    142 ibm ex5 implementation guide table 4-13 shows the dimm population order for memory-mirroring mode without the mezzanine installed. Table 4-12 mirror dimm installation: two processors and no memory mezzanine installed table 4-13 dimm population order: memory-mirroring mode without the mezzanine i...

  • Page 163

    Chapter 4. Ibm system x3690 x5 143 table 4-15 dimm population order: memory-mirroring mode with the mezzanine installed dimm installation: max5 table 4-16 shows the installation guide for max5 memory mirroring. Table 4-16 max5 memory mirroring setup 4.8.7 memory sparing sparing provides a degree of ...

  • Page 164

    144 ibm ex5 implementation guide these options are configured by using the uefi during the boot sequence. 4.8.8 effect on performance of using mirroring or sparing to understand the effect on performance of selecting various memory modes, we use a system configured with x7560 processors and populate...

  • Page 165: 4.9  Storage

    Chapter 4. Ibm system x3690 x5 145 4.9 storage the x3690 x5 has internal capacity of up to sixteen 2.5-inch drives, as shown in figure 4-20. The server supports 2.5-inch disk drives or solid-state drives (ssds), or 1.8-inch ssds. Figure 4-20 front of the x3690 x5 with sixteen 2.5-inch drive bays thi...

  • Page 166

    146 ibm ex5 implementation guide table 4-17 x3690 x5 hard drive backplanes as listed in table 4-17, the backplane option part numbers include the necessary cables to connect the backplane to the sas controller. The short sas cable is needed when installing a hard drive backplane for 2.5-inch bays 1 ...

  • Page 167

    Chapter 4. Ibm system x3690 x5 147 figure 4-21 and figure 4-23 on page 150 show the backplanes and their cable connections. Figure 4-21 the 2.5-inch sas backplanes (rear view) using 2.5-inch disk drives table 4-19 lists the 2.5-inch sas 10k and 15k rpm disk drives and the 2.5-inch ssds that are supp...

  • Page 168

    148 ibm ex5 implementation guide self-encrypting drives (seds) are also an available option as listed in table 4-19 on page 147. Seds provide cost-effective advanced data security with advanced encryption standard (aes) 128 disk encryption. To make use of the encryption capabilities, you must also u...

  • Page 169

    Chapter 4. Ibm system x3690 x5 149 follow these installation steps for the single sata drive: 1. Install the single sata hdd bay assembly into the last bay for drive bays 12 - 15. 2. Disconnect the optical drive’s cable from the planar connector. 3. Plug the cable from the single sata hdd bay into t...

  • Page 170

    150 ibm ex5 implementation guide listed in “workload-optimized x3690 x5 models” on page 125. You can add two more exflash backplanes to increase the supported number of ssds to 24. The ibm exflash 8x 1.8-inch hs sas ssd backplane, part number 60y0360, supports eight 1.8-inch high iops ssds, as shown...

  • Page 171

    Chapter 4. Ibm system x3690 x5 151 table 4-25 controllers supported with the exflash ssd backplane option if you have already set up the serveraid controller that you plan to use and you want to leave the battery attached, you can still disable the write back cache by going into the megaraid web bio...

  • Page 172

    152 ibm ex5 implementation guide serveraid m5000 series performance accelerator key serveraid m5000 series performance accelerator key for system x enables performance enhancements that are needed by the emerging ssd technologies being used in a mixed sas and ssd environment. You can enable these pe...

  • Page 173

    Chapter 4. Ibm system x3690 x5 153 serveraid br10i controller the serveraid-br10i has the following specifications: lsi 1068e-based adapter two internal mini-sas sff-8087 connectors sas 3 gbps pcie x8 host bus interface fixed 64 kb stripe size supports raid-0, raid-1, and raid-1e no battery and no o...

  • Page 174

    154 ibm ex5 implementation guide – performance optimization for ssds – lsi safestore: support for self-encrypting drive services, such as instant secure erase and local key management (which requires the use of self-encrypting drives) support for the optional m5000 advanced feature key, which enable...

  • Page 175

    Chapter 4. Ibm system x3690 x5 155 ibm 6gb ssd host bus adapter the ibm 6gb ssd host bus adapter is an ideal host bus adapter (hba) to connect to high-performance ssds. With two x4 sff-8087 connectors and a high performance powerpc i/o processor, this hba can support the bandwidth that ssds can gene...

  • Page 176

    156 ibm ex5 implementation guide battery trays on top of the memory tray or the dimm air baffle (whichever one is installed in the server). The battery trays are standard with the server. Each battery tray holds up to two batteries, to support a maximum of four raid adapters with attached batteries ...

  • Page 177

    Chapter 4. Ibm system x3690 x5 157 4.9.5 serveraid expansion adapter the serveraid expansion adapter, which is also known as the ibm x3690 x5 raid expansion adapter or ibm 4x4 drive backplane serveraid expansion adapter, is a sas expander. It allows you to create raid arrays of up to 16 drives and a...

  • Page 178

    158 ibm ex5 implementation guide 4.9.6 drive combinations the x3690 x5 drive subsystem is divided into four backplanes; each backplane can connect to either four 2.5-inch drives or eight 1.8-inch ssds. This section describes the supported combinations. A configuration with four drives figure 4-28 sh...

  • Page 179

    Chapter 4. Ibm system x3690 x5 159 figure 4-30 x3690 x5 with one ibm 8x 2.5-inch hs sas hdd backplane figure 4-31 illustrates the ibm exflash 8x sas ssd backplane, which requires two sas cables. With the exflash, eight drives can be used in the same space as four 2.5-inch drives. Figure 4-31 x3690 w...

  • Page 180

    160 ibm ex5 implementation guide figure 4-34 x3690 with one 8x 2.5-inch backplane and one exflash 8x 1.8-inch ssd backplane configurations with 16 drives figure 4-35 and figure 4-36 both show the full sixteen 2.5-inch drive configuration. Both configurations require four sas cables. Figure 4-35 x369...

  • Page 181

    Chapter 4. Ibm system x3690 x5 161 figure 4-38 x3690 with two 4x 2.5-inch backplanes and one ibm exflash 8x 1.8-inch ssd backplane figure 4-39 shows two 8x exflash backplanes. Using these two backplanes requires four sas cables. Figure 4-39 also shows the use of a single sata drive. Figure 4-39 x369...

  • Page 182

    162 ibm ex5 implementation guide configurations with 24 drives figure 4-42 shows two 4x backplanes and 2 8x exflash backplanes. The 8x backplane can be used here instead of two 4x backplanes. Six sas cables are required. Figure 4-42 x3690 x5 with two 4x 2.5-inch backplanes and two ibm exflash 8x 1.8...

  • Page 183

    Chapter 4. Ibm system x3690 x5 163 6 gbps throughput per sas port. 800 mhz powerpc processor with lsi sas2108 6 gbps raid on chip (roc) controller. Pci express 2.0 x8 host interface. 512 mb onboard data cache (ddr2 running at 800 mhz). Intelligent lithium polymer battery backup unit standard with up...

  • Page 184: 4.10  Pcie Slots

    164 ibm ex5 implementation guide 4.10 pcie slots the x3690 x5 provides five pcie 2.0 slots for add-in cards. Figure 4-44 shows the location of the slots as viewed from the rear of the server. Figure 4-44 x3690 x5 pcie slots these slots are connected to the planar through two riser cards, both of whi...

  • Page 185

    Chapter 4. Ibm system x3690 x5 165 4.10.1 riser 1 in standard x3690 x5 models, riser slot 1 has the 2x8 riser card installed (60y0329, feature 9285) which has the following slots: slot 1, pcie 2.0 x8 full height, full length slot slot 2, pcie 2.0 x8 full height, half length slot the 2x8 riser can be...

  • Page 186

    166 ibm ex5 implementation guide 4.10.3 emulex 10gb ethernet adapter as described in 4.3, “models” on page 124, certain models include the emulex 10gb ethernet adapter as standard. The card is installed in pcie slot 5. Slot 5 is a nonstandard x8 slot, which is slightly longer than normal. It accepts...

  • Page 187

    Chapter 4. Ibm system x3690 x5 167 table 4-33 emulex adapter part numbers general details about this card are in emulex 10gb virtual fabric adapter for ibm system x, tips0762, which is available at the following website: http://www.Redbooks.Ibm.Com/abstracts/tips0762.Html the emulex 10gb ethernet ad...

  • Page 188

    168 ibm ex5 implementation guide sfp+ transceivers are not included with the server and must be ordered separately. Table 4-34 lists compatible transceivers. Table 4-34 transceiver ordering information 4.10.4 i/o adapters table 4-35 on page 168 shows the current list in the configuration and options...

  • Page 189: 4.11  Standard Features

    Chapter 4. Ibm system x3690 x5 169 4.11 standard features in this section, we describe the standard, onboard features of the x3690 x5. This section covers the following topics: 4.11.1, “integrated management module” on page 169 4.11.2, “ethernet subsystem” on page 170 4.11.3, “usb subsystem” on page...

  • Page 190

    170 ibm ex5 implementation guide serial over lan (sol) active energy manager power/reset control (power on, hard/soft shutdown, hard/soft reset, and schedule power control) 4.11.2 ethernet subsystem the x3690 x5 has an embedded dual 10/100/1000 ethernet controller. The bcm5709c is a single-chip, hig...

  • Page 191

    Chapter 4. Ibm system x3690 x5 171 figure 4-47 x3690 x5 light path diagnostics panel light path diagnostics can alert the user to the following errors: over current faults fan faults power supply failures pci errors you can obtain the full details about the functions and operation of light path diag...

  • Page 192

    172 ibm ex5 implementation guide figure 4-48 x3690 x5 fans figure 4-49 shows the top of the server and the hot-swap fan access panel. Figure 4-49 hot-swap fan access panel the following conditions affect system fan-speed adjustments: inlet ambient temperature cpu temperatures fan 1 fan 5.

  • Page 193: 4.12  Power Supplies

    Chapter 4. Ibm system x3690 x5 173 dimm temperatures altitude 4.12 power supplies this section covers the power subsystem of the x3690 x5 and the max5. 4.12.1 x3690 x5 power subsystem the x3690 x5 power subsystem consists of up to four hot-pluggable 675w auto-sensing power supplies. The modules are ...

  • Page 194

    174 ibm ex5 implementation guide table 4-37 power supply ac input requirements 4.12.2 max5 power subsystem the max5 power subsystem consists of one or two hot-pluggable 675w power supplies. The power subsystem is designed for n+n (fully redundant) operation and hot-swap replacement. Most standard mo...

  • Page 195

    Chapter 4. Ibm system x3690 x5 175 figure 4-50 low profile x8 riser card with hypervisor flash usb connector the ibm usb memory key for virtualization is included in the virtualization-optimized models that are listed in 4.3, “models” on page 124, but it can be added to any x3690 x5 system. Table 4-...

  • Page 196: 4.15  Rack Mounting

    176 ibm ex5 implementation guide vmware esxi 4.1 4.15 rack mounting the x3690 x5 is a 2u-high device (1u is one rack unit and is 1.75 inches). The max5 memory expansion unit is an additional 1u high unit. Both devices are designed to be installed in standard 19-inch racks. Three slide kits are avail...

  • Page 197: Ibm Bladecenter Hx5

    © copyright ibm corp. 2011. All rights reserved. 177 chapter 5. Ibm bladecenter hx5 the ibm bladecenter hx5 blade server showcases the ex5 architecture and technology in a blade form factor. This chapter introduces the server and describes its features and options. This chapter contains the followin...

  • Page 198: 5.1  Introduction

    178 ibm ex5 implementation guide 5.1 introduction the ibm bladecenter hx5 supports up to two processors using intel xeon 6500 (nehalem ex) 4-core, 6-core, or 8-core processors or four processors using the intel xeon 7500 nehalem ex 4-core, 6-core, or 8-core processors. The hx5 supports up to 40 dual...

  • Page 199

    Chapter 5. Ibm bladecenter hx5 179 figure 5-2 shows the components on the system board of the hx5. Figure 5-2 layout of hx5 (showing a 2-node 4-socket configuration) the max5 memory expansion blade, which is shown in figure 5-3 on page 180, is a device with the same dimensions as the hx5. When the m...

  • Page 200

    180 ibm ex5 implementation guide unit occupies two blade bays in the bladecenter chassis. The max5 cannot be removed separately from the hx5. Figure 5-3 max5 memory slots 5.1.1 comparison to the hs22 and hs22v the bladecenter hs22 is a general-purpose 2-socket blade server and the hs22v is a virtual...

  • Page 201: 5.2  Target Workloads

    Chapter 5. Ibm bladecenter hx5 181 5.2 target workloads the hx5 is designed for business-critical workloads, such as database and virtualization. Virtualization provides many benefits, including improved physical resource utilization, improved hardware efficiency, and reduced power and cooling expen...

  • Page 202: 5.3  Chassis Support

    182 ibm ex5 implementation guide 5.3 chassis support the hx5 is supported in bladecenter chassis s, h, and ht, as listed in table 5-3. Table 5-3 hx5 chassis compatibility (bc is bladecenter) the number of hx5 servers supported in each chassis depends on the thermal design power of the processors tha...

  • Page 203: 5.4  Models

    Chapter 5. Ibm bladecenter hx5 183 5.4 models the base models of the bladecenter hx5, with and without the max5 memory expansion blade, are shown in table 5-5. In the table, opt indicates optional and std indicates standard. Table 5-5 models of the hx5 also available is a virtualization workload-opt...

  • Page 204: 5.5  System Architecture

    184 ibm ex5 implementation guide table 5-6 workload-optimized models of the hx5 model 7872-68x is a virtualization-optimized model and includes the following features in addition to standard hx5 and max5 features: forty dimm sockets, all containing 4 gb memory dimms for a total of 160 gb of availabl...

  • Page 205: 5.6  Speed Burst Card

    Chapter 5. Ibm bladecenter hx5 185 figure 5-4 shows the hx5 block diagram. Figure 5-4 hx5 block diagram 5.6 speed burst card to increase performance in a 2-socket hx5 server (that is, with two processors installed), install the ibm hx5 1-node speed burst card. The 1-node speed burst card takes the q...

  • Page 206

    186 ibm ex5 implementation guide figure 5-5 hx5 1-node speed burst card block diagram figure 5-6 shows where the speed burst card is installed on the hx5. Figure 5-6 installing the speed burst card 5.7 ibm max5 for bladecenter ibm max5 for bladecenter, which is shown in figure 5-3 on page 180, is a ...

  • Page 207

    Chapter 5. Ibm bladecenter hx5 187 attachment to a single hx5 using the ibm hx5 max5 1-node scalability kit, part number 59y5877, as described in 5.8.3, “hx5 with max5” on page 190 communication with the processors on the hx5 using high-speed qpi links max5 is standard with certain models, as listed...

  • Page 208: 5.8  Scalability

    188 ibm ex5 implementation guide 5.8 scalability this section explains how the hx5 can be expanded to increase the number of processors and the number of memory dimms. The hx5 blade architecture allows for a number of scalable configurations, including the use of a max5 memory expansion blade, but t...

  • Page 209

    Chapter 5. Ibm bladecenter hx5 189 figure 5-8 2-node hx5 with the 2-node scalability card indicated the double-wide configuration consists of two connected hx5 servers. This configuration consumes two blade slots and has the 2-node scalability card attached. The scaling is performed through qpi scal...

  • Page 210

    190 ibm ex5 implementation guide figure 5-9 shows the block diagram of a 2-node hx5 and the location of the hx5 2-node scalability card. Figure 5-9 block diagram of a 2-node hx5 ensure that all firmware is up-to-date before attaching the blades together. For the minimum firmware requirements, see 8....

  • Page 211

    Chapter 5. Ibm bladecenter hx5 191 figure 5-10 single-node hx5 + max5 the card that is used to connect the max5 to the hx5 is the ibm hx5 max5 1-node scalability kit, which is extremely similar in physical appearance to the 2-node scalability kit that was shown in figure 5-8 on page 189. Table 5-10 ...

  • Page 212: 5.9  Processor Options

    192 ibm ex5 implementation guide having only one processor installed in the hx5 instead of two processors is supported; however, the recommendation is for two processors to maximize memory performance. Ensure that all firmware is up-to-date before attaching the max5 to the blade. For minimum firmwar...

  • Page 213

    Chapter 5. Ibm bladecenter hx5 193 follow these processor configuration rules: all installed processors must be identical. In a 2-node configuration, both two and four processors are supported. That is, each node can have one or two processors installed. All processors must be identical. If you only...

  • Page 214: 5.10  Memory

    194 ibm ex5 implementation guide 5.10 memory the hx5 2-socket has eight dimm sockets per processor (a total of 16 dimm sockets), supporting up to 128 gb of memory when using 8 gb dimms. With the addition of the max5 memory expansion blade, a single hx5 blade has access to a total of 40 dimm sockets ...

  • Page 215

    Chapter 5. Ibm bladecenter hx5 195 for optimal performance, populate all dimm slots on the hx5 before filling the max5. Each processor controls eight dimms and four memory buffers in the server, as shown in figure 5-12. To make use of all 16 dimm sockets, you must install both processors. If only on...

  • Page 216

    196 ibm ex5 implementation guide figure 5-14 dimm layout on the max5 system board max5 memory runs at 1066, 978, or 800 mhz ddr3 speeds. The memory speed is dependent on the processor qpi speed in the hx5: a qpi speed of 6.4 ghz means the speed of the max5 memory is 1066 mhz. A qpi speed of 5.8 ghz ...

  • Page 217

    Chapter 5. Ibm bladecenter hx5 197 table 5-14 numa-compliant dimm installation for a single-node hx5 in a 2-node (4-socket) configuration with two hx5 servers, follow the memory installation sequence in both nodes. You must populate memory to have a balance for each processor in the configuration. F...

  • Page 218

    198 ibm ex5 implementation guide max5 memory population order with the configuration of an hx5 connected to a max5, follow these rules: install at least two dimms in the hx5 (four dimms if the hx5 has two installed processors). For the best memory performance, fully populate the hx5 by using the seq...

  • Page 219

    Chapter 5. Ibm bladecenter hx5 199 this list shows the correct dimms in each power domain: power domain a: 1 - 4 and 5 - 8 power domain b: 13 - 16 and 17 - 20 power domain c: 9 - 12 and 21 - 24 for the best memory performance, install the dimms by spreading them among all six memory buffers and all ...

  • Page 220

    200 ibm ex5 implementation guide fashion to provide a balanced system and populating all processors identically is also required by vmware. Looking at figure 5-16 as an example, processor 0 has dimms populated, but no dimms are populated that are connected to processor 1. In this case, processor 0 h...

  • Page 221

    Chapter 5. Ibm bladecenter hx5 201 table 5-17 dimm installation for memory mirroring: one processor table 5-18 lists the dimm installation sequence for memory-mirroring mode when two processors are installed. Table 5-18 dimm installation for memory mirroring: two processors dimm placement: max5 tabl...

  • Page 222

    202 ibm ex5 implementation guide 5.10.5 memory sparing the hx5 supports dimm sparing, but only on the dimms that are installed in the hx5, not in the max5. For more information about memory sparing, see “memory sparing” on page 29. Table 5-20 shows the installation order when one processor is instal...

  • Page 223: 5.11  Storage

    Chapter 5. Ibm bladecenter hx5 203 mirroring or sparing effect on performance to understand the effect on performance of selecting various memory modes, we use a system that is configured with x7560 processors and populated with sixty-four 4 gb quad-rank dimms. Figure 5-17 shows the peak system-leve...

  • Page 224

    204 ibm ex5 implementation guide figure 5-18 ssd expansion card for the hx5 (left: top view; right: underside view) the ssd expansion card can be installed in the hx5 in combination with a ciov i/o expansion card and cffh i/o expansion card, as shown in figure 5-19. Figure 5-19 placement of ssd expa...

  • Page 225

    Chapter 5. Ibm bladecenter hx5 205 table 5-23 supported ssds for more information about ssd drives and their advantages, see 2.8.1, “ibm exflash price-performance” on page 49. 5.11.2 lsi configuration utility figure 5-20 shows the lsi sas configuration utility window running on a 2-node hx5 with one...

  • Page 226

    206 ibm ex5 implementation guide figure 5-21 raid choices using the lsi configuration utility the following options are presented: create im volume: creates a raid-1 array raid 1 drives are mirrored on a 1 to 1 ratio. If one drive fails, the other drive takes over automatically and keeps the system ...

  • Page 227

    Chapter 5. Ibm bladecenter hx5 207 5.11.3 determining which ssd raid configuration to choose using an industry standard io tool that measures hard drive performance, we tested each available configuration type. We used two separate tests: one test using 50% sequential reads and 50% random writes (fi...

  • Page 228

    208 ibm ex5 implementation guide figure 5-24 connecting sas connectivity card and external sas solution 5.12 bladecenter pci express gen 2 expansion blade the ibm bladecenter pci express gen 2 expansion blade provides the capability to attach selected pci express cards to the hx5. This capability is...

  • Page 229: 5.13  I/o Expansion Cards

    Chapter 5. Ibm bladecenter hx5 209 the expansion blade has the following features: support for pcie 2.0 adapters in an expansion blade the expansion blade lets you install one or two standard form factor pcie 2.0 adapter cards in a bladecenter environment, enabling the use of specialized adapters or...

  • Page 230

    210 ibm ex5 implementation guide table 5-25 lists the ciov expansion cards that are supported in the hx5. Table 5-25 supported ciov expansion cards see the ibm serverproven compatibility website for the latest information about the expansion cards that are supported by the hx5: http://ibm.Com/server...

  • Page 231

    Chapter 5. Ibm bladecenter hx5 211 table 5-26 lists the supported cffh i/o expansion cards. Table 5-26 supported cffh expansion cards see the ibm serverproven compatibility website for the latest information about the expansion cards that are supported by the hx5: http://ibm.Com/servers/eserver/serv...

  • Page 232

    212 ibm ex5 implementation guide in a bladecenter s chassis, the cffh i/o expansion card requires a supported switch module in bay 2. When used in a bladecenter s chassis, a maximum of two ports are routed from the cffh i/o expansion card to the switch module in bay 2. See the ibm bladecenter intero...

  • Page 233

    Chapter 5. Ibm bladecenter hx5 213 5.14.3 integrated management module (imm) the hx5 blade server includes an imm to monitor server availability, perform predictive failure analysis and so forth, and trigger ibm systems director alerts. The imm performs the functions of the baseboard management cont...

  • Page 234

    214 ibm ex5 implementation guide 5.15 integrated virtualization the hx5 offers an ibm 2 gb usb flash drive option that is preloaded with either vmware esxi 4.0 update 1 or vmware esxi 4.1: esxi 4.0 is for 1-node hx5 configurations only. Esxi 4.1 is required for configurations with the max5 memory ex...

  • Page 235

    Chapter 5. Ibm bladecenter hx5 215 figure 5-28 two unpartitioned hx5s for information about setup options and common errors, see 8.6, “creating an hx5 scalable complex” on page 402. 5.17 operating system support the hx5 supports the following operating systems: microsoft windows server 2008 r2 micro...

  • Page 236

    216 ibm ex5 implementation guide suse linux enterprise server 11 with xen for amd64/em64t vmware esx 4.0 update 1 vmware esxi 4.0 update 1 vmware esx 4.1 vmware esxi 4.1 see the serverproven website for the most recent information: http://ibm.Com/systems/info/x86servers/serverproven/compat/us/nos/em...

  • Page 237: Implementing

    © copyright ibm corp. 2011. All rights reserved. 217 part 2 implementing scalability in this second part of the book, we provide detailed configuration and setup information to get your servers operational. We particularly focus on setting up max5 configurations of all three ex5 servers, as well as ...

  • Page 238

    218 ibm ex5 implementation guide.

  • Page 239

    © copyright ibm corp. 2011. All rights reserved. 219 chapter 6. Ibm system x3850 x5 and x3950 x5 the system x3850 x5 and x3950 x5 are enterprise-class intel processor-based servers for mission-critical applications. Scaling two of these servers together and virtualizing the resources allow this serv...

  • Page 240

    220 ibm ex5 implementation guide 6.1 before you apply power for the first time after shipping before you begin populating your server with all of its processors, memory, and pci adapters and before you install an operating system, perform the recommendations described in the following sections. 6.1....

  • Page 241

    Chapter 6. Ibm system x3850 x5 and x3950 x5 221 figure 6-1 cmos battery location 3. Use your finger to pry up the battery on the side closest to the neighboring ic chip. The battery easily lifts out of the socket. 4. After 30 seconds, insert one edge of the battery, with the positive side up, back i...

  • Page 242

    222 ibm ex5 implementation guide same server. Any operational differences in stepping levels are handled by the microcode of the processor, integrated management module (imm), and uefi. 6.2.1 minimum processors required the minimum number of processors required for the server to boot into any operat...

  • Page 243

    Chapter 6. Ibm system x3850 x5 and x3950 x5 223 memory is installed on memory cards 1, 2, 7, and 8. Performance improves as dimms are added to evenly populate all ranks on each memory card. All memory on memory cards 7 and 8 is not visible if processor 4 fails. The operational configuration for vmwa...

  • Page 244

    224 ibm ex5 implementation guide 6.2.3 processor installation order the following list shows the processor installation requirements: the server ships with a minimum of two processors, which are installed in sockets 1 and 4. You can install additional processors in sockets 2 and 3 in either order. Y...

  • Page 245

    Chapter 6. Ibm system x3850 x5 and x3950 x5 225 figure 6-2 processor installation and removal tool to use the tool, perform the following steps: 1. When the processor is not installed in the server, ensure that you place the processor flat on an antistatic bag or mat, with the contact pads facing do...

  • Page 246

    226 ibm ex5 implementation guide processing nodes in a high-performance cluster, database servers, or print servers for graphics printers require the best memory performance possible. Remember these considerations when installing memory in the system x3850 x5: due to the fact that no single processo...

  • Page 247

    Chapter 6. Ibm system x3850 x5 and x3950 x5 227 the dimm configuration is invalid. See the dimm placement tables in “x3690 x5 memory population order” on page 133 for details. A nonfunctional dimm, failed dimm slot, bent processor pin, or resource conflict with a pcie adapter. Swap the dimm with a f...

  • Page 248

    228 ibm ex5 implementation guide full memory test is if you have an intermittent memory problem that you are trying to isolate. Because the server identifies which specific dimms are experiencing excessive single-bit failures, it is far more efficient to swap reported dimms with similar dimms inside...

  • Page 249

    Chapter 6. Ibm system x3850 x5 and x3950 x5 229 6.3.2 memory fault tolerance for servers with high availability requirements, using the memory mirroring or memory sparing configuration allows the server to continue to function normally in the rare event of a memory failure. See 2.3.6, “reliability, ...

  • Page 250

    230 ibm ex5 implementation guide figure 6-6 on page 229 shows the memory configuration panel with your choice of performing memory sparing (memory spare mode) or memory mirroring (memory mirror mode), but not both options at the same time. Select the desired option and reboot the server. If your mem...

  • Page 251

    Chapter 6. Ibm system x3850 x5 and x3950 x5 231 fpga firmware flashed. The recommendation is to reflash the fpga on your system x3850 x5 after connecting the max5 to ensure that the fpga levels on the max5 and server match. The server might fail to complete post because of a significant difference i...

  • Page 252

    232 ibm ex5 implementation guide keep a set of these plastic boots for times when you want to remove the qpi cables for the movement of equipment from rack to rack or for servicing the unit. Figure 6-7 reusable qpi cable connector protective boot there are four qpi cables connecting all of the qpi p...

  • Page 253

    Chapter 6. Ibm system x3850 x5 and x3950 x5 233 table 6-3 qpi port relationship to the processor socket figure 6-9 shows the back of the x3850 x5 with the attached max5. Figure 6-9 lab photo of an x3850 x5 (top) attached to a max5 (bottom) 6.4.4 accessing the dimms in the max5 to add or subtract mem...

  • Page 254

    234 ibm ex5 implementation guide chassis, it is not a requirement to remove the qpi cables before removing the memory board. 2. Remove the front bezel of the max5 by pressing in on the tab buttons on both sides of the bezel. The bezel can be pulled away from the max5. 3. As shown in figure 6-10, the...

  • Page 255

    Chapter 6. Ibm system x3850 x5 and x3950 x5 235 after you install and configure the max5 properly, a successful link between the server and the max5 can be confirmed when both units power on and off as one complete unit and the memory in the max5 can be seen by the server. You also see the following...

  • Page 256

    236 ibm ex5 implementation guide 6.5.2 processor requirements to maximize performance, both nodes must have all four processors and all memory cards installed and populated. However, the x3850 x5 does support 2-node configurations with only two processors installed in each node. The following list s...

  • Page 257

    Chapter 6. Ibm system x3850 x5 and x3950 x5 237 there are four qpi cables connecting all of the qpi ports of both x3850 x5 servers. Figure 6-13 shows the cable routing. Figure 6-13 cabling diagram for 2-node x3850 x5 consider this important information: the qpi cables are keyed to be inserted only o...

  • Page 258

    238 ibm ex5 implementation guide the cable end must slide into the port until it clicks into place. You can disengage the retainer that holds the cable in place by pressing on the blue tab on the top of the qpi cable connection. Qpi cables for connecting qpi ports 1 to 1 and 4 to 4 must be installed...

  • Page 259

    Chapter 6. Ibm system x3850 x5 and x3950 x5 239 stand-alone database server with built-in fault tolerance for power and cooling, processors, memory, and pcie buses. 6.6.1 generation 2 and generation 1 pcie adapters all of the pcie slots in the x3850 x5 are at generation 2 ( gen 2 ) specification. Be...

  • Page 260

    240 ibm ex5 implementation guide figure 6-15 qpi link frequency setting pcie adapter compatibility ibm serverproven tests ibm and non-ibm adapters that have been proven to function correctly in the system x3850 x5 server. See the following page for specifics: http://ibm.Com/systems/info/x86servers/s...

  • Page 261

    Chapter 6. Ibm system x3850 x5 and x3950 x5 241 figure 6-16 pcie slot speed selection panel to force gen1 compliance non-uefi adapters in an uefi environment a number of gen1 pcie adapters were designed prior to the implementation of uefi. As a result, these adapters are not recognized or might not ...

  • Page 262

    242 ibm ex5 implementation guide figure 6-17 legacy support panel another possible solution to this problem is booting the server in legacy only mode. This mode allows both non-uefi-aware operating systems and pcie adapters to function as they function on a non-uefi server. Many of the advanced memo...

  • Page 263

    Chapter 6. Ibm system x3850 x5 and x3950 x5 243 figure 6-18 selecting legacy only for a boot option besides adding legacy only to the boot manager, you must also change the sequence to place legacy only at the top of the boot sequence. To change the boot order from within the boot manager panel, sel...

  • Page 264

    244 ibm ex5 implementation guide 6.6.2 pcie adapters: slot selection a single-node x3850 x5 provides two independent zones of i/o interfaces. A 2-node x3850 x5 provides four independent zones of i/o interfaces. Each zone contains two processors and one i/o hub. Table 6-7 shows which i/o devices and ...

  • Page 265

    Chapter 6. Ibm system x3850 x5 and x3950 x5 245 qpi link. If two matched processors of any qpi link speed are installed, this limit is no longer an issue. Of all of the i/o adapters that can be installed on the server, the serveraid and 6 gb sas controllers managing ssd drives are the only adapters ...

  • Page 266

    246 ibm ex5 implementation guide figure 6-20 legacy option rom states when booting from san with multiple paths for redundancy, you will need to enable the legacy option rom for both hbas. The default uefi mode on the x3850 x5, you can sequence the order that the uefi will search the various attache...

  • Page 267

    Chapter 6. Ibm system x3850 x5 and x3950 x5 247 figure 6-21 disabling pxe boot of the onboard network ports other pci adapters can have their boot option rom disabled from within their configuration panels. To access individual adapter configuration panels from f1-setup, select system settings adapt...

  • Page 268

    248 ibm ex5 implementation guide to enter the configuration of a specific adapter, select the pciroot directly under the adapter name. When you have multiple controllers of the same type, selecting the pciroot of any of the same adapter types will select all of them and display a panel that allows y...

  • Page 269

    Chapter 6. Ibm system x3850 x5 and x3950 x5 249 installed in the server. To control the boot sequence, from within f1-setup, select system settings device and i/o protest option rom execution order. Figure 6-25 shows what the panel looks like after pressing enter on the list of possible choices. Use...

  • Page 270

    250 ibm ex5 implementation guide this precaution will eliminate the possibility of a single breaker or circuit fault from taking down the entire server. Think of the power supplies in your server as shock absorbers in your car. They are designed to absorb and overcome a wide variety of power conditi...

  • Page 271

    Chapter 6. Ibm system x3850 x5 and x3950 x5 251 6.8.1 imm network access the greatest strength of the imm is the ability to completely monitor and manage the server from over the network. How much functionally you have through this remote access depends entirely on your configuration of the imm. Imm...

  • Page 272

    252 ibm ex5 implementation guide management network traffic. Separating the traffic is done when you connect your public network port to switch ports that belong to a public access virtual lan (vlan) and the management port is connected to a switch port defined by a separate management vlan. Figure ...

  • Page 273

    Chapter 6. Ibm system x3850 x5 and x3950 x5 253 6.8.3 imm communications troubleshooting the integrated management module user guide is an excellent guide to help you with every aspect of configuring and using the imm. Download the guide from this website: http://ibm.Com/support/entry/portal/docdisp...

  • Page 274

    254 ibm ex5 implementation guide figure 6-30 integrated management module system status when an actual hardware error is detected in the server, the system status is represented by a red x. The system health summary will provide information about the errors presently unresolved in the server, as sho...

  • Page 275

    Chapter 6. Ibm system x3850 x5 and x3950 x5 255 figure 6-31 imm system status with a hard drive failure virtual light path diagnostics if you are standing in front of the server, it is easy to track this problem by noticing the first tier of light path diagnostics, with the error light on the operat...

  • Page 276

    256 ibm ex5 implementation guide most servers are not physically located near the people who manage them. To help you see the event from a remote location, the imm provides the capability of looking at all tiers of light path diagnostics, as shown in figure 6-33. Figure 6-33 integrated management mo...

  • Page 277

    Chapter 6. Ibm system x3850 x5 and x3950 x5 257 figure 6-34 integrated management module hardware event log remote control certain problems require that you get into the operating system or f1-setup to detect them or fix them. For remotely managed servers, you can use the remote control feature of t...

  • Page 278

    258 ibm ex5 implementation guide imm remote control provides the following features: the remote control provides you with the same capability that you have with a keyboard, mouse, and video panel directly connected to the server. You have the ability to encrypt the session when used over public netw...

  • Page 279: 6.9  Uefi Settings

    Chapter 6. Ibm system x3850 x5 and x3950 x5 259 6.9 uefi settings the unified extensible firmware interface (uefi) is the interface between the operating system (os) and platform firmware. Uefi provides a modern, well-defined environment for booting an os and running pre-boot applications. Uefi is e...

  • Page 280

    260 ibm ex5 implementation guide table 6-8 uefi settings, asu values, and default settings uefi value asu value asu settings default processor settings turbomode uefi.Turbomodeenable enable disable enable turboboost power optimization uefi.Turbboost traditional poweroptimized poweroptimized processo...

  • Page 281

    Chapter 6. Ibm system x3850 x5 and x3950 x5 261 6.9.1 settings needed for 1-node, 2-node, and max5 configurations use these settings for each configuration: 1-node configuration if you use the factory default uefi settings, the machine can work in a 1-node configuration. You can also change uefi set...

  • Page 282

    262 ibm ex5 implementation guide pooled this option presents the additional memory in the max5 as a pool of memory without being assigned to any particular processor. Use this setting with linux oss. 6.9.2 uefi performance tuning tuning the x3850 x5 for performance is a complicated topic, because it...

  • Page 283: 6.10  Installing An Os

    Chapter 6. Ibm system x3850 x5 and x3950 x5 263 a depending on the processor workload, enabling turbomode might also increase power consumption. The actual processing performance boost that you get from turbomode depends on the environment that the server is in terms of temperature and humidity, bec...

  • Page 284

    264 ibm ex5 implementation guide novell netware 6.5 for more information, see the integrated management module user's guide at the following web page: http://ibm.Com/support/entry/portal/docdisplay?Lndocid=migr-5079770 follow these steps to mount a drive through imm: 1. Connect to the imm with your ...

  • Page 285

    Chapter 6. Ibm system x3850 x5 and x3950 x5 265 figure 6-40 virtual media session window 5. Select the virtual media session window. 6. Click add image if you want to map an img or iso image file. 7. Select the check box next to the drive that you want to map and click mount selected, as shown in fi...

  • Page 286

    266 ibm ex5 implementation guide for linux, look on the vendor websites. They contain information about installation with a usb flash drive. For example, the following web pages provide details for using a usb key as an installation medium: installing red hat linux from a usb flash drive: http://ibm...

  • Page 287

    Chapter 6. Ibm system x3850 x5 and x3950 x5 267 3. Decompress the ibm_utl_sgtkwin_x.Xx_windows_32-64.Zip file to the directory that you have created, for example, c:\sgshare\sgdeploy . Creating a deployment image follow these steps to create a windows installation image. 1. Start the toolkit configu...

  • Page 288

    268 ibm ex5 implementation guide figure 6-44 define the source and target 5. When the copy process is finished, as shown in figure 6-45, click finish. Figure 6-45 the copy process successfully completes 6. Open a command prompt and change to the c:\sgshare\sgdeploy\sgtkwinpe directory. Use the follo...

  • Page 289

    Chapter 6. Ibm system x3850 x5 and x3950 x5 269 figure 6-46 build process is finished preparing the usb key follow these steps to create a bootable usb key with the windows installation image that was created in “creating a deployment image” on page 267: 1. Insert your usb key. 18:26:21 - creating t...

  • Page 290

    270 ibm ex5 implementation guide 2. Enter diskpart to format the usb key using fat32. All files on the usb key will be deleted. At the command prompt, type the commands that are listed in figure 6-47. Figure 6-47 using diskpart to format the usb memory key 3. Copy the contents from c:\sgshare\sgdepl...

  • Page 291

    Chapter 6. Ibm system x3850 x5 and x3950 x5 271 linux and vmware installation the procedure for linux and vmware is similar to the windows procedure: 1. Install the serverguide scripting toolkit. 2. Create a deployment image. 3. Prepare a usb key. For more information, see the ibm serverguide script...

  • Page 292

    272 ibm ex5 implementation guide setting the boot order to ensure that you can boot esxi successfully, you must change the boot order. The first boot entry must be legacy only and the second boot entry must be embedded hypervisor. Follow these steps: 1. Press f1 for the uefi setup. 2. Select boot ma...

  • Page 293

    Chapter 6. Ibm system x3850 x5 and x3950 x5 273 2. In the loading vmware hypervisor panel, press shift+o when the progress bar is displayed. 3. Enter the following command at the prompt: esxcfg-advcfg -k true allowinterleavednumanodes 4. After the system boots, connect to the system using the vspher...

  • Page 294

    274 ibm ex5 implementation guide figure 6-52 editing the wmkernel settings in vsphere client advanced settings 7. Click ok to save the settings. Updating esxi you can install the latest version of esxi 4 on ibm hypervisor keys and it is supported by ibm. Use the following vmware upgrade mechanisms f...

  • Page 295

    Chapter 6. Ibm system x3850 x5 and x3950 x5 275 to order a recovery cd, contact your local support center at the following website: http://www.Ibm.Com/planetwide/region.Html 6.10.3 installing the esx 4.1 or esxi 4.1 installable onto x3850 x5 before installing any vmware os, always refer to the lates...

  • Page 296

    276 ibm ex5 implementation guide installing the esx 4.1 os vmware esx 4.1 is not uefi aware, and therefore, you must use the legacy only option as the first boot option. To decrease the boot time, the optimal minimum configuration is legacy only, cd/dvd rom, and hard disk 0. Use the following steps ...

  • Page 297

    Chapter 6. Ibm system x3850 x5 and x3950 x5 277 9. The boot options line will appear on the panel. Type the following parameter at the end of the boot options line: allowinterleavednumanodes=true the edited result looks like figure 6-55. Press enter to proceed. Figure 6-55 editing the boot options 1...

  • Page 298

    278 ibm ex5 implementation guide 11.Proceed through the installer to the set bootloader options page and type the following parameter in the kernel arguments text box, as shown in figure 6-57: allowinterleavednumanodes=true figure 6-57 editing the bootloader options 12.Complete the remainder of the ...

  • Page 299

    Chapter 6. Ibm system x3850 x5 and x3950 x5 279 figure 6-59 installing esxi 4.1 installable edition 9. Add the following line after vmkboot.Gz: allowinterleavednumanodes=true ensure that you leave a space at the beginning and the end of the text that you enter (as shown in figure 6-60); otherwise, t...

  • Page 300

    280 ibm ex5 implementation guide figure 6-61 loading vmware hypervisor panel figure 6-62 esxi 4.1 installable numa error 12.Enter the following command at the prompt after you have pressed shift+o: esxcfg-advcfg -k true allowinterleavednumanodes your output looks like figure 6-63 on page 281. Import...

  • Page 301

    Chapter 6. Ibm system x3850 x5 and x3950 x5 281 figure 6-63 loading vmware hypervisor boot command 13.Press enter after the command has been entered. Press enter again to continue to boot. 14.After the system boots, connect to it using the vsphere client. 15.Select the configuration tab of the host ...

  • Page 302

    282 ibm ex5 implementation guide figure 6-65 editing the vmkernel settings in vsphere client attaching max5 to an existing vmware esx/esxi 4.1 installation next, we add the max5 memory option to an existing vmware esx/esxi installation on the x3850 x5. Perform these actions: 1. To successfully scale...

  • Page 303

    Chapter 6. Ibm system x3850 x5 and x3950 x5 283 figure 6-66 editing the vmkernel settings in vsphere client 4. Power off the x3850 x5 and unplug all power cords. 5. Attach the max5 to the x3850 x5. For more information, see 6.4, “attaching the max5 memory expansion unit” on page 230. 6. Replug the p...

  • Page 304

    284 ibm ex5 implementation guide figure 6-68 imm system health summary 14.Power on the x3850 x5. 15.At the uefi panel, press f1 for the uefi menu. 16.Select system settings memory and change the max5 memory scaling affinity to non-pooled, as shown in figure 6-69. Figure 6-69 uefi setting for max5 me...

  • Page 305

    Chapter 6. Ibm system x3850 x5 and x3950 x5 285 figure 6-71 vmware esx grub 2. Add the following line at the beginning: allowinterleavednumanodes=true ensure that you leave a space at the beginning and the end of the text that you enter (as shown in figure 6-72); otherwise, the command will fail to ...

  • Page 306

    286 ibm ex5 implementation guide 3. Click vmkernel in the left pane and select vmkernel.Boot.Allowinterleavednumanodes. Click ok. Figure 6-73 editing the wmkernel settings in vsphere client 4. Power off the x3850 x5 and unplug all power cords. 5. Attach the max5 to the x3850 x5. For more information...

  • Page 307

    Chapter 6. Ibm system x3850 x5 and x3950 x5 287 13.Wait approximately 10 minutes to power on the x3850 x5. There must be no leds lit on the light path diagnostics panel. The system health status in the imm web interface looks like figure 6-75. Figure 6-75 imm system health summary 14.Power on the x3...

  • Page 308

    288 ibm ex5 implementation guide if this error occurs, follow these steps to change the boot code: 1. In the loading vmware hypervisor panel, press shift+o when the progress bar is displayed, as shown in figure 6-78. Figure 6-78 loading wmware hypervisor panel 2. Enter the following command at the p...

  • Page 309

    Chapter 6. Ibm system x3850 x5 and x3950 x5 289 microsoft windows server 2008 this section provides useful information to assist with the installation of windows server 2008. The installation process has not been covered in this ibm redbooks publication, because there are no particular deviations fr...

  • Page 310

    290 ibm ex5 implementation guide consider these useful tips: the dual-port emulex 10gbe nic driver is not included on the windows server 2008 dvd and must be downloaded separately (or use serverguide instead). You must ensure that the flow control is enabled under network adapter properties after up...

  • Page 311

    Chapter 6. Ibm system x3850 x5 and x3950 x5 291 the following websites contain additional os installation guides: installing vmware esx server 4.1 - ibm system x3850 x5 (7145): http://ibm.Com/support/entry/portal/docdisplay?Lndocid=migr-5086213 installing vmware esxi server 4.1 installable - ibm sys...

  • Page 312

    292 ibm ex5 implementation guide the following websites contain additional os installation guides: installing red hat enterprise linux version 6 - ibm system x3850 x5 (type 7145) http://ibm.Com/support/entry/portal/docdisplay?Lndocid=migr-5086423 installing red hat enterprise linux version 5 update ...

  • Page 313

    Chapter 6. Ibm system x3850 x5 and x3950 x5 293 the recommendation is to review all retain tips related to this system and the os you are installing. See the following website: http://ibm.Com/support/entry/portal/problem_resolution/hardware/systems/system_ x/system_x3850_x5 6.10.5 downloads and fixe...

  • Page 314

    294 ibm ex5 implementation guide operating system updates, fixes, and patches the performance and reliability of an x3850 x5 tightly relate to the os running on it. Ibm supports an assortment of modern and widely used oss capable of utilizing the system’s potential. Each vendor supports their os by ...

  • Page 315

    Chapter 6. Ibm system x3850 x5 and x3950 x5 295 fibre channel over ethernet (fcoe) fcoe is the transport, or mapping, of encapsulated fc frames over the ethernet. The ethernet provides the physical interface, and fc provides the transport protocol. System setup for fcoe requires converged network ad...

  • Page 316

    296 ibm ex5 implementation guide if you do not have internal drives, disable the onboard sas raid controller by selecting system settings devices and io ports enable/disable onboard devices and disabling the sas controller or planar sas. Set the hba as the first device in the option rom execution or...

  • Page 317

    Chapter 6. Ibm system x3850 x5 and x3950 x5 297 ibm xiv storage system: architecture, implementation, and usage, sg24-7659 this book describes the concepts, architecture, and implementation of the ibm xiv® storage system: http://www.Redbooks.Ibm.Com/abstracts/sg247659.Html ibm midrange system storag...

  • Page 318

    298 ibm ex5 implementation guide figure 6-81 imm web interface from the system status pages, you can perform these tasks: monitor the power status of the server and view the state of the os. View the server temperature readings, voltage thresholds, and fan speeds. View the latest server os failure s...

  • Page 319

    Chapter 6. Ibm system x3850 x5 and x3950 x5 299 for more information and to download electronic service agent, go to the following website: https://www-304.Ibm.Com/support/electronic/portal/navpage.Wss?Category=5&locale=en _us 6.11.4 recovery process this section provides an overview of several reco...

  • Page 320

    300 ibm ex5 implementation guide 5. If the working node does not have hard drives installed, add the hard drives with the operating system from the second node. At the boot sequence, you will get a prompt to import the raid configuration. 6. Reconnect the power cords. 7. Power on the system. Power f...

  • Page 321: Ibm System X3690 X5

    © copyright ibm corp. 2011. All rights reserved. 301 chapter 7. Ibm system x3690 x5 the ibm system x3690 x5 offers flexibility in design to support a wide variety of uses. The x3690 x5 supports up to16 internal drive bays and offers up to 7.5 tb of redundant storage. This system is ideal for most ge...

  • Page 322

    302 ibm ex5 implementation guide 7.1 before you apply power for the first time after shipping before you begin populating your server with all of its processors, memory, and pci adapters, and before you install an operating system (os), perform the recommendations provided in this section. 7.1.1 ver...

  • Page 323

    Chapter 7. Ibm system x3690 x5 303 figure 7-2 location of switch block sw2 and the cmos memory battery c. The numbers on the switch block represent the off side of the switch. They are located on the side of the switch block that is closest to the front of the server. To clear cmos, slide switch 1 (...

  • Page 324

    304 ibm ex5 implementation guide e. Push the battery back into the socket with your finger and clip it back into place. 7.1.3 verify that the server will complete post before adding options when you have ordered options for your server that have not yet been installed, it is a good idea to ensure th...

  • Page 325

    Chapter 7. Ibm system x3690 x5 305 figure 7-4 processor locations in relation to surrounding components 7.2.2 processor operating characteristics table 7-1 describes the operating characteristics of the server based on the number of processors and how the memory is installed. The table also describe...

  • Page 326: 7.3  Memory Considerations

    306 ibm ex5 implementation guide 7.3 memory considerations section 4.8, “memory” on page 131 covers all of the various technical considerations regarding memory configuration for the system x3690 x5. There is a great deal of flexibility regarding the memory configuration of this server. As a result,...

  • Page 327

    Chapter 7. Ibm system x3690 x5 307 any memory installed on the optional mezzanine board will not be seen by processor 1 when processor 2 is not installed or operational. For nonuniform memory access (numa)-aware oss, when two processors are installed, you must install the same amount of memory on bo...

  • Page 328

    308 ibm ex5 implementation guide figure 7-5 how to access diagnostics in post 2. When the built-in diagnostics start, the diagnostics start the quick memory test for all of the memory in the server, as shown in figure 7-6 on page 309. You can stop the quick memory test at any time and run a full mem...

  • Page 329

    Chapter 7. Ibm system x3690 x5 309 figure 7-6 the start-up panel for the built-in diagnostics 3. The quick diagnostics continue to run, reporting each test that the quick diagnostics are currently performing and the length of time that it will take to complete that test. If an error occurs, the quic...

  • Page 330

    310 ibm ex5 implementation guide 7.3.3 memory fault tolerance for servers with high availability requirements, using the memory mirroring or memory sparing configuration allows the server to continue to function normally in the rare event of a memory failure. See 2.3.6, “reliability, availability, a...

  • Page 331: 7.4  Max5 Considerations

    Chapter 7. Ibm system x3690 x5 311 7.4 max5 considerations on top of the half of a terabyte (tb) of memory that can be configured in the x3690 x5, an additional half of a tb of memory can also be configured in the max5 memory expansion unit and attached to the server to increase overall memory acces...

  • Page 332

    312 ibm ex5 implementation guide the recommended sequence of updates is shown in table 7-2 on page 311. To update the server to this firmware level or a higher level, you can use one of the following choices: when you have a compatible os installed, you can use the updatexpress system packs installe...

  • Page 333

    Chapter 7. Ibm system x3690 x5 313 figure 7-9 reusable qpi cable connector protective boot there are only two qpi cables connecting the two qpi ports of the x3690 x5 to two of the four qpi ports on the max5. Figure 7-10 shows how the cables must be connected between the qpi ports of the two units. F...

  • Page 334

    314 ibm ex5 implementation guide figure 7-11 shows the back of the x3690 x5 with the max5 attached. Figure 7-11 lab photo of an x3690 x5 (top) attached to an ex5 max5 (bottom) 7.4.4 accessing the dimms in the max5 to access the memory dimms in the max5, slide the memory board of the max5 forward out...

  • Page 335

    Chapter 7. Ibm system x3690 x5 315 figure 7-12 blue release tabs for the memory tray cam levers 4. You can then pull the memory tray out about 30% before it stops. This design allows you to get a better grip on the tray on either side and then use your fingers to push in another set of blue release ...

  • Page 336

    316 ibm ex5 implementation guide 7.5 pcie adapters and riser card options this section describes considerations for determining how to use your pcie slots, depending on the types of pcie riser cards that you have installed. The x3690 x5 is designed to function in a wide variety of tasks, starting fr...

  • Page 337

    Chapter 7. Ibm system x3690 x5 317 figure 7-14 qpi link frequency setting backward compatibility of gen2 pcie slots to gen1 adapters although all gen2 pcie slots are backward compatible to gen1 adapters, not all pcie adapter vendors adopted the optional specification of gen1 that allows a gen2 pcie ...

  • Page 338

    318 ibm ex5 implementation guide figure 7-15 pcie slot speed selection panel to force gen1 compliance non-uefi adapters in an uefi environment a number of gen1 pcie adapters were designed prior to the implementation of uefi. As a result, these adapters are not recognized or might not have uefi drive...

  • Page 339

    Chapter 7. Ibm system x3690 x5 319 figure 7-16 legacy thunk support in uefi another possible solution to this problem is booting the server in legacy only mode. This mode allows both non-uefi-aware oss and pcie adapters to function as though they are on a non-uefi server. Many of the advanced memory...

  • Page 340

    320 ibm ex5 implementation guide figure 7-17 selecting legacy only for a boot option if you add legacy only to the boot manager, you must also change the boot sequence to place legacy only at the top of the boot sequence. To change the boot order from within the boot manager panel, select change boo...

  • Page 341

    Chapter 7. Ibm system x3690 x5 321 7.5.2 pcie adapters: slot selection the x3690 x5 offers a wide variety of pcie adapter configurations among possible riser card combinations, as described in 4.10, “pcie slots” on page 164. The two riser slots have these riser card options: riser slot 1 can contain...

  • Page 342

    322 ibm ex5 implementation guide of all of the i/o adapters that can be installed on the server, the serveraid and 6 gbps sas controllers, when managing solid-state drives (ssds), are the only adapters that can approach the theoretical limits of the x8 pcie 2.0 slot. When you use ssd drives, connect...

  • Page 343

    Chapter 7. Ibm system x3690 x5 323 figure 7-19 legacy option rom states when booting from san with multiple paths for redundancy, you need to enable the legacy option rom for both hbas. The default uefi mode on the x3690 x5, you can sequence the order in which the uefi will search the various attach...

  • Page 344

    324 ibm ex5 implementation guide figure 7-20 disabling pxe boot of the onboard network ports other pci adapters can have their boot option rom disabled from within their configuration panels. To access individual adapter configuration panels from f1-setup, select system settings adapters and uefi dr...

  • Page 345

    Chapter 7. Ibm system x3690 x5 325 to enter the configuration of a specific adapter, select the pciroot directly beneath the adapter name. When you have multiple controllers of the same type, selecting the pciroot of any of the same adapter types will select all of them and present a panel that allo...

  • Page 346

    326 ibm ex5 implementation guide rom execution order regardless of whether you are booting in legacy only mode or uefi mode, you can control from which device you want to boot. This capability is important when multiple storage adapters are installed in the server. To control the boot sequence from ...

  • Page 347

    Chapter 7. Ibm system x3690 x5 327 ensure that half of the power supplies of both units are plugged into one utility power source and the remaining half are plugged into a separate utility power source. This approach eliminates the possibility of a single breaker or circuit fault taking down the ent...

  • Page 348

    328 ibm ex5 implementation guide figure 7-26 integrated management module configuration panel 7.7.1 imm network access the greatest strength of the imm is the ability to completely monitor and manage the server from over the network. The configuration of the imm determines the amount of functionalit...

  • Page 349

    Chapter 7. Ibm system x3690 x5 329 panel of f1-setup. In f1-setup, you can access this panel by selecting system setting integrated management module network configuration, as shown in figure 7-27. Figure 7-27 imm network configuration panel when configured as dedicated, you connect to the network v...

  • Page 350

    330 ibm ex5 implementation guide figure 7-29 onboard ethernet port used when imm network interface is shared although this design eliminates a physical switch port and patch cable configuration, both the media access control (mac) address for the second ethernet port and the mac address for the imm ...

  • Page 351

    Chapter 7. Ibm system x3690 x5 331 auto-negotiate medium-dependent interface crossover (mdix) when they auto-negotiate speed and duplex settings. If you can ping the imm, you have a good direct network link. If the web session fails, perform the following steps: 1. Try a separate web browser. 2. Dir...

  • Page 352

    332 ibm ex5 implementation guide figure 7-30 integrated management module system status when an actual hardware error is detected in the server, the system status is represented by a red x and the system health summary provides information about the errors that are presently unresolved in the server...

  • Page 353

    Chapter 7. Ibm system x3690 x5 333 figure 7-31 imm system status with a hard drive failure virtual light path diagnostics it is easy to pinpoint a hardware fault when standing in front of the server, by noticing the first tier of light path diagnostics, with the error light on the front of the opera...

  • Page 354

    334 ibm ex5 implementation guide servers are not typically located near the people who manage them. To help you see the event from a remote location, the imm provides the capability to view all tiers of light path diagnostics, as shown in figure 7-33. Figure 7-33 integrated management module virtual...

  • Page 355

    Chapter 7. Ibm system x3690 x5 335 figure 7-34 integrated management module hardware event log remote control certain problems require that you view os logs or enter uefi via f1-setup to detect them or fix them. For remotely managed servers, you have the remote control feature of the integrated mana...

  • Page 356

    336 ibm ex5 implementation guide the remote control feature provides the following advantages: it gives you the same functionality that you have with a keyboard, mouse, and video panel directly connected to the server. You can encrypt the session when using the remote control feature over public net...

  • Page 357: 7.8  Uefi Settings

    Chapter 7. Ibm system x3690 x5 337 7.8 uefi settings hardware settings for the ibm system x3690 x5 are accessible through the uefi. It also provides lower-level hardware settings that will most likely be transparent to the os. Uefi replaces the old bios firmware interface. You can obtain more inform...

  • Page 358

    338 ibm ex5 implementation guide figure 7-38 uefi system configuration and boot management settings panel 7.8.1 scaled system settings when you attach the max5 memory expansion unit, additional settings are enabled in the uefi. Specifically, it adds the max5 memory scaling option in system settings ...

  • Page 359

    Chapter 7. Ibm system x3690 x5 339 figure 7-39 max5 memory scaling option in system x3690 x5 uefi the max5 memory scaling setting provides two options to determine how the system will present the memory capacity in the max5 unit to the running os: non-pooled the default option splits the memory in t...

  • Page 360

    340 ibm ex5 implementation guide you can obtain a complete list of supported windows server oss at ibm serverproven: http://ibm.Com/systems/info/x86servers/serverproven/compat/us/nos/microsoft.Html linux server settings the following linux server versions are supported: red hat enterprise linux 6 x6...

  • Page 361

    Chapter 7. Ibm system x3690 x5 341 figure 7-40 uefi operating modes panel as seen in figure 7-40, the operating modes settings provide the user with several pre-configured processor and memory settings that are defined by system characteristics, whether it is efficiency, acoustic, performance, or cu...

  • Page 362

    342 ibm ex5 implementation guide table 7-4 comparison of operating modes a complete description of power-related and performance-related system settings is available in 2.7.3, “performance-related individual system settings” on page 43. System settings for specific server workloads setting the indiv...

  • Page 363

    Chapter 7. Ibm system x3690 x5 343 7.8.4 optimizing boot options uefi systems, including the x3690 x5, can take more time to start up when scaled with the max5 expansion. The number of installed adapters directly affects the time that it takes for uefi to initialize. A simple step to reduce the over...

  • Page 364

    344 ibm ex5 implementation guide figure 7-41 uefi boot manager menu 3. Select delete boot option from the boot manager menu. 4. Select all items that you do not want to boot from by using the spacebar to select them. When you have selected all of the items, scroll down to the end of the page using t...

  • Page 365

    Chapter 7. Ibm system x3690 x5 345 figure 7-42 deleting boot options 5. Press esc when you have finished to return to the boot manager menu. 6. Select change boot order from the boot manager menu. 7. Press enter to make the device list active. 8. Use the up and down arrow keys to navigate to the dev...

  • Page 366

    346 ibm ex5 implementation guide 7.9 operating system installation scaling an x3690 x5 with max5 allows the os to access additional system memory that is installed in the max5. Most of the configurations are done on the hardware and firmware level, so that the os installation method is similar to th...

  • Page 367

    Chapter 7. Ibm system x3690 x5 347 figure 7-43 default uefi boot order in a typical configuration, these server functions are required to allow the server to perform pxe network boot and installation: dhcp server, providing ip address trivial file transfer protocol (tftp) server, providing initial b...

  • Page 368

    348 ibm ex5 implementation guide image loading using imm remote control integrated management module (imm) on ibm system x3690 x5 is equipped with a remote control function that is capable of mounting an installation image on the management workstation. It is accessed using the imm remote control op...

  • Page 369

    Chapter 7. Ibm system x3690 x5 349 figure 7-45 imm virtual media session interface follow this sequence to mount an image as virtual cd/dvd media at the controlled system: 1. Click add image. 2. Choose an image in . Iso or . Img format. 3. Check the box under map of the image to mount. 4. Click moun...

  • Page 370

    350 ibm ex5 implementation guide for the full details, see the ibm serverguide scripting toolkit, windows edition user’s reference and ibm serverguide scripting toolkit, linux edition user’s reference at the following website: http://ibm.Com/support/entry/portal/docdisplay?Lndocid=serv-toolkit windo...

  • Page 371

    Chapter 7. Ibm system x3690 x5 351 figure 7-46 ibm serverguide scripting toolkit 3. Choose the os type that you want and click next, as shown in figure 7-47. Figure 7-47 select the os type 4. Insert the correct os installation media or select the folder that contains the installation files for the s...

  • Page 372

    352 ibm ex5 implementation guide figure 7-48 define the source and target 5. When the file copy process completes successfully, as shown in figure 7-49, click finish. Figure 7-49 copy process completes successfully 6. Open a command prompt and change to the c:\sgshare\sgdeploy\sgtkwinpe directory. U...

  • Page 373

    Chapter 7. Ibm system x3690 x5 353 figure 7-50 build process is finished preparing the usb key follow these steps to create a bootable usb key with the windows installation image that was created in “creating a deployment image” on page 350: 1. Insert your usb key. 18:26:21 - creating the winpe x64 ...

  • Page 374

    354 ibm ex5 implementation guide 2. Use diskpart to format the usb key using fat32. All files on the usb key will be deleted. At the command prompt, type the commands that are listed in figure 7-51. Figure 7-51 using diskpart to format the usb memory key 3. Copy the contents from c:\sgshare\sgdeploy...

  • Page 375

    Chapter 7. Ibm system x3690 x5 355 linux and vmware installation the procedure for the linux and vmware installation is similar to the windows procedure: 1. Install the serverguide scripting toolkit. 2. Create a deployment image. 3. Prepare a usb key. For more information, see the ibm serverguide sc...

  • Page 376

    356 ibm ex5 implementation guide see 4.13, “integrated virtualization” on page 174 for ordering information for the integrated virtualization option. 7.9.3 windows server 2008 r2 windows server 2008 r2 is currently the latest version of the popular windows server os. It provides enhancements in seve...

  • Page 377

    Chapter 7. Ibm system x3690 x5 357 figure 7-53 ibm serverguide partition setup window we recommend that you map san storage logical drives after the os installation process completes, except in boot from san configurations. You can access more information about the windows server 2008 r2 installatio...

  • Page 378

    358 ibm ex5 implementation guide 7.9.4 red hat enterprise linux 6 and suse linux enterprise server 11 ibm system x3690 x5 with max5 memory expansion fully supports the latest version of two major linux server distributions: rhel6 from red hat and sles11 from novell. The installation of rhel6 and sle...

  • Page 379

    Chapter 7. Ibm system x3690 x5 359 4. Press enter after doing the modification. Figure 7-54 shows the modified boot menu that allows the esxi installation to proceed with the enabled interleaved numa nodes parameter. Figure 7-54 modified vmware vmvisor boot menu installation panel without this addit...

  • Page 380

    360 ibm ex5 implementation guide figure 7-56 edited esxi hypervisor loading panel 7. Set the allowinterleavednumanodes parameter to true in all subsequent system reboots. You are required to have vmware vsphere client to perform the following steps: a. After the esxi hypervisor is running, connect t...

  • Page 381

    Chapter 7. Ibm system x3690 x5 361 figure 7-57 vmware esxi 4.1 software advanced settings panel in addition to these required steps, use the tips in the following list for the vmware vsphere esxi installation on the x3690 x5: vmware vsphere 4.1 is the last release to support esx; all subsequent rele...

  • Page 382

    362 ibm ex5 implementation guide create a drive raid array before installation. The serveraid management interface is accessible via uefi settings, by selecting system settings adapters and uefi drivers. When running the esxi hypervisor from a usb flash drive, plug it in to the internal usb slot to ...

  • Page 383

    Chapter 7. Ibm system x3690 x5 363 figure 7-58 esx installation options window 3. The boot options line appears on the panel. Add the allowinterleavednumanodes=true parameter at the boot options line, so that the line reads in the following manner: boot options initrd=initrd.Img debuglogtoserial=1 v...

  • Page 384

    364 ibm ex5 implementation guide figure 7-59 edited esx installation boot options 4. Proceed with esx installation until you get to the setup type page, as shown in figure 7-60. Click advanced setup, clear configure bootloader settings automatically (leave checked if unsure), and click next. Figure ...

  • Page 385

    Chapter 7. Ibm system x3690 x5 365 figure 7-61 set bootloader options 6. Continue with the esx installation. 7.9.7 downloads and fixes for the x3690 x5 and max5 it is common during the support lifetime of a product that updates are released to provide users with enhanced capabilities, extended funct...

  • Page 386

    366 ibm ex5 implementation guide useful links: ibm bootable media creator (bomc) is a tool that simplifies the ibm system x firmware update process without an os running on the system. You can obtain more information about this tool at the following website: http://ibm.Com/support/entry/portal/docdi...

  • Page 387

    Chapter 7. Ibm system x3690 x5 367 table 7-6 internet links to support and downloads 7.9.8 san storage reference and considerations the ibm system x3690 x5 with its max5 memory expansion capability is considered a high-end server in the ibm system x product line. Target workloads for this design inc...

  • Page 388

    368 ibm ex5 implementation guide fibre channel over ethernet (fcoe) this relatively new interface was introduced to provide an easier method for fc storage environments to take advantage of the high performance 10 gb ethernet interface. Data communication uses fc protocol over a 10 gb ethernet inter...

  • Page 389

    Chapter 7. Ibm system x3690 x5 369 ibm midrange system storage implementation and best practices guide, sg24-6363 this book represents a compilation of best practices for deploying and configuring ibm midrange system storage servers, which include the ds4000 and ds5000 family of products. Http://www...

  • Page 390

    370 ibm ex5 implementation guide figure 7-62 example of imm error status system event logs system event logs in uefi and imm maintain histories of detected error events. Ibm dynamic system analysis (dsa) ibm dsa is a tool to collect and analyze complete system information to aid in diagnosing system...

  • Page 391

    Chapter 7. Ibm system x3690 x5 371 you can download the problem determination and service guide for the ibm system x3690 x5 from the following website: http://www.Ibm.Com/support/entry/portal/docdisplay?Lndocid=migr-5085205 7.10.2 system recovery in the unlikely event of a max5 power loss or severe ...

  • Page 392

    372 ibm ex5 implementation guide.

  • Page 393: Ibm Bladecenter Hx5

    © copyright ibm corp. 2011. All rights reserved. 373 chapter 8. Ibm bladecenter hx5 in this chapter, we take a closer look at the three scalable configuration options that are offered with the hx5. We describe the requirements that must be met for each of the scaling options with reference to hardwa...

  • Page 394

    374 ibm ex5 implementation guide 8.1 before you apply power for the first time after shipping before you begin populating your server with processors, memory, and expansion adapters, and before you install an operating system, perform the recommendations that are described in the following sections:...

  • Page 395

    Chapter 8. Ibm bladecenter hx5 375 8.1.2 clearing cmos memory when a server is shipped from one location to another location, you have no idea what the server was exposed to. For all you know, it might have been parked next to a large magnet or electric motor and everything in the server that stores...

  • Page 396

    376 ibm ex5 implementation guide 2. The numbers on the switch block represent the off side of the switch. Locate switch 5, which is the rtc switch. To clear the cmos, remove the plastic sheet from the jump block, as shown in figure 8-3 on page 375. Set switch 5 on switch block sw2 to the on position...

  • Page 397

    Chapter 8. Ibm bladecenter hx5 377 installed option rather than having to look at the entire server to try to find a good starting point during problem determination. 8.2 planning to scale: prerequisites as described in 5.8, “scalability” on page 188, the hx5 blade architecture allows for three scal...

  • Page 398

    378 ibm ex5 implementation guide table 8-1 shows the recommended installation guide to install the memory in a nonuniform memory access (numa)-compliant dimm installation for the hx5 2-node configuration. Table 8-1 numa-compliant dimm installation for the hx5 2-node for an hx5 with max5 configuratio...

  • Page 399

    Chapter 8. Ibm bladecenter hx5 379 8.2.3 required firmware of each blade and the amm if you plan to have a 2-node hx5 or hx5+max5 configuration, be prepared to install the latest versions of all firmware on the blade and the amm in the chassis. Without the proper versions of firmware, several issues...

  • Page 400

    380 ibm ex5 implementation guide figure 8-8 amm firmware version table 8-3 shows the minimum firmware levels that the hx5 must be running for a 2-node scaled configuration. It also shows the necessary firmware levels to scale with a max5. Table 8-3 minimum code levels required before scaling hx5 wit...

  • Page 401

    Chapter 8. Ibm bladecenter hx5 381 another characteristic of mismatched firmware is the following message when attempting to create the scale in the scalable complex, as shown in figure 8-10: “the complex state data cannot be read at this time. Check the event log to verify the completion of prior c...

  • Page 402: 8.3  Recommendations

    382 ibm ex5 implementation guide 8.3 recommendations in this section, we describe the considerations when creating a 2-node scale or adding a max5 to an hx5 blade. These settings help you to make decisions for performance and maximum uptime. 8.3.1 power sharing cap if installing a max5, ensure that ...

  • Page 403

    Chapter 8. Ibm bladecenter hx5 383 figure 8-13 blade showing power sharing cap and location 8.3.2 bladecenter h considerations blade server placement in a chassis is vitally important in balancing the power consumption and ensuring reliability. This section is only relevant if you use a bladecenter ...

  • Page 404

    384 ibm ex5 implementation guide figure 8-14 power management as shown from the amm figure 8-15 shows a breakdown of each domain and the current amount of power that is currently being used for each domain. Figure 8-15 main view when looking at the amm power management option on a bladecenter h you ...

  • Page 405

    Chapter 8. Ibm bladecenter hx5 385 figure 8-16 power consumption in one power domain a further consideration is when a 2-node or hx5+max5 configuration straddles the two power domains. If you install the primary blade server of a scalable blade complex in blade server bay 7 of a bladecenter h chassi...

  • Page 406

    386 ibm ex5 implementation guide the hx5 offers two methods to create raid arrays: either via the lsi setup utility in uefi or through the use of ibm serverguide. 8.4.1 launching the lsi setup utility the hx5 offers two ways of launching the lsi setup utility. Using uefi to launch the lsi setup util...

  • Page 407

    Chapter 8. Ibm bladecenter hx5 387 4. Select lsi logic sas controller and press enter, as shown in figure 8-18. Figure 8-18 adapter list in uefi you can now create an array. We describe creating a raid-1 array in 8.4.2, “creating a raid-1 mirror using the lsi setup utility” on page 389. Enabling leg...

  • Page 408

    388 ibm ex5 implementation guide figure 8-19 uefi system configuration and boot management menu 3. On the boot manager menu, select add boot option, as shown in figure 4 on page 389. Figure 8-20 uefi boot manager menu.

  • Page 409

    Chapter 8. Ibm bladecenter hx5 389 4. Select legacy only, as shown in figure 8-21, and press enter. The selection disappears when selected. If this option is not available, the legacy only option might already be part of the boot list. Figure 8-21 legacy only option in the boot list 5. When the syst...

  • Page 410

    390 ibm ex5 implementation guide figure 8-23 lsi array creation options you can use one of the following options to create arrays: create im volume: creates an integrated mirror (im) raid-1 array raid 1 drives are mirrored on a 1 to 1 ratio. If one drive fails, the other drive takes over automatical...

  • Page 411

    Chapter 8. Ibm bladecenter hx5 391 figure 8-24 creating an im volume 2. In figure 8-24, click no under raid disk to form the array. For each disk, you are prompted to confirm by pressing d, as shown in figure 8-25. Pressing d ensures that you understand that all data on this disk will be deleted. Fi...

  • Page 412

    392 ibm ex5 implementation guide figure 8-26 array with both drives created 4. After you have set both disks to yes , press c to create the array. This step completes creating and initializing a raid-1 array. 8.4.3 using ibm serverguide to configure the lsi controller ibm serverguide also lets you c...

  • Page 413

    Chapter 8. Ibm bladecenter hx5 393 figure 8-27 lsi setup using serverguide if desired, you can install an operating system on each node. Then, you can use this configuration to boot each blade individually if you need increased flexibility. You can use the temporary boot in stand-alone mode in the c...

  • Page 414

    394 ibm ex5 implementation guide figure 8-28 complex scale toggle for stand-alone mode 8.4.4 speed burst card reinstallation in the event that you decide to split a 2-node complex permanently in order to use the blades independently, always reinstall the speed burst card for performance reasons. The...

  • Page 415

    Chapter 8. Ibm bladecenter hx5 395 figure 8-29 hx5 1-node speed burst card block diagram figure 8-30 shows where the speed burst card is installed on the hx5. Figure 8-30 installing the speed burst card smi links qpi links 1 6 d d r 3 m e m o ry dim m s : one dim m per ch a n n e l memory buffer mem...

  • Page 416: 8.5  Uefi Settings

    396 ibm ex5 implementation guide 8.5 uefi settings unified extensible firmware interface (uefi) is an interface that takes care of handing over the pre-boot environment to the operating system. Uefi is effectively the replacement for bios. Bios has been around for many years but it was not designed ...

  • Page 417

    Chapter 8. Ibm bladecenter hx5 397 figure 8-32 system configuration and boot management uefi menu you can obtain more general information about uefi at the following website: http://www.Uefi.Org/home/ for an explanation of uefi settings, see 2.7, “uefi system settings” on page 36. Generally, the hx5...

  • Page 418

    398 ibm ex5 implementation guide a server is only as good as its weakest link, so you can have a well-tuned server, but a poorly configured san to which it connects. Generally, you need to look at an entire solution, including the san and network infrastructure, to fully achieve the best performance...

  • Page 419

    Chapter 8. Ibm bladecenter hx5 399 a simple step to reduce the overall system start-up time is to remove any unnecessary boot devices from the boot order list. In particular, ensure that pxe network boot is removed from the boot order list if it is not used. Pxe network boot can increase start-up ti...

  • Page 420

    400 ibm ex5 implementation guide figure 8-34 deleting boot options note that, in our example in figure 8-34, we also have selected hard disk 0 for removal. Our system is running windows 2008 with guid partition table (gpt) disk, and therefore, windows boot manager is being used to boot the operating...

  • Page 421

    Chapter 8. Ibm bladecenter hx5 401 8.5.4 hx5 2-node uefi settings no specific uefi settings are required for the hx5 to operate as a 2-node complex. The configuration of the complex is provided by the advanced management module (amm). See 8.6, “creating an hx5 scalable complex” on page 402 for instr...

  • Page 422

    402 ibm ex5 implementation guide use these general settings for each operating system type: pooled: use this setting with linux operating systems. Non-pooled: vmware and microsoft operating systems must be configured to use the non-pooled setting. 8.5.6 operating system-specific settings in uefi whe...

  • Page 423

    Chapter 8. Ibm bladecenter hx5 403 figure 8-36 scalable complex configuration panel 3. Ensure that both hx5s are powered off by looking at the status column in theunassigned nodes section. If the systems are not in a powered off state, the following warning message (figure 8-37) displays when you at...

  • Page 424

    404 ibm ex5 implementation guide figure 8-38 powering off an hx5 from the scalable complex configuration page 5. Select the two hx5 systems in the unassigned nodes area by clicking the check boxes next to them. The create partition action in the available actions pull-down list box is preselected fo...

  • Page 425

    Chapter 8. Ibm bladecenter hx5 405 figure 8-40 available actions to perform against the complex the following options are available: power off partition this option powers off all blades in the partition (in this example, it powers off blade 1 and blade 2). Power on partition this option powers on a...

  • Page 426

    406 ibm ex5 implementation guide 8.6.1 troubleshooting hx5 problems for general troubleshooting advice for the hx5, see the hx5 problem determination and service guide, which is available at this website: http://ibm.Com/support/entry/portal/docdisplay?Lndocid=migr-5084529 scalability card issues the...

  • Page 427

    Chapter 8. Ibm bladecenter hx5 407 8.7 operating system installation installing an operating system on an hx5 is practically identical to installing an operating system on any other ex5 system. The only difference, when installing from a remote location, is that the amm is used for remote control in...

  • Page 428

    408 ibm ex5 implementation guide figure 8-42 remote control function in the amm by clicking the remote drive icon (as highlighted in figure 8-43 on page 409), you can mount any local cd\dvd rom drive from the management workstation from which you are running the remote control session. You can also ...

  • Page 429

    Chapter 8. Ibm bladecenter hx5 409 figure 8-43 mounting physical media or disk images use the following steps to mount an image as virtual cd/dvd media at the controlled system: 1. Click select image under the available resources area of the remote disk panel, as shown in figure 8-43. 1. Click add i...

  • Page 430

    410 ibm ex5 implementation guide for linux, look on the vendor websites. They contain information about installation with a usb flash drive. For example, the following websites provide details for using a usb key as an installation medium: installing red hat linux from a usb flash drive: http://ibm....

  • Page 431

    Chapter 8. Ibm bladecenter hx5 411 b. Select add operating system installation files, as shown in figure 8-44. Figure 8-44 ibm serverguide scripting tool window c. Choose the operating system type that you want and click next, as shown in figure 8-45. Figure 8-45 select the type of the operating sys...

  • Page 432

    412 ibm ex5 implementation guide figure 8-46 define the source and target e. When the file copy process completes, as shown in figure 8-47, click finish. Figure 8-47 the file copy process completed successfully f. Open a command prompt and change to the c:\sgshare\sgdeploy\sgtkwinpe directory. Use t...

  • Page 433

    Chapter 8. Ibm bladecenter hx5 413 figure 8-48 build process is finished 3. Prepare the usb key follow these steps to create a bootable usb key with the windows installation image that was created in step 2 on page 410: a. Insert your usb key. B. Use diskpart to format the usb key using fat32. All f...

  • Page 434

    414 ibm ex5 implementation guide figure 8-49 using diskpart to format the usb memory key c. Copy the contents from c:\sgshare\sgdeploy\winpe_scenariooutput\local_win2008_r2_x64_ee\iso to the usb key. The usb key includes the folders and files that are shown in figure 8-50. Figure 8-50 contents of th...

  • Page 435

    Chapter 8. Ibm bladecenter hx5 415 linux and vmware installation the procedure for linux and vmware is similar to the windows procedure: 1. Install the serverguide scripting toolkit. 2. Create a deployment image. 3. Prepare a usb key. For more information, see the ibm serverguide scripting toolkit, ...

  • Page 436

    416 ibm ex5 implementation guide installing vmware esxi to successfully complete the installation of a supported version of esxi onto the hx5, complete the following tasks: 1. Install the system memory in a balanced configuration when installing the esxi operating system on the hx5, the memory must ...

  • Page 437

    Chapter 8. Ibm bladecenter hx5 417 b. Physically attach the max5 by using the instructions that are provided in the ibm bladecenter hx5 installation and user’s guide, which is available at this website: http://ibm.Com/support/entry/portal/docdisplay?Lndocid=migr-5084612 c. Reflash the fpga code to e...

  • Page 438

    418 ibm ex5 implementation guide figure 8-54 editing the wmkernel settings in the vsphere client k. Click ok. At this point, you have completed the process of configuring the embedded esxi 4.1 on the hx5 with max5. Attaching max5 to an existing embedded esxi 4.1 configuration the easiest way to add ...

  • Page 439

    Chapter 8. Ibm bladecenter hx5 419 figure 8-55 configuration tab software advanced settings 3. Click vmkernel in the left pane and select the check box next to vmkernel.Boot.Allowinterleavednumanodes,as shown in figure 8-56 on page 420..

  • Page 440

    420 ibm ex5 implementation guide figure 8-56 editing the wmkernel settings in vsphere client 4. Click ok. 5. Shut down the hx5. 6. Ensure that you have the latest fpga code installed on the hx5 by updating it, if required. 7. Physically attach the max5 by using the instructions that are provided in ...

  • Page 441

    Chapter 8. Ibm bladecenter hx5 421 to order a recovery cd, contact your local support center at the following website: http://www.Ibm.Com/planetwide/region.Html updating esxi you can install the latest version of esxi 4 on ibm hypervisor keys, and it is supported by ibm. Use the following vmware upg...

  • Page 442

    422 ibm ex5 implementation guide installing on an hx5 with max5 attached to correctly configure and install the esx 4.1 or esxi 4.1 installable editions of the esx server, follow these instructions. Common steps for both esx 4.1 and esxi 4.1 installable editions perform the following steps for both ...

  • Page 443

    Chapter 8. Ibm bladecenter hx5 423 figure 8-57 example of a boot order f. Select commit changes and press enter to save the changes. 2. Boot the host from the esx installation media. 3. Press f2 when you receive the installation options panel, as shown in figure 8-58. Figure 8-58 esx installation op...

  • Page 444

    424 ibm ex5 implementation guide figure 8-59 editing the boot options 5. Proceed through the installer until you reach the setup type page. Click advanced setup and clear configure boot loader automatically (leave checked if unsure),as shown in figure 8-60 on page 425..

  • Page 445

    Chapter 8. Ibm bladecenter hx5 425 figure 8-60 modifying the esx installation 6. Proceed through the installer to the set boot loader options page and type the following parameter in the kernel arguments text box, as shown in figure 8-61 on page 426: allowinterleavednumanodes=true.

  • Page 446

    426 ibm ex5 implementation guide figure 8-61 editing the boot loader options 7. Complete the remainder of the esx installation and reboot the host. For esxi 4.1 installable edition hosts use the following steps to install the esxi 4.1 installable onto the hx5 with max5 attached. To ensure that you c...

  • Page 447

    Chapter 8. Ibm bladecenter hx5 427 c. Select cd/dvd rom, legacy only, and hard disk 0, as shown in figure 8-62, for example, if you are using internal drives that have been configured as an integrated mirror with the onboard raid controller. If you cannot find these options, the options are already ...

  • Page 448

    428 ibm ex5 implementation guide figure 8-64 editing the boot load command 5. Complete the esxi installation and reboot when prompted. Ensure that you remove the media or unmount the installation image before the system restarts. 6. In the loading vmware hypervisor panel, when the gray progress bar ...

  • Page 449

    Chapter 8. Ibm bladecenter hx5 429 figure 8-65 loading vmware hypervisor boot command 8. Press enter after the command has been entered. Press enter again to continue to boot. 9. After the system boots, connect to it using the vsphere client. 10.Select the configuration tab of the host and click adv...

  • Page 450

    430 ibm ex5 implementation guide figure 8-66 configuration tab software advanced settings 11.Click vmkernel in the left pane and select the check box next to vmkernel.Boot.Allowinterleavednumanodes,as shown in figure 8-67 on page 431..

  • Page 451

    Chapter 8. Ibm bladecenter hx5 431 figure 8-67 editing the vmkernel settings in vsphere client 12.Click ok. At this point, you have completed the process of installing esxi 4.1 installable on the hx5 with max5. Attaching max5 to hx5 with an existing installation of esx 4.1 or esxi 4.1 installable ed...

  • Page 452

    432 ibm ex5 implementation guide figure 8-68 editing the wmkernel settings in vsphere client 4. Shut down the hx5. 5. Ensure that you have the latest fpga code installed on the hx5 by updating it, if required. 6. Physically attach the max5 using the instructions that are provided in the installation...

  • Page 453

    Chapter 8. Ibm bladecenter hx5 433 figure 8-69 vmware esx grub 3. Add the following line at the beginning of the boot load command: allowinterleavednumanodes=true ensure that you leave a space at the beginning and at the end of the text that you enter, as shown in figure 8-70. Otherwise, the command...

  • Page 454

    434 ibm ex5 implementation guide we recommend that you review all of the retain tips, which relate to this system and the operating system that you are installing, which are available at the following website: http://ibm.Com/support/entry/portal/troubleshooting/hardware/systems/bladecente r/bladecen...

  • Page 455

    Chapter 8. Ibm bladecenter hx5 435 the hx5 has not been zoned to external san storage at the time of installation unless you are performing boot from san. Failure to follow this rule potentially will result in unwanted data loss. Installing microsoft windows server 2008 r2 on hx5 without the use of ...

  • Page 456

    436 ibm ex5 implementation guide 8.7.5 red hat enterprise linux installation tips and settings this section provides useful information to assist you with the installation of red hat enterprise linux (rhel). We do not describe the red hat installation process in this chapter, because there are no pa...

  • Page 457

    Chapter 8. Ibm bladecenter hx5 437 red hat linux 5 software, firmware, drivers, and fixes for hx5 see the following website for all the latest information relating to software, firmware, drivers, and fixes for hx5 when used with red hat linux 5: http://www-933.Ibm.Com/support/fixcentral/systemx/quic...

  • Page 458

    438 ibm ex5 implementation guide troubleshooting information for hx5 and suse linux enterprise server we recommend that you review all retain tips relating to the hx5 and the operating system that you are installing. Go to the following website for troubleshooting information pertaining to hx5: http...

  • Page 459

    Chapter 8. Ibm bladecenter hx5 439 device drivers device drivers are software that controls the hardware server components on the operating system level. They are specific to the operating system version, and therefore, critical device drivers are included with the installation media. Ibm, operating...

  • Page 460

    440 ibm ex5 implementation guide 8.7.8 san storage reference and considerations bladecenters are designed to provide blades with usable storage via the external storage attachment due to the blade’s compact design. We highly recommend that you have an understanding of the terms and methods that rela...

  • Page 461

    Chapter 8. Ibm bladecenter hx5 441 figure 8-71 adapters visible in uefi if you do not have internal drives installed, disable the onboard sas raid controller in uefi by navigating to system settings devices and io ports enable/disable onboard devices and disabling the sas controller or planar sas. S...

  • Page 462

    442 ibm ex5 implementation guide ibm redbooks references for san-related information the following ibm redbooks publications describe ibm system storage products and their various implementations, including with the ibm system x product lines: ibm system storage solutions handbook, sg24-5250 this bo...

  • Page 463

    Chapter 8. Ibm bladecenter hx5 443 8.8.1 tools to aid hardware troubleshooting for the hx5 use the following tools when troubleshooting problems on the hx5 in any given configuration. Advanced management module the first place to start troubleshooting the hx5 is typically the advanced management mod...

  • Page 464

    444 ibm ex5 implementation guide system event log this log contains post and system management interrupt (smi) events and all events that are generated by the baseboard management controller (bmc) that is embedded in the imm. You can view the system event log through the uefi utility by pressing f1 ...

  • Page 465

    Chapter 8. Ibm bladecenter hx5 445 you might see the following effects of split power domains for hx5 with max5 configurations: a loss of power to either power domain results in complete system shutdown. The hx5 will not be able to power on if the attached max5 in slot 8 has no power, because the hx...

  • Page 466

    446 ibm ex5 implementation guide.

  • Page 467: Management

    © copyright ibm corp. 2011. All rights reserved. 447 chapter 9. Management in the information technology sector, server systems management has received greater focus over recent years. The ability to maintain and manage systems efficiently is essential to business it operations. We briefly describe ...

  • Page 468: 9.1  Introduction

    448 ibm ex5 implementation guide 9.1 introduction ibm provides a number of tools to successfully deploy, manage, and maintain the ex5 range of systems. The collective name for these tools is ibm toolscenter. You can access these tools at the following link: http://ibm.Com/support/entry/portal/docdis...

  • Page 469

    Chapter 9. Management 449 9.2 integrated management module (imm) the integrated management module (imm) offers these overall features and functions: provides diagnostics, virtual presence, and remote control to manage, monitor, troubleshoot, and repair from anywhere securely manages servers remotely...

  • Page 470

    450 ibm ex5 implementation guide figure 9-1 shows the location of the available ports to manage the imm on the x3690 x5. Figure 9-1 ports over which the imm can be managed out-of-band on an x3690 x5 figure 9-2 shows the location of the ports available to manage the imm on the x3850 x5. Figure 9-2 po...

  • Page 471

    Chapter 9. Management 451 figure 9-3 network configuration menu showing the network interface port settings 5. Set the network interface port setting to shared to allow the imm to use ethernet port 1. 6. For dhcp control, choose the static ip option. 7. For ip address, enter the relevant ip address....

  • Page 472

    452 ibm ex5 implementation guide figure 9-4 imm login web page enter the default user name, which is userid . This user name is case-sensitive. Enter the default password, which is passw0rd . You must use a zero in place of the letter “o” in the password. See the user’s guide for integrated manageme...

  • Page 473

    Chapter 9. Management 453 figure 9-5 amm login page after you log in to the amm, you can manage and retrieve information from the hx5, such as hardware vital product data (vpd), power usage, and so on. We describe more of this information in 9.3, “advanced management module (amm)” on page 454. 9.2.2...

  • Page 474

    454 ibm ex5 implementation guide 9.2.3 updating firmware using the imm through the integrated management module (imm), you can apply the following updates: imm uefi field programmable gate array (fpga) preboot dynamic system analysis (pdas) you can use the . Exe or . Sh file to make an update. For t...

  • Page 475

    Chapter 9. Management 455 figure 9-7 shows an amm. Figure 9-7 amm all bladecenter chassis come standard with at least one amm. Each chassis also supports a second management module for redundancy. One of the management modules is active, and the second management module, if installed, remains on sta...

  • Page 476

    456 ibm ex5 implementation guide discover other bladecenter units on the network and enable access to them through their management-module web interfaces control the bladecenter unit, blade servers, and other bladecenter components: – power on/off – firmware update – configuration settings – serial ...

  • Page 477

    Chapter 9. Management 457 figure 9-8 advanced management module login web page the system status summary page displays. This page provides you with information about the overall health of the chassis and its installed components, as shown in figure 9-9 on page 458..

  • Page 478

    458 ibm ex5 implementation guide figure 9-9 system status summary page all high-level tasks that can be performed against the various components within the bladecenter chassis are shown in the left pane, as highlighted in figure 9-9. Of particular importance to this ibm redbooks publication is the s...

  • Page 479

    Chapter 9. Management 459 event log and any configured alerts are sent. The information that is gathered by service advisor is the same information that is available if you save service data from the amm web interface. After gathering the information, service advisor automatically initiates a call t...

  • Page 480

    460 ibm ex5 implementation guide figure 9-10 service advisor settings 7. It is advisable to generate a test call to ibm to ensure that the bladecenter chassis can call home correctly. Select the test call home tab and click test call home. The test call home tab only appears if the service advisor i...

  • Page 481

    Chapter 9. Management 461 installation guide if you experience difficulties configuring the service advisor. You can obtain this guide at the following website: http://ibm.Com/support/entry/portal/docdisplay?Lndocid=migr-5073392 9.3.3 updating firmware using the amm through the amm, you can apply fo...

  • Page 482: 9.4  Remote Control

    462 ibm ex5 implementation guide figure 9-12 overview of the blades that you select for update 4. Use browse to locate the update file. 5. Click update to start the update. 9.4 remote control the ability to control the ex5 systems remotely is provided by the imm, in conjunction with the ibm virtual ...

  • Page 483

    Chapter 9. Management 463 4. If you want exclusive remote access during your session, click start remote control in single user mode. If you want to allow other users remote console (kvm) access during your session, click start remote control in multi-user mode. Figure 9-13 imm remote control page 5...

  • Page 484

    464 ibm ex5 implementation guide figure 9-14 imm remote control video viewer showing power control options.

  • Page 485

    Chapter 9. Management 465 figure 9-15 imm remote control virtual media session 9.4.2 accessing the remote control feature for the hx5 you manage the remote control feature for the hx5 through the advanced management module. Perform these steps: 1. To gain control of an hx5 blade, log in to the amm. ...

  • Page 486

    466 ibm ex5 implementation guide figure 9-16 remote control options page 3. To select a blade that you want to control, click start remote control. 4. To view the video of a blade, select the blade from the pull-down list box, as shown in figure 9-17 on page 467. You can obtain additional instructio...

  • Page 487

    Chapter 9. Management 467 figure 9-17 selecting which blade’s video output to display 9.5 ibm systems director 6.2 ibm systems director is a platform manager that offers the following benefits: enables the management of ex5 physical servers and virtual servers that are running on the ex5 platform. H...

  • Page 488

    468 ibm ex5 implementation guide ibm systems director provides the following functionality for ex5 systems, as well as many other system types: discovery and inventory monitoring and reporting software updates configuration management virtual resource management remote control, which includes power ...

  • Page 489

    Chapter 9. Management 469 figure 9-18 ibm systems director login web page 2. After you log in to the console, navigate to inventory system discovery in the left navigation pane. In the right pane, select a discovery option in the select a discovery option pull-down list. Ibm systems director default...

  • Page 490

    470 ibm ex5 implementation guide figure 9-19 ibm systems director system discovery pane 8. To rename the imm, right-click the imm server object in the discovered manageable systems area and click rename, as shown in figure 9-20. Figure 9-20 renaming an imm 9. A rename display box opens, as shown in ...

  • Page 491

    Chapter 9. Management 471 10.To authenticate to the imm, right-click no access under the access column and click request access, as shown in figure 9-22. Figure 9-22 request access option 11.The request access pane opens, as shown in figure 9-23. Enter the user name and password of an account that h...

  • Page 492

    472 ibm ex5 implementation guide 12.A successful authentication to an imm yields an ok access status, as shown in figure 9-24. Click close when finished. Figure 9-24 successful authentication to an imm 13.Click navigate resources in the left pane of ibm systems director. In the right pane, under gro...

  • Page 493

    Chapter 9. Management 473 complete these steps to discover and manage an x3850 x5 complex running a supported operating system via ibm systems director: 1. Verify that your systems are configured correctly as a complex by logging in to the imm web console and navigating to scalable partitioning mana...

  • Page 494

    474 ibm ex5 implementation guide figure 9-26 scalable complex with correctly configured access in ibm systems director 8. Discover the ip address of the operating system running on the primary node by navigating to inventory system discovery. 9. Specify the ip address of the operating system under s...

  • Page 495

    Chapter 9. Management 475 figure 9-27 deploying an agent to a managed system 14.The agent installation wizard window opens. Click next. 15.Click the platform agent packages group in the left pane and select the platform agent package that is relevant to your operating system. For linux and wmware op...

  • Page 496

    476 ibm ex5 implementation guide figure 9-28 platform agent selection pane 16.Click next. 17.Click next again. 18.Click finish. 19.You are prompted to either schedule the deployment of the agent or deploy it immediately. For our example, we deployed the agent immediately. Ibm systems director defaul...

  • Page 497

    Chapter 9. Management 477 9.5.3 discovering a single-node hx5 via ibm systems director managing an hx5 with ibm systems director provides you added benefits, including centralized management of hardware alerts and inventory capabilities. The methods that are used to discover the hx5 imm are similar ...

  • Page 498

    478 ibm ex5 implementation guide 7. Enter the user name and password credentials of an account that has supervisor access to the amm and click request access when finished. 8. Click close after the process completes. You return to the system discovery pane. 9. Close this pane and select navigate res...

  • Page 499

    Chapter 9. Management 479 for the purposes of this exercise, we demonstrate how to install a platform agent on an hx5 blade running windows server 2008. The method of deploying a platform agent to a linux or wmware esx server is the same. Complete these steps to discover and deploy a platform agent ...

  • Page 500

    480 ibm ex5 implementation guide 7. Specify the ip address of the operating system under select a discovery option. 8. Specify operating system as the resource type to discover under the select the resource type to discover list box. You must specify operating system to enable ibm systems director t...

  • Page 501

    Chapter 9. Management 481 17.You are prompted to either schedule the deployment of the agent or deploy it immediately. For our example, we deploy the agent immediately. Ibm systems director defaults to run now for all tasks unless specified otherwise. Click ok when ready to deploy the agent. Click d...

  • Page 502

    482 ibm ex5 implementation guide service and support manager prerequisites before installing service and support manager, you must meet the following prerequisites: the ibm systems director server must have access to the internet. The access can be granted via a proxy server, but the following confi...

  • Page 503

    Chapter 9. Management 483 figure 9-33 installing service and support manager 5. Select the language again that you want to use from the list box and click ok. 6. The introduction pane appears. Accept the license terms if prompted and click next. 7. Click the restart ibm systems director check box if...

  • Page 504

    484 ibm ex5 implementation guide figure 9-34 restart ibm systems director option 8. Click install. The service and support manager installer stops the ibm systems director server service. The installation completes. Click done when finished. The ibm systems director server service starts after the i...

  • Page 505

    Chapter 9. Management 485 figure 9-35 service and support manager plug-in 11.Click getting started with electronic service agent, as shown in figure 9-35. 12.The service and support manager wizard welcome pane opens. Click next. 13.Enter your company contact information in the provided fields. The m...

  • Page 506

    486 ibm ex5 implementation guide figure 9-36 service and support manager company contact details 15.Provide default details for the physical location of your systems, as shown in figure 9-37. You can change individual system location details at a later stage using the details that are provided in “c...

  • Page 507

    Chapter 9. Management 487 16.On the connection pane, leave the default settings if your ibm systems director server has a direct connection to the internet, as shown in figure 17. Enter the proxy server details if you are required to connect to the internet via a proxy server. Always ensure that you...

  • Page 508

    488 ibm ex5 implementation guide information center, which is available at this website, for further details regarding service and support manager: http://publib.Boulder.Ibm.Com/infocenter/director/v6r2x/index.Jsp changing the location settings of a system perform these steps to change the location ...

  • Page 509

    Chapter 9. Management 489 figure 9-40 powering on a complex via the operating system object 2. Click ok when the task launch dialog pane appears to run the task immediately. The system powers on normally. The power control menus within ibm systems director are adaptive, which means that they change ...

  • Page 510

    490 ibm ex5 implementation guide 3. To view the installed software, right-click the respective scalable partition object, select related resources, and then select installed software. All software, including system drivers, is displayed for that system. You can also view additional information about...

  • Page 511

    Chapter 9. Management 491 3. Right-click the imm of the server or blade that you want to identify and select system status and health system identification led on, as shown in figure 9-43. Figure 9-43 identifying a system within the ibm systems director console updating the firmware of a complex usi...

  • Page 512

    492 ibm ex5 implementation guide consider the following items when updating the system firmware on a complex or scalable partition: before starting any system firmware update processes, ensure that the multinode systems are discovered with both in-band and out-of-band methods. For more information, ...

  • Page 513

    Chapter 9. Management 493 8. A summary of the updates to be installed will be displayed, as shown in figure 9-45. Click finish when ready to proceed. Figure 9-45 summary of firmware to be installed 9. Click ok to run the firmware update immediately. The server will need to be rebooted so the update ...

  • Page 514

    494 ibm ex5 implementation guide the esa inventory does not include the following information: collection or transmission of any of your company’s financial, statistical, or personnel data client information your business plans the web component of ibm electronic services offers a single location fo...

  • Page 515

    Chapter 9. Management 495 9.7 advanced settings utility (asu) the advanced settings utility (asu) allows you to modify your server firmware settings from a command line. It supports multiple operating systems, such as linux, solaris, and windows (including windows preinstallation environment (pe)). ...

  • Page 516

    496 ibm ex5 implementation guide use the following procedure to download, install, and connect to the imm using a windows operating system: 1. Create a directory named asu. 2. Download the asu tool for your operating system (32-bit or 64-bit) at the following website and save it in the asu directory...

  • Page 517

    Chapter 9. Management 497 figure 9-47 files installed in the asu folder 4. Run a command, such as the asu show command, either in-band or out-of-band, using the commands that are listed in table 9-4. This command confirms that the connection and utility work. Table 9-4 steps to run asu both in-band ...

  • Page 518

    498 ibm ex5 implementation guide 9.7.2 common problems this section describes the problems that you might encounter running asu. The following reasons for errors are the most common: incorrect password or user firewall issues insufficient rights for the operating system link error example 9-2 shows ...

  • Page 519

    Chapter 9. Management 499 example 9-5 user rights in linux error ibm advanced settings utility version 3.61.70i licensed materials - property of ibm (c) copyright ibm corp. 2007-2010 all rights reserved user authority level is not sufficient. You must invoke asu over the network (--host --user --pas...

  • Page 520

    500 ibm ex5 implementation guide show all uefi settings on the command line, enter asu show uefi . See example 9-8. Example 9-8 output uefi.Operatingmode=custom mode uefi.Quietboot=enable uefi.Turbomodeenable=enable uefi.Turboboost=power optimized uefi.Processoreistenable=enable uefi.Processorccxena...

  • Page 521: 9.8  Ibm Serverguide

    Chapter 9. Management 501 enable hyper-threading on the command line, enter asu set uefi.Hyperthreading enable . See example 9-13. Example 9-13 output uefi.Hyperthreading=enable waiting for command completion status. Command completed successfully. Disable energy manager on the command line, enter a...

  • Page 522

    502 ibm ex5 implementation guide process. If you boot from san, make sure that you have only one path to the device because serverguide has no multipath support. See “booting from san” on page 295 for more details. After the serverguide installation procedure, you can attach external storage or acti...

  • Page 523

    Chapter 9. Management 503 figure 9-50 date and time settings 11.Create a raid configuration. If you previously created a raid adapter, serverguide detects this configuration and displays it under keep current adapter configuration. Select a raid configuration and click next, as shown in figure 9-51....

  • Page 524

    504 ibm ex5 implementation guide figure 9-52 raid configuration panel 13.Restart the server. After the server restarts, the server will restart again to complete the serverguide setup, as shown in figure 9-53. If you previously created a raid configuration and kept the configuration, restarting the ...

  • Page 525

    Chapter 9. Management 505 figure 9-54 selection for format and partition 15.When the process has completed, click next, as shown in figure 9-55. Figure 9-55 confirmation about completing the creation and formatting of the partition 16.Review the configuration, as shown in figure 9-56. Click next. Fi...

  • Page 526

    506 ibm ex5 implementation guide figure 9-57 serverguide copying files 18.When the process is finished, click next. Figure 9-58 confirmation about file copy success 19.Insert the operating system installation dvd and click next, as shown in figure 9-59 on page 507. Serverguide searches for the disc..

  • Page 527

    Chapter 9. Management 507 figure 9-59 prompt to insert the install medium 20.When serverguide successfully finds the installation medium, click next, as shown in figure 9-60. Figure 9-60 confirmation about finding installation medium 21.The windows setup installation procedure starts. Follow the ins...

  • Page 528

    508 ibm ex5 implementation guide if used with ibm serverguide and ibm updatexpress, the serverguide scripting toolkit provides a total solution for deploying ibm system x and bladecenter x86-based hardware in an unattended mode. The serverguide scripting toolkit enables you to create a bootable cd, ...

  • Page 529

    Chapter 9. Management 509 to download the scripting toolkit or the ibm serverguide scripting toolkit user’s reference, select the ibm serverguide scripting toolkit website: http://ibm.Com/support/entry/portal/docdisplay?Lndocid=serv-toolkit 9.10 firmware update tools and methods multiple methods exi...

  • Page 530

    510 ibm ex5 implementation guide figure 9-61 uefi setting for lan over usb for the hx5, from the amm web interface, click in the left panel blade tasks configuration advanced blade policy settings and ensure that the server’s ethernet over usb interface is enabled. See figure 9-62. Figure 9-62 amm s...

  • Page 531

    Chapter 9. Management 511 partition from which they are executed. You must update the firmware for each blade independently before rebooting either system. With imm firmware version 1.15, a dynamic host configuration protocol (dhcp) server is included in the imm. This dhcp assigns an ip address for ...

  • Page 532

    512 ibm ex5 implementation guide figure 9-63 welcome window 4. Accept the default update the local machine, as shown in figure 9-64. Click next. Figure 9-64 selecting the update task 5. Accept the default check the ibm web site, as shown in figure 9-65. Click next. Figure 9-65 checking the ibm websi...

  • Page 533

    Chapter 9. Management 513 figure 9-66 selecting the type of updates 7. Select the directory in which you want to store the downloaded files, as shown in figure 9-67. Click next. Figure 9-67 selecting your target directory 8. Enter the settings for an http proxy server, if necessary, or leave the che...

  • Page 534

    514 ibm ex5 implementation guide figure 9-71 overview of possible updates 12.When the update is finished, a message displays confirming the updates. Click next. Figure 9-72 updates are successful 13.Click finish to close the uxspi. 14.Restart the system to complete the update process. 9.12 bootable ...

  • Page 535

    Chapter 9. Management 515 creating an update media perform these steps to create an update media: 1. Create a folder named bomc . 2. Download the latest version of bomc from the following website and save it in the bomc folder: http://ibm.Com/support/entry/portal/docdisplay?Lndocid=tool-bomc 3. From...

  • Page 536

    516 ibm ex5 implementation guide figure 9-73 welcome panel 6. Check updates and click next, as shown in figure 9-74. Figure 9-74 selected update task 7. Select latest available individual updates and click next, as shown in figure 9-75 on page 517..

  • Page 537

    Chapter 9. Management 517 figure 9-75 select source of updates 8. Enter the settings for an http proxy server, if necessary, or check do not use proxy, as shown in figure 9-76. Click next. Figure 9-76 http proxy settings 9. Select one or more machine types that are contained on the bootable media an...

  • Page 538

    518 ibm ex5 implementation guide figure 9-77 the targeted systems panel 10.Select the directory where you want to store the downloaded files, as shown in figure 9-78. Click next. Figure 9-78 example of a target directory 11.By default, bomc creates an iso file, as shown in figure 9-79. You can choos...

  • Page 539

    Chapter 9. Management 519 figure 9-80 unattended mode panel 13.Review the selections and confirm that they are correct. Figure 9-81 shows an example. You can click save to save this configuration information to a file. Click next. Figure 9-81 confirm choices panel 14.Bomc acquires the files. In the ...

  • Page 540

    520 ibm ex5 implementation guide figure 9-82 downloading the files 15.After the update completes (figure 9-83), click next. Figure 9-83 confirmation that the creation process is finished 16.Click finish. Figure 9-84 finish panel 17.You now have a bootable image with the updates. You can mount or bur...

  • Page 541

    Chapter 9. Management 521 9.13 megaraid storage manager in this section, we provide an overview of the megaraid storage manager (msm) software. For more information, see the installation and user’s guide at the following website: http://ibm.Com/support/entry/portal/docdisplay?Lndocid=serv-raid with ...

  • Page 542

    522 ibm ex5 implementation guide to install the msm software, read the installation and user’s guide for msm at the following website. Http://ibm.Com/support/entry/portal/docdisplay?Lndocid=serv-raid you must have administrator or root equivalent operating system privileges to install and fully acce...

  • Page 543

    Chapter 9. Management 523 9.13.3 virtual drive states a virtual drive is a partition in a drive group that is made up of contiguous data segments on the drives. There are multiple virtual drive states. The following list describes all of the possible virtual drive states: optimal the virtual drive o...

  • Page 544

    524 ibm ex5 implementation guide sequence number: 4 media error count: 0 other error count: 0 predictive failure count: 0 last predictive failure event seq number: 0 pd type: sas raw size: 136.731 gb [0x11176d60 sectors] non coerced size: 136.231 gb [0x11076d60 sectors] coerced size: 135.972 gb [0x1...

  • Page 545: 9.14  Serial Over Lan

    Chapter 9. Management 525 example 9-16 output from command megacli -cfgldadd -r1[252:1,252:2] -a0 adapter 0: created vd 1 adapter 0: configured the adapter!! Exit code: 0x00 3. The virtual drive is successfully created. Additional command examples the following command examples use the megacli comma...

  • Page 546

    526 ibm ex5 implementation guide in this section, we describe these topics: 9.14.1, “enabling sol in uefi” on page 526 9.14.2, “bladecenter requirements” on page 527 9.14.3, “enabling sol in the operating system” on page 529 9.14.4, “how to start a sol connection” on page 533 9.14.1 enabling sol in ...

  • Page 547

    Chapter 9. Management 527 9.14.2 bladecenter requirements the bladecenter chassis must be correctly configured before you can use the cli and sol. Amm is able to communicate and manage the blades effectively using management virtual lan (vlan) 4095 through switches in bays 1 and 2. By default, all i...

  • Page 548

    528 ibm ex5 implementation guide 3. Ensure that the sol is enabled for the chassis. To verify, from the amm, select blade tasks serial over lan serial over lan configuration. Verify that serial over lan is enabled, as shown in figure 9-88. Figure 9-88 serial over lan configuration page 4. Finally, c...

  • Page 549

    Chapter 9. Management 529 9.14.3 enabling sol in the operating system in this section, we describe the settings to enable sol in these operating systems: “windows server 2008” “windows server 2003” on page 531 “linux” on page 533 the onboard broadcom nic driver is required to be at the latest versio...

  • Page 550

    530 ibm ex5 implementation guide c:\users\administrator> 3. Enter the command bcdedit /ems on . Example 9-18 shows the output from this command. Example 9-18 output of bcdedit /ems on c:\users\administrator>bcdedit /ems on the operation completed successfully. 4. Modify the ems settings to match the...

  • Page 551

    Chapter 9. Management 531 imm setting you can change the cli mode for the com port for ems. Use the following procedure: 1. Log in to the web interface of imm. 2. Navigate to imm control serial port. 3. Change the cli mode to cli with ems compatible keystroke sequences (figure 9-90). Figure 9-90 ser...

  • Page 552

    532 ibm ex5 implementation guide example 9-22 output of the bootcfg /ems on /port com2 /baud 115200 /id 1 command c:\>bootcfg /ems on /port com2 /baud 115200 /id 1 success: changed the redirection port in boot loader section. Success: changed the redirection baudrate in boot loader section. Success:...

  • Page 553

    Chapter 9. Management 533 linux you must edit two files in linux to ensure that the console redirection still works after the operating system has loaded. The same files are changed for red hat linux (rhel) and suse linux. Edit these files: /boot/grub/menu.Lst /etc/inittab menu.Lst or grub.Conf add ...

  • Page 554

    534 ibm ex5 implementation guide connecting to the x3690 x5 or x3850 x5 you can use telnet or ssh to connect to the x3690 x5 or x3850 x5 servers. Follow these steps: 1. Start a telnet/ssh session to the imm ip address. 2. Log in to the imm. The default user id is userid and the default password is p...

  • Page 555

    Chapter 9. Management 535 sample output after you connect to the system using sol and power on the server, you see the output in the telnet/ssh window. Figure 9-94 shows the x3850 x5 booting. Figure 9-94 booting the x3850 x5 figure 9-95 shows the logo window via sol. The ibm logo is not displayed. F...

  • Page 556

    536 ibm ex5 implementation guide figure 9-97 shows the windows sac console. Figure 9-97 microsoft windows special administration console (windows only) figure 9-98 shows linux booting. Figure 9-98 linux boot sequence as seen in sol computer is booting, sac started and initialized. Use the “ch -?” co...

  • Page 557: Abbreviations and Acronyms

    © copyright ibm corp. 2011. All rights reserved. 537 ac alternating current aes advanced encryption standard aik automated installation kit amm advanced management module api application programming interface apic advanced programmable interrupt controller asu advanced settings utility bc bladecente...

  • Page 558

    538 ibm ex5 implementation guide ip internet protocol ipmb intelligent platform management bus ipmi intelligent platform management interface is information store iso international organization for standards it information technology itso international technical support organization jbod just a bunc...

  • Page 559

    Abbreviations and acronyms 539 ssct standalone solution configuration tool ssd solid-state drive ssh secure shell ssic system storage interoperation center ssl secure sockets layer stg server and technology group tb terabyte tcg trusted computing group tco total cost of ownership tcp transmission co...

  • Page 560

    540 ibm ex5 implementation guide.

  • Page 561: Related Publications

    © copyright ibm corp. 2011. All rights reserved. 541 related publications the publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this book. Ibm redbooks publications the following ibm redbooks publications provide addition...

  • Page 562: Other Publications

    542 ibm ex5 implementation guide vista deployment using tivoli provisioning manager for os deployment, redp-4295 you can search for, view, or download ibm redbooks, redpapers, webdocs, draft publications and additional materials, as well as order hardcopy ibm redbooks publications, at this website: ...

  • Page 563: Online Resources

    Related publications 543 online resources the following websites are also relevant as further information sources: ibm ex5 home page http://ibm.Com/systems/ex5 ibm system x3850 x5 home page http://ibm.Com/systems/x/hardware/enterprise/x3850x5 ibm bladecenter hx5 home page http://ibm.Com/systems/blad...

  • Page 564

    544 ibm ex5 implementation guide.

  • Page 565: Index

    © copyright ibm corp. 2011. All rights reserved. 545 index numerics 10gb ethernet hx5 183 x3690 x5 166 x3850 x5 104 39r6529 101, 162 39r6531 101, 162 39y9306 211 41y8278 175, 214 41y8287 175, 214 42c1830 211 42d0632 92, 147 42d0637 92, 147 42d0672 92, 147 42d0677 92, 147 42d0709 147 42d0747 92 42d07...

  • Page 566

    546 ibm ex5 implementation guide 60y0333 148 60y0337 165 60y0339 146 60y0360 150 60y0366 165 60y0381 146 675w redundant power supply kit 173 69y2322 146 69y2323 146 69y2344 176 69y2345 176 69y2347 176 69y4389 176 69y4390 176 69y4403 176 6gb ssd host bus adapter x3850 x5 99, 155 81y4426 92, 149 a act...

  • Page 567

    Index 547 serverguide 502 updatexpress system pack installer 511 dvd drive x3690 x5 163 x3850 x5 102 dynamic infrastructure 1 dynamic system analysis 448 e electronic service agent 481, 493 emergency messaging service 529 emulex 10gb ethernet adapter hx5 183 x3690 x5 166 x3850 x5 104 energy efficien...

  • Page 568

    548 ibm ex5 implementation guide latency settings 398 layout 179 light path diagnostics 443 local storage 385 log 444 lsi configuration utility 205, 386 lsi controller 207 max5 180, 186, 401 block diagram 187 population order 198 max5 and esxi 418 media to install 407 memory 194 balance 199 block di...

  • Page 569

    Index 549 login 469 no access 471 partitions 473 power control 488 renaming systems 470 request access 471 scalable partitions 473 service and support manager 481 ibm toolscenter 448 imm 449–454 see also amm asu 495 asu commands 500 discovery 468 features 449 firmware updates 454 hx5 213 in-band man...

  • Page 570

    550 ibm ex5 implementation guide megaraid storage manager 521 drive states 522 installation 521 megacli 523 virtual drive states 523 memcached 52 memory 22 chipkill 29 dimm placement 23 giga transfers 22 hemisphere mode 26 hx5 194 lock step mode 30 mirroring 28 performance 23 qpi link 22 ranks 24 re...

  • Page 571

    Index 551 s sas drives hx5 203 x3690 x5 145 x3850 x5 90 sas expansion adapter 146 scalability 33 hx5 188 scalable memory buffers 30 self-encrypting drives x3690 x5 148 serial over lan 525 connecting 533 enabling 526 telnet 534 serveraid megaraid storage manager 521 serveraid b5015 x3690 x5 155 x3850...

  • Page 572

    552 ibm ex5 implementation guide hx5 212, 396 hyper-threading 45 intel virtualization technology 45 main window 37 mapper policy 47 max5 memory scaling affinity 47 memory data scrambling 47 memory mirror mode 46 memory spare mode 46 memory speed 41, 46 operating modes 21 page policy 47 patrol scrub ...

  • Page 573

    Index 553 dvd drive 163 dynamic system analysis 370 efficiency mode 341 electronic service agent 370 embeded hypervisor 355 emulex 10gb ethernet adapter 166 energy efficiency 341 ethernet 170 event log 334, 370 exflash 149 external storage 162 fans 171 features 118 firmware 365 friction slide kit 17...

  • Page 574

    554 ibm ex5 implementation guide serveraid b5015 155 serveraid controllers 149 serveraid expansion adapter 146, 157 serveraid m1015 154 serveraid m5014 153 serveraid m5015 153 serveraid m5025 162 serveraid performance key 152 serveraid-br10i 153 serverguide 501 serverguide scripting toolkit 349 serv...

  • Page 575

    Index 555 remote control 257 settings 250 shared port 252 system status 253 troubleshooting 253 userid 251 using 250 virtual light path diagnostics 255 virtual media 265 installation order processors 224 integrated virtualization 114 internal disks 90 internal view 58 legacy option roms 246 legacy t...

  • Page 576

    556 ibm ex5 implementation guide serveraid support 91 serveraid-br10i x3850 x5 96 serverguide 501 serverguide scripting toolkit 266 serverproven 108 service processor 250 shared port 252 slot differences 244 slots 238 ssd controllers 96 ssd drives 93 ssd support 91 standard processors 74 storage 90 ...

  • Page 577: Ibm Ex5 Implementation Guide

    (1.0” spine) 0. 875”1.49 8 ” 460 788 pages ibm ex5 implementation guide ibm ex5 implementation guide ibm ex5 implementation guide ibm ex5 implementation guide.

  • Page 578: Ibm Ex5 Implementation Guide

    Ibm ex5 implementation guide ibm ex5 implementation guide.

  • Page 580: Ibm Ex5 Implementation

    ® sg24-7909-00 isbn 0738435643 international technical support organization building technical information based on practical experience ibm redbooks are developed by the ibm international technical support organization. Experts from ibm, customers and partners from around the world create timely te...