Warning: This guide is for properly trained service personnel and technicians who are required to install the IPU-POD If you have any questions then please contact your Graphcore representative or use the resources on the Graph- core support portal: https:/ /www.graphcore.ai/support.
IPU-POD128 build and test guide 1.1 Acronyms and abbreviations This is a short list that describes some of the most commonly used terms in this document. Table 1.1: Glossary Term Description Active optical cable Baseboard Management Controller: standby power domain service processor doing system...
CHAPTER IPU-POD DESIGN COMPONENTS This section describes the components in the IPU-POD . Each IPU-POD is made from two IPU-POD logical racks with GW-Links connected between them. 2.1 IPU-POD components Each IPU-POD has the following: • 16 IPU-M2000s • 1 (default con guration is one host server, up to four can be supported) Server •...
IPU-POD128 build and test guide • 2 power connectors per IPU-M2000 • 5 fan units • 5 LED indicators • Unit QR code Fig. 2.2: Back panel 2.2.2 QR code label There is a QR code label on the back panel of each IPU-M2000. The QR code contains the following information for each IPU-M2000: •...
PowerEdge R6525 server but up to four servers can be used. Contact Graphcore sales for details of other supported server types. This document describes the default server (PowerEdge R6525) installation only. Other servers may have di erent installation requirements.
(ML) jobs as a data-plane, connecting the host servers running the Poplar® SDK with the IPUs running the ML model in the IPU-M2000s. The default ToR switch is an Arista DCS-7060CX-32S-F. Contact Graphcore sales for details of other supported switch types. This document describes the default switch (7060CX) installation only.
IPU-POD128 build and test guide 2.8.1 RJ45 cables • Red: IPU-M2000 to IPU-M2000 within-rack IPU-Link connectivity • Blue: Connecting IPU-M2000s to the management switch (BMC + IPU-Gateway management) • Blue: Connecting servers to the management switch • Yellow: Connecting IPU-M2000s to the management switch (BMC only management) 2.8.2 OSFP cables...
CHAPTER THREE IPU-POD RACK ASSEMBLY You will need to follow the instructions in this section for each of the two IPU-POD logical racks you need for the IPU-POD Note the correct orientation of the IPU-M2000, server and switch units in the rack to ensure correct air ow. The front interface of the IPU-M2000 units (connectivity ports) should be matched with the front door of the rack (cold aisle).
Page 13
IPU-POD128 build and test guide Fig. 3.2: Completed rack: hot aisle (four-server version) Note that Fig. 3.2 shows three blue RJ45 cables in each R6525 server. In the default build, servers 2 to 4 only have two blue RJ45 cables. See Section 3.6.7, Dell R6525 server(s) cabling...
IPU-POD128 build and test guide 3.2 Document reproduction Ideally you should reference this document from a tablet device to allow you to zoom in on photographs. If you reproduce this document on paper it should be done in colour otherwise you will not be able to see cable colours and other notations properly.
IPU-POD128 build and test guide 3.4.2 Unpacking the rack Follow the instructions to remove the outer packaging of the APC AR3300SP rack, ensuring that you safely store these materials for later repackaging. Do not remove the rack from the shock pallet. Remove the white bag (Fig.
Page 17
IPU-POD128 build and test guide Fig. 3.4: Rack earth straps Remove the top and bottom side panels. The vertical accessory channels should be positioned at the very front and very rear of the rack. If necessary, move these from their shipping positions (Fig.
IPU-POD128 build and test guide 3.4.4 Removing the vertical accessory channels Using a Torx TX30 screwdriver, remove two accessory channels from the rack (Fig. 3.6). Fig. 3.6: Accessory channel removal 3.4.5 Adjusting the rear accessory channels Set the rear accessory channel to the furthest position in the rack. Tighten up the screws ensuring the teeth engage into the slots in the rail, as shown in Fig.
IPU-POD128 build and test guide 3.4.6 Adjusting the rear vertical rails Using a Torx TX30 screwdriver, make both rear vertical rack rails loose and freely movable. Position the rear vertical rack rails such that there is 20mm of distance between the rear face of the vertical rack rail and the racks rear frame.
IPU-POD128 build and test guide Fig. 3.9: Front vertical rail alignment Secure the rail into position by moving the TX30 screws back upwards such that the teeth engage with both the supporting rails. This must be done at the top and bottom of the bracket.
Page 21
IPU-POD128 build and test guide Fig. 3.11: M2000 rack rail kit: extended rails 2) Whilst pulling on the thumb screw end of the rails, push the white plastic release tab towards the thumb screw end (Fig. 3.12): Fig. 3.12: M2000 rack rail kit: white release tab 3) The inner and outer rails will now separate (Fig.
Page 22
IPU-POD128 build and test guide Fig. 3.14: M2000 rack rail kit: xing pins 2) Push the inner rail towards the end of the IPU-M2000 containing the network ports, you should hear a click as the latching mechanism locks behind the head of a xing pin (Fig.
IPU-POD128 build and test guide Fig. 3.17: M2000 rack rail kit: attaching inner rail to M2000 The inner rails are now securely a xed to the IPU-M2000 body. Place the outer rails to one side for later use (Fig. 3.18): Fig.
Page 24
IPU-POD128 build and test guide Fig. 3.20: Bottom PDU bracket cage nuts Screw the PDU support brackets to the inside of the cabinet. The PDU brackets should be installed at the rear of the rack: one bracket on top with 9cm distance from the top of rack and one bracket on the bottom with 12cm distance from the bottom of rack.
IPU-POD128 build and test guide 3.5 Installing the equipment The following sections describe the installation of the IPU-M2000s, PDUs, servers, ToR and management switches into the rack. 3.5.1 Installing the IPU-M2000s Earlier in the guide we a xed the inner rack rails to the IPU-M2000 body. The next step in installing the IPU- M2000s is to install the outer rack rails into the rack.
Page 26
IPU-POD128 build and test guide Fig. 3.23: IPU-M2000 outer rail latching mechanism 3) Locate the rear end of the outer rail and slightly open the large metal latch, then press the upper and lower locating pins into the square holes in the vertical rack rail. Release the large metal latch and the outer rail will now be secured to the vertical rack rail (Fig.
Page 27
IPU-POD128 build and test guide Fig. 3.25: IPU-M2000 rack rail kit: outer rail screws and washers Fig. 3.26: IPU-M2000 rack rail kit: outer rail screws and washers attached Next, to install an IPU-M2000 unit into the rack rails, work your way through the following steps in order:...
Page 28
IPU-POD128 build and test guide 2) Place the IPU-M2000 onto an appropriate server lift and adjust the height such that it is suitable for the sliders (Fig. 3.28). If a lift is not available, this is a two person operation.
Page 29
IPU-POD128 build and test guide Fig. 3.30: Sliding IPU-M2000 into rack Fig. 3.31: Blue tab release mechanism 6) Screw the captive thumb screw into the inner rack rail (Fig. 3.32): Fig. 3.32: Screwing captive thumb screw into inner rack rail...
IPU-POD128 build and test guide 3.5.2 Installing the management switch Insert 2 cage nuts, inside the rack, on either side of the rack into the top and bottom positions of location 17. Place the management switch on top of the last IPU-M2000 and screw it into position using four M6 screws (Fig.
IPU-POD128 build and test guide 3.5.4 Installing the PDUs Install the two PDUs vertically at the rear of the rack, one on the left side and one on the right side. Push the mains cable through the roof of the rack and then clip the PDUs onto the PDU bracket as shown in Fig.
IPU-POD128 build and test guide 3.5.5 Installing the Dell R6525 server(s) Remove and discard the cable management arm brackets from the rear of each tool-less sliding rail (Fig. 3.36). Fig. 3.36: Cable management arm brackets - to be removed Install the tool-less sliding rail kit(s). The reference design has a single server installed in rack slot #19; in four server con gurations they are installed in rack slots #19-#22.
Page 33
IPU-POD128 build and test guide Remove the Velcro tape from the light pipes on the rear of the servers. Remove the small plastic tab on the left front side of the server bezel and clip the bezel in place on the front of the...
IPU-POD128 build and test guide 3.6 Cabling the rack The following sections detail the cabling of the rack and the dressing of the cables within the rack. For reference, the IPU-M2000s and server(s) are numbered as shown in Fig. 3.39: Fig.
IPU-POD128 build and test guide 3.6.1 IPU-M2000 to IPU-M2000 IPU-Link connectivity (OSFP) There are eight OSFP IPU-Link ports on the right side of each IPU-M2000. Using the supplied 60x 0.3M OSFP cables, and starting at IPU-M2000 #1 (bottom-most IPU-M2000 in the rack), link the top row of four ports (5-8) to the bottom row of four ports (1-4) in the IPU-M2000 that is installed directly above (see Fig.
Page 36
IPU-POD128 build and test guide Fig. 3.42: Check IPU-M2000 to IPU-M2000 IPU-Link cabling Note: The white tab is on the top of the cable when inserted into the IPU-M2000. Table 3.2 shows the IPU-M2000 OSFP port mapping. Table 3.2: OSFP IPU-M2000 to IPU-M2000 IPU-Links...
Page 37
IPU-POD128 build and test guide Fig. 3.43: Final IPU-M2000 to IPU-M2000 IPU-Link cabling Version: latest (2021-11-25)
IPU-POD128 build and test guide 3.6.2 IPU-M2000 to IPU-M2000 Sync-Link cabling Using the 0.15m red Ethernet RJ45 cable, wire the 16 IPU-M2000s as follows: Starting from IPU-M2000 #1 at the bottom of the rack, insert one side of a cable into port 6 and one side of another cable into port 7.
Page 39
IPU-POD128 build and test guide Table 3.3 gives the IPU-M2000 Sync-Link port mapping. Table 3.3: IPU-M2000 to IPU-M2000 Sync-Links From Cables IPU-M2000 #15 Sync-Link ports 6,7 IPU-M2000 #16 Sync-Link ports 2,3 RJ45 0.15 red IPU-M2000 #14 Sync-Link ports 6,7 IPU-M2000 #15 Sync-Link ports 2,3 RJ45 0.15 red...
IPU-POD128 build and test guide 3.6.3 IPU-M2000 to management switch cabling (RJ45) There are two Ethernet ports in the middle of each IPU-M2000 (see Fig. 3.46). One of them is a BMC + GW SoC port (upper port) and the other is a BMC port (lower port).
Page 41
IPU-POD128 build and test guide Ports Allocation Customer datacentre uplink for BMC + GW Customer datacentre uplink for BMC-only (future update) Customer management interface PDU management Server 1 PDU management switch ToR to management switch 9-12 1GbE server management (iDRAC)
IPU-POD128 build and test guide 3.6.4 Management switch: BMC cabling Start cabling using a 1.0m yellow cable and insert one end into port 48 of the management switch. Run the cable through the loop in the OSFP connector as shown in the left hand picture in Fig.
Page 43
IPU-POD128 build and test guide Table 3.5: IPU-M2000 management BMC-only mapping From Cables IPU-M2000 #16 BMC port Management switch port 48 RJ45 1.0m yellow IPU-M2000 #15 BMC port Management switch port 47 RJ45 1.0m yellow IPU-M2000 #14 BMC port Management switch port 46 RJ45 1.0m yellow...
IPU-POD128 build and test guide 3.6.5 Management switch: BMC + GW SoC cabling Start cabling using a 1.0m blue RJ45 cable and insert one end into port 32 of the management switch. Run the cable through the loop in the OSFP connector as shown in the left hand picture in Fig.
Page 45
IPU-POD128 build and test guide Table 3.6: IPU-M2000 management BMC + GW port mapping From Cables IPU-M2000 #16 BMC + GW Soc port Management switch port 32 RJ45 1.0m blue IPU-M2000 #15 BMC + GW Soc port Management switch port 31 RJ45 1.0m blue...
IPU-POD128 build and test guide 3.6.6 IPU-M2000 to ToR switch cabling (QSFP) The next step is to connect the IPU-M2000s to the ToR switch. There are two RNIC ports on the left side of each IPU-M2000, as shown in Fig.
Page 47
IPU-POD128 build and test guide Table 3.7: IPU-M2000 RNIC port mapping IPU-M2000 port ToR switch port Cables IPU-M2000 #16 port 2 ToR switch port 9 QSFP 1.0m IPU-M2000 #15 port 2 ToR switch port 10 QSFP 1.0m IPU-M2000 #14 port 2 ToR switch port 11 QSFP 1.0m...
Page 48
IPU-POD128 build and test guide Fig. 3.53: Final IPU-M2000 to ToR switch cabling Version: latest (2021-11-25)
IPU-POD128 build and test guide 3.6.7 Dell R6525 server(s) cabling All cables should be routed from the rear of the server to the right-hand side when viewed from the rear (see Fig. 3.54), then along the side of the rack using the cable management holes in the vertical rack rails.
IPU-POD128 build and test guide 3.6.8 ToR switch to Dell server(s) Using 2 of the 1.5m QSFP cables per server, connect the ToR switch to the server(s) as follows. 1) Feed the cables through the cut-outs in the side of the rack rst (Fig.
Page 51
IPU-POD128 build and test guide Fig. 3.56: ToR switch to server cabling: cable connections (4 server version) Version: latest (2021-11-25)
IPU-POD128 build and test guide 3.6.9 Management switch to Dell server(s): iDRAC Using four of the 1.5m blue RJ45 cables connect the management switch to the Dell server(s) as follows: • Ports 9 to 12 on the management switch connect to the iDRAC connector on the server(s) - Fig.
IPU-POD128 build and test guide 3.6.11 Management switch to Dell server(s): switch management Using one of the 1.5m blue RJ45 cables connect the management switch to server 1 as follows: • Port 8 from the management server is connected to the lowest server (server 1) - the correct port to connect to on the server is shown in Fig.
IPU-POD128 build and test guide 3.6.12 Management switch to PDUs Using the two 2m purple RJ45 cables connect the management switch to the two PDUs as given in Table 3.9: Table 3.9: Management switch to PDU connections Ethernet port | Management switch port...
IPU-POD128 build and test guide 3.7 Power cabling The designated sockets on the PDUs for servers and IPU-M2000s are shown in Fig. 3.61. Fig. 3.61: Designated PDU sockets for server(s) and IPU-M2000s Fig. 3.62 Fig. 3.63 show the nal power cabling.
Page 56
IPU-POD128 build and test guide Fig. 3.62: Final power cabling (direct view) Fig. 3.63: Final power cabling (angled view) Version: latest (2021-11-25)
IPU-POD128 build and test guide 3.7.1 IPU-M2000 power cabling Start by cabling IPU-M2000 #1 using 0.5m power cables. Ensure that only three IPU-M2000s are connected to the same bank on the PDU. Red cables to the left PDU and blue cables to the right PDU (as seen looking at the rear of the rack).
IPU-POD128 build and test guide 3.7.2 Server power cabling: Dell R6525 Using 1m C13 to C14 power cables (selecting the correct coloured cable to match the PDU colour), connect the server(s) to the PDUs as shown in Fig. 3.66. Fig. 3.66: Server power cabling Note: The photo above shows the four-server version.
IPU-POD128 build and test guide 3.8 Completing the rack The following steps describe completing the rack: tting blanking panels and re-installing the doors and side panels. 3.8.1 Blanking panels Install the supplied APC 1U blanking panels in every unoccupied rack slot at the front of the rack.
IPU-POD128 build and test guide 3.8.4 PDU plugs The power plugs for the PDUs must not exit the top of the rack, they must stay inside the rack and be secured to the vertical mounting anges. You need to ensure that there are no cables on the outside of the rack.
Note: Ansible playbooks for the server installation and les for switch con guration can be provided on request: please contact Graphcore support. These templates may need to be adapted to match any site-speci c di erences. 4.1 Server con guration The server and POD con guration consists of the following main steps: •...
IPU-POD128 build and test guide • Single/dual port Mellanox ConnectX-5 EN 100Gb/s Ethernet 4.1.2 Storage con guration recommendations The recommendation is to have two types of server storage: SSD-SATA for the operating system and NVME SSD for data storage. Operating system: •...
Page 63
IPU-POD128 build and test guide Setting BIOS 1.2.11 BIOS 1.4.8 or later LogicalProc Enabled Enabled ProcVirtualization Enabled Enabled IommuSupport Disabled See below L1StreamHwPrefetcher Enabled Enabled L2StreamHwPrefetcher Enabled Enabled MadtCoreEnumeration Linear Linear NumaNodesPerSocket CcxAsNumaDomain Disabled Disabled CpuMinSevAsid ProcCcds CcdCores EmbSata AhciMode AhciMode BIOS 1.4.8 or later does not con gure IOMMU directly.
IPU-POD128 build and test guide 4.1.5 Operating system installation Please contact your Graphcore representative or use the support portal for information about operating system support. This document describes the following operating systems: • Ubuntu 18.04.4 LTS (bionic) (Ubuntu 18.04 packages) •...
Page 65
IPU-POD128 build and test guide CentOS 7 packages In order to have a stable system where IPU related software can run, several packages need to be installed (see Table 4.3) via the yum con guration manager. Other packages may be required for local requirements.
Server as IT admin user itadmin Server iDRAC port root 100GbE RDMA switch admin 1GbE Management switch admin The default passwords are available from Graphcore support portal. Table 4.7 contains the required groups provided on the IPU-POD Table 4.7: Required user groups Groups Description root A root group to locate the root account is mandatory.
IPU-POD128 build and test guide 4.1.7 DHCP Service (Dynamic Host Con guration Protocol) An ISC-DHCP-Server service (Table 4.8) is recommended to provide DHCP network con guration to IPU-M2000s. It can be installed from the Ubuntu or CentOS public repositories. Table 4.8: DHCP service...
Page 68
IPU-POD128 build and test guide • /etc/dhcp/dhcpd.d/vlan-13.conf #Example of content. Add all BMCs and Gws below using this format #host ipum1bmc { hardware ethernet a0:b0:c0:d0:e0:f0; fixed-address 10.1.1.1;} #host ipum1gw { hardware ethernet a1:b1:c1:d1:e1:f1; fixed-address 10.1.2.1;} #host ipum2bmc { hardware ethernet a0:b0:c0:d0:e0:f0;...
IPU-POD128 build and test guide 4.1.8 Rsyslog service The rsyslog service (Table 4.9) is a software utility for forwarding log messages in an IP network. Table 4.9: Rsyslog service Name Type User Group Access Description /etc/rsyslog.d folder root root 0755 Rsyslog tool con guration folder /etc/rsyslog.conf...
IPU-POD128 build and test guide 4.1.9 NTP service (Network Time Protocol) NTP service is recommended to provide network time con guration to IPU-M2000s. It can be installed from the Ubuntu or CentOS public repositories. NTP le structure • /etc/ntp.conf ( le) –...
IPU-POD128 build and test guide 4.2 Network con guration 4.2.1 Overview Fig. 4.2 gives a logical overview of the network setup within the IPU-POD Version: latest (2021-11-25)
Page 74
IPU-POD128 build and test guide Fig. 4.2: IPU-POD network overview Option: VLAN 14 and the cabling to the dedicated BMC port can be provided as an upgrade for customers that want an isolated BMC network. This separates the BMC and GW tra c inside the IPU-M2000 onto the two ports - BMC on the lower port and GW on the upper port.
IPU-POD128 build and test guide 4.2.2 IPU-POD network interfaces Table 4.11: Network con guration Port Role Speed IP address Con g VLAN from (see note) IPU-M2000: BMC BMC only management 1GbE 10.1.1.1-16/22 Static (future) DHCP IPU-M2000: GW BMC+GW management 1GbE 10.1.2.1-16/22...
Page 76
IPU-POD128 build and test guide Fig. 4.3: Management switch VLAN assignments These port groups are members of the switch’s internal port based VLANs. The VLAN assignments are given below. • VLAN 13: IPU-M2000 BMC and GW tra c • VLAN 14: IPU-M2000 BMC only tra c (option) •...
IPU-POD128 build and test guide 4.2.4 ToR switch con guration The default ToR switch is an Arista DCS-7060CX-32S-F. The ports are allocated as follows: 1) One port per server for tra c between the server(s) and IPU-M2000s, on VLAN11 2) One port per server for uplink tra c from the server(s) to datacentre network-attached storage. This is disabled in the default setup le since NAS con guration will be site speci c.
IPU-POD128 build and test guide Note: The VLAN assignment for the datacentre uplink ports will be site and con guration speci c. In the default con guration le these ports are disabled. A switch con guration le can be provided – please contact Graphcore support for details.
IPU-POD128 build and test guide 4.2.6 Server network con guration • It is recommended to use the Netplan network manager to con gure the server using netplan con guration • Two 1GbE baseT connections are required to the 1GbE management switch for Server 1. Only one 1GbE baseT connection is required to the 1GbE management switch for additional servers.
By default this is server 1. The following Graphcore software packages need to be installed on the management server: • V-IPU software: contains management and control software for IPU resource control, Built In Self Test (BIST) and monitoring of the IPU-M2000s and IPUs.
If you want to revert to the previous software version, the standby image can be upgraded to the previous version in the same way as described above. Note: Graphcore has only quali ed the IPU-M2000 software release with exactly the documented set of software sub-component versions and any other version combinations of software components are not guaranteed.
IPU-POD128 build and test guide 5.3.1 Download IPU-M2000 software update bundle The management server needs to be loaded with the correct IPU-M2000 software update bundle before the software update of the IPU-M2000s can be performed. To perform the download, follow these steps: 1.
IPU-POD128 build and test guide The IPU-M2000 Gateway upgrade itself is destructive in the sense that it will overwrite all executables and cor- responding le systems for the standby image. The content of the overlay con g les is maintained and stored persistently on the management server.
./rack_tool.py bist This test will generate a very low level hardware veri cation report that will need to be analysed by Graphcore support in case any errors are reported. The logs are located at “./logs” relative to the current directory where the command is executed.
IPU-POD128 build and test guide 6.2.2 V-IPU built in self tests ./rack_tool.py vipu-test The following section is based on excerpts from the V-IPU Admin Guide which should be consulted for a detailed and updated overview of BISTs. The V-IPU User Guide is also useful.
Page 86
IPU-POD128 build and test guide • There is no persistence to the cluster test results. Only the results of the last test can be retrieved with the –status command, as long as the V-IPU server has not been restarted. IPU-Link cabling test In order to verify that external IPU-Link cables are connected as expected and properly inserted, the cabling test can be utilized.
Page 87
IPU-POD128 build and test guide Sync-Link test The Sync-Link test veri es the external Sync-Link cabling between IPU-M2000s. You can run a Sync-Link test by passing the --sync option to the test cluster command. A failing Sync-Link test reports the cables which failed to satisfy the cluster topology that is being tested by pointing to the IPU-M2000s and Sync-Link port numbers of the failing Sync-Link.
Page 88
This example shows a situation where the IPU-link tra c test has failed due to too many correctable errors being detected. Should this occur please try reseating the IPU-Link cables associated with the referenced IPU-M2000 units. If that does not resolve the issue, please contact Graphcore support. Version: latest (2021-11-25)
CHAPTER SEVEN IPU-POD INSTALLATION An IPU-POD system consists of two IPU-POD logical racks with optical GW-Link cables cross connected between them. Table 7.1 shows the BOM for an IPU-POD Table 7.1: BOM for an IPU-POD Item Product number Description Quantity 900-0020 IPU-POD Logical Rack...
Page 90
IPU-POD128 build and test guide Fig. 7.1: IPU-POD topology with cross connected GW-links Fig. 7.2 shows the location of the GW-Link ports on the front of each IPU-M2000. Fig. 7.2: IPU-M2000 front panel showing GW-Link ports Version: latest (2021-11-25)
Page 91
IPU-POD128 build and test guide Table 7.2 gives the IPU-M2000 GW-Link port mappings between the two IPU-POD racks, with the GW-Links cross connected: • GW-Link port 1 of each IPU-M2000 in the left-hand IPU-POD connects to GW-Link port 2 of the matching IPU-M2000 (same rack index) in the right-hand IPU-POD •...
CHAPTER EIGHT IPU-POD NETWORK CONFIGURATION This section describes how to set up the network con guration on the management server and IPU-M2000s for your IPU-POD 8.1 Overview This section describes how two IPU-POD racks can be merged into a new IPU-POD .
IPU-POD128 build and test guide be changed to vlan-13. 8.2 Useful resources For more details on using V-IPU you can reference the V-IPU user guide V-IPU administrator guide. The user guide also contains relevant information. 8.3 IP addressing The previous template for management network IP addresses (10.x.y.z) has been modi ed since logical rack (lrack) numbers are now needed in the IP address.
IPU-POD128 build and test guide user: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether f4:02:70:b9:18:3e brd ff:ff:ff:ff:ff:ff inet 10.1.7.150/16 brd 10.1.255.255 scope global eno1 valid_lft forever preferred_lft forever inet 10.2.7.150/16 brd 10.2.255.255 scope global eno1 valid_lft forever preferred_lft forever inet 10.3.7.150/16 brd 10.3.255.255 scope global eno1...
Page 95
IPU-POD128 build and test guide Step 1: For lrack1 and lrack2 1a) Copy rack_config.json to rack_config.json_pod128 in the same directory: cp /home/ipuuser/.rack_tool/rack_config.json /home/ipuuser/.rack_tool/rack_config.json_pod128 You then need to rewrite the rack_config.json_pod128 con g le to match the IPU-POD IP addresses as shown in Section 8.7.3, Lrack1: rack_con g.json...
Page 96
IPU-POD128 build and test guide sudo scp /etc/dhcp/dhcpd.d/vlan-11.conf ipuuser@pod7:/etc/dhcp/dhcpd.d/lrack2 Step 3: Edit DHCP con g les for lrack1 You need to edit the following management network DHCP les on lrack1: • /etc/dhcp/dhcpd.d/lrack1/vlan-13.conf • /etc/dhcp/dhcpd.d/lrack2/vlan-13.conf Step 4: netplan setup There are examples of the netplan les that you are required to edit in Section 8.7, /etc/netplan...
IPU-POD128 build and test guide server RNIC addresses also by the Pod7 management server’s DHCP server ONLY: rack has servers host lr8-server1mx { hardware ethernet 1c:36:da:4b:ea:ef; fixed-address 10.5.8.150; } host lr8-server2mx { hardware ethernet 1c:36:da:4b:ea:ef; fixed-address 10.5.8.151; } host lr8-Server3mx { hardware ethernet 0c:44:a1:20:7c:83; fixed-address 10.5.8.152; } host lr8-Server4mx { hardware ethernet 0c:44:a1:20:80:a3;...
Page 98
IPU-POD128 build and test guide sudo systemctl restart isc-dhcp-server Step 8: lrack1 DHCP server On lrack1 you need to edit /etc/dhcp/dhcpd.conf and include /etc/dhcp/dhcpd.d/ipum-dhcp.conf as described Section 8.6.1, Lrack1 and lrack2: /etc/dhcp/dhcpd.conf. This dhcpd.conf le will then start pointing to the new /etc/dhcp/dhcpd.d/lrack1 and /etc/dhcp/dhcpd.d/lrack2 con g les.
Page 99
IPU-POD128 build and test guide Step 12: Restart IPU-M2000s on lrack2 12a) Restart the IPU-M2000s on lrack2 by using the BMC controller to manually power cycle them with the following commands run on lrack2: rack_tool.py power-cycle rack_tool.py run-command c reboot d bmc This will use the old /home/ipuuser/.rack_tool/rack_config.json le.
Page 100
IPU-POD128 build and test guide Step 15: Test access to V-IPU agents Use the V-IPU test command to test access to V-IPU agents for both lrack1 and lrack2. For example: vipu-admin test cluster cl128 For more details on how to use this command refer to the V-IPU guides...
IPU-POD128 build and test guide Step 21: Refresh overlay les on lrack1 Use rack_tool to refresh the overlay les. rack_tool.py update-root-overlay Step 22: Check IPU-M2000s logging to lrack1 You need to check whether all 32 IPU-M2000s are logging to the lrack1 syslog. This will either be located in /var/log/syslog or in a speci ed location if you have lters in place for IPU-M2000 logs.
IPU-POD128 build and test guide rack_tool.py update-root-overlay 8.5.2 Syslog on BMC 8.6 DHCP les 8.6.1 Lrack1 and lrack2: /etc/dhcp/dhcpd.conf This setup assumes that the IPU-POD switches, server BMCs and PDUs are connected to the IT network (see Section 3, IPU-POD...
IPU-POD128 build and test guide (continued from previous page) subnet 10.5.0.0 netmask 255.255.0.0 { option subnet-mask 255.255.0.0; default-lease-time 600; max-lease-time 7200; range 10.5.7.240 10.5.7.244; list all lrack numbers that is served by this host here include "/etc/dhcp/dhcpd.d/lrack1/ipum-rnic.conf"; 8.6.3 Lrack2: /etc/dhcp/dhcpd.d/ipum-dhcp.conf cat /etc/dhcp/dhcpd.d/ipum-dhcp.conf...
IPU-POD128 build and test guide 8.7 /etc/netplan les 8.7.1 1GbE management interface on lrack1 server The 1GbE management interface is required to have the setup described in this section. The eno1 interface has three IP subnets for communicating with: BMC ports, IPU-Gateway ports, and between servers (for example V-IPU CLI to V-IPU controller on the management server, and Poplar instance to V-IPU controller).
IPU-POD128 build and test guide (continued from previous page) ethernets: eno2: dhcp4: yes enp161s0f0: dhcp4: yes enp161s0f1: addresses: - 10.5.7.151/16 Server 3 would have enp161s0f1 as 10.5.7.152/16 and server 4 10.5.7.153/16. 8.7.3 Lrack1: rack_con g.json le This le on lrack1 is required for software upgrades and connectivity checks. With a spine switch installed, this le will contain entries for all 32 IPu-M2000s in the IPU-POD , as shown in the code example below.
CHAPTER NINE SYSTEM INTEGRATION TESTING 9.1 Cluster tests Use the test cluster command built in to the vipu-admin tool to perform tests on a newly created cluster before any partitions are created. More details about cluster test commands can be found in the V-IPU administrator guide.
CHAPTER REVISION HISTORY This document’s revision history is as follows: Version Date Notes 10th of October 2021 First release Version: latest (2021-11-25)
Need help?
Do you have a question about the IPU-POD128 and is the answer not in the manual?
Questions and answers