Example Of The Cabling For A 128-Way System - IBM RS/6000 SP Problem Determination Manual

This soft copy for use by IBM employees only.
Actually, for 128-way systems, while all 16 available connections on the node
switch boards are cabled to the intermediate switch boards (4 to each of the
ISBs), only half of these cables are actually used to transfer data. It is by taking
out these redundant cables and using the connection on this ISB to connect to
other ISBs that it is possible to go beyond the 128-way to larger switch sizes.
The jack socket on the NSB is covered with a blanking plate in this instance.
Figure 42 shows how ISBs are cabled for two of the four ISBs (the diagram
would be rather confusing if all 4 were included). In addition, there is reference
to connections between ISB frames. These are only required for systems larger
than the 128-way.
Figure 42. Example of the Cabling for a 128-Way System
The connectivity on 128-way systems is excellent because there are always 8
cables that data can travel down to reach the node frame. The intermediate
switch boards provide sufficient routes to ensure that this is an optimum
configuration. If a 128-way configuration was supported without the use of
intermediate switch boards, then only 2 cables could be used to connect the
majority of the switch boards and the bandwidth would be compromised.
When using the intermediate switch boards, there will be more hops required to
travel between the node switch boards because the route must always go
through one of these intermediate boards rather than directly from one node
switch board to another. This increases the latency of the network, but because
the latency is so low on the switch, this does not become a significant factor in
the speed of the network. The determining factor is whether there is contention
for routes. As soon as packets are delayed due to all available routes being
busy, then network delays occur. By using intermediate switch boards, packet
contention is kept to a minimum. In the hypothetical example given for a
128-way system with all switch boards directly attached, packet contention is
likely to occur on a more regular basis.
89
Chapter 4. The Switch