Table of Contents
Advertisement
For example, on a node with two adapter links in a configuration in which each link is part of a
separate network, the result is a window on each of the two networks that are independent
paths from one node to others. For IP communication and for messages that use the user
space FIFO mechanism (in which PAMI/LAPI creates packets and copies them to the user
space FIFOs for transmission), striping provides no performance improvement. Therefore,
PAMI/LAPI does not perform striping for short messages, non-contiguous messages, and all
communication in which bulk transfer is disabled through environment variable settings.
For large contiguous messages that use bulk transfer, striping provides a vast improvement in
communication performance. Bandwidth scaling is nearly linear with the number of adapters
(up to a limit of eight) for sufficiently large messages. This improvement in communication
bandwidth stems from the following factors:
The low overhead that is needed to initiate the remote direct memory access (RDMA)
operations that are used to facilitate the bulk transfer.
The major proportion of RDMA work that is done by the adapters.
High levels of concurrency in the RDMA operations for various parts of the contiguous
messages that are transferred by RDMA by each of the adapters.
To activate striping or failover for an interactive parallel job, you must use the following
settings for the MP_EUIDEVICE and MP_INSTANCES environment variables:
For instances from multiple networks:
MP_EUIDEVICE=sn_all, which guarantees that the assigned adapters are from different
networks.
For instances from a single network:
MP_EUIDEVICE=sn_single and MP_INSTANCES=n (in which n is greater than 1 and less
than max_protocol_instances), which features improved striping performance by using
RDMA that is seen only if windows are allocated from multiple adapters on the single
network. Such an allocation might not be possible if there is only one adapter on the
network or if there are multiple adapters, but there are resources available on only one of
the adapters.
To activate striping for a parallel job that is submitted to the resource manager batch system,
the network statement of the resource manager command file must be coded by using the
following network statements:
For a PAMI/LAPI User Space job: #@ network.pami = sn_all,shared,us
For an MPI and PAMI/LAPI User Space job on multiple networks and shares adapter
windows: #@ network.mpi_pami = sn_all,shared,us
The value of MP_INSTANCES ranges from 1 to the maximum value specified by
max_protocol_instances, as defined for the class in the LoadLeveler LoadL_admin file or
database configuration. The default value of max_protocol_instances is 1. For more
information, see Tivoli® Workload Scheduler LoadLeveler: Using and Administering,
SA22-7881-06.
Chapter 2. Application integration
125
Advertisement
Table of Contents
