Troubleshooting The Infiniband Network - HP StoreAll Series Installation Manual

/lib/modules/2.6.18-194.el5/updates/kernel/net/sunrpc/sunrpc.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/nfsd/nfsd.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/nfs/nfs.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/lockd/lockd.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/nfs_common/nfs_acl.ko
/lib/modules/2.6.18-194.el5/updates/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/exportfs/exportfs.ko
3. Rename all of the above files to use the following suffix: /path/name.ofed. For example:
mv /lib/modules/2.6.18-194.el5/updates/kernel/fs/nfs/nfs.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/nfs/nfs.ko.ofed
4. Clean up the modules with the depmod -a command, and reboot the nodes. A reboot is
necessary for changes to take effect.
"depmod -a" , "reboot"
5. Execute the following commands on each node to ensure that the modules are loaded at
startup:
chkconfig openibd on
service openibd start
6. Confirm that the Subnet Manager opensmd is running on at least one node.
7. Do one or both of the following:
NOTE:
If the subnet manager runs on managed switches, skip this step.
a. Run the command /usr/sbin/sminfo as root to determine whether opensmd is
running on the IB network.
b. If opensmd is not running, issue the following commands:
chkconfig opensmd on
service opensmd start
8. Verify the status of the HCA.
NOTE:
9. Run the following checks:
* ofed_info
* ibstat
* ibclearcounters
* ibdiagnet -lw 4x -ls 10 -r
10. Verify that the link is up, and the state is active. If the state is initializing, there is no
subnet manager running on the fabric. See

Troubleshooting the InfiniBand network

This section provides solutions to common InfiniBand installation and setup issues.
Force connected mode for a node:
/sys/class/net/ib0
"echo connected > /sys/class/net/ib0/mode"
"ifconfig ib0 mtu 65520"
NOTE: For Windows WinOF (OFFED) IB client connectivity, verify Windows Sockets Direct (wsd).
This must be enabled for Windows.
Troubleshoot physical errors (logical, system information message errors, and so on). Be sure to:
Use ibstat to look for errors on InfiniBand nodes.
Use ibclearcounters to watch for error counter increments.
This step is needed only if you have unmanaged InfiniBand switches in your network.
If you are using Host Based SM, by default it is tied to port 1 of the HCA.
Step 7.
Setting up InfiniBand couplets 155