Table of Contents
Advertisement
Quick Links
Advertisement
Table of Contents
Summary of Contents for Ammasso 1100
- Page 1 The Ammasso 1100 High Performance Ethernet Adapter User’s Guide Ammasso, Inc. 345 Summer St. Boston, MA 02210 Main: 617-532-8100 Fax: 617-532-8199 http://www.ammasso.com Revision: 1.2Update 1 July 22, 2005 Copyright © 2005 Ammasso, Inc. All rights reserved.
- Page 2 Information in this document is subject to change without notice. This document is provided for information only. Ammasso, Inc. makes no warranties of any kind regarding the Ammasso 1100 High Performance Ethernet Adapter except as set forth in the license and warranty agreements. The Ammasso 1100 High Performance Ethernet Adapter is the exclusive property of Ammasso, Inc.
-
Page 3: Table Of Contents
ISTRIBUTION PECIFIC UILD ETAILS 3.4.1 3.4.2 3.4.3 BUILD 1100 C MMASSO OMMANDS AND TILITIES 1100 A ONFIGURING THE MMASSO DAPTER 3.5.1 ONFIGURATION NTRIES 3.5.2 AMPLE ONFIGURATION ERIFYING THE DAPTER OFTWARE NSTALLATION EMOVING AN DAPTER OFTWARE NSTALLATION THE AMMASSO MPI LIBRARY... - Page 4 INARY FOR LUSTER EPLOYMENT 6.2.5 AMSO LEAN UP DIRECTORIES AND FILES TEPS ON THE CLUSTER NODE SYSTEMS LUSTER EPLOYMENT USING THE AMMASSO 1100 WITH PXE BOOT HEORY OF PERATION EQUIREMENTS BIOS S ETTINGS RAMDISK UILDING THE 7.4.1 ONFIGURING AND UILDING PPLICATIONS 7.4.2...
- Page 5 ONFIGURING A ERVER PXELINUX UILDING ISKLESS INUX OOT VIA 7.8.1 ONFIGURE A YSTEM 7.8.2 NFS S ONFIGURE THE ERVER 1100 O ROM I PDATING THE MMASSO PTION MAGE APPENDIX A: SUPPORT BTAINING DDITIONAL NFORMATION ONTACTING MMASSO USTOMER UPPORT ETURNING A...
-
Page 6: Overview
The Ammasso 1100 High Performance Ethernet Adapter is a RDMA enabled Ethernet Server adapter offering high performance for server to server networking environments. The Ammasso 1100 is also a gigabit Ethernet adapter and works within any gigabit Ethernet environment supporting existing standard wiring, switches, and other Ethernet adapters. -
Page 7: Theory Of Operation
1.2 Theory of Operation 1.2.1 How the Ammasso Adapter Works The Ammasso 1100 is an Ethernet adapter designed to take advantage of various standard interfaces, including MPI, DAPL, and traditional BSD sockets. For use with MPI and/or DAPL, the Ammasso 1100 leverages its RDMA capabilities and... - Page 8 For each RDMA command, the Ammasso 1100 adapter implements all of the required RDMA operations as well as the processing of the TCP/IP state machine, thus reducing demand on the CPU and providing a significant advantage over standard adapters while maintaining TCP/IP standards and functionality.
-
Page 9: What Is Mpi And Why Is It Used
MPI is considered the de-facto standard for writing parallel applications. A version of the MPI library MPICH-1.2.5 is available that is compatible with the Ammasso 1100 adapter. 1.2.4 What Is DAPL and Why Is It Used? Direct Access Programming Library (DAPL) defines a single set of user APIs used in Database and Storage RDMA Enabled environments. -
Page 10: Product Components
1.3 Product Components • AMSO-1100 High Performance Ethernet Adapter • A .tgz archive file containing all software files • The Ammasso 1100 High Performance Ethernet Adapter User’s Guide (this file) • Current release notes 1.4 Specifications 1.4.1 Performance • 1 Gigabit Ethernet •... -
Page 11: Hardware System Requirements And Installation
This is a Class A Product. In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures. Observe the following precautions when working with your Ammasso 1100 High Performance Ethernet Adapter. WARNING: Only trained and qualified personnel should be allowed to install or replace this equipment. -
Page 12: System Hardware Requirements
2.3 Adapter Hardware Installation 2.3.1 Choosing a Slot for Installation When choosing a connector slot in your computer, be aware that the Ammasso 1100 Ethernet Adapter must be inserted into a PCI-X slot. Before inserting the adapter, use your system documentation to determine which slots support PCI-X. -
Page 13: Adapter Insertion Procedure
RJ45 connector. The Green “Link” LED should be constantly illuminated once link is established. The Yellow “Activity” LED will flash when there is network traffic. The Ammasso 1100 will not establish link if connected to a 10/100 Ethernet switch as it requires a port speed of 1 gigabit. -
Page 14: Adapter Software Installation
This must be installed to utilize the Ammasso 1100 adapter. AMSO_MPICH.tgz is the MPICH 1.2.5 package and requires the AMSO1100 package. This package must be installed to enable MPI applications over the Ammasso 1100 adapter. AMSO_DAPL.tgz is the DAPL package and requires the AMSO1100 package. This package must be installed to enable DAPL applications over the Ammasso 1100 adapter. - Page 15 At this point you have successfully installed the AMSO1100 package on your system. If you are deploying the Ammasso software on many machines in a cluster, please see chapter 6 for more details. The various commands, libraries, and drivers are installed in the directory chosen in step 3.
-
Page 16: Makefile Targets
3.3.1 Makefile Targets The AMSO1100 package top-level Makefile supports the following targets: all: Configures and builds AMSO1100 (default target). config: Configures the AMSO1100 source tree for building. The result is a file Config.mk that contains various environment variables needed to build the product. build: Builds AMSO1100 -- depends on config target. -
Page 17: Makefile Configuration Variables
3.3.2 Makefile Configuration Variables The configuration file Config.mk has the following variables: GCC_PATH The command name of the compiler program to use and optionally its associated pathname. LD_PATH The command name of the loader program to use and optionally its associated pathname. PLATFORM The target build platform. -
Page 18: Distribution Specific Build Details
Ammasso support website (www.ammasso.com/support) for an up to date list of issues. The list of packages provided below will ensure that the system will be able to take advantage of all the Ammasso 1100 features – such as being able to support 32-bit MPI applications on 64-bit platforms. -
Page 19: Suse
For 2.4 Kernels: # make dep For 2.6 Kernels: # make prepare 3.4.2 SuSE The following section lists SuSE distribution specific details. While the exact keystrokes may vary slightly from release to release, the following are offered as guidelines for these distributions. -
Page 20: Kbuild
For 2.6 Kernels: # make prepare # make 3.4.3 Kbuild When compiling against 2.6 kernels, the 'kbuild' style Makefiles allow for an O= option to specified. This option tells kbuild where to put the configured kernel files. If the kernel is configured with O=, then all external modules must be built with the same parameter. -
Page 21: Ammasso 1100 Commands And Utilities
Command to dump Ammasso registers for debug. ccroute(8) Ammasso RNIC route configuration command. crash_dump(8) Dump a crashed RNIC to a file to enable Ammasso support personnel to help debug it. rnic_cfg(8) Describe the Ammasso 1100 configuration file and its variables. -
Page 22: Configuring The Ammasso 1100 Adapter
3.5 Configuring the Ammasso 1100 Adapter The Ammasso 1100 configuration file stores the necessary networking information for both the RDMA and Linux NetDev-style ccilnet interfaces. A single file is used to store the entire Ammasso 1100 configuration for a given system. -
Page 23: Configuration Entries
[mask] [mtu] This is the maximum transmittable unit (MTU) or frame size for the interface. This is not required. If it is not specified, Ammasso will set the MTU to the default value of 1500. [gw] This is the network gateway IP address for this interface. This is specified in network `dotted quad’... -
Page 24: Sample Configuration File
AMSO_MASK=255.255.255.0 3.6 Verifying the Adapter Software Installation Once your system has been installed and configured with the Ammasso 1100 hardware and software, you can verify correct installation with the following procedures. To verify CCIL configuration, you can use the ifconfig(8)command. The CCIL interface is called ccilnet0. -
Page 25: Removing An Adapter Software Installation
RDMA IP address. You will need two machines with Ammasso hardware and software installed in order to verify RDMA connectivity. In this example, node-A has RDMA IP address 10.40.32.52 and node-B has RDMA IP address 10.40.32.53. -
Page 26: The Ammasso Mpi Library
4.1.1 Compiler Support The Ammasso 1100 is designed to work with the standard compiler suites available in the Linux community. We have tested our MPI implementation with the GNU C and the F77 suites that are bundled with the traditional Linux distributions such as Red Hat and SuSE. - Page 27 Q7: Enter path to remote shell [/usr/bin/rsh] Argonne, MPICH, from which Ammasso MPICH derives, assumes the use of the BSD rsh(1) command, hence Ammasso’s choice to leave that as the default. However, best security practices recommend a stronger system; such as Secure Shell and Ammasso recommends sites consider its use.
-
Page 28: M Akefile T Argets
Another case would be in a cluster environment which has both 32-bit and 64- bit systems. At this point Ammasso MPICH is configured and will begin the compilation and installation process. The files are installed into the <installation_dir>/mpich-1.2.5 directory. If you are on a 64-bit machine and have taken the option to build the 32-bit version of Ammasso MPICH, there will also be <installation_dir>/mpich-... -
Page 29: M Akefile C Onfiguration V Ariables
Configures and builds Ammasso MPICH, this is the default target for make. config: Configures the Ammasso MPICH source tree, this rule creates the file Config. build: Builds Ammasso MPICH and depends on config target. install: Installs Ammasso MPICH and depends on build target. -
Page 30: L Ocating The Mpi L Ibraries And F Iles
INSTALL_DIR This is the full pathname into which Ammasso MPICH will be installed. It must be an absolute pathname not relative. RSHCOMMAND This is the full pathname to the remote shell command. It must be an absolute path pathname not relative. -
Page 31: C Ompiling And L Inking A Pplications
The libraries and files associated with using MPICH and the Ammasso 1100 are located in a directory within the Ammasso installation environment: <install_dir>/mpich-1.2.5 If you chose to install a 32b version of MPICH on a 64b system, there will be an additional directory: <install_dir>/mpich-1.2.5-32... -
Page 32: C Reating A Mpi C Luster M Achines F Ile
TCP/IP address associated with another NIC (e.g. eth0). The Ammasso MPICH implementation does not wrap the machine file hostnames. If you specify a command line to run with four processes (-np 4), the machines.LINUX file must have four host processors listed. -
Page 33: V Erifying Mpi I Nstallation
# export P4_RSHCOMMAND=/usr/bin/ssh 4.5 Verifying MPI Installation Once an application has been compiled and linked with the Ammasso MPICH driver, it can be run like any other MPI application. The example cpi.c program that was compiled and linked above on 2 systems can be used to verify that the MPI installation is correct. -
Page 34: R Emoving The A Mmasso Mpi I Nstallation
<install_dir>/mpich- 1.2.5/share directory. These files are left on the system for use in the future if needed. # . /etc/ammasso.conf # $INSTALL_DIR/mpich-1.2.5/mpich_uninstall Uninstall mpich found at /usr/opt/ammasso/mpich-1.2.5 (y or n)? [no] y The following files were saved: /usr/opt/ammasso/mpich-1.2.5/share/machines.sample /usr/opt/ammasso/mpich-1.2.5/share/machines.LINUX If a 32b library exists on a 64b system, remove that installation as well: # . -
Page 35: Viadev E Nvironment V Ariables
NP is the total number of processes in the MPI run. Assuming each node runs the same number of processes, NP is computed as: NP = (NC * LP) RP is defined as the number of remote processes relative to any given MPI process in an MPI run. - Page 36 VIADEV_MAX_RENDEZVOUS 209715200 Bytes The maximum amount of user buffer memory that will be locked down at any point in time for zero-copy 4.7.1.1 VIADEV_NUM_RDMA_BUFFERS This parameter specifies how many RDMA Write buffers will be setup per connection. The amount of memory consumed per process for RDMA Write buffers is: 2 * VIADEV_NUM_RDMA_BUFFERS * 8360 * RP Thus, by default, on a 16 node run with 1 process on each node (NC=16, LP=1, NP=16, RP=15), the memory used for RDMA Write buffers would be 2 * 256 * 8360 * 15 =...
-
Page 37: T Uning Suggestions
Each MPI process is allowed to lock down up to VIADEV_MAX_RENDEZVOUS amount of application buffer memory. Once this limit is reached, IO buffers are evicted and unregistered on a “least recently used” basis. The default value for this parameter is 209715200 bytes (200MB). 4.7.2 Tuning suggestions Based on the default values, one can determine how much memory is being consumed by each MPI process in a run. -
Page 38: The Ammasso Dapl Library
5 The Ammasso DAPL Library 5.1 Overview The Ammasso 1100 DAPL release is composed of one source code tar package. AMSO_DAPL.tgz is the DAPL package. This package must be installed to enable both user and kernel mode uDAPL and kDAPL applications over the Ammasso 1100 adapter. -
Page 39: M Akefile T Argets
5.2.1 Makefile Targets The DAPL Makefile supports the following targets: all: Configures and builds DAPL (default target). config: Configures the DAPL source tree. This rule creates the file Config. build: Builds DAPL -- depends on config target. install: Installs DAPL -- depends on build target. uninstall: Uninstall DAPL if it is installed. -
Page 40: C Onfiguring Dapl
/etc/dat.conf. The file is created with the Ammasso DAPL provider already registered, so no modifications are needed by default. If an /etc/dat.conf file already exists at install time, the Ammasso entry will be appended into the /etc/dat.conf file to allow multiple providers on one system. -
Page 41: K Dapl I Nstallation V Erification
Use the dapl_uninstall command to remove the AMSO_DAPL installation: # . /etc/ammasso.conf # $INSTALL_DIR/dapl-1.2/dapl_uninstall Uninstall dapl found at /usr/opt/ammasso/dapl-1.2 (y or n)? [no] y Dapl in /usr/opt/ammasso/dapl-1.2 has been removed. 5.6 Ammasso DAPL Compatibility Settings The following shell environment variables are provided to enable compatibility with other DAPL provider libraries. -
Page 42: Dapl_1_1_Rdma_Iov_Defaults
Infiniband DAPL providers that are ported to the Ammasso DAPL provider. The solution is to explicitly set local read access on memory regions which is valid for both IB and iWARP. Ammasso provides a workaround for this issue: Set CCAPI_ENABLE_LOCAL_READ=1 in your environment before executing your uDAPL application. -
Page 43: Cluster Installation
6 Cluster Installation 6.1 Introduction The purpose of this chapter is to provide a sample install session for the Ammasso 1100 adapter software (AMSO1100), MPICH and DAPL packages. From the install on one node, the software is then deployed to several nodes across a cluster. The steps to install on an initial node in the cluster are different than those used for the follow on cluster nodes. -
Page 44: I Nstall Amso1100 And B Uild B Inary For C Luster D Eployment
* The installer has detected /usr/opt/ammasso/data/app64-01/rnic_cfg. * Please answer 'no' below to keep this configuration. Configure interfaces of the Ammasso 1100 adapter? (y or n) [y] Configure the RDMA network interface IP address settings? (y or n) [y] Please enter the RDMA IP address (10.40.32.53): Please enter the RDMA network mask (255.255.240.0):... - Page 45 # ls /usr/opt/ammasso/data app64-01 default # ls /usr/opt/ammasso/data/app64-01 mode rnic_cfg # tail -12 /usr/opt/ammasso/data/app64-01/rnic_cfg function amso_rdma_0_0 { AMSO_IPADDR=10.40.32.53 AMSO_MASK=255.255.240.0 AMSO_GW= AMSO_MTU=1500 function amso_ccil_0_0 { AMSO_IPADDR=10.40.48.53 AMSO_MASK=255.255.240.0 AMSO_GW= AMSO_MTU=1500 AMSO1100 is now installed on the build system. Note the file rnic_cfg which is used to set the IP settings of the RNIC.
-
Page 46: I Nstall Mpich And B Uild B Inary For C Luster D Eployment
# make config Enter the AMSO1100 build path (/tmp/AMSO1100): Base directory to install mpich (/usr/opt/ammasso): Enter path to c compiler (/usr/bin/gcc): Enter path to c++ compiler (/usr/bin/g++): Enter path to fortran 77 compiler (/usr/bin/g77):... - Page 47 Therefore the make binary will prompt to overwrite the existing installation. Say yes at this point if the MPICH installation directory (/usr/opt/ammasso/mpich-1.2.5) can be over-written and the make binary will re-install and make the binary images. If the MPICH installation directory has been modified it will be necessary to save that directory prior to doing the make binary and restore afterwards.
-
Page 48: I Nstall Dapl And B Uild B Inary For C Luster D Eployment
The proper response is to answer ‘y’. # cd /tmp/AMSO_DAPL/ # make install ..output of build, make takes approximately 5 minutes ..# ls /usr/opt/ammasso/ data lib64 mpich-1.2.5 scripts support dapl-1.2... -
Page 49: C Lean Up Amso Directories And Files
# scp app64-01:/tmp/ammasso1100.bin /tmp 3. On each cluster node, install the adapter software. The –d flag specifies which directory to use to hold the rnic_cfg file. This can be any directory on the system. # cd /tmp/ # ls ammasso* ammasso1100.bin... - Page 50 # /tmp/ammasso1100.bin –d /usr/opt/ammasso/data/app64-02 Ammasso 1100 binary installer created on ..Using /usr/opt/ammasso/data/app64-02 as the configuration directory. Installation complete. Reboot the system to activate the AMSO1100 board and its software # ls /usr/opt/ammasso data lib64 release scripts starcore_cset support # ls /usr/opt/ammasso/data...
- Page 51 This can be done, for example, using scp(1) or rcp(1), for example: # scp foo:/tmp/image-mpich-1.2.5.bin /tmp # cd /tmp/ # ls image* image-mpich-1.2.5.bin # /tmp/image-mpich-1.2.5.bin Ammasso Mpich binary installer created on ..Mpich has been installed into /usr/opt/ammasso/mpich-1.2.5. # ls /usr/opt/ammasso data lib64 mpich-1.2.5...
-
Page 52: C Luster D Eployment
6.4 Cluster Deployment For deployment on a large number of clusters, a cluster administrator will want to script an installation procedure with: 1) an uninstall of previous Ammasso release software 2) an install of ammasso1100.bin, image-mpich-1.2.5.bin, and image-dapl-1.2.bin 3) place a new rnic_cfg for the system... -
Page 53: Using The Ammasso 1100 With Pxe Boot
7.2 Requirements In order to use the Ammasso 1100 for PXE booting you will need the following set up. This list is provided for reference. Examples are provided in later sections which describe how to accomplish each of these. - Page 54 1. A target machine that is PXE capable. The Ammasso 1100 card must be selected as the network boot device. 2. The Ammasso driver, ccil.{o,ko}, compiled against the kernel that will be PXE booted. 3. A ramdisk image that contains the Ammasso driver and utilities. The following files are required for the ramdisk to load.
-
Page 55: B Uilding The Ramdisk
ROM if it is enabled in the BIOS. If PXE is enabled in your motherboard’s BIOS, the Ammasso option ROM will begin to execute the PXE support. However, each user must enable PXE boot in the boot menu of your motherboard’s BIOS. See your motherboard documentation on how to do this. -
Page 56: C Onfiguring And B Uilding B Usy B Ox A Pplications
To simplify building, all applications are shown to be linked statically. For the next few steps we assume that the shell variables, BUSYBOX and MODUTILS, have been set to the path of the sources for their namesakes. The variable INITRD should be set to a temporary directory where the ramdisk will be built. -
Page 57: P Opulating The R Amdisk
7.4.3 Populating the Ramdisk Create an initrd directory structure to be used for the ramdisk. For this example, the initrd directory structure is shown below: # ls ${INITRD} ccore linuxrc modules proc sbin share Copy the BusyBox files: # cp -a ${BUSYBOX}/_install/bin/* ${INITRD}/bin # cp -a ${BUSYBOX}/_install/sbin/* ${INITRD}/sbin Copy insmod from modutils: # cp ${MODUTILS}/insmod/insmod.static ${INITRD}/bin/insmod... - Page 58 /modules/net/sunrpc.ko insmod /modules/net/lockd.ko insmod /modules/net/nfs.ko # Load the RAW socket support driver #insmod /modules/net/af_packet.ko # Load the Ammasso 1100 firmware echo /ccore/ccos.boot /ccore/ccos.bin 0x`cat /ccore/ccos.bin_start` /ccore/ccos.boot /ccore/ccos.bin 0x`cat /ccore/ccos.bin_start` # Load the ccilnet driver insmod /modules/net/ccil.ko # Define the interface name export IFACE=ccilnet0 ifconfig lo 127.0.0.1 up...
-
Page 59: B Uilding The R Amdisk I Mage
# Display the routing table route -n # Mount what will become the root filesystem mount -t nfs -o nolock,rsize=8192,wsize=8192,hard,intr ${NFSDIR} /mnt # We don't need these anymore, and they will get remounted later, anyway umount /dev/pts umount /proc # Switch the mount points - the NFS mounted FS will become / cd /mnt pivot_root . - Page 60 There needs to be one entry for each MAC address in the "group" section of the file. A sample dhcpd.conf configuration file is shown here: authoritative; ddns-update-style interim; option domain-name "ammasso.com"; shared-network lab { # cn0 subnet subnet 10.40.48.0 netmask 255.255.240.0 { option routers 10.40.48.1;...
-
Page 61: B Uilding Pxelinux
7.7 Building PXELINUX In this example, we use PXE to boot Linux with PXELINUX from the syslinux package. This can be obtained from the following website: http://www.kernel.org/pub/linux/utils/boot/syslinux/ To build PXELINUX, you will need nasm(1), an x86 assembler; which can be obtain from the following website: http://sourceforge.net/projects/nasm You only need to build pxelinux.0. - Page 62 7.8.1 Configure a Root File System The idea behind a diskless node is to use the distribution that has been installed on a remote server via NFS. We used a two step process to create the root file system for a client on the server.
-
Page 63: U Pdating The A Mmasso 1100 O Ption Rom I Mage
7.9 Updating the Ammasso 1100 Option ROM Image The Ammasso 1100 network adapter was factory flashed with an option ROM image. If you are updating from a previous release, this image will need to be updated (see HOWTO_UPDATE.txt for more specifics on updating the software and hardware). -
Page 64: Appendix A: Support
Once received, the customer is expected to properly package the product to ensure safe return to the designated facility. Ammasso, Inc will not be responsible for any damage to the adapter incurred during shipment, or due to inadequate packaging. Be sure to use the... -
Page 65: Appendix B: Warranty
Ammasso High Performance Server Adapter LIMITED LIFETIME HARDWARE WARRANTY Ammasso warrants to the original owner that its adapter product will be free from defects in material and workmanship. This warranty does not cover the adapter product if it is damaged in the process of being installed or improperly used. - Page 66 Ammasso warrants the media to be free from material physical defects for a period of ninety (90) days after delivery by Ammasso. If such a defect is found, return the media to Ammasso for replacement or alternate delivery of the Software as Ammasso may select. Software licenses for the Ammasso 1100 Ethernet Adapter can be found at www.ammasso.com, or...
Need help?
Do you have a question about the 1100 and is the answer not in the manual?
Questions and answers