Sun Microsystems Dual Port 4x IB Host Channel Adapter PCI-ExpressModule User Manual
  1. Manuals
  2. Brands
  3. Sun Microsystems Manuals
  4. Control Unit
  5. Dual Port 4x IB Host Channel Adapter...
  6. User manual
Sun Microsystems Dual Port 4x IB Host Channel Adapter PCI-ExpressModule User Manual

Sun Microsystems Dual Port 4x IB Host Channel Adapter PCI-ExpressModule User Manual

Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
Sun
Dual Port 4x IB Host Channel
™
Adapter PCI-ExpressModule User's
Guide
Sun Microsystems, Inc.
www.sun.com
Part No. 819-6182-11
July 2006, Revision A
Submit comments about this document at:
http://www.sun.com/hwdocs/feedback

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Dual Port 4x IB Host Channel Adapter PCI-ExpressModule and is the answer not in the manual?

Questions and answers

Related Manuals for Sun Microsystems Dual Port 4x IB Host Channel Adapter PCI-ExpressModule

Summary of Contents for Sun Microsystems Dual Port 4x IB Host Channel Adapter PCI-ExpressModule

  • Page 1 Dual Port 4x IB Host Channel ™ Adapter PCI-ExpressModule User’s Guide Sun Microsystems, Inc. www.sun.com Part No. 819-6182-11 July 2006, Revision A Submit comments about this document at: http://www.sun.com/hwdocs/feedback...
  • Page 2 Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and Sun™ Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun’s licensees who implement OPEN...
  • Page 3 Contents Preface xxiii Overview 1 Product Features 1 Hardware Overview 2 I/O Interfaces 3 InfiniBand Interface 3 PCI-Express Interface 3 LED Displays 3 Hardware, Software, and Cable Requirements 4 Installing the Adapter 5 Installing the Hardware 5 ▼ To Install the Adapter 5 Verifying the Installation 6 ▼...
  • Page 4 InfiniBand Software Overview 13 InfiniBand Software for the Solaris Operating System 13 User Direct Access Programming Library 14 IP over InfiniBand Kernel 14 InfiniBand Transport Framework 15 InfiniBand Management Framework 15 Tavor HCA Driver 15 Related Software Documentation 16 Manuals 16 Man Pages 16 Downloading the Software for the Linux Operating System 17 InfiniBand Cable Pin Assignments 19...
  • Page 5 Dennis P. Symanski DATE Donald Cameron DATE Worldwide, Compliance Engineering Program Manager/Customer Quality Sun Microsystems, Inc. Sun Microsystems Scotland, Limited 4150 Network Circle, MPK15-102 Blackness Road, Phase I, Main Bldg. Santa Clara, CA 95054 U.S.A. Springfield, EH49 7LR Tel: 650-786-3255...
  • Page 6 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...
  • Page 7 Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. Modifications: Any modifications made to this device that are not approved by Sun Microsystems, Inc. may void the authority granted to the user by the FCC to operate this equipment.
  • Page 8 ICES-003 Class A Notice - Avis NMB-003, Classe A This Class A digital apparatus complies with Canadian ICES-003. Cet appareil numérique de la classe A est conforme à la norme NMB-003 du Canada. ICES-003 Class B Notice - Avis NMB-003, Classe B This Class B digital apparatus complies with Canadian ICES-003.
  • Page 9 BSMI Class A Notice The following statement is applicable to products shipped to Taiwan and marked as Class A on the product compliance label. CCC Class A Notice The following statement is applicable to products shipped to China and marked with “Class A” on the product’s compliance label.
  • Page 10 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...
  • Page 11 Do not make mechanical or electrical modifications to the the equipment’s electrical rating label. equipment. Sun Microsystems is not responsible for Never push objects of any kind through openings in ■ regulatory compliance of a modified Sun product.
  • Page 12 instructions provided in the product service Caution – Not all power cords have the same manual. Do not disassemble batteries or current ratings. Do not use the power cord attempt to recharge them outside the system. provided with your equipment for any other Do not dispose of batteries in fire.
  • Page 13 Assurez-vous que la tension et la fréquence de votre ■ source d'alimentation correspondent à la tension et à Caution – To prevent extreme operating la fréquence indiquées sur l'étiquette de la tension temperatures due to reduced airflow électrique nominale du matériel consideration should be made to the amount N'introduisez jamais d'objets quels qu'ils soient dans of air flow that is required for a safe operation...
  • Page 14 Modification du matériel utilisés avec des ordinateurs. N'utilisez jamais N'apportez aucune modification mécanique ou électrique de cordon d'alimentation à usage domestique au matériel. Sun Microsystems décline toute responsabilité avec les produits Sun. quant à la non-conformité éventuelle d'un produit Sun modifié.
  • Page 15 L'avertissement suivant s'applique uniquement aux Couvercle de l'unité systèmes équipés d'un interrupteur Veille: Pour ajouter des cartes, de la mémoire ou des périphériques de stockage internes, vous devez retirer le couvercle de Attention – L'interrupteur d'alimentation de votre système Sun. Remettez le couvercle supérieur en ce produit fonctionne uniquement comme un place avant de mettre votre système sous tension.
  • Page 16 Treffen Sie zu Ihrem eigenen Schutz bei der Installation des Nehmen Sie keine elektrischen oder mechanischen Geräts die folgenden Sicherheitsvorkehrungen: Gerätemodifikationen vor. Sun Microsystems ist für die Einhaltung der Sicherheitsvorschriften von modifizierten Beachten Sie alle auf den Geräten angebrachten ■...
  • Page 17 Aufstellung von Sun-Geräten Netzkabel an eine frei zugängliche geerdete Steckdose in der Nähe des Systems ange- schlossen ist. Schließen Sie das Stromkabel Achtung – Geräteöffnungen Ihres Sun- nicht an, wenn die Stromversorgung vom Produkts dürfen nicht blockiert oder Systemchassis entfernt wurde. abgedeckt werden.
  • Page 18 Kippschutz am Rack, um ein Umkippen zu ella se explican las medidas de seguridad que debe tomar al vermeiden. instalar un producto de Sun Microsystems. Medidas de seguridad Achtung – Um extreme Betriebstemperaturen Para su protección, tome las medidas de seguridad im Rack zu vermeiden, stellen Sie sicher, dass siguientes durante la instalación del equipo:...
  • Page 19 No realice modificaciones de tipo mecánico ni eléctrico en el cuentan con protección frente a sobrecargas y equipo. Sun Microsystems no se hace responsable del no están diseñados para su utilización con cumplimiento de normativas en caso de que un producto sistemas informáticos.
  • Page 20 hace las veces de dispositivo de desconexión principal del sistema. Asegúrese de que Precaución – No ponga en funcionamiento los productos Sun que no tengan colocada la conecta el cable de alimentación a una toma cubierta. De lo contrario, puede sufrir lesiones de tierra situada cerca del sistema y de fácil personales y ocasionar daños en el sistema.
  • Page 21 Aviso de cumplimiento de la normativa para Danmark la utilización de láser Los productos Sun que utilizan tecnología láser cumplen los Advarsel! – Litiumbatteri — Eksplosionsfare requisitos establecidos para los productos láser de clase 1. ved fejlagtig håndtering. Udskiftning må kun ske med batteri af samme fabrikat og type.
  • Page 22 xxii Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...

  • Page 23: How This Document Is Organized

    Preface The Sun™ Dual Port 4x IB Host Channel Adapter PCI-ExpressModule User’s Guide provides detailed procedures for installing the hardware and software for the Sun IB-Host Channel Adapter PCI-ExpressModule. This manual includes an overview of the adapter, supported platforms, cable information, and related documentation.

  • Page 24: Using Unix Commands

    Using UNIX Commands ® This document might not contain information about basic UNIX commands and procedures such as shutting down the system, booting the system, and configuring devices. Refer to the following for this information: Software documentation that you received with your system ■...

  • Page 25: Typographic Conventions

    Typographic Conventions Typeface Meaning Examples The names of commands, files, Edit your.login file. AaBbCc123 and directories; on-screen Use ls -a to list all files. computer output % You have mail. What you type, when contrasted AaBbCc123 with on-screen computer output Password: AaBbCc123 Book titles, new words or terms,...

  • Page 26: Related Documentation

    Related Documentation The documents listed as online are available at: http://www.sun.com/products-n-solutions/hardware/docs/ Part Application Title Number Format Location Installation Sun Dual Port 4x IB Host Channel Adapter 819-6183 Online PCI-ExpressModule Release Notes HTML 817-5093 Service System Administration Guide: Devices and Online Files System HTML 816-4555...

  • Page 27: Documentation, Support, And Training

    Documentation, Support, and Training Sun Function Documentation http://www.sun.com/documentation/ Support http://www.sun.com/support/ Training http://www.sun.com/training/ Third-Party Web Sites Sun is not responsible for the availability of third-party web sites mentioned in this document. Sun does not endorse and is not responsible or liable for any content, advertising, products, or other materials that are available on or through such sites or resources.
  • Page 28 xxviii Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...

  • Page 29: Product Features

    C H A P T E R Overview This chapter provides an overview of the Sun IB-Host Channel Adapter PCI- ExpressModule. It includes the following sections: “Product Features” on page 1 ■ “Hardware Overview” on page 2 ■ “Hardware, Software, and Cable Requirements” on page 4 ■...

  • Page 30: Hardware Overview

    The standard PCI-Express interface is used for data traffic, management traffic, and to generate and receive inbound configuration packets Sun Dual Port 4x IB Host Channel Adapter PCI-ExpressModule FIGURE 1-1 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...
  • Page 31 I/O Interfaces The IB-EM includes the following interfaces: Two 4x InfiniBand copper connectors ■ PCI-E x8-lane edge connector ■ InfiniBand Interface The IB-EM provides two 4x InfiniBand v1.1 connector (or ports) for external copper cables and is IBTA specification 1.1-compliant. PCI-Express Interface The PCI-Express bus is a PCI-Express x8 version 1.0a-compliant interface.
  • Page 32 Sun 4x-to-12x InfiniBand cable, 5 meter length Note – Use only supported cables to ensure reliable InfiniBand interface connections. These cables can be ordered from Sun Microsystems, Inc. Connecting to other IB switch vendor ports may require 4x-4x IB cables. Refer to the documentation for the IB switch you use.

  • Page 33: Installing The Adapter

    C H A P T E R Installing the Adapter This chapter describes how to install both the Sun IB-Host Channel Adapter PCI- ExpressModule hardware and verify that it has been installed correctly. It contains the following section: “Installing the Hardware” on page 5 ■...

  • Page 34: Verifying The Installation

    Caution – Avoid putting unnecessary stress on the connection. Do not bend or twist the cable near the connectors and avoid cable bends of more than 90 degrees. 4. If not already connected, connect the 12x end of the InfiniBand I/O cable(s) to the appropriate ports on the InfiniBand switch.
  • Page 35 5. Verify that the IB-EM ports are up and the driver is attached. To do this, type the following command: # dmesg | grep tavor The output shows recently printed system diagnostic messages that have the string tavor in the message. Included in the output is a message that tells whether the port is up or down.
  • Page 36 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...
  • Page 37 C H A P T E R Configuring IP Over InfiniBand This chapter describes how to configure IP over InfiniBand (IPoIB). This chapter contains the following sections: “IPoIB Configuration” on page 9 ■ IPoIB Configuration The following procedure can be used when making changes to your configuration. See System Administration Guide: Basic Administration (817-1985) for Solaris setup information.
  • Page 38 For example: Feb 17 20:30:42 hostname px_pci: [ID 370704 kern.info] PCI-device: pciex15b3,6278@0, tavor0 Feb 17 20:30:42 hostname genunix: [ID 936769 kern.info] tavor0 is /pci@7c0/pci@0/pci@8/pciex15b3,6278@0 Feb 17 20:30:42 hostname tavor: [ID 548581 kern.info] tavor0: FW ver: 0004.0007.0190, HW rev: a0 Feb 17 20:30:42 hostname tavor: [ID 827127 kern.info] tavor0: MT23108 Sun Microsystems InfiniBand HCA (0x0003ba0001001c18) Feb 17 20:30:45 hostname genunix: [ID 979957 kern.info] NOTICE: tavor0: port 1 up Feb 17 20:30:45 hostname genunix: [ID 611667 kern.info] NOTICE: tavor0: port 2 up...
  • Page 39 4. Plumb each of the IB devices. To do this, type the following command for each IB-EM port in the system: # ifconfig ibdn plumb host-ipv4-address up where is the instance number of the interface and is the designated host-ipv4-address IPv4 address for the device.
  • Page 40 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...
  • Page 41 C H A P T E R InfiniBand Software Overview This chapter provides an overview of the IB software for both Solaris and Linux Operating Systems. This chapter includes the following topics: “InfiniBand Software for the Solaris Operating System” on page 15 ■...
  • Page 42 Solaris 10 OS IPoIB uDAPL IBTF IBMF Tavor HCA Driver IB-HCA Port 1 Port 2 InfiniBand Software Stack FIGURE 4-1 User Direct Access Programming Library This library supports service providers written according to the uDAPL 1.2 specification. Sun provides default service provider modules to support the Tavor HCA driver.
  • Page 43 InfiniBand Transport Framework IBTF is the Solaris implementation of the InfiniBand Transport Layer (IBTL). IBTF consists of InfiniBand Device Manager (IBDM), InfiniBand Communications Manager (IBCM) and an InfiniBand Nexus driver. The Nexus driver along with IBDM creates and manages InfiniBand devices. The following devices are supported: IB I/O controller (IOC) devices ■...

  • Page 44: Related Software Documentation

    Related Software Documentation Manuals System Administration Guide: Devices and Files System (817-5093) ■ For information on InfiniBand devices, see the System Administration Guide: Devices and Files System (817-5093) System Administration Guide: Network Services (816-4555) ■ For general information on the InfiniBand network, see the System Administration Guide: Network Services (816-4555).
  • Page 45 Downloading the Software for the Linux Operating System If you are using the Linux operating system, use the Mellanox software to operate the IB-EM adapter. The Mellanox software and documentation are available at the following web site: https://docs.mellanox.com/dm/ibg2/ReadMe.html http://www.openfabrics.org/downloads.html Chapter 4 InfiniBand Software Overview...
  • Page 46 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...

  • Page 47: Cable Pin Assignments

    A P P E N D I X InfiniBand Cable Pin Assignments This appendix lists the cable pin assignments for the Sun IB-Host Channel Adapter PCI- ExpressModule. It contains the following section: “Cable Pin Assignments” on page 19 ■ Cable Pin Assignments shows the InfiniBand cable connectors and pin assignments.
  • Page 48 P1 (12X) P2 (4X) PAIR 1 PAIR 1 PAIR 2 PAIR 2 PAIR 3 PAIR 3 PAIR 4 PAIR 4 PAIR 5 PAIR 5 PAIR 6 PAIR 6 PAIR 7 PAIR 7 PAIR 8 PAIR 8 SHELL SHELL Sun 4x-to-12x InfiniBand Cable Pin Assignments FIGURE A-1 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide •...

  • Page 49: Diagnostic Software And Troubleshooting

    A P P E N D I X Diagnostic Software and Troubleshooting This appendix provides an overview of the SunVTS™ diagnostic application and troubleshooting tips for checking the adapter. There is also a section outlining some common troubleshooting issues. This appendix contains the following sections: “SunVTS Diagnostic Testing”...
  • Page 50 SunVTS Documentation TABLE B-1 Title Descriptions SunVTS 6.0 PS1 Documentation Describes the new SunVTS features and tests, including Supplement (819-1804) the ibhcatest. SunVTS 6.0 User’s Guide Describes the SunVTS diagnostic environment. (817-7664) SunVTS 6.0 Test Reference Manual Describes each SunVTS test and describes the various (817-7665) test options and command-line arguments.

  • Page 51: Troubleshooting Tasks

    Using the ibhcatest ▼ 1. Ensure that the SunVTS software and the Tavor driver are installed on your system. To do this, type the following command: # pkginfo SUNWvts SUNWvtsx SUNWtavor If a SunVTS software package is not installed, refer to the SunVTS User’s Guide for installation instruction.
  • Page 52 If an InfiniBand software package is not installed, check your Solaris Operating System documentation for software package information. Check the system log or console for Tavor driver error messages. ■ See tavor(7D) for error messages and descriptions. When the driver is attached to a port on the adapter, the following message is sent.
  • Page 53 snoop The snoop program captures and inspects network packets. See the snoop(1M) man page for details. For example: # snoop -d ibd1 Using device /dev/ibd1 (promiscuous mode) ib-1-167 -> * ARP C Who is 199.1.1.168, ib-1-168 ? ib-1-168 -> ib-1-167 ARP R 199.1.1.168, ib-1-168 is 0:2:4:7:0:0:0:0:a:4:7c:4f:0:2:c9:2:0:0:55:91 ib-1-167 ->...
  • Page 54 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...
  • Page 55 Index InfiniBand, 3 PCI-X, 3 adapter IPoIB, 13 features, 1 configuring, 9 cable requirements, 4 LEDs cables descriptions, 3 connectors, 19 layout, 4 part numbers, 4 verification, 6 pin assignments, 19 operating systems, 4 diagnostics, ibhcatest, 22 documentation man pages, 16 requirements related, 16 hardware, 4...
  • Page 56 tasks, 23 utilities, 24 uDAPL, 13, 14 utilities, 24 cfgadm, 24 kstat, 25 netstat, 25 snoop, 25 Sun IB Host Channel Adapter PCI-ExpressModule User’s Guide • July 2006...

Table of Contents