1. Manuals
  2. Brands
  3. Voltaire Manuals
  4. Switch
  5. ISR 9024
  6. User manual

Voltaire ISR 9024 User Manual

Industrial rackmount swithes
Hide thumbs
Table of Contents

Advertisement

Quick Links

Voltaire Switch User Manual
ISR 9024, ISR 9096, and ISR 9288 Switches
Business Headquarters
Voltaire Inc.
6 Fortune Drive, Suite 301
Billerica, MA USA 01821
Tel: 978-439-5400
Fax: 978-439-5401
Release AAA-BAA
May 2006
P/N 399Z00038
Israel Office
Voltaire Ltd.
9 Hamenofim St.
Bldg. A Herzeliya
46725, Israel
Tel: +972 (9) 971-7666
Fax: +972 (9) 971-7660

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ISR 9024 and is the answer not in the manual?

Questions and answers

Related Manuals for Voltaire ISR 9024

Summary of Contents for Voltaire ISR 9024

  • Page 1 Voltaire Switch User Manual ISR 9024, ISR 9096, and ISR 9288 Switches Release AAA-BAA May 2006 P/N 399Z00038 Business Headquarters Israel Office Voltaire Inc. Voltaire Ltd. 6 Fortune Drive, Suite 301 9 Hamenofim St. Billerica, MA USA 01821 Bldg. A Herzeliya...
  • Page 2 Voltaire Ltd. Voltaire is a trademark or registered trademark of Voltaire Ltd. or its subsidiaries in the United States and other countries.

  • Page 3: Audience

    About this Manual This preface describes the audience, organization and command syntax conventions of the Voltaire Switch User Manual. It also provides information on how to obtain related documentation and technical assistance. Information about hardware installation of Voltaire switches can be found in ISR 9024 Installation Manual and in the ISR 9XXX Installation Manual, which includes the ISR 9096 and the ISR 9288.

  • Page 4: Document Conventions

    Text set off in this manner indicates important information regarding a specific feature. CAUTION Text set off in this manner indicates that failure to follow directions could result in damage to equipment or loss of information. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 5: Document Organization

    This guide contains the following chapters: Chapter 1 – Introduction: Describes the InfiniBand technology and provides an introduction to the ISR 9024, ISR 9096 and ISR 9288 switches. Chapter 2 – CLI Overview: Provides a brief introduction on the Voltaire Command Line Interface (CLI).
  • Page 6 Appendix A – CLI Commands Reference: Explains the CLI basic concept and details the CLI command menus. The CLI provides configuration and management functions for Voltaire switches, as well as a wide range of provisioning, configuration and maintenance functions for the InfiniBand fabric.

  • Page 7: What's New In This Manual

    Rack Grouping (Creating, renaming, deleting rack groups, and viewing rack information via the GUI and the CLI; groups.csv file) The Voltaire switch software supports a new hardware module: sLB-8-12 Diagnostics Enhancements: Enhanced Event Notification - extended notification mode, New alarm category in current alarm mechanism: Fabric Config Error, Enhanced Performance Manager (PM) [PortCounters files;...

  • Page 8: Table Of Contents

    Document Organization ......................iii What's New in this Manual? ....................... v Chapter 1. Introduction ......................1-1 About InfiniBand......................1-2 ISR 9024 System Overview ..................1-3 ISR 9288/9096 System Overview .................1-4 Voltaire Switch Management ..................1-10 Chapter 2. CLI Overview ......................2-1 Interfaces for CLI Connection..................2-2 CLI Command Modes ....................2-2...
  • Page 9 The Device Manager Main Window ................5-6 Voltaire Device Manager Operating Conventions ............5-16 Viewing Switch Information ..................5-16 Viewing Module Information..................5-27 Using the Voltaire Device Manager for Switch Management........5-40 Reset Operations......................5-45 Chapter 6. Voltaire Fabric Manager (VFM) ................6-1 Fabric Manager Description ..................6-2 Fabric Manager Installation...................6-3...
  • Page 10 Chapter 12. Replacing Modules .................... 12-1 12.1 Replacing sFBs (Spines)....................12-2 12.2 Replacing sLBs......................12-6 12.3 Replacing sMBs ......................12-7 12.4 Replacing an Unmanaged ISR 9024................12-8 Chapter 13. Configuring Cluster Monitor................13-1 13.1 What is Ganglia? ......................13-2 13.2 Cluster Configuration ....................13-2 Chapter 14. Fabric Diagnostic....................14-1 14.1 Fabric Diagnostic Overview..................14-2...
  • Page 11 Contents Appendix A CLI Commands Reference ................. A-1 Appendix B Port Counters Fields................... B-1 Appendix C Supported Traps and Events................C-1 Appendix D Glossary....................... D-1 Appendix E Index........................E-1 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 12 Figures Figure 1-1. Voltaire ISR 9024 Grid Switch™ Functional Diagram........1-4 Figure 1-2. Voltaire ISR 9288 Grid Backbone™ Functional Diagram ......1-5 Figure 1-3. Voltaire ISR 9096 Grid Backbone™ Functional Diagram ......1-5 Figure 1-4. ISR 9288 InfiniScale Switching Topology - Fat Tree Topology..... 1-8 Figure 1-5.
  • Page 13 Figure 5-33. Configuring User Authorization ..............5-43 Figure 5-34. Configuring FTP ..................5-44 Figure 5-35. Warning popup ..................... 5-45 Figure 6-1. Voltaire Fabric Manager Initiate Window............6-5 Figure 6-2. Login Window ....................6-6 Figure 6-3. Exit Warning Message..................6-7 Figure 6-4. Fabric Manager Main Window................ 6-8 Figure 6-5.
  • Page 14 Figure 6-35. Event Log - Example ................... 6-46 Figure 6-36. Error Log – Example ................... 6-47 Figure 6-37. Bad Ports Log ....................6-48 Figure 6-38. PortCounters.csv File - Example ..............6-49 Figure 6-39. Cluster Monitor Dialog Box ................ 6-49 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 15 Figure 8-10. Rack-Groups – Table View ................8-9 Figure 8-11. Absent Elements ..................8-10 Figure 9-1. QoS SL to VL Mapping Table................. 9-3 Figure 9-2. QoS Arbitration Table ..................9-5 Figure 10-1. Routing ......................10-4 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 16 Figure 14-4. Port Counters Graph .................. 14-12 Figure 14-5. Trap Table – 1 X Link ................14-20 Figure 14-6. VFM Event Data Window ................. 14-20 Figure 14-7. Nodes Info Window................... 14-22 Figure 14-8. Bad Ports Log .................... 14-25 viii P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 17: Chapter 1. Introduction

    Introduction In This Chapter This chapter provides a brief introduction to InfiniBand technology and describes Voltaire ISR 9024, ISR 9096 and ISR 9288 switches. The following information is included in these pages: About InfiniBand, on page 1-2 ISR 9024 System Overview, on page 1-3...

  • Page 18: About Infiniband

    CPU, freeing their power to run applications. Offering advanced fault isolation controls, nonexistent in previous protocols, InfiniBand provides a high level of fault tolerance. And most importantly, because the InfiniBand system is modular, processing power based on commodity servers can be added as needed. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 19: Isr 9024 System Overview

    Using the Voltaire ISR 9024 Grid Switch, you can build high performance clusters and grids that scale from several to hundreds of nodes. The ISR 9024 serves as a building block for Fat Tree topologies of hundred of nodes for scientific, commercial HPC and enterprise grid applications. When combined with the Voltaire ISR 9288/9096 high port density switch, the Voltaire ISR 9024 enables the formation of clusters scaling to thousand of nodes.

  • Page 20: Isr 9288/9096 System Overview

    Figure 1-1. Voltaire ISR 9024 Grid Switch™ Functional Diagram ISR 9288/9096 System Overview The Voltaire ISR 9288/9096 Grid Backbone is a modular platform for building high levels of performance and scalability for large high performance computing (HPC) clusters. The Voltaire ISR 9288/9096 enables high performance applications to run on distributed server, storage and network resources.

  • Page 21: Figure 1-2. Voltaire Isr 9288 Grid Backbone™ Functional Diagram

    Introduction ISR 9288/9096 System Overview 420 nanoseconds make the Voltaire ISR 9288/9096 the highest performing switch available. In addition to scalable InfiniBand switching, the Voltaire ISR 9288/9096 offers powerful multi-protocol connectivity in a single, flexible system. In specific configurations, the chassis hosts the Voltaire Router Modules, providing seamless connectivity between server clusters, FC SANs, NAS appliances, IP SANs and TCP/IP networks (LANs).
  • Page 22 N:1 or N:N redundancy. Fan Units (sFU) • The ISR 9288 (only) accommodates a vertical fan unit with 4 fans (sFU-4). • The ISR 9288/9096 accommodates a horizontal fan unit with 8 fans (sFU-8). P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 23 The number of available ports is limited when router modules are installed in the system. The VFM groups all the Line and Fabric Boards in the Chassis and identifies them as a one switch entity, represented as a single icon. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 24: Figure 1-4. Isr 9288 Infiniscale Switching Topology - Fat Tree Topology

    The ISR 9288 features six symmetrical Line Boards (sLB-24) at both ends and four interconnect Fabric Boards (sFB) Each sLB includes two 24 4X IB port chips. Each sFB has three 4X IB port chips. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 25: Figure 1-5. Isr 9096 Infiniscale Switching Topology - Fat Tree Topology

    (alternating between the orange and the green) if the system detects a link that is lower than 12X. However, this does not affect link behavior. Regardless of the blinking LED, the link is still operating and fully functional (data can flow over the link). P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 26: Voltaire Switch Management

    Graphic User Interface (GUI). The Device Manager can be connected via HTTP session. Voltaire Fabric Manager, described in Chapter 6 of this manual. Provides a fabric-wide view of the switch fabric, with an advanced Graphic User Interface (GUI).

  • Page 27: Figure 1-6. Out-Of-Band Management

    Default Gateway Figure 1-7. In-band Management When using in-band management, user Applications (GUI and CLI) manage the Switch internally via a router module and IPoIB traffic from the router to the sMB. 1-11 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 28 Subnet Management temporarily freezes all applications until the sFB is replaced. If there is another sMB installed in the system, the latter takes over the management of the system. 1-12 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 29: Management Interfaces

    CLI and are detailed on (as detailed in Appendix A, FTP Mode Section A.4.5). The Fast interface also provides an interface to the Voltaire Fabric Manager and Voltaire Device Manager GUI applications via HTTP session. (Out of band interface) The Serial interface (RS-232) provides an interface to the CLI.
  • Page 30 The user can change the subnet address of the Internal interface, which is set by default to 10.9.9.0. The IP address of the internal interface of each sMB module is (by default) 10.9.9.1 for the left-hand sMB and 10.9.9.2 for the right-hand sMB. 1-14 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 31: Chapter 2. Cli Overview

    Keyboard Usage in CLI Operation, on page 2-3 CLI Command Structure, on page 2-5 CLI Case Sensitivity, on page 2-5 CLI Command Prompts, on page 2-6 NOTE For the CLI Commands Reference refer to Appendix A. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 32: Interfaces For Cli Connection

    Privileged mode – Allows the user to configure basic parameters such as date/time and reload (reset) switch, and to perform software and firmware updates. Config mode – Allows the user to configure all configuration parameters. Debug mode – Intended for use by Voltaire personnel only. CLI Command Command Prompt Access Method...

  • Page 33: Cli Usage Tips

    Moves the cursor one character to the left. When you enter a command that extends beyond a single line, you can press the Left Arrow key repeatedly to scroll back toward the system prompt and verify the beginning of the command entry. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 34 Deletes from the cursor to the end of the word. Esc F Moves the cursor forward one word. Esc L Changes the word to lowercase at the cursor to the end of the word. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 35: Cli Command Structure

    The CLI is case sensitive. All keywords must be entered in lower case. Any user- defined strings, such as names or descriptions, may appear in any case, including mixed case. Case information for user-defined strings is preserved in the configuration. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 36: Cli Command Prompts

    Exec or Privileged configuration mode, as shown in the Table 2-3. Table 2-3. Command Prompts CLI Command Mode Command Prompt Exec ISRXXxx> Privileged ISRXXxx# Configuration (Config) ISRXXxx(config)# Debug For use by Voltaire personnel only. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 37 Configuring the Management IP Address in ISR 9288/9096/9024 Switches, on page 3-5 Configuring the ISR 9288/9096 with I/O Modules, on page 3-7 Configuring Time and Date, on page 3-9 Configuring Passwords, on page 3-9 Verifying Configuration, on page 3-10 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 38: Chapter 3. Initial System Setup

    Before configuring system parameters, make sure you have completed the hardware installation according to the relevant Installation Guide. Information about hardware installation of Voltaire switches can be found in the ISR 9024 Installation Manual and in the ISR 9XXX Installation Manual (the latter includes the ISR 9096 and the ISR 9288 Switches).

  • Page 39: Connecting A Console

    123456 To change to Privileged mode, type enable Step 5 Type the password at the prompt (the password is voltaire Step 6 Issue the appropriate CLI commands to complete the required actions. Step 7 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 40: Configuring The Management Interface Ip Address

    Sets the IP address of the IB interface. ip-address-ib set The active management interface can be <192.168.70.60> set to the Fast or IB interface. This <255.255.255.0> command can be accessed via the Config/Interface IB Mode. See next section for further information. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 41: Configuring The Management Ip Address In Isr 9288/9096/9024 Switches

    IP address of the Fast interface. However, CLI sessions can be established through both interfaces simultaneously. The following sections describe how to configure the initial management interface for GUI sessions, although this configuration also impacts on the CLI. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 42: Management Interface

    IB The last line in the CLI output above specifies that the IB interface is the active management interface. To set the IB interface as the active management interface: ISR 9288/9096/9024 (config-if-IB)# management-interface set P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 43: Configuring The Isr 9288/9096 With I/O Modules

    Make sure the IP address of each module is on the same subnet as the FAST interface. • Configure each module using the CLI command. io ip-address set The configured modules can be viewed using the CLI command. io show P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 44 Sets the IP address of the I/O module in a ip-address-ib set Step 5 specific slot and specific position in the <slot> <position> ISR 9288/9096 switch <192.168.70.60> <255.255.255.0> Exits the cluster mode. exit Step 6 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 45: Configuring Time And Date

    The factory default password for Exec mode is 123456 and for Privileged mode is voltaire. Command Description Enable [password] Enters Privileged mode (from Exec mode) Step 1 password update [admin | Sets the Privileged and Exec mode passwords Step 2 enable] exit Exits Privileged mode Step 3 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 46: 3.11 Verifying Configuration

    Initial System Setup Verifying Configuration Example: # password update admin Insert new (up to 8 characters) password: [voltaire] Please retype new password: [voltaire] 3.11 Verifying Configuration From the Exec/Privileged mode, verify the system parameters using the steps below. Command Description...

  • Page 47: Chapter 4. Basic Cli Maintenance Commands

    Updating Software, on page 4-2 Prerequisites for Updating Software, on page 4-3 Backing up the Switch Configuration, on page 4-4 Software and Firmware Upgrade Process, on page 4-5 CLI Commands Reference, on page 4-6 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 48: Prerequisites

    Updating Software The same software image can be used to manage either an ISR9096 or the ISR 9288 system. The ISR 9024 has its own software image. The software automatically detects the system type and executes the suitable device management operations.

  • Page 49: Prerequisites For Updating Software

    Configure the user name for FTP server access. Step 5 password Set the user password to access the FTP server. Step 6 ftp show After the update is complete, execute this command to verify the Step 7 new settings. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 50: Backing Up The Switch Configuration

    Uploads the configuration settings of the switch to the FTP Step 7 path> server. exportFile group Uploads the group.cvs file to the remote FTP server, using Step 8 [remote path] the previously defined server name, user name, and password P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 51: Software And Firmware Upgrade Process

    12 unit connected to the sMB. If the version is older than 0.5.0, it is automatically upgraded to the new revision. Note: When performing an automatic firmware upgrade (as described above), the system issues a request to power cycle the ISR 9XXX chassis. This request appears on all terminals. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 52: Cli Commands Reference

    Power down the system and install both sMB modules into the chassis. Step 5 Power up the system with both sMBs. Step 6 CLI Commands Reference Appendix A provides a comprehensive CLI reference detailing all the menus, commands, and usage. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 53: Chapter 5. Voltaire Device Manager (Vdm)

    Voltaire Device Manager (VDM) In This Chapter This chapter provides information about working with the Voltaire Device Manager (VDM) application. It includes information on the following topics: Device Manager Description, on page 5-2 Getting Started with the Device Manager, on page 5-3...

  • Page 54: Device Manager Description

    The Voltaire Device Manager (VDM) is an embedded application with a web-based GUI that provides InfiniBand management functionality for Voltaire ISR 9096 and ISR 9288 and ISR 9024 switches. It enables users to view a graphical representation of the modules, its LEDs and ports, and to monitor and configure device parameters.

  • Page 55: Getting Started With The Device Manager

    Chapter 3. Open the Internet Browser and connect to the fast Ethernet IP address (for Step 2 example: http://172.25.3.13). The Voltaire Device Manager Initiate window is displayed. Figure 5-1. Accessing the Voltaire Multi Protocol Switch Window P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 56: Figure 5-2. Login Dialog Box

    Voltaire Vision GUI. NOTE When opening the Voltaire Device Manager Initiate window for the first time, click to start the Sun Java Web Start installation. The computer that is being used must be connected to the Internet for successful Java Web Start package download from Sun’s website.
  • Page 57 User Settings Section 5.7.3. To Login into the Device Manager: Select the User Type. Step 1 Type in the Password and click OK. The Voltaire Device Manager Main window Step 2 is displayed, as shown in Figure 5-4 and Figure 5-5. NOTE Login password is case sensitive.

  • Page 58: The Device Manager Main Window

    ISR 9024 Voltaire Device Manager application, including basic functions and operations. The Main window provides access to all Voltaire Device Manager functions. It also provides a graphical representation of the connected switch and components, with color-coded visual indicators that display alarms and important events in the switch.

  • Page 59: Figure 5-4. Isr 9288 Voltaire Device Manager Main Window

    Figure 5-4 shows the ISR 9288 Main window: Device Status Bar Toolbar Main Menu ISR 9288 Device Tree ISR 9288 Information Device Area Graphical Representation Trap Table Figure 5-4. ISR 9288 Voltaire Device Manager Main Window P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 60: Figure 5-5. Isr 9096 Voltaire Device Manager Main Window

    Voltaire Device Manager (VDM) The Device Manager Main Window 5.3.2 ISR 9096 Main Window Figure 5-5 shows the ISR 9096 Main window: ISR 9096 Device Graphical Representation Figure 5-5. ISR 9096 Voltaire Device Manager Main Window P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 61 Voltaire Device Manager (VDM) The Device Manager Main Window Main Menu The main menu provides access to the following Voltaire Device Manager functions: Table 5-3. Main Menu Options Menu Submenu Action File Exit–Logs out of the Voltaire Device Manager and terminate all Voltaire Device Manager processes.

  • Page 62: Figure 5-6. Device Tree

    Figure 5-6. Device Tree You can expand/collapse the navigation tree to display the various components of the switch. You can show/hide the device tree from view by selecting View/View Tree from the Main Menu. 5-10 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 63 Telnet– Opens a Telnet window, allowing you to connect to the device through Telnet. VFM– Launches a VFM (Voltaire Fabric Manager) session; see Chapter 6 for a description of the VFM. Notice the little green triangle at the upper left corner of the Icons. Green means that the status is normal.

  • Page 64: Figure 5-7. Isr 9096 Switch Vdm Graphical Representation (Front View)

    LEDs and ports. The ISR 9288/9096 Voltaire Device Manager provides two views of this switch, a front view and a rear view. (This does not apply to the ISR 9024) NOTE When two Management boards (sMB) are installed in failover configuration, the active sMB board is displayed with a border line around it (as shown on the sMB at the left of the ISR 9096 Switch Director, in Figure 5-7).

  • Page 65: Figure 5-9. Isr 9288 Switch Graphical Representation (Front View)

    Figure 5-9 and Figure 5-10 show the graphical representations of the ISR 9288 switch. Figure 5-9. ISR 9288 Switch Graphical Representation (Front View) Click to display the rear view of the ISR 9288 switch, shown in Figure 5-10. 5-13 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 66: Figure 5-10. Isr 9288 Switch Graphical Representation (Rear View)

    The Device Manager Main Window Figure 5-10. ISR 9288 Switch Graphical Representation (Rear View) ISR 9024 Figure 5-11 shows the graphical representation of the ISR 9024 switch. Figure 5-11. ISR 9024 Switch Graphical Representation 5-14 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 67: Trap Table

    IB ports—click on one of these ports to obtain further information on the port operating status; see 5.5.2, Section 5.5.2. ISR 9024—clicking on any point in the graphical representation, except the ports, displays device general information. ISR 9288/9096—clicking on boards in the graphical representation zooms in on the board and displays information windows on the board.

  • Page 68: Voltaire Device Manager Operating Conventions

    The following are specific operation conventions that apply to the Voltaire Device Manager. 5.4.1 Sorting Tables You can sort the data in the Voltaire Device Manager tables by clicking the column headers. An arrow indicates the sort order (ascending or descending). Use Ctrl-click for a secondary sort order. 5.4.2...

  • Page 69: Figure 5-12. Device Information

    Displays a text description of the switch. Switch Description Displays the switch description. Device Revision Displays the device revision number. Software Version Displays the software version of the switch. Hardware Version Displays the hardware version of the switch. 5-17 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 70 Displays the user-defined contact person assigned to this device. Location Displays the user-defined location assigned to this device. Click advanced to obtain detailed module information (Refer to Obtaining Step 2 Detailed Module Information Section 5.5.6 for further information.) 5-18 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 71: Figure 5-13. Port Information

    To view port information: In the Main Window Switch graphical representation view, click on a port ( Step 1 The following information is displayed in the information area of the main window. Figure 5-13. Port Information 5-19 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 72 Displays the Head of Queue lifetime of the port. The port discards a packet from transmission if the Head of Queue life has expired. Port MTU Displays the Maximum Transmission Unit value of the port. 5-20 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 73: Figure 5-14. Cpu Information

    Figure 5-14. CPU Information For ISR 9288/9096 switches, the status bar displays the relevant sMB card for which the CPU information is displayed; it also indicates whether the sMB card is the active or standby card. 5-21 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 74 CPU. Idle Time The percentage of time (of the total usage time) the CPU is idle. Waiting Time The percentage of time (of the total usage time) the CPU is waiting to process. 5-22 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 75: Figure 5-15. Temperature Readings

    All fields in this dialog box are read only. Module temperatures are displayed both in Fahrenheit and in Celsius. The status bar at the top of the dialog box indicates an overheating alarm condition, should one exist. 5-23 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 76: Figure 5-16. Local Sm Information

    5.5.5 Local SM Information To view local SM information: Click (SM icon) on the Device toolbar. The Local SM window is displayed in the Information area. Figure 5-16. Local SM Information 5-24 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 77 The collection and use of vital product data allows the switch system to be understood and service provided more quickly. Detailed VPD module information can be obtained by clicking the Advanced button. The detailed Module Information Window is displayed. 5-25 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 78: Figure 5-17. Detailed Module Information

    Port Number The part number, if applicable. Hardware Version The hardware version of the module, if applicable. Serial Number The module serial number. Production Date The date on which the module was manufactured. 5-26 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 79: Viewing Module Information

    The information described in this section varies according to the module selected, as described below: sLB (Line Board) Information on page 5-28 Port Information, on page 5-18 CPU Information, on page 5-21 Temperature Information, on page 5-23 Local SM Information, on page 5-23 5-27 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 80: Figure 5-18. Slb Information

    Hardware Version. Additional VPD can be obtained by clicking the Advanced Button. (Refer to Obtaining Detailed Module Information Section 5.5.6 for further information). The center of the window displays the following information on each of the Line Board ports. 5-28 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 81 Displays the Maximum Transmission Unit value of the port. Displays the Virtual Lanes supported by the port. Current support is for VLs 0 to 7. The lower part of the window displays the status of each port and related events. 5-29 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 82: Figure 5-19. Slb-24 With Optical Adapters

    VDM to differentiate them from the IB connector, as shown below. Connected Connected Empty Optical Adapter IB Connector IB Connector Figure 5-19. sLB-24 with Optical Adapters Further detail is available in the ISR 9XXX Installation Manual. 5-30 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 83: Figure 5-20. Isr 9288 Fabric Board Information

    The blue Hot Swap LED is active when the sFB board is ready to be removed. The board’s VPD (Vital Product Data) is also displayed. Additional VPD can be obtained by clicking the Advanced Button (refer to Obtaining Detailed Module Information, Section 5.5.6 for further information). 5-31 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 84: Figure 5-22. Smb Information

    Management are active and the status of the power and info LED. The blue Hot Swap LED is active when the sMB board is ready to be removed. The board’s VPD (Vital Product Data) is also displayed. Additional VPD can be 5-32 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 85: Figure 5-23. Smb Detailed Module Information Window

    The unique identifier of the module. CPLD Version The version of the Logic Device PR-PMC Version The version of the PowerPC Processor PCI Mezzanine Card Refer to Obtaining Detailed Module Information, Section 5.5.6 for further information. 5-33 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 86: Figure 5-24. Sctrl Information

    Table 5-12 describes the features of the sCTRL information window. Table 5-12. sCTRL Information Window Features Feature Description sMB1 and 2 LEDs show the status of the Subnet Manager and the Chassis Management Shows how many sFBs are installed 5-34 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 87: Figure 5-25. Spsu Information

    Fan speed can be either turbo or normal. Refer to the ISR 9288/9096 Fan Speed Algorithm Section, on page 5-37 for further detail. 5-35 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 88: Figure 5-26. Isr 9288 Fan Information

    Should a specific fan unit be faulty, the fan unit is displayed in red in the dialog box and the Status bar at the top of the window is also displayed in red. When this is the case, consult the ISR 9XXX Installation Manual for maintenance instructions. 5-36 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 89: Figure 5-28. Faulty Fan Information

    Fan in the unit is failing, all the other Fans are set to Turbo. Each module has three temperature thresholds: RELAX threshold WARNING threshold ALARM threshold After a successful scan of the sensors of all chassis modules, the fan speed is set to 5-37 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 90 167ºF 55ºC 60ºC 65ºC 131ºF 140ºF 149ºF 69ºC 100ºC 156ºF 212ºF 45ºC 50ºC 113ºF 122ºF The table of temperatures for each module can be viewed via the VDM. See Temperature Information, Section 5.5.4. 5-38 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 91: Figure 5-29. Router Module Information

    When the GUI is closed, the VDM is automatically updated. When the GUI is open, the user gets a warning message. The user must then exit the GUI and reenter it in order for the change to take effect. 5-39 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 92: Using The Voltaire Device Manager For Switch Management

    Voltaire Device Manager (VDM) Using the Voltaire Device Manager for Switch Management Using the Voltaire Device Manager for Switch Management The Voltaire Device Manager allows you to configure the following device settings: 675HFTP Configuration IP Settings Remote Management Setting User Settings FTP Configuration 5.7.1...

  • Page 93: Figure 5-31. Remote Manager Settings - Snmp Tab

    Voltaire Device Manager (VDM) Using the Voltaire Device Manager for Switch Management 5.7.2 Remote Management Setting The Remote Management Settings dialog box enables you to set SNMP parameters to interface with remote management systems, and to provide general details of the remote management system.

  • Page 94: Figure 5-32. Remote Manager Settings - General Tab

    Voltaire Device Manager (VDM) Using the Voltaire Device Manager for Switch Management Figure 5-32. Remote Manager Settings - General tab Fill in the following fields: Step 5 Device Name: Describes the device name in a read-only field. SNMP Version: Displays the software version of the SNMP in a read-only field.

  • Page 95: Figure 5-33. Configuring User Authorization

    Using the Voltaire Device Manager for Switch Management 5.7.3 User Settings The Voltaire Device Manager allows you to configure three types of user authorization: The Administrator level enables you to change user password configuration, to export or import configurations and more. The default password is 123456 The Manager level enables you to set the parameters.

  • Page 96: Figure 5-34. Configuring Ftp

    Voltaire Device Manager (VDM) Using the Voltaire Device Manager for Switch Management 5.7.4 FTP Configuration The FTP configured for use with the switch is used for the following: Upgrading software versions Uploading log files to the remote server Uploading/downloading the repository file.

  • Page 97: Reset Operations

    Click OK or Apply to save the new entry and close the window. Step 5 Reset Operations The Voltaire Device Manager has several reset options. Resets the entire switch (hardware reset, in ISR 9288/9096 switches only) Resets a Line or Fabric Board (in ISR 9288/9096 switches only) Reset the CPU (software reset), available in all switches.
  • Page 98 CAUTION This is a traffic affecting operation. To reset the Management Board (software reboot): This operation will typically be performed when faulty operation of the Voltaire Device Manager software is suspected. This operation does not affect the InfiniBand traffic. From the main window toolbar, click From the main menu, select Action—Reset.

  • Page 99: Chapter 6. Voltaire Fabric Manager (Vfm)

    Voltaire Fabric Manager (VFM) In This Chapter This chapter provides system information and requirements for working with the Voltaire Fabric Manager (VFM) application. It includes information on the following topics: Fabric Manager Description, on page 6-2 Fabric Manager Installation, on page 6-3...

  • Page 100: Fabric Manager Description

    Voltaire Fabric Manager (VFM) Fabric Manager Description Fabric Manager Description The Voltaire Fabric Manager (VFM) is an embedded application with a web-based GUI that provides InfiniBand fabric management functionality. The Voltaire Fabric Manager provides a color-coded topology map of the InfiniBand fabric which allows users to view the status of the managed InfiniBand fabric.

  • Page 101: Fabric Manager Installation

    Voltaire Fabric Manager (VFM) Fabric Manager Installation 6.1.2 Voltaire Fabric Manager Main Functionality Voltaire Fabric Manager provides the following functionality: Alarms Color-coded display of alarmed objects in all relevant Fabric Manager display windows Current Alarms reports and ability to manually clear alarms...
  • Page 102 Installing the Sun Java Web Start Application Install the Sun Java Web Start application (Version 1.0.1) on the Workstation; Install it via the Internet using the link in the GUI, or via the Voltaire Installation 6.2.3 Installing the Sun Java Web Start Application on...

  • Page 103: Getting Started With The Fabric Manager

    Open the Internet Browser and connect to the fast Ethernet IP address (for Step 2 example: http://192.168.1.2). The Voltaire Fabric Manage Initiate Window is displayed. This window varies according to the device type, as shown in the example in Figure 6-1.

  • Page 104: Figure 6-2. Login Window

    Voltaire Vision GUI. NOTE When opening the Voltaire Fabric Manager Initiate Window for the first time, click to start the Sun Java Web Start installation. The computer that is being used must be connected to the Internet for successful Java Web Start package download from Sun’s website.

  • Page 105: Figure 6-3. Exit Warning Message

    To login into the Voltaire Fabric Manager: Select the User Type from the drop-down menu. Step 1 The Voltaire Fabric Manager supports three User Types, each with a different authorization level. Administrator user type. This user can perform all configuration and provisioning operations.

  • Page 106: The Fabric Manager Main Window

    Figure 6-4 shows the main areas in the Main Window: Status Main Toolbar Menu Subnet Tree Node/ Host/ Link Information Topology Map Node/ Host/ Properties Topology Map Trap Table Figure 6-4. Fabric Manager Main Window P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 107: Figure 6-5. Example Of A System With 1104 Nodes

    You can also right click the alarm in the Trap Table and select Go To Node from the context sensitive menu. The map focuses on the selected alarmed node/host (small rectangles drawn around the node on the topology map) and the focus of the P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 108: Figure 6-6. Topology Map

    Subnet Tree and the Node Information. Figure 6-6. Topology Map The Topology Map displays the following types of icons, depending on the configuration of the fabric: Icons are color coded to indicate their alarm status. 6-10 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 109 This icon can be displayed on a device icon; when displayed, it indicates that the device is a Voltaire device; this can either be a device manufactured by Voltaire (such as an ISR 9288 switch for example), or a device with the Vendor ID Voltaire set in the device configuration.
  • Page 110 Searching the Topology Map 2. Type in Search string 1. Select Search criteria 3. Click Search You can perform a search for a specific node in the Topology Map by using the toolbar search box. 6-12 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 111 Node Info and Node Properties areas focus to display the information of selected node. 6.4.4 Main Menu The main menu provides access to Voltaire Fabric Manager. In general, when a menu heading is not available (disabled), it appears grayed out. Table 6-4 details the Main Menu options. Table 6-4. Main Menu Options...
  • Page 112 FTP. Note that the import will not be performed if errors were found in the file while it was checked. The Grouping info Log shows the group.csv file after it was created (exported). 6-14 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 113 Node Open–opens the Node information view of the node selected in the topology map. Launch Device Manager–launches the Voltaire Device Manager application with a connection to the switch selected in the topology map. See Chapter 5 for descriptions of the relevant Voltaire Device Manager applications.

  • Page 114: Figure 6-7. Right-Click A Node - Topology Map

    Attached to–Displays the nodes which the port is connected to, at both ends of the link. Help About–Displays the Voltaire Fabric Manager software version. 6.4.5 Right Click/Context Sensitive Menu You can use the right click menu in the Topology map, in the event log and in other Fabric Manager windows and dialog boxes.

  • Page 115: Figure 6-9. Right-Click A Port - Information Area (Right Panel)

    The Fabric Manager Main Window Figure 6-9. Right-Click a Port – Information Area (Right Panel) Figure 6-10. Right-Click a Port – Table View Figure 6-11. Right-Click an Event Figure 6-12. Right-Click the Fabric Subnet 6-17 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 116: Figure 6-13. Right-Click Racks

    The Subnet Tree displays all the elements monitored by the Fabric Manager in a top down, hierarchical format. Figure 6-14. Subnet Tree The IP address displayed at the bottom of the subnet tree window is the fast Ethernet IP address of the switch. 6-18 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 117 I/O modules) inserted in the sRBD slots, up to three modules per sRBD. I/O Modules can be either the Voltaire Fibre Channel Router (FCRs) or IP Router (IPR). The user also has the ability to see in which slot they are installed.
  • Page 118 Table 6-6. Fabric Manager Toolbar Icons - General General Action Buttons Refresh–Refreshes the Voltaire Fabric Manager main window with the most updated information. Events Log–Displays the online Events Log (Events Table), described in Section 6.7.2. The Events Log provides information on the events and alarms in the system.
  • Page 119 Hierarchical Layout – Rearranges the Topology Map to the default arrangement. Nodes Information– Displays the Node Information window. Main Fabric – When clicked from within the Rack Map, brings you back to the main fabric map. 6-21 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 120: Figure 6-15. Topology Table View

    Normal, Minor, Warning or Critical. GUID The GUID of the node/host. Name The user-assigned name of the node/host. IP Address The IP address of the node/host. Description A description of the type of node/host. 6-22 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 121: Figure 6-16. Trap Table

    The Trap Table is available in both, the VFM and the CLI. Refer to Appendix C for a list of the reported traps and events. Figure 6-16. Trap Table 6-23 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 122 Appendix C for a list of the events reported in the system. Rearranging the Trap Table You can rearrange columns positions within the Trap Table by dragging and dropping a selected column in the new desired position. 6-24 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 123: Figure 6-17. Events Log Right-Click Menu

    Right-Click the event listing in the Trap Table. The following menu is displayed. Figure 6-17. Events Log Right-Click Menu Viewing Event Information Select View Event. The Event Data window is displayed. Figure 6-18. Events Data 6-25 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 124 Additional information, if applicable, regarding the location of the port. Port GUID Displays the Global Unique Identification number (GUID) of the port. IB Port Displays the IB port number. Displays the port LID. 6-26 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 125: Figure 6-19. Go To Node

    By applying event filters, you can limit the types of events that are reported in the Fabric Manager Trap Table. Refer to the Fabric Inspect Feature, Section, 6.7.2. for more information on how to set up Event Filters. 6-27 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 126: Viewing Properties Data

    Displays whether or not a Subnet Manager is installed in the node. Node Device Displays the identification number of the device. Node Vendor Displays the name of the vendor that manufactured the device. Node IP Displays the IP address of the node. 6-28 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 127: Figure 6-21. Port Properties

    Displays the location of the node in the rack (assigned by the user according to needs). 6.5.2 Port Properties The following information is displayed when selecting a port in the Node/Host Information panel. Figure 6-21. Port Properties 6-29 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 128 Active Link Speed Displays the current signaling rate of the link: current default is 2.5 Gbps. Supported Link Displays the maximum signaling rate the link can support: current Speed default is 2.5 Gbps. 6-30 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 129: Information Windows

    Displays the connection width. Information Windows The Voltaire Fabric Manager allows you to get an overall view of the InfiniBand fabric, and to drill down to specific information views all the way down to the individual port. At each view different information is displayed.

  • Page 130: Figure 6-24. Racks Components

    Number of Nodes Displays the number of nodes contained in the Rack. Click a Rack in the subnet tree or in the List of Racks, as shown in Figure 6-24. Figure 6-24. Racks Components 6-32 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 131: Figure 6-25. Hosts List

    Displays the Global Unique Identification number (GUID) of the device. Name Displays the user-assigned device name. The device name is either set by the ATS mechanism, or can be set manually via the CLI or the Voltaire Device Manager. Local ID of the Host/Adapter node.

  • Page 132: Figure 6-26. Hosts Ports

    Displays the current signaling rate of the link: current default is 2.5 Gbps. Displays the Maximum Transmission Unit value of the port. Displays the VLs supported by the port. Current support is for VLs 0 to7. 6-34 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 133: Figure 6-27. List Of Switches

    Displays the Global Unique Identification number (GUID) of the switch. Name Displays the user-assigned switch name. The switch name is either set by the ATS mechanism, or can be set manually via the CLI or the Voltaire Switch Manager. Displays the Local ID of the switch.

  • Page 134: Figure 6-28. Switch/Slb Information

    Displays the active width of the link: 1X, 4X, or 12X. Speed Displays the signaling rate of the link: 2.5 or 5 Gbps(for future use). Displays the Maximum Transmission Unit for the link. Displays the number of Virtual Lanes enabled on the link. 6-36 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 135: Figure 6-29. Port Information

    Displays the state of the port, which can be one of the following: ACT (port is active), DWN (down). This field is color coded to indicate the state of the port. Defines the LID Mask Control value of the port. Possible values are 0 to 7. 6-37 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 136 Displays the Maximum Transmission Unit value of the port. Displays the VLs enabled on the port. Has SM Indicates if a Subnet Manager is present in the node. MKey Displays the Management Key Value of the port. 6-38 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 137: Figure 6-30. Link Information

    Section 6.4.1, of the ports, the switch that the port belongs to and (if applicable) the board which the port belongs to. Color coding is defined according to severity. For more information, refer to The Event Notification Mechanism, Section 14.1.1. 6-39 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 138: Figure 6-31. Grouping Information

    To display grouping information: Select the switch in the main window topology view and selecting Node/Grouping Info from the Main Menu. The following is an example of the window displayed. Figure 6-31. Grouping Information 6-40 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 139: Configuring The Fabric

    Configuring the Fabric Configuring the Fabric 6.7.1 Subnet Manager (SM) Properties Window This section describes viewing and configuring SM parameters via the Voltaire Fabric Manager. To view Subnet Manager general parameters: Select Fabric/SM Properties from the main menu. The following information is displayed: Figure 6-32.
  • Page 140 Read-only field that displays the SM Send window, that is, the maximum PIPELINE number of outstanding MADs on the wire. 16 is the default value for switch platforms ISR 9288/9096, and ISR 9024. Polling Defines the polling retries interval for a standby SM to poll the master SM Retries for its activity.
  • Page 141 Subnet Manager (SM) Mode, Appendix A.4.13 for more information. 6.7.2 Fabric Inspect Features Fabric Inspect is used for diagnostics purposes. The Fabric Inspect feature is fully detailed and illustrated in Chapter 14, on Fabric Diagnostic. 6-43 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 142: Figure 6-33. Event Filters Settings

    Click Reset All to deselect all events. Click Select All to select all events. Click Apply to save any changes made in the dialog box. Click to return to the previously saved settings. 6-44 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 143: Figure 6-34. Pm Settings

    You can also disable selected PM counters by clearing the checkbox displayed next to the counter. In the Sample Interval field, set the frequency at which the PM data is collected Step 2 from the counters; range: 60 seconds to 65535 seconds. 6-45 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 144: Figure 6-35. Event Log - Example

    To show the Event Log: Select Fabric/Fabric Inspect/Event Log from the VFM main menu. The following window is displayed. Figure 6-35. Event Log - Example Refer to Section 14.3.3 for further information on Event Log. 6-46 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 145: Figure 6-36. Error Log - Example

    Select Fabric/Fabric Inspect/Error Log from the VFM main menu. The following window is displayed. Figure 6-36. Error Log – Example Refer to Section 14.3.3 for further information on Fabric Diagnostic using the Event/Error Log. 6-47 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 146: Figure 6-37. Bad Ports Log

    This information enables faster location of the faulty link and faster stabilization of the IB fabric. 6-48 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 147: Figure 6-38. Portcounters.csv File - Example

    On the topology map, right-click the host and select Monitor. The browser Step 4 window opens with the Ganglia link directed to the specific host, displaying a detailed host report, as shown in the example below. 6-49 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 148: Figure 6-40. Ganglia Cluster Monitor

    Voltaire Fabric Manager (VFM) Configuring the Fabric Example: http://172.25.2.175/ganglia/?m=load_one&r=hour&s=descending&c= Support+Cluster&h=$HOST&sh=1&hc=4 In the example below, the Host is called hydra3.voltaire.com Figure 6-40. Ganglia Cluster Monitor Refer to Chapter 13 for further detail on Cluster Monitor Configuration. 6-50 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 149: Figure 6-41. Qos Sl To Vl Mapping Table

    Select the new value and click Apply to save the setting. Repeat as required. Step 5 Click Activate to send the new settings to the switch. Step 6 6-51 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 150: Figure 6-42. Qos Vl High Arbitration Table

    Click Apply to save the configuration in the Fabric manager. Step 5 Click Default to display the default settings in the dialog box. Step 6 Refer to Chapter 9 for further information on Configuring Quality of Service (QoS). 6-52 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 151: Figure 6-43. Routing

    Voltaire Fabric Manager (VFM) Configuring the Fabric 6.7.5 Routing Configuration To perform a routing configuration: Select Fabric/Routing Configuration from the main menu. The following dialog Step 1 box is displayed. Figure 6-43. Routing 6-53 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 152: Figure 6-44. Routing Algorithm

    If you selected either, the Rearrangeable or Up-down algorithm, then select a value from Step 3 the Topology drop-down menu. This value specifies the depth of the CLOS network (e.g., 3-stage CLOS, 5-stage CLOS, etc…) 6-54 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 153: Figure 6-45. Add To Spine List

    Balanced routing configuration is automatically saved, ignoring your configuration. The Status shows which configuration was applied. Click Activate Routing. Step 6 Click OK. Step 7 Refer to Chapter 10 for further information on Configuring Routing Algorithms. 6-55 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 154: Figure 6-46. Current Alarms

    You can filter the Current Alarms to display all alarm states or only specific Alarm States. By default the table view shows all nodes regardless of alarm state (The All box is checked). Check one or more boxes. 6-56 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 155: Figure 6-47. Alarm Data

    To find out more specific details pertaining to an event, right click the event listing in the Current Alarms Window and select Alarm Data. The following of window is displayed. Figure 6-47. Alarm Data 6-57 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 156: Clearing Alarms

    Current Alarms window and select Clear All. NOTE After clearing an alarm in the Fabric Manager, if the actual alarm still persists physically, the alarm will reappear in the Current Alarms window. 6-58 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 157: Figure 6-48. Config Ftp Dialog Box

    Downloads group.csv from the remote FTP server, using the previously defined server name, user name, and password Store grouping Uploads group.cvs File to the remote FTP server, using the previously defined server name, user name, and password 6-59 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 158 The Grouping info Log shows the group.csv file after it was created (exported). The Grouping error Log shows the error found in the group.csv file. Refer to Chapter 8 for detailed information on Rack Grouping. 6-60 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 159: Configuring Nodes

    From the Main Menu, select Node/Open. The contents of the specified node appears. From the Graphics display, select the desired node, right-click and select Open. From the Subnet menu click the desired node. The following window is displayed. Figure 6-49. Viewing Nodes 6-61 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 160: Figure 6-50. Nodes Grouping Info Window

    From the Main Menu, select Node/Grouping Info. The grouping details of the specified switch appears. From the Graphics display, select the desired node (ISR 9096/9288 switches only), right-click and select Grouping Info. The following window is displayed. Figure 6-50. Nodes Grouping Info Window 6-62 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 161: Figure 6-51. Group Properties

    Nodes Menu - Cluster Monitor The browser window opens with the Ganglia link directed to the specific IP, displaying a detailed host report. Example: http://172.25.2.175/ganglia/?m=load_one&r=hour&s=descending&c= Support+Cluster&h=$HOST&sh=1&hc=4. Refer to Chapter 13 for further detail on Cluster Monitor. 6-63 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 162: Figure 6-53. Current Alarms Of A Single Node

    This window shows the Current Alarms of the specified single node and not from the whole fabric. Figure 6-53. Current Alarms of a Single Node 6.8.7 Node - Clear Alarms The Node/Clear Alarms feature clears all the alarms within the specified node. 6-64 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 163: Chapter 7. Names Settings

    CLI and the GUI. It includes information on the following topics: CLI Switch Name Settings, on page 7-2 GUI Node Name Settings, on page 7-3 Naming an Unmanaged ISR 9024, on page 7-4 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 164: Cli Switch Name Settings

    CLI Switch Name Settings From the CLI, you can only rename the Switch. Each managed switch has a node name. This name is advertised to the Voltaire Fabric Manager and introduced in the fabric topology map. Former versions have assigned the switch platform name as the default node name (e.g., ISR9288).

  • Page 165: Gui Node Name Settings

    The change here will only be applied when looking from the Subnet Manager under which the naming was performed. However, when moving the node to another Subnet Manager, the Node will be automatically be set back to its default Name. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 166: Naming An Unmanaged Isr 9024

    Naming an Unmanaged ISR 9024 Naming an Unmanaged ISR 9024 You can rename an unmanaged ISR 9024 from the GUI the exact same way it is performed for a regular switch. See Section 7.2 for further detail on how to name a switch or node via the GUI.

  • Page 167: Chapter 8. Rack Grouping

    Rack Grouping Settings via the VFM, on page 8-3 Elements of the Group.csv File, on page 8-10 Backing up and Editing Groups Settings via the CLI, on page 8-12 Rack Grouping Installation Suggestion, on page 8-14 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 168: Rack Grouping Overview

    Add/remove physical elements from/to rack-groups. Add/edit elements and group information. The CLI enables to: Import & export the group.csv file for backup and update purposes. The figure below provides an example of Rack groups in the VFM. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 169: Rack Grouping Settings Via The Vfm

    Several elements can be selected by pressing SHIFT+Click. Note: In the Table view, you can only select a single element at the time. Right-click and choose the Properties option. Step 2 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 170: Figure 8-2. Rack Grouping - Topology Map

    Rack Grouping Rack Grouping Settings via the VFM Figure 8-2. Rack Grouping – Topology Map Figure 8-3. Rack Grouping – Table View P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 171: Figure 8-4. Rack Grouping - Switches/Host Table

    Figure 8-4. Rack Grouping – Switches/Host Table The Node Properties dialog box is displayed. Figure 8-5. Node Properties Dialog Box In the Properties dialog box, click New; the New Rack dialog box appears. Step 3 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 172: Figure 8-6. New Rack Dialog Box

    (Optional) In the Location field, set the Rack position, to define the physical Step 5 position of the rack according the user’s needs. Click OK; the Properties (Node/Rack) dialog box drop-down menu displays the Step 6 new Rack that you just created. Figure 8-7. New Rack P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 173: Figure 8-8. Group Properties Dialog Box

    Under the Name field, type in the new name for the specified rack. Step 3 (Optional) In the Location field, you can change the Rack’s position. Step 4 Click OK to confirm and save your settings; the rack will be renamed. Step 5 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 174: Figure 8-9. Rack Group Contents

    Double-click a Rack-group in the Topology Map to open a new window showing only the elements in the specific rack, as shown in the example below. Shows all rack-groups in the fabric and their contents Shows Rack details Shows Rack Contents Figure 8-9. Rack Group Contents P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 175: Figure 8-10. Rack-Groups - Table View

    To display absent elements (disconnected element): In the table view of the fabric show the element’s information by checking the “Absent Filter” only. The absent elements will be displayed. Note: The absent elements are taken from the group.csv file. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 176: Elements Of The Group.csv File

    1 – does not show as part of the rack. Rack_ID Unique key for the rack-group [int]. `0` means: not in a group. Group_item_Location: Index indicating the location of the element within the 8-10 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 177 HCA elements and one SWITCH element. DATE: Sun Jul 3 09:40:30 2005 # VER1.0 # Type,Id/guid,name,Don't show in group,Rack Id,Location in rack,U RACK,1,DemoRack,0,0,1,0 HCA,8f10403965218,Nestor0,0,1,0,U HCA,8f10403962b84,Nestor1,0,1,0,U HCA,8f10403962af8,Nestor9,0,1,0,U SWITCH,8f104004105ab,0008f104004105ab,0,1,0,U HCA,8f10403965214,Nestor13,0,1,0,U HCA,8f10403962c5c,Nestor4,0,1,0,U HCA,8f104039651a8,Nestor11,0,1,0,U HCA,8f10403962b88,Nestor5,0,1,0,U HCA,8f10403965210,Nestor7,0,1,0,U HCA,8f10403965204,Nestor14,0,1,0,U HCA,8f10403965200,Nestor10,0,1,0,U HCA,8f104039651f4,Nestor8,0,1,0,U HCA,8f104039651e4,Nestor15,0,1,0,U HCA,8f104039651b4,Nestor12,0,1,0,U 8-11 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 178: Backing Up And Editing Groups Settings Via The Cli

    Upload group.cvs File to remote FTP server,using previously defined server name, user name and password. importFile group Download group.csv from the remote FTP server, using previously defined server name, user name and password. 8-12 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 179 FTP server to the local library. Use the group import command to update the current configuration with the rack Step 4 grouping defined in the group.csv file imported from the FTP. 8-13 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 180: Rack Grouping Installation Suggestion

    Section 8.4 for more information.) Optionally edit/update the group.csv file; add physical-position and elements. Step 4 Import the group.csv file back to the system; the system will be updated. (See Step 5 Section 8.4 for more information.) 8-14 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 181: Chapter 9. Configuring Quality Of Service (Qos)

    This chapter provides information about how to configure the Quality of Service feature. It includes information on the following topics: QoS Overview, on page 9-2 SL to VL Mapping, on page 9-2 QoS Arbitration, on page 9-4 Example of SM Parameter Configuration, on page 9-7 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 182: Qos Overview

    Number of Data VL Numbering VLs Supported VL0, VL1 VL0 - VL3 VL0 - VL7 VL0 - VL14 There are two types of VLs, one for normal traffic called a data VL and one P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 183: Figure 9-1. Qos Sl To Vl Mapping Table

    Step 4 and click the mouse button. Select the new value and click Apply to save the setting. Repeat as required. Step 5 Click Activate to send the new settings to the switch. Step 6 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 184: Qos Arbitration

    Step 1 Fabric/Quality of Service from the main menu. Click the VL Arbit High tab, or the VL Arbit Low tab (the procedure is identical Step 2 for both). The following dialog box is displayed. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 185: Figure 9-2. Qos Arbitration Table

    LRH is used to determine the number of units in the packet. (Note, the VCRC and also the symbols between packets introduced by the physical layer should not be included in VL arbitration weight calculations.) The calculation is maintained to 4 byte increments. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 186 1 2 1 Save the Quality of Service by using qos-apply-configuration set. Changes made Step 6 to the QoS configuration will be saved. Type qos-info show to view the QoS parameter information. Step 7 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 187: Example Of Sm Parameter Configuration

    (config-sm)# sm-info mopvl set [vl0,vl0-1,vl0-3,vl0-7,vl0-14] (config-sm)# mc (config-sm-mc)# mc auto-join set [disable, enable] SM Auto Join mode (Mini SM-like operation). This mode is disabled by default When enabled, the Subnet Manager will automatically add each active end port P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 188 Configuring Quality of Service (QoS) Example of SM Parameter Configuration (HCA/SW) to the default IPoIB multicast group (MGID 0xFF12401BFEFF000000000000FFFFFFFF). Refer to the CLI Subnet Manager (SM) Mode, Section A.4.13 for further detail. P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 189: Chapter 10. Configuring Routing Algorithms

    Balanced Routing Scheme, on page 10-2 Rearrangeable Routing (Multi-Path with sFB Consistency) , on page 10-2 Up/Down Routing Scheme, on page 10-3 Configuring Routing via the VFM, on page 10-4 Configuring Routing via the CLI, on page 10-7 10-1 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 190: 10.1 Supported Algorithms

    Configuring Routing Algorithms Supported Algorithms 10.1 Supported Algorithms The Voltaire Fabric Manager supports the following routing algorithms: Balanced-routing Rearrangeable routing (multi-path with sFB consistency) Up-down routing 10.2 Balanced Routing Scheme This is the default routing scheme and is applicable to all Fabric configurations.

  • Page 191: 10.4 Up/Down Routing Scheme

    For example, in a 3-stage CLOS network, if both end-ports connect to the same switching chip on the first level of the CLOS, then the route should not traverse the CLOS back-bone level. 10-3 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 192: 10.5 Configuring Routing Via The Vfm

    Configuring Routing via the VFM 10.5 Configuring Routing via the VFM To configure routing via the Fabric Manager: Select Fabric/Routing Configuration from the main menu. The following dialog Step 1 box is displayed. Figure 10-1. Routing 10-4 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 193: Figure 10-2. Routing Algorithm

    If you selected either, the Rearrangeable or Up-down algorithm, then select a value from Step 3 the Topology drop-down menu. This value specifies the depth of the CLOS network (e.g., 3-stage CLOS, 5-stage CLOS, etc…) 10-5 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 194: Figure 10-3. Add To Spine List

    Note: This step is very important. If you do not click Apply (or if you make a mistake while configuring), the Balanced routing configuration is automatically saved, ignoring your configuration. The Status shows which configuration was applied. Click Activate Routing. Step 6 Click OK Step 7 10-6 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 195: 10.6 Configuring Routing Via The Cli

    Set to Algorithm by using the sm info algorithm set command, as follows:: Step 1 sm-info algorithm set Command: Description: Sets the routing algorithm to be implemented by the Voltaire Subnet Manager. The Voltaire Subnet Manager supports the following routing algorithms: multipath with balanced-routing and multi-path with spine consistency. Syntax:...
  • Page 196 VFM automatically adds all GUIDs of the internal backbone switch chips to the spine list, considerably simplifying your task. Refer to The CLI Subnet Manager (SM) Mode, Appendix A.4.13 for a list of Routing Configuration commands. 10-8 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 197: Chapter 11. Configuring Partitioning

    This chapter provides information about how to perform Partitioning. It includes information on the following topics: Partitioning Overview, on page 11-2 Mechanisms Used to Create and Enforce Partitions, on page 11-3 Setting Partitioning, on page 11-3 11-1 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 198: 11.1 Partitioning Overview

    The pkey table can be altered via the CLI. On the first Join operation that uses a specific pkey (other than the default), partitioning automatically creates the IPoIB multicast group that corresponds to this pkey. 11-2 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 199: Mechanisms Used To Create And Enforce Partitions

    Table 11-1. Pkey Table in Repository Port-GUID Pkey0 Pkey1 Pkey2 … Pkey14 Pkey15 0 (default) 0xFFFF GUID 1 … GUID N The SM either configures the HCA with the Pkey table as configured in the 11-3 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 200 [row > 1] [guid] pkey set pkey set [pkey-number] [row] [range > 0] ['0x'hex] pkey update pkey update 11-4 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 201 0000000000000000 ffff 0 0x11111111 Once created use the command pkey set to define the Pkey value. Step 6 For example: ISR9XXX(config-sm)# pkey set pkey0 2 0xf1f1 Change will be effective only after 'pkey update' command 11-5 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 202 ISR9XXX(config-sm)# pkey low show entry Port-GUID Pkey0 Pkey1 Pkey2 Pkey3 Pkey4 Pkey5 Pkey6 Pkey7 |---|----------------|----|-----|-----|-----|-----|----|-----|----- 0000000000000000 ffff Run the command to update the Pkey Table. pkey update Step 3 11-6 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 203 |---|----------------|----|----|-----|-----|-----|-----|-----|------ 0000000000000000 ffff 0 Run the command to update the Pkey Table. pkey update Step 3 Refer to The CLI Subnet Manager (SM) Mode, Appendix A.4.13 for a list of Partitioning (Pkey) commands. 11-7 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 204: Chapter 12. Replacing Modules

    It includes information on the following topics: Replacing sFBs (Spines), on page 12-2 Replacing sLBs, on page 12-6 Replacing sMBs, on page 12-7 Replacing an Unmanaged ISR 9024, on page 12-8 12-1 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 205: Replacing Sfbs (Spines)

    All modules are shipped with firmware on-board. When replacing modules, make sure that the firmware version on the substitute module received from Voltaire matches the other similar modules in your system. If not, a firmware upgrade is required. 12.1 Replacing sFBs (Spines)
  • Page 206 You can also perform a board hardware reset from the VDM by right-clicking the specific board on the picture and select Reset Module Name from the drop-down menu, as shown in the Figure below. 12-3 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 207: Figure 12-1. Resetting A Specific Board Via The Vdm

    Routing configuration must thus be reestablished for the sFB being replaced. This command reconfigures and rebalances all the routing of the traffic the fabric in a non-intrusive way. 12-4 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 208 When replacing an sFB, make sure that the algorithm is set back to the exact same mode the system was in before sFB replacement. 12-5 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 209: 12.2 Replacing Slbs

    It is possible to burn a specific module (but this cannot be performed via the CLI). Call your local representative for further support. If you upgraded the firmware, perform a hardware reset as described in Section Step 3 12.1. 12-6 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 210: 12.3 Replacing Smbs

    <ftproot-path> Example: export pub/sw-config-19-june/ This can also be performed via the VDM (Device\Config FTP and click the Store Configuration button). Once the sMB has been replaced, you can import the back-up configuration. 12-7 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 211: 12.4 Replacing An Unmanaged Isr 9024

    If there are two sMBs, the data is replicated on the other sMB. 12.4 Replacing an Unmanaged ISR 9024 When the ISR 9024 does not have a CPU or Software on-board, it is referred to as unmanaged. The ISR 9024 firmware version of the newly inserted ISR 9024 must be identical to the firmware of all the other ISR 9024 boards installed in the system.
  • Page 212 ISR9288-0b5b(utilities) In this example, the firmware version is fw_version = 00.08.00 NOTE If the firmware version of the ISR 9024 is not identical to the other ISR 9024 installed in the system, contact your local representative for firmware upgrade. 12-9...

  • Page 213: Chapter 13. Configuring Cluster Monitor

    Monitor About this Chapter This chapter provides instructions on how to configure the Cluster Monitor. It includes information on the following topics: What is Ganglia?, on page 13-2 Cluster Configuration, on page 13-2 13-1 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 214: 13.1 What Is Ganglia

    On the topology map, right-click the host and select Monitor. The browser Step 4 window opens with the Ganglia link directed to the specific host, displaying a detailed host report, as shown in the example below. 13-2 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 215: Figure 13-2. Ganglia Cluster Monitor

    Configuring Cluster Monitor Cluster Configuration Example: http://172.25.2.175/ganglia/?m=load_one&r=hour&s=descending&c= Support+Cluster&h=$HOST&sh=1&hc=4 In the example below, the Host is called hydra3.voltaire.com Figure 13-2. Ganglia Cluster Monitor 13-3 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 216: Chapter 14. Fabric Diagnostic

    Appendix B - lists the available Port Counters, describes the PortCounter Fields and Summary report. Appendix C – Defines the trap structure and lists the Supported Traps and Events in the Alarm Table. 14-1 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 217: 14.1 Fabric Diagnostic Overview

    Fabric Diagnostic Fabric Diagnostic Overview 14.1 Fabric Diagnostic Overview Voltaire’s Fabric Manager enables fabric connectivity debugging using the built-in Performance Manager (PM). PM has two major capabilities: Port Counters Monitoring and Report. The PM generates a periodic port counters report file (in CSV format) that can be loaded to Excel and further analyzed by the user.

  • Page 218: Figure 14-1. Diagnostic Block Diagram

    In addition, events may be generated in the fabric and sent to the SM by fabric elements. The SM reports those events as well. The event mechanism can do the following actions with each event: Log the event in the event log 14-3 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 219 Applied routing scheme Port State Change The user may filter the events using the GUI and CLI. The filtering policy determines for each event whether it is logged and whether a trap is generated. 14-4 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 220 14.1.4 Procedures and Tools used for Problem Identification Tools available to perform diagnostic: Use the Topology Map to see current problems The Error Log The Bad Ports Log The Current Alarms Table The Fabric Statistics portcounters.csv file 14-5 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 221 9. Verify that the problem is fixed 10. Repeat the above until all fabric nodes have No bad link error (there might be error counters that are > 0 and not problem related) 14-6 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 222: 14.2 Fabric Inspect Settings

    1. dump of the fabric - running netdiscover and capturing the output NOTE Set the SM verbosity level to "verbose" before re-generating the problem and exporting the logs. Sometimes the "debug" level is needed. 2. CATC dump, if available. 14-7 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 223 Step 5 Wait for three minutes to allow error counters enable file to be updated. Refer to Appendix A on page A-1 for the complete list of the CLI PM commands. 14-8 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 224: Figure 14-2. Pm Settings

    Select Fabric/Fabric Inspect/Settings from the VFM main menu. The following Step 1 dialog box is displayed. Figure 14-2. PM Settings By default, PM counters are monitored for all devices in the fabric (Scope field is set to All). 14-9 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 225: 14.3 Diagnostic Tools

    14.3 Diagnostic Tools 14.3.1 Displaying and Checking Port Counters using the GUI Both Voltaire's Fabric and Device Managers allow you to view statistics for active ports and detect faulty ports; the statistics are collected according to a user-defined time interval.

  • Page 226: Figure 14-3. Port Counters Information

    Step 4 Check the port’s health status Step 5 Step 6 If required, click to refresh the dialog box. NOTE The counters displayed in the dialog box vary according to the selected port. 14-11 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 227: Figure 14-4. Port Counters Graph

    At the bottom of the dialog box, select the measurement units to display the Step 3 counters in (MBps or Gbps). You can also display the counters for TX or RX separately. Click Start. To stop counting, click Stop. Step 4 14-12 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 228: Log Files

    Inspect Feature, Section 6.7.2. Bad Ports Log - Displays a log of the Bad Ports in the switch. Select Device—Bad Ports Log from the Voltaire Device Manager main menu. A browser window is displayed, showing the bad ports. For further information, refer to Identifying a Bad Port, Section 14.4.3.
  • Page 229 Show topology log. umServer-log show Show log file vsm-internal-log show Show Voltaire Subnet Manager (VSM) internal log. vsm-log show Show Voltaire Subnet Manager (VSM) log. Logs can be exported to the remote FTP server using the following CLI commands:...
  • Page 230 Refer to Appendix A.3.3 for a list of CLI logs mode and commands. NOTE The Event Log only saves information since the last device reboot. 14-15 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 231 PM_ SymbolError PM_ LinkErrorRecovery PM_ LinkDownedCounter PM_ PortRcvErrors PM_ PortRcvRemotePhysicalErrors PM_ PortRcvSwitchRelayErrors PM_ PortXmitDiscards PM_ PortXmitConstraintErrors PM_ PortRcvConstraintErrors PM_ LocalLinkIntegrityErrors PM_ ExcessiveBufferOverrunErrors PM_ VL15Dropped Alarm: Fabric Config Error. This alarm unifies the following events: 14-16 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 232: Viewing Alarm Information

    (All box is checked). Check one or more boxes. By applying event filters, you can limit the types of events that are reported in the Fabric Manager Event Log. 4. Identify the Problem and Troubleshoot as Required Refer to Appendix B.3 for a list of errors. 14-17 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 233 The RcvBytes and XmitBytes port counters are maintained as 64 bit counters. True 64 bit value are available for ports of switch chips (Infiniscale-III) only • Calculates the average throughput per port over the last interval: XmitBytes Rate (MB/Sec) RcvBytes Rate (MB/Sec) XmitPkts/Sec RcvPkts/Sec 14-18 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 234 Open the CSV format file with a suitable application (e.g., MS Excel). Step 3 Refer to Appendix B.1 for a complete description of ports counters fields, a description of the CSV Summary Report and the port counters currently available in the fabric. 14-19 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 235: 14.4 Additional Tools And Examples

    In the VFM, right-click the trap and select View Event. The following Event Data Step 1 Information is displayed, showing which ports are connected on both ends with the 1X link: Figure 14-6. VFM Event Data Window 14-20 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 236 If it remains use the second port on the HCA (if available) or use another HCA. Step 2 If it remains replace the ISR 9288/9096 Line Board (sLB) or the ISR 9024 Switch, Step 3 as required.

  • Page 237: Figure 14-7. Nodes Info Window

    This window allows the user to find the bad sLB or sFB by LID or GUID. Node information combined and portcounters.csv file, allows identify any fabric port connectivity failure that occurred during a pre-configured interval. 14-22 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 238 As a result, the SM ignores the “remote” part of the link (“remote” is the side of the link that is the most distant from the SM). 14-23 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 239 PortGUID, IntP#, Name, IP, Status 256, 8f104003f0670, 4, 0008f104003f0670, 0.0.0.0, Unknown Peer 256, 8f104003f0670, 6, 0008f104003f0670, 0.0.0.0, Unknown Peer 384, 8f104003f0671, 4, 0008f104003f0671, 0.0.0.0, Unknown Peer 384, 8f104003f0671, 6, 0008f104003f0671, 0.0.0.0, Unknown Peer 14-24 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 240: Figure 14-8. Bad Ports Log

    CLI command: # sm-info sm-initiate-fabric-configuration set To display the bad port table: Select Device—Bad Ports Log from the Voltaire Device Manager main menu. A browser window is displayed, showing the bad ports (see Figure 14-8). Figure 14-8. Bad Ports Log 14-25 P/N 399Z00038 ©...
  • Page 241 On each port in the InfiniBand fabric there are a few very important parameters which indicate the health status of a specific port (port-link), such as port-link- speed, port-link-width, port counters (12 counters), etc. 14-26 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 242 (default) or on a specific (unique) LID. NOTE Peer-connections are not listed. Therefore, when using this utility to verify a specific LID, the width/speed values are checked against the supported maximum. 14-27 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 243 # port-verify -z (all fabric : just zero port counters and exit) # port-verify (all fabric : verify port counters , Width and Speed) Refer to port-verify CLI command under the Utilities Mode, Appendix A.3.4. 14-28 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 244 HCAs. This script is very easy to use and is helpful if you want to start off with a "clean" baseline of your fabric after many changes have occurred. 14-29 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 245 5 guid 0008f104003f0723 ports 24 port 22 xmitdiscards:....4 port 10 linkdowned:......1 port 13 lid 4 guid 0008f104003f0722 ports 24 port 14 errs.sym:......83 Refer to error-find CLI command under the Utilities Mode, Appendix A.3.4. 14-30 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 246: Appendix A Cli Commands Reference

    Appendix A CLI Commands Reference This section provides information about the following CLI topics: Voltaire CLI Basic Concept, on page A-2 Exec (Admin) Mode, on page A-3 Privileged Mode, on page A-9 Configuration (Config) Mode, on page A-38 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 247 CLI Commands Reference Voltaire CLI Basic Concept Voltaire CLI Basic Concept The Voltaire CLI concept implements a hierarchical organization, as described in the diagram below. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 248 Shows the SNMP communities. version show Shows the software version information. The following details the commands available in Exec mode: Command: Description: Displays a list of available commands in the current CLI mode/menu. Syntax P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 249 IB denotes that the IB interface is the active interface). Syntax: fast-interface show Example: # fast-interface show fast ip 172.25.2.206 ip mask is 255.255.0.0 broadcast ip is 172.25.255.255 management interface is eth0:1 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 250 10 of the ISR 9288 rear panel, in the left router slot. NOTE The IP address of the router module must be on the same subnet as the sMB card with which the CLI session is established. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 251 Anafa No. Node GUID System GUID FW Version --------- --------- ----------- ---------- Line 2 -------- 8f104003f0682 8f104003f0682 0.8.0 8f104003f0683 8f104003f0683 0.8.0 Line 4 -------- 8f104003f0764 8f104003f0764 0.8.2 8f104003f0765 8f104003f0765 0.8.2 Spine 1 -------- 8f10400403009 8f10400403009 0.8.2 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 252 Returns the current parameter settings of the Subnet Manager in the ISRXXXX. Syntax: sm-info show Example: subnet manager info is: smName= LAVISM port guid= 0008f1040040061b topology=CLOS-stage-3 active topology=CLOS-stage-0 algorithm= balanced-routing active algorithm= balanced-routing sm KEY = 0000000000000000 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 253 Command: Description: Displays software version information. Syntax: version show Example: ISR 9288 version: 3.3.0 date: Feb 10 2005 10:51:52 AM build Id: 208 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 254 [admin, enable] Changes admin or enable password. ping Sends echo messages. qos-info show Shows the QoS parameters. rear show Shows the blade module information in the rear of the chassis. reload Restarts the switch software. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 255 [update-file-dir] update software update software [update-file-dir] utilities utilities version show version show A-10 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 256 (Config) Mode is detailed in Appendix A.4. Syntax: config debug Command: Description: Changes to Debug mode, which is intended for the use of Voltaire personnel only. Syntax: debug disable Command: Description: Disables the configuration mode and switches to the Exec mode, which allows view-only access of the switch configuration parameters.
  • Page 257 Shows the ftp client configuration Syntax: ftp show Example: ftp configuration ----------------------- remote server: 172.25.0.16 user: swget ISR9096-3009# logs Command: Description: Changes to the Logs Mode. The Logs Menu is detailed in Appendix A.3.4. Syntax: logs A-12 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 258 Line in slot #6: HW Version is "AAI"; Serial Number is "AAPV43040007" Line in slot #9: HW Version is "AAI"; Serial Number is "AAPV43040075" IO Drawer in slot #11: HW Version is "AAD"; Serial Number is "AARM44040006" A-13 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 259 This allows the user to quickly detect counter errors during connection of nodes. Syntax: sLBPortCounter show [slot #] Example: Switch# sLBPortCounter show 2 IO Drawer in this position A-14 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 260 = 0xfe80000000000000 port-state-change-trap = enable bad ports mode = enable pm mode = enable grouping mode = enable NOTE Refer to Section 6.7.1 for a description of the Subnet Manager parameters. A-15 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 261 Utilities Mode. The Utilities Menu is detailed in Appendix A.3.4. Syntax: utilities version show Command: Description: Displays hardware and software version information Syntax: version show Example: ISR 9288 version: 3.3.0 date: Feb 10 2005 10:51:52 AM build Id: 208 A-16 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 262 Show therm log. topology-log show Show topology log. umServer-log show Show log file vsm-internal-log show Show Voltaire Subnet Manager (VSM) internal log. vsm-log show Show Voltaire Subnet Manager (VSM) log. Logs syntax usage is as follows: NMS-log show NMS-log show...
  • Page 263 7 17:50:38 0 vsm[1013]: SM STATE CHANGE: Entering SM Discovering state, previous state SMINFO_INIT LID 18 IB Port 21. Peer at: 0008f104003f5886, port 2 Port-GUID 8f104003f5886 LID 12 IB Port 2. ] A-18 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 264 Command: Description: Shows the event log mask. Syntax: log-mask show Example: Switch(logs) log-mask show volPmSymbolErrorEvent is ON. volPmLinkErrorRecoveryEvent is ON. volPmLinkDownedCounterEvent is ON. volPmPortRcvErrorsEvent is ON. volPmPortRcvRemotePhysicalErrorsEvent is ON. volPmPortRcvSwitchRelayErrorsEvent is OFF. A-19 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 265 Shows the smAgent log file. Syntax: smAgent-log show Example: Switch(logs) smAgent-log show 8 08:29:15 0 sudo: root : TTY=unknown ; PWD=/usr/voltaire ; USER=root ; COMMAND=/usr/voltaire/scripts/ftpCli.sh get root br6000 172.25.3.94 /var/www/html/jarPropertyFile.txt /tmp/jarPropertyFile.txt /var/ftpLog.tmp 8 08:29:47 0 sudo: root : TTY=unknown ; PWD=/usr/voltaire ;...
  • Page 266 7 17:50:04 0 umServer[832]: getFromRepository#39: IN 7 17:50:05 0 umServer[832]: getFromRepository#52: OUT 7 17:50:05 0 umServer[832]: Starting umServer vsm-internal-log show Command: Description: Shows the status of Voltaire Subnet Manager (VSM) internal counters. Syntax: vsm-internal-log show Example: ISR9096-302f(logs) vsm-internal-log show Sun Feb...
  • Page 267 CLI Commands Reference Privileged Mode vsm-log show Command: Description: Shows the Voltaire Subnet Manager (VSM) internal log, listing the traps and events according to the user configuration of verbosity. Syntax: vsm-log show Example: Switch(logs) vsm-log show [17:50:36.327][1013] SM BUILD VERSION: IBSW_OSM_2_377 Nov 30 2005 18:21:02 [17:50:36.328][1013] osm_opensm_init: [...
  • Page 268 [-h] - Show this help Example: error-find.sh -H find_bad_ports find_bad_ports find_bad_ports findpath findpath [-d(ebug)] [-v(erbose)] [-s(imple)] [-f(astmode)] [-p(arser-output)][-r(eturn path show)] [-t timeout_ms] <portguid> getpathrecord getpathrecord [-d(ebug)] [-p gid_prefix] [-s smlid] [-t timeout_ms] <srcguid> <destguid> A-23 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 269 -m(od) -D(irect)] [-t timeout_ms] <dlid | path> <num> [win] switchlock Switchlock vendstat vendstat [-d] [-t timeout_ms] <Gpio|iNfo|portCause|Perormance> <lid> [args...] width-check width-check [lid |or nothing = pass on all fabric] zero-counters zero-counters [-m counter_mask] [-u uniq_lid] [-h] A-24 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 270 This command is useful in case the active Width/Speed of a specific port must be changed without actual cable reconnection. Syntax: port-manage [-v] [-f] <-d|-e|-r> <LID> <PORT> Options: [-v] - Increase output verbosity level A-25 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 271 (unique LID: zero port counters, wait 4 sec., verify port counters and Width) # port-verify.sh -z (all fabric : just zero port counters and exit) # port-verify.sh (all fabric : verify port counters , Width and Speed) A-26 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 272 Prints port path between two nodes. The first node is the management board itself, the end node is defined by the port guid. Syntax: findpath [-d(ebug)] -v(erbose) -s(imple) - f(astmode)[-t timeout_ms] <portguid> Example: findpath 0x8f104039651cd A-27 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 273 4) Entered second switch chip transmitted from switch chip via port 15. 5) Entered third switch chip transmitted from switch chip via port 2. Conclusion: on the requested path 3 switch chips and one entity (Voltaire HCA - see guid range) getpathrecord Command: Description: Checks the existence of a path between nodes.
  • Page 274 Q <lid|path> [<port>] [<set value>] - Get/Set port hQlife madstat L - Get local addr madstat I <lid|path> [<set value>] - Get/Set switch life time madstat G <lid|path> [<outport>] [<inport>] - Get Port SL to VL mapping table A-29 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 275 MC (multicast) group(s) and verifies whether the remote port has joined the MC group(s). Syntax: mcmember [-d(ebug)] [-t timeout_ms] [-s scope] [-m mgid] [-p portguid] [-pro(x)y] < [-j(oin)] [-g(et)] [-l(eave)] > Example mcmember -m 0xFF12401BFeFF000000000000FFFFFFFF -p 0x8f10400400707 -j A-30 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 276 Example: mctrace 0xc000 22 24 Output: From hca 0x8f10403962740 port 1 lid 0x16-0x16 "Voltaire HCA400" [1] -> switch 0x8f104003f07d9[14] lid 0xb "ISR9288 Voltaire sLB-24" [16] -> hca 0x8f10403962e24[1] lid 0x18 "Voltaire HCA400" To hca 0x8f10403962e24 port 1 lid 0x18-0x18 "Voltaire HCA400"...
  • Page 277 7 [24] "S-0008f104003f074f"[22] [22] "S-0008f104003f074f"[24] [10] "S-0008f10400403002"[10] "S-0008f1040040300e"[10] [23] "S-0008f104003f0765"[24] [15] "S-0008f104003f0765"[22] "S-0008f10400403009"[10] switchguids=0x8f104003f074e Switch 24 "S-0008f104003f074e" # ISR9288/ISR9096 Voltaire sLB-24 smalid 6 [24] "S-0008f104003f074e"[22] [22] "S-0008f104003f074e"[24] [10] "S-0008f10400403002"[7] "S-0008f1040040300e"[7] [16] "S-0008f104003f0765"[17] "S-0008f10400403009"[7] A-32 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 278 A direct routed MAD is sent to a node element in the fabric that corresponds to the specified direct-path. A direct routed mad can work in a fabric that is not configured. A path is specified using the following format: <port0,port1, port2, ...>. A-33 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 279 LID is assigned to 1 is indeed an IB switch. A direct routed MAD is sent to a node element in the fabric that corresponds to the specified direct-path. A-34 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 280 -a 18 -D 0,8 992 16 Output: ## start: win 16 direct route path 0,8 attr 18 mod 0 ## END: total sent 992 replied 992 resend 0 err 0 ms 119 avg ms 0.12 A-35 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 281 G <lid> - Get GPIO Data vendstat N <lid> - Get IS3 General Info vendstat C <lid> - Get IS3 Port Cause Bits vendstat P <lid> <port> - Get Port Performance Counters A-36 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 282 Zero lid 6 port 255 mask 0xffff Zero lid 10 port 255 mask 0xffff Zero lid 11 port 255 mask 0xffff Zero lid 5 port 2 mask 0xffff Zero lid 4 port 1 mask 0xffff A-37 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 283 Shows the Watchdog timeout. Config syntax usage is as follows: alarms alarms cluster cluster factory-default factory-default group group interface IB interface IB interface LOCAL interface LOCAL interface fast interface fast names names route route A-38 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 284 FTP server. Among other uses, the active FTP server is used for download of new switch software versions. See Appendix A.4.5 for a list of commands available under this menu. Syntax: A-39 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 285 This mode is also used to view all node names that were manually changed by the user via the GUI. See Appendix A.4.10 for a list of commands available under this A-40 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 286 WDT is to reset the switch in the event that no response is detected on the system. The watchdog timer is disabled by default. Syntax: watchdog mode set [enable | disable] Example: Switch(config)# watchdog mode set enable A-41 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 287 WDT performs the reset. The WDT timeout is factory set, and can not be changed by the user. Syntax: watchdog timeout show Example: Switch(config)# watchdog timeout show watchdog timeout 10200 [msec] A-42 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 288 2 minor Port Counter Switch, "ISR 9288" sFB-12 1 port 51 13/09/2005 18:11:13 6 PortRcvErrors 3 minor Port Counter Switch, "ISR 9288" sFB-12 4 port 36 13/09/2005 18:14:13 8 Link-Error-Recovery alarm clear-all Command: Description: Clears the all the current alarm entries. Syntax: alarm clear-all A-43 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 289 This command sets the IP address of the router module installed in the center position of the sRBD, which installed in slot 12 of the switch rear panel. io show Command: Description: Shows existing I/Os. Syntax: io show A-44 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 290 CLI Commands Reference Configuration (Config) Mode Example: Switch(config-cluster)# io show slot #2, position #2, I/O Type FCR, IP: 172.25.3.98 Mask: 255.255.0.0 slot #2, position #3, I/O Type FCR, IP: 172.25.3.94 Mask: 255.255.0.0 A-45 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 291 [remote path]. exportFile topology exportFile topology [remote path]. exportLOGs exportLOGs [remote path]. exportSm exportSm [remote path]. import import [remote-path ]. importFile group importFile group [remote path]. password password [password] A-46 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 292 This may take a few minutes. Syntax: exportLOGs [remote path] Example: ISR9288-0967(config-ftp)# exportLOGs /pub/ exportSm Command: Description: Uploads to the FTP server a log file that stores the Subnet Manager A-47 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 293 Configuration (Config) Mode activity. You should change the name of the file on the FTP server, as the next export action will overwrite the existing file. (SM = Subnet Manager – VSM – Voltaire Subnet Manager) Syntax: exportSm <ftproot-path> Example:...
  • Page 294 CLI Commands Reference Configuration (Config) Mode Description: Defines the user name that the switch uses to access the file server. Syntax: username [username] A-49 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 295 ISR9288-0957(config-group)# group import Summary report: Racks Elements Normal events :0 Warning events:0 Error events Error in input file importGroup failed ISR9288-0957(config-group)# group error-log show Error in row:8 Duplicate Guid -> = 8f10403960860 -- Reject A-50 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 296 Checks the group information for errors or warnings. It is recommended to run this command before performing a group import. Syntax: group verify Example: ISR9288-0957(config-group)# group import Summary report: Racks Elements Normal events :0 Warning events:0 Error events A-51 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 297 IB interface. Syntax: broadcast-ib set [broadcast ip address] Example: broadcast-ib set 172.25.255.255 Expected Results: ISR9096-3009(config-if-IB)# ip-address-ib show IB interface ip is 172.25.3.16 ip mask is 255.255.0.0 broadcast ip is 172.25.255.255 management interface is eth0:1 A-52 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 298 Sets the Maximum Transmission Unit of the IB interface. Syntax: mtu set [1500 | 2400] mtu show Command: Description: Shows the present Maximum Transfer Unit (MTU) settings. Syntax: mtu show Example: Switch(config-if-IB)# mtu show current mtu = 1500 A-53 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 299 Returns the IP address of the Local Interface, its subnet mask and the IP subnet to which the Local interface broadcasts. Syntax: ip-address-local show Example: Switch(config-if-LOCAL)# ip-address-local show local ip is 172.25.7.10 ip mask is 255.255.0.0 A-54 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 300 Syntax: ip-address-fast set [ip-address] [netmask] Example: ISR9096-3009(config-if-fast)# ip-address-fast set 172.25.3.16 255.255.0.0 Expected Results: ISR9096-3009(config-if-fast)# ip-address-fast show fast interface ip is 172.25.3.16 ip mask is 255.255.0.0 broadcast ip is 172.25.255.255 management interface is eth0:1 A-55 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 301 Fast. When under config/interface IB menu, the current management interface will be set to InfiniBand). Syntax: management-interface set Example: Switch(config-if-fast)# management-interface set A-56 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 302 [name] Example: Switch(config-names)# system-name set Switch1 Change will be effective next time CLI is activated Expected results: Switch(config-names)# system-name show system name is Switch1 At the next CLI activation: connecting Switch1> A-57 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 303 Switch(config-names)# unique-default-name set enable Change will be effective only after next reboot Expected results: Switch(config-names)# unique-default-name show enable unique-default-name show Command: Description: Displays the unique default name option. Syntax: unique-default-name show Example: Switch(config-names)# unique-default-name show enable A-58 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 304 Resets the sLB port counter settings. pm sLBPortCounter show Shows the sLB port counter settings. pm scope set Sets the scope for the PM. pm show Shows PM settings pm symbol-error-counter set A-59 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 305 [slot #] pm scope set pm scope set [all, all_hca ,all_sw , node_list] pm show pm show pm symbol-error-counter set pm symbol-error-counter set [0,65535] pm vl15-dropped set pm vl15-dropped set [0,65535] A-60 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 306 In this case, it is not recommended to change the default settings. pm link-error-recovery-counter set Command: Description: Defines the error threshold for this counter. Setting to 0 disables counter monitoring (default: 1) Syntax: pm link-error-recovery-counter set [0 - 255] A-61 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 307 Defines the error threshold for this counter. Setting to 0 disables counter monitoring (default: 5). Syntax: pm port-rcv-remote-physical-errors set [0 - 65535] Note: In this case, it is not recommended to change the default settings. A-62 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 308 Sets a reset scope for specific nodes in the fabric. For example, only HCAs counters will be reset. You can also set the counters of a specific node to reset. Syntax: pm reset-scope set [none|all|hca|switch|node_list] Example: pm reset-scope set hca A-63 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 309 Shows the current PM (Performance Management) configuration. Syntax: pm show Example: ISR9096-300b(config-pm)# pm show pm mode enable Trap mask [33554428] Polling interval Scope Reset-scope Counter operation delta Symbol error counter threshold Link error recovery counter threshold A-64 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 310 Defines the error threshold for this counter. Setting to 0 disables counter monitoring (default: 5). Syntax: pm vl15-dropped set [0 - 65535] Note: In this case, it is not recommended to change the default settings. A-65 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 311 [ip address] Example: default-gw fast set 172.25.0.1 default-gw show Command: Description: Shows the default Gateway of the fast Ethernet Syntax: default-gw show Example: ISR9096-300b(config-route)# default-gw show default-gw Ip is:192.168.60.93 default-gw interface is:IB A-66 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 312 Given a complex environment with multiple managed switch platforms, a subnet manager running over an ISR 9288/9096 platform will have priority over a subnet manager running over an ISR 9024 platform. NOTE Any configuration parameter, specifically the subnet manager priority parameter, is assigned a default value during the very first installation or after invoking the factory-default CLI.
  • Page 313 [error|info|verbose|debug|function| frames] sm-info verbosityTopology set sm-info verbosity Topologyset set [none|error|info|verbose| debug|function-|frames] spines add [row] [GUID] spines del [row] spines show trap-mask set trap-mask set [none, default, all] A-68 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 314 See page A-80 for a list of commands available under this menu. Syntax: sm-info algorithm set Command: Description: Sets the routing algorithm to be implemented by the Voltaire Subnet Manager. The Voltaire Subnet Manager supports the following routing algorithms: multi-path with balanced-routing, multi-path with sFB consistency, and the up-down algorithm. Syntax:...
  • Page 315 Sets the maximum number of Virtual Lanes operational on this port. The actual number of VL is the minimum between the switch capability (VLcap) and this parameter set by the user. Syntax: sm-info mopvl set [vl0,vl0-1,vl0-3,vl0-7,vl0-14] Example: sm-info mopvl set vl0 A-70 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 316 = verbose sm topology verbosity = none sm mads-pipeline = 1 sm polling-retries = 12 sm activity = 2283 sm state = master sm mode = enable sm LMC = 0 A-71 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 317 Sets the subnet prefix. The subnet prefix is used as the most significant 64 bit of the GID of each IB node in the subnet. Syntax: sm-info subnet-prefix set [hex int] Example: ISR9096-302f(config-sm)# sm-info subnet-prefix set 022 A-72 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 318 Adds an sFB to the sFB list. Used for Clos topology configuration. Syntax: spine add <row><Guid> Example: Switch(config-sm)# spines add 6 0008f104004004c4 Change will take effect after next Fabric Reconfiguration. Expected Result: Switch(config-sm)# spines show entry GUID |------|---------------- 0008f104004004eb 0008f104004004c7 0008f104004004c5 0008f104004004ea 0008f104004004e9 0008f104004004c4 A-73 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 319 Sets the trap mask. Syntax: trap-mask set [none, default, all] Example: Switch(config-sm)# trap-mask set all trap-mask show Command: Description: Shows the traps mask. Syntax: trap-mask show Example: Switch(config-sm)# trap-mask show volSmaMulticastGroupCreated volSmaMulticastGroupDeleted volSmaPortLinkStateChange A-74 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 320 ON volPmPortRcvConstraintErrorsEvent is ON volPmLocalLinkIntegrityErrorsEvent is ON volPmExcessiveBufferOverrunErrors is ON volPmVL15DroppedEvent is volOsm1xEvent is volOsmInvalidLinkEvent is volOsmBrokenLinkEvent is volOsmNoAvailableLidEvent is volOsmLmcTooBigEvent is volDupHcaGuidEvent is volDupSwitchEvent is volGidAddrInService is volGidAddrOutService is volOsm4xEvent is A-75 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 321 0.0.0.0, Unknown Peer 2944, 8f104004101c2, 15, 0008f104004101c2, 0.0.0.0, Unknown Peer 2944, 8f104004101c2, 17, 0008f104004101c2, 0.0.0.0, Unknown Peer 8f1040041119b, ISR9024-119b, 192.168.2.100, Ignored Peer 8f1040041119b, ISR9024-119b, 192.168.2.100, Duplicate HCA Peer GUID N/A, 8f10403962e25, N/A, N/A, User Ignored A-76 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 322 Refer to page 14-23 for a detailed description of this feature. Syntax: bad-ports remove [guid] [port (1..255)] Example: ISR9288-0967(config-sm-badports)# bad-ports remove 0008f104004101c2 Change will take effect after next Fabric Configuration ISR9288-0967(config-sm-badports)# sm-initiate-fabric-configuration A-77 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 323 In Multicast Auto-join mode a multicast group is created by the SM. HCAs can join or leave the Multicast group. Syntax: mc auto-join set [disable, enable] Example: Switch(config-sm-mc)# mc auto-join set enable Expected result: Switch(config-sm-mc)# mc show auto join enable A-78 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 324 CLI Commands Reference Configuration (Config) Mode mc show Command: Description: Shows the Multicast mode setting, enabled to disabled. Syntax: mc show Example: Switch(config-sm-mc)# mc show auto join enable A-79 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 325 16 partitions (Current HCA capability is limited to 8). Row 1 in the table corresponds to the default Pkey configuration. This entry is protected and can not be removed. This command enables or disables partitioning. Syntax: partitioning-enable set [enable, disable] A-80 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 326 Example: Switch(config-sm-pkey)# pkey high show entry Port-GUID Pkey8 Pkey9 Pkey10 Pkey11 Pkey12 Pkey13 Pkey14 Pkey15 |-----|----------------|------|-----|------|------|------|------|------|----- 0000000000000000 0 8f10001a10001 8f10001a10002 8f10001011302 0002C9000100D151 0 0002C9000100D152 0 8f100012b0001 8f100012b0002 8f10001011301 8f100010c0001 8f100010c0002 A-81 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 327 Change will be effective only after 'pkey update' command Expected results: Switch(config-sm-pkey)# pkey update Switch(config-sm-pkey)# pkey low show entry Port-GUID Pkey0 Pkey1 Pkey2 Pkey3 Pkey4 Pkey5 Pkey6 Pkey7 |-----|----------------|-----|-----|-----|-----|-----|-----|-----|------- 0000000000000000 8003 8f10001a10002 8f10001a10001 A-82 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 328 VoltaireHCA400 0x000C 0008f10403971a24 castor VoltaireHCA400 0x0001 0008f1040040300b N/A ISR9096VoltairesFB-4 0x0009 0008f1040040300a N/A ISR9096VoltairesFB-4 0x0007 0008f104003f06b4 N/A ISR9288/ISR9096VoltairesLB-24 0x0008 0008f104003f06b5 N/A ISR9288/ISR9096VoltairesLB-24 0x0005 0008f104003f0678 N/A ISR9288/ISR9096VoltairesLB-24 GUID NAME DESCRIPTION 0x0006 0008f104003f0679 N/A ISR9288/ISR9096VoltairesLB-24 ISR9096-300b(config-sm-pkey)# A-83 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 329: Qos Mode

    The following details the commands available under the sm QoS sub-menu. qos-apply-configuration set Command: Description: Changes made to the QoS configuration will be saved after typing qos-apply-configuration set yes Syntax: qos-apply-configuration set [yes, no] A-84 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 330 Command: Description: Sets VL in the SL-VL Table, as described on page 9-2 Syntax: qos-sl-vl set [row] [virtual lane] Example: Switch(config-sm-qos)# qos-sl-vl set 1 1 Expected Result: Switch(config-sm-qos)# qos-sl-vl show entry sl |-----|---|--- A-85 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 331 Sets the High Arbitration table, as described on page 9-4 Syntax: qos-vl-high-arbitration-limit set [row] [virtual lane] [priority weight] Example: Switch(config-sm-qos)# qos-vl-high-arbitration set 1 1 5 (config-sm-qos)# qos-vl-high-arbitration show Expected Result: entry vl weight |-----|---|--- A-86 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 332 Sets the Low Arbitration table, as described on page 9-4 Syntax: qos-vl-low-arbitration set [row] [virtual lane] [weight] Example: Switch(config-sm-qos)# qos-vl-low-arbitration set 1 2 1 Expected Result: Switch(config-sm-qos)# qos-vl-low-arbitration show entry vl weight |-----|---|--- A-87 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 333 LV is the LifeTimeValue component of the SwitchInfo attribute. If LV > 19, then SLL is to be interpreted as infinite. HLL is defined as 4.096us * 2^HL if 0 ≤ HL ≤ 19, +5% / -55%. A-88 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 334 SM Auto Join mode (Mini SM-like operation). This mode is disabled by default When enabled, the Subnet Manager will automatically add each active end port (HCA/SW) to the default IPoIB multicast group (MGID 0xFF12401BFEFF000000000000FFFFFFFF). A-89 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 335 In your browser, enter the string as follows: 1. Cluster Head Node IP Address 2. Ganglia tool 3. Cluster Name 4. Name of the Host Syntax: snmp cluster-monitor set [string] Example: snmp cluster-monitor set A-90 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 336 CLI Commands Reference Configuration (Config) Mode http://172.25.2.175/ganglia/?m=load_one&r=hour&s=descending&c =Support+Cluster&h=$IP&sh=1&hc=4 In the example below, the Host is called hydra3.voltaire.com snmp cluster-monitor show Command: Description: Shows the SNMP cluster monitor string. Syntax: snmp cluster-monitor show Example: ISR9096-3009(config-snmp)# snmp cluster-monitor show cluster-monitor is http://172.25.2.175/ganglia/?m=load_one&r=hour&s=descending&c= Support+Cluster&h=$HOST&sh=1&hc=4...
  • Page 337 ISR9096-3009(config-snmp)# snmp contacts set Voltaire Expected Results: ISR9096-3009(config-snmp)# snmp contacts show Contacts is Voltaire snmp contacts show Command: Description: Shows the SNMP agent contacts. Syntax: snmp contacts show Example: ISR9096-3009(config-snmp)# snmp contacts show Contacts is Voltaire A-92 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 338 Sets the SNMP location. Syntax: snmp location set [location] Example: ISR9288-0957(config-snmp)# snmp location set snmp location show Command: Description: Shows the SNMP agent location. Syntax: snmp location show Example: ISR9288-0957(config-snmp)# snmp location show A-93 P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 339: Appendix B Port Counters Fields

    Initialize—the switch has been powered up and is in initializing. Armed—the port is in armed mode. Active—the switch is operational. Displays the Maximum Transmission Unit value of the port. Num VLs Displays the Virtual Lanes supported by the port. Current support is P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 340 SymErr, LinkRecovers, LinkDowned, RcvErr, RcvRemotePhys, RcvSwitchRelay, XmtDiscards, XmtConstraint, RevConstraint, LocalLinkInteg, ExcessBufOvrrun, VL15Dropped Data Counters include: XmitBytes (GB), XmitBytes Rate (MB/Sec), RcvBytes (GB), RcvBytes (MB/Sec), XmitPkts, XmitPkts/Sec, Rcv Pkts, Rcv Pkts/Sec, XmitWaits. PeerLID The remote connected Ports LID P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 341 Total active switch chips ports Switch Alert Ports: Total current switch ALERT flagged ports Total HCAs: Total active HCA chips HCA Ports: Total active HCA chips ports HCA Alert Ports: Total current HCA ALERT flagged ports P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 342 0 – 255 PortRcvConstraintErrors 0 – 255 LocalLinkIntegrityErrors 0 – 15 ExcessiveBufferOverrunErrors 0 – 15 VL15Dropped 0 – 65535 PortXmitData 0 – 65535 PortRcvData 0 – 65535 PortXmitPackets 0 – 65535 PortRcvPackets 0 – 65535 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 343 Please contact FilterRawOutbound is true and packet is raw Voltaire Customer Support. PartitionEnforcementOutbound is true and packet fails partition check, IP version check, or transport header version check. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 344 Total number of data octets, divided by 4, transmitted on all VLs from the port selected by PortSelect. This includes all octets between (and not including) the start of packet delimiter and VCRC. It excludes all link packets. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 345 VCRC. It excludes all link packets. PortXmitPackets Total number of packets, excluding link packets, transmitted on all VLs from the port. PortRcvPackets Total number of packets, excluding link packets, received on all VLs from the port. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 346 Appendix C Supported Traps and Events An event corresponds to an occurrence within the fabric. Traps are generated, some of which are translated into alarms, per severity, according to Voltaire or user- defined thresholds. Trap Structure Each trap is characterized by the parameters listed below.
  • Page 347 PM_SymbolError Symbol-Error counter threshold Object: port warning enable Manual was exceeded. New value is XYZ, previous value was XYZ. Parameters: current value, last value Link-Error-Recovery counter PM_LinkErrorRecovery minor enable manual threshold was exceeded. New P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 348 New value is XXX, previous value was XYZ. Parameters: current value, last value FABRIC_CONFIG_ERROR Found invalid link from %s OSM_EVENT_INVALID_LINK critical enable manual GUID %016Lx port#XYZ to %s Object: port GUID %016Lx port#XYZ. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 349 Local Link Integrity threshold minor enable Manual reached. Object: port FLOW_CONTROL_ERROR SRV__EXCESSIVE_BUF_OVERRUN_TRA Excessive Buffer Overrun P_130 minor enable Manual threshold reached. Object: port SRV__FLOW_CONTROL_TIMER_EXPIRE Switch Flow Control Update D_TRAP_131 minor enable Manual watchdog timer expired. Object: port P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 350 /QP XYZ> on SL XYZ at switch LID XYZ port#XYZ. (prefix, id), to qp, sl SRV__SM_INFO_EVENT OSM_EVENT_OUT_OF_RESOURCE_STA Rejected multicast %s for port warning disable manual GUID %016Lx. P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 351 Get Subnet Port Counters OMPLETED normal Completed. Object: SM OSM_EVENT_MCAST_CONFIG_REPORT Object: SM Configured MLID 0x%X for %u normal ports, max tree depth = %u. Parameters: mlid, number of ports (int32), max tree depth (int16) SRV__SM_STATE_CHANGE_EVENT P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 352 Object: SM minor Enable manual number of sFBs (XYZ). Parameters: number of sFBs (int32), predefined number of sFBs (int32) OSM_EVENT_MINIMAL_LMC_VALUE_R Minimal LMC value of XYZ is warning enable manual EQUIRED required for persistent CLOS P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 353 %016Lx%016Lx. Object: SM OSM_EVENT_COMPLETE_SUBNET_REC normal Complete subnet reconfiguration. Object: SM OSM_EVENT_APPLIED_ROUTING Object: SM normal Applied %s routing scheme Parameters: routing scheme (char*) OSM_EVENT_PORT_STATE_CHANGE Port State Change. Reporting normal Switch LID XYZ. Object: node P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 354: Appendix D Glossary

    Direct Memory Access Fibre Channel Fast Ethernet File Transfer Protocol Field Replaceable Unit Gigabit Ethernet Gigabit Ethernet Graphical User Interface GUID Global Unique ID Gateway Host Control Adaptor High-performance computing Head of Queue InfiniBand Input Output P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 355 SNMP also supports multiple network management workstations organized in a hierarchical fashion. sFB12 Fabric Board with 12-port support sLB24 Line Board with 24 ports sFU-x Fan Unit Service Level Subnet Manager SNMP Simple Network Management Protocol P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 356 Glossary Transmission Control Protocol Unreliable Datagram Protocol Voltaire Device Manager Voltaire Fabric Manager Virtual Lane Voltaire Subnet Manager Work Queue Elements P/N 399Z00038 © 2006 Voltaire, Inc.

  • Page 357: Appendix E Index

    14-26, A-74 CLI, Fast Interface Mode, A-52 Baud, 3-3 CLI, IB Interface Mode, A-49 broadcast-fast set, A-49, A-52 CLI, Local Interface Mode, A-51 CLI, sm-info commands, A-67 CLI-log show, A-18 clock show, A-4, A-11 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 358 Filtering, 6-56 exportSm, A-46 Current Alarms window, 6-56 Fabric Board, 1-6 Data center reconstruction, 1-2 Fabric Diagnostic Data Clusters, 1-2 1X Cable Width, 14-20 deadlock, avoiding, 9-2 1X link, 14-20 Debug mode, CLI, A-11 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 359 A-26 High-speed interprocessor communication, 1- findpath, A-27 Firmware Updates, A-9 Host Channel Adapter, 1-3 Firmware Upgrade, 4-5 Host Information, viewing in VFM, 6-34 First-Time Configuration of switch, 3-2 Hosts List, in VFM, 6-33 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 360 Memory sharing, 1-2 ISR 9288 Switch, 5-13, 5-14 MIB, 1-10 ISR9288 FRUs, 1-6 MKey, 6-38 MKEY, 6-30 module-firmware show, A-6 Mozilla, 6-3 Java Web Start application, Sun, 6-4 MTU, 5-20, 5-29, 6-29, 6-34, 6-36, 6-38 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 361 Remote Management Setting using the port connection matrix, displaying, 14-22 Device Manager, 5-41 Port Counters, in VFM, 14-10 Remote Path field, 5-44 Port GUID, 6-29 remote-logger delete, A-20 Port information, Infiniband. See Infiniband remote-logger set, A-20 port information P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 362 10-7, A-66 Switches List, in VFM, 6-31, 6-35 sm-info CLI Commands, A-67 Switching Topology, 1-7 sm-info show, A-7, A-8, A-15 switchlock, A-35 SM-KEY, 6-42 System Requirements, Fabric Manager, 6-3 smpdump, A-34 P/N 399Z00038 © 2006 Voltaire, Inc.
  • Page 363 VL arbitration, 9-2, 9-4, 9-5 therm-log show, A-20 VLs, definition, 9-2 Time and Date configuration, CLI, 3-9 Voltaire Device Manage Initiate window, 5-3 Tool Tips, 5-16, 6-12 Voltaire Device Manager, 5-1 Tool tips, in VFM Topology map, 6-12 Voltaire Device Manager , terminating, 5-6...

This manual is also suitable for:

Isr 9096Isr 9288

Table of Contents