Page 2
You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.
® Revision History—Intel Omni-Path Fabric Revision History For the latest documentation, go to: http://www.intel.com/omnipath/SwitchPublications. Date Revision Description June 2019 10.0 • Updated the FM Switch LED description. • Edge switch product naming. Updated remaining instances of "unmanaged" to "externally-managed" wherever applicable.
Page 5
® Contents—Intel Omni-Path Fabric 2.5.3 Edge Switch Cabling Recommendations............71 2.6 Setting Up the USB Console................... 73 2.7 Bringing Up the System For the First Time..............73 2.7.1 Start-up Procedures.................. 73 2.7.2 Changing the Switch IP Address and Default Gateway through the CLI..... 73 2.7.3 Updating the Management Module IP Addresses in a Director Class...
Omni-Path Software Installation, User, and Reference Guides (includes HFI documents) http://www.intel.com/omnipath/FabricSoftwarePublications • Drivers and Software (including Release Notes) http://www.intel.com/omnipath/Downloads ® Use the tasks listed in this table to find the corresponding Intel Omni-Path document. Task Document Title Description Key: Shading indicates the URL to use for accessing the particular document.
Page 10
This document ® ® Using the Intel Intel Omni-Path Fabric Quick Start ® outlines the most basic steps for getting your Intel Omni- documentation set Guide ® Path Architecture (Intel OPA) cluster installed and operational.
Running MPI and Message-Passing Interface (MPI) application applications on programmers. ® Intel ® Provides a reference for programmers working with the Intel Writing and running ® Intel Performance Scaled Messaging PSM2 Application Programming Interface (API). The middleware that 2 (PSM2) Programmer's Guide Performance Scaled Messaging 2 API (PSM2 API) is a low- ®...
Procedures and information may be marked with one of the following qualifications: • (Linux) – Tasks are only applicable when Linux* is being used. ® • (Host) – Tasks are only applicable when Intel Omni-Path Fabric Host Software or ® Intel Omni-Path Fabric Suite is being used on the hosts.
If you do not wish to so agree, do not install or use the software. Technical Support ® Technical support for Intel Omni-Path products is available 24 hours a day, 365 days a year. Please contact Intel Customer Support or visit http://www.intel.com/omnipath/ support for additional detail. ® Intel...
Omni-Path Fabric—Installation Installation This section describes how to install and configure for first-time use: ® • The Intel Omni-Path Fabric Series switches in a network environment. Planning the Installation ® The Intel Omni-Path switches are designed for installation in standard 19-inch equipment racks.
® Installation—Intel Omni-Path Fabric Figure 1. Edge Switch Port Connection Limitations 2.1.1.2 Cable Guidelines NOTE Building and electrical codes vary depending on the location. Comply with all code specifications when planning the site and installing cable. When running cable to the equipment, consider the following: •...
® Intel Omni-Path Fabric—Installation 2.1.1.3 Cable Handling and Bend Radius ® • Intel Omni-Path Copper Cable 100 Series: — Recommended minimum bend radius is >69mm ® • Intel Omni-Path Active Optical Cable 100 Series: — Recommended minimum bend radius is >105 mm within 100 mm of either cable end module, and >60 mm on the rest of the cable...
® Installation—Intel Omni-Path Fabric Figure 4. Fiber Optic Bend Radius 2.1.2 Uninterruptible Power Supply Equipment Consider the following when selecting uninterruptible power supply (UPS) equipment: • The minimum amperage requirements for a UPS: — Calculate VA (volt-amps): Locate the voltage and amperage requirements for each piece of equipment (usually located on a sticker on the back or bottom of the equipment).
Torque wrench • Open sockets or open wrenches ® • Lifting device (e.g., pallet jack) for the Intel Omni-Path Director Class Switch 100 Series 2.2.2 Check the Installation Site The switches are designed to be installed in an existing server cabinet (not a telco cabinet), where it can be mounted in a standard equipment rack.
Omni-Path Fabric Edge Switch Installation ® This section describes the rack mounting instructions for the 24- and 48-port Intel Omni-Path Fabric Edge Switches in a four-post standard-depth rack. The installation of either switch is very similar; the majority of steps apply to both switches. Where applicable, any specifics for either switch are called out.
® Intel Omni-Path Fabric—Installation Figure 6. Rail Installation Pin a. Fasten the rail using two (2) M6 screws. b. Torque screws to 30 in-lb. Repeat this step for the second rail. 2. Fasten the port-side rails using four (4) M6 screws.
® Installation—Intel Omni-Path Fabric ® Figure 8. Intel Omni-Path Fabric Edge Switch Mounted in a Four Post Standard Rack 3. The rails are designed for different mounting positions, from flush mount to recessed (in 1 inch increments) as shown in the following figure.
® Intel Omni-Path Fabric—Installation Figure 9. Switch Rail Adjustments 4. To change the depth of the switch, insert your fingers into the rail adjustment notches and pull away from the switch as shown in the following figure. This disengages the rail latches from the switch hat section.
This is also true for patch panels. • Top-of-rack and bottom-of-rack transitions must be closed out ® The Intel Omni-Path Edge switch is generally shorter than the associated servers installed in the same cabinet. Therefore, the following additional considerations should be considered: •...
® Intel Omni-Path Fabric—Installation Figure 11. Edge Baffle (Top View) NOTE Installation of the baffle(s) take place after the edge switch has been rack mounted. 1. Unlock the rail system by pulling the handles out. 2. Guide the baffle onto the rails, then slide the baffle into the rack until it stops against the edge switch.
® Installation—Intel Omni-Path Fabric Figure 12. Secure the Top Baffle to the Switch 3. After baffle is fully inserted, push the handles back until the rail locks in place. To vertically secure the baffles, insert the baffle clamp (one on each side) as shown: 1.
® Intel Omni-Path Fabric—Installation Figure 15. Clamp over Rail Post 4. Reattach the rail post, which locks the clamp into place. Figure 16. Reattach the Rail Post 2.3.4 Installing the Switch Fascia The Edge switch fascia guards against ESD as well as accidental contact with the fans and power supplies.
® Installation—Intel Omni-Path Fabric Figure 17. Switch Fascia The fascia is installed by snapping it in place onto the switch rail posts. Figure 18. Rail Posts for Fascia Installation 2.3.5 Reversing the Fan Airflow This section details the procedure to reverse the airflow on the switch fan assembly from the default port-side exhaust position.
® Intel Omni-Path Fabric—Installation 1. Loosen the 4 thumbscrews from the switch cover. There are 3 screws on the top, and 5 screws on the back. Lift the cover on a 45-degree angle, then pull to remove. Figure 19. Switch Cover Thumbscrews (Top View) 2.
® Installation—Intel Omni-Path Fabric Figure 21. Wire Assembly Strain Relief and Airflow Direction Indicators 4. Reattach the switch cover. 2.3.6 Installing the Q7 Management Module This section details the procedure to install the Q7 Management Module into an existing externally-managed edge switch.
® Intel Omni-Path Fabric—Installation 1. Loosen the 4 thumbscrews from the switch cover. There are 3 screws on the top, and 5 screws on the back. Lift the cover on a 45-degree angle, then pull to remove. Figure 23. Switch Cover Thumbscrews (Top View) 2.
® Installation—Intel Omni-Path Fabric Figure 25. Q7 Module Mounting Screw 4. Reattach the switch cover. NOTE For information on upgrading Q7 board controller firmware (BCFW) and BIOS, refer to Q7 BIOS Update on page 86. ® Intel Omni-Path Director Class Switch 100 Series Installation 2.4.1...
® Installation—Intel Omni-Path Fabric Figure 32. Unpacking the Switch (7) 6. Remove the plastic bag, then remove the support shelf from the top of the switch. 2.4.3 Installation Tasks The following section details rack mounting instructions for the 6-slot and 24-slot ®...
® Intel Omni-Path Fabric—Installation Figure 33. Measure the Rack 2. Before installing into the cabinet, preset the adjustable rails on the baffle base, leaving the screws loose to the dimension measured on the cabinet (27" in this example). In order to be able to fit the baffle base into the cabinet, set the rails to slightly less than the measured size (i.e., <...
® Installation—Intel Omni-Path Fabric Figure 35. Install Fixture Brackets 4. Secure the shelf to the rack using M6 mounting screws, 3 per shelf rail (1 in the front and 2 in the back) as displayed in the following figure. Torque screws to 30 in-lb.
® Intel Omni-Path Fabric—Installation Figure 37. Secure the Baffles to Top of Shelf Rails 6. Secure the top of the baffles to the rack rail as displayed in the following figure: • 6-Slot Chassis: use 2 mounting nuts, 1 per rail.
® Installation—Intel Omni-Path Fabric Figure 38. Secure the Top of the Baffles to the Rack Rails 7. The completed Shelf assemblies for both the 6- and 24-slot switches are displayed in the following figure. ® Intel Omni-Path Fabric Switches Hardware...
® Intel Omni-Path Fabric—Installation Figure 39. Completed Shelf Assemblies 2.4.3.2 Installing the Director Switch Chassis WARNING To avoid injury, do not lift a fully-loaded Director Class switch manually. Use a mechanized lift only. 1. Align the chassis with the rack.
® Installation—Intel Omni-Path Fabric Figure 40. Slide the Chassis into the Rack 4. Fasten the chassis to the rack with M6 screws: • 6-Slot Director: six (6). • 24-Slot Director: twelve (12). ® Intel Omni-Path Fabric Switches Hardware June 2019 Installation Guide Doc.
® Intel Omni-Path Fabric—Installation Figure 41. Fasten the Chassis to the Rack 5. The completed switch installations for both the 6- and 24-slot switches are displayed in the following figure. ® Intel Omni-Path Fabric Switches Hardware Installation Guide June 2019...
® Installation—Intel Omni-Path Fabric Figure 42. Completed Switch Installations 2.4.4 Installing the Optional Cable Management Guide The optional Cable Management Guides are attached to the rack rails with M6 cage nuts and screws as shown in the following figure. ®...
® Installation—Intel Omni-Path Fabric 2.4.5 Installing Director Class Modules 2.4.5.1 Spine, Leaf and Management Modules NOTE In order to maintain thermal integrity, all open Spine and Leafs slots must be populated with filler panels. ® Intel Omni-Path Fabric Switches Hardware...
® Intel Omni-Path Fabric—Installation NOTE When installing the Leaf Module, there is no "correct" orientation (i.e., no top or bottom). These modules all have release arms that function in a similar way during installation. The Management Module (MM) is used here as the example.
® Installation—Intel Omni-Path Fabric Figure 45. Module Installation 2 2. Slide the module straight into an applicable open slot until it engages with the backplane connector. CAUTION Never push on the release arms when sliding the modules into the applicable slots.
® Intel Omni-Path Fabric—Installation Figure 47. Module Installation 4 5. Chassis management slot leaf only: If the leaf module is in one of the chassis management slots (i.e., L101 or L102) the mechanical release latch (MRL) must be moved in order to install the leaf. To move the MRL, loosen the thumbscrew attached to the Serial, Ethernet, Chassis EEPROM Board (SEEB) module, then pull down to release the MRL as shown in the following figure.
® Installation—Intel Omni-Path Fabric 2.4.5.2 Fans 1. Slide the fan module straight into an applicable open slot until it engages with the backplane connector. 2.4.5.3 Power Supplies 1. Loosen the thumbscrew until it disengages from the power supply unit. 2. Pull the handle down.
® Intel Omni-Path Fabric—Installation NOTE ® The following instructions are for the Intel Omni-Path Edge Switch with hot swap power and fan modules (Part Numbers 100SWE48UFH and 100SWE48QFH). 2.4.6.1 Power Supply 1. Grasp the module by the handle and slide into an open slot until the unit engages with the connector.
® Installation—Intel Omni-Path Fabric Figure 50. Edge Power Supply Install 2 2.4.6.2 Fans 1. Grasp the module by the handle and slide into an open slot until the unit engages with the connector. Figure 51. Edge Fan Supply Install 1 2.
Connect Equipment to the Ports and Power On the System NOTE Before connecting equipment, it is important to understand the locations of the serial ® and Ethernet ports on the Intel Omni-Path Fabric series switches. For the edge switch, these are located on the switch port side. Figure 53.
® Installation—Intel Omni-Path Fabric 1. Connect a Category 5 or 6 (Cat 5/6) Ethernet cable to the RJ-45 connector(s) on the switch. Connect the other end of the Cat 5/6 to an OOB LAN workstation, another switch or a hub.
® Intel Omni-Path Fabric—Installation 2.5.2 Leaf Module Port Numbering The physical module contains two 16-port leafs-A and B-stacked on top of each other. The physical port numbering is shown below. Figure 56. Leaf Module Port Numbering Figure 57. 48-Port Leaf Module Port Numbering...
® Intel Omni-Path Fabric—Installation Figure 59. 48-Port Edge Switch Cabling Recommendations 2.5.3.1 Connecting Power NOTE For each power supply installed, a power cord must be installed in corresponding power inlet. 1. Provide strain relief for the power cable(s). 2. If necessary, replace the closeout over the switch fan side.
® Installation—Intel Omni-Path Fabric NOTE For the Director switches, make certain the DC ON/OFF switch is illuminated. If it is not, press the button to supply power. The switch is located in the lower left corner on the chassis spine side.
Page 74
® Intel Omni-Path Fabric—Installation NOTE If using a serial port on a Director switch, make certain to connect to the SEEB serial port associated with the Management Module (MM) on the opposite side of the chassis. 2.7.2.1 Using the Serial Connection 1.
® Installation—Intel Omni-Path Fabric where is the new default gateway IP address in dotted decimal -h <ipaddress> format. 6. The changes are effective immediately. 2.7.3 Updating the Management Module IP Addresses in a Director Class Redundant Management Configuration Each management module (MM) must have a unique IP address that is different than the chassis IP address of the switch.
® Intel Omni-Path Fabric—Installation 2.8.1 Edge Switch LEDs Figure 60. Edge Switch LEDs 2.8.1.1 RJ45 LEDs The RJ45 connectors have two LEDs, Link and Speed. The Speed LED is Green when a 100 Mbps link is connected. The Link LED is...
® Installation—Intel Omni-Path Fabric 2.8.1.3 Ports, Fabric Manager (FM), Managed (M), and FM Switch LEDs Figure 61. Ports, Fabric Manager (FM), Managed (M), and FM Switch LEDs Ports Each switch cable port has a Green link status LED that provide the following indications: •...
® Intel Omni-Path Fabric—Installation Figure 62. Activating the FM Enable Switch LED The FM Enable Switch is used as follows: NOTE Actions take place approximately 1 second after releasing the button. • Pressing the button for 4 seconds does a "Restore Factory Defaults" and reboots the ASIC.
® Installation—Intel Omni-Path Fabric 2.8.1.4 Power Supply LEDs Figure 63. Power Supply LEDs PSU1/PSU2: • Green indicates that DC power is normal. • Off indicates a DC power failure or no DC power is present. 2.8.1.5 Airflow Direction LED Figure 64.
® Intel Omni-Path Fabric—Installation 2.8.2.1 Management Module LEDs Figure 65. Management Module LEDs Module Status (M_Stat) The status LED indicates one of the following conditions: • Steady Green: the module is operating normally. • Blinking Green: LED test state. •...
® Installation—Intel Omni-Path Fabric Service (SVC) The Chassis Service (SVC) LED indicates the following condition: • Green: Chassis beacon is active • Off: Not selected Master • Green: the module is acting as the master Management Module • Off: Management Module is not the Master MRL Active •...
® Intel Omni-Path Fabric—Installation • Off: the system functioning normally. • Steady Amber: the system requires some attention, which could indicate one of the following conditions: — The switch temperature is at a warning level — The switch silicon temperature is at a warning level —...
® Installation—Intel Omni-Path Fabric 2.8.2.4 SEEB Module LEDs Figure 68. SEEB Module LEDs RJ45 LEDs The RJ45 connectors have two LEDs, Link and Speed.The Speed LED is Green when a 100Mbps link is connected. The Link LED is Amber when an Ethernet link has been established, and blinking when the link is active.
® Intel Omni-Path Fabric—Installation 2.8.2.5 Fans and Power Supplies LEDs Fan LEDs Figure 69. Director Class Fan LEDs Fan LEDs indicate the following status(es): • Green indicates that the fan is functioning properly. • Amber indicates that the following warning condition exists: —...
® Installation—Intel Omni-Path Fabric Power Supply LEDs Figure 70. Director Class Power Supply LEDs Power LEDs indicate the following status(es): • Green indicates that AC power is present. • Green indicates that DC power is present. • FLT: indicates that DC power is off/interrupted (e.g., power cord not seated properly).
® Intel Omni-Path Fabric—Q7 BIOS Update Appendix A Q7 BIOS Update Before Getting Started • Ensure that the Q7 unit being updated is a QA3E. This unit can be found in either a managed Edge switch or a Director Class switch management module.
® Q7 BIOS Update—Intel Omni-Path Fabric • If you have a QA3E card(s) that does not have BIOS version QA3ER962 and BCFW version CGBCP429, you will need to perform the update. Run the CLI command . If BCWF and BIOS updates are required, the system returns...
® Intel Omni-Path Fabric—Q7 BIOS Update You can check the progress of the update with the CLI command . The system returns information similar to the showLastBcFwRetCode -all following: Edge-> showLastBcFwRetCode -all Module Board Controller Firmware Last Exit Code: 3: In Progress Edge->...
Page 89
® Q7 BIOS Update—Intel Omni-Path Fabric 2. At the master MM prompt, enter the following command to update the master and slave: sftp> put CGBCP429.q7.spkg A:/firmware/bcfw.pkg The system returns information similar to the following: Uploading CGBCP429.q7.spkg to /firmware/A:/firmware/bcfw.pkg CGBCP429.q7.spkg 100% 193KB 11.3MB/s...
® Intel Omni-Path Fabric—Director Switch Cable Management Appendix B Director Switch Cable Management This example explains cabling a Director switch in a raised floor cable management scenario. 1. Cable the Leaf modules from the top rows of the chassis down toward the floor.