Table of Contents
Advertisement
Quick Links
Advertisement
Table of Contents
Related Manuals for Nvidia Skyway GA100
Summary of Contents for Nvidia Skyway GA100
- Page 1 NVIDIA Skyway InfiniBand-to-Ethernet Gateway User Manual...
-
Page 2: Table Of Contents
InfiniBand-to-Ethernet Gateway Operational Description ......8 1.4.2 Load Balancing and High Availability Operational Description...... 8 Operating System ................9 Certifications ...................9 System Layout and Interfaces............10 NVIDIA Skyway Front and Rear Panel............10 Interfaces Detailed Description............11 2.2.1 Power-On LED ................11 2.2.2 USB 3.0 Interfaces ................11 2.2.3... - Page 3 System Connectivity ................. 21 Initial Power-On ................21 Configuring the Gateway for the First Time ........23 Gateway Initialization............... 23 4.1.1 Rerunning the Wizard ..............27 Starting the Command Line Interface (CLI)..........28 Networkwide Deployment Guidelines ..........29 Configuring High Availability (HA) ............29 5.1.1 Before Configuring HA..............29 5.1.2...
- Page 4 About this Manual This manual describes the installation and basic use of NVIDIA Skyway™ InfiniBand-to-Ethernet gateway. Ordering Part Numbers The table below provides the ordering part number (OPN) for the available NVIDIA Skyway gateway. NVIDIA SKU Legacy OPN Marketing Description 920-9B020-00FA-0D2 ...
-
Page 5: Introduction
1 Introduction This is the user guide for the NVIDIA Skyway InfiniBand-to-Ethernet gateway. This document contains the complete product overview, installation and initialization instructions, and product specifications. This document is preliminary and subject to change. 1.1 Product Overview NVIDIA Skyway GA100 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet storage or other Ethernet-based communications to access the InfiniBand datacenter, and vice versa. -
Page 6: Main System Components
These power supply units can be removed from the system only if they are being replaced. 1.2.3 Fans 1.2.3.1 Power Supply Fans NVIDIA Skyway is equipped with one fan per power supply unit on the rear panel of the appliance. ... -
Page 7: Package Contents
1.2.3.2 Internal Fans NVIDIA Skyway is equipped with six internal fans for cooling the CPU and expansion cards. Under normal operation, the cooling fans operate at a constant speed. If the system module fails or one of the temperature thresholds is exceeded, the cooling fans automatically raise their rotation speeds to draw more airflow. -
Page 8: System Features
ConnectX cards and gateway appliances. A single NVIDIA Skyway supports a maximum bandwidth of 1.6Tb/s, utilizing 16 ports, each of which reaches 100Gb/s traffic. In terms of connectivity, the InfiniBand ports can be connected to the InfiniBand network using HDR/HDR100 or EDR speeds, while the Ethernet ports can be connected to the Ethernet network using 200Gb/s or100Gb/s. -
Page 9: Operating System
The same GID and LID remain, even when handled by a different ConnectX HCA. 1.5 Operating System NVIDIA Skyway includes the NVIDIA Gateway operating system, MLNX-GW, which manages the appliance and handles the high availability and load balancing between the ConnectX cards and between gateway appliances. -
Page 10: System Layout And Interfaces
2 System Layout and Interfaces The figures below show the front and rear sides of NVIDIA Skyway. Each numbered interface that is referenced in the figures is described in the following table with a link to detailed information. 2.1 NVIDIA Skyway Front and Rear Panel ... -
Page 11: Interfaces Detailed Description
2.2 Interfaces Detailed Description 2.2.1 Power-On LED There is one I/O LED (green) on the front panel, to indicate if the system is powered. • For Power-On LEDs definitions, please refer to Power-On LEDs Specifications 2.2.2 USB 3.0 Interfaces Skyway offers four USB 3.0 ports on the system's rear panel. The USB interfaces are USB 3.0 compliant and can be used to provide the bandwidth up to 500MB/s to shorten the time for data transmission. -
Page 12: Redundant Power Module
2.2.6 Fans Modules 2.2.6.1 Power Supply Fans NVIDIA Skyway is equipped with one fan per power supply unit on the rear panel of the appliance. 2.2.6.2 Internal Fans NVIDIA Skyway is equipped with six internal fans for cooling the CPU and expansion cards. Under normal operation, the cooling fans operate at a constant speed. If the system module fails, or one... -
Page 13: Hardware Installation
3 Hardware Installation Installation of the NVIDIA Skyway gateway requires attention to the mechanical and power elements of the appliance and precautions must be taken for the rack-mounted equipment. The system platform can be rack-mounted and is designed for installation in a standard 19” rack. -
Page 14: System Requirements
The operating environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level. 3.1.2.2 Airflow Requirements NVIDIA Skyway appliance is offered with one airflow pattern: from the front panel to the rear panel. Refer to the Technical Specifications section for airflow numbers. -
Page 15: Rack Mounting
3.1.4 Rack Mounting The NVIDIA Skyway appliance can be mounted in a rack using the optional rack mounting kit. We strongly recommend that the minimum depth of cabinet is 1100mm. 3.1.4.1 Installing the Server in a Rack Before mounting the NVIDIA Skyway appliance in a rack, ensure that all internal components have been installed and that the unit has been fully tested. - Page 16 The server slides are developed for 1U or 2U applications of which system load does not exceed 75lbs. The slide length is 1041±3.0mm. The rear bracket is extendable to a max/min post-to-post distance of 670-1042 mm. The slide extension is 610.0±3.0mm. Step 1: Remove inner member. Pull inner member out as in the illustration. Step 2: Mount the inner member onto the chassis. Place the key slot on T stud, and push the inner member toward the back.
- Page 17 Step 4: Release the locking latch upward. Step 5: Push the middle member forward to the rear of the slide. Step 6: Install the chassis. As shown, insert the inner member to the cabinet member. Make sure the ball retainer is in the...
- Page 18 open position. If the ball retainer is not on the front position, it might cause damage to the slides. After the inner member goes in, push up/down the disconnect lever to unlock the slides and keep pushing the chassis to the fully-closed position. Step 7: Screw the system in the cabinet.
-
Page 19: Cable Installation
3.2 Cable Installation 3.2.1 Power Cable The NVIDIA Skyway appliance is shipped with two power supply units. Each supply unit has a separate AC receptacle. The appliance accepts voltages of 100-127 VAC and 200-240 VAC for all possible power supply units. The power cords should be a standard 3-wire AC power cards, including a safety ground, and rated for 15A or higher. - Page 20 3.2.2.1 Identifying Ethernet and InfiniBand/VPI Ports Networking Cable Installation All cables can be inserted or removed with the unit powered on. To insert a cable and press the connector into the port receptacle until the connector is firmly seated. Support the weight of the cable before connecting the cable to the adapter card. Do this by using a cable holder or tying the cable to the rack.
-
Page 21: System Connectivity
The LED indicator for that port will turn off when the cable is unseated. For full cabling guidelines, ask your NVIDIA Networking representative for a copy of NVIDIA Cable Management Guidelines and FAQs Application Note. - Page 22 If no obstacles were found and the problem persists, call your NVIDIA Networking representative for assistance.
-
Page 23: Configuring The Gateway For The First Time
MAC address. Connect a VGA monitor and USB keyboard directly to the NVIDIA Skyway appliance. To enter the BIOS, reboot the NVIDIA Skyway appliance and press <DEL> during bootup until the BIOS window pops up. Go to “Server Mgmt.” tab → “BMC network configuration."... - Page 24 Use the following IPMI command to remote access serial console (user and password should be “admin” by default). ipmitool -I lanplus -H <IPMI_CONTROLLER_IP> -U <user> -P <password> sol activate The command should be run on a Linux console with the “ipmitool” application installed.
- Page 25 At this point, make sure to disconnect the VGA monitor and USB keyboard, or else the following error may appear: TSC_DEADLINE disabled due to Errata; Please update microcode to version : 0xffffffff or later Log in as admin and use admin as password, using IPMI. ipmitool -I lanplus -H <IP Address>...
- Page 26 Wizard Session Display (Example) Comments Step 2: Use DHCP on mgmt0 interface? [yes] Perform this step to obtain an IP address for the gateway (mgmt0 is the management port of the gateway). • Typing “yes” will have the DHCP server assign the IP address •...
-
Page 27: Rerunning The Wizard
Wizard Session Display (Example) Comments You have entered the following information: The wizard displays a summary of choices Hostname: <gateway name> and then asks to confirm the choices or to Use DHCP on mgmt0 interface: yes re-edit them. Enable IPv6: yes •... -
Page 28: Starting The Command Line Interface (Cli)
-l <username> <ip address> Log in to the gateway (default username and password are both "admin"). Read and accept the EULA, when prompted. Once the following prompt appears, the system is ready to use. NVIDIA Gateway Password: Last login: <time> from <ip-address> gateway >... -
Page 29: Networkwide Deployment Guidelines
5 Networkwide Deployment Guidelines Unable to render include or excerpt-include. Could not retrieve page. 5.1 Configuring High Availability (HA) This section explains how to configure a HA cluster with multiple appliances. 5.1.1 Before Configuring HA • For all appliances in the HA cluster, the MLNX-GW version must the same. •... -
Page 30: Configuring Ha On Skyway Appliance
eth_router > enable eth_router # configure terminal eth_router (config) # protocol mlag eth_router (config) # lacp eth_router (config) # vlan eth_router (config vlan 999) # exit eth_router (config) # interface vlan ip address 192.17.10.3/24 primary eth_router (config) # interface port-channel eth_router (config interface port-channel 1) # exit... - Page 31 Type 'YES' to confirm the HA domain id change: YES After this step, the Skyway appliances will be rebooted. Once all systems complete the initialization, verify that all Skyway appliances were added properly to the HA cluster by running "show gw ha" from one of the Skyway appliances. Verify domain ID appears as configured and all Skyway appliances appear in the output of the command.
- Page 32 5.1.2.1 High Availability LAG/MLAG Setup...
-
Page 33: Configuring Partition Keys (Pkeys)
5.1.2.2 Skyway Connectivity to the Ethernet Using L2 Ethernet Switches In this above use case, every Skyway-facing port on the side of the L2 Ethernet switches should be configured as a router port. In addition, a private network should be established (in the example above, 3.3.0.0/16) between the router ports mentioned above and the Skyways Ethernet port channel. ... - Page 34 switch (config) # ib partition <partition name> member ALL type full Example: switch (config) # ib partition pkey_0x1 pkey switch (config) # ib partition pkey_0x1 ipoib switch (config) # ib partition pkey_0x1 member ALL type full Configure PKEY on the InfiniBand host. #### pkey_full = pkey_id (hex) + 8000 (hex)
-
Page 35: System Monitoring
6 System Monitoring 6.1 Front Panel Monitoring Components 6.1.1 Power-On LED There is one I/O LED (green) on the front panel to indicate if the system is powered. LED State Color Description Green System is turned on Blinking Green System is under S4 state Power off Rear Panel LEDs... -
Page 36: Lan Interfaces Leds
6.2.1 LAN Interfaces LEDs 6.2.1.1 LAN3/LAN4 Rear I/O LED Interface There are two I/O LEDs (green and amber) to indicate LAN link and activity. Left LED Right LED Description Green 10M bps linked Blinking Green 10M bps active Amber Green 100M bps linked Amber Blinking Green... -
Page 37: Network Interface Cards Leds
LED State Color Description Blinking Amber Power supply warning event Blinking Green AC present standy output on Amber AC unplug to this module or power supply critical event Green Power supply DC output ON and OK No AC power to both power modules 6.2.3 Network Interface Cards LEDs There are two I/O LEDs per port: ... - Page 38 LED Color and Description State Blinking green Indicates a valid logical link with active traffic.
-
Page 39: System Maintenance
7 System Maintenance This chapter contains the installations and un-installation instructions of the following customer replaceable units. 7.1 Power Supply Units Skyway is equipped with two replaceable power supply units (PSU) that work in a redundant configuration. The figure below shows the power side of the system which includes a hot-swap PSU. Item Description Power Socket... -
Page 40: Slide Rail Kit
Do not run the appliance with openings due to missing parts. This may cause overheating due to improper airflow. Step 2. Insert the PSU by sliding it into the opening, until a slight resistance is felt. Step 3. Continue pressing the PSU until it seats completely. The latch will snap into place, confirming proper installation. -
Page 41: Troubleshooting
The Activity LEDs do not come on Check if the NVIDIA Skyway appliance has been started. The appliance is off Press the Power Button w/Integrated LED If that does not work, do the following: Unplug the appliance. -
Page 42: Technical Specifications
Dimensions (W x H x D): 438 x 88 x 760 (17.24" x 3.46" x 29.92") Physical Weight: • NVIDIA Skyway gateway: 21kg • NVIDIA Skyway gateway with ACC and package: 32kg Mounting: 19” rack mount InfiniBand: IBTA v1.3 Protocol Support Auto-Negotiation: SDR (2.5Gb/s per lane), DDR (5Gb/s per lane), EDR (25Gb/s per lane) -
Page 43: System Dimensions
All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. 9.3 Thermal Threshold Definitions There are two thermal threshold definitions for NVIDIA Skyway which impact the overall system operation state: Critical—When the device crosses this temperature, the firmware will automatically shut down the device. - Page 44 2. Emergency—The temperature threshold is set by the CPU's internal thermal trip. It is impossible to change the temperature value through a software interface.
-
Page 45: Inventory Information
10 Inventory Information The system’s inventory parameters (such as the serial number or part number) can be extracted from labels on the system's bottom side. -
Page 46: Field Replaceable Units
11 Field Replaceable Units Ordering Number Part Description MGA100-PS Power supply for NVIDIA Skyway Infiniband-to-Ethernet appliance MGA100-RKIT Rail kit for NVIDIA Skyway Infiniband-to-Ethernet appliance... -
Page 47: Revision History
12 Revision History Date Description of Changes Jun. 2023 Added the following sections: • Configuring High Availability (HA) • Configuring Partition Keys (PKEYs) Nov. 2022 Updated Hardware Installation Jun. 2022 Added partition PKeys to the Networkwide Deployment Guidelines Nov. 2021 Updated supported protocols across the document. Jun. - Page 48 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
- Page 49 Technologies Ltd. in the U.S. and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated. Copyright © 2024 NVIDIA Corporation & affiliates. All Rights Reserved.
Need help?
Do you have a question about the Skyway GA100 and is the answer not in the manual?
Questions and answers