Related Manuals for Nvidia Spectrum-3 SN4000 Series
Summary of Contents for Nvidia Spectrum-3 SN4000 Series
- Page 1 NVIDIA Spectrum-3 SN4000 1U and 2U Switch Systems Hardware User Manual NVIDIA Spectrum-3 SN4000 1U and 2U Switch Systems Hardware User Manual, SN4700, SN4600C, SN4600, SN4410 Exported on Sep/19/2022 02:33 PM...
-
Page 2: Table Of Contents
Splitter (Breakout) Cables and Adapters ........... 33 Initial Power On ..................36 System Bring-Up..................38 Configuring Network Attributes Using NVIDIA Onyx (MLNX-OS) ......38 Configuring Network Attributes Using Cumulus Linux ........41 FRU Replacements ................. 41 Power Supplies ................. 42 Fans .................... - Page 3 Unit Identification LED................ 51 Port LEDs ..................52 Inventory Information ................55 Software Management ...............57 Software Upgrade.................. 57 NVIDIA Onyx (MLNX-OS) Software Upgrade..........57 Switch Firmware Update..............57 Cumulus Linux Software Upgrade ............57 Troubleshooting ................58 Specifications..................60 SN4410 Specifications ................60 SN4600/SN4600C Specifications ..............
- Page 4 This document is preliminary and subject to change. Relevant for Models: SN4410, SN4600/SN4600C and SN4700 About this Manual This manual describes the installation and basic use of NVIDIA Ethernet switches based on the NVIDIA® Spectrum®-3 ASIC. Ordering Information See Ordering Information.
-
Page 5: Ordering Information
Cumulus Linux, 24 QSFP-DD28 and 8 QSFP-DD ports, 2 Power Supplies (AC), x86 CPU, standard depth, P2C airflow, Rail Kit 920-9N312-00F MSN4410- NVIDIA Spectrum-3 based 400GbE 1U Open Ethernet Switch with ONIE, B-0N0 WS2FO 24 QSFP-DD28 and 8 QSFP-DD ports, 2 Power Supplies (AC), x86 CPU,... -
Page 6: Sn4700 Ordering Part Numbers
Part Model Number 920-9N302-00F MSN4600- A-0N0 VS2FO NVIDIA Spectrum-3 based 200GbE 2U Open Ethernet Switch with ONIE, 64 QSFP56 ports, 2 Power Supplies (AC), x86 CPU, standard depth, P2C airflow, Rail Kit 920-9N302-00F SN4600C 7-0X0 MSN4600- NVIDIA Spectrum-3 based 100GbE 2U Open Ethernet Switch with... - Page 7 Part Model Number 920-9N301-00F MSN4700- B-0C0 WS2FC NVIDIA Spectrum-3 based 400GbE 1U Open Ethernet Switch with Cumulus Linux, 32 QSFPDD ports, 2 Power Supplies (AC), x86 CPU, standard depth, P2C airflow, Rail Kit 920-9N301-00 MSN4700- RB-0C0 WS2RC NVIDIA Spectrum-3 based 400GbE 1U Open Ethernet Switch with...
-
Page 8: Introduction
This document is preliminary and subject to change. The SN4000 series switches are the 4th generation of NVIDIA Spectrum® switches, purpose-built for leaf/spine/super-spine datacenter applications. Allowing maximum flexibility, SN4000 series provides port speeds spanning from 1GbE to 400GbE, and a port density that enables full rack connectivity to any server at any speed. -
Page 9: Speed And Switching Capabilities
SN4600 Front View SN4600C Front View SN4600/SN4600C Rear View SN4700 Front View SN4410/SN4700 Rear View Speed and Switching Capabilities The table below describes maximum throughput and interface speed per system model. -
Page 10: Management Interfaces, Psus And Fans
System Interfaces Supported Rates Max Throughput Model SN4410 32 x QSFP-DD 8Tb/s 8 x 400GbE 16 x 40/100/200GbE 32 x 1/10/25/50GbE SN4600 64 x QSFP56 25.6Tb/s 64 x 40/200GbE 128 x 1/10/25/50/100GbE SN4600C 12.8Tb/s 64 x QSF28 64 x 40/100GbE 128 x 1/10/25/50/100GbE ... -
Page 11: Certifications
Certifications The list of certifications (such as EMC, Safety and others) per system for different regions of the world is located on the NVIDIA website at http://www.mellanox.com/page/ environmental_compliance. -
Page 12: Installation
Unless otherwise specified, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operation environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level. -
Page 13: Safety Warnings
Safety Warnings. Note that some warnings may not apply to all models. Air Flow NVIDIA systems are offered with two air flow patterns: the images are provided for illustration purposes only. The design may slightly vary in different systems •... -
Page 14: Package Contents
All FRU components need to have the same air flow direction. A mismatch in the air flow will affect the heat dissipation. The table below provides an air flow color legend and respective OPN designation. Direction Description and OPN Designation Connector side inlet to power side outlet. -
Page 15: Mounting Options
If anything is damaged or missing, contact your sales representative at Networking- support@nvidia.com. Mounting Options By default, the systems are sold with fixed rail-kits. Telescopic Rail-kits are available for some systems, and should be purchased separately. For installation instructions, refer to the relevant links in the following table: ... - Page 16 Prerequisites: Before mounting the system to the rack, select the way you wish to place the system. Pay attention to the airflow within the rack cooling, connector and cabling options. While planning how to place the system, consider the two installation options shown in the figures below, and review the following points: •...
- Page 17 • Rear side (FRUs) 2. Front side (ports) Short Racks (430-580 mm) Installation - Side View Front side (ports): Rear side (FRUs): Standard Racks (580-800 mm) Installation - Side View Front side (ports): Rear side (FRUs):...
- Page 18 To mount the system into the rack: The following steps include illustrations that show front side (ports) installation, yet all instructions apply to all installation options. Attach the left and right rack mount rails (A) to the switch, by gently pushing the switch chassis’...
- Page 19 Install 8 cage nuts (D) in the desired 1U slots of the rack: 4 cage nuts in the non-extractable side and 4 cage nuts in the extractable side. Installing the Cage Nuts While each rack U (unit) consists of three holes, the cage nut should be installed vertically with its ears engaging the top and bottom holes only.
- Page 20 Slide the switch with the rails (A) and ears (C) installed on it into the left and right rails (B) on the rack. Use four M6 screws (E) to fix the rack mount ears (C) to the rack. Do not tighten the screws yet.
-
Page 21: Removing The System From The Rack
Removing the System from the Rack To remove a unit from the rack: Turn off the system and disconnect it from peripherals and from the electrical outlet. While your installation partner is supporting the system’s weight: Loosen the screws attaching the rack mount ears (C) to the rack. Do not remove them yet. Loosen the screws attaching the rack mount blades (B) to the rack, and pull the blades towards you, while your partner is holding the system. - Page 22 Rack Rail Kit Parts Prerequisites The rails must be separated prior to the installation procedure. To separate the rails: Separate rail C from sliders A/B + D. Extend the rail assembly by pulling the extension outwards (D).
- Page 23 Rails Separation Before mounting the system to the rack, select the way you wish to place the system. Pay attention to the airflow within the rack cooling, connector and cabling options. While planning how to place the system, review the following points: •...
- Page 24 If cable accommodation is required, disassemble any of the inner rails from the brackets attached to them, by removing and scraping the connecting screws. Disassembling the Inner Rails for Cable Accommodation Route the power cable through either of the inner rails, and reassemble the brackets by screwing the 3 screws (per rail) provided with the rail-kit (H) with a torque of 0.7±0.05Nm. ...
- Page 25 Securing the Chassis in the Inner Rails Slide the switch into the rack by carefully pushing the inner rails into the outer rails installed on the rack. Sliding the Switch into the Rack When fully inserted, fix the switch by closing the remaining 2 screws in the middle and tightening the 8 screws inserted in Step 2 with a torque of 4.5±0.5 Nm.
-
Page 26: Removing The System From The Rack
Removing the System from the Rack To remove a unit from the rack: Turn off the system and disconnect it from peripherals and from the electrical outlet. Unscrew the two M6 screws securing the front of the inner rails’ ears to the outer rails and to the rack. ... -
Page 27: Sn4410/Sn4700 Fixed Rail Kit
Locking Mechanism SN4410/SN4700 Fixed Rail Kit This document is preliminary and subject to change. Kit OPN Legacy OPN Rack Size and Rack Depth Range 930-9NRKT-00JN-000 MTEF-KIT-J 430-800 mm The following parts are included in the fixed rail kit (see figure below): •... - Page 28 Prerequisites: Before mounting the system to the rack, select the way you wish to place the system. Pay attention to the airflow within the rack cooling, connector and cabling options. While planning how to place the system, consider the two installation options shown in the figures below, and review the following points: •...
- Page 29 Rear side (FRUs): 2. Front side (ports): Standard Racks (580-800 mm) Installation Options Rear side (FRUs): 2. Front side (ports): In short racks, the system’s ventilation openings should be framed by the designated windows in the rails, as shown below. Short Racks (430-580 mm) Installation - Side View Front side (ports): Rear side (FRUs):...
- Page 30 Standard Racks (580-800 mm) Installation - Side View Front side (ports): Rear side (FRUs): To mount the system into the rack: At least two people are required to safely mount the system in the rack. The following steps include illustrations that show front side (ports) installation, yet all instructions apply to all installation options.
- Page 31 Attaching the Brackets to the Chassis Install 8 cage nuts (D) in the desired 1U slots of the rack: 4 cage nuts in the non-extractable side and 4 cage nuts in the extractable side. Installing the Cage Nuts While each rack U (unit) consists of three holes, the cage nut should be installed vertically with its ears engaging the top and bottom holes only.
- Page 32 Slide the switch with the rails (A) and ears (C) installed on it into the left and right rails (B) on the rack. Use four M6 screws (E) to fix the rack mount ears (C) to the rack. Do not tighten the screws yet.
-
Page 33: Cable Installation
To remove a cable, disengage the locks and slowly pull the connector away from the port receptacle. The LED indicator for that port will turn off when the cable is unseated. For full cabling guidelines, ask your NVIDIA representative for a copy of NVIDIA Cable Management Guidelines and FAQs Application Note. - Page 34 Using Splitter (Breakout) Cables with NVIDIA Onyx (MLNX-OS) When using this feature, you should log into the NVIDIA Onyx (MLNX-OS) CLI and configure the individual ports to be ‘split-2’ or ‘split-4’. For further information on NVIDIA’s cable, visit http:// www.mellanox.com/page/interconnect_overview.
- Page 35 SN4410 Splitting Options...
-
Page 36: Initial Power On
SN4600/SN4600C Splitting Options SN4700 Splitting Options Initial Power On Each system’s input voltage is specified in the Specifications chapter. The power cords should be standard 3-wire AC power cords including a safety ground and rated for 15A or higher. - Page 37 If the Fan Status LED is not green, unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and/or obstacles. If no obstacles were found and the problem persists, call your NVIDIA representative for assistance.
-
Page 38: System Bring-Up
Serial Terminal Program Configuration Setting Parameter Baud Rate 115200 Data bits Stop bits Parity None Flow Control None Step 3. Login as admin and use admin as password. On the first login, the NVIDIA Onyx (MLNX-OS) configuration wizard will start. - Page 39 Configuration Wizard Session Comments Wizard Session Display You must perform this configuration the first time you NVIDIA configuration wizard operate the system or after resetting the system. Type ‘y’ and then press <Enter>. Do you want to use the wizard for initial...
- Page 40 Step 6. Check the software version embedded in your system, using the command ‘show version’. Compare this version to the latest version that can be retrieved from NVIDIA support site. To upgrade software, please refer to the NVIDIA Onyx (MLNX-OS) User Manual.
-
Page 41: Configuring Network Attributes Using Cumulus Linux
Login as admin (default username is admin, password is admin). Once you get the CLI prompt, you are ready to use the system. For additional information about NVIDIA Onyx (MLNX-OS), refer to the NVIDIA Onyx (MLNX-OS) User Manual located on the NVIDIA Networking Documentation Website. -
Page 42: Power Supplies
Power Supplies NVIDIA systems that are equipped with two replaceable power supply units work in a redundant configuration. Either unit may be extracted without bringing down the system. Make sure that the power supply unit that you are NOT replacing is showing green for the power supply unit LED. -
Page 43: Fans
Make sure that the board connector is on the lower side of the FRU module, and Insert the unit by sliding it into the opening, until a slight resistance is felt. Continue pressing the power supply unit until it seats completely. The latch will snap into place, confirming the proper installation. - Page 44 Do not remove a fan unit (faulty or operational), and leave its slot open while the system is running. Grasping the handle with your right hand, push the latch release with your thumb while pulling the handle outward. As the fan unit unseats, the fan unit status LEDs will turn off. Remove the fan unit.
-
Page 45: Interfaces
Switching Capabilities. As detailed in the following table, for additional data interfaces, each QSFP28/56/DD port can be connected with a QSFP28/56/DD cable or connector through NVIDIA QSFP28/56/DD to SFP (Dynamix QSA™) adapters, hybrid or split cables*. Model Family... - Page 46 Model Family Ports Maximum Speed 40GbE 50GbE 25GbE 10GbE 1GbE SN4700 400GbE 200GbE 40GbE 100GbE 50GbE 25GbE 10GbE 1GbE *In the SN4600, SN4700 and SN4410 systems, when interconnecting Switch to Switch and Switch to NIC in 50GbE, 100GbE 200GbE and 400GbE based PAM4 speeds, the supported length of PAM4 passive copper and breakout cables is up to 2.5m. ...
-
Page 47: Rs232 (Console)
Make sure you use only FCC compliant Ethernet cables. The USB interface is USB 2.0 compliant and can be used by NVIDIA Onyx software to connect to an external disk for software upgrade or file management. The connector complies with the USB 3.0 type A standard. -
Page 48: Status And Port Leds
When using an NVIDIA Onyx (MLNX-OS) based system, keeping the reset button pressed for more than 15 seconds will reset the system and the “admin” password, this should allow you to login without a password and set a new password for the “admin” user. -
Page 49: Fan Status Led
*The figure is provided for Illustration purposes only. The design may slightly vary in different systems. It may take up to five minutes to turn on the system. If the System Status LED shows amber after five minutes, unplug the system and call your NVIDIA representative for assistance. System Status LED Assignments LED Behavior... -
Page 50: Power Supply Status Leds
LED Behavior Description Action Required Solid Green All fans are up and running. Solid Amber Error, one or more fans are not operating The faulty FRUs should be replaced. properly. Fan Status Rear LED Assignments (One LED per Fan) LED Behavior Description Action Required Solid Green... -
Page 51: Unit Identification Led
Solid Green The PSU is running normally. Flashing Refer to Troubleshooting. For further assistance, Green 1Hz AC Present, Standby – On, Main Output - call your NVIDIA representative. Flashing PSU warning - events where the PSU Amber 1Hz continues to operate... -
Page 52: Port Leds
switch (config) # show leds Module LED Status -------------------------------------------------------------------------- MGMT UID Blues To deactivate the UID LED on a switch system, run: switch (config) # led MGMT uid off Port LEDs SN4600/SN4600C Port LEDs SN4600/SN4600C SN4600/SN4600C Port LEDs Indications LED Behavior Description Action Required Link is down... - Page 53 SN4410/SN4700 Port LEDs Each QSFP module can be used as two 4X ports/four 2X ports/eight 1X ports. Each QSFP-DD has one dedicated bi-color LED. In order to provide link information for more than one port by using one LED, LED splitting control button is available. You may use the button to select between 8 indication states.
- Page 54 State State Indication QSFP Module LED Comments LEDs Indication [/1 /2 /3 /4] 8x/1xE 8x/1xF 8x/1xG 8x/1xH The port LED behavior indicates the port state, as follows: SN4410/SN4700 Port LEDs Indications Description Action Behavior Required No 8x/4x/2x/1x link was established on this QSFP module Solid Green At list one link was established: 8x/4x/2xA/2xB/1xA/1xB/1xC/1xD/ 1xE/1xF/1xG/1xH...
-
Page 55: Inventory Information
Description Action Behavior Required Flashing Link is up with traffic Green Flashing A problem with the link Refer Amber to Troubleshooting Inventory Information The system’s inventory parameters (such as serial number, part number and GUID address) can be extracted from the inventory pull-out tab on the lower left side of the rear panel. Pull-out Tab SN4600/SN4600C: SN4410/SN4700:... -
Page 57: Software Management
Software and firmware updates are available from the NVIDIA Support website. Check that your current revision is the latest one available on the NVIDIA Support website. If you do not have the latest revision, upgrade your software using the CLI or the GUI. Copy the updated software to a known location on a remote server within the user’s LAN. -
Page 58: Troubleshooting
Cause and Solution Indicator LEDs System Status LED is Cause: NVIDIA Onyx (MLNX-OS) software did not boot properly and blinking for more only firmware is running. than 5 minutes Solution: Connect to the system via the console port, and check the software status. - Page 59 Problem Symptoms Cause and Solution Indicator System boot failure Software upgrade while using failed on x86 based Monitoring and Troubleshooting Cumulus Linux User Guide. Cumulus Linux systems...
-
Page 60: Specifications
Global Power Consumption: Typical power with passive cables (ATIS): 460W Main CPU: Intel x86 2.20GHz Quad Core Devices PCIe: 4x Gen 3.0 Switch: NVIDIA Spectrum®-3 Memory: 16GB RAM , 60GB SSD Throughput 8Tb/s... -
Page 61: Sn4600/Sn4600C Specifications
Typical power with passive cables (ATIS): 466W Main CPU: Intel x86 2.20GHz Quad Core Devices PCIe: 4x Gen 3.0 Switch: NVIDIA Spectrum®-3 Memory: SN4600 - 16GB RAM, 60GB SSD SN4600C - 8GB RAM, 30GB SSD Throughput SN4600 - 25.6Tb/s SN4600C - 12.8Tb/s... - Page 62 Global Power Consumption: Typical power with passive cables (ATIS): 630W Main Devices CPU: Intel x86 2.20GHz Quad Core PCIe: 4x Gen 3.0 Switch: NVIDIA Spectrum®-3 Memory: 16GB RAM , 60GB SSD Throughput 25.6Tb/s...
-
Page 63: Appendixes
NVIDIA fan module, 80 x 80 [mm], P2C SN4600 T-000 I Airflow, For SN4600 switch 930-9NFAN-00 MTEF-FANR-I NVIDIA fan module, 80 x 80 [mm], C2P SN4600 J7-000 Airflow, For SN4600 switch 930-9BFAN-00I MTEF-FANF- NVIDIA fan module, 40 x 40 [mm], P2C SN4700/SN4410 X-000 ... -
Page 64: Thermal Threshold Definitions
Networks Help Center. Interface Specifications Small Form Factors Specifications NVIDIA switch systems come in a flexible range of form factors - SFP/QSFP, SFP28/QSFP28, SFP56/ QSFP56 and SFP-DD/QSFP-DD. All form factors specification documents are available on the Storage Networking Industry Association (SNIA) Website. ... -
Page 65: Rj45 To Db9 Harness Pinout
RJ45 to DB9 Harness Pinout In order to connect a host PC to the Console RJ45 port of the system, a RS232 harness cable (DB9 to RJ45) is supplied. RJ45 to DB9 Harness Pinout Disassembly and Disposal Disassembly Procedure To disassemble the system from the rack: Unplug and remove all connectors. -
Page 66: Disposal
7. Remove the caged nuts. Disposal According to the WEEE Directive 2002/96/EC, all waste electrical and electronic equipment (EEE) should be collected separately and not disposed of with regular household waste. Dispose of this product and all of its parts in a responsible and environmentally friendly way. Follow the instructions found at http://www.mellanox.com/page/dismantling_procedures proper disassembly and disposal of the switch, according to the WEEE directive. -
Page 67: Document Revision History
Document Revision History Date Revisi Description July 24, 2022 Updated OPNs in: • Ordering Information • Installation • Accessory and Replacement Parts Updated Interface Specifications February 3, 2022 Updated Interfaces December 16, 2021 Updated LED Notifications November 1, 2021 Added SN4410 to the User Manual. August 10, 2021 Updated: •... - Page 68 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
- Page 69 Copyright © 2022 NVIDIA Corporation & affiliates. All Rights Reserved.
Need help?
Do you have a question about the Spectrum-3 SN4000 Series and is the answer not in the manual?
Questions and answers