Subscribe to Our Youtube Channel
Related Manuals for Mellanox Technologies MT51336-EN-EVB
Summary of Contents for Mellanox Technologies MT51336-EN-EVB
- Page 1 SwitchX Evaluation Board Hardware User Manual PN: MT51336-EN-EVB, MT51336-VPI-EVB Rev 1.0 www.mellanox.com...
- Page 2 THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PRO- VIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUS- TOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
-
Page 3: Table Of Contents
6.1 Chassis Management Using the MLNX-OS Stack 6.2 Network Management and Clustering Software 6.3 Chassis Management 6.3.1 Open SM 6.4 Fabric Management with UFM 6.5 Configuring the Switch for the First Time Chapter 7 Connecting to the Switch Platform Mellanox Technologies... - Page 4 7.1 Starting an SSH Connection to the Switch (CLI) 7.2 Starting a WebUI Connection to the Switch 7.2.1 Managing the System 7.2.2 I2C Connector 7.2.3 How to Get Mellanox Firmware Tools (MFT) 7.3 Updating Firmware Chapter 8 Troubleshooting Mellanox Technologies...
- Page 5 Figure 30: Unscrew the Management Board Figure 31: Remove the Management Board Figure 32: Holes for the Extender Board Interfaces Figure 33: Install the Extender Board Figure 34: Connecting the EVB to the Server Side Board Figure 35: EVB Board Connections View 1 Mellanox Technologies...
- Page 6 Rev 1.0 Figure 36: Reset Button Figure 37: Management Interfaces Figure 38: Transfer Power Cord Figure 39: Install the Power Cord Clip Figure 40: Transfer Power Cord Finished Figure 41: Connector Pinout Mellanox Technologies...
- Page 7 Configuration Wizard Session - IP Configuration by DHCP Table 10: Configuration Wizard Session - IP Zeroconf Configuration Table 11: Configuration Wizard Session - Static IP Configuration Table 12: Adapter Cards Certification Status Table 13: Specifications for MT51336-NUDT Table 14: Connector Pinout Mellanox Technologies...
-
Page 8: Revision
Added the word “Hardware” to the name. Added correct graphics of the server side card Added tie down the cables Added Caution Note do not use as a shelf. fixed graphics More fixes Added Initial configuration section Preliminary Release Mellanox Technologies... -
Page 9: Chapter 1 Overview
• Initial Ethernet and IB switch testing for L2/L2+ This EVB can be used both while rack mounted (standard 19” rack) and in a tabletop setup(for Signal Integrity evaluation). Figure 1: Connector and Power Side Panels MT51336-EN-EVB CONSOLE Mellanox Management CPU The EVB comes with a Power PC 460E CPU for management. -
Page 10: Pcie/1Gbe Extender Kit
EVB to the Server. One cable is for clocking and reset and two cables are for data TX and RX. 1.2.2 1GbE Extender Kit The PCIE extender allows for remote management through the 10/100/1000Mb RJ45 Ethernet connection on the EVB Extender side board. This is connected to any standard Ethernet port fabric management port. Mellanox Technologies... -
Page 11: Compliance Ports
Ports 1 and 2 are SMAports and ports 35 and36 are mini- SMP ports. These reference traces are designed to have identical insertion loss as the corresponding compli- ance ports, and those traces are routed in the same geometry and length as the traces of the compli- ance ports. Mellanox Technologies... -
Page 12: Features
• Upload/download configuration • System alarms • VLAN aware • ETS • Flow Control • 36 FDR (56Gb/s) ports in a 1U switch • IBTA 1.3 and 1.21 complaint • SDR/DDR/QDR/FDR10/FDR link speed • Congestion control • Adaptive routing Mellanox Technologies... -
Page 13: Serial Number And Product Version Information
The Serial number and GUID for the switch and the MAC for the Management PC are found on the pull out tab below the USB interface connection. Figure 2: Pull Out Tab S/N: MT1117X00014 P/N: MT51336-EN-EVB Rev:A1 MAC: 0002C927051C Made in IL... -
Page 14: Chapter 2 40 Gb/S Ethernet
Up to 24 QSFP ports are capable of 1/2/4/8G Fibre Channel with a QSA. All ports are capable of virtual switching (VEPA+). All usable virtual switching ports are grouped together. The port protocol can only be obtained from the management interface. Mellanox Technologies... -
Page 15: Chapter 3 Fdr
FDR10 supports 20% more bandwidth over regular QDR using the same QSFP cables/connectors. Both FDR and FDR10 support Forward Error Correction (FEC), as described in IEEE 802.3ap chapter 74. FDR10 is only guaranteed to work with approved Mellanox ConnectX-3 adapters. Mellanox Technologies... -
Page 16: Chapter 4 Basic Operation And Installation
Switch Platform Hardware Overview Figure 4 shows the power side panel and connector side panel views of the switch EVB. Figure 3: Connector and Power Side Panel MT51336-EN-EVB CONSOLE The figure shows port configurations for the switch systems, including: • 2 – Ethernet RJ45 connectors •... -
Page 17: Status Leds
For more information See “Troubleshooting” on page 64. If the PSU LED is not green, this indicates a problem with the power supply. The switch is opera- tional only if the PSU LED is green. Mellanox Technologies... -
Page 18: Figure 5: Power, Fan, And System Leds
Red — Major fault 4.1.1.3 Port Connector LED Assignment Above the ports are two LEDs one for the upper port and one for the lower port . The fol- lowing table shows the port status according to the LED indication. Mellanox Technologies... -
Page 19: Figure 7: Power Side Panel
The power supply unit (PSU) is located on the right side of the power side panel. The PSU has a single 2 color LED on the right side of the PSU, that indicates the internal status of the unit. Mellanox Technologies... -
Page 20: Figure 8: Psu Status Leds
Figure 9: Fan Status LED Connector Side This is a two color LED Green — OK Red — Major fault Off — No power to the system The LED indicator on the Fan Module is on the right side of the module. Mellanox Technologies... -
Page 21: Figure 10: Fan Status Led Power Side
For more information See “Troubleshooting” on page 64. 4.1.1.6 Bad Port LED The Bad Port indicator is located on the left side of the connector side panel of the unit. The fol- lowing Bad Port conditions are possible: Mellanox Technologies... -
Page 22: Reset Button
UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future. For details please contact Mellanox Technologies support. 4.1.2 Reset Button On the connector side panel under the system LEDs is a reset button. This reset button requires a tool to be pressed, a paper clip will do. -
Page 23: Qsfp Cable Power Budget Classification
Reference Traces 1,2 One Trace and compliance pair 1, 3 One Trace and compliance pair 33, 35 One Trace and compliance pair 2, 4 One Trace and compliance pair 34, 36 Compliance Ports 3,4 Compliance Ports 33, 34 Mellanox Technologies... -
Page 24: Management And Firmware Updating Interfaces
• 1 connector that is labelled “CONSOLE”. Use this connector to connect to the host PC. There is one I2C interface to connect to the SwitchX MT51336 on the power side, but this is for Mellanox FAEs and advances users only. Figure 14: Management Interfaces MT51336-EN-EVB CONSOLE 4.4.2.1 RJ-45 Connector (CONSOLE) The port labelled “CONSOLE”... - Page 25 4.4.2.4 I2C Interface There is an I2C connector (banana pin type connector) on the far left of the power side of the switch. This con- This interface is for Debug and Troubleshooting only by a certified Mellanox FAE. Mellanox Technologies...
-
Page 26: Switch Platform Installation And Operation
8cm (3 inches) of clearance around the ventilation openings. 3. Stacking the Chassis The chassis should not be stacked on any other equipment. If the chassis falls, it can cause bodily injury and equipment damage. Mellanox Technologies... - Page 27 9. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 10. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Mellanox Technologies...
-
Page 28: Mechanical Installation
There are two possible air flow directions. Be sure that the switch air flow direction is compatible with your system, rack, and PSU. The connector side of the switch has the data ports, system LEDs, and management connection ports. The switch platform contains auto-sensing 100 - 240 VAC connections for the PSU. Mellanox Technologies... -
Page 29: Package Contents And Installation
If anything is damaged or missing, contact your customer representative immediately. 4.6.1 Installing the Switch in the Rack Tools and Customer Supplied Parts • Phillips Screwdrivers #1 and #2 • grounding screw • ESD strap • grounding wire sufficient to reach a valid ground. Mellanox Technologies... -
Page 30: Figure 15: Rack Rail Kit Parts
To Transfer the power cord from the connector side to the power side through the rail kit see Section B, “Transferring the Power Cord,” on page 71. Things to consider before choosing where to mount the rails and rail slides. Mellanox Technologies... -
Page 31: Figure 16: Mounting Options
The installation kit allows for a 2” recess of the switch past the vertical support. Figure 16: Mounting Options Vertical Support Connector side Power side Rack door Connector side Power side Power side Connector side Rack door Power side Connector side Vertical Support Decide which mounting option you want to use. Mellanox Technologies... -
Page 32: Figure 17: Screwing On The Rail
Clip 4 more caged nuts into the holes on the opposite side of the rack Check that both sides of the switch, left and right, are the same level in the rack. Slide the rail into the rail slide. Mellanox Technologies... -
Page 33: Grounding The Switch
If so, connect a ground wire to one of the casing screws and connect the other end to a valid ground. If you choose to not use the ground screw, make sure that the rack is properly Mellanox Technologies... -
Page 34: Power Connections And Initial Power On
Power supply unit has directional air flow similar to the fan module. The Fan module airflow must coincide with the airflow of the PSU. If the PSU airflow direction is dif- ferent from the fan module airflow direction the system will shut down. Mellanox Technologies... -
Page 35: Figure 20: Power Supply Unit Extraction
Insert the power cord into the supply connector. Insert the other end of the power cord into an outlet of the correct voltage. The green PSU indicator should light. If not, repeat the whole procedure to extract the PSU and re-insert it. Mellanox Technologies... -
Page 36: Cable Installation
This switch can operate indefinitely with one of the three fans in the fan module inoperable so long as the ambient temperature is below 45° Celsius. Operation without a fan unit should not exceed two minutes. During fan hot-swap, if both indicators are OFF then the fan unit is disconnected. Mellanox Technologies... -
Page 37: Figure 23: Air Flow Labels
The green fan status indicator should light. If not, extract the fan unit and reinsert it. After two unsuccessful attempts to install the fan unit, power off the switch before attempting any system debug. Mellanox Technologies... -
Page 38: Chapter 5 Installing The Pcie/1Gbe Server Side Board
Ethernet connection. The interfaces are: Interface For PCIE clocking and reset of the PCI Express QSFP for connection to the server side board QSFP for connection to the server side board Interface For Ethernet RJ45 for remote Ethernet connection Mellanox Technologies... -
Page 39: Figure 25: Server Side Card
CX4 connector connection PCIE SERVER TX PCIE SERVER RX The PCB server board has 3 interfaces: Interface For PCIE clocking and reset QSFP for connection to the server side board QSFP for connection to the server side board Mellanox Technologies... -
Page 40: Removing The Top Cover
ESD strap is touching your skin and that the other end is connected to a verified ground. Remove the 6 screws from the power side of the top. Slide the top towards the power side. With the cover off you will need a clip on fan cooling the chip heat sink. Mellanox Technologies... -
Page 41: Figure 27: Screw Locations
SwitchX Switch EVB Hardware User Manual Rev 1.0 Figure 27: Screw Locations Figure 28: Slide the Top Mellanox Technologies... -
Page 42: Removing The Management Cpu
Rev 1.0 Installing the PCIE/1GbE Server Side board Removing the Management CPU Figure 29: Management Board Installed Figure 30: Unscrew the Management Board Mellanox Technologies... -
Page 43: Installing The Pcie/1Gbe Extender
ESD strap is touching your skin and that the other end is connected to a verified ground. Put the connectors for the extender daughter board through the holes in the frame. Figure 32: Holes for the Extender Board Interfaces Set the connectors through these holes. Mellanox Technologies... -
Page 44: Figure 33: Install The Extender Board
Select only one of these two options. Either use the RJ45 Ethernet connection option or the PCIE option. DO NOT TRY AND USE BOTH OPTIONS AT THE SAME TIME! When running the SwitchX EVB without the top cover use portable fans to keep the heat sinks cool. Mellanox Technologies... -
Page 45: Installing The Server Side Board
Connect the QSFP cables from the daughter board to the server board. connect green to green and blue to blue. Connect the CX4 cable from the daughter board to the server board. Tie down the cables using the cable ties in the EVB. Mellanox Technologies... -
Page 46: Board Connectors Numbering And Locations
Rev 1.0 Installing the PCIE/1GbE Server Side board Board Connectors Numbering and Locations Figure 35: EVB Board Connections View 1 Mellanox Technologies... -
Page 47: Table 7: Port Connector Numbering
Connects to HW port 27 on SwitchX Mini SMP Data Connects to HW port 27 on SwitchX Mini SMP Data Connects to HW port 27 on SwitchX Mini SMP Data Connects to HW port 27 on SwitchX 12V PS1 Mellanox Technologies... - Page 48 Connects to Port 36, Ref to compliance port 34 Mini SMP Reference Connects to Port 36, Ref to compliance port 34 Mini SMP Reference Connects to Port 35, Ref to compliance port 33 Mini SMP Reference Connects to Port 35, Ref to compliance port 33 Mellanox Technologies...
-
Page 49: Switch Shut Down Procedure
To restart the switch you must physically go to the switch and unplug and replug in all of the power cords to the chassis. Disassembly of the Switch from the Rack To disassemble the switch from the rack: Unplug and remove all connectors. Mellanox Technologies... -
Page 50: Disposal
15 seconds the switch is reset and the password is deleted. The LEDs will flicker for a fraction of a second to indicate reset is occurring. You will then be able to enter without a password and make a new password for the user admin. Mellanox Technologies... -
Page 51: Chapter 6 Management And Tools Overview
The manager also has the ability to burn new firmware on the switch. Network Management and Clustering Software Download and install, on all nodes, the Mellanox OpenFabric software package for Linux, Win- dows, or other operating systems from the Mellanox software website: Mellanox Technologies... -
Page 52: Chassis Management
LIDs. A typical installation using the OFED package will run the OpenSM subnet manager at system start up after the OpenIB drivers are loaded. This automatic OpenSM is resident in memory, and Mellanox Technologies... -
Page 53: Fabric Management With Ufm
• Advanced Fabric-wide Real-time and History Monitoring • Quality of Service • Traffic Aware Routing Algorithm (TARA) • Central Device Management • Task automation • Threshold based alerts • Fabric segmentation/isolation • Subnet Manager Configuration • Logging • High-Availability Mellanox Technologies... -
Page 54: Configuring The Switch For The First Time
Connect the host PC to the CONSOLE (RJ-45) port of the switch system using the supplied cable. Step 1. The CONSOLE ports for the EVB is shown below. Figure 37: Management Interfaces MT51336-EN-EVB CONSOLE Make sure to connect to the CONSOLE RJ-45 port of the switch and not to the (Ether- net) MGT port. -
Page 55: Table 9: Configuration Wizard Session - Ip Configuration By Dhcp
Step 3: Admin password (Press <Enter> to leave To avoid illegal access to the machine, please type a unchanged)? <new_password> password and then press <Enter>. Then confirm the Step 4: Confirm admin password? <new_password> password by re-entering it. Note that password characters are not printed. Mellanox Technologies... - Page 56 Choice: <Enter> To run the command “configuration jump-start” you must be in Config mode. Configuration changes saved. To return to the wizard from the CLI, enter the “con- figuration jump-start” command from configuration mode. Launching CLI... <switch name> [> Mellanox Technologies...
-
Page 57: Table 10: Configuration Wizard Session - Ip Zeroconf Configuration
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI... switch-1 > Mellanox Technologies... -
Page 58: Table 11: Configuration Wizard Session - Static Ip Configuration
Before attempting a remote (for example, SSH) connection to the switch, check the mgmt0 inter- Step 5. face configuration. Specifically, verify the existence of an IP address. To check the current mgmt0 configuration, enter the following commands: sx-43 [standalone: master] > enable sx-43 [standalone: master] # show interfaces mgmt0 Mellanox Technologies... - Page 59 Comment: RX bytes: 1343502058 TX bytes: 313920869 RX packets: 17589211 TX packets: 992717 RX mcast packets: TX discards: RX discards: TX errors: RX errors: TX overruns: RX overruns: TX carrier: RX frame: TX collisions: TX queue len: 1000 Mellanox Technologies...
-
Page 60: Chapter 7 Connecting To The Switch Platform
Set up an Ethernet connection between the switch and a local network machine (“the remote Step 1 machine” henceforth) using a standard RJ-45 connector. Note: The Ethernet supported link speeds are 10/100 Mbps. Start a Web browse – Google Chrome, Microsoft Internet Explorer 7.0 or Mozilla Firefox 3.0. Step 2. Mellanox Technologies... - Page 61 BIP see “Box IP Centralized Location” section in the SW user manual. You will receive the login window for remote management of the switch. The following figure Step 4. shows an example. Note that the default username is admin. Mellanox Technologies...
-
Page 62: Managing The System
7.2.3 How to Get Mellanox Firmware Tools (MFT) Mellanox Firmware Tools (MFT) and documentation are available for download via http://www.mellanox.com > Downloads > Firmware Tools. The MFT kit includes: • mlxburn • flint • spark • debug utilities See “Related Documentation” on page 7. Mellanox Technologies... -
Page 63: Updating Firmware
Be sure to read and follow all of the instructions regarding the updating of the firmware on your switch system. Firmware for the HCA cards connected to this switch system can be downloaded from the same site. Mellanox Technologies... -
Page 64: Chapter 8 Troubleshooting
Make sure that the latest firmware version is installed on both the HCA and the switch. If media adapters are used, check that the all connections are good, tight, and secure. The activity LED does not come on: Check that the Subnet Manager has been started. Mellanox Technologies... - Page 65 EBC=100 MHz) Security/Kasumi support Bootstrap Option H - Boot ROM Location I2C (Addr 0x52) Internal PCI arbiter disabled 32 kB I-Cache 32 kB D-Cache Board: Mellanox PPC460EX Board FDEF: I2C: ready DRAM: 2 GB (ECC enabled, 400 MHz, CL3) Mellanox Technologies...
- Page 66 PCIE1: successfully set as root-complex 15b3 bd34 0c06 Net: ppc_4xx_eth0, ppc_4xx_eth1 Hit Ctrl+B to stop autoboot: Mellanox MLNX-OS Boot Menu: 1. EFM_PPC_M460EX EFM_1.1.1000 2010-06-24 16:32:03 ppc 2. EFM_PPC_M460EX EFM_1.1.1200 2010-06-25 18 :00:03 ppc 3. U-Boot prompt Choice: Select the image to boot. Mellanox Technologies...
-
Page 67: Table 12: Adapter Cards Certification Status
Table 12 lists the approved certification status per card in different regions of the world. Table 12 - Adapter Cards Certification Status Adapter Card P/N VCCI ICES cTUVus MT51336-EN-EVB YES MT51336-VPI-EVB YES A.1.1 FCC Statements (USA) Class A Statements: § 15.19(a)(4) This device complies with Part 15 of the FCC Rules. - Page 68 EMC testing and certification for many electronic products. Korean EMC certifications are issued by Radio Research Laboratory (RRL), which is organized under the Ministry of Information and Communications (KCC). EMC testing includes electromagnetic emissions (EMI) and susceptibil- ity (EMS). Certified equipment is labeled with the KCC mark and certification number. Mellanox Technologies...
-
Page 69: Table 13: Specifications For Mt51336-Nudt
Passive cables: 100W – 2.8W/port Typ@Typ: 40GigE Active Cables: 186W – 5.2 W/port Passive cables: 104W – 2.9 W/port Max@Max: 40GigE Active Cables: 224W – 6.2 W/port Passive cables: 119W – 3.3W/port Temperature: 0°C to 45°C Protocol Support Regulatory Mellanox Technologies... - Page 70 VCCI V-3/2007.04, class A EN 55022:1998+A1:2000+A2:2003 class A, EN 61000-3-2:2000+A2:2005, EN61000-3-3:1995+A2:2005, EN 55024:1998 + A1:2001+A2:2003 standards, harmonized under EMC Directive 2004/108/EC Article 6(2); AS/NZS 3548 Safety: IEC/EN 60950-1:2006 ETSI EN 300 019-2-2 IEC 60068-2- 64, 29, 32 RoHS: RoHS-R5 Mellanox Technologies...
-
Page 71: Figure 38: Transfer Power Cord
Push the cord end through the hole in the switch slide. Leave ~ 7” of cord hanging out from the hole. Screw the switch slides to the switch. Make sure to use two cord clamps per switch slide. These will be needed to hold the cord during the switch installation process. Mellanox Technologies... -
Page 72: Figure 39: Install The Power Cord Clip
Rev 1.0 Figure 39: Install the Power Cord Clip Push the power cord into the cord clamps. Figure 40: Transfer Power Cord Finished Return to step 4 on page 29 and continue the installation. Mellanox Technologies... -
Page 73: Figure 41: Connector Pinout
PCIe1_Tx2_n GP_HV PWR_OK GBE1_MDI0+/TX0+ 101 3.3V USB_0_OC# GBE0_MDI0+/TX0+ 102 PCIe1_Rx1_p 3.3V PCIe1_Tx1_p Sys_Err INT1# 2.5VA_GBE PCIe1_Rx1_n USB0- 2.5VA_GBE PCIe1_Tx1_n INT2# GBE1_ACT# USB0+ GBE0_ACT# CPU_TDO GBE1_LINK# PCIe1_Rx0_p RS232_DCD GBE0_LINK# PCIe1_Tx0_p CPU_TD1 GBE1_MDI3- INT3# PCIe1_Rx0_n RS232_RX GBE0_MDI3- CPU_Present# PCIe1_Tx0_n GBE1_MDI3+ Mellanox Technologies... - Page 74 Rev 1.0 Table 14 - Connector Pinout Signal Signal Pin number Signal number number RS232_TX GBE0_MDI3+ CPU_TCK PCIE_CLK_REF1+ PCIe0_Rx0_p RS232_DTR PCIE_CLK_REF0+ PCIe0_Tx0_p CPU_TMS GBE1_MDI2- PCIE_CLK_REF1- PCIe0_Rx0_n RS232_DSR GBE0_MDI3- PCIE_CLK_REF0- PCIe0_Tx0_n SYS_HALT# GBE1_MDI2+ RS232_RTS GBE0_MDI3+ Mellanox Technologies...
Need help?
Do you have a question about the MT51336-EN-EVB and is the answer not in the manual?
Questions and answers