Summary of Contents for Mellanox Technologies InfiniBand X
Page 1
SwitchX 36-Port QSFP InfiniBand- Ethernet Gateway System Hardware User Manual P/N: MSX6036G-2SFS, MSX6036G-2SRS Rev 1.0 www.mellanox.com...
Page 2
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.
This requires a customer support login. Look up the relevant SwitchX®- based switch system/series release note file. This document contains information regarding configuring and manag- Mellanox MLNX-OS® Software User ing Mellanox Technologies Switch Platforms. Manual Command Reference Guide for MLNX-OS® listing all of the commands MLNX-OS® Software Command Ref- available through MLNX-OS®...
FDR — Fourteen Data Rate – Used to indicate a 4X port running at 56 Gb/s. FRU — Field Replaceable Unit Mellanox Part Numbering Legend Place Field Decoder Mellanox Technologies System Type SwitchX® switch Family Data Transfer Protocol Size of box 0 = 1U 1 = 1.5U...
The SX6036G is a high-performance, low-latency 56Gb/s FDR InfiniBand to 40Gb/s Ethernet gateway built with Mellanox's 6th generation SwitchX®-2 VPI switch device. Installation, hot-swapping components and hardware maintenance is covered in “Basic Opera- tion” on page 16. Figure 1: Connector Side View of the Gateway 427.9 mm Standard depth 628.9 mm Short depth 428.9 mm 43.6 Mellanox Technologies...
RoHS 6 Advanced Features • Congestion control** • Adaptive routing** • Port mirroring** • Forward Error Correction -(FEC) • Link Layer Retransmission -(LLR) • Multiple SWIDs ** ** This feature will be available to customers in the near future. Mellanox Technologies...
4X port runs a bit rate of 14.0625 Gb/s with a 64b/66b encoding, resulting in an effective band- width of 56.25 Gb/s. The FDR physical layer is an IBTA specified physical layer using advanced block types, deskew mechanism and framing rules for supporting higher speeds and increased reliability. Mellanox Technologies...
Page 15
FDR10 supports 20% more bandwidth over QDR using the same QSFP cables/connectors. Both FDR and FDR10 support Forward Error Correction (FEC), as described in IEEE 802.3ap chapter 74. FDR and FDR10 are only guaranteed to work with approved Mellanox Cables. Mellanox Technologies...
All connectivity except for power cords is via the connector side panel. All connectors can sup- port active optical cables. Status LEDs The System Status LEDs are located to the left of the connectors on the connector side panel, and on the power side at the far right. Mellanox Technologies...
Figure 5: Gateway Status LEDs Front and Back Both of these LEDs show the switch status. Power side Connector side status LED status LED Both of these LEDs give identical information. The system status LEDs should display as follows: Mellanox Technologies...
If the PS LEDs are not green, this indicates a problem with the power supplies. The gateway is operational only if at least one of the PS LEDs is green. 2.2.2 Gateway Status LED Figure 6: Gateway Status LED Mellanox Technologies...
If the shuts down due to over temperature, unplug the , wait 5 min- gateway utes and replug in the . For more information See page 57. Figure 7: Fan Status LED Connector Side Mellanox ® Mellanox Technologies...
The gateway is available with one factory installed Power Supply Unit. A second Power Supply Unit can be added to support hot-swap ability and to add redundancy. See “Accessory and Replacement Parts” on page 66 for ordering part numbers. Mellanox Technologies...
Error – The PS unit is not operational. Off – There is no power to the system (neither PS unit is receiving power). If one PS unit is showing green and the second PS unit is unplugged it will show a red indication. Mellanox Technologies...
The UID LED is a debug feature that lights a blue LED on the gateway connector side panel for ease in finding a particular gateway within a cluster. 1. This feature will be available in a coming software release. Mellanox Technologies...
Table 11 - Port Connector Physical and Logical Link Assignments for Ethernet Mode LED Configuration LED Description Physical link is down / Default. Solid Green Physical link is up with no traffic. Flashing Green Physical link is up with traffic. Flashing Orange Physical errors. Mellanox Technologies...
On the power side panel the PS unit and fan modules air flow direction can be seen. All compo- nents need to have the same air flow. A mismatch in the air flow will affect the heat dissipation. All PS units must have the same air flow as the fan module. Mellanox Technologies...
These air flow labels must be the same. Any mismatch of these labels will affect the gateway temperature. Table 13 shows the icons that indicate airflow using arrows and colors. All components need the same colored airflow icons. Mellanox Technologies...
• 1 – connector that is labeled “CONSOLE” Use this connector to connect to the host PC. • 1 – reset button • 1 – I2C banana connector on the power supply side is for FAE use only. Mellanox Technologies...
There is a single USB connector. This connector can be used to install software and or firmware upgrades using a memory device that has a USB connector. This connector is USB 2.0 compli- ant. USB 1.0 is not supported. Mellanox Technologies...
The ports are labelled as shown in Figure 19. The bottom row ports are flipped from the top row. See Figure 20 for bottom row - top row port orientation. See page 51 for more informa- tion regarding breakout (fanout) cables. Mellanox Technologies...
To change the port speed configuration use the GUI, or use the CLI and go to the interface con- figuration mode, then use the command “speed”. See the Mellanox MLNX-OS user manual for instructions on port speed re-configuration. Mellanox Technologies...
3. Over-temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: 45°C (113°F). Moreover, to guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation openings. Mellanox Technologies...
In general you should fill the rack with equipment starting from the bottom to the top. 10. Equipment Installation This equipment should be installed, replaced, and/or serviced only by trained and qualified personnel. Mellanox Technologies...
Page 32
“<HAR>”, 3 - conductor, minimum 1.0 mm wire, rated at 300 V, with a PVC insulated jacket. The cord must have a molded plug rated at 250 V, 10 A. 16. High Leakage Current Warning: High leakage current; Earth connection essential before connecting supply. Mellanox Technologies...
Page 33
Caution: Slide/rail mounted equipment is not to be used as a shelf or a work space. The rails are not intended for sliding the unit away from the rack. It is for permanent installation at final resting place only, not used for service and maintenance Mellanox Technologies...
Norway only. 3.1.1 Battery Replacement Mellanox Technologies does not support battery replacement. Customer removal of the gateway cover will void the warrantee. Only Remove the cover to comply with WEEE directives or to dis- assemble for environmentally approved disposal. Customer removal of the cover will void the warrantee.
1. Place the ESD mat on the floor where you will be working and put on the ESD strap. Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground. Mellanox Technologies...
• Will the connector side be recessed past other equipment in the rack and will this be problematic? • The installation kit allows for a 2” (5cm) recess of the gateway past the vertical support. • Make sure the gateway air flow is compatible with your installation selection. Mellanox Technologies...
Figure 24: Screwing on the Rail 5 screws per side are needed for the short gateway. 7 screws per side are needed for the standard gateway. There are 16 screws in the kit; you will have left over screws. Mellanox Technologies...
Figure 26: Slide the Rail into the Rail Slide This side of the rail kit goes on the side of the rack you will slide the gateway into. This is the same side of the gateway that will be next to the vertical support. Mellanox Technologies...
If so, connect a ground wire to one of the casing screws and connect the other end to a valid ground. If you choose to not use the ground screw, make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the gateway and the Mellanox Technologies...
Einhieten spannungs- ensure a powered down state “Alimentation Stabilise’e”. frei zu machen sind die inside of the switch platform. Pour isoler completement le Netzstecker aller Netzteile module en cause, Il faut zu entfernen de’brancher toutes les alimen- tations stabilise’es. Mellanox Technologies...
Figure 30: Power Supply Unit Extraction Push Here Latch release If you are not immediately installing a new PSU you must install a blank so that the air flow for cooling the gateway is not compromised. Mellanox Technologies...
45° Celsius. This unit is intended for installation in a Restricted Access Location. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security. Mellanox Technologies...
These two latches must be pushed towards each other at the same time while the module is pulled out. To insert a FAN Unit: 1. Make sure the mating connector of the new unit is free of any dirt and/or obstacles. Mellanox Technologies...
Hook up the supplied harness cable (HAR00028) from the connector labelled CONSOLE to the DB9 connector of the local host PC. Figure 34: Console Port SX6036G CONSOLE Make sure to connect to the Console RJ-45 port of the gateway and not to the (Ether- net) MGT port. Mellanox Technologies...
‘zeroconf’ configuration or not. If you enter ‘no’ (no Zeroconf), then you need to enter a static IP, and the session will continue. Step 3: Enable IPv6? [yes] The management interface will be able to use IPv6 addresses. Mellanox Technologies...
Page 46
Note: <enter> to save changes and exit. To re-run the configuration wizard run the command Choice: <Enter> “configuration jump-start” in Config mode. Configuration changes saved. The table below shows an example of static IP configuration for mgmt0 interface. Mellanox Technologies...
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the “configuration jump-start” command from configure mode. Launching CLI... > Mellanox Technologies...
The following is an example of the output: Interface mgmt0 state Admin up: Link up: IP address: 192.168.10.43 Netmask: 255.255.255.0 Speed: 1000Mb/s (auto) Duplex: full (auto) Interface type: ethernet Interface source: physical MTU: 1500 HW address: 00:02:C9:11:2A:AE Comment: Mellanox Technologies...
2. Start a remote secured shell (SSH) to the gateway using the command: “ssh -l <username> <gateway _IP_address>”, where the <gateway _IP_address> is the IP address of the gateway . rem_mach1 > ssh -l <username> <ip address> Mellanox MLNX-OS Switch Management Last login: Thu Aug 28 11:24:13 2013 from 192.168.10.1 Mellanox Technologies...
Page 50
Guide for additional information on the CLI commands. 5. You can enter any supported command now. Rerunning the Wizard 6. If you want to rerun the wizard run the following commands: switch > enable switch# configure terminal switch (config) # configuration jump-start Mellanox Technologies...
2 10 GbE ports then only SFP+ connectors #1 and #2 are used. Connectors #3 and #4 are left unconnected. Using Figure 36 and Table 18, with breakout cables you can achieve 13 ports of 10 Gb/s. Figure 35: Breakout or Fanout Cable Mellanox Technologies...
Can be split to 2 — — — — — — — — — — — — — Future Software versions will support full flexibility to include individual port configuration with splittable ports as per Figure 37 and Table 19. Mellanox Technologies...
= 00 10GbE ports + 00 40GbE ports THESE PORTS 12 x 40GbE ports split to 4 10GbE ports = 48 10GbE ports 12 x 40GbE ports used as 40GbE ports = 12 40GbE ports Total = 48 x 10GbE and 12 x 40GbE Mellanox Technologies...
Use the CLI or the GUI in order to perform software upgrades. For further information please refer to the MLNX-OS Software User Manual section Upgrading MLNX-OS® Software. Be sure to read and follow all of the instructions regarding the updating of the software on your gateway system. Mellanox Technologies...
Plug in the gateway . If the gateway does not come on, check the power supplies. If the gateway comes on, Use the MLNX-OS® management CLI to determine the cause of the Shutdown. Check the temperature. Check the Fan status. Mellanox Technologies...
Temperature: Cable power / MAX 2.0W Operating 0° to 45° Celsius Temperature TYP: 1.5W Non-operating -40° to 70° Celsius Shock and Humidity: Shock and Vibration: ETSI EN 300 019-2-2: 1999-09 Operating 5% - 90% non-condensing Vibration/ Humidity Mellanox Technologies...
1+1 Redundant: Power Supplies Approved Cables For a list of all approved cables see: http://www.mellanox.com/related-docs/user_manuals/Mellanox_approved_cables.pdf EMC Certifications The list of approved certifications per switch in different regions of the world is located on the Mellanox Website at: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.pdf Mellanox Technologies...
SwitchX 36-Port QSFP InfiniBand- Ethernet Gateway System Hardware User Manual Rev 1.0 EMC statements are also in the Regulatory and Compliance Guide. China CCC Warning Statement Mellanox Technologies...
When the SwitchX® device crosses this temperature, the firmware will automatically shut down the device. 3. Emergency – 130C In case the firmware fails to shut down the SwitchX® device upon crossing the Critical thresh- old, the SwitchX® device will auto-shutdown upon crossing the Emergency (130C) threshold. Mellanox Technologies...
+3.3 V Power supply transmitter Vcc 1 +3.3 V Power Supply LPMode Low Power Mode Ground Tx3p Transmitter Non-Inverted Data Input Tx3n Transmitter Inverted Data Input Ground Tx1p Transmitter Non-Inverted Data Input Tx1n Transmitter Inverted Data Input Ground Mellanox Technologies...
Rev 1.0 Figure 39: QSFP Connector Male and Female Views 18.35 View into Rear of Connector 8.50 18.35 View into Front of Cage 8.50 Mellanox Technologies...
Fan module with connector side to power supply side air flow MSX60-FR 300W Power Supply w/ Power Supply Side to Connector side air flow MSX60-PF 300W Power Supply w/ Connector side to Power Supply side air flow MSX60-PR Power cord Type C13-C14 ACC000500 Mellanox Technologies...
Dispose of this product and all of its parts in a responsible and environmentally friendly way. Follow the instructions found in the Mellanox Web site at mellanox.com for proper instructions to disassemble and dispose of the Switch according to the WEEE directive. Mellanox Technologies...
Cet équipement ne doit pas être en service dans un local dont la température dépasse le maximum recommandé de 45°C (113°F). En outre et pour garantir une circulation d'air correcte, laisser un espace d'au moins 8 cm (3") autour des orifices de ventilation. Mellanox Technologies...
Page 69
Lors du montage ou de la maintenance de ce produit dans un rack, il faut faire spéciale- ment attention pour s'assurer que l'ensemble reste stable. En règle générale, le rack doit être rempli en commençant par le bas. Mellanox Technologies...
Page 70
300 V, avec une gaine isolante en PVC. Le cordon doit avoir une prise moulée 250 V 10 A. 20. Courant de fuite élevé Avertissement : courant de fuite élevé, une connexion à la terre est indispensable avant de brancher l'alimentation. Mellanox Technologies...
Page 71
: IEC 60 825-1:1993 + A1:1997 + A2:2001 et EN 60825-1:1994+A1:1996+ A2:2001 Pour les produits disposant de ports optiques Armoire de protection appropriée Une enceinte appropriée aux niveaux électrique, mécanique et incendie doit être four- nie par le constructeur du produit fini ou par l'utilisateur. Mellanox Technologies...
Page 72
Ce produit et toutes ses pièces doivent être mis au rebut d'une manière responsable, respectant l'environnement. 28. Restrictions concernant l'alimentation pour la Norvège Cet appareil est prévu pour être relié à un système d'alimentation TN et un système d'alimentation informatique de Norvège uniquement. Mellanox Technologies...
Temperatur von 45°C (113°F) betrieben werden. Es ist ein Luft- strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich. Außerdem sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen einwandfreien Luftstrom zu gewährleisten. Mellanox Technologies...
Page 74
Stabilität des Systems zu gewährleisten. Im Allgemeinen sollten Sie das Gestell von unten nach oben mit Geräten füllen. 13. Geräteinstallation Diese Gerät sollte nur von geschultem und qualifiziertem Personal installiert, ausge- tauscht oder gewartet werden. Mellanox Technologies...
Page 75
19. Hoher Ableitstrom WARNUNG: Hohe Ableitstrom; Earth Verbindung, bevor Sie die Verbindung von wesentlicher Bedeutung werden. 20. Infomationen zum Erdungsanschluss Bevor Sie dieses Gerät an das Stromnetz, die Schutzerde Terminal Schrauben dieses Gerät muss an den Schutzleiter in der Gebäudeinstallation. Mellanox Technologies...
Page 76
Instandhaltung und Wartung. 26. WEEE-Direktive Gemäß WEEE Directive 2002/96/EC müssen alle elektrischen und elektronischen Abfallgeräte (EEE) separat gesammelt und nichit mit normalem Haushaltsmüll ensorgt werden. Dieses Produkt und alle seine Teile in verantwortungsvoller und umweltfreundlicher Art und Weise entsorgen. Mellanox Technologies...
No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 45°C. Además, para garantizar una circulación de aire adec- uada, se debe dejar como mínimo un espacio de 8 cm (3 pulgadas) alrededor de las aberturas de ventilación. Mellanox Technologies...
Page 78
Al instalar o realizar el mantenimiento de este aparato en un bastidor, es preciso adop- tar precauciones especiales para garantizar que el sistema se mantenga estable. En gen- eral, en un bastidor, los equipos se deben instalar comenzando desde abajo hacia arriba. Mellanox Technologies...
Page 79
300 voltios nominal, con cobertura protectora aislante de PVC. El cable debe tener un enchufe moldeado con capuchón de 250 voltios nominal, 10 A. 20. Alta corriente de fuga ADVERTENCIA: Alta corriente de fuga. Es esencial efectuar la conexión a tierra antes de conectar la alimentación. Mellanox Technologies...
Page 80
PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1:2007/03 y EN 60825-1:2007 Recinto adecuado El fabricante del producto final o el usuario final deberán suministrar un confinamiento adecuado para componentes eléctricos y mecánicos y contra incen- dio. Mellanox Technologies...
Page 81
(EEE) se deben recolectar por separado y no se deben eliminar junto con residuos domésticos. Al deshacerse de este producto y de todas sus partes, hágalo de una manera respon- sable y respetuosa con el medio ambiente. Mellanox Technologies...
НЕ вставлять инструменты или части тела в углубление вентиляторного модуля. 6. Перегрев Не эксплуатировать это оборудование в помещении с температурой окружающей среды, превышающей максимально рекомендуемое значение: 45 °C (113 °F). Более того, для надлежащей вентиляции следует обеспечить зазор вокруг вентиляционных отверстий не менее 8 см (3 дюйма). Mellanox Technologies...
Page 83
подсоединять и отсоединять. За особыми предупреждениями и указаниями следует обратиться к производителю кабеля. 13. Установка или обслуживание в стойке При установке или обслуживании этого изделия в стойке следует обеспечить устойчивость системы. Как правило, стойка заполняется оборудованием снизу вверх. Mellanox Technologies...
Page 84
Подключение к электропитанию в Европе выполняется с помощью гармонизированного шнура питания с маркировкой <HAR>, 3-жильного, с сечением жилы не менее 1,0 мм², рассчитанного на номинальное напряжение 300 В, с ПВХ оболочкой. Шнур должен иметь литую вилку, рассчитанную на 250 В, 10 А. Mellanox Technologies...
Page 85
ЛАЗЕРНОЕ ИЗДЕЛИЕ КЛАССА 1 и ссылка на последние стандарты по безопасности лазерных изделий IEC 60 825-1:1993 + A1:1997 + A2:2001 и EN 60825-1:1994 + A1:1996 + A2:2001. Соответствующий кожух Производитель конечного изделия и/или конечный пользователь должны обеспечить соответствующий электрический, механический и противопожарный кожух. Mellanox Technologies...
Page 86
используются для обслуживания устройства. 27. Директива WEEE В соответствии с Директивой 2002/96/EC (WEEE) отходы электрического и электронного оборудования должны собираться и утилизироваться отдельно от обычных бытовых отходов. Следует утилизировать это изделие и все его части ответственным и экологически безопасным способом. Mellanox Technologies...
Acest echipament nu trebuie să fie acţionat într-o zonă unde temperatura ambiantă depăşeşte valoarea maximă recomandată: 45°C (113°F). În plus, pentru a asigura un flux de aer adecvat, lăsaţi un spaţiu liber de cel puţin 8 cm (3 inchi) în jurul fantelor de ventilare. Mellanox Technologies...
Page 88
Când acest produs este montat sau depanat într-un rack, trebuie să fie luate măsuri de precauţie speciale pentru a se asigura că sistemul rămâne stabil. În general, trebuie să umpleţi rack-ul cu echipamente începând de jos în sus. Mellanox Technologies...
Page 89
PVC. Cordonul de alimentare trebuie să fie prevăzut cu o fişă turnată cu putere nominală egală cu 250 V, 10 A. 20. Curent de scurgere de înaltă frecvenţă Avertisment: Curent de scurgere de înaltă frecvenţă; Împământarea este esenţială înainte de a conecta sursa de alimentare. Mellanox Technologies...
Page 90
Atenţie: Echipamentul montat pe o linie de alunecare/şină nu va fi utilizat ca raft sau spaţiu de lucru. Scopul şinelor nu este de a glisa unitatea de pe rack. Acestea sunt des- tinate instalării permanente numai la punctul final de oprire şi nu vor fi folosite pentru depanare şi întreţinere Mellanox Technologies...
Page 91
În conformitate cu Directiva DEEE 2002/96/CE, toate deşeurile de echipamente elec- trice şi electronice (EEE) trebuie colectate separat şi nu trebuie eliminate împreună cu deşeurile menajere obişnuite. Eliminaţi acest produs şi toate componentele sale în mod responsabil şi ecologic. Mellanox Technologies...