Related Manuals for Mellanox Technologies MTS3610Q-6UNC
Summary of Contents for Mellanox Technologies MTS3610Q-6UNC
- Page 1 324 PORT QDR InfiniBand Switch Platform User’s Manual PN: MTS3610Q-6UNC, MTS3610D-6UNC Rev 1.7 www.mellanox.com...
- Page 2 THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
-
Page 3: Table Of Contents
5.5.1 Leaf Fan Module 5.5.2 Spine Fan Modules 5.6 Management Module 5.6.1 Extracting a Management Module 5.6.2 Inserting a Management Module 5.7 Replacing the Battery 5.8 Over-temperature Shutdown Mechanism Chapter 6 Disassembly and Disposal 6.1 Disassembling the Chassis Mellanox Technologies... - Page 4 Appendix D Ethernet RJ 45 Connector Appendix E Sample In-band FW Upgrade Appendix F Replacement Parts Ordering Numbers Appendix G Avertissements de sécurité d’installation (French) Appendix H Installation - Sicherheitshinweise (German) Appendix I Special Regulations Regarding Finland, Sweden, Denmark, and Norway Mellanox Technologies...
-
Page 5: Table Of Figures
Figure 29: Mechanical Lifting Device Figure 30: Shelf Removal Figure 31: Rail Slide Removal Figure 32: Front Panel Figure 33: Rear Panel Leaf Side of the Switch Figure 34: QSFP Connector Male and Female Views Figure 35: Ethernet RJ45 Connector Mellanox Technologies... -
Page 6: Table Of Lists
Switch Rack Mechanical and Environmental Requirements (Fully Populated Chassis) 10 Table 4: LEDs Display for Normal Operation Table 5: HCA Cards Certification Status Table 6: Switch Specification Sheet MTS3610D and MTS3610Q Table 7: InfiniBand QSFP Connector Pinout Table 8: Replacement Parts Ordering Numbers Mellanox Technologies... -
Page 7: About This Manual
> Downloads > Firmware Tools. Note that the Switch System described in this manual is based on ® Mellanox Technologies’ InfiniScale IV switch device. FabricIT Enterprise Fabric Management Software Talk to your Mellanox representative for information regarding CLI User’s Manual licensing and implementation of the FabricIT Enterprise Fabric Management Software System. -
Page 8: Revision History
Updated installation with new parts 2009 Mar, 2009 Updated installation with new parts Feb, 2009 added max time a fan module can be removed, and LED reset of the fan leds Jan, 2009 Added Installation instructions and added info on EMGT. Mellanox Technologies... -
Page 9: Chapter 1 Overview
This is the System Image GUID according to the IB spec. It is burned on the board which is in the chassis. All the boards and the management SW looks for this GUID in addition to their own Node GUID. Mellanox Technologies... - Page 10 The switch ships in a minimum base configuration plus additional modules depending on the cho- sen customer configuration. Optional modules include: • Leaf boards • Power supply units • Management modules Figure 2 shows the front and rear sides fully populated. Mellanox Technologies...
-
Page 11: Figure 2: Switch Views
The chassis needs all nine spines for the switch system to run properly. Four spines to the right of the fan module and five spines to the left of the fan mod- ule. The leaf fan module contains 7 internal fans. Mellanox Technologies... -
Page 12: Chapter 2 Safety Warnings And Led Indications
6. During Lightning - Electrical Hazard During periods of lightning activity, do not work on the equipment or connect or disconnect cables. Mellanox Technologies... - Page 13 This equipment should be installed in compliance with local and national electrical codes. 12. Battery Replacement There is a risk of explosion should the battery be replaced with a battery of an incorrect type. Dispose of used batteries according to the instructions. Mellanox Technologies...
- Page 14 According to the WEEE Directive 2002/96/EC, all waste electrical and electronic equipment (EEE) should be collected separately and not dis- posed of with regular household waste. Dispose of this product and all of its parts in a responsible and environ- mentally friendly way. Mellanox Technologies...
-
Page 15: Led Status Indicators
Each of the 18 connectors on the Leaf board has two LEDs that indicate if there is a physical and logical connection. The figure below explains the connector status according to its LED display. Mellanox Technologies... -
Page 16: Spine Board Led Indicators
These are displayed on the spine as link A and link B. The two status LEDs for the spine have the configuration shown in Figure 7. The LEDs indicate as follows: Mellanox Technologies... -
Page 17: Figure 7: Spine Board Status Indications
Each of the leafs is connected to each of the spines through two links. • Yellow – indicates that there is a logical connection to the spines through this link. • Green – indicates that there is a physical connection to the spines through this link. Mellanox Technologies... -
Page 18: Management Module Led Indicators
Green = operational Red = PS fault detected. User should check individual power supplies for fault indications. FAN SPINE off = No power to fan Green Green = operational Red = Fault detected (fan broken, can't reach full RPMs) Mellanox Technologies... - Page 19 Green = Management module is operating as a master (i.e., the other management module is a Slave) Yellow = Management module is operating as a slave (other management module is the master). When only one management is present in the system, default is Green Mellanox Technologies...
-
Page 20: Chapter 3 Installation
~350 lbs (155 kg). Make sure that proper manpower and equipment is used for transporting and moving the chassis. Note: You will need a mechanical lift, to move and insert the chassis into the rack. Mellanox Technologies... -
Page 21: Esd Connection
Make sure that this is a tight fit. Figure 9: ESD Connector ESD connec- 3.2.2 Installing the Chassis You will need: • #2 phillips screwdriver • a mechanical lift • a grounding lug Mellanox Technologies... -
Page 22: Figure 10: Installation Kit Parts
• 1 left hand cable holder Figure 10: Installation Kit Parts Lock down bar Rail slide Eye bolts Caged nuts and bolts Right and Left Cable holders Before starting the procedure, put the ESD strap on connect it to a valid ground. Mellanox Technologies... -
Page 23: Figure 11: Vertical Rails To Cabinet Measurement
2. Insert four caged nuts into the rack on the side you plan to have the connectors. Note the U num- ber on the rack. Figure 12: Caged Nut Insertion 3. Screw the rail slides onto the rack. Mellanox Technologies... -
Page 24: Figure 13: Rail Slide Installation
1. Screw the four Eye bolts to the four corners of the top of the chassis. 2. Connect the eye bolts to a Mechanical lifting device. Figure 15: Mechanical Lifting Device 3. Raise the chassis 2 cm (1”) above the level of the shelf. Mellanox Technologies... -
Page 25: Figure 16: Face Plate Mounting Bolt Locations
9. Screw the bolts through the faceplate into the caged nuts, and tighten all ten bolts and nuts. 10.Place the lock down bar over the lip of the MTS3610 and align the holes in the lock down bar with the holes in the shelf. Mellanox Technologies... -
Page 26: Ground Connections
Test the ground using an Ohm meter. Figure 18: Ground Connection Grounding post Grounding lug Mellanox Technologies... -
Page 27: Installing The Cable Holder
60Hz. The output voltage for the PSUs is 48V. The power cords should be standard 3-wire AC power cords including a safety ground. Before starting any procedure on the MTS3610 Switch system put an ESD prevention wrist strap on your wrist and connect to the MTS3610 Chassis. Mellanox Technologies... -
Page 28: Figure 19: Electrical Cable Clamp
PSUs fail, the total current consumption will then be divided by the remaining working PSUs. When the failed PSU is hot swapped the new PSU will ramp up and pass its share of cur- rent, so that the total current is always divided by the number of working PSUs. Mellanox Technologies... -
Page 29: Figure 20: Multiple Power Inlets - Electric Caution Notification
4. Go around to the other side of the switch chassis and check the leaf board status LEDs. This LED should be off. The green connector The yellow connector LEDs should be on. LEDs should be blink- ing. This LED should be on. Mellanox Technologies... -
Page 30: Infiniband Qsfp Cable Installation
Warning: Take care to not impede the air exhaust flow through the ven- tilation holes next to the InfiniBand ports. Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack. Mellanox Technologies... -
Page 31: Chapter 4 Switch Power Up
Subnet Managers. The switch supports the SystemImage GUID feature of the InfiniBand specification which means that software management tools that support this feature can recognize all the internal devices as a single switch system. Mellanox Technologies... - Page 32 A fully populated 324 port system features up to 25.9 Tbit/sec (IB ports in QDR mode) Constant Bisectional Bandwidth (CBB) Fat Tree topology that the Subnet Manager will discover and con- figure to enable a full cluster to operate. For more information refer to the documentation of the Subnet Manager being used. Mellanox Technologies...
-
Page 33: Chapter 5 Insertion And/Or Hotswap Extraction
PSU may be extracted without bringing down the system. To extract a PSU: 1. Remove the power cord from the power supply unit. 2. Unscrew the cable clip and put the clip and screws aside for later. Mellanox Technologies... -
Page 34: Leaf Boards
1. Disconnect all cables connected to the leaf board. 2. Unscrew the retaining screws on the outside of the connectors. These screws are captive and will remain hanging in place. See Figure 22. Mellanox Technologies... -
Page 35: Inserting A Leaf Board
1. Unscrew the locking screws on both of the latches. 2. Press the latch button on each of the spine ejectors and pull open both latches. Open both latches simultaneously until they are 90 degrees from the face of the spine. Mellanox Technologies... -
Page 36: Inserting A Spine Board
4. Make sure the spine ejectors are open all the way. Now press both ejectors inward simultane- ously. You should feel the ejectors are hooked into the front panel. 5. Press both ejectors applying some force to close them completely. 6. Screw in the locking screws on each latch. Mellanox Technologies... -
Page 37: Fan Modules
5.5.1.1 Extracting the Leaf Fan Module 1. Unscrew the four captive screws that hold the leaf fan module to the chassis. See Figure 24. 2. Slowly pull out the fan module while holding both its upper and lower handles. Mellanox Technologies... -
Page 38: Spine Fan Modules
• outside air is brought in through vent holes below the spines • pushed through the spines • pushed out of the chassis by the spine fan module above the leafs. Air flow through the spines is independent of the air flow through the leafs. Mellanox Technologies... -
Page 39: Management Module
3. Press the red buttons to unlock the ejectors. 4. Press the ejectors outward so that the board pops out, then slowly slide the board out. Note: The board is short, therefore do not let go of it while sliding it out. Mellanox Technologies... -
Page 40: Inserting A Management Module
Replace the battery by: 1. Remove the Management module from the chassis. 2. Put on an ESD strap and ground the strap to a valid ground. 3. Remove the battery from the holder. 4. Insert the new battery + side up. Mellanox Technologies... -
Page 41: Over-Temperature Shutdown Mechanism
Each leaf board and each spine board has its own on-board thermal sensor to track PCB tempera- º ture. If a PCB’s temperature exceeds 70 C, the thermal sensor shuts down the over-heated card, and notifies the management module. Mellanox Technologies... -
Page 42: Chapter 6 Disassembly And Disposal
9. Remove the lock down bar. 10.Remove the 10 screws holding the chassis to the rack. These screws are located in the faceplate on the rear of the chassis. Mellanox Technologies... -
Page 43: Removing The Chassis
3. Raise the chassis 2 cm (1”) above the level of the shelf. 4. Move the chassis away from the rack. 5. Remove the mechanical lifting device. 6. Remove the ten caged nuts from the vertical rack support. Mellanox Technologies... -
Page 44: Removing The Bottom Shelf
According to the WEEE Directive 2002/96/EC, all waste electrical and electronic equipment (EEE) should be collected separately and not dis- posed of with regular household waste. Dispose of this product and all of its parts in a responsible and environ- mentally friendly way. Mellanox Technologies... -
Page 45: Chapter 7 Switch Management Tools
The chassis manager will give the user access to: • switch temperatures • power supply voltages • fan unit information • power unit information • Flash memory The manager also has the ability to • burn new FW on the switch system. Mellanox Technologies... -
Page 46: Chapter 8 Fabricit Management
Without the license, you will not be able to run a Subnet Manager on the switch nor run advanced fabric diagnostics. For information on installation, please refer to the FabricIT Enterprise Fabric Management Soft- ware CLI User’s Manual. Mellanox Technologies... -
Page 47: Chapter 9 Troubleshooting
3. Make sure that the latest FW version is installed on both the HCA card and the switch. 4. If media adapters are used check that the all connections are good, tight, and secure. 5. Replace the cable. Mellanox Technologies... -
Page 48: Management Module
3. Remove and reinstall the spine board. Make sure the mating connectors of the unit is free of any dirt and/or obstacles. See Section 5.4 on page 35. 4. Make sure that the spine and the Leafs both have the same version of FW. Mellanox Technologies... -
Page 49: Management Module
See Section 5.6 on page 39. 3. Make sure that the spine and the Leafs both have the same version of FW. 4. Reburn the FW and remove and reinstall the management module. 5. Replace the Management module. Mellanox Technologies... -
Page 50: Appendix A 19U Qsfp Ddr/Qdr Ib Switch Specification Data
SerDes Speeds: 20 and 40 Gb/s per port QSFP: MTS3610Q Connector Types: QSFP Power Consump- 5170W @ 220VAC (fully pop- tion: ulated) Dissipated power: Dissipated power: 3820W Power through QSFP: 1350W (3.5W available per port) Protocol Support Regulatory Compliance Mellanox Technologies... - Page 51 A.1 FCC Statements (USA) Class A Statements: § 15.21 Statement Warning! Changes or modifications to this equipment not expressly approved by the party responsi- ble for compliance (Mellanox Technologies) could void the user's authority to operate the equipment. §15.105(a) Mellanox Technologies...
- Page 52 (Translation - "This is a Class A product based on the standard of the Voluntary Control Council for Inter- ference by Information Technology Equipment (VCCI). If this equipment is used in a domestic environ- ment, radio interference may occur, in which case the user may be required to take corrective actions.") Mellanox Technologies...
-
Page 53: Appendix B Mechanical Drawing
Figure 32: Front Panel Feedback: L eaf Fan MTS3610 MTS3610 ATTN MTS3610 MTS3610 MTS3610 MTS3610 MTS3610 MTS3610 MTS3610 ATTN ATTN ATTN ATTN ATTN ATTN ATTN ATTN Eight Power Modules 100-240 VAC 50-60 Hz Nine spines Air inlets to Spines Mellanox Technologies... -
Page 54: Figure 33: Rear Panel Leaf Side Of The Switch
Each Leaf has 18 QSFP connector ports. The switch can hold up to 18 leafs. Leaf Blanks must be in place when a slot is not populated with a Leaf. This is necessary to ensure proper air flow for cooling the leafs and spines. Mellanox Technologies... -
Page 55: Appendix C Qsfp Interface
Receiver Non-Inverted Data Output 3 Ground ModPrsL Module Present IntL Interrupt Vcc Tx +3.3 V Power supply transmitter Vcc 1 +3.3 V Power Supply LPMode Low Power Mode Ground Tx3p Transmitter Non-Inverted Data Input Tx3n Transmitter Inverted Data Input Mellanox Technologies... -
Page 56: Figure 34: Qsfp Connector Male And Female Views
Rev 1.7 Figure 34: QSFP Connector Male and Female Views 18.35 View into Rear of Connector 8.50 18.35 View into Front of Cage 8.50 Mellanox Technologies... -
Page 57: Appendix D Ethernet Rj 45 Connector
19U QSFP SDR/DDR/QDR InfiniBand Switch Platform User’s Manual Rev 1.7 Appendix D: Ethernet RJ 45 Connector Figure 35: Ethernet RJ45 Connector Mellanox Technologies... -
Page 58: Appendix E Sample In-Band Fw Upgrade
/dev/mst/mt25418_pci_uar0 - PCI direct access. bus:dev.fn=07:00.0 bar=0xde000000 size=0x2000000 /dev/mst/mtusb-1: - USB to I2C adapter as I2C master Running the mst ib add for adding the in-band devices, (you should run opensm before this opera- tion) Mellanox Technologies... - Page 59 -I- skip option set. no report will be issued -I--------------------------------------------------- -I- Links With Logical State = INIT -I--------------------------------------------------- -I- skip option set. no report will be issued -I--------------------------------------------------- -I- PM Counters Info -I--------------------------------------------------- -I- skip option set. no report will be issued Mellanox Technologies...
- Page 60 -I- Stages Status Report: STAGE Errors Warnings Bad GUIDs/LIDs Check Link State Active Check Performance Counters Report Partitions Check IPoIB Subnets Check Please see /tmp/ibdiagnet.log for complete log ---------------------------------------------------------------- -I- Done. Run time was 14 seconds. -I- Added 32 in-band devices Mellanox Technologies...
- Page 61 - PCI direct access. bus:dev.fn=07:00.0 bar=0x00000000 size=0x0 /dev/mst/mt25418_pci_uar0 - PCI direct access. bus:dev.fn=07:00.0 bar=0xde000000 size=0x2000000 /dev/mst/mtusb-1: - USB to I2C adapter as I2C master Inband devices: ------------------- /dev/mst/CA_MT25418_mtlsqt127_HCA-1_lid-0x0020 /dev/mst/CA_MT25418_mtlsqt128_HCA-1_lid-0x000A /dev/mst/CA_MT25418_mtlsqt131_HCA-1_lid-0x001E /dev/mst/CA_MT25418_mtlsqt135_HCA-1_lid-0x001B /dev/mst/CA_MT25418_mtlsqt137_HCA-1_lid-0x0001 /dev/mst/CA_MT25418_mtlsqt138_HCA-1_lid-0x000F /dev/mst/SW_MT48438_lid-0x0002 /dev/mst/SW_MT48438_lid-0x0003 /dev/mst/SW_MT48438_lid-0x0004 /dev/mst/SW_MT48438_lid-0x0005 /dev/mst/SW_MT48438_lid-0x0006 Mellanox Technologies...
- Page 62 /dev/mst/SW_MT48438_lid-0x0016 /dev/mst/SW_MT48438_lid-0x0017 /dev/mst/SW_MT48438_lid-0x0018 /dev/mst/SW_MT48438_lid-0x0019 /dev/mst/SW_MT48438_lid-0x001A /dev/mst/SW_MT48438_lid-0x001C /dev/mst/SW_MT48438_lid-0x001D /dev/mst/SW_MT48438_lid-0x001F mtlsqt131:/home/folder # flint -d /dev/mst/SW_MT48438_lid-0x0002 q Image type: FW Version: 7.1.903 Device ID: 48438 Chip Revision: A0 Description: Node Sys image GUIDs: 0002c90200405a10 0002c90200405a10 Board ID: (MT_0C60000003) VSD: PSID: MT_0C60000003 Mellanox Technologies...
- Page 63 -I- Using auto detected hash file: /mswg/release/fw-IS4/last_revision/hash.csv -I- Generating image ... Current FW version on flash: 7.1.903 New FW version: 7.1.918 Burning second FW image without signatures - OK Restoring second signature - OK -I- Image burn completed successfully. Mellanox Technologies...
-
Page 64: Appendix F Replacement Parts Ordering Numbers
Power cord Type E/F for Sweden, France, Germany, ACC000207 Netherlands, Russia Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Mellanox Technologies... - Page 65 19U QSFP SDR/DDR/QDR InfiniBand Switch Platform User’s Manual Rev 1.7 Part Description Power cord Type B for Japan, ACC000212 Power cord Type I for Australia ACC000213 Mellanox Technologies...
- Page 66 Rev 1.7 Mellanox Technologies...
-
Page 67: Appendix G Avertissements De Sécurité D'installation (French)
7. Montage et entretien sur baie Lorsque ce produit est monté ou entretenu sur baie, il faut prendre des précau- tions spéciales pour s’assurer que le système reste stable. En général, il faut remplir la baie avec du matériel de bas en haut. Mellanox Technologies... - Page 68 être certifiés UL de type DP-1 ou DP-2. (Note - lorsqu’il existe dans un circuit non LPS) Protection contre la surintensité : Un appareil de protection répertorié facile- ment accessible contre la surintensité du circuit de branchement et calibré à 20A doit être incorporé dans le câblage électrique du bâtiment. Mellanox Technologies...
- Page 69 à 3 conducteurs, d’un diamètre de fil minimum de 0,75 mm2, d’une tension nominale de 300 V, avec une gaine isolée en PVC. Le cordon doit avoir une fiche moulée d’une tension nominale de 250 V et d’une intensité nominale de 10 A. Mellanox Technologies...
-
Page 70: Appendix H Installation - Sicherheitshinweise (German)
7. Rack-Montage und Wartung Wenn dieses Produkt in einem Rack montiert oder gewartet wird, sind beson- dere Vorsichtsmaßnahmen zu ergreifen, um die Stabilität des Systems zu gewährleisten. Im Allgemeinen sollten Sie das Gestell von unten nach oben mit Geräten füllen. Mellanox Technologies... - Page 71 3 - Leiter, mindestens 0,75 mm2 Draht, bewertet mit 300 V, mit einem PVC- Mantel isoliert. Das Kabel muss eine angespritztem Stecker bewertet bei 250 V, 10 A. " 13. Ableitstrom> 3.5mA WARNUNG: Hohe Ableitstrom; Earth Verbindung, bevor Sie die Verbindung von wesentlicher Bedeutung werden. Mellanox Technologies...
- Page 72 Kabel für den Anschluss an das Gerät RS232-und Ethernet-Schnittstellen müssen UL zertifiziert Typ DP-1 oder DP-2. (Hinweis-, wenn nicht mit Wohn- sitz in LPS-Schaltung) Überstromschutz: Eine leicht zugängliche Auflistung Abzweigleitung Übers- trom-Schutzeinrichtung 20 A bewertet werden müssen in dem Gebäude Verka- belung. Mellanox Technologies...
-
Page 73: Appendix I Special Regulations Regarding Finland, Sweden, Denmark, And Norway
2. Finland - “Laite on liitettävä suojamaadoituskoskettimilla varustettuun pistora- siaan” 3. Norway - “Apparatet må tilkoples jordet stikkontakt” Unit is intended for connection to IT power systems for Norway only. 4. Sweden - “Apparaten skall anslutas till jordat uttag.” Mellanox Technologies...
Need help?
Do you have a question about the MTS3610Q-6UNC and is the answer not in the manual?
Questions and answers