A full-featured, storage-rich, flexible 2 socket compute sled that is ideal for many workloads including virtualization, collaborative, and software-defined workloads (43 pages)
Summary of Contents for Dell EMC PowerEdge MX Series
Page 1
Dell EMC PowerEdge MX Networking Architecture Guide January 2019 Architecture Guide Abstract This architecture guide provides an overview of the architecture, features, and functionality of the PowerEdge MX networking infrastructure. Dell EMC Solutions...
Dell EMC PowerEdge MX7000 chassis PowerEdge MX7000 (front) Overview Figure 2 shows the front view of the Dell EMC PowerEdge MX7000 chassis. The left side of the chassis can have one of three control panel options: • LED status light panel •...
Page 6
Chapter 1: Dell EMC PowerEdge MX Platform Overview • Touch screen LCD panel equipped with Dell EMC PowerEdge iDRAC Quick Sync 2 The bottom of Figure 2 shows six hot-pluggable, redundant, 3,000-watt power supplies. Above the power supplies eight slots are available. In the example below, the slots contain: •...
Page 7
Figure 3. Dell EMC PowerEdge MX740c sled with six 2.5-inch SAS drives Dell EMC The Dell EMC PowerEdge MX840c is a powerful four-socket, full-height, double-width PowerEdge server that features dense compute, exceptionally large memory capacity, and a highly MX840c compute expandable storage subsystem.
Page 8
Two PCIe mini-mezzanine card slots for connecting to storage Fabric C • iDRAC9 with Lifecycle Controller Figure 4. Dell EMC PowerEdge MX840c sled with eight 2.5-inch SAS drives Dell EMC The PowerEdge MX5016s storage sled delivers scale-out, shared storage within the PowerEdge PowerEdge MX architecture.
One Dell EMC Networking MX7116n Fabric Expander Module (FEM) installed in fabric slot A2 • Two Dell EMC Networking MX5108n Ethernet switches installed in fabric slots B1 and B2 • Two Dell EMC Networking MXG610s Fibre Channel switches installed in fabric slots C1 and C2 •...
Page 10
Chapter 1: PowerEdge MX Platform Overview Dell EMC PowerEdge MX7000 – rear Figure 6. Dell EMC The Dell EMC PowerEdge MX9002m management module controls the overall chassis PowerEdge power, cooling, and hosts the OpenManage Enterprise - Modular (OME-M) console. Two MX9002m...
Page 11
ID button and health status LED Power status LED Micro-B USB port Dell EMC The Dell EMC Networking MX9116n Fabric Switching Engine (FSE) is a scalable, high- Networking performance, low latency 25 GbE switch purpose-built for the PowerEdge MX platform. MX9116n Fabric...
Page 12
Two QSFP28 ports Two QSFP28 unified ports 12 QSFP28-DD ports Dell EMC The Dell EMC Networking MX7116n Fabric Expander Module (FEM) acts as an Ethernet Networking repeater, taking signals from an attached compute sled and repeating them to the MX7116n Fabric associated lane on the external QSFP28-DD connector.
Page 13
Chapter 1: Dell EMC PowerEdge MX Platform Overview The ports can be used to provide a combination of network uplinks, VLT interconnects (VLTi), or for FCoE connectivity. The MX5108n supports FCoE Initilization Protocol (FIP) Snooping Bridge (FSB) mode but does not support NPG or direct attach FC capabilities.
Page 14
Note: For information about PTM port to compute sled-mapping, see Appendix A. Dell EMC The Dell EMC Networking MXG610s is a high-performance, 32 Gb Fibre Channel switch Networking based on Brocade Gen 6 technology. The MXG610s features 8/16/32 GbE capable MXG610s Fibre interfaces.
Page 15
Eight 32Gb FC SFP ports Two 4x 32Gb FC QSFP ports Dell EMC The Dell EMC PowerEdge MX5000s SAS module supports x4 SAS internal connections PowerEdge to all eight front-facing slots in the PowerEdge MX7000 chassis. The MX5000s uses T10 MX5000s SAS SAS zoning to provide multiple SAS zones/domains for the compute sleds.
Chapter 1: Dell EMC PowerEdge MX Platform Overview Figure 14. Dell EMC PowerEdge MX5000s SAS module The following MX5000s components are labeled in Figure 14: Express service tag Module insertion/removal latch Power and indicator LEDs Six SAS ports PowerEdge MX7000 chassis fabric slots...
Page 17
Chapter 1: Dell EMC PowerEdge MX Platform Overview Figure 15. Dell EMC PowerEdge MX740c mezzanine cards Mini-mezzanine The MX7000 chassis also provides Fabric C, shown in Figure 16, supporting redundant card MXG610s FC switches, or MX5000s SAS modules. This fabric uses a mid-plane connecting the C1 and C2 modules to each compute or storage sled.
Chassis Deployment Wizard enables you to configure the following: • System time • Alert settings • iDRAC9 Quick deployment settings • Network IOM access settings • Firmware updates • Proxy settings • MCM group definition PowerEdge MX7000 Multi-Chassis Management groups Dell EMC PowerEdge MX Networking Architecture Guide...
Page 20
OME-M console to create the group. Figure 17 shows the supported MX9002m module cabling method. Chassis one, two, and three are shown while chassis n represents the last chassis in the group. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 21
Chapter 2: OpenManage Enterprise - Modular Console Figure 17. Dell EMC PowerEdge MX9002m module daisy chain cabling Table 1 lists each MX9002m module and the associated connections that are shown in Figure 17. Table 1. MX9002m module cable requirements CxMMx...
REST API endpoint to OME-M to manage all switches in a chassis or across all chassis in an MCM group. This SmartFabric management endpoint exists even if the switches are not running in SmartFabric mode. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 23
Figure 19. OME-M console – I/O Modules inventory Figure 20 shows using the OME-M console to change the Breakout Type for port- group1/1/1 on the IOM-A1, a MX9116n FSE. This behavior is similar for the MX5108n. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 24
Chapter 2: OpenManage Enterprise - Modular Console Figure 20. OME-M console – MX9116n FSE breakout mode options Note: To change the breakout on a port, the HardwareDefault option must be selected and applied before choosing the appropriate breakout type. Dell EMC PowerEdge MX Networking Architecture Guide...
This chapter presents the following topics: Scalable Fabric architecture overview ............26 QSFP28 double density connectors ..............28 Interfaces and port groups ................29 Embedded Top of Rack switching ..............33 Scalable Fabric OS10EE CLI Commands ............35 Dell EMC PowerEdge MX Networking Architecture Guide...
3 through 10 houses only the MX7116n FEM. All connections in Figure 21 use QSFP28-DD connections. Figure 21. Scalable Fabric example using Fabric A Table 2 shows the recommended IOM slot placement when creating a Scalable Fabric Architecture. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 27
MX9116n FSE in chassis 1 fabric slot A1, then the second MX9116n FSE should be in chassis 1 fabric slot B1. This only applies if the 2xMX9116n in different chassis topology is selected. Dell EMC PowerEdge MX Networking Architecture Guide...
The equivalent QSFP28-DD description is easily identifiable: • Q28DD-200G-2SR4: Dell Networking Transceiver, 2x100GbE QSFP28-DD, 2SR4, MPO12-DD, MMF Note: See Appendix A for more information about the various media types and their use in the PowerEdge MX7000. Dell EMC PowerEdge MX Networking Architecture Guide...
To view the backplane port connections to servers, use the show inventory media command. In the output, a server-facing interface displays INTERNAL as its media. A FIXED port does not use external transceivers and always displays as Dell EMC Qualified true. OS10# show inventory media...
Page 30
MX7116n FEM appear as standard Ethernet interfaces from the perspective of the MX9116n FSE. Figure 25 illustrates how the QSFP28-DD cable provides 8x25GbE lanes between the MX9116n FSE and a MX7116n FEM. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 31
4x 25GbE – Breakout a QSFP28 port into four 25GbE interfaces • 2x 50GbE – Breakout a QSFP28 port into two 50GbE interfaces • 1x 100GbE – Reset the unified port back to the default, 100GbE mode Dell EMC PowerEdge MX Networking Architecture Guide...
Page 32
When connecting a pair of PowerEdge MX IOMs in SmartFabric mode to an upstream switch pair, ensure the following: • Both upstream switches must be connected using technologies such as Dell EMC VLT or Cisco Nexus vPC. • The upstream switch ports must be in a port channel using LACP.
Top of Rack (ToR) switches and blade chassis connecting to a different set of ToR switches. If the storage array was Ethernet-based, it can be connected to the core/spine. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 34
In Figure 26, above, there are eight switches in total. In the following figure, using embedded ToR, switch count is reduced to the two MX9116n FSE in the two chassis: Dell EMC PowerEdge MX Networking Architecture Guide...
– displays the unit ID and service tag of the MX7116n FEM attached to an MX9116n FSE • unit-provision – Provisions the unit ID of a Fabric Expander Module that is attached to a Fabric Switching Engine in Full Switch mode Dell EMC PowerEdge MX Networking Architecture Guide...
Chapter 4: PowerEdge MX Ethernet Switch Operating Modes Operating modes description PowerEdge MX Ethernet switches support Dell EMC Networking OS10EE Enterprise Edition (OS10EE). OS10EE is a Network Operating System supporting multiple architectures and environments. When running OS10EE, the MX9116n FSE and the MX5108n operate in one of two modes: •...
Page 38
– Disable spanning tree globally ▪ mac-flush-timer – Set the time used to flush MAC address entries ▪ mode – Enable a spanning-tree mode, such as RSTP or MST ▪ mst – Configure multiple spanning-tree (MST) modes Dell EMC PowerEdge MX Networking Architecture Guide...
Figure 28. Two MX9116n FSEs, different chassis topology Two MX9116n Figure 29 shows two MX9116n FSEs in either fabric slots A1/A2 or B1/B2. The blue links FSEs in the represent two QSFP28-DD ports in a VLTi configuration. same chassis Dell EMC PowerEdge MX Networking Architecture Guide...
Page 40
QSFP28 port. In SmartFabric mode the QSFP28 port is configured to operate at same chassis 40GbE allowing for a valid port channel. Note: See Appendix A for more information. Dell EMC PowerEdge MX Networking Architecture Guide...
QSFP28-DD port groups can be used as uplinks by changing the breakout type and cabling as described in Section 3.2.1. Note: See Appendix A for information about connecting to the Dell EMC Networking Z9100-ON ToR switch.
Page 42
NPIV Proxy Gateway (NPG) and direct attach F_PORT mode, not shown. In this example, two Dell Networking S4148U-ON switches are shown but any FC-capable switch can be used. Note: FCoE uplinks must be physically separate from Ethernet uplinks. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 43
Figure 33 and Figure 34 show two methods for using the MX5108n in a LAN. The two Ethernet only diagrams highlight using either 10GBASE-T ports or QSFP28 ports with breakout for 10GbE or 25GbE uplinks. Apart from the ports and cables that are used, the configuration is the same. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 45
The MX5108n is configured to operate in FSB mode. The MX5108n snoops FIP Bridge packets on FCoE-enabled VLANs and discovers the following information: • End nodes (ENodes) • Fibre channel forwarder (FCF) • Connections between ENodes and FCFs • Sessions between ENodes and FCFs Dell EMC PowerEdge MX Networking Architecture Guide...
Create uplinks from the fabric to the existing network and assign VLANs to those uplinks. Deploy the appropriate server templates to the compute sleds. Note: For more information about deploying a SmartFabric, see the Dell EMC PowerEdge MX SmartFabric Deployment Guide. VLAN...
Page 47
Storage - Data Replication Used for VLANs supporting storage data replication such as for VMware VSAN VM Migration Used for VLANs supporting vMotion and similar technologies VMware FT Logging Used for VLANs supporting VMware Fault Tolerance Dell EMC PowerEdge MX Networking Architecture Guide...
Page 48
OS10EE Version Parameter Value Recommended max VLANs per fabric 10.4.0E(R3S) Recommended max VLANs per uplink Recommended max VLANs per server port Note: These maximums are considered a best practice, not an enforced limit. Dell EMC PowerEdge MX Networking Architecture Guide...
19 and 20 Port group 8 31 and 32 Port group 3 21 and 22 Port group 9 33 and 34 Port group 4 23 and 24 Port group 10 35 and 36 Dell EMC PowerEdge MX Networking Architecture Guide...
Page 51
Dell EMC connections to a Top of Rack (ToR) switch. This example shows how to optimize Networking connections from the MX9116n FSEs on the PowerEdge MX7000 to the Dell EMC Z9100-ON spine Networking Z9100-ON ToR switch. The Z9100-ON groups interfaces into “tiles” with shared memory buffers as shown in Figure 38.
Page 52
Table 7 outlines the recommended port order for connecting chassis with MX9116n FSEs with a pair of Dell EMC Z9100-ON spine switches to optimize buffer utilization. Chassis one and two are in the first group of 10 chassis, the second set of 10 chassis starts with chassis 11 and 12.
Page 54
Figure 40. Two MX5108n in a single chassis When using the SmartFabric wizard for 2xMX5108n Ethernet Switches in same chassis, there is a printout button on the Summary tab of the wizard. The Print option is shown in Figure 41. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 55
Management (MCM) daisy-chain as shown in Figure 43. A password is not needed as the machine to machine authentication trust is used. • Ensure that each member chassis is added to the group using the automatic or manual method. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 56
Note: More information about the configuration of MCM groups can be found in the Dell EMC OpenManage Enterprise-Modular Edition Version 1.00.01 for PowerEdge MX7000 Chassis as well PowerEdge MX7000 Chassis Management Networking Cabling. Dell EMC PowerEdge MX Networking Architecture Guide...
Appendix A: Additional Guidelines and Equipment Details Figure 43. Dell EMC PowerEdge MX9002m module daisy chain cabling Supported cables and optical connectors PowerEdge The PowerEdge MX7000 supports various optics and cables. The sections in this MX700 appendix provide a brief summary of the specified industry standards and the use case supported optics regarding the chassis.
Page 58
8 channels QSFP28-DD = 2x100Gb 16 fibers or wires QSFP56-DD = 2x200Gb MPO12DD 16 fiber parallel optical connector Table 10 shows the location within the chassis, where each type of media is relevant. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 59
Each type of media has a specific use case regarding the MX7000, with each type of media there are various applications. The following sections outline where in the chassis each type of media is relevant. Note: See the Dell EMC Networking Transceivers and Cables document for more information about supported optics and cables. SFP+/SFP28...
Page 60
Figure 47. QSFP+ Transceiver The QSFP+ media type has several uses in the PowerEdge MX7000. While the MX9116n does not have interfaces that are dedicated to QSFP+, ports 41 through 44 can be broken Dell EMC PowerEdge MX Networking Architecture Guide...
Page 61
DAC with a fixed fiber optic cable in between the attached QSFP+ transceivers. Figure 48. QSFP+ cables: Direct Attach Copper (DAC) Figure 49. QSFP+ cables: Multi-fiber Push On (MPO) Cable Dell EMC PowerEdge MX Networking Architecture Guide...
Page 62
Figures 51 and 52 show the DAC and MPO cables, which are two variations of breakout cables. The MPO cable in this example attaches to one QSFP+ transceiver and four SFP+ transceivers. Figure 51. QSFP+ to SFP+ Breakout cables: Direct Attach Copper (DAC) Breakout Dell EMC PowerEdge MX Networking Architecture Guide...
Page 63
Figure 53. Figure 53. QSFP28 transceiver There are three variations of cables for QSFP28 connections. The variations are shown in Figures 54 through 56. Figure 54. QSFP28 cables: Direct Attach Copper (DAC) Dell EMC PowerEdge MX Networking Architecture Guide...
50 shows one of the unified interfaces, port groups 15 and 16. The port group is set to 4x 32 Gb FC mode. However, each of the four lanes is 25 Gbps, not 28 Gbps. When these lanes are mapped from the Network Processing Unit (NPU) to the FC ASIC, for Dell EMC PowerEdge MX Networking Architecture Guide...
112 Gbps for both. Compared to the 200 Gbps using 4x 32 Gb FC. Note: Rate limited ports are not oversubscribed ports. There is no FC frame drop on these ports and buffer to buffer credit exchanges ensure flow consistency. Spanning Tree Protocol recommendations Dell EMC PowerEdge MX Networking Architecture Guide...
RSTP tree processes these packets accordingly leading to the potential of unexpected trees. Note: Dell EMC Networking recommends, that when connecting to an existing environment that is not using RPVST+, to change to the existing STP protocol before connecting a OS10EE switch.
10GBASE-T PTM 25G PTM 25G PTM PTM port mapping The following figures show the port mapping between compute sleds and PTM interfaces. This mapping applies to both 25 GbE and 10 GbE PTMs. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 70
Appendix A: Additional Guidelines and Equipment Details Figure 59. Ethernet PTM Dual Port NIC Mapping Figure 60. Ethernet PTM Quad Port NIC Mapping Note: Ports 9 through 16 are reserved for future expansion. Dell EMC PowerEdge MX Networking Architecture Guide...
MX9116n FSE in slot A2, the same rule applies to devices in B1 and B2. It is not supported to have an MX7116n FEM in slot A2 connected to an MX9116n FSE in slot A1, for example. The following examples are not supported configurations. Dell EMC PowerEdge MX Networking Architecture Guide...
Page 73
Figure 64. Unsupported A1-B1, A2-B2 The following configurations are supported in full switch mode; however, they are not supported configurations for SmartFabric mode. Figure 65. Unsupported VLT configuration Figure 66. Unsupported inter-chassis configuration Dell EMC PowerEdge MX Networking Architecture Guide...
Appendix B: Additional Information Appendix B Additional Information This appendix presents the following topics: Documentation and Support ................75 Dell EMC PowerEdge MX Networking Architecture Guide...
Appendix B: Additional Information Documentation and Support Dell EMC The following Dell EMC documentation provides additional and relevant information. documentation Access to these documents depends on your login credentials. If you do not have access to a document, contact your Dell EMC representative.
Need help?
Do you have a question about the PowerEdge MX Series and is the answer not in the manual?
Questions and answers