Prior to deployment ....................10 CNA’s Used for this deployment document ..............10 Scenario 1: Deploying the Dell Force10 MXL FSB in a Cisco 5000 series switch (NPIV) environment . 11 Configuring the Dell Force10 MXL for FIP-Snooping............14 Configuring the Cisco 5000 series switch with firmware ver 5.x for a single connection from the Dell Force10 MXL ....................
Page 4
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL PFC ........................68 ETS........................69 IOM ........................69 NPIV ........................69 NPV ........................69 VSAN ........................69 Appendix B – Full CLI examples ................... 70 Force10 MXL CLI example ..................70 Cisco Nexus 5548UP CLI example ................
Page 5
Dell Force10 MXL Blade Switch (10/40GbE) and available modules....... 7 Figure 2. FlexIO Modules that can be used as part of the Dell Force10 MXL switch ....... 7 Figure 3. Port mapping for Dell Force10 MXL expansion modules ..........8 Figure 4.
Page 6
FCoE configuration (can be copied and pasted) ........... 54 Figure 47. Flowchart of MXL configuration to connection into dual Nexus vPC configuration... 56 Figure 48. CLI configuration example: configuring Dell Force10 MXL (MXL-1) for uplinking into Cisco vPC(can be copied and pasted) ................. 57 Figure 49.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Introduction The new Dell™ Force10 MXL has the T11, BB-5 compliant FIP- Snooping Bridge featureset. With an available FCF (Fiber Channel Forwarder), this switch will pass FCoE traffic for storage availability.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 3. Port mapping for Dell Force10 MXL expansion modules Dell Force10 MXL - Port Mapping QSFP+ 2 X 40Gb QSFP+ 8 X 10 SFP+ SFP+ 4 X 10Gb 10G-BaseT 4 X 10Gb...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Configuration scenarios The following sections will present very basic examples of deploying the 10G switches for FIP-Snooping and will provide step-by-step explanations of the CLI commands as a guide. Consult the table of contents above for a list of examples covered in this document.
Switches. If the switches are in a stacked configuration stacking should be disabled. Please refer to the Dell Force10 MXL user’s guide or the Dell Force10 MXL Stacking guide for further details on disabling or changing stacking ports. If the configuration is used in this manner proper congestion behavior, lossless Ethernet and reliability cannot be guaranteed.
Scenario 1: Deploying the Dell Force10 MXL FSB in a Cisco 5000 series switch (NPIV) environment This first example is a basic, single connection between devices example using the Dell Force10 MXL. This configuration is being shown for the purposes of simplification and potentially easing into the progression of a more in-depth setup.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL The flowchart in the following figure is a general overview of how the deployment will occur. This includes the basic planning that will need to take place in order for most of the steps in the rest of the document to fall into place.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 7. Deployment planning and configuration continued from previous flowchart – Check basic ethernet connectivity on the applicable VLAN Missing Basic Basic Ethernet Ethernet Connectivity Connectivity Good · Check physical links.
Configuring the Dell Force10 MXL for FIP-Snooping The Dell Force10 MXL switches will monitor FIP packets and will establish the proper filtering, and priorities for the FCoE traffic that is passed through the configured links. To see an example of the full...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Command-Line Interface Method Figure 9. Example commands for Dell Force10 MXL (can be copied and pasted) feature fip-snooping protocol lldp exit service-class dynamic dot1p default vlan-id 20 interface TenGigabitEthernet 0/1 portmode hybrid...
Page 16
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Step by step explanation of CLI example configure – this brings the prompt into the configuration interface. feature fip-snooping – This configures the global enablement of the FIP-Snooping feature on the switch.
Page 17
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL tagged TenGigabitEthernet 0/1,52 – This adds these ports to the tagged VLAN 1000. fip-snooping enable – Enables FIP-snooping for this VLAN which will set all ports involved to have MTU or Jumbo Frames (mini) of size 2500 no shutdown –...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Configuring the Cisco 5000 series switch with firmware ver 5.x for a single connection from the Dell Force10 MXL Figure 10. Cisco Nexus 5000 series configuration sequence Connect Cisco Configure FC ports and...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL The following CLI commands are necessary for an un-configured Cisco 5548UP. The CLI will show additional lines that are either default or can’t be changed and are not added for this example. The CLI will also show the lines in a different order after they have been entered.
Page 20
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Step by step explanation of the CLI example Configure – this brings you into the configuration interface. feature FCOE – enables the feature for FCOE as long as the licensing and FC modules are installed.
Special instructions concerning deployment with Intel x520 CNA The Intel x520 CNA is one of the various choices for CNA’s that can be used with the Dell PowerEdge Blade Servers. With the card installed it will necessary to insure that the ISCSI portion of the driver is not installed.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL This is what the downlink (server facing port) additional settings should look like. These additional command line “dcbx version CEE” will configure the port to only use the CEE version of DCBX and not negotiate other values. This setting is currently necessary for the Intel x520 CNA for use with FCoE and applicable storage.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Basic Validation for the Cisco Nexus 5000 series switch configuration In order to have a fully supported configuration the cables or SFP+ transceivers must only be Cisco- branded products for the Cisco Nexus. The following command is used to give a port status overview: show interface brief.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Check the spanning tree configuration. Blocking ports should be understood and when they show in this entry it should be the number expected. Otherwise it could be that an unintentional cable loop has been created that will need to be resolved.
In other words if the Cisco Nexus 5548 has been configured based on the CLI examples in this document but the Dell Force10 MXL has not been configured many of the results shown here will not match. The storage device and SAN switches will also need to be configured if using NPV mode.
------- -------- 1000 TRUE 0X0EFC00 The Force10 MXL’s FIP-snooping FCF connection information can be checked by using: show fip-snooping fcf. This command will display the FCF’s MAC address, the connecting interface and applicable VLAN configured for the connection. Figure 21.
NOTE: The above figure “show fip-snooping sessions” has some display overlap which will be somewhat typical based on screen size of the terminal being used. Using the Force10 MXL, fip-snooping statistics can be validated using: show fip-snooping statistics. This command will display the various counters applicable to the fip- snooping feature Figure 24.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL NOTE: the previous figure “show fip-snooping statistics” was continued from previous page. Also only the internal port that is currently connected to an FCoE CNA is displayed here. When the command is run all the ports will be shown. If the ports are not involved in FCoE the counters will display 0, These counters are also cumulative so they will capture more and more results over time.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL The following command will show vital information on the DCBX configuration. Type: show interfaces dcbx detail. This command will show more specific detail about the configuration that has been negotiated between the devices. In this example interface TenGigabitEthernet 0/3 is used as a downlink to a host server and CNA installed in that system.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 27. “show interfaces dcbx detail” - continued Local DCBX Status ----------------- DCBX Operational Version is 0 DCBX Max Version Supported is 0 Sequence Number: 1 Acknowledgment Number: 1 Protocol State: In-Sync...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 28. “show interfaces pfc detail” MXL-B1-Rack3015# show interfaces pfc detail Interface TenGigabitEthernet 0/3 Admin mode is on Admin is enabled Remote is enabled, Priority list is 3 Remote Willing Status is enabled...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL The following command will show vital information on the ETS settings in the configuration. Type: show interfaces ets detail. This command will show more specific detail about the ETS configuration that has been negotiated between the devices. In this example interface TenGigabitEthernet 0/3 is used as a downlink to a host server and CNA installed in that system.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 30. “show interfaces ets detail” - continued Remote Willing Status is enabled Local Parameters : ------------------ Local is enabled TC-grp Priority# Bandwidth ------------------------------------------------ ------------------------------------------------ 0,1,2,4,5,6,7 50 % 50 % Oper status is internally propagated...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Basic Troubleshooting Areas Figure 32. Basic troubleshooting areas Problem Potential Problem Area Potential Fix Basic Connectivity Between CNA and MXL Series switch is not present (ping Connect or Change cable and...
Page 36
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Problem Potential Problem Area Potential Fix Check VFC entries applicable to the connection and make sure "no ENode's or FCF entries are not present The VFC entry may not be enable (no shutdown"...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Scenario 2: Configuring multiple uplinks into LAG for Cisco Nexus 5000 series switch (switch - NPIV) environment Figure 33. Multiple port link (LAG) configuration between switches and storage CISCO NEXUS N5548P...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Configuring the Cisco Nexus 5000 series switch for a multiple link LAG/PortChannel (link aggregation) connection at the Top-of-Rack. The typical scenario in a business environment consists of more than one connection or uplink for bandwidth and redundancy.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Command-Line Interface Figure 35. Multiple port link (port-channel/LAG) Cisco Nexus 5548 configuration feature lacp vlan 1000 fcoe vsan 2 vsan database vsan 2 vsan 2 interface vfc1 vsan 2 interface fc2/15...
Page 40
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Step by step explanation of CLI example (only covering the differences from the single port configuration) interface port-channel1 switchport mode trunk - switchport trunk native vlan 20 switchport trunk allowed vlan 1000 ...
Configuring the Dell Force10 MXL for FIP-Snooping This is an example of the necessary CLI commands for Dell Force10 MXL switch configured with two links in a LAG or port-channel uplinked to the Cisco Nexus 5548UP. This provides more bandwidth and fail-over capability which would be more typical in the larger enterprise or datacenter infrastructure.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Command-Line Interface Method Figure 37. Multiple port uplink (LAG) MXL configuration (can be copied and pasted) configure default vlan-id 20 interface TenGigabitEthernet 0/1 no ip address mtu 2500 portmode hybrid switchport...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 38. Multiple port uplink (LAG) MXL configuration (can be copied and pasted) - continued interface Port-channel 5 no ip address mtu 2500 portmode hybrid switchport fip-snooping port-mode fcf no shutdown...
Page 44
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Step by step explanation of Dell Force10 MXL CLI example with multiple port (LAG) uplink (this only covers the differences from the single port configuration). interface TenGigabitEthernet 0/1 – configure internal interfaces that connect to the blade servers.
Page 45
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL dcbx port-role auto-upstream – with this command the port-role is established as a dcbx capable upstream port to communicate with top-of-rack FCF switches such as the Cisco Nexus 5000 series. no shutdown – configures the port as active.
Cisco Nexus ToR is many times set in an aggregation mode or NPV to simply pass FC out to another terminating switch (in our example Brocade 300 or 5100).The following pictures and examples describe a two link LAG from a Dell Force10 MXL to the Cisco 5548UP with the Cisco switch set in NPV mode. Command-Line Interface Method Figure 40.
The FC interfaces will no longer show up and the “show flogi database” command is no longer available. Configuring the Dell Force10 MXL for FIP-snooping with Cisco Nexus 5000 series switch in NPV mode. The configuration will be the same as previously mentioned in the single or multiple port...
The following scenarios will step through setting up the vPC configuration and then adding FCoE to the configuration. The naming conventions for the two Cisco Nexus 5000 Series Switches will be Nex5K-1 for the first switch and Nex5K-2 for the second. The Dell Force10 MXL switches are named similarly. MXL-1, and MXL-2.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Configuring the Cisco 5000 series switch for vPC and connections from the Dell Force10 MXL Figure 43. Initial vPC configuration without FCoE links Management LAN Peer Keep-Alive Peer Keep-Alive vPC Peer Link...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 44. Flow of vPC configuration on Cisco Nexus 5000 series switches Connect to first Configure vpc Cisco Nexus 5000 Enable features domain and peer- Series Switch keepalive feature vpc vpc domain 1...
Page 52
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Key Items: · vPC domain number can be any chosen number as long as it does not conflict with other switches in the infrastructure, but it also match on both vPC peer switches.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Configuring the Dell Force10 MXL switch for a PortChannel connection into Cisco Nexus 5000 series switch with vPC configured Figure 47. Flowchart of MXL configuration to connection into dual Nexus vPC configuration.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 48. CLI configuration example: configuring Dell Force10 MXL (MXL-1) for uplinking into Cisco vPC(can be copied and pasted) configure default vlan-id 20 interface TenGigabitEthernet 0/1 no shutdown exit interface Port-channel 5...
Page 58
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Step by step explanation of Dell Force10 MXL CLI example with multiple port (LAG) uplink to Cisco vPC configuration. configure default vlan-id 20 – globally established VLAN 20 as the default VLAN interface TenGigabitEthernet 0/1 no shutdown –...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Example commands to validate vpc settings With the Cisco Nexus 5000 series switch, one way the vPC configuration can be validated is by using: show vpc brief. This command will display the following items that are key to the vPC behavior: ·...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Configuring the Cisco 5000 series switch with vPC and connections from the Dell Force10 MXL with FCoE Figure 50. vPC Configuration with FCoE links Management LAN Peer Keep-Alive Peer Keep-Alive vPC Peer Link...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 51. Flow of additional configuration for FCoE on Cisco Nexus 5000 series switch with vPC Configure FC Connect to first Enable additional ports and settings Configure VSAN(s) Cisco Nexus 5000...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 52. CLI configuration example: vPC on Cisco Nexus 5000 Series switch (secondary switch), with FCoE configuration (can be copied and pasted) feature fcoe feature npiv slot 2 port 15-16 type fc...
Page 63
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL feature FCOE – enables the feature for FCOE as long as the licensing and FC modules are installed. slot 2 – enter into the slot 2 configuration port 15-16 type fc – at slot 2 port 15-16 set the unified ports to be type FC vlan 20 –...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Configuring the Dell Force10 MXL switch with connections to Cisco Nexus vPC configuration with FCoE Figure 53. Flowchart for MXL FCoE configuraton to Nexus vPC Turn on FCoE Connect to (fip-snooping...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Figure 54. CLI example of MXL FCoE configuration with LAG/PortChannel uplinked to Cisco Nexus vPC configuration feature fip-snooping protocol lldp exit service-class dynamic dot1p default vlan-id 20 interface TenGigabitEthernet 0/1 portmode hybrid...
Page 66
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Step by step explanation of CLI example service-class dynamic dot1p – This sets the switch to globally honor dot1p priorities for traffic on the switch. feature fip-snooping – This configures the global enablement of the FIP-Snooping feature on the switch.
Page 67
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL portmode hybrid - configuring portmode hybrid will allow for the native VLAN untagged, and a tagged VLAN to exist on the port together. This is similar to switchport mode trunk in industry standard CLI.
Appendix A – Basic terminology Command line interface (CLI). This is the text based telnet, ssh, or serial type interface that will be used for entering commands into the Dell Force10™ MXL model switch. At release the MXL is only configurable via CLI.
(IEEE, 2011). IO module (IOM) refers to the modules on the rear of the Dell PowerEdge M1000e chassis that will receive and transmit I/O (Ethernet, FC, Infiniband, etc) from the blade servers. The Dell Force10 MXL switch is as an IOM for the M1000e blade server chassis.
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Appendix B – Full CLI examples Force10 MXL CLI example MXL-B1-Rack3015#show run Current Configuration ... ! Version E8-3-16-123 boot system stack-unit 0 primary system: B: boot system stack-unit 0 default system: B: boot system gateway 172.25.151.254...
Page 71
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL no ip address shutdown interface TenGigabitEthernet 0/6 no ip address shutdown interface TenGigabitEthernet 0/7 no ip address shutdown interface TenGigabitEthernet 0/8 no ip address shutdown interface TenGigabitEthernet 0/9 no ip address...
Page 72
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL shutdown interface TenGigabitEthernet 0/20 no ip address shutdown interface TenGigabitEthernet 0/21 no ip address shutdown interface TenGigabitEthernet 0/22 no ip address shutdown interface TenGigabitEthernet 0/23 no ip address shutdown interface TenGigabitEthernet 0/24...
Page 73
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL interface fortyGigE 0/37 no ip address shutdown interface TenGigabitEthernet 0/41 no ip address no shutdown interface TenGigabitEthernet 0/42 no ip address no shutdown interface TenGigabitEthernet 0/43 no ip address no shutdown...
Page 74
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL no advertise dcbx-tlv ets-reco dcbx port-role auto-upstream no shutdown interface TenGigabitEthernet 0/53 no ip address shutdown interface TenGigabitEthernet 0/54 no ip address shutdown interface TenGigabitEthernet 0/55 no ip address shutdown interface TenGigabitEthernet 0/56...
Page 75
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL ---------------- additional MXL config example with vPC on Cisco Nexus and separate LAG/PortChannel configured for FCoE MXL-1 SHOW RUNNing-config Current Configuration ... ! Version E8-3-16-166 ! Last configuration change at Thu May...
Page 76
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL shutdown interface TenGigabitEthernet 0/6 no ip address portmode hybrid switchport shutdown interface TenGigabitEthernet 0/7 no ip address portmode hybrid switchport shutdown interface TenGigabitEthernet 0/8 no ip address portmode hybrid switchport shutdown...
Page 77
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL no ip address portmode hybrid switchport shutdown interface TenGigabitEthernet 0/16 no ip address portmode hybrid switchport shutdown interface TenGigabitEthernet 0/17 no ip address portmode hybrid switchport shutdown interface TenGigabitEthernet 0/18 no ip address...
Page 78
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL shutdown interface TenGigabitEthernet 0/25 no ip address portmode hybrid switchport shutdown interface TenGigabitEthernet 0/26 no ip address portmode hybrid switchport shutdown interface TenGigabitEthernet 0/27 no ip address portmode hybrid switchport shutdown...
Page 79
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL interface TenGigabitEthernet 0/41 no ip address shutdown interface TenGigabitEthernet 0/42 no ip address shutdown interface TenGigabitEthernet 0/43 no ip address shutdown interface TenGigabitEthernet 0/44 no ip address shutdown interface TenGigabitEthernet 0/45...
Page 80
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL port-channel-protocol LACP port-channel 12 mode active no shutdown interface TenGigabitEthernet 0/53 no ip address shutdown interface TenGigabitEthernet 0/54 no ip address shutdown interface TenGigabitEthernet 0/55 no ip address shutdown interface TenGigabitEthernet 0/56...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL line console 0 line vty 0 exec-timeout 60 0 line vty 1 line vty 2 line vty 3 line vty 4 line vty 5 line vty 6 line vty 7 line vty 8...
Page 82
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL service-policy type network-qos fcoe-default-nq-policy slot 1 port 31-32 type fc snmp-server user admin network-admin auth md5 0x18e86ba84d1d511fcb6fe4b5e02dc408 priv 0x18e86ba84d1d511fcb6fe4b5e02dc408 loca lizedkey snmp-server enable traps entity fru vrf context management ip route 0.0.0.0/0 172.25.188.254...
Page 83
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL vsan 2 interface fc1/32 feature npv feature npiv interface fc1/31 switchport trunk mode on no shutdown interface fc1/32 switchport trunk mode on no shutdown interface Ethernet1/1 interface Ethernet1/2 interface Ethernet1/3 interface Ethernet1/4...
Deploying FCoE (FIP Snooping) on Dell Force 10 MXL Appendix B - Network switch versions Version information for the network switches used in creating this document are as follows: Table 1. Switch Firmware Versions Network switch Dell Force10 MXL Cisco 5020...