Page 1
Front cover Nortel Networks L2/3 Ethernet Switch Module for server BladeCenter Full Layer 2 switching and Layer 3 routing Six external multimode fiber or copper GbE interfaces Hot pluggable switch modules Rufus Credle Stephan Fleck Scott Lorditch Jeremy Oliver Redpaper...
Page 3
International Technical Support Organization Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter September 2005...
IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead.
Cisco, Cisco IOS, Cisco Systems, the Cisco Systems logo, EtherChannel are Registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries. Other company, product, and service names may be trademarks or service marks of others. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter viii...
IBM Bermuda Become a published author Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners, or customers.
Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an email to: redbook@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HQ7 Building 662 P.O. Box 12195 Research Triangle Park, NC 27709-2195 Preface...
Page 14
Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
IBM Eserver BladeCenter. The customer can install as few as one Nortel Networks L2/3 GbESM or as many as four Nortel Networks L2/3 GbESMs in one BladeCenter chassis. With four Nortel Networks L2/3 GbESMs installed, you can obtain 24 GbE uplink interfaces as well as 56 GbE internal switching capability.
Page 16
The Nortel Networks Layer 2/3 Copper and Fiber GbE Switch Modules for IBM Eserver BladeCenter is an integral part of these solutions. With the Nortel Networks L2/3 GbESM, you have the investment protection and price performance of a solution behind which the world’s leading server and networking companies stand.
BladeCenter architecture. If you seek to know more about the IBM Eserver BladeCenter and its components, visit: http://www.ibm.com/products/us/ Figure 2-1 on page 5 shows the IBM Eserver BladeCenter chassis, HS40, HS20, JS20, and LS20: IBM Eserver BladeCenter chassis...
IBM Eserver JS20 Figure 2-1 IBM Eserver BladeCenter and blade modules Blade development is ongoing for the BladeCenter platform. Therefore, we suggest that you regularly visit the following Web site for the latest information about IBM Eserver BladeCenter: http://www.ibm.com/servers/eserver/bladecenter/index.html 2.1.1 IBM Eserver BladeCenter storage solutions IBM delivers a wide range of easy-to-install, high-capacity, tested storage products for the IBM Eserver BladeCenter to meet your demanding business needs.
For more information about advanced server management, see: http://www-1.ibm.com/servers/eserver/xseries/systems_management/xseries_sm.html Deployment and update management IBM deployment tools help minimize the tedious work that can be involved in getting your servers and clients ready to run. These tools include: Remote Deployment Manager Software Distribution Premium Edition ServerGuide™...
Page 21
Module Figure 2-2 Midplane view It should be noted that the upper and lower midplane sections in an IBM Eserver BladeCenter are independent of each other (see Figure 2-3). Having a dual midplane ensures that there is no single point of failure and the blades remain operational.
Each switch module has two 100 Mb Ethernet interfaces, one for each Management Module. Note: On the Nortel Networks L2/3 GbESM, the management Ethernet ports on the switch are referred to as MGT1 and MGT2. For more information beyond this generic illustration, see Chapter 4, “Nortel Networks Layer 2/3 GbE Switch Module architecture”...
Each switch module (SW Module) receives one LAN input from each processor blade, for a total of 14 inputs. Note: On the Nortel Networks L2/3 GbESM, the internal Ethernet ports on the switch are referred to as MGT1 and MGT2. For more information beyond this generic illustration, see Chapter 4, “Nortel Networks Layer 2/3 GbE Switch Module architecture”...
Figure 2-5 Gigabit Ethernet path 2.3 IBM Eserver HS20 architecture In this section, we discuss the architectural design of the IBM Eserver BladeCenter HS20. This is presented as just one example of the blade design for a typical dual-processor server.
Page 25
The LPC bus is used to connect to the POST/BIOS EEPROM on the 8843. The size of the EEPROM is 4 MB x 8, and it contains primary BIOS, backup BIOS, and blade diagnostics. Chapter 2. IBM Eserver BladeCenter overview...
High-bandwidth, low pin-count implementations for optimized performance 2.4 Stand-alone configuration tools IBM Eserver BladeCenter hardware can be configured using standard software, such as a Web browser and a Telnet client, which are available on all the mainstream operating system platforms. This is possible by exploiting Web and American National Standards Institute (ANSI) interfaces that are embedded in both the management and the Ethernet Switch Modules.
If Module is plugged into If Module is plugged into Rear Bay 2 Rear Bay 2 192.168.70.129 192.168.70.129 Rear Bay 3 Rear Bay 3 192.168.70.130 192.168.70.130 Rear Bay 4 Rear Bay 4 Figure 2-7 Stand-alone configuration tools Chapter 2. IBM Eserver BladeCenter overview...
Page 28
Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
I/O module bays of the BladeCenter chassis. The modules can be hot-plugged into an IBM Eserver BladeCenter without disrupting normal operations. The Nortel Networks L2/3 GbESM connects to the server blades via the 14 internal GbE interfaces (server ports) over the BladeCenter midplane. It supplies six external copper or multimode fiber GbE interfaces for outside communication (shown in Figure 3-1).
Page 31
Unmatched High Availability support and field proven Resiliency High availability support is built in at both Layer 2 and Layer 3 in the Nortel Networks L2/3 GbESM to reduce single points of failure when it comes to enabling reliability and performance of the network.
Leadership features and function The leadership features and function include: IBM Eserver BladeCenter delivers with the Nortel GbESM, full Layer 2 switching and Layer 3 switching (routing) functionality as well as Layer 4 filtering and related services. The switch module runs Alteon Operating System and appears as any other product from Nortel's Alteon product line to the data center’s network management tools.
3.3 Supported hardware Table 3-1 lists the following IBM hardware platforms which support Nortel Networks Layer 2/3 Copper Gigabit Ethernet Switch Module for IBM Eserver BladeCenter (26K6530) and Nortel Networks Layer 2/3 Fiber Gigabit Ethernet Switch Module for IBM Eserver BladeCenter (26K6531).
Page 34
Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
GbESM 2. The links connecting the blade servers to the Nortel GbESMs are on the backplane of the BladeCenter chassis. The Nortel GbESM has two links to the Management Modules. Each link connects to a different Management Module. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 37
Management Module if Spanning Tree is enabled for the Management Module ports. This is rarely necessary. 6 External ports Hard-coded filter 2 Management CPU/AOS Module ports 14 internal blade ports Figure 4-3 Layer 2 frames flow in the Nortel Networks L2/3 GbESM Chapter 4. Nortel Networks Layer 2/3 GbE Switch Module architecture...
The two GEAs are interconnected with 10 Gigabit proprietary link, which is shown as the 10G HiGig link in Figure 4-4. HiGig is a proprietary protocol from IBM. GEA0 supports eight Gigabit Ethernet ports (two internal connections-5421s to the Management Modules and six external ports).
4.2.1 Nortel Networks L2/3 GbESM ports specific roles Figure 4-5, Figure 4-6, and Figure 4-7 on page 26 show different examples of the port connections to the Nortel GbESM(s) within the IBM Eserver BladeCenter. We then discuss the specific roles and restrictions for the various ports.
Page 40
BladeCenter Chassis (BSS = Blade Server Slot) Figure 4-7 Overall view of port connections within an IBM Eserver BladeCenter Ports INT1 through INT14: Connects to blade server slots 1 through 14, respectively: Preset default values for ports going to the blade servers (includes ports INT1 through INT14): –...
Page 41
Preset default values for ports going to external connections (includes ports EXT1 through EXT6): Untagged and configured on VLAN 1. These ports default to Disabled when in a new IBM Eserver BladeCenter. You must use the Management Module Web interface, under I/O Module tasks Advanced settings, to set External Ports to Enabled to bring them up the first time.
Page 42
Switch bay 3: 192.168.70.129/24 Switch bay 4: 192.168.70.130/24 Based on certain interactions within the IBM Eserver BladeCenter, it is usually recommended to change the management IP address directly on the Nortel GbESM, but instead, only change it through the Management Module Web-based GUI.
In this chapter, we discuss tools, techniques, and applications that help with the management and deployment of the Nortel GbESM in an IBM Eserver BladeCenter. We also discuss the management paths and rules for connecting to and accessing the Nortel GbESM.
In this section, we look at the basic management connectivity and management pathways to the Nortel GbESM., as shown in Figure 5-1. Important: Properly managing the Nortel GbESM in the IBM Eserver BladeCenter actually requires proper management of the Management Module within the BladeCenter chassis.
Management Module supports an external Ethernet interface, which is used to manage the Blade servers, Ethernet switches, and the Management Module itself. Within the IBM Eserver BladeCenter, management traffic flows through a different bus, the I2C bus, as shown in the Figure 5-1 on page 30.
Further configuration of the Nortel Networks L2/3 GbESM module is performed by using Telnet (for the Command Line Interface) or a Web browser (for the Browser Based Interface) and accessing the address of the MGT1 or 2 ports. Note: It is recommended (and easier) to use a server or mobile computer that is external to the IBM Eserver BladeCenter chassis to perform initial configuration of the Nortel Networks L2/3 GbESM module.
Page 47
In order to use in-band management paths, you must configure at least one additional IP address on the Nortel Networks L2/3 GbESM beyond the address that is provided through the Management Module and attached to VLAN 4095. This additional IP address should be attached to one of the active VLANs configured on the switch and is discussed in detail in 7.5.3, “Base configuration common to all examples”...
The boot menu contains the commands to control the booting of the switch, from which image to boot , which config to boot, and the gtimg and ptimg commands for getting and putting firmware files to the switch. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 49
Figure 5-4 shows what each of the commands does. The help command is also global and lists all the global commands, as shown in Figure 5-5. Figure 5-5 CLI global command list Chapter 5. Nortel Networks L2/3 GbESM management and administration...
Page 50
Opens a Telnet session to the designated IP address. Tailors the level of messages displayed on your session. verbose lines Controls the number of lines per screen for display purposes. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 51
Figure 5-6 Display of a firmware update using CLI The firmware for the Nortel Networks L2/3 GbESM is contained in two files: one is a boot image file and the other is the OS image file. Use the following steps to upgrade the firmware on the Nortel Networks L2/3 GbESM via the Telnet session: 1.
Page 52
Example 5-1 Example configuration file dump >> Main# /cfg/dump script start "Layer 2-3 Gigabit Ethernet Switch Module for IBM eServer BladeCenter" 4 /**** DO NOT EDIT THIS LINE! /* Configuration dump taken 2:49:25 Sun Jan 4, 2070 /* Version 1.1.0.6, Base MAC address 00:11:f9:36:b7:00 /c/sys/access/user/uid 1 name "USERID"...
Configuring user accounts This section describes the user accounts on the switch. There are multiple modes of authentication which are supported on the Nortel Networks L2/3 GbESM: The default mode is to support passwords without individual user IDs. When accessing the CLI in this mode, there is only a single prompt to enter the password.
5.2.3 Browser Based Interface We now take a brief look at the Browser-Based Interface (BBI) on the Nortel Networks L2/3 GbESM. Almost everything that can be done via the CLI can also be done in the BBI. In the remainder of this book, more emphasis is placed on configuring the switch using the CLI rather than using the BBI.
SNMPV3 as well as support for SNMP versions 1 and 2. SNMP Management Information Base (MIB files are provided with every software image. These files can be imported to the MIB compiler, which is included with IBM Director and Chapter 5. Nortel Networks L2/3 GbESM management and administration...
5.3 Multiple Nortel Networks L2/3 GbESMs in a BladeCenter If there are two (or more) switches in a single IBM Eserver BladeCenter chassis, the management (MGTx) interfaces of all of the switches are on VLAN 4095. This has the...
6.1 IBM Eserver BladeCenter system In this section, we discuss the stages of preparing IBM Eserver BladeCenter for operation. 6.1.1 Management Module firmware After the required hardware has been installed in your IBM Eserver BladeCenter, you should update the Management Module using IBM Eserver® BladeCenter - Management Module Firmware Update Version 1.18 or later.
In this section, we configure the Management Module external and internal network interfaces to exist upon the management subnet. The external network interface IP address is attached to the network outside of the IBM Eserver BladeCenter. This is the address used to contact the Management Module from an external device.
Page 60
IP connectivity from the Management Module to your switches modules. Figure 6-3 Management Module Internal Network Interface window Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Nortel Networks L2/3 GbESM setup and configuration You can install the Nortel Networks L2/3 GbESM into any of the four BladeCenter switch bays in the rear of the chassis. Bay 1 is attached to one of the Ethernet Network Interfaces Controllers (NIC) on the blade HS20.
Page 62
Management Module. If this field is set to Enabled, be sure a valid IP configuration is entered for this switch module in the Management Module settings for Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 63
In this section, we load the latest version of the switch module’s firmware. Determining the level of Nortel switch software After you install the Nortel Networks L2/3 GbESM in your BladeCenter unit, make sure that the latest Nortel switch operating system is installed on the module. To determine the level of the Nortel switch operating system software that is installed on the switch module: 1.
Otherwise, you must use a router. Use the Management Module graphical interface to configure the IP addresses of the Management Module external network interface (eth0) and the Nortel Networks L2/3 GbESM so that they are on the same subnet as the TFTP server. Note: Alternatively, you can use FTP instead of TFTP.
Page 65
Monitors → Firmware VPD on the Management Module Web interface, which opens a window similar to the one shown in Figure 6-6 on page 52. Chapter 6. IBM Eserver BladeCenter system initial setup...
In this section, we prepare the use of our operating systems for the BladeCenter HS20s. Note you can use IBM Director and Remote Deployment Manager (RDM) to customize and deploy your network operating systems to the HS20s. However, if you are building your network operating system manually, after Windows 2003 Server has been successfully installed, go the next section 6.2.3, “Broadcom Advanced Control Suite installation”...
IBM Eserver 325, 326 IBM Eserver MXE-460 IBM Eserver xSeries 205, 225 (Type 8647), 226, 235, 236, 255, 305, 335, 336, 346, 365, 440, 445, 450, 455, 460 IBM IntelliStation A Pro (Type 6224) IBM IntelliStation E Pro (Type 6216, 6226)
Page 68
2. Click MANAGEMENT PROGRAMS, and a window similar to Figure 6-8 opens. Figure 6-8 Select Features window 3. Select Control Suite and BASP. 4. Click Next to continue, and then click Finish. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Cisco refers to this as a and uses the switchport trunk command to configure it. On the Nortel Networks L2/3 GbESM, the internal ports have tagging enabled by default to provide support for Serial over LAN functionality. The external ports have tagging disabled.
RIP. All of the products tested support version 2 of OSPF. To configure OSPF on the GbESM, use commands on the /cfg/l3/ospf menu. Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
The remaining sections in this chapter present several sample configurations which exploit different capabilities of the Nortel Networks L2/3 GbESM. It is not intended that any of these samples be copied exactly and used in a real network. However, portions of one or more of them can serve as the basis for a real configuration.
These configurations are provided to show that the GbESM switch modules can participate in RIP or OSPF networks if it is decided that this is the best way to integrate them with the Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
In most cases, you should configure trunk failover on all Nortel Networks L2/3 GbESM in the IBM Eserver BladeCenter if the server blades are running NIC Teaming. These two features work together to provide a High Availability design.
In the IBM Eserver BladeCenter, NIC teaming will detect the failure of a NIC chip on the server blade, the loss of connection to a switch module via the midplane, and the failure of a switch module (including intentional removal or power-off).
Use of these options can allow a backup switch to take over even if the current master is still running but has lost some of the tracked resources. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Based on this, we strongly recommend that you use a cross-over cable between the Nortel Networks L2/3 GbESMs and upstream switches. This helps ensure that the link always works under all possible conditions.
The fiber connectors on the Nortel Networks Layer 2/3 Fiber Gigabit Ethernet Switch Module for IBM Eserver BladeCenter must be Multi-Mode Fiber (MMF). Either 50 or 62.5 micron fiber can be used. Single-Mode (9 micron) fiber is not supported. The fiber should be terminated with LC connectors on both ends.
This document was prepared using the features and syntax from the aforementioned revisions of code, and as such, might vary from other revisions. For complete and current lists of available features and commands for these products, visit the IBM or Nortel Web sites.
This section lays out the first configuration from which each continuing section builds. This topology uses 802.1Q tagging for carrying several different networks. The topology also includes static port aggregation (Etherchannel) with trunk failover. Something that should be Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Server Management 10.10.0.3 10.99.0.4 Workstation 10.20.0.1 10.10.0.2 10.20.0.3 10.20.0.2 BladeCenter Links between Management Modules and GbESMs not shown Figure 7-3 Basic Layer 2 topology with 802.1Q tagging and trunk failover Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 85
Gi0/7, Gi0/8, Gi0/9, Gi0/10 Gi0/11, Gi0/12, Gi0/13, Gi0/14 Gi0/15, Gi0/16, Gi0/17, Gi0/18 Gi0/19, Gi0/20, Gi0/21, Gi0/22 Gi0/25, Gi0/26, Gi0/27, Gi0/28 native active vlan_green active vlan_red active mgmt active 1002 fddi-default act/unsup Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 86
Cisco, # for Extreme) are for the counterpart switch of the same type, GbESM_2 or Core2. To start setting up the ports: 1. Access the interface level for g0/1 and g0/2 with the following: interface range g0/1-2 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 87
1. Tagging on the INT ports is enabled by default due to all INT ports being members of VLAN 1 and 4095: /c/port EXT1 tag ena /c/port EXT2 tag ena Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 88
EXT1 rem EXT2 Unlike the Cisco switch, VLAN information is included when dumping the switch configuration with the /c/d command. VLAN information can be displayed with the /i/l2/vlan command. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 89
Additionally, the sysName was set in order to easily identify the switch. All configuration changes were applied and then saved to the flash: /c/sys/ssnmp name “GbESM_1” /* name “GbESM_2” apply save Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 90
Example 7-9 Basic topology GbESM configuration - tagging and trunk failover >> Main# /c/d script start "Layer 2-3 Gigabit Ethernet Switch Module for IBM eServer BladeCent er" 4 /**** DO NOT EDIT THIS LINE! /* Configuration dump taken 13:44:45 Wed Jun 22, 2005 /* Version 1.0.1.6, Base MAC address 00:11:f9:36:b7:00...
Page 91
Core1 ! hostname Core2 enable password cisco no aaa new-model ip subnet-zero no ip domain-lookup no file verify auto spanning-tree mode pvst Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 93
! ip address 10.99.0.246 255.255.255.0 ip default-gateway 9.42.171.3 ip classless ip http server ip http secure-server control-plane line con 0 line vty 0 4 password cisco no login line vty 5 15 no login Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
7.6, “Basic Layer 2 entry topology” on page except that IEEE 802.ad dynamic port aggregation (LACP) is used on all the upstream connections rather than the static Etherchannel. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 95
Blade Blade Blade Server Server Server Server Management Workstation 10.10.0.3 10.99.0.4 10.20.0.1 10.10.0.2 10.20.0.3 10.20.0.2 BladeCenter Links between Management Modules and GbESMs not shown Figure 7-5 Advanced Layer 2 topology Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
G0/11-12 on Core1 are configured as LACP trunks as well, including IEEE 802.1Q tagging (VLAN 5 untagged/native). Remember to shut down the ports before starting the configuration changes as mentioned before. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 97
69, the static port aggregation for ports EXT1 and EXT2 must be disabled with the /c/trunk 1/dis command. Finally apply must be entered to execute the configuration changes. Remember that the ports EXT1, EXT2, EXT5, and EXT6 are still disabled. Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
The GbESM interoperates with Cisco Per VLAN Spanning Tree (PVST) protocol using up to 16 Spanning Tree groups. This section shows how to configure separate Spanning Tree Groups (STG) for each VLAN using Cisco’s proprietary PVST and IEEE 802.1D (Common/Mono Spanning Tree). Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 99
VLANs. This is done by setting different root bridge priorities or port costs for different VLANs Some known issues which arise when connecting PVST and plain Mono Spanning Tree can be avoided. For more information, ssee IBM Eserver BladeCenter Layer 2-7 Network Switching, REDP-3755, at: http://www.redbooks.ibm.com/redpapers/pdfs/redp3755.pdf...
Page 100
1000 full down INT11 1000 full down INT12 1000 full down INT13 1000 full down INT14 1000 full down MGT1 full MGT2 full disabled EXT1 1000 full EXT2 1000 full Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 103
For Cisco IOS devices, some commands you can use to verify the desired configuration and operation of the equipment are: show int status show etherchannel port show lacp internal show spanning-tree Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 104
/* Version 1.0.1.6, Base MAC address 00:11:f9:36:b7:00 /* GbESM_1 /c/sys hprompt ena /c/sys/access/user/uid 1 name "USERID" pswd "cbd31b7a4b020a2a86b6f2b388b2a9d8e8b4271b97d91dc22045f70228a24127" cos admin /c/sys/ssnmp name "GbESM_1” name "GbESM_2" on GbESM_2 /c/port INT1 pvid 20 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 105
/c/l2/stg 5/add 99 /c/l2/lacp/port EXT1 mode active /c/l2/lacp/port EXT2 mode active adminkey 17 /c/l2/lacp/port EXT5 mode active /c/l2/lacp/port EXT6 mode active adminkey 21 /c/l3/if 99 addr 10.99.0.243 addr 10.99.0.244 on GbESM_2 Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 108
10.10.0.246 255.255.255.0 on Core2 interface Vlan20 ip address 10.20.0.245 255.255.255.0 ip address 10.20.0.246 255.255.255.0 on Core2 interface Vlan99 ip address 10.99.0.245 255.255.255.0 ip address 10.99.0.245 255.255.255.0 on Core2 ip default-gateway 9.42.171.3 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
In our example, we divide the Root Bridge over the VLANs to share the load, Core1 should be the Root Bridge for VLANs 5 and 10: spanning-tree vlan 1-10 root primary spanning-tree vlan 11-4094 root secondary Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 110
A drawback of this topology is that in some cases half of the bandwidth available between the IBM Eserver BladeCenter and the core is not used, due to constraints of the Spanning Tree Algorithm blocking the traffic on the backup connections.
Page 112
Example 7-16 Advanced Layer 2 Topology with Rapid-PVST configuration for the Cisco3560 Switches version 12.2 no service pad service timestamps debug uptime service timestamps log uptime no service password-encryption hostname Core1 hostname Core2 on Core2 enable password cisco no aaa new-model Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Configurations for use of MST The MST (802.1s) protocol, like the prestandard multiple spanning tree group capability of the GbESM, allows for multiple instances of spanning tree to run at the same time. Each Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 116
Example 7-17 MSTP Information Display - GbESM >> Main# /info/l2/stg ------------------------------------------------------------------ Spanning Tree Group 1: On (MSTP) VLANs: 10 30 Current Root: Path-Cost Port Aging 1000 00:12:7f:ea:76:00 20000 EXT1 Parameters: Priority Aging Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 118
(4095) is also active by default. The STP region name is INTEROP. Example 7-19 Sample MSTP Configuration - GbESM script start "Layer 2-3 Gigabit Ethernet Switch Module for IBM eServer BladeCenter" 4 /**** DO NOT EDIT THIS LINE! /* Configuration dump taken 0:05:23 Thu Jan 1, 2070 /* Version 1.0.1.6, Base MAC address 00:11:f9:38:1b:00...
Page 119
/c/l3/if 30 addr 172.16.30.3 mask 255.255.255.0 broad 172.16.30.255 vlan 30 /c/l3/if 40 addr 172.16.40.3 mask 255.255.255.0 broad 172.16.40.255 vlan 40 /c/l3/frwd/off dirbr disabled script end /**** DO NOT EDIT THIS LINE! Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Examples of the use of static and dynamic routing are included in this section. High Availability designs using Layer 3 require different techniques than those using solely Layer 2. Examples of HA designs are included in this section. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Layer 3 switch (router). It would then flow through one or more additional devices until it returned to the IBM Eserver BladeCenter, crossing the external links for a second time.
Page 124
Shut down the ports on GbESM1 and GbESM2 /oper/port EXT1/dis /oper/port EXT2/dis /oper/port EXT5/dis /oper/port EXT6/dis Shut down the ports on Core1 and Core2 conf t int range g0/1-2, g0/11-12 shut Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 125
Server Server Server Server 10.10.0.3 10.99.0.4 10.20.0.1 10.10.0.2 10.20.0.3 10.20.0.2 BladeCenter Management Workstation Links between Management Modules and GbESMs not shown Figure 7-6 Layer 3 sample configuration (static routing, VRRP) Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 126
/cfg/port ext1/tag d /cfg/port ext2/tag d /cfg/port ext5/tag d /cfg/port ext6/tag d /cfg/port ext3/tag e /* cross-over to other GbESM needed for VRRP /* for GbESM 1 /cfg/l2/vlan 35/def ext1 ext2 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 127
/* clean up VLAN 1 - remove unused ports /c/l2/vlan 1 rem EXT1 rem EXT2 rem EXT3 rem EXT5 rem EXT6 /* clean up VLAN 5 - unused in this configuration /c/l2/vlan 5 Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 128
46 switchport mode access interface range g0/11-12 channel-group 2 mode on interface range Po2, g0/11-12 switchport access vlan 46 switchport mode access interface vlan 46 ip address 10.46.0.246 255.255.255.0 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
/* the above is only on GbESM 1; defaults to 100 on GbESM 2 /cfg/l3/vrrp/vr 4 vrid 4 if 46 addr 10.46.0.100 prio 101 /* the above is only on GbESM 1; defaults to 100 on GbESM 2 Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 130
The Core switches in this sample are not running VRRP. It is possible to develop a similar configuration using VRRP or Cisco’s proprietary HSRP to allow the Core routers to present a Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 131
VLAN, use the /i/port, /i/l2/trunk, and /i/link commands. The ports should be shown as being up. The trunks should be forwarding, and VLAN membership should be as shown in the configuration with tagging off. Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 132
EXT4 down EXT5 1000 full EXT6 1000 full >> GbESM_1 - Information# /i/l2/trunk Trunk group 1: Enabled failover dis, port state: EXT1: STG 1 forwarding EXT2: STG 1 forwarding Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 133
It should be possible to ping the server blades from all of the switches and from a mobile computer that is connected to the network. This should continue to work if GbESM 1 is removed or powered off, validating the proper function of VRRP. Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 134
Example 7-27 Layer 3 GbESM configuration - static routing and VRRP >> GbESM_1 - Main# /c/d script start "Layer 2-3 Gigabit Ethernet Switch Module for IBM eServer BladeCenter" 4 /**** DO NOT EDIT THIS LINE! /* Configuration dump taken 0:09:11 Thu Jan 1, 2070 /* Version 1.0.1.6, Base MAC address 00:11:f9:36:b7:00...
Page 136
! these statements are not displayed by the “sh run” command Core1#sh run Building configuration... Current configuration : 3019 bytes version 12.2 service config no service pad service timestamps debug uptime service timestamps log uptime Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 138
10.20.0.0 255.255.255.0 10.35.0.100 ! -- points to 10.46.0.100 on core 2 ip http server ip http secure-server control-plane line con 0 line vty 0 4 password cisco no login Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
GbESM modules to forward traffic between the two switches: the secondary switch, which connects the IBM Eserver BladeCenter to the outside world, and the primary switch, which continues to forward traffic to and from the server blades.
Page 140
Blade Server Server Server Server 10.10.0.2 10.10.0.3 10.20.0.1 10.99.0.4 10.20.0.2 10.20.0.3 Management Workstation BladeCenter Links between Management Modules and GbESMs not shown Figure 7-7 Layer 3 Sample Configuration (Dynamic routing) Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 141
10.35.0.243 mask 255.255.255.0 /cfg/l3/if 36 vlan 36 addr 10.36.0.243 mask 255.255.255.0 /* the below lines clean up unused items from the previous sample /cfg/vlan 46/del /cfg/l3/if 46/del /cfg/l3/vrrp/vrid 3/del Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 142
/cfg/l3/if 46 vlan 46 addr 10.46.0.244 mask 255.255.255.0 /* the below lines clean up unused items from the previous sample /cfg/vlan 46/del /cfg/l3/if 46/del /cfg/l3/vrrp/vrid 3/del /cfg/l3/vrrp/vrid 4/del /cfg/l3/vrrp/hot dis /cfg/l3/vrrp/group/dis Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 143
46 switchport mode access interface range Po2, g0/11-12 switchport access vlan 36 switchport mode access interface vlan 36 ip address 10.36.0.246 255.255.255.0 interface vlan 46 ip address 10.46.0.246 Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 144
OSPF is begun. The configuration is identical on the two GbESMs except as noted. Example 7-33 OSPF configuration for GbESMs /c/l3/ospf/on /c/l3/ospf/aindex 0 areaid 0.0.0.9 /c/l3/ospf/if 10 aindex 0 /c/l3/ospf/if 20 aindex 0 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 146
Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Redistributing: rip Default version control: send version 2, receive version 2 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 148
10.36.0.0/24 via 10.36.0.0 10.45.0.0/24 via 10.20.0.244 10.45.0.0/24 via 10.10.0.244 10.46.0.0/24 via 10.20.0.244 10.46.0.0/24 via 10.10.0.244 10.46.0.0/24 via 10.36.0.246 >> GbESM_1 - OSPF Information# dump OSPF Information: OSPF Version 2 Router ID: 10.10.0.243 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 150
0x80000003 0x00491C 4 10.56.0.246 10.56.0.246 1532 0x80000010 0x0023D5 2 Net Link States (Area 0.0.0.9) Link ID ADV Router Seq# Checksum 10.10.0.244 9.42.171.244 1577 0x80000007 0x00405B 10.20.0.244 9.42.171.244 1577 0x80000007 0x00C7C9 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 151
Example 7-40 Layer 3 GbESM configuration with dynamic routing >> GbESM_1 - Main# /c/d script start "Layer 2-3 Gigabit Ethernet Switch Module for IBM eServer BladeCenter" 4 /**** DO NOT EDIT THIS LINE! /* Configuration dump taken 0:04:01 Thu Jan 1, 2070 /* Version 1.0.1.6, Base MAC address 00:11:f9:36:b7:00...
Page 152
255.255.255.0 broad 10.36.0.255 vlan 36 /c/l3/vrrp/on /c/l3/vrrp/vr 1 vrid 1 if 10 prio 101 /* -- above only on GbESM 1, defaults to 100 on GbESM 2 addr 10.10.0.100 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 153
10 dead 40 trans 5 retra 5 /c/l3/ospf/if 35 /* use if 45 instead of 35 for GbESM 2 aindex 1 prio 1 cost 1 hello 10 dead 40 Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 155
9.42.171.245 255.255.255.0 ! -- .246 on core 2 interface Vlan1 no ip address shutdown ! core 2 uses vlans 36 and 46 and addresses ending in .246 Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 156
0.0.0.0 0.0.0.0 9.42.171.3 ip http server ip http secure-server control-plane line con 0 line vty 0 4 password cisco no login line vty 5 15 no login Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
# No IP address is configured for VLAN Mgmt. # Config information for VLAN VLAN10. configure vlan "VLAN10" tag 10 # VLAN-ID=0xa Global Tag 4 configure stpd s0 add vlan "VLAN10" configure vlan "VLAN10" qosprofile "QP1" Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration...
Page 158
1 # identical lines for remaining ports omitted. enable stpd s0 # MAC FDB configuration and static entries configure fdb agingtime 300 configure ipfdb agingtime 0 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
This chapter provides a brief introduction to the Serial over LAN (SOL) feature for the IBM Eserver BladeCenter. It also discusses the configuration of the Nortel Networks L2/3 GbESM along with rules to follow for establishing an SOL connection.
Ethernet interface(s) for administration (SSH for example). If BOOTP is needed for JS20, again, the second network interface is recommended. There are no specific rules that apply to the Nortel Networks L2/3 GbESM itself. The switch by default reserves VLAN 4095 for SOL traffic.
As long as the SOL VLAN ID value in the Management Module Web interface is set to 4095, there are no other specific configuration steps for the Nortel Networks L2/3 GbESM, because the Nortel Networks L2/3 GbESM reserves VLAN 4095 for SOL. All other configuration steps involve either the Management Module or the blade server.
Page 168
5. To return to the <SAC> prompt at anytime, type exit. The Linux and AIX® the SOL sessions look the same as when using SSH to administer the blade server. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
4095 and that ports INT1-14 are each members of at least one such data-bearing VLAN 3. Make sure the Nortel Networks L2/3 GbESM firmware code is upgraded to the most current version of the Alteon OS. You can find the latest code for the Nortel Networks L2/3 GbESM at: http://www.ibm.com/support When you go to this Web site, do the following: a.
9.2.1 General comments on troubleshooting Because of the highly integrated nature of the Nortel Networks L2/3 GbESM within the IBM Eserver BladeCenter, it is usually necessary to engage several teams for anything beyond basic hardware troubleshooting. Experience has shown that the greater the communication between administrative groups, the more likely an issue can be resolved sooner rather than later.
What are the symptoms of the problem? What problems could cause these symptoms? When does the problem occur? Where in the network are the symptoms seen? Is it a hardware or software issue? Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
9.3.2 Data collection In this particular step, it might be necessary to collect additional information to help isolate problem cause as noted in the following sections. Test connectivity Use connectivity testing tools such as ping, traceroute, and Telnet Note error messages that appear Details from affected users Ask more detailed questions: Has anything changed recently?
Ensure that your changes have not inadvertently caused other problems. If another problem is created, try to troubleshoot that problem (if you have time and approval). Otherwise, initiate the back-out plan. Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
9.3.7 Problem resolution Document the problem and resolution for future reference. 9.4 Troubleshooting tools This section lists troubleshooting tools for your use. Ping This tool, also known as the Packet INternet Groper, allows you to check connectivity between network devices. This tool is included with Windows, all variants of UNIX, and is also available on many network device CLIs.
Page 176
GbESM firmware. Those listed here have been tested and verified to work properly. Syslog server A syslog messaging server for application, network, and operating-system logging is available for Linux and Windows. That tool is 3Com 3CDaemon (TFTP/FTP/Syslog). Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 177
Web load generator A Web load generator allows you to test and analyze the performance characteristics and bottlenecks of your Web site under various load conditions. Tools available for Windows: – Microsoft Web Stress Tool – WebBench – LoadRunner – Socrates –...
Page 178
GbE Switch feature release, a feature release that adds new functionality and sometimes new commands GbE Switch patch release, a patch release that is available as bug fixes are created Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
10.1 Placing a call to IBM For U.S., AP, CAN, and EMEA, use one of the following numbers when calling IBM for technical support: Within the United States, call the IBM Support Center at 1-800-IBM-SERV (426-7378). Within Canada: – For support, call HelpPC at 800-426-7378.
10.4 Other support sites Listed here are other helpful Web sites: Nortel Networks Technical Support http://www130.nortelnetworks.com/cgi-bin/eserv/cs/main.jsp Nortel Networks Products http://products.nortel.com/go/product_index.jsp Chapter 10. Service and support...
Page 182
Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this Redpaper. IBM Redbooks For information about ordering these publications, see “How to get IBM Redbooks” on page 173. Note that some of the documents referenced here may be available in softcopy only.
Page 186
IBM UpdateExpress http://www.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-53046 Support for IBM xSeries products http://www.ibm.com/servers/eserver/support/xseries/index.html Broadcom NetXtreme Gigabit Ethernet drivers http://www.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-43815 Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM Eserver BladeCenter http://www.redbooks.ibm.com/redpapers/pdfs/redp3869.pdf Broadcom Advanced Control Suite (BACS) http://www.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-43815 IBM Driver matrixes http://www-307.ibm.com/pc/support/site.wss/DRVR-MATRIX.html IEEE Standards Organization http://standards.ieee.org/getieee802...
IBM Directory of worldwide contacts http://www.ibm.com/planetwide/ IBM Latin America http://www.ibm.com/pc/la Nortel Networks Technical Support http://www130.nortelnetworks.com/cgi-bin/eserv/cs/main.jsp Nortel Networks Products http://products.nortel.com/go/product_index.jsp How to get IBM Redbooks You can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at this Web site: ibm.com/redbooks...
Page 188
Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter...
Page 190
Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter Redpaper Full Layer 2 switching This IBM Redpaper positions the Nortel Networks Layer 2/3 Fiber INTERNATIONAL and Copper GbE Switch Modules for IBM Eserver BladeCenter and Layer 3 routing TECHNICAL...