Page 2
Product provided by Avaya including the selection, arrangement and to users of its products. Documentation does not include marketing design of the content is owned either by Avaya or its licensors and is materials. Avaya shall not be responsible for any modifications,...
Page 3
See the Avaya Support website: http://support.avaya.com for product notices and articles, or to report a problem with your Avaya product. For a list of support telephone numbers and contact addresses, go to the Avaya Support website: http://support.avaya.com, scroll to the bottom of the page, and select Contact Avaya Support.
Page 4
Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 1: Introduction...................... Purpose of this document......................... 17 Related resources............................. 17 Documentation..........................17 Training............................17 Avaya Mentor videos........................17 Support..............................18 Chapter 2: New in this release................... Features..............................19 Chapter 3: Troubleshooting planning fundamentals............Proper installation and routine maintenance..................... 21 Network configuration..........................
Page 6
Trace..............................42 Route Switch Processor Packet Tracing....................43 CP to COP messaging........................44 Interval.............................. 44 ERCD Records Dump..........................44 CP to COP messaging........................45 Service Level Agreement Monitor......................45 Chapter 6: Log and trap fundamentals................Simple Network Management Protocol..................... 49 Overview of traps and logs........................
Page 7
Running a ping test........................... 97 Viewing ping probe history........................99 Viewing ping results..........................100 Running a traceroute test.......................... 101 Viewing traceroute results......................... 103 Viewing the traceroute history........................104 Running a loopback test........................... 105 Configuring Ping Snoop for R series modules..................106 Configuring the SLA Monitor agent......................
Page 8
Roadmap of PCAP CLI commands....................162 Accessing the secondary CPU......................164 Configuring PCAP global parameters....................164 Enabling PCAP on a port......................... 166 Configuring PCAP capture filters...................... 168 Configuring VLAN MAC filters for PCAP..................173 Example PCAP configuration......................174 Using the captured packet dump...................... 175 Copying captured packets to a remote machine................
Page 9
Configuring Chassis KHI........................218 Configuring Performance KHI......................219 Configuring Protocol KHI........................220 Configuring Forwarding KHI......................222 Configuring IP interface KHI......................224 Configuring Port KHI........................225 Enabling and disabling the Route Switch Processor Packet Tracing............227 Dumping RSP Packet Tracing........................229 Dumping specified ERCD records......................
Page 10
Configuring an SNMP notify table......................272 Configuring SNMP notify filter profile table parameters................273 Configuring SNMP notify filter table parameters..................274 Enabling SNMP trap logging........................275 Chapter 14: Log configuration using Enterprise Device Manager......... Configuring the system log........................277 Configuring the system log table and severity level mappings..............278 Chapter 15: SNMP trap configuration using CLI..............
Page 11
Starting system message logging to a PCMCIA or external flash card............ 329 Configuring system message control......................330 Extending system message control......................331 Configuring ACLI logging.......................... 332 Chapter 19: Recovery trees and procedures..............Recovery trees............................335 IST failure............................335 DHCP Relay failure.......................... 336 SNMP failure............................
Page 12
Multicast VLAN Registration........................369 Unable to add a VLAN as a receiver VLAN..................369 Traffic is not passing from the source to the receiver............... 369 Enabling trace messages for MVR troubleshooting................. 370 IGMP Layer 2 querier..........................370 Querier not elected........................... 370 Enabling trace messages for IGMP Layer 2 querier troubleshooting..........
Page 13
Viewing multicast group trace information for IGMP snoop..............411 Viewing PGM interface errors........................412 Viewing PGM negative acknowledgement errors..................413 Viewing multicast routes........................... 414 Showing the hardware resource usage....................416 Viewing multicast routing process statistics....................417 Determing the protocol configured on the added VLAN................419 Determining the data stream learned when IP multicast over SPBM is configured on the VLAN....
Page 14
Customer unable to log on by any means (Telnet, rlogin, SSH, and PPP)........460 Administrator unable to obtain accounting information from the TACACS+ server......460 Administrator unable to receive trap packets from the Avaya Ethernet Routing Switch 8800/8600 461 User unable to login......................... 461 Avaya Secure Network Access troubleshooting..................
Ongoing product training is available. For more information or to register, you can access the website at http://avaya-learning.com/. Avaya Mentor videos Avaya Mentor is an Avaya-run channel on YouTube that includes technical content on how to install, configure, and troubleshoot Avaya products. Go to http://www.youtube.com/AvayaMentor...
• Scroll down Playlists, and click the name of a topic to see the available list of videos posted on the site. Support Visit the Avaya Support website at http://support.avaya.com for the most up-to-date documentation, product notices, and knowledge articles. You can also search for release notes, downloads, and resolutions to issues.
Chapter 2: New in this release The following section details what's new in Avaya Ethernet Routing Switch 8800/8600 Troubleshooting, NN46205-703 for Release 7.2: • Features on page 19 Features See the following sections for information about feature changes: IPv6 traceroute The traceroute command now supports IPv6 addresses.
Page 20
New in this release Troubleshooting July 2013 Comments? infodev@avaya.com...
• the normal behavior of your network Proper installation and routine maintenance Follow proper maintenance and installation procedures to prevent problems. For information about routine maintenance procedures, see Avaya Ethernet Routing Switch 8800/8600 Routine Maintenance, NN46205-312. Network configuration To keep track of your network configuration, gather the information described in the following sections.
You can use the map to systematically search each part of your network for problems. Logical connections The Avaya Ethernet Routing Switch 8800/8600 supports virtual LANs (VLAN). With VLANs, you must know how your devices are connected both logically and physically. Device configuration information Maintain online and paper copies of your device configuration information.
Normal behavior on your network Maintain a change control system for all critical systems. Permanently store change control records. • contact details Store the details of all support contracts, support numbers, engineering details, and telephone and fax numbers to save time when troubleshooting. Normal behavior on your network When you are familiar with your fully operational network, you can more effectively troubleshoot problems that arise.
Page 24
Troubleshooting planning fundamentals Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 4: Troubleshooting fundamentals This section provides conceptual information about common problems. Connectivity problems To troubleshoot connectivity problems, always provide source and destination IP pairs. Ten pairs usually provide enough information for troubleshooting (five working pairs and five pairs with connectivity issues). You can capture a dump of the hardware records from the ingress OctaPID.
Page 26
Troubleshooting fundamentals Troubleshooting July 2013 Comments? infodev@avaya.com...
An interface that supports DDM is called a Digital Diagnostic Interface (DDI). These devices provide real-time monitoring of individual DDI SFPs and XFPs on a variety of Avaya products. The DDM software provides warnings or alarms when the temperature, voltage, laser bias current, transmitter power, or receiver power fall outside of vendor-specified thresholds during initialization.
Port mirroring Port mirroring navigation • Overview on page 29 • Port mirroring and modules on page 29 • R modules on page 32 • RS modules on page 32 • ACLs, ACEs, and port mirroring on page 34 • Port mirroring considerations and restrictions on page 34 Overview...
Page 30
OctaPID ID to different destination ports, CLI prompts you with an error message. The following table describes ingress mirroring functionality for R and RS/8800 modules. Only one type of mirroring destination is supported. You cannot mirror the same port to multiple classes of destinations, for example, MLT and VLAN.
Page 31
Port mirroring Table 2: Egress mirroring functionality for R and RS/8800 modules Function Support information Egress port mirroring and egress flow mirroring One port to one port • R module—restriction: one egress source in each lane • RS/8800 module—no restrictions in each lane One to MLT groups (for TPS •...
ACL parameters. You can configure the mirroring action globally in an ACL, or for a specific ACE by using the ACE Debug actions. However, regardless of the ingress or egress mode, you configure the mirroring destination by using an ACE. Troubleshooting July 2013 Comments? infodev@avaya.com...
The 8800 series modules provide the same functionality as their RS module equivalents. The 8800 modules however, use the enhanced network processor — the RSP 2.7. R and RS modules continue to be supported and you can install a mix of R, RS and 8800 modules in the same chassis.
Use filters to reduce the amount of mirrored traffic. To use filters with port mirroring for an R or RS/8800 module, you must use an ACL-based filter. Apply an ACL to the mirrored port in the egress, ingress, or both directions. Traffic patterns that match the ACL/ACE with an action of permit are forwarded to the destination and to the mirroring port.
Remote mirroring debugging, or for security purposes such as packet sniffing, intrusion detection, or intrusion prevention. Mirroring does not affect IPFIX actions. After duplication, the packet proceeds along its original path. You can configure as many ingress mirroring flows as you have filters. In flow-based remote mirroring, the RMS encapsulates all flow-based mirroring packets, and does not distinguish between RMTs based on flows.
• Only one port in an OctaPID can act as an RMS. • On R modules, you can mirror only one port in each egress lane. This does not apply to RS/8800 modules. • The RMS port must be a port mirroring destination port because only mirrored packets are remote mirrored.
ICMP traffic from going to the CPU as a result of enabling Ping Snoop. For R and RS/8800 modules, there is a pre-configured Ping Snoop ACT and ACL. If you have an R series module installed, you must use the ACL filter option.
CPU), or on the network. You can then use FTP to download the stored packets to an offline analyzer tool such as EtherReal or Sniffer Pro. The following figure illustrates how to use the PCAP tool to configure PCAP filters and enable them on ports. Figure 1: PCAP example Troubleshooting July 2013 Comments? infodev@avaya.com...
IP traffic use Global IP filter sets. You can use PCAP to capture packets that match criteria based on MAC address filters. Avaya recommends that you use PCAP with MAC filters because it reduces traffic flow on the PCAP engine.
Page 40
OctaPID. For 10/100 ports, there is one OctaPID for every 8 ports. For example, ports 1-8 use one OctaPID, ports 9-16 use another OctaPID, and ports 17-24 use another OctaPID. For all Gigabit ports, each port has its own OctaPID. Troubleshooting July 2013 Comments? infodev@avaya.com...
General diagnostic tools The Avaya Ethernet Routing Switch 8800/8600 has diagnostic features available with Enterprise Device Manager (EDM), the Command Line Interface (CLI), and the Avaya CLI (ACLI). Use these diagnostic tools to troubleshoot operational and configuration issues. You can configure and display log files, view and monitor port statistics, trace a route, run loopback and ping tests, test the switch fabric, and view the address resolution table.
Even if a router is reachable, it can have improperly working interfaces or corrupted routing tables. Trace Trace commands provide detailed data collection about software modules on the Avaya Ethernet Routing Switch 8800/8600. The trace toolset traces multiple modules simultaneously and provides options to specify the verbosity level of the output.
CPU utilization. Route Switch Processor Packet Tracing The Avaya Ethernet Routing Switch 8800/8600 supports Route Switch Processor (RSP) Packet Tracing on R and RS/8800 modules, which provides CLI and ACLI support for the following COP debug commands: • ercdIngressPktTraceEnable / ercdIngressPktTraceDisable •...
Packet Tracing after the interval number of seconds. When Packet Tracing is disabled, the COP sends a trace disable message to the CP. ERCD Records Dump Ethernet Routing Switch 8800/8600 provides CLI and ACLI support for the following COP debug commands: • dump ercdRecord arp •...
Server and agent Ethernet Routing Switch 8800/8600 supports the SLA Monitor agent. You must have an SLA Monitor server in your network to use the SLA Monitor feature. Most of the SLA Monitor configuration occurs on the server;...
Page 46
An agent only accepts commands from the server to which it is registered. An agent can use alternate servers to provide backup for timeout and communication issues with the primary server. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 47
Service Level Agreement Monitor QoS tests SLA Monitor uses two types of tests to determine QoS benchmarks: • Real Time Protocol (RTP) This test measures network performance, for example, jitter, delay, and loss, by injecting a short stream of UDP packets from source to destination (an SLA Monitor agent). •...
Page 48
Troubleshooting tool fundamentals Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 6: Log and trap fundamentals Use the information in this section to help you understand Simple Network Management Protocol (SNMP) traps and log files, available as part of the Avaya Ethernet Routing Switch 8800/8600 System Messaging Platform. Simple Network Management Protocol The Simple Network Management Protocol (SNMP) provides facilities for managing and monitoring network resources.
(for example, Warning or Fatal) from specific applications, and send them to a trap server for further processing. For example, you can configure the Ethernet Routing Switch 8800/8600 to send SNMP traps to a server when a port is unplugged or when a power supply fails.
Log file transfer on page 54 Log message format The log messages for the Avaya Ethernet Routing Switch 8800/8600 have a standardized format. All system messages are tagged with the following information: • Module ID—software module from which the log is generated •...
Page 52
• local log file • designated printer • one or more remote hosts Internally, the Avaya Ethernet Routing Switch 8800/8600 has four severity levels for log messages: Info, Warning, Error, Fatal. The system log supports eight different severity levels: • Debug •...
PCMCIA (or external flash) card; this information is not saved to DRAM. Avaya recommends that you log to a PCMCIA (or external flash) card and always keep a PCMCIA (or external flash) card in each SF/CPU. The DRAM has limited memory allocated to SMP.
The system logs contain important information for debugging and maintaining your Avaya Ethernet Routing Switch 8800/8600. When logging to the PCMCIA card (or external flash on the 8895 SF/CPU), the log file is automatically transferred to a remote host when it reaches your specified size parameters.
Page 55
The following examples show how SMP determines when to transfer the log files and whether to continue logging to the external memory card. Example 1 The Avaya Ethernet Routing Switch 8800/8600 is operating with an 8 MB (8192 KB) PCMCIA card installed. The configured parameters are •...
Page 56
SMP is so much greater than the maxsize parameter. Example 2 The Avaya Ethernet Routing Switch 8800/8600 is operating for some time with an 8 MB (8192 KB) PCMCIA card installed. The configured parameters are •...
Page 57
System Messaging Platform The configured parameters are • minsize: 500 KB • maxsize: 2048 KB • maxoccupyPercentage: 90 Current operating parameters are • PCMCIA card size: 8192 KB • Current log file size: 200 KB There are some image and configuration files on the PCMCIA card that take up a total of 7782 KB.
Page 58
Log and trap fundamentals Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 7: Common error log messages The following table describes the most frequently seen error log messages on the Avaya Ethernet Routing Switch 8800/8600, and the associated remedial action. Table 7: Common error log messages Name Description Remedial Action The usage of counter...
Page 60
0, broadcast 11133 packet per second The message is expected If this message appears more than under normal operating once an hour, contact Avaya Technical rarCheckConsistency: conditions. If the Hash bins Support. Record size 90 in hash 0x00002ff9 Next...
Page 61
The error is stating that the If you see these message frequently, SWIP is being reset contact Avaya Technical Support. HwCheck: Fad CRR Failed, because CRR failed on the Reset swip. Total=3240 HwCheck: Fad CRR Failed, FAD.
Page 62
16) []: 202067 ? Enter a Date Code new Part Number (string max len 16) []: ? Press Enter to keep the original Deviations (string max value. len 16) []: ? Press Enter to keep the original value. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 63
FAD Errors: The Fabric If you see this message frequently, Access Device (FAD) is a contact Avaya Technical Support. HW ERROR FAD Mis-Align module in the SSF that detected, SWIP Reset Status=8. Total=4 participates in sending packets to the Backplane.
Page 64
OCTAPIDS to show where the failure is. The Fabric Access Device If you see this message frequently, (FAD) is a module in the contact Avaya Technical Support. HW WARNING HwCheck: Fad SSF which sends packets to CRR Failed, Reset swip. Total=6<000>...
Page 65
8600 4.1.4 COP SW 4.1.6.2..Please update to Release ercdProcIpRecMsg: Failed to Delete IP Record 4.1.6.2 or later. The RSP (Route Switch Contact Avaya Technical Support. Processor) is not up. For rcdIndexReadEntry: The Lane 0, the Co=Processor RSP 0 is not up ercdGetLaneEqStats:...
Page 66
If you see HwCheck: Clock Recovery backplane trap to slot 2. The three or more messages a day contact Failed Reset tmux on slot 2 tapmux was reset and Avaya Technical Support. synchronized with the CPU module. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 67
Description Remedial Action Normally less than 10 errors If these messages appear frequently, in a very short period of time contact Avaya Technical Support. Continuous fad history were seen on TMUX (and errors, Reset swip. Total=x globally logged as history) such as clock recovery or TMUX lockup.
Page 68
Common error log messages Troubleshooting July 2013 Comments? infodev@avaya.com...
The following sections provide troubleshooting information for common problems you may encounter with the Avaya Ethernet Routing Switch 8800/8600 chassis. LED indications of problems The following table lists possible problems indicated by the LEDs on Avaya Ethernet Routing Switch 8800/8600 modules and suggests corrective action. Table 8: LED problem indicators...
If a module fails during module initialization and the replacement module is the same module type, in rare cases, the new module may not initialize. To work around this issue, follow the steps in either workaround 1 or 2. Troubleshooting July 2013 Comments? infodev@avaya.com...
1. Remove the faulty module. 2. Insert a new module. 3. Reboot the chassis. If the module still fails to operate, contact the Avaya Technical Solutions Center for assistance. Failure to get a logon prompt from the Console port 1. If you connect a terminal to the console port of the 8692 or 8895 SF/CPU module and you fail to get a logon prompt, the port may have an incorrect DCE/DTE setting.
Cabling for 10BASE-T networks can consist of two-pair Category 3, 4, or 5 unshielded twisted pair (UTP) wiring. However, to prepare for future upgrades to Fast Ethernet, Avaya strongly recommends that you use all Category 5 cable in your network.
SFP, XFP, and GBIC cables Cables for the optical transceivers vary depending on the specific device type. For information about the cable requirements for SFPs, XFPs, and GBICs, see Avaya Ethernet Routing Switch 8800/8600 Installation — SFPs, XFPs, GBICs, and OADM Hardware Components, NN46205-320.
Page 74
When the number of files stored on the PCMCIA card exceeds 100, the I/O modules may continuously reset after rebooting the chassis. If there are more than 100 files on the PCMCIA card, delete any unnecessary files and reboot the chassis. Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 9: Software troubleshooting This section contains general troubleshooting tools for Avaya Ethernet Routing Switch 8800/8600 software. Enterprise Device Manager (EDM) troubleshooting If you are experience difficulties with Enterprise Device Manager, collect the following information for troubleshooting: Procedure steps 1. Define the problem symptoms, with configuration error, if viewed.
• Hosts can send packets to the 8800/8600 VLAN destined for networks beyond the same VLAN firewalls and routers. • Hosts, servers, routers, firewalls, and the 8800/8600 VLAN can all be on the same VLAN in a legacy network design.
Page 77
How to stop ICMP redirects from causing high CPU utilization ip icmp redirect (ACLI Global Configuration mode) Troubleshooting July 2013...
Page 78
Software troubleshooting Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 10: Software troubleshooting tool configuration using Enterprise Device Manager Use the procedures in this section to help you use Avaya Ethernet Routing Switch 8800/8600 troubleshooting tools. Flushing routing tables by VLAN For administrative and troubleshooting purposes, sometimes you must flush the routing tables.
MirroredPortList Specifies the port or ports to be mirrored (the source ports). MirroringPortList Specifies the destination port or ports (the ports to which the mirrored packets are forwarded). Used to configure the mirroring ports. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring ACLs for mirroring Variable Value Mode Specifies the traffic direction of the packet being mirrored: • tx mirrors egress packets. • rx mirrors ingress packets. • both mirrors both egress and ingress packets. • rxFilter mirrors and filters ingress packets. •...
Page 82
6. For R modules in Tx modes, choose Edit > Diagnostics > General > Port Mirrors, and configure the mirroring ports. For RS/8800 or R modules in Rx mode: specify mirroring ports in the ACE Debug tab. On the ACL tab, select an ACL, click ACE, select an ACE, then click Action/ Debug.
• For R modules in Tx modes: specify ports in the Edit, Diagnostics, Port Mirrors tab • For RS/8800 or R modules in Rx modes: specify ports in the ACE Debug tab State Enables or disables all of the ACEs in the ACL. The default value is enable.
Page 84
PCAP, the Packet Capture Tool, rather than selecting the parameter copyToPrimaryCp. 8. For R and RS/8800 modules in Rx mode : configure one of: DstPortList, DstVlanId, or DstMltId. For R modules in Tx mode: configure the Edit, Diagnostics, Port Mirrors tab.
Page 85
Configuring ACEs for mirroring Variable Value RemarkDscp Specifies whether the DSCP parameter marks nonstandard traffic classes and local-use Per-Hop Behavior (PHB). The default is disable. RemarkDot1Priority Specifies whether Dot1 Priority, as described by Layer 2 standards (802.1Q and 802.1p), is enabled. The default is disable.
Variable Value If you enable mirroring, ensure that you configure the appropriate parameters: • For R and RS/8800 modules in Rx mode: DstPortList, DstVlanId, or DstMltId. • For R modules in Tx mode: configure the Edit, Diagnostics, Port Mirrors tab.
VLAN in which the destination MAC address resides. Configuring PCAP globally Use the Packet Capture Tool (PCAP) to capture packets for troubleshooting and security purposes. Configure PCAP globally to define how PCAP operates on the Avaya Ethernet Routing Switch 8800/8600. Prerequisites •...
Page 88
This field is valid only if the device is network. CopyFileName Specifies the file name to use when copying the PCAP file from the PCAP engine DRAM or a PCMCIA (or external flash) device to a remote client (user local machine). Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring PCAP on a port Configuring PCAP on a port Configure PCAP on a port so that the port supports PCAP, and to apply filters to the captured data. You can apply IP- or Access Control List (ACL)-based filters. Prerequisites •...
Specifies the source MAC address mask that specifies an address range. IsInverseSrcMac Specifies the source MAC address inverse. When set, all MAC addresses other than the one specified are matched. DstMac Specifies the destination MAC address. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 91
Configuring PCAP filters Variable Value DstMacMask Specifies the destination MAC address mask that specifies an address range. IsInverseDstMac Specifies the destination MAC address inverse. When set, all MAC addresses other than the one specified are matched. VlanId Specifies the VLAN ID of the packet to match. ToVlanId Specifies the destination VLAN ID;...
Specifies a range of UDP ports. IsInverseUdpPort Indicates that all other values other than the specified range of UDP ports are matched. TcpPort Specifies the TCP port of the packet to match. ToTcpPort Specifies a range of TCP ports. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring VLAN MAC filters for PCAP Variable Value IsInverseTcpPort Indicates that all other values other than the specified range of TCP ports are matched. UserDefinedData Specifies the user-defined data to match. UserDefinedDataSize Specifies the length of user-defined data. UserDefinedOffset Specifies the offset from which the match must start. IsInverseUserDefined Indicates that all data other than the specified user- defined data is matched.
More specific information about the test result (for example, an error code after a failed test): none, NoReceive (timeout on a send), BadSeq (packets received out of sequence), BadLen (packet length mismatch), BadData (packet data mismatch) Troubleshooting July 2013 Comments? infodev@avaya.com...
These statistics are debugging aids, and you should use them only when consulting with Avaya support personnel. The statistic of most interest is the NoSpace counter, which indicates the number of entries the AR table could not add because of lack of space.
Page 96
Specifies the number of IP Subnet entries deleted from the AR table. RsvpsAdded Specifies the number of Resource Reservation Setup Protocol (RSVP) entries added to the AR table. RsvpsDeleted Specifies the number of RSVP entries deleted from the AR table. Troubleshooting July 2013 Comments? infodev@avaya.com...
Running a ping test Running a ping test Use Ping to determine if an entity is reachable. Procedure steps 1. In the navigation tree, open the following folders Configuration > Edit > Diagnostics. 2. Click Ping/Trace Route. 3. Click Insert. 4.
Page 98
Describes the remote ping test. SourceAddressType Specifies the type of the source address used at a remote host when performing a ping operation. SourceAddress Specifies the IP address (a.b.c.d) as the source address in outgoing probe packets. Troubleshooting July 2013 Comments? infodev@avaya.com...
Viewing ping probe history Variable Value IfIndex Setting this object to an interface's ifIndex, prior to starting a remote ping operation, directs the ping probes to be transmitted over the specified interface. ByPassRouteTable Enables (optionally) the bypassing of the route table. DSField Specifies the value to store in the Differentiated Services (DS) field in the IP packet used to encapsulate the ping...
Specifies the maximum ping RTT received. A value of 0 means that no RTT is received. AverageRtt Specifies the current average ping RTT. ProbeResponses Specifies the number of responses to probes. SentProbes Specifies the number of sent probes. Troubleshooting July 2013 Comments? infodev@avaya.com...
Running a traceroute test Variable Value RttSumOfSquares Specifies the sum of squares of RTT for all probes received. LastGoodProbe Specifies the date and time when the last response is received for a probe. Running a traceroute test Use traceroute to determine the route packets take through a network to a destination. Procedure steps 1.
Page 102
Specifies the number of seconds to wait before repeating a trace route test as defined by the value of the various objects in the corresponding row. The default is 0. StorageType Specifies the storage type for this row. Troubleshooting July 2013 Comments? infodev@avaya.com...
Viewing traceroute results Variable Value AdminStatus Specifies the desired state for TraceRouteCtlEntry. The options are enabled or disabled. MaxRows Specifies the maximum number of entries allowed in the TraceRouteProbeHistoryTable. TrapGeneration Determines when to generate a notification for this entry. The options are: •...
2. Click Ping/Trace Route. 3. Click the Trace Route Control tab. 4. Select an entry. 5. Click the Trace Route Probe History button. Variable definitions Use the information in the following table to understand the history parameters. Troubleshooting July 2013 Comments? infodev@avaya.com...
Running a loopback test Variable Value OwnerIndex Identifies the Trace Route entry to which a probe result belongs. TestName Specifies the test name. Index Specifies the Index. HopIndex Indicates for which hop in a traceroute path the probe results are intended. ProbeIndex Specifies the index of a probe for a particular hop in a traceroute path.
10. From the Mode options, select permit. 11. From the Flags options, select copyToPrimaryCp. 12. Click Insert. 13. For the ACE, configure a source IP address, a destination IP address, or both: select the ACE, and then click the IP button. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring the SLA Monitor agent 14. To configure the source IP address, in the Source Address tab, click Insert. Configure Oper to eq and type the IP address in the List box, and then click Insert. 15. To configure the destination IP address, in the Destination Address tab, click Insert.
If you install a certificate on the agent, you must ensure a matching configuration on the server. By default, the agent uses an Avaya SIP certificate to secure communications with the server. ConfiguredAgentAddrType Specifies the address type of the agent: IPv4.
Page 109
Configuring the SLA Monitor agent Name Description ConfiguredServerPort Restricts the agent to use of this registration port only. The default is 0, which means the agent disregards the source port information in server traffic. The server must use the same port. ConfiguredAltServerAddrType Specifies the address type of the secondary server: IPv4.
Page 110
Software troubleshooting tool configuration using Enterprise Device Manager Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 11: Software troubleshooting tool configuration using CLI Use the procedures described in this section to configure troubleshooting tools using CLI. General troubleshooting This section provides information about general troubleshooting using CLI. Roadmap of general CLI troubleshooting commands The following roadmap lists some of the CLI commands and their parameters that you can use to complete the procedures in this section.
You can capture a dump of the hardware records for an ingress OctaPID to aid in troubleshooting. Generally, a verbosity level of 1 suffices. The dump ar command displays the hardware registers of the RaptARU attached to an OctaPID. Troubleshooting July 2013 Comments? infodev@avaya.com...
General troubleshooting Procedure steps To dump hardware record information, enter the following command: dump ar <octapid> <vlan|ip_subnet|mac_vlan|mac|arp|ip|ipx| ipmc|ip_filter|protocol|sys_rec|all> <verbosity> For example, dump all hardware records from OctaPID 0 slot 1 port 1 with a verbosity level of 3: dump ar 0 all 3 Variable definitions Use the information in the following table to help you use the dump command.
Page 116
<slot#> trace grep [<keyword>] Variable definitions Use the information in the following table to help you use the trace command. Variable Value clear Clears any previous trace output. filter Filters the trace output. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 117
General troubleshooting Variable Value grep [keyword] Performs a comparison of trace messages (get regular expression and print [GREP]). info [tail] [tail] Shows the trace output. shows the last results first. level <modid> <level> Starts the trace by specifying the module ID and level. <modid>...
When the configured utilization is reached and sustained for the configured amount of time, a CP trace is performed and saved to the PCMCIA (or external flash on the 8895 SF/CPU). Procedure steps 1. Configure the module and verbosity: trace auto-enable add-module <modid> <level> For example: Troubleshooting July 2013 Comments? infodev@avaya.com...
Removes a module ID from the auto-trace instance. Using trace Ethernet to diagnose problems The trace command is the main debugging tool for the ERS 8800/8600. To alleviate some of the overhead and make tracing more efficient, Avaya enhanced the current tracing Troubleshooting...
Page 120
Ethernet trace packets and that you start the packet capture. (OR) If packets are going to be decoded real time, then ensure that you execute the following command in the Linux or Solaris PC in superuser mode: Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 121
General troubleshooting • In Linux: ./eth-trace-linux [-q -s 0 –n] –i <PC ethernet port (eg: eth0)> [cpu <base|mezz> ether dst <mac address to be filtered/decoded>] • In Solaris: ./eth-trace-solaris [-q -s 0 –n] –i <PC ethernet port (eg: eth0)> [cpu <base|mezz> ether dst <mac address to be filtered/decoded>] Where cpu <base|mezz>...
Shows the configured values for Ethernet tracing. Using trace syslog to diagnose problems The trace command is the main debugging tool for the ERS 8800/8600. To alleviate some of the overhead and make tracing more efficient, Avaya added Syslog tracing to enhance the current tracing mechanism.
Page 123
General troubleshooting • The trace syslog enhancement has nothing to do with the existing syslog functionality. • In each syslog trace, the management IP address of the switch is part of the syslog header. This enables you to differentiate between multiple trace messages generated from different switches.
Using the shell debug wrapper commands Important: In some cases, these debug commands can take a long time to display the information on the screen. To avoid displaying messages on the console screen, enter config log screen off. You can only enter one command for any group at a given point of time. Using the show debug generic command Use the show debug generic [verbose] command for debugging purposes only.
Important: The verbose option displays four additional commands that are prone to some amount of risk to your switch. Avaya recommends that you execute the verbose command during a maintenance window only. Table 11: show debug generic verbose commands...
Using the shell debug wrapper commands Command Definition dumpPcapFrame Displays PCAP output but only on the slave shell, not on the master. portRemoteMirrorTblDump Displays information about the remote mirroring port table Using the show debug bridging command Use the show debug bridging <value> command for debugging bridging problems only. To use this command, you must enter one of the following subgroup parameters: eap high-port <value>...
• vrrp verbose <value> Collecting Key Health Indicator (KHI) information The Key Health Indicators (KHI) feature of the Avaya Ethernet Routing Switch 8800/8600 provides a subset of health information that allows for quick assessment of the overall operational state of the device.
Configuring global KHI Use the following procedure to configure KHI at the global level. You can enable or disable KHI globally. In addition, the Avaya Ethernet Routing Switch 8800/8600 provides a global boot delay parameter for KHI. If the system begins collecting statistics immediately at boot-up, the transitions that the system initially experiences do not provide an appropriate baseline of normal operations against which to compare.
Management KHI tracks TCP connections, CLI users, and KHI log status. Procedure steps 1. To enable the KHI feature globally, enter: config sys set khi khi-enable <true|false> 2. To enable the management KHI feature, enter: Troubleshooting July 2013 Comments? infodev@avaya.com...
Collecting Key Health Indicator (KHI) information config sys set khi mgmt-khi-enable <true|false> 3. To display the management KHI information, enter: show khi mgmt [all] [history] Variable definitions Variable Value <true|false> Enables or disables the specified KHI feature. [all] Displays all management KHI information, including the event history.
It supports VRFs. Every protocol has a large number of parameters that can be tracked, but only the key parameters are tracked by the KHI. Protocol information is collected and displayed on-demand, Troubleshooting July 2013 Comments? infodev@avaya.com...
When you display IST/SMLT information, the information (especially the SMLT table) is computed on demand. Carefully consider the frequency of issuing the show khi protocol-stats when the setup is a large IST/SMLT setup. Avaya recommends issuing the command when the network stabilizes.
Page 134
The memory used for forwarding KHI information is minimal, however, collection of forwarding KHI information can have an impact when the system is busy. Avaya recommends to enable forwarding KHI when the System stabilizes. To ensure the validity of the KHI information, verify that the timestamps are relevant.
Collecting Key Health Indicator (KHI) information Clear command allows to establish the last clear timestamps. Variable definitions Variable Value <true|false> Enables or disables the specified KHI feature. Displays all forwarding KHI information. current-status Displays the current status of forwarding by slot. asic Displays ASIC health information.
While the memory used for port KHI is minimal, collecting system statistics when a system is busy can have a system impact. Avaya recommends you enable port KHI when a system stabilizes. Troubleshooting July 2013...
Enabling and disabling the Route Switch Processor (RSP) packet tracing The clear command allows you to establish a reference list of UP ports and to establish the last clear timestamps. It also clears any existing port error information. To provide a useful reference point for the UP ports list, use the clear command when the network is stable.
Page 138
Software troubleshooting tool configuration using CLI Configure the Route Switch Processor (RSP) packet tracing to observe the behavior of the RSP on each R and RS/8800 module. The RSP is the programmable Application Specific Integrated Circuit (ASIC) that controls the ports and traffic flow.
Page 139
Enabling and disabling the Route Switch Processor (RSP) packet tracing Variable definitions The following table describes variables that you enter in the config rsp-trace ingress- pkt-trace port <value> state <value> or the config rsp-trace egress- pkt-trace port <value> state <value> command. Variable Value port <value>...
The CLI command accepts only a single port. Important: RSP packet tracing displays only the last 1024 packets captured. 1. Display the specific egress RSP packet by using the following command: dump rsp-trace egress-display-pkt port <value> pkt-id <value> Troubleshooting July 2013 Comments? infodev@avaya.com...
Dumping specified ERCD records Display the specific ingress RSP packet by using the following command: dump rsp-trace ingress-display-pkt port <value> pkt-id <value> 2. Display the ingress packet tracing by using the following command: dump rsp-trace ingress-pkt-trace port <value> [start-pkt <value>] [end-pkt <value>] Display the egress packet tracing by using the following command: dump rsp-trace egress-pkt-trace port <value>...
Page 142
VLANs to which this port belongs and the corresponding ingress VLAN records of this port. slot <value> Specifies the slot number to which you send the query. port <value> Specifies the port number {slot/port} for which you get the records. Troubleshooting July 2013 Comments? infodev@avaya.com...
Using PIM debugging commands Variable Value [verbose <value>] Specifies an expanded display. value is in the range of 0 – 3. Using PIM debugging commands Use PIM traces to aid in PIM troubleshooting. Procedure steps 1. To start debug trace message output, enter: config ip pim debug-pimmsg pimdbgtrace 1 2.
Use global and peer debug commands to display specific debug messages for your global and peers BGP configuration, including the BGP neighbors. You can use these commands to troubleshoot your BGP configuration. Procedure steps 1. Display specific debug messages for your global BGP configuration using the following command: Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 145
Using BGP debugging commands config ip bgp global-debug mask <value> 2. Display specific debug messages for your global BGP neighbors using the following command: config ip bgp neighbor-debug-all mask <value> 3. Display specific debug messages for BGP peers or peer group using the following command: config ip bgp neighbor <nbr_ipaddr|peer-group-name>...
You can use port mirroring to aid in troubleshooting procedures. Port mirroring configuration navigation • Roadmap of port mirroring CLI commands on page 147 • Configuring port mirroring on page 148 • Configuring global mirroring actions with an ACL on page 151 Troubleshooting July 2013 Comments? infodev@avaya.com...
Port mirroring configuration • Configuring ACE debug actions to mirror on page 152 • Example of port mirroring configuration with ACLs (rx-filter mode) on page 154 Roadmap of port mirroring CLI commands The following roadmap lists some of the CLI commands and the parameters that you can use to complete the procedures in this section.
1. Create a port mirroring instance: config diag mirror-by-port <id> create in-port <value> [out- port <value>] [mode <value>] [enable <value>] [remote-mirror- vlan-id <value>] [mirroring-mlt <value>] [mirroring-vlan <value>] <id> specifies the mirror-by-port entry ID in the range of 1 to 383. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 149
Port mirroring configuration Mirroring is not operational until you issue the enable parameter. 2. Add mirroring entries as required: config diag mirror-by-port <id> add {mirrored-ports <port]| mirroring-mlt <mid>|mirroring-port <port>|mirroring-vlan <vid>} 3. Use the following variable definitions tables to configure mirroring as required. 4.
Page 150
Configures the remote mirror VLAN ID. <vid> • <vid> is the ID of the VLAN in the range of 0 to 4094. remove {mirrored-ports Removes ports, an MLT, or a VLAN from the mirroring instance. <port>|mirroring-mlt Troubleshooting July 2013 Comments? infodev@avaya.com...
Port mirroring configuration Variable Value <mid>|mirroring-ports To change a port mirroring configuration, first disable it. <port>|mirroring-vlan <vid>} • mirrored-ports <port> specifies the source port for mirrored packets. • mirroring-mlt <mid> specifies the destination MLTs for mirrored packets. • mirroring-ports <port> specifies the destination ports for mirrored packets.
• For R modules in Tx mode, use config diag mirror-by-port commands to specify mirroring ports. • For R or RS/8800 modules in Rx mode, use the config filter acl <acl-id> ace <ace-id> debug commands to specify mirroring ports. Configuring ACE debug actions to mirror Use debug actions to use filters for troubleshooting procedures.
Page 153
Value mirror <enable| Enables or disables mirroring for the ACE. If you enable mirroring, ensure that you configure the disable> appropriate parameters: • For R and RS/8800 modules in Rx mode, use mirroring-dst-ports mirroring-dst- vlan mirroring-dst-mlt , or config diag •...
Page 154
• Use port 3/48 as the monitoring port. • Configure an ACL so that TCP traffic from ports 20 to 500, and ICMP frames are mirrored to the monitoring port. The configuration shown in the following figure is used. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 155
Port mirroring configuration Figure 3: Port mirroring setup Procedure steps 1. Create a new ACT to filter on ICMP frames and TCP destination ports. Configure a new ACT with ID = 2: ERS-8610:5# config filter act 2 create 2. Select the IP attributes of the IP protocol type: ERS-8610:5# config filter act 2 ip ipProtoType 3.
Remote mirroring configuration Remote mirroring configuration navigation • Configuring remote mirroring on page 157 • Example of remote mirroring configuration using ACLs on page 159 Configuring remote mirroring Use remote mirroring to monitor many ports from different switches using one network probe device.
Page 158
VLAN so that all nodes in the network can learn the DstMac. ether-type <ether-type> Specifies the Ethertype of the remote mirrored packet. The default value is 0x8103. info Displays the remote mirroring configuration of the port. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 159
Remote mirroring configuration Variable Value mode <source|termination> Specifies whether the port is an RMT (mode is termination) or an RMS (mode is source). remove-vlan-id <vlan-id> Removes a VLAN from the VLAN list. Used only for RMT ports. When the RMT port is removed from the last VLAN in the list, RMT is disabled from the port.
The following table lists the commands and their parameters that you use to perform the procedures in this section. Command Parameter config {ethernet} add set <value> <ports> pcap add-acl-filter <acl-id> enable <true|false> [mode <value>] info Troubleshooting July 2013 Comments? infodev@avaya.com...
1. Log on to the Primary CPU. 2. Access the Secondary CPU by entering the following command: peer telnet Configuring PCAP global parameters Configure PCAP globally to define how PCAP operates on the Avaya Ethernet Routing Switch 8800/8600. Troubleshooting July 2013...
Page 165
PCAP configuration Prerequisites • The secondary SF/CPU is installed and active. • If saving to external memory, a PCMCIA card (or external flash on the 8895 SF/CPU) is installed. Procedure steps 1. Enable PCAP using the following command: config diag pcap enable true 2.
• If required, ACLs with a global action of mirror exist. Procedure steps 1. To apply filter sets or ACLs to captured packets, enter: config {ethernet} <ports> pcap add set <value> config {ethernet} <ports> pcap add-acl-filter <acl-id> 2. To enable PCAP on Ethernet ports, enter: Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 167
PCAP configuration config {ethernet} <ports> pcap enable true [mode <value>] 3. To ensure PCAP is correctly configured, enter: show diag pcap port Variable definitions Use the information in the following table to complete the config {ethernet} <ports> pcap command. Variable Value add set <value>...
Configuring PCAP capture filters Use capture filters to better define the match criteria used on packets. Avaya recommends using PCAP with IP or MAC filters to reduce the load on the PCAP engine. To create a functional capture filter that captures specific packets, create two filters. Use one filter to capture specific packets, and another filter to drop all other packets.
Page 169
PCAP configuration Variable Value • trigger-on indicates to start capturing the packet when a packet matches this filter. PCAP is enabled globally and the trigger filter is disabled. trigger-off • indicates to stop capturing the packet when a packet matches this filter. PCAP is disabled globally and the trigger filter is disabled.
Page 170
The default is 00:00:00:00:00:00, which means this option is disabled. mask <value> is the mask of the source MAC address. This parameter specifies an address range. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 171
PCAP configuration Variable Value means that the filter matches for all other values than the range of values defined. tcp-port <tcport> [to Specifies the TCP port of the packet. <tcport> can be one value or a range of TCP port <value>] [not] values.
Page 172
5. Stop packet capture when the PCAP engine buffer is full: config diag pcap buffer-wrap false 6. Save captured packets after the PCAP engine buffer is full: config diag pcap auto-save true file-name pcap_test.cap device pcmcia Troubleshooting July 2013 Comments? infodev@avaya.com...
PCAP engine. Prerequisites • A VLAN exists. • For more information about VLANs and MAC filters, see Avaya Ethernet Routing Switch 8800/8600 Configuration — VLANs and Spanning Tree, NN46205-517. Procedure steps 1. Enable PCAP with the mode set to rxFilter.
6. Disable PCAP at the port level: config ether 2/10 pcap enable false 7. Copy captured packets to a file: copy PCAP00 /pcmcia/pcap_packets.cap 8. Use Ethereal or Sniffer Pro to analyze the packets. Troubleshooting July 2013 Comments? infodev@avaya.com...
CPU intensive. The switch does not respond to any commands while the dump is in progress. Avaya recommends you use this command only when it is absolutely necessary. However, there is no degradation in normal traffic handling or switch failover.
You can clear the PCAP DRAM buffer and the PCAP counters. Procedure steps 1. Log on to the Secondary SF/CPU. 2. Disable PCAP: config diag pcap enable false 3. Reset the PCAP engine DRAM buffer: Troubleshooting July 2013 Comments? infodev@avaya.com...
PCAP configuration 3. Create capture-filter 2 and configure to drop all traffic. This is required so that you only capture traffic using the criteria from step 2. ERS8600-B:5# config diag pcap capture-filter 2 create ERS8600-B:5# config diag pcap capture-filter 2 action drop ERS8600-B:5# config diag pcap capture-filter 2 enable true 4.
FTP server to their client machine. However, they are having a problem connecting to the FTP server. The FTP client resides on client 1 and the FTP server is on client 2. The FTP server is connected to an Avaya Ethernet Routing Switch 8800/8600 (R1) through port interface 2/10.
Page 181
PCAP configuration Method 1 In this solution, PCAP is configured to capture all packets on port interface 2/10 and packets are saved on a PCMCIA device. The file containing captured packets is then copied using FTP for analysis at a later time. Procedure steps 1.
Page 182
Configure a capture filter: config diag pcap capture-filter 7 create config diag pcap capture-filter 7 action drop config diag pcap capture-filter 7 protocol-type 6 not config diag pcap capture-filter 7 tcp-port 20 to 21 not Troubleshooting July 2013 Comments? infodev@avaya.com...
Testing the switch fabric config diag pcap capture-filter 7 enable true Method 4 If the amount of traffic flowing between client 1 and client 2 is still too large for analysis, start packet capture when the first TCP/FTP packet arrives at the port, which also enables PCAP automatically.
You can test the Address Resolution Protocol (ARP) address table for consistency. Procedure steps 1. Test the address table: test artable 2. Stop the test after a few seconds: test stop artable 3. View the results of the test: show test artable Troubleshooting July 2013 Comments? infodev@avaya.com...
Clearing ARP information for an interface Clearing ARP information for an interface You can clear the ARP cache as part of ARP problem resolution procedures. Procedure steps Clear ARP information using the following commands: clear ip arp ports <port> clear ip arp vlan <vid> Flushing routing, MAC, and ARP tables for an interface Use this procedure to clear or flush routing, MAC, and ARP tables for an interface.
<ipxhost> [<count>] [-s] [-q] [-t <value>] <ipxhost> specifies the IPX host in the net.node format: 0x00:0x00:0x00:0x00.0x00:0x00:0x00:0x00:0x00:0x00 3. Ping an MPLS device: mplsping ipv4 <prefix/len> [ttl <value>] [source <value>] [count <value>] mplsping rsvp <lsp-name> [ttl <value>] [source <value>] [count <value>] Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 187
Running a ping test <prefix/len> specifies the IPv4 address and prefix length; <lsp-name> specifies the name of the label-switched path. Variable definitions Use the information in the following table to help you use the ping <HostName/ ipv4address/ipv6address> command. Variable Value Sets the ping debug flag.
1446 bytes from 100.100.1.1: icmp_seq=5. time=1.589 ms 1446 bytes from 100.100.1.1: icmp_seq=6. time=1.577 ms 1446 bytes from 100.100.1.1: icmp_seq=7. time=1.588 ms 1446 bytes from 100.100.1.1: icmp_seq=8. time=1.590 ms 1446 bytes from 100.100.1.1: icmp_seq=9. time=1.535 ms ----100.100.1.1 PING Statistics---- Troubleshooting July 2013 Comments? infodev@avaya.com...
Running a traceroute test 10 packets transmitted, 10 packets received, 0% packet loss round-trip (ms) min/avg/max = 1.535/1.580/1.605 Running a traceroute test Use traceroute to determine the route packets take through a network to a destination. Procedure steps To use traceroute, enter the following command: traceroute <ipaddr>...
The predefined ACL and ACTs for ping snoop are numbered 4096. You can use your own ACT, ACL, and ACE instead, but you are duplicating the ACT, ACL, and ACE that the system predefines. Configure the ACE action, debug action, and the IP addresses that you require. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 191
Configuring ping snoop for R series modules By default, ping snoop messages are echoed only to the serial console port. If you do not have access to the serial port and are connecting via Telnet (or other means such as SSH or Rlogin), to see the messages in your session, enter: config log screen on This setting is specific to the CLI session where it is executed.
<port> 5. (Optional) Control the port used for Real Time Protocol (RTP) and New Trace Route (NTR) testing: config slamon agent-comm-port <port> 6. (Optional) Install a Secure Socket Layer (SSL) certificate for the agent: Troubleshooting July 2013 Comments? infodev@avaya.com...
Verify the configuration. ERS-8800:1# config ERS-8800:1/config# slamon agent ip-address 192.0.2.1 ERS-8800:1/config# slamon oper-mode enable ERS-8800:1/config# slamon server ip-address 192.0.2.25 ERS-8800:1/config# slamon server port 50156...
Page 194
If you install a certificate on the agent, you must ensure a matching configuration on the server. By default, the agent uses an Avaya SIP certificate to secure communications with the server. deinstall-cert-file <file> Uninstalls an SSL certificate. <file>specifies the file name and path of the certificate to uninstall.
Page 195
Configuring the SLA Monitor agent Variable Value server port <file> Restricts the agent to use of this registration port only. The default is 0, which means the agent disregards the source port information in server traffic. The server must use the same port. Troubleshooting July 2013...
Page 196
Software troubleshooting tool configuration using CLI Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 12: Software troubleshooting tool configuration using ACLI Use the tools described in this section to perform troubleshooting procedures using ACLI. General troubleshooting This section provides information about general troubleshooting using ACLI. Roadmap of general ACLI troubleshooting commands The following roadmap lists some of the ACLI commands and their parameters that you can use to complete the procedures in this section.
You can use ACLI to help provide diagnostic information. Prerequisites • Access Privileged EXEC mode. Procedure steps 1. Prior to capturing data it is useful to disable scrolling of the output display. To do this, issue the following command: Troubleshooting July 2013 Comments? infodev@avaya.com...
General troubleshooting terminal more disable 2. You can view configuration file information using the more command, for example: more boot.cfg 3. The following command output should be captured when any switch problem is observed. show tech show running-config [verbose] [module <cli|sys|web|rmon| vlan|port|qos|traffic-filter|mlt|stg|ip|ipx|diag|dvmrp| radius|ntp|lacp|naap|cluster|bootp|filter|ipv6>] show interfaces FastEthernet statistics...
9 3 Wait approximately 30 seconds. The default trace settings for CPU utilization are: High CPU Utilization: 90%, High Track Duration: 5 seconds, Low CPU Utilization: 75%, and Low Track Duration: 5 seconds. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 203
General troubleshooting 3. Stop tracing: trace shutdown 4. View the trace results: show trace file [tail] 5. You can save the trace file to the PCMCIA card for retrieval. save trace The file is saved with a file name of systrace.txt. R series modules use different trace commands: r-module <1-10>...
General troubleshooting Using auto-trace to diagnose problems You can use auto-trace to automatically perform the trace function when a parameter reaches a certain threshold. For example, if the SF/CPU fluctuates and accessing the switch to perform a CP trace is not possible, use auto-trace to automatically perform this function.
Using trace ethernet to diagnose problems The trace command is the main debugging tool for the ERS 8800/8600. To alleviate some of the overhead and make tracing more efficient, Avaya enhanced the current tracing mechanism by adding Ethernet tracing. With this enhancement, trace messages are transmitted out as Ethernet packets thereby avoiding the overhead caused by the traces for displaying the messages on the switch console and logging to a file.
Page 207
General troubleshooting Prerequisites • Connect a Linux or Solaris PC to the port that will generate the Ethernet trace packets. • Use a packet capture tool to capture the Ethernet trace packets the switch generates. (This is optional). Procedure steps 1.
Page 208
. The default is 01:02:03:04:05:06 enable Enables trace ethernet Note: Before enabling Ethernet tracing, you must configure a port number. Ethernet tracing is disabled by default. disable Disables trace ethernet Ethernet tracing is disabled by default. Troubleshooting July 2013 Comments? infodev@avaya.com...
General troubleshooting Using trace syslog to diagnose problems The trace command is the main debugging tool for the ERS 8800/8600. To alleviate some of the overhead and make tracing more efficient, Avaya enhanced the current tracing mechanism by adding Syslog tracing. With this enhancement, trace messages are transmitted out as Syslog packets thereby avoiding the overhead caused by the traces for displaying the messages on the switch console and logging to a file.
Page 210
The range is 1..4094. enable Enables trace syslog Note: Before enabling Syslog tracing, you must configure a destination IP address. Syslog tracing is disabled by default. disable trace syslog Disables Syslog tracing is disabled by default. Troubleshooting July 2013 Comments? infodev@avaya.com...
shell debug wrapper commands shell debug wrapper commands The shell debug wrapper commands group multiple shell commands under one command. With this feature, you can enter one command to display the output of all the related shell commands in that group. This eliminates the need for you to remember the names of all the individual commands and makes it easier to enter just one command, which should facilitate debugging.
Page 212
MacInfo frames. Important: The verbose option displays four additional commands that are prone to some amount of risk to your switch. Avaya recommends that you execute the verbose command during a maintenance window only. Troubleshooting July 2013 Comments? infodev@avaya.com...
shell debug wrapper commands Table 16: show debug generic verbose commands Command Definition For best results sysCliShowPerf ; Displays system performance parameters like CPU/SF/ execute twice netStackSysPool Buffer utilizations and DRAM info; displays memory Show buffer information. fbufDump Displays statistics about all queues. execute once cppQShow Displays CPP queue statistics.
• vrrp verbose <value> Collecting Key Health Indicator (KHI) information The Key Health Indicators (KHI) feature of the Avaya Ethernet Routing Switch 8800/8600 provides a subset of health information that allows for quick assessment of the overall operational state of the device.
The following sections describe the various KHI options. Configuring global KHI You can enable or disable KHI globally. In addition, the Avaya Ethernet Routing Switch 8800/8600 provides a global boot delay parameter for KHI. If the system begins collecting statistics immediately at boot-up, the transitions that the system initially experiences do not provide an appropriate baseline of normal operations against which to compare.
Collecting Key Health Indicator (KHI) information Prerequisites • Log on to Global Configuration mode. Procedure steps 1. To enable KHI globally, enter: [no] khi enable 2. To configure the boot delay, enter: khi boot-delay <minutes> 3. To display high-level KHI information, enter: show khi info 4.
Displays the event history (max 10). Configuring Chassis KHI Chassis KHI displays the chassis key health indicators, such as temperature, fans, power supply, slots and CPU state. To configure chassis KHI, use the following procedure. Troubleshooting July 2013 Comments? infodev@avaya.com...
Collecting Key Health Indicator (KHI) information Prerequisites • Log on to Global Configuration mode. Procedure steps 1. To enable KHI globally, enter: [no] khi enable 2. To enable chassis KHI, enter: [no] khi chassis 3. To display chassis KHI information, enter: show khi chassis [all] [history] 4.
Displays all performance KHI information, including the event history. [history] Displays the event history (max 10). Configuring Protocol KHI Protocol KHI tracks the health of the following protocols: • OSPF • BGP • IST/SMLT • PIM • IGMP Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 221
When you display IST/SMLT information, the information (especially the SMLT table) is computed on demand. Carefully consider the frequency of issuing the show khi protocol-stats when the setup is a large IST/SMLT setup. Avaya recommends issuing the command when the network has stabilized.
The memory used for Forwarding KHI information is minimal, however, collection of Forwarding KHI information can have an impact when the system is busy. Avaya recommends to enable Forwarding KHI when the System has stabilized. To ensure the validity of the KHI information, verify that the timestamps are relevant.
Page 223
Collecting Key Health Indicator (KHI) information Prerequisites • Log on to Global Configuration mode. Procedure steps 1. To enable the KHI feature globally, enter: [no] khi enable 2. To enable the forwarding KHI feature, enter: [no] khi forwarding 3. To display the forwarding KHI information, enter: show khi forwarding [<all|asic|current-status|f2x|history| rsp-state|rsp-stats>] [slot <value>] 4.
[no] khi enable 2. To display the IP interface KHI information, enter: show khi ip-interface 3. To clear the IP interface KHI statistics, enter: clear khi ip Variable definitions Variable Value [no] Disables the specified KHI feature. Troubleshooting July 2013 Comments? infodev@avaya.com...
While the memory used for Port KHI is minimal, collecting system statistics when a system is busy can have a system impact. Avaya recommends to enable Port KHI when a system has stabilized. The clear command allows you to establish a reference list of UP Ports and to establish the last clear timestamps.
Page 226
Displays the event history (max 10). state Displays port state (up, down, and health), including the port state for SMLT and IST ports. system-stats Displays system port statistics for unicast, multicast, and broadcast packets. Troubleshooting July 2013 Comments? infodev@avaya.com...
Tracing Configure the Route Switch Processor (RSP) Packet Tracing to observe the behavior of the RSP on each R and RS/8800 module. The RSP is the programmable Application Specific Integrated Circuit (ASIC) that controls the ports and traffic flow. Prerequisites •...
Page 228
1024 packets captured. [<interval-value>] Indicates the time interval for which the Packet Tracing is to remain enabled. <interval-value> specifies a value of 1, 10, 30 60, 120, or 300 seconds. The default value is 1 second. Troubleshooting July 2013 Comments? infodev@avaya.com...
Dumping RSP Packet Tracing Variable Value The interval is an optional parameter. If you do not configure the interval, the default value is 1 second. If you do configure it, the time interval changes immediately. On all subsequent occasions when you enable rsp- trace, if you do not specify a new interval value, it is set to the previously set interval value.
Page 230
<slot/port> [<start-pkt> <end-pkt>] Variable definitions Use the information in the following table to help you complete the preceding procedure steps. Variable Value <slot/port> Specifies a port in the lane for which to display the trace. Troubleshooting July 2013 Comments? infodev@avaya.com...
Dumping specified ERCD records Variable Value <pkt-id> Specifies the ID as an integer of the packet to display. value specifies the packet ID as an integer in the range of 1–1024. <start-pkt> Specifies the packet ID of the first packet to display as an integer in the range of 1–...
Specifies an expanded display. value is in the range of 0–3. Using PIM debugging commands Use PIM traces to aid in PIM troubleshooting. Prerequisites Access Global Configuration mode. Procedure steps 1. Start debug trace message output. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 233
Using PIM debugging commands debug ip pim pimdbgtrace 2. Stop debug trace message output. no debug ip pim pimdbgtrace 3. Display trace messages forwarded by the switch. debug ip pim send 4. Display trace messages received by the switch. debug ip pim rcv-dbg-trace 5.
To remove specific debug messages use no neighbor-debug-all mask <WORD 1-100>. 3. Display specific debug messages for BGP peers or peer groups using the following command: neighbor <nbr_ipaddr|peer-group-name> neighbor-debug mask <WORD 1-100> 4. You can also run BGP trace using the following command: Troubleshooting July 2013 Comments? infodev@avaya.com...
Bridging (SPB) configuration. You can use these commands to troubleshoot your SPB configuration. SPBM debugging is done through Connectivity Fault Manager (CFM). For information about using CFM, see Avaya Ethernet Routing Switch 8600/8800 Configuration—Shortest Path Bridging MAC (SPBM), NN46205–525. Prerequisites •...
The following roadmap lists some of the ACLI commands and their parameters that you can use to complete the procedures in this section. Command Parameter Privileged EXEC mode show filter acl debug [<1-4096>] [<1-1000>] show mirror-by-port Global Configuration mode Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 238
Value in-port <portList> Creates a new mirror-by-port table entry. {monitor-mlt <1-256>| • in-port <portList> specifies the monitor-vlan <0-4094>| out- mirrored port. port <portList>} • monitor-mlt <1-256> specifies the mirroring MLT ID from 1 to 256. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 239
Port mirroring configuration Variable Value monitor-vlan <0-4094> • specifies the mirroring VLAN ID from 0 to 4094. • out-port <portList> specifies the mirroring port. enable Enables or disables a mirroring instance already created in the mirror-by-port table. mode <tx|rx|both|rxFilter| Sets the mirroring mode. The default is rx. txFilter|bothFilter>...
• For R modules in Tx modes: use mirror-by-port commands to specify mirroring ports. • For R and RS/8800 modules in Rx modes: use the filter acl ace debug commands to specify Troubleshooting July 2013 Comments? infodev@avaya.com...
Port mirroring configuration Variable Value mirroring ports. See Configuring ACE debug actions to mirror on page 241. Configuring ACE debug actions to mirror Use debug actions to use filters for troubleshooting or monitoring procedures. If you use the mirror action, ensure that you specify the mirroring destination: MLTs, ports, or VLANs.
Enables the ability to count matching packets. mirror enable Enables mirroring. If you enable mirroring, ensure that you configure the appropriate parameters: monitor- • For R and RS/8800 modules in Rx mode: dst-ports monitor-dst-vlan , or monitor- dst-mlt • For R modules in Tx mode: use the...
Configuring remote mirroring [srcMac <0x00:0x00:0x00:0x00:0x00:0x00>] [dstMac <0x00:0x00:0x00:0x00:0x00:0x00>] [ether-type <0x00-0xffff>] [vlan-id <1-4094>] Variable definitions Use the information in the following table to use the remote-mirroring command. Variable Value dstMac Sets the destination MAC address for use in the remote mirroring encapsulation header. The <0x00:0x00:0x00:0x00:0x00:0 mirrored packet is sent to this MAC address.
Roadmap of PCAP ACLI commands The following table lists the commands and their parameters that you use to perform the procedures in this section. Command Parameter Privileged EXEC mode show pcap <cr> capture-filter [id <1-1000>] dump Troubleshooting July 2013 Comments? infodev@avaya.com...
1. Log on to the Primary CPU. 2. Access the Secondary CPU by entering the following command: peer telnet Configuring PCAP global parameters Configure PCAP globally to define how PCAP operates on the Avaya Ethernet Routing Switch 8800/8600. Troubleshooting July 2013...
Page 247
PCAP configuration Prerequisites • The Secondary SF/CPU is installed and active. • If saving to external memory, a PCMCIA card (or external flash on the 8895 SF/CPU) is installed. • Access Global Configuration mode. Procedure steps 1. Enable PCAP using the following command: pcap enable 2.
• If required, ACLs with a global action of mirror exist. • Access Global Configuration mode. Procedure steps 1. Apply filter sets or ACLs to captured packets: pcap <1-1000> pcap acl-filter <1-4096> 2. To enable PCAP on Ethernet ports, use the following command: Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring PCAP capture filters Use capture filters to better define the match criteria used on packets. Avaya highly recommends using PCAP with IP or MAC filters to reduce the load on the PCAP engine. To create a functional capture filter that captures specific packets, create two filters. Use one filter to capture specific packets, and another filter to drop all other packets.
Page 250
PCAP is enabled globally and the trigger filter is disabled. • trigger-off indicates to stop capturing the packet when a packet matches this filter. PCAP is disabled globally and the trigger filter is disabled. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 251
PCAP configuration Variable Value dscp <0-63> [<0-63>] Specifies the DSCP value of the packet. <0-63> is the DSCP from 0 to 63. The default is 0, [match-zero] which means this option is disabled. Use the second <0-63> to specify a range. When match-zero is set, 0 is considered a valid...
The default is 0, which means that this option is [<0-4092>] disabled. Use the second <0-4092> to specify a range. Configuring VLAN MAC filters for PCAP Use PCAP with VLAN MAC address (forwarding database) filters to reduce traffic flow on the PCAP engine. Troubleshooting July 2013 Comments? infodev@avaya.com...
CPU intensive. The switch does not respond to any commands while the dump is in progress. Avaya recommends you use this command only when it is absolutely necessary. However, there is no degradation in normal traffic handling or switch failover.
These commands can be executed in the Primary CPU. copy PCAP00 <WORD 1-99> ftp> get PCAP00 <WORD 1-99> For example: copy PCAP00 /pcmcia/file.cap Variable definitions Use the information in the following table to help you perform this procedure. Troubleshooting July 2013 Comments? infodev@avaya.com...
PCAP configuration Variable Value <WORD 1-99> Specifies pcmcia, flash, or an IP host by IP address and specifies the PCAP file (.cap). Formats include: • a.b.c.d:<file> • /pcmcia/<file> • /flash/<file> Resetting the PCAP DRAM buffer You can clear the PCAP DRAM buffer and the PCAP counters. Prerequisites •...
2. Stop the test after a few seconds: test stop fabric 3. View the results of the test: show test fabric Currently no test is running. Last test results: IfIndex: 0 Result: success Troubleshooting July 2013 Comments? infodev@avaya.com...
Testing the ARP address table PassCount: 62115 FailCount: 0 Job aid Use the information in the following table to understand the test parameters. Field Description IfIndex Specifies the interface index, if applicable. Result Shows the result of the most recently run (or current) test: none, success, inProgress, notSupported, unAbleToRun, aborted, failed.
Prerequisites • Access Interface Configuration mode. Procedure steps 1. Flush IP routing tables by port by entering the following command: action flushIp 2. You can also flush the MAC address and ARP tables: action flushArp Troubleshooting July 2013 Comments? infodev@avaya.com...
Running a ping test action flushMacFdb 3. Clear a routing table using the following commands in Privileged EXEC mode: clear ip route interface fastethernet <portList> clear ip route interface gigabitethernet <portList> clear ip route interface vlan <1-4094> Job aid: ping and traceroute considerations Ping and traceroute may fail for VRF, IPVPN, or MPLS routes if large packet sizes are used for the operation.
Page 260
16 to 4076, or 16 to 65487. The default is 16 octets. interface-port <value> Specifies the interface slot/port number. interface-mgmt <value> Specifies the interface Ethernet management port number. interface-vlan <value> Specifies the interface VLAN ID (1 to 4094). Troubleshooting July 2013 Comments? infodev@avaya.com...
Running a traceroute test Variable Value interface-tunnel Specifies the interface tunnel ID (1 to 2000). <value> scopeid <1-9999> Specifies the circuit scope ID for IPv6 from 1 to 9999. source <WORD 1-256> Specifies the source IP address for use in IP VPN pings.
Specifies the source IP address for use in IP VPN traceroutes. vrf <WORD 0-64> Specifies the VRF instance by VRF name. Running an IPv6 traceroute test Use traceroute to determine the route packets take through a network to a destination. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring Ping Snoop for R series modules Procedure steps 1. Access Privileged EXEC mode. 2. To use traceroute, enter the following command: traceroute <A.B.C.D> [<56-1232>] [-m <1-255>] [-p <1-65535>] [-q <1-255>] [-w <1-255>] [-v] [source <WORD 1-256>] Variable definitions Use the information in the following table to help you use the traceroute <A.B.C.D> command.
Page 264
8. Ensure your configuration is correct: show filter acl 4096 Variable definitions Use the information in the following table to help you use these commands. Variable Value <A.B.C.D> Specifies the source or destination IP address. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring the SLA Monitor agent Configuring the SLA Monitor agent Configure the SLA Monitor agent to communicate with an SLA Monitor server to perform Quality of Service (QoS) tests of the network. Before you begin • To use the SLA Monitor agent, you must have an SLA Monitor server in your network. About this task To configure the agent, you must enable it and assign an IP address.
If the agent task functions as expected, perform typical troubleshooting steps to verify agent accessibility: • Verify IP address assignment and port use. • Ping the server IP address. • Verify the server configuration. Variable definitions Use the data in the following table to use the slamon command. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 267
If you install a certificate on the agent, you must ensure a matching configuration on the server. By default, the agent uses an Avaya SIP certificate to secure communications with the server. oper-mode enable Enables the SLA Monitor agent.
Page 268
Software troubleshooting tool configuration using ACLI Troubleshooting July 2013 Comments? infodev@avaya.com...
Open Shortest Path First (OSPF) generate SNMP traps, use the following command: config ip ospf trap enable. For more information about how to configure SNMP community strings, and for related topics, see Avaya Ethernet Routing Switch 8800/8600 Security, NN46205-601.
Page 270
SnmpV3 > Target Table > Target Params Table). TMask Specifies the mask. The value can be empty or in six- byte hex string format. Tmask is an optional parameter that allows an entry in the TargetAddrTable to specify multiple addresses. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring target table parameters Variable Value Specifies the maximum message size. The size can be zero, or 484 to 2147483647. The default is 484. Although the maximum MMS is 2147483647, the switch supports the maximum SNMP packet size of 8192. Configuring target table parameters Configure the target table to set parameters such as SNMP version and security levels.
Identifies the IP address for the trap sender. Configuring an SNMP notify table Configure the notify table to select management targets to receive notifications, as well as the type of notification to send to each management target. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring SNMP notify filter profile table parameters Procedure steps 1. In the navigation tree, open the following folders: Configuration > Edit > SNMPv3. 2. Click Notify Table. 3. Click Insert. 4. In the Name box, type a notify table name. 5.
1. In the navigation tree, open the following folders: Configuration > Edit > SNMPv3. 2. Click Notify Table. 3. Click the Notify Filter Table tab. 4. Click Insert. 5. In the NotifyFilterProfileName box, type a name for the notify filter profile. Troubleshooting July 2013 Comments? infodev@avaya.com...
Enabling SNMP trap logging 6. In the Subtree box, type subtree location information in x.x.x.x.x.x.x.x.x.x. format. 7. In the Mask box, type the mask location in hex string format. 8. From the Type options, select included or excluded to set filter flag. 9.
Page 276
Enables or disables the sending of traps after an error occurs. LastErrorCode Specifies the last reported error code. LastErrorSeverity Specifies the last reported error severity: 0= Informative Information 1= Warning Condition 2= Error Condition 3= Manufacturing Information 4= Fatal Condition Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 14: Log configuration using Enterprise Device Manager Use log files and messages to help perform diagnostic and fault management functions. Configuring the system log Use the system log to track all user activity on the switch. The system log can send messages to up to ten syslog hosts.
Variable Value Specifies the ID for the syslog host. IpAddr Specifies the IP address of the syslog host. UdpPort Specifies the UDP port to use to send messages to the syslog host (514 to 530). Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 279
Configuring the system log table and severity level mappings Variable Value Enable Enables or disables the sending of messages to the syslog host. HostFacility Specifies the syslog host facility used to identify messages (LOCAL0 to LOCAL7). The default is LOCAL7. Severity Specifies the message severity for which to send syslog messages.
Page 280
Log configuration using Enterprise Device Manager Troubleshooting July 2013 Comments? infodev@avaya.com...
OSPF to generate SNMP traps, use the following command: config ip ospf trap enable. For information about how to configure SNMP community strings and for related topics, see Avaya Ethernet Routing Switch 8800/8600 Security, NN46205-601. Roadmap of SNMP trap CLI commands The following roadmap lists some of the CLI commands and their parameters that you can use to complete the procedures in this section.
3. To specify the required type for an existing notification, enter the following command: config snmp-v3 notify type <Notify Name> new-type <value> 4. Ensure that the configuration is correct: config snmp-v3 notify info show snmp-v3 notify Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring an SNMP host target address Variable definitions Use the information in the following table to complete the config snmp-v3 notify command. Variable Value create <Notify Name> [tag Creates an SNMP trap notification entry. <value>] [type <value>] • <Notify Name> is the index of the notify table with a string length of 1 to 32.
Page 286
<Target Name> new- Specifies a new mask for the target. mask <value> mms <Target Name> new- Specifies a new maximum message size (MMS) mms <value> associated with an entry in the target address table. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring SNMP target table parameters Variable Value Although the maximum value for the MMS is 2,147,483,647, the device supports the maximum SNMP packet size of 8192 (8K). parms <Target Name> new- Specifies a new string value that identifies target address parms <value>...
Page 288
Specifies a new security level. The valid options are seclevel <value> noAuthNoPriv, authNoPriv, and authPriv. sec-name <Tparm Name> [new- Specifies a new security name (readview or writeview), secname <value>] which identifies the principal that generates SNMP messages. Example of configuring additional target parameters Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring an SNMP notify filter table Procedure steps Configure target table parameters: config snmp-v3 target-param create TparamV2 mp--model snmpv2c sec-level noAuthNoPriv sec-name readview Configuring an SNMP notify filter table Configure the notify table to select management targets to receive notifications, as well as the type of notification to send to each management target.
• <subtree oid> identifies the filter subtree with a string length of 1 to 32. • new-type <value> specifies include or exclude. Configuring SNMP interfaces If the Avaya Ethernet Routing Switch 8800/8600 has multiple interfaces, configure the IP interface from which the SNMP traps originate. Troubleshooting July 2013...
Page 291
Configuring SNMP interfaces Procedure steps 1. Configure the destination and source IP addresses for SNMP traps: config sys set snmp sender-ip <dest-ipaddr> <src-ipaddr> 2. If required, send the source address (sender IP) as the sender network in the notification message: config sys set snmp force-trap-sender true 3.
Use the information in the following table to help you use the config snmp snmplog command. Variable Value <enable|disable> Enables or disables the logging of traps. info Displays information about SNMP logging. maxfilesize <64-256000> Specifies the maximum file size for the trap log. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring a UNIX system log and syslog host Configuring a UNIX system log and syslog host Use the syslog commands to control a facility in UNIX machines that logs SNMP messages and assigns each message a severity level based on importance. Procedure steps 1.
Page 294
<info|warning| error| Specifies the severity levels for which to send syslog fatal> [<info|warning| error| messages for the specified modules. <severity> is the fatal>] [<info|warning| error| severity for which syslog messages are sent. fatal>] [<info|warning| error| fatal>] Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 295
Configuring a UNIX system log and syslog host Variable Value udp-port <port> Specifies the User Datagram Protocol port number on which to send syslog messages to the syslog host. <port> is the UNIX system syslog host port number (514 to 530). Troubleshooting July 2013...
Page 296
SNMP trap configuration using CLI Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 16: Log configuration using the CLI Use log files and messages to help perform diagnostic and fault management functions. Roadmap of log commands The following roadmap lists some of the command line interface (CLI) commands and the associated parameters for completing the procedures in this section. Command Parameter config bootconfig logfile...
1. Define which messages are logged: config log level [<level>] 2. Write the log file from memory to a file: config log write <str> 3. Use the following table to help you configure other parameters as required. Troubleshooting July 2013 Comments? infodev@avaya.com...
Viewing logs Variable definitions Use the information in the following table to help you use the config log commands. Variable Value clear Clears the log file. info Displays the current log settings. level [<level>] Shows and sets the logging level. <level>...
Page 300
IP with severity levels of ERROR and WARNING. ERS 8610:5# show log file category OSPF|IP severity ERROR|WARNING cpu CPU5 The following example shows you how to display the log messages from a specific log file. ERS 8610:5# show log file name-of-file /pcmcia/sample.txt Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring the remote host address for log transfer Configuring the remote host address for log transfer Configure the remote host address for log transfer. The system transfers the current log file to a remote host when the log file size reaches the configured maximum size. Prerequisites •...
Specify the name of the file stored in the remote host. If not configured, the current log file name is the default. Important: Avaya recommends that you do not set this option. If this option is set, the previously transferred log file is overwritten on the remote server.
Page 303
Configuring system logging to a PCMCIA or external flash Caution: Risk of data loss Before removing the PCMCIA (or external flash) card from your primary SF/CPU, you must stop the logging of system messages. Failure to do so may corrupt the file system on the PCMCIA (or external flash) card and cause your log file to be permanently lost.
To begin or stop logging system messages on the PCMCIA (or external flash) card, use the following command: config log logToPCMCIA <true|false> Variable definitions Use the date in the following table to complete the config log logToPCMCIA command. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring system message control Variable Value <true|false> Begin or stop the logging of system messages on the PCMCIA (or external flash) card. If true is specified, the following message appears: Logging to PCMCIA STARTED . If false is specified, the following message appears: Logging to PCMCIA STOPPED Configuring system message control...
<str> Adds a forced message control pattern, where <str> is a string of four characters. You can add a four-byte pattern into the force-msg table. The software and the hardware log messages that use Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring logging Variable Value the first four bytes matching one of the patterns in the force-msg table undergo the configured message control action. You can specify up to 32 different patterns in the force-msg table including a wild-card pattern (****). When specifying the wild- card pattern, all messages undergo message control.
Page 308
Use the information in the following table to help you use the show clilog file commands. Variable Value tail Shows the last results first. grep <value> Performs a string search in the CLI log file. <value> is the string, of up to 256 characters in length, to match. Troubleshooting July 2013 Comments? infodev@avaya.com...
Specify which protocols and processes generate traps by enabling traps for that protocol. For example, to allow SNMP traps to be generated for OSPF, use the following command: ip ospf trap enable. For information about configuring SNMP community strings and related topics, see Avaya Ethernet Routing Switch 8800/8600 Security, NN46205-601.
Job aid: SNMP configuration in the ACLI Command Parameter mapwarning {emergency|alert| critical|error|warning|notice| info|debug} severity <info|warning|error| fatal> [<info|warning|error| fatal>] [<info|warning|error| fatal>] [<info|warning|error| fatal>] udp-port <514-530> Job aid: SNMP configuration in the ACLI SNMP is configured differently in the ACLI than in the CLI. Auto-generation of several parameters and command structure changes means that several configuration procedures are no longer required in the ACLI.
There are two preconfigured entries in the snmpNotifyTable. These entries cannot be modified or deleted. The ACLI command set does not allow you to create or delete entries in the snmpNotifyTable; this table is automatically generated. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring SNMP notifications Configuring SNMP notifications The SNMP notification table (snmpNotifyTable) is preconfigured and nonconfigurable in the ACLI. Configuring an SNMP host Configure an SNMP host so that the switch can forward SNMP traps to a host for monitoring. You can use SNMPv1, SNMPv2c, or SNMPv3. Prerequisites •...
Page 314
• target-name <WORD 1-32> is the target name with a string length of 1 to 32. v3 {noAuthnoPriv| Creates a new SNMPv3 entry for the target address table. authNoPriv|AuthPriv} <WORD 1-32> [inform [retries <0-255>] [timeout <0-2147483647>]] Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring SNMP target table parameters Variable Value [filter <WORD 1-32>] {noAuthnoPriv|authNoPriv|AuthPriv} • [target-name <WORD specifies the security level. 1-32>] • <WORD 1-32> specifies the security name, which identifies the principal that generates SNMP messages. • inform indicates that SNMP notifications should be sent as inform (rather than trap).
OID with a string length of 1 to 32. If the Subtree OID parameter uses a + prefix (or no prefix), this indicates include. If the Subtree OID uses the - prefix, this indicates exclude. Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring SNMP interfaces Configuring SNMP interfaces If the Avaya Ethernet Routing Switch 8800/8600 has multiple interfaces, configure the IP interface from which the SNMP traps originate. Prerequisites • Access Global Configuration mode. Procedure steps 1. Configure the destination and source IP addresses for SNMP traps: snmp-server sender-ip <A.B.C.D>...
1. To enable SNMP trap logging, use the following command: snmplog enable 2. To configure the maximum log file size: snmplog maxfilesize <64-256000> 3. To view the contents of the SNMP log, use the following command: show snmplog Troubleshooting July 2013 Comments? infodev@avaya.com...
Configuring a UNIX system log and syslog host Variable definitions Use the information in the following table to help you use the snmplog command. Variable Value enable Enables or disables the logging of traps. maxfilesize Specifies the maximum file size for the trap log. <64-256000>...
Page 320
{local0| Specifies the UNIX facility used in messages to the syslog host. {local0|local1|local2| local1|local2|local3| local3|local4|local5|local6|local7} local4|local5|local6| is the UNIX system syslog host facility (LOCAL0 to local7} LOCAL7). enable Enables the syslog host. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 321
Configuring a UNIX system log and syslog host Variable Value maperror {emergency| Specifies the syslog severity to use for Error messages. alert|critical|error| warning|notice|info| debug} mapfatal {emergency| Specifies the syslog severity to use for Fatal messages. alert|critical|error| warning|notice|info| debug} mapinfo {emergency| Specifies the syslog severity level to use for Information messages.
Page 322
SNMP trap configuration using the ACLI Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 18: Log configuration using the ACLI Use log files and messages to help perform diagnostic and fault management functions. Roadmap of ACLI log commands The following roadmap lists some of the ACLI commands and their parameters that you can use to complete the procedures in this section.
1. Define which messages are logged: logging level <0-4> 2. Write the log file from memory to a file: logging write <WORD 1-1536> 3. Use the following table to help you configure other parameters as required. Troubleshooting July 2013 Comments? infodev@avaya.com...
Viewing logs Variable definitions Use the information in the following table to help you use the logging commands. Variable Value level <0-4> Shows and sets the logging level. The level is one of these values: 0 = Information; all messages are recorded.
Configuring the remote host address for log transfer Configure the remote host address for log transfer. The system transfers the current log file to a remote host when the log file size reaches the configured maximum size. Troubleshooting July 2013 Comments? infodev@avaya.com...
0-255> Important: Avaya recommends that you do not set this option. If this option is set, the previously transferred log file is overwritten on the remote server. Configuring system logging to a PCMCIA or external flash System logs are a valuable diagnostic tool.
Page 328
Use the information in the following table to help you use the boot config commands in this procedure. Variable Value flags logging Enables or disables logging to a PCMCIA (or external flash) card. The log file is named using an 8.3 (xxxxxxxx.sss) format. The first six characters Troubleshooting July 2013 Comments? infodev@avaya.com...
Starting system message logging to a PCMCIA or external flash card Variable Value of the file name contain the last three bytes of the chassis base MAC address. The next two characters specify the slot number of the SF/CPU that generated the logs. The last three characters denote the sequence number of the log file.
2. Configure the maximum number of messages by using the following command: sys msg-control max-msg-num <2-500> 3. Configure the interval by using the following command: sys msg-control control-interval control-interval <1-30> 4. Enable message control: sys msg-control Troubleshooting July 2013 Comments? infodev@avaya.com...
Extending system message control Variable definitions Use the information in the following table to complete the sys msg-control command. Variable Value <cr> Activates system message control. Enabling this command suppresses duplicate error messages. action <suppress-msg| Configures the message control action. The default value is suppress-msg.
Procedure steps 1. To enable ACLI logging, enter the following command: clilog enable 2. To change the maximum file size used for ACLI logs: clilog maxfilesize <64 to 256000> 3. Ensure that the configuration is correct: Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 333
Configuring ACLI logging show clilog 4. To view the ACLI log: show clilog file [tail] [grep <WORD 1-256>] Variable definitions Use the information in the following table to help you use the clilog commands. Variable Value enable Enables ACLI logging. To disable, use the clilog enable command.
Page 334
Log configuration using the ACLI Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 19: Recovery trees and procedures This section provides problem scenarios and recovery procedures. Recovery trees Recovery trees provide a quick reference for troubleshooting without procedural detail. They are meant to quickly guide you through some common failure scenarios and provide a solution.
Recovery trees and procedures Figure 5: IST failure recovery tree DHCP Relay failure Use the following flowchart to help provide guidance if DHCP Relay fails. For more information, DHCP troubleshooting on page 438. Troubleshooting July 2013 Comments? infodev@avaya.com...
Recovery trees Figure 6: DHCP Relay failure recovery tree SNMP failure Use the following flowchart to help provide guidance if Simple Network Management Protocol (SNMP) access fails. For more information, see SNMP troubleshooting on page 437. Troubleshooting July 2013...
Use the following flowchart to help provide guidance if you cannot write to the flash. Caution: Risk of file loss Before you format the flash, ensure you back up all files. Formatting the flash deletes all files, including configuration and license files. Troubleshooting July 2013 Comments? infodev@avaya.com...
Licensing problems and recovery The following sections describe licensing problems you may encounter and provides solutions. For more information about installing licenses and accessing the License Bank, see Avaya Ethernet Routing Switch 8800/8600 Administration, NN46205-605. Licensing problems and recovery navigation •...
Flash memory, but only one is used to unlock features. If two license files are installed on an Avaya Ethernet Routing Switch 8800/8600, for example, one Advanced and one Premier license file, the Premier License takes precedence over the Advanced License.
- incorrect MAC address specified when the license file was generated Prerequisites • Use the show license command to double-check that the license is not installed. • For more information about transferring a license and installing a license, see Avaya Ethernet Routing Switch 8800/8600 Administration, NN46205-605. Procedure steps 1.
2. Check the flash directory for the presence of the license file: The license file must be stored in the flash directory. 3. Ensure that the license file name and location follows the guidelines given in Avaya Ethernet Routing Switch 8800/8600 Administration, NN46205-605.
Page 343
Licensing problems and recovery 4. Check switch log messages for errors during boot: show clilog file 5. After the license filename and location criteria for the license file is correct, restart the switch: boot Troubleshooting July 2013...
Page 344
Recovery trees and procedures Troubleshooting July 2013 Comments? infodev@avaya.com...
If a log or trap is present, take remedial actions to clear the alarm, depending on the type of alarm. Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting DWDM XFPs Additional useful commands The following CLI commands also provide useful information relating to DWDM XFPs: • show sys pluggable-optical-module info [<portlist>] • show sys pluggable-optical-module info [<portlist>] [detail] • show sys pluggable-optical-module temperature [<portlist>] • show sys pluggable-optical-module voltage [<portlist>] •...
Page 348
Layer 1 troubleshooting Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 21: Layer 2 troubleshooting Use this section to help you troubleshoot Layer 2 (VLAN, spanning tree, link aggregation, multilink trunking) problems. Troubleshooting SMLT failure using the CLI or ACLI In typical SMLT topologies, critical network traffic runs over the aggregated links. Therefore, SMLT failure can cause a partial network outage when network protocols such as VRRP, OSPF, and RIP, and link layer protocols such as VLACP register SMLT links going down and (potentially) recovering.
Page 350
ACLI command: show mlt 7. After the SMLT links recover, to inspect forward database (FDB) table software records for their integrity, enter the following CLI command: show vlan info fdb-e [<vid>] enter the following ACLI command: Troubleshooting July 2013 Comments? infodev@avaya.com...
When the VLACP protocol goes up and down (flaps) rapidly, this behavior can cause system-wide instability, including high CPU utilization. For more information on how to resolve VLACP flapping, see Avaya Ethernet Routing Switch 8800/8600 Configuration — Link Aggregation, MLT, and SMLT (NN46205-518).
When the VLACP protocol goes up and down (flaps) rapidly, this behavior can cause system-wide instability, including high CPU utilization. For more information on how to resolve VLACP flapping, see Avaya Ethernet Routing Switch 8800/8600 Configuration — Link Aggregation, MLT, and SMLT (NN46205-518).
Troubleshooting IstSessionDown message using CLI or ACLI If the Avaya Ethernet Routing Switch 8800/8600 shows the log message IstSessionDown for no apparent reason or additional information as to why, use show mlt ist stats (CLI) or show ist stat (ACLI) to determine if a switch is having difficulty sending or receiving the IST messages.
Troubleshooting BPDU filtering In this output, BPDU filtering is enabled on port 4/1, BPDU packets are received, port 4/1 is disabled, and the TimerCount is incrementing, but no SNMP trap is ever received. 2. Configure the correct SNMP target information on the switch. config snmp-v3 target-addr create <Target Name>...
Page 356
Layer 2 troubleshooting Troubleshooting July 2013 Comments? infodev@avaya.com...
339. The following table describes the license required to use specific features. The Premier license enables all licensed features on the Ethernet Routing Switch 8800/8600. For more information about installing or transferring licenses, see Avaya Ethernet Routing Switch 8800/8600 Administration (NN46205-605).
OSPF down state or no state problems on page 361 • OSPF Init state problems on page 362 • OSPF ExStart/Exchange problems on page 363 Viewing OSPF errors Check OSPF errors for administrative and troubleshooting purposes. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 359
OSPF troubleshooting Procedure steps 1. In the CLI, to display extended information about OSPF errors: show ports error ospf [<ports>] 2. In the ACLI, to display extended information about OSPF errors: show interfaces fastEthernet ospf [<portList>] [<1-4094>] show interfaces fastEthernet ospfv3 [<portList>] [<1-4094>] show interfaces gigabitEthernet ospf [<portList>] [<1-4094>] show interfaces gigabitEthernet ospfv3 [<portList>] [<1-4094>]...
Indicates the link state database (LSDB) is exchanged Loading Indicates the processing state of the LSDB for input into the routing table. The router can request LSA for missing or corrupt routes. Full Indicates the normal full adjacency state. Troubleshooting July 2013 Comments? infodev@avaya.com...
OSPF troubleshooting OSPF down state or no state problems If no state information appears for a neighbor when you enter the show ip ospf neighbor command, then the router has not received valid OSPF hello packets from the neighbor. If a neighbor appears in the down state, that neighbor is typically a manually-configured neighbor (using the config ip ospf neighbor command).
Check that the path is not reachable due to access lists implemented on routers. Ensure the multicast address of 224.0.0.5 is able to traverse the link. If multicast traffic is blocked, you must to configure the Avaya Ethernet Routing Switch 8800/8600 for OSPF NBMA instead of Broadcast.
MTU size for OSPF is 1500 bytes by default. Incoming OSPF database description (DBD) packets are dropped if their MTU size is greater than 1500 bytes. To allow the Avaya Ethernet Routing Switch 8800/8600 to accept OSPF database description packets with a different MTU size, enable mtu-ignore.
(CLI or ACLI) 6. Check whether the nexthop-self is enabled. (In case of eBGP peers it is always safe to have nexthop-self enabled) show ip bgp neighbor info (CLI) show ip bgp neighbor (ACLI) Troubleshooting July 2013 Comments? infodev@avaya.com...
BGP+ troubleshooting Additional considerations when troubleshooting MSTP/SMLT co-existence 1. Verify that both boxes are operating in the same mode (4 or 2byte). If 4Byte, the 4- byte-as flag needs to be enabled globally. 2. Attempt to ping the destination network to ensure it is reachable. 3.
<neighbor-ip> neighbor-debug mask <value> (ACLI BGP Router Configuration mode) 3. In addition to BGP trace and debug, you can also run RTM traces to see if the routes are being lost somewhere during the RTM processing. trace level 45 3 Troubleshooting July 2013 Comments? infodev@avaya.com...
1. Ensure that iBGP neighborship is properly established between the two provider edge (PE) routers. 2. Ensure that the proper I/O modules are used. IP VPN Lite requires R or RS/8800 modules. 3. Ensure that the Import and Export route targets (RT) are matched between PEs 4.
Page 368
OSPF) or properly redistributed (if the interior protocol is RIP or Static). 6. If you use a sniffer, the VPN traffic should have an outer IP header with the RD as the source and destination IP address. Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 23: Multicast feature troubleshooting Use the information in this section to help you troubleshoot multicast feature problems. For additional multicast routing troubleshooting information, see: • Multicast troubleshooting using EDM on page 387 • Multicast troubleshooting using CLI on page 397 •...
If a querier is not elected, use the following procedure to troubleshoot the issue. IGMP Layer 2 Querier is based on IGMP snoop, check whether IGMP snoop is enabled on the VLAN: config vlan <vid> ip igmp info (CLI) show ip igmp interface vlan <vid> (ACLI) Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting static mroute IGMP snoop and IGMP Layer 2 querier must be enabled for the feature to function. Enabling trace messages for IGMP Layer 2 querier troubleshooting If the preceding information does not address your issue, you can also use the following trace command to view additional information related to Layer 2 querier: trace level 23 <1-4>...
Page 372
If the multicast route is not the selected route, then note the preference value of the selected route. The following sample output shows that the route to destination 20.20.20.2 takes a unicast route. 8600# show ip mroute rpf Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 373
Troubleshooting static mroute ================================================================ ======== Mroute Route - GlobalRouter ================================================================ ======== DEST MASK PREF OWNER ---------------------------------------------------------------- -------- 20.20.20 255.255.255.255 2.2.2.1 OSPF After you properly configure the static-mroute, the route to destination 20.20.20.2 takes the Multicast route, as shown in the following sample output. 8600# show ip mroute rpf ================================================================ ========...
Page 374
Use this command to verify whether the RPF address in the route is a PIM neighbor. If it is not, then the route is not selected. The following shows sample output for this command. Troubleshooting July 2013 Comments? infodev@avaya.com...
PIM tree. To display the configured RP sets, enter the following CLI command: show ip pim rp-set Enter the following ACLI command: show ip pim rp-hash Troubleshooting July 2013 Comments? infodev@avaya.com...
1. Verify that the source is reachable form the host. If not, verify that the unicast routing protocol (like OSPF) has fully converged. 2. Check that the SSM group falls within the SSM range configured on the 8800/8600 switch. If the multicast group range does not fall in the SSM group range, the (S,G) record is not created on the receiver side.
• MD5 password configuration mismatch. • MSDP peer is not reachable through the routing table. Possible reason for MSDP SAs not being received and advertised: • no active RPs • no source reachability • no RP reachability Troubleshooting July 2013 Comments? infodev@avaya.com...
MSDP • originator ID not configured on RP switch • RPF check failure See the following sections to troubleshoot a specific MSDP issue: • MSDP peer not established on page 379 • MSDP peer established, but no MSDP local cache and foreign cache entries on page MSDP peer not established If the MSDP peer is not established, perform the following procedure.
Page 380
CLI or ACLI: show ip msdp count 9. To display the configured mesh groups, enter the following command using the CLI or ACLI: show ip msdp mesh-group Troubleshooting July 2013 Comments? infodev@avaya.com...
Multicast virtualization Multicast virtualization In most cases, a virtualized multicast instance functions in an identical manner to a nonvirtualized multicast instance. There are some exceptions to avoid. For example, a virtualized multicast instance cannot function properly in a PIM-DM or DVMRP only network, because the protocols are not supported for virtualization.
PIM instance before you can begin PIM configuration. To create a PIM instance, perform the following procedure. Procedure steps To create a PIM instance, enter the following CLI command: config ip vrf <vrf-name> pim create From VRF Router Configuration mode, enter the following ACLI command: Troubleshooting July 2013 Comments? infodev@avaya.com...
Multicast virtualization ip pim enable Cannot create a PIM instance on a VRF If you cannot create a PIM instance on a VRF, possible causes include the following: • The VRF does not exist. • The maximum PIM instance count is reached. To troubleshoot, perform the following procedure.
Cannot enable IGMPv3 on a VLAN If you cannot enable IGMPv3 on a VLAN, a possible cause is that the VLAN is associated with a VRF which does not have a PIM instance created. To troubleshoot, perform the following procedure. Troubleshooting July 2013 Comments? infodev@avaya.com...
Multicast virtualization Procedure steps 1. To check the VRF association for the VLAN, enter the following CLI command: show vlan info ip enter the following ACLI command: show interface vlan ip 2. To ensure the VRF PIM instance is in SSM mode, enter the following CLI command: show ip pim info vrf <vrf-name>...
Page 386
Multicast feature troubleshooting Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 24: Multicast troubleshooting using EDM Use the information in this section to help you troubleshoot multicast routing problems using Enterprise Device Manager (EDM). Viewing IGMP snoop trace information Use this procedure to view multicast trace information for IGMP snoop, which tracks the data flow path of the multicast streams.
You can use the Source tab to view these sources. Invlan Specifies the ingress VLAN ID where the traffic emanates for the multicast stream. Pruned Specifies whether the route had been pruned. True indicates that the multicast stream has been pruned. Troubleshooting July 2013 Comments? infodev@avaya.com...
Viewing pruned multicast routes Viewing pruned multicast routes The Prunes tab shows all of the prunes received for the Group address in the multicast stream selected from the Mroute-HW table. Procedure steps 1. In the navigation tree, open the following folders: IP > Multicast. 2.
Use the information in the following table to help you use the EgressVlans tab. Variable Value EgressVlan All the egress VLANs for the particular multicast stream selected from the Mroute-HW table. EgressVlanPorts The corresponding ports for the particular multicast stream selected from the Mroute-HW table. Troubleshooting July 2013 Comments? infodev@avaya.com...
Protocol (PAP) Join requests received for members of this interface. BasicQuery Indicates the number of Basic Query messages sent by the Avaya Ethernet Routing Switch 8800/8600 on an IGAP-enabled interface. BasicLeave Indicates the number of Basic Leave messages received by this interface.
• snoop-spb • routed-spb • pim Procedure steps 1. In the navigation tree, open the following folders: Configuration > IP > IGMP. 2. Click the Interface tab. The protocol is shown under the Mode column. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 393
Determining the protocol configured on the added VLAN Variable definitions Use the information in the following table to help you understand the fields in the Interface tab. Field Description IfIndex Indicates the interfaces where IGMP is configured QueryInterval Indicates the frequency at which the interface transmits IGMP host query packets Mode Specifies the protocol used on the added VLAN.
Indicates the multicast mode. Determining the data stream learned when IP multicast over SPBM is configured on the VLAN To determine the data stream learned when IP multicast is configured on the VLAN, use the following procedure: Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 395
Determining the data stream learned when IP multicast over SPBM is configured on the VLAN Procedure steps In the navigation tree, open the following folders: Configuration > IP > Multicast. Variable definitions Use the information in the following table to help you use the Multicast tab. Field Description Group...
Indicates that the data is received at the UNI ports Network Recv Indicates that the data is received at the SPBM cloud Total number of SPBM Specifies the number of SPBM IP multicast route entries IP_MULTICAST Route entries Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 25: Multicast troubleshooting using CLI Use the procedures in this section to help you troubleshoot multicast routing problems using the command line interface (CLI). Viewing multicast group trace information for IGMP snoop Multicast group trace tracks the data flow path of the multicast streams. Procedure steps Display the multicast group trace for an IGMP snoop-enabled interface using the following command:...
Table 21: show ip pgm interface error general parameters Field Description Displays the circuit number of the selected interface IN_SPM PORT ERRORS Displays the number of SPMs discarded because they were received on the wrong interface Troubleshooting July 2013 Comments? infodev@avaya.com...
Viewing PGM negative acknowledgement errors Field Description IN_RDATA PORT Displays the number of RDATA packets discarded because ERRORS they were received on the wrong interface IN_RDATA Displays the number of RDATA packets discarded because NO_SESSION ERRORS there was no active session IN_NCF PORT ERRORS Displays the number of NCFs discarded because they were received on the wrong interface IN_NCF NO_SESSION...
• When you follow the command with <value> <value>, the output includes only the specified source-group pair. group-prune-state [grp Use the show ip mroute-hw group-prune- <value>] state command as follows: Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 401
Viewing multicast routes Variable Value • When you use it by itself, the output includes all the group entries found in the hardware records. • When you follow the command with <value>, the output includes all the entries corresponding to the specified group address.
Showing the hardware resource usage The Avaya Ethernet Routing Switch 8800/8600 can query the number of ingress and egress IP Multicast streams traversing your switch. After setting the thresholds for ingress and egress records, if the record-usage goes beyond the threshold, you are notified by way of a trap on the console, logged message, or both.
Viewing multicast routing process statistics Field Description SEND TRAP ONLY Indicates the status of sending traps only SEND TRAP AND LOG Indicates the status of sending traps and log messages Viewing multicast routing process statistics To provide additional troubleshooting information, you can enable the collection and display of multicast routing process statistics.
Page 404
Subsequent outputs provide valid values. DropPackets Specifies the number of dropped packets for the associated VRF and Group IP address DropBytes Specifies the number of dropped bytes for the associated VRF and Group IP address Troubleshooting July 2013 Comments? infodev@avaya.com...
Determining the protocol configured on the added VLAN Determining the protocol configured on the added VLAN The protocol configured on the added VLAN can be one of the following values: • snoop • snoop-spb • routed-spb • pim Procedure steps Determine the protocol that is configured on the VLAN added by using the following command: show ip igmp interface [vrf <value>] [vrfids <value>]...
Page 406
Use this value to modify the leave latency of the network. A reduced value results in reduced time to detect the loss of the last member of a group. This does not apply if igmpInterface version is 1. Troubleshooting July 2013 Comments? infodev@avaya.com...
Determining the data stream learned when IP multicast over SPBM is configured on the VLAN Determining the data stream learned when IP multicast over SPBM is configured on the VLAN To determine the data stream learned when IP multicast is configured on the VLAN, use the following procedure: Procedure steps Determine the data stream learned using the following command:...
Specifies the IP multicast group for which this entry specifies a next hop on an outgoing interface VSN-ISID Specifies the VSN I-SID Data ISID Specifies the VRF ID for the multicast route BVLAN Specifies the backbone VLAN Troubleshooting July 2013 Comments? infodev@avaya.com...
Defining the IS-IS trace flag for IP multicast Field Description Source-BEB Specifies the source backbone edge bridge Defining the IS-IS trace flag for IP multicast To define the IS-IS trace flag for IP multicast, use the following procedure: Procedure steps Define the IS-IS trace flag by using the following command: trace flags isis [set ip-multicast] [remove ip-multicast] Troubleshooting...
Page 410
Multicast troubleshooting using CLI Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 26: Multicast troubleshooting using the ACLI Use the information in this section to help you troubleshoot multicast routing problems using the Avaya command line interface (ACLI). Viewing multicast group trace information for IGMP snoop Multicast group trace tracks the data flow path of the multicast streams.
Indicates the outgoing port number. Viewing PGM interface errors Display general information about PGM errors that occurred on the selected interface. Prerequisites • Access Privileged EXEC mode. Procedure steps Show PGM interface errors using the following command: Troubleshooting July 2013 Comments? infodev@avaya.com...
Viewing PGM negative acknowledgement errors show ip pgm interface error Job aid: show ip pgm interface error general command The following table describes fields for this command. Table 30: show ip pgm interface error general parameters Field Description Displays the circuit number of the selected interface. IN_SPM PORT ERRORS Displays the number of SPMs discarded because they were received on the wrong interface.
TG window. Viewing multicast routes The multicast route table contains multicast routing information for all group addresses. Prerequisites • Access Privileged EXEC mode. Procedure steps 1. Show group trace information using the following command: Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 415
Viewing multicast routes show ip mroute hw-group-trace [<A.B.C.D>] [<A.B.C.D>] 2. Show group prune state information using the following command: show ip mroute hw-group-prune-state [<A.B.C.D>] Variable definitions Use the information in the following table to use the show ip mroute commands. Variable Value hw-group-trace...
Showing the hardware resource usage The Avaya Ethernet Routing Switch 8800/8600 can query the number of ingress and egress IP Multicast streams traversing your switch. After setting the thresholds for ingress and egress records, if the record-usage goes beyond the threshold, you are notified by way of a trap on the console, logged message, or both.
Viewing multicast routing process statistics Prerequisites • Access Privileged EXEC mode. Procedure steps Show the hardware resource usage using the following command: show ip mroute hw-resource-usage Job aid: show ip mroute hw-resource-usage command The following table shows the field descriptions for this command. Table 33: show ip mroute hw-resource-usage parameter Field Description...
Page 418
Specifies the VRF name for which to display statistics. [vrfids <vrfids>] Specifies the VRF ID for which to display statistics. Job aid: show ip mroute stats command The following table shows the field descriptions for this command. Troubleshooting July 2013 Comments? infodev@avaya.com...
Determing the protocol configured on the added VLAN Table 34: show ip mroute stats <grp-ip-list> parameters Field Description GroupAddress Specifies the multicast group IP address for which statistics are displayed. SourceCounter Specifies the source number corresponding to the associated VRF and group IP address in the whole multicast route records.
Page 420
This object configures a router capable of running either version. For IGMP to function correctly, you must configure all routers on a LAN to run the same version of IGMP. OPER VERS Indicates the operational version of IGMP. Troubleshooting July 2013 Comments? infodev@avaya.com...
Determining the data stream learned when IP multicast over SPBM is configured on the VLAN Field Description QUERIER Indicates the address of the IGMP querier on the IP subnet to which this interface is attached. QUERY MAXRSPT Indicates the maximum query response time (in tenths of a second) advertised in IGMPv2 queries on this interface.
Page 422
For IP multicast over SPBM, this value is sb-access or spb-network. Spb-access indicates the datastream learned was from the UNI ports. Spb-network indicates that the datastream learned was from the SPBM cloud. Troubleshooting July 2013 Comments? infodev@avaya.com...
Showing the SPBM multicast database Showing the SPBM multicast database You can determine the database used by the SPBM multicast module by using the following procedure: Procedure steps Show the SPBM multicast database by using the following command: show isis spbm ip-multicast-route [all | detail | group {A.B.C.D}| vlan <0–4094>...
Page 424
Multicast troubleshooting using the ACLI Procedure steps Define the IS-IS trace flag for IP multicast by using the following command: trace flags isis [set ip-multicast] [remove ip-multicast] Troubleshooting July 2013 Comments? infodev@avaya.com...
WCP and the WSP. The commands in this section are specific to the Unified Access. For more information about other troubleshooting tools and commands, go to Avaya WLAN 8100 Troubleshooting Guide (NN46251–700) and Avaya Ethernet Routing Switch 8800/8600 Troubleshooting (NN46205–703) Troubleshooting WCP and AP problems Use the procedures in this section to troubleshoot WCP and AP problems.
DHCP option 43 for APs provides the non-default base port to the AP. 6. If none of the above steps resolves the problem, get the switch technical dump and send it to Avaya Support. show tech Joining the mobility domain After you enable the wireless system, you should be able to join the mobility domain.
If a WSP MAC entry is not listed, then that WSP is not being managed by any of the WCPs in the mobility domain. 2. Verify that a circuitless IP address is configured in the ERS 8800/8600 and that wireless is enabled.
3. Check to see if the WSP MAC address is present in the domain WSP database: show wireless domain wsp database 4. Check the WCP load balancing table to see which WCP is assigned to manage the WSP and the current joining status: show wireless domain load-balance wsp-lb-table Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting WCP and AP problems Managing an AP Use the following steps to troubleshoot why an AP is not being managed by a WCP. Procedure 1. Check to see if the AP MAC entry is present in the managed AP: show wireless ap status 2.
802.11 frames like probe-request, authentication and association frames to the Access Point. For more information on this feature, see Avaya WLAN 8100 Fundamentals (NN47251–102) and the Avaya WLAN 8100 WC 8180 CLI Reference (NN47251–107) for a complete list of configuration commands for this feature. Procedure 1.
5. If the SSID is not broadcasting, make sure the radio profile is mapped properly in the AP profile. 6. If the SSID is not broadcasting, get the AP technical dump and send it to Avaya Support: wireless ap tech-dump <ap-mac> <ap-ip> filename <name>.txt Troubleshooting WSP problems Use the procedures in this section to troubleshoot problems with a WSP.
Page 432
9. Check the WSP's peer device information: (CLI) show wireless-switch peer-devices [ap | switch] (ACLI) show wireless switch peer-devices [ap | switch] 10. Check the WLAN VLAN FDB entries that the WSP has learned: Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting WSP problems (CLI) show wireless-switch fdb-entry [vid <value>] [mac <value>] (ACLI) show wireless switch mac-address-entry [vid <value>] [mac <value>] 11. Verify that the wireless tunnel is added to the VLAN that is mapped or what is configured in the VLAN reservation in case of L3 Roaming. The wireless tunnel appears like any other port member except it uses a WT-n designation.
5. Determine the egress port of the tunnel and then mirror that port to verify that the keepalive is sent out. Identify the keepalive packet in the trace by looking for a UDP packet with an IP header length of 36: Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting WSP problems config rsp-trace egress-pkt-trace port <value> state enable interval <value> 6. If the data packets are not tunnelled, check the state of the tunnel and the tunnel statistics: dump wireless tunnel dump wireless stats 7. Check ERCD ( Enterprise RSP Control Driver) Record Tables to verify if the wireless tunnels are configured as destinations: dump ercdRecord arp slot <value>...
Page 436
Troubleshooting the Unified Access solution Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 28: Upper layer troubleshooting This section describes troubleshooting for Layer 4 to 7 applications. SNMP troubleshooting This section describes methods to troubleshoot the following Simple Network Management Protocol (SNMP) scenario: • network management station (NMS) not receiving traps Verify your management configurations for your management station. Also verify your management station setup.
6. Obtain sniffer traces where the traffic ingresses and egresses the switch and also on the client side of the network. Troubleshooting IPv6 DHCP Relay For a DHCP Relay recovery flowchart, see DHCP Relay failure on page 336 Troubleshooting July 2013 Comments? infodev@avaya.com...
With DHCP Relay, the Avaya Ethernet Routing Switch 8800/8600 only participates in forwarding the requests and replies to and from the client and the DHCP server. The Ethernet Routing Switch 8800/8600 always acts as the relay agent, on which you configure the forward path to the server.
See the following sections to troubleshoot the specified BFD issue: • BFD session stays in down state on page 441 • BFD enabled on OSPF or BGP, but session not created on page 441 • BFD session flaps on page 442 Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting BFD • BFD session goes down when MLT member ports are enabled or disabled on page • BFD with trace on on page 444 BFD session stays in down state If the BFD session stays in the down state, perform the following steps: Procedure steps 1.
2. If the multiplier parameter is less than 3, to change it to 3 or higher, enter the following CLI command: config {vlan <vid> | ethernet <portlist>} ip bfd multiplier <2-20> enter the following ACLI command: (config-if)# ip bfd [port <portlist> | vlan <vid>] multiplier <2-20> Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting BFD 3. Verify whether the detect time is small while too many BFD sessions exist. To increase the detect time by changing the rx-interval and tx-interval parameters, enter the following CLI command: config {ethernet <portlist> | vlan <vlan-id>} {rx-interval <milliseconds>...
<mid> member <portlist> mlt <mid> member <portlist> BFD with trace on The Avaya Ethernet Routing Switch 8800/8600 supports only one static or OSPF session with trace on (trace level 115 3) (CR Q018374647). Troubleshooting IPv6 VRRP The following sections describe troubleshooting information for IPv6 VRRP...
Page 445
Troubleshooting IPv6 VRRP Type value Type definition Master to Backup Backup to Master Initialize to Master Master to initialize Initialize to Backup Backup to Initialize Backup to Backup Master Backup Master to Backup The following table describes the VRRP transition causes. Table 39: Transition cause Cause value Cause definition...
If the Master-Backup relationship is not formed between the VRRP virtual routers, look for the following trace messages to ensure that the master is sending the advertisements correctly and the backup is processing them. Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting IPv6 VRRP On the master router, look for the following RCIP6 trace messages. • tMainTask RCIP6: rcip6_vrrp.c: 5118: VRF name: GlobalRouter (VRF id 0): ipv6VrrpTic: Am Master for Vrid 200 on IfIndex 2053 Timer If VRRP is enabled on the interface, this timer kicks off every second and shows the state for the VRID.
• You must configure IST peers with the same IPv6 subnets on the SMLT VLANs (same as for IPv4). • Make sure that IST MLTs on the RSMLT peers contain the same set of links (this is very difficult to catch through regular troubleshooting) Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting IPv6 RSMLT • Do not enable transmission of IPv6 ICMP Redirect messages on RSMLT VLANs (ICMP redirect is disabled by default). • No IPv6 connectivity is expected when the IST is down. Traffic recovery from an IST going down takes multiple seconds. RSMLT peers not up If, after a series of reconfigurations, RSMLT peers do not transition to the up state, use the following procedure to troubleshoot the issue.
To troubleshoot RADIUS issues on the Ethernet Routing Switch 8800/8600, use the following procedure. 1. Make sure the RADIUS server is reachable from the NAS (or 8800/8600 switch). 2. Make sure authentication and accounting are enabled globally in the address configuration: radius enable radius server create <ipv4|ipv6>...
Note: Access-level’s are defined in the file dictionary.passport, which should be present under the directory “dictionary” Sample RADIUS configuration in the clients.conf file The following shows a sample IPv4 client configuration in the clients.conf file: client 192.168.152.125 Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting DHCP Snooping { secret = secret shortname = localhost } The following shows a sample IPv6 client configuration in the clients.conf file: client 3001::4/64 { secret = secret shortname = localhost } Sample RADIUS configuration in the radiusd.conf The following shows a sample IPv4 configuration in the radiusd.conf file: listen { ipaddr = * #ipv6addr = ::...
DHCP Snooping configured properly but client not assigned IP If DHCP Snooping is configured properly, but the client is still not assigned an IP address, you can enable a trace for DHCP Snooping events to troubleshoot the source of the problem. Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting DHCP Snooping 1. Enable trace: trace level 116 4 trace screen on (CLI or ACLI) 2. Verify the trace shows the switch received the DHCP Discover request from the client. 3. Verify the trace shows the switch received the DHCP Offer response from the server.
(CLI) show port vlans (ACLI) Enabling trace messages for Dynamic ARP Inspection troubleshooting You can enable Dynamic ARP Inspection trace for troubleshooting purposes using the following command: trace level 117 <1-4> (CLI or ACLI) Troubleshooting July 2013 Comments? infodev@avaya.com...
Troubleshooting IP Source Guard Note that this command only traces for configuration changes and HA synchronization. It does not provide ARP message tracing. Troubleshooting IP Source Guard To troubleshoot specific IP Source Guard issues, use the following procedure. 1. If IP Source Guard cannot be enabled on the port, verify that DHCP Snooping is enabled globally, on the port, and on all VLANs.
4. On the TACACS+ server, check whether the Avaya Ethernet Routing Switch 8800/8600 IP address is configured in the trust list. 5. If you can log on to the Avaya Ethernet Routing Switch 8800/8600, check whether the TACACS+ server configured on the router has the correct IP address using the...
8800/8600, and the TACACS+ server is reachable. 2. Verify whether the SSH client is configured correctly. 3. To verify whether the SSH function is enabled and configured correctly on the Avaya Ethernet Routing Switch 8800/8600, enter the following CLI command:...
8800/8600 and the modem is well connected. 4. If customers use the serial port to connect to the Ethernet Routing Switch 8800/8600, ensure the DTE port is used on the Ethernet Routing Switch 8800/8600 side, and the DCE port is used on the customer PC side...
2. Check whether accounting is enabled on TACACS+ server. Administrator unable to receive trap packets from the Avaya Ethernet Routing Switch 8800/8600 If the administrator is unable to receive trap packets from the Avaya Ethernet Routing Switch 8800/8600, perform the following steps. Procedure steps 1.
Network Access Switch (SNAS), DHCP and DNS servers. • VoIP access, which restricts VoIP traffic to VoIP controllers and authenticated IP phone peers. For more information about ASNA, see Avaya Ethernet Routing Switch 8800/8600 Security, NN46205-601. Avaya Secure Network Access troubleshooting navigation •...
Avaya Secure Network Access troubleshooting • Issue: client not registered by switch on page 465 • Issue: PC client Web page displays Cannot contact Web Server on page 466 Monitoring DHCP requests You can monitor DHCP requests to aid in troubleshooting ASNA.
5. Perform the failover again. Issue: TG page does not open when client is in Red VLAN Use the following information if the TG page does not open when a client is in a Red VLAN. Troubleshooting July 2013 Comments? infodev@avaya.com...
Avaya Secure Network Access troubleshooting Procedure steps 1. Check if the client in the Red VLAN can obtain an IP address correctly. 2. On the client, check if Java is installed correctly. 3. If the error “Unknown IP address” appears, then the edge switch does not know the IP address of the client and cannot give the information to the SNAS server.
The following sections provide information for troubleshooting HTTPS connectivity issues. HTTPS configurable port Beginning with Release 7.1.3, you can change the port assignment for the HTTPS port. Avaya strongly recommends that you discover the ports that UDP and TCP are already using before you select a port for HTTPS.
Page 467
OpenSSL tool and loaded into the DUT/flash/.ssh folder. 2. Verify that HTTPS is using port 443 (default). ERS 8800/8600 7.1 does not support changing the HTTPS port. 3. Verify the reachablility of the box by ping.
Page 468
Upper layer troubleshooting Troubleshooting July 2013 Comments? infodev@avaya.com...
This section described where to download software or documentation. Downloading Avaya Ethernet Routing Switch 8800/8600 software Use the following procedure to download software from the Avaya Web site 1. Go to the Avaya technical support portal: http://www.avaya.com/support 2. To download software you must do one of the following: •...
Software download Downloading Avaya Ethernet Routing Switch 8800/8600 documentation Use the following procedure to download documentation from the Avaya Web site 1. Go to the Avaya technical support portal: www.avaya.com/support 2. From the list on the left side of the Support pane, expand the Documentation tree.
Chapter 30: Data collection required for Technical Support cases Use the following sections to learn about gathering information before you contact Avaya for technical support. Gathering critical information Before contacting Avaya Technical Support, you must gather information that can help the technical support personnel when troubleshooting.
Description show ports error show interface Displays port collision and extended error information. show-all <port-type> error collision show interface <port- type> error verbose show ports stats show interface Displays port statistics. interface main <port-type> statistics Troubleshooting July 2013 Comments? infodev@avaya.com...
Data collection commands CLI command ACLI command Description show port stats stg show interface Displays spanning tree topology changes. <port-type> statistics stg show mlt info show mlt Displays MLT configuration. show port info show interface Displays port state information. state <port-type>...
Flags errors on the physical layer for MLT. CPU spike issues To collect information to troubleshoot a CPU spike issue, use the commands in the following table. Troubleshooting July 2013 Comments? infodev@avaya.com...
Page 475
Using the trace tool inappropriately can cause primary CPU lockup conditions, loss of access to the switch, loss of protocols, and service degradation. While these occurrences are uncommon, when using the trace level tool, minimize this risk by following these Avaya recommendations: •...
Data collection required for Technical Support cases Contacting support For a detailed list of options for contacting Avaya Technical Support, go to the Avaya Website: www.avaya.com. Click the Support tab. Troubleshooting July 2013 Comments? infodev@avaya.com...
Chapter 31: Safety messages This section describes the different precautionary notices used in this document. This section also contains precautionary notices that you must read for safe operation of the Avaya Ethernet Routing Switch 8800/8600. Notices Notice paragraphs alert you about issues that require your attention. The following sections describe the types of notices.
Electrostatic alert: PRECAUCIÓN ESD (Descarga electrostática) El aviso de ESD brinda información acerca de cómo evitar una descarga de electricidad estática y el daño posterior a los productos Avaya. Electrostatic alert: CUIDADO ESD Os avisos do ESD oferecem informações sobre como evitar descarga de eletricidade estática e os conseqüentes danos aos produtos da Avaya.
Page 479
Notices Los avisos de Precaución brindan información acerca de cómo evitar posibles interrupciones del servicio o el daño a los productos Avaya. Caution: CUIDADO Os avisos de cuidado oferecem informações sobre como evitar possíveis interrupções do serviço ou danos aos produtos da Avaya.
Page 480
Safety messages Troubleshooting July 2013 Comments? infodev@avaya.com...
Appendix A: Traps reference The Avaya Ethernet Routing Switch 8800/8600 generates alarms, traps, and logs. For information about Traps, see the Avaya Ethernet Routing Switch 8800/8600 Traps Reference (NN46205–706). Troubleshooting July 2013...
Page 482
Traps reference Troubleshooting July 2013 Comments? infodev@avaya.com...
Resolution an IP address to an Ethernet media access control (MAC) address. Protocol (ARP) Avaya command A textual user interface. When you use ACLI, you respond to a prompt line interface by typing a command. After you enter the command, you receive a (ACLI) system response.
Page 484
Administratively monitors network performance through agents and Management management stations. Protocol (SNMP) Split MultiLink An Avaya extension to IEEE 802.1AX (link aggregation), provides nodal Trunking (SMLT) and link failure protection and flexible bandwidth scaling to improve on the level of Layer 2 resiliency. Trivial File...