Page 1 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router Cisco_IOS_XR_Software Release_3.9 October, 2010 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 Text Part Number: OL-21483-02...
Page 2 OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at www.cisco.com/go/trademarks.
Page 3: Table Of Contents
Local and Global Configurations 1-35 Collecting Configuration Information 1-37 Verifying the Running Configuration 1-38 Using the show configuration failed Command 1-47 Startup Failed Configuration 1-47 Commit Configuration Failed 1-48 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router OL-21483-02...
Page 4 Troubleshooting Transient Traffic Drop in Forwarding 3-88 Troubleshooting Control Plane Information 3-90 Troubleshooting the Interface Manager 3-95 Interface Manager Control Process 3-96 Troubleshooting the Trace Logs for the Interface Manager 3-100 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router OL-21483-02...
Page 5 Troubleshooting the Multishelf System Router Topology 6-153 Examples 6-156 Troubleshooting the Catalyst Switch 6-158 Restrictions 6-158 Troubleshooting the CRS-1, 4-slot, 8-slot, or 16-slot System Router Topology 6-159 Examples 6-162 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router OL-21483-02...
Page 6 Using show process and top processes Commands 8-186 Troubleshooting a Process Block 8-188 Troubleshooting a Process Crash on Line Cards 8-192 Troubleshooting a Memory Leak 8-193 Troubleshooting a Hardware Failure 8-194 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router OL-21483-02...
Page 7 Validating and Troubleshooting ROM Monitor Software Installation 10-209 Verifying and Troubleshooting the ROM Monitor Version 10-210 Troubleshooting Upgrading and Downgrading ROM Monitor Software on Cisco CRS-1 Routers 10-213 Troubleshooting Upgrading and Downgrading Cisco IOS XR Software Packages 10-214 Troubleshooting the Statistics Infrastructure...
Page 8 Troubleshooting Layer 2 Load Balancing Example 13-246 Verifying the Bundle Status 13-246 Checking the IGP Routing Table and CEF Database 13-247 Verifying Bundle Load Balancing 13-247 N D E X Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router OL-21483-02...
Page 9 Preface This guide describes how to troubleshooting a router using the Cisco IOS XR software. This preface contains the following sections: Changes to This Document, page vii • • Obtaining Documentation and Submitting a Service Request, page viii Changes to This Document Table 1 lists the technical changes made to this document since it was first issued.
Page 10: Obtaining Documentation And Submitting A Service Request
Obtaining Documentation and Submitting a Service Request For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at: http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html...
Page 11: Troubleshooting Techniques And Approaches
Capture and save the output of all commands. This information is useful when contacting • Cisco Technical Support. For information on contacting Cisco Technical Support, see the “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface.
Page 12: Documenting The Network
Verifying and Troubleshooting CLI Access Ensure that the system has been booted. If the system has not booted, see Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for information on booting a router running Cisco IOS XR software.
Page 13: Chapter 1 General Troubleshooting
If you have the correct username and password, but are locked out of the router, you may need to perform password recovery to access the system again. See Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for password recovery procedures.
Page 14: Cli Access Through The Management Ethernet Interface
The type of client software you use depends on the server process you want to use. See Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for information on the client and server services supported by the Cisco IOS XR software.
Page 15 If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router OL-21483-02...
Page 16 172.29.52.72 0011.93ef.e8fe Interface ARPA MgmtEth0/RP1/CPU0/0 172.29.52.75 00:11:14 0011.93ef.e8e2 Dynamic ARPA MgmtEth0/RP1/CPU0/0 172.29.52.77 00:11:14 0011.93ef.e8e2 Dynamic ARPA MgmtEth0/RP1/CPU0/0 172.29.52.76 02:12:54 0011.93ef.e8e6 Dynamic ARPA MgmtEth0/RP1/CPU0/0 172.29.52.78 01:42:02 0014.a8de.6700 Dynamic ARPA MgmtEth0/RP1/CPU0/0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router OL-21483-02...
Page 17: Basic Cisco Ios Xr Verification And Troubleshooting Tools
Each man page contains the command name, syntax, command mode, usage, examples, and related commands. The Cisco IOS XR Documentation Package - Man pages for Cisco IOS XR CLI commands must be Note loaded in order to run the man command.
Page 18 To use this command, you must be in a user group associated with a task group that includes the proper task IDs. For detailed information about user groups and task IDs, see the Configuring AAA Services on Cisco IOS XR Software module of the Cisco IOS XR System Security Configuration Guide.
Page 19: Describe Command
: Built on Tue Jan 31 10:56:38 UTC 2006 Source : By edde-bld1 in /files/3.3.0/hfr/workspace fo8 Card(s): RP, DRP, DRPSC Component: ipv4-bgp V[fwd-33/53] IPv4 Border Gateway Protocol (BGP) User needs ALL of the following taskids: Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router OL-21483-02...
Page 20: Show Platform Command
0.20% top 65555 10 Rcv 0:00:00 0.10% devb-ata 55 Rcv 0:22:50 0.09% eth_server 10 Rcv 0:05:13 0.09% eth_server 10 Rcv 0:15:38 0.07% shelfmgr Press ‘q’ to exit the command. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-10 OL-21483-02...
Page 21: Show Context Command
#12 0x4820d140 Registers info 00000000 481ff7b0 4824a55c 00000000 0000f054 00000001 00000006 00000000 00000000 fc220000 481fffc0 00000000 4823be90 4824a4a0 48230000 00000000 00000048 00000001 00000019 48256520 00000000 00000000 00000003 00000045 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-11 OL-21483-02...
Page 22 The command output shows a crashed on a node. The process is pkg/bin/tcam_mgr. Collect the following information and sent it to Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface.
Page 23: Show Users Command
161.44.1925 vty1 User-B telnet 00:00:03 161.44.1929 Verifying the System To verify the general status and state of a router using Cisco IOS XR software, perform the following procedure. SUMMARY STEPS admin show platform [node-id] show version show running-config show logging...
Page 24 Analyze the logged events and their order of happening. RP/0/RP0/CPU0:router(admin)# show logging Check for anything out of the ordinary such as errors, tracebacks, or crashes. Also check for any Severity 1 or Severity 2 errors. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-14 OL-21483-02...
Page 25 Step 13 Displays the contents of the currently running show running-config configuration. Verify that the contents of the current running configuration Example: are what is expected. RP/0/RP0/CPU0:router# show running-config Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-15 OL-21483-02...
Page 26 The output from the show version command indicates the version of software being run on the nodes and from which location (disk or network). Check that the expected software version and images are installed. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-16 OL-21483-02...
Page 27 Built on Tue Jan 31 10:02:02 UTC 2006 By edde-bld1 in /files/3.3.0/workspace for c2.95.3-p8 The example output shows that the Cisco IOS XR software version is 3.3.0 and that the installed pie versions are also 3.3.0. The output from the show running-config command displays hardware module power status, secure domain router (SDR) configuration, fabric configuration, and rack numbers and serial numbers.
Page 28 The syslog will provide details on any logged problems. RP/0/RP0/CPU0:router(admin)# show environment Temperature Information --------------------------------------------- Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-18 OL-21483-02...
Page 29 -10,75 -15,85 spa0 Inlet0 0,50 -10,60 -15,70 spa0 Hotspot0 0,64 -10,74 -15,84 spa0 Hotspot1 0,69 -10,79 -15,88 spa5 Inlet0 0,58 -10,71 -15,81 spa5 Hotspot0 0,60 -10,73 -15,83 0/RP0/* Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-19 OL-21483-02...
Page 30 1254 1240 1508 1297 1807 2545 3328 5000 1254 1226 1807 2535 3336 4992 3268 4940 0/SM0/* host 1762 2496 3302 1804 2496 0/SM1/* host 1762 2483 3285 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-20 OL-21483-02...
Page 31 The output from the show context command displays core dump context information. See the “show context Command” section on page 1-11 for more information on the show context command output. RP/0/RP0/CPU0:router# show context node: node0_1_CPU0 ------------------------------------------------------------------ Crashed pid = 61524 (pkg/bin/tcam_mgr) Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-21 OL-21483-02...
Page 32 Package: hfr-fwdg, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor kspace for c2.95.3-p8 Package: hfr-admin, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo rkspace for c2.95.3-p8 Package: hfr-base, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-22 OL-21483-02...
Page 33 CRS-8_X1 line console exec-timeout 120 0 session-timeout 120 line default exec-timeout 120 0 session-timeout 120 telnet vrf default ipv4 server max-servers no-limit domain ipv4 host x1 172.16.52.72 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-23 OL-21483-02...
Page 34 GigabitEthernet0/6/5/1 description Connected to CRS-8_X2 GE 0/6/5/1 ipv4 address 10.55.48.0 255.255.255.0 controller SONET0/1/0/0 delay trigger line 100 clock source internal controller SONET0/1/0/1 clock source internal controller SONET0/6/0/0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-24 OL-21483-02...
Page 35 Unable to get current LC status info - Verified QNET Status : [FAIL] - Verifying GSP Fabric Status - Verified GSP Fabric Status : [OK] - Verifying GSP Ethernet Status Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-25 OL-21483-02...
Page 36 GigabitEthernet0/1/5/2 unassigned Shutdown Down GigabitEthernet0/1/5/3 unassigned Shutdown Down GigabitEthernet0/1/5/4 unassigned Shutdown Down GigabitEthernet0/1/5/5 unassigned Shutdown Down GigabitEthernet0/1/5/6 unassigned Shutdown Down GigabitEthernet0/1/5/7 unassigned Shutdown Down POS0/6/0/0 10.50.8.1 POS0/6/0/1 10.50.36.1 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-26 OL-21483-02...
Page 37: Validating And Troubleshooting Cisco Ios Xr Software Package Installation
Validating and Troubleshooting Cisco IOS XR Software Package Installation The Cisco IOS XR software is divided into software packages allowing you to select which features run on your router. Each package contains the components to perform a specific set of router functions, such as routing, security, or Modular Services card support.
Page 38 Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for information on installing and upgrading Cisco IOS XR software packages. The following example shows that the Cisco IOS XR software and active packages are version 3.3.0. RP/0/RP0/CPU0:router# show version Cisco IOS XR Software, Version 3.3.0 Copyright (c) 2006 by cisco Systems, Inc.
Page 39 V 3.3.0, Cisco Systems, at disk0:hfr-base-3.3.0 Built on Tue Jan 31 09:48:20 UTC 2006 By edde-bld1 in /files/3.3.84.2I/hfr/workspace for c2.95.3-p8 The following example shows that the Cisco IOS XR software and active packages are version 3.3.0. RP/0/RP0/CPU0:router# show install Node 0/1/SP [SP] [SDR: Owner] Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm...
Page 40: Validating The Installation
See the Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for details on installing, activating, and upgrading software packages.
Page 41 [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful. Info: Node 0/SM2/SP Info: [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful. Info: [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful. Info: [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful. Info: [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful. Info: Node 0/SM3/SP Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-31 OL-21483-02...
Page 42: Show Install Active Command
If the expected active software packages are not displayed, install the packages (if required) and activate the packages. See Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for information on installing and activating Cisco IOS XR software packages. The following example output shows the...
Page 43: Show Install Committed Command
The active packages for each node are on disk0, and for all nodes, the composite package hfr-os-mbi-3.3.0 is active. Additional packages shown are optional packages that have been activated after the initial loading of the Cisco IOS XR Unicast Routing Core Bundle. show install committed Command Use the show install committed command to display committed software packages.
Page 44 The show install committed command is useful to ensure software is installed and committed after a router reload. If the expected software is not installed and committed, see Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for information on installing and committing Cisco IOS XR software packages.
Page 45: Validating And Troubleshooting Cisco Ios Xr Software Configuration
Validating and Troubleshooting Cisco IOS XR Software Configuration Validating the Cisco IOS XR software configuration includes collecting configuration information on the router to determine configuration changes and verifying the current running configuration. When a configuration fails during a commit, the failed configuration can be viewed to help determine why the configuration was not committed.
Page 46 656 wrapping entries (4096 possible, 3440 filtered, 6460 total) Aug 29 06:14:38.443 38446 apply reply '--' Aug 29 06:14:38.442 1139 Apply/abort called 'cfg/if/act/POS0_5_0_0/keepalive' Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-36 OL-21483-02...
Page 47: Collecting Configuration Information
172.29.52.78 no domain ipv4 host pe1 no domain ipv4 host pe2 domain ipv4 host pe6 172.29.52.128 domain ipv4 host pe7 172.29.52.182 interface GigabitEthernet0/1/5/1 no negotiation Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-37 OL-21483-02...
Page 48: Verifying The Running Configuration
| include path show sysdb trace verification location node-id show cfgmgr trace show config commit history show config commit changes show config failed startup cfs check Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-38 OL-21483-02...
Page 49 Displays details of recent verification sysDB transactions and changes on local plane configurations. Verify that changes to the SysDB were verified and Example: accepted. RP/0/RP0/CPU0:router# show sysdb trace verification location 0/3/CPU0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-39 OL-21483-02...
Page 50 192.0.2.73 domain ipv4 host ce7 192.0.2.78 domain ipv4 host pe6 192.0.2.128 domain ipv4 host pe7 192.0.2.182 domain-lookup vty-pool default 0 25 ipv4 virtual address 192.0.2.72 255.255.255.0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-40 OL-21483-02...
Page 51 Source : By edde-bld1 in /vws/3.3.0./file for c2.95.3-p8 Card(s): RP, DRP, DRPSC, OC3-POS-4, OC12-POS, GE-3, OC12-POS-4, OC48-POS , E3-OC48-POS, E3-OC12-POS-4, E3-OC3-POS-16, E3-OC3-POS-8, E3-OC3-POS-4, E3-OC48 -CH, E3-OC12-CH-4, E3-OC3-CH-16, E3-GE-4, E3-OC3-ATM-4, E3-OC12-ATM-4, E5-CEC, L C, SP Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-41 OL-21483-02...
Page 52 '--' 7 20:10:35.471 4912 Verify called 'cfg/if/act/GigabitEthernet0_3_4_0.1/a/sub_vlan/0x2/________/Gigab itEthernet0_3_4_0/________' 7 20:10:35.471 8782 apply reply '--' 7 20:10:35.471 8782 apply reply '--' 7 20:10:35.471 8782 apply reply Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-42 OL-21483-02...
Page 53 '0x2' 5 14:47:57.471 cfgmgr/common 0/RP0/CPU0 t4 Attempting to apply ascii admin startup config from file '/qsm/cfsroot/admin/admin.cfg'. 5 14:48:09.156 cfgmgr/common 0/RP0/CPU0 t4 Clean all admin files since adm Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-43 OL-21483-02...
Page 54 (Attempting to apply binary LR s tartup config.). Several invalid argument errors were returned when attempting to restore the startup configuration from the binary checkpoints. If there are invalid argument errors, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface.
Page 55 Connected to P2_CRS-8 Bundle-POS 24 ipv4 address 10.12.24.1 255.255.255.0 bundle minimum-active links 1 bundle minimum-active bandwidth 2488320 no interface Loopback0 interface Loopback0 ipv4 address 10.1.1.1 255.255.255.255 interface MgmtEth0/RP0/CPU0/0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-45 OL-21483-02...
Page 56 POS0/1/0/1 mpls ldp router-id Loopback0 neighbor graceful-restart interface POS0/1/0/1 interface GigabitEthernet0/1/5/2 mpls oam router igmp version 1 ssh server xml agent tty xml agent corba http server Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-46 OL-21483-02...
Page 57: Using The Show Configuration Failed Command
The load configuration failed startup command can be used in configuration mode to load the failed startup configuration into the target configuration session, then the configuration can be modified and committed. See Cisco IOS XR Getting Started Guide for information on committing a configuration.
Page 58: Commit Configuration Failed
The following example shows an invalid task ID configuration that fails to commit. The show configuration failed command provides information on why the configuration failed. RP/0/RP0/CPU0:router(config)# taskgroup isis Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-48 OL-21483-02...
Page 59: Asic Errors
Address—Address that encountered the SBE – Syndrome—Syndrome if available – Multiple Bit Errors—Uncorrectable multiple bit error in memory. • Reported to PM as Major and ALDEMS for each occurrence. Error data: Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-49 OL-21483-02...
Page 60 : 0x105051d Thresh/period(s): 0/0 Alarm state: OFF Error count : 3703662240 Last clearing : Thu Mar 29 11:59:32 2007 Last N errors : 50 -------------------------------------------------------------- First N errors. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-50 OL-21483-02...
Page 61 : 0x105081d Thresh/period(s): 0/0 Alarm state: OFF Error count : 332349486 Last clearing : Mon Apr 16 10:44:39 2007 Last N errors : 21 -------------------------------------------------------------- First N errors. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-51 OL-21483-02...
Page 62 Fri Oct 14 13:52:22 2005 ASIC-ERROR The following example shows how to display node-specific ASIC errors. RP/0/RP0/CPU0:router# dir harddisk:/ASIC-ERROR Directory of harddisk:/ASIC-ERROR 1881 drwx 4096 Thu Jun 16 08:32:14 2005 node0_3_CPU0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-52 OL-21483-02...
Page 63 Parity error count CRC error count Generic error count : 0 Reset error count -------------------- ************************************************************ Pse ASIC Error Summary ************************************************************ Instance Number of nodes SBE error count Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-53 OL-21483-02...
Page 64 Reset error count -------------------- Instance Number of nodes SBE error count MBE error count Parity error count CRC error count Generic error count : 0 Reset error count -------------------- Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-54 OL-21483-02...
Page 65 Instance Number --- Instance Name SHARQ RP/0/RP0/CPU0:router# show controllers asic metro instance help-instance location 0/0/CPU0 Wed May 2 13:47:51.817 EST EDT Total 2 Instance Instance Number --- Instance Name IngressPSE Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-55 OL-21483-02...
Page 66: Trace Commands
EgressPSE Trace Commands Trace commands provide an ‘always on’ debug feature. Many major functions in Cisco IOS XR software have “trace” functionality to show the last actions it conducted allowing you to analyze function events. Use the show trace commands to display the trace data for a specific feature or process. Use the ? in the CLI to determine if a command has the trace keyword.
Page 67: Packets
To turn on packet capture and view capture packet output, perform the following procedure. SUMMARY STEPS configure interface type instance capture software packets commit show captured packets {ingress | egress} [interface type instance] [hexdump] [last number] [single-line] location node-id Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-57 OL-21483-02...
Page 68 {ingress | egress} [hexdump] [interface type instance ] [last punted in the software. number ] [single-line] location node-id Example: RP/0/RP0/CPU0:router# show captured packets ingress location cpu 0/1/cpu0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-58 OL-21483-02...
Page 69 [IPV4: source 172.19.90.6, dest 172.19.90.2 ihl 5, ver 4, tos 0 id 0, len 56, prot 61, ttl 64, sum 6e5a, offset 0] 00000000 00000000 00000000 00000000 00100fff 002403b2 0d3de784 3bed359f a39c0000 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-59 OL-21483-02...
Page 70: Logging Archive For Harddisk
In order to view entire MIB table, the community string needs to have ‘systemowner’. This allows the user to view admin plane objects as well as LR plane. To locate and download MIBs, use the Cisco MIB Locator found at the following URL and choose a platform under the Cisco Access Products menu: http://cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml.
Page 71: Error File Locations
Use the more command to view the contents of a file. harddisk: The following folders are located on the harddisk: • shutdown—Contains shutdown scripts collected for nodes var/log—Contains any syslog archive data to harddisk if enabled • Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-61 OL-21483-02...
Page 72: Default Disk Location
Wed Nov 23 17:32:34 2005 hfr-mpls-p.pie-3.2.2 1024655360 bytes total (859799552 bytes free) Sysmgr Collection Scripts The sysmgr is responsible for starting, monitoring, stopping, and if necessary, restarting most processes on the system. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-62 OL-21483-02...
Page 73: Wdsysmon Collection Scripts
Directory of disk1:/wdsysmon_debug 196736 -rwx 37151 Wed Dec 21 17:47:40 2005 debug_evm.364641 196832 -rwx 39422 Wed Dec 21 21:42:10 2005 debug_evm.364640 196928 -rwx 39577 Wed Dec 21 21:57:14 2005 debug_evm.307296 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-63 OL-21483-02...
Page 74 0xfc139000 0x00007adc 0xfc133270 0x00000148 /pkg/lib/libinfra.dll 0xfc141000 0x000341a4 0xfc1333b8 0x00000bbc /pkg/lib/cerrno/libinfra_error.dll 0xfc1121dc 0x00000cd8 0xfc176000 0x000000a8 /pkg/lib/libios.dll 0xfc177000 0x0002dc38 0xfc1a5000 0x00002000 /pkg/lib/cerrno/libevent_manager_error.dll 0xfc1a7000 0x00000e88 0xfc133f74 0x00 000088 /pkg/lib/libc.dll 0xfc1a8000 0x00079d70 0xfc222000 0x00002000 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-64 OL-21483-02...
Page 75 0.000 0.00 0.00 shelfmgr 77896 1.054 0.175 0.063 0.000 0.00 0.00 gsp 32792 0.484 0.100 0.012 0.000 0.00 0.00 bcm_process 32794 0.097 0.020 0.011 0.000 0.00 0.00 stp_process Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-65 OL-21483-02...
Page 76: Shutdown Collection Scripts
Monitoring allows you to view interface, controller fabric, or controller SONET counters, and auto-updating statistics on processes and threads in real-time. The following commands are used for monitoring: monitor interface Command, page 1-67 • monitor controller Command, page 1-68 • Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-66 OL-21483-02...
Page 77: Monitor Interface Command
Use the monitor interface command to monitor interface counters. The following example shows the output of the monitor interface command. The Cisco IOS XR Manageability Package is required to use the monitor interface command. Note The following example shows the output of the monitor interface command with a specified interface.
Page 78: Monitor Controller Command
Monitor Time: 00:00:08 SysUptime: 19:02:27 CRS-8_X1 Monitor Time: 00:00:10 SysUptime: 19:02:29 CRS-8_X1 Monitor Time: 00:00:12 SysUptime: 19:02:31 CRS-8_X1 Monitor Time: 00:00:14 SysUptime: 19:02:33 CRS-8_X1 Monitor Time: 00:00:16 SysUptime: 19:02:35 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-68 OL-21483-02...
Page 79: Monitor Processes Command
0.01% lpts_pa 756K 0:00:00 0.01% bcdl_agent 65554 0:02:33 0.01% devb-ata 828K 0:00:01 0.01% raw_ip Chapter 8, “Process Monitoring and Troubleshooting,” for more information on processes and process monitoring. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-69 OL-21483-02...
Page 80: Monitor Threads Command
Gathering Information for Technical Support The following commands are useful for gathering information about your system before you contact Cisco Technical Support. We recommend that you run these commands if time permits, especially if your system has experienced a major problem.
Page 81 • admin show redundancy • admin show controllers fabric plane all detail • Before contacting Cisco Technical Support, review the information provided at the following URL: http://www.cisco.com/web/services/ts/access/index.html. For more information on gathering system information, see Chapter 7, “Collecting System Information”.
Page 82 Chapter 1 General Troubleshooting Gathering Information for Technical Support Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 1-72 OL-21483-02...
Page 83: Chapter 2 Troubleshooting Booting
Boot the route processor (RP) of the designated shelf controller (DSC). For Cisco CRS-1 Multishelf Systems, when the RP of the DSC boots up, if the RP or shelf controller (SC) on other chassis are configured to autoboot (for example, they are in a loop sending minimum boot image [MBI] requests), they boot automatically.
Page 84: Verifying Successful Bootup
The default configuration selects the route processor (RP) in the lowest numbered slot. For the Cisco CRS-1 Multishelf System, the RP in the lowest numbered slot in the LCC configured as Rack 0 is selected as the primary RP and the designated shelf controller (DSC).
Page 85: Verifying And Troubleshooting Route Processor And Shelf Controller Cards
Requests, page 2-79 Troubleshooting RP and SC Cards Not Booting To troubleshoot the RP and SC cards in a router running Cisco IOS XR software, perform the following procedure. Troubleshooting procedures for the following cards is provided in this section:...
Page 86 Chapter 2 Troubleshooting Booting Verifying and Troubleshooting Route Processor and Shelf Controller Cards • Cisco CRS-1 4-Slot Line Card Chassis/Cisco CRS-1 8-Slot Line Card Chassis/Cisco CRS-1 16-Slot Line Card Chassis – Primary (DSC) and standby route processors (RPs) Cisco CRS-1 Multishelf System •...
Page 87 Step 4 Swap RP (or SC) slots. Swap the RP (or SC) card in the chassis: RP slots—RP0 and RP1 • SC slots—SC0 and SC1 • Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 2-77 OL-21483-02...
Page 88: Troubleshooting Rp And Sc Cards Resetting While Booting
Request” section on page viii in the Preface. Troubleshooting RP and SC Cards Resetting While Booting To troubleshoot resetting RP and SC cards on Cisco CRS-1 systems, perform the following procedures. SUMMARY STEPS Place both RPs in ROMMON mode. Verify that all three LEDs on power modules are green.
Page 89: Troubleshooting Blocked Fcc Shelf Controller Or Lcc Route Process Minimum Boot Image Requests
Step 1 Place both RPs in ROMMON mode. See Router Recovery with ROM Monitor mode. See the Cisco IOS XR ROM Monitor Guide for the Cisco CRS-1 Router for information on entering ROMMON mode. Step 2 Displays the environment variable settings for the card.
Page 90 Router# show spanning-tree connected. If the port is not listed, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface.
Page 91: Troubleshooting Ipv4 Cef Information
C H A P T E R Troubleshooting Forwarding Cisco Express Forwarding (CEF) is the mechanism that enables packet forwarding. CEF information is examined when data forwarding is not occurring as expected. Troubleshooting CEF involves comparing the Routing Information Base (RIB) information to the software Forwarding Information Base (FIB), verifying that the hardware is programmed correctly, verifying that the adjacencies are built correctly, verifying the control plane is built correctly, and gathering any necessary trace information.
Page 92: C H A P T E R 3 Troubleshooting Forwarding
ROUTES Netio PPE-1 PPE-n To troubleshoot IPv4 CEF information on Cisco IOS XR software, perform the following procedure. SUMMARY STEPS show route ipv4 prefix mask show cef ipv4 prefix mask detail show cef ipv4 prefix mask detail location node-id (on ingress line card)
Page 93 For example, if it is a remote adjacency, then the adjacency information exists on another node. • Check that the expected hash (load balance) and egress interfaces are listed. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-83 OL-21483-02...
Page 94 255.255.255.0 hardware detail egress location Check that the hardware and software representations • 0/13/cpu0 in hex format match. For example: SW: 0x0c000000 00000020 00000000 00000000 HW: 0x0c000000 00000020 00000000 00000000 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-84 OL-21483-02...
Page 95: Troubleshooting Adjacency Information
“Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface. Troubleshooting Adjacency Information To troubleshoot adjacency information on Cisco IOS XR software, perform the following procedure. SUMMARY STEPS show arp location node-id show arp traffic location node-id...
Page 96 Contact Cisco Technical Support if the problem is not resolved DETAILED STEPS Command or Action Purpose...
Page 97 If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-87 OL-21483-02...
Page 98: Troubleshooting Transient Traffic Drop In Forwarding
1500, flags 0 0 0 2894 packets, 156876 bytes 0xffffffff Troubleshooting Transient Traffic Drop in Forwarding To troubleshoot a momentary drop in IPV4 packet forwarding on Cisco CRS-1 router, perform the following procedure. SUMMARY STEPS show interfaces interface name show controller <ingressq/egressq>...
Page 99 [ingress|egress] location node_id show route <prefix> det show cef <prefix> det location location-id show cef <prefix> hardware ingress detail location location-id Contact Cisco Technical Support if the problem is not resolved DETAILED STEPS Command or Action Purpose...
Page 100: Troubleshooting Control Plane Information
6, then it indicates that the route has been programmed as drop in hardware. Troubleshooting Control Plane Information To troubleshoot control plane information on Cisco IOS XR software, perform the following procedure. SUMMARY STEPS show netio idb interface-type interface-instance location node-id...
Page 101 • correct. Compare the output to the configuration of the interface and expected features. Use this command for the ingress and egress interfaces for the local line card. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-91 OL-21483-02...
Page 102 0x01180020 interface global index: physical media type: dchain ptr: <0x482ae8e0> echain ptr: <0x482d791c> fchain ptr: <0x482d79b8> driver cookie: <0x4824ad58> driver func: <0x4824ad44> number of subinterfaces: 4096 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-92 OL-21483-02...
Page 103 <0xfcedda4c, 0x482dbee4> < 0> ether_sock <98> <0xfd7b1874, 0x48302824> < 0> Fixup: l2_adj_rewrite <86> <0xfcec745c, 0x00000000> < 0> queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0> txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0> Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-93 OL-21483-02...
Page 104 MINIMUM MASK DESTINATION: 0 / 0 MINIMUM MASK SOURCE: 0 / 0 BYTES OF SNIFF PACKET: 0 SUPPRESS PUNT ACL: 0 MPLS PROPAGATE TTL FLAG: 1 PARITY: 0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-94 OL-21483-02...
Page 105: Troubleshooting The Interface Manager
• Interface Manager Control Process, page 3-96 Troubleshooting the Trace Logs for the Interface Manager, page 3-100 • • Troubleshooting the Client for the Interface Manager, page 3-101 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-95 OL-21483-02...
Page 106: Interface Manager Control Process
Increases in proto max: ------------------------------------------------------------------------- Client requests Request Type Requests: Operations: Max Ops: ----------------------------------------------------------------------- SPECIFY_CONTROL NOTIFY_GET INTERFACE_REPLICATE INTERFACE_CREATE INTF_CREATE_WITH_BCAPS INTERFACE_DELETE DELETE_CONFIRM CAPS_BASE_DEFINE RESOURCE_ALLOC_DONE CAPS_ADD CAPS_REMOVE ADOPT_CONFIRM PROTO_LOOKUP PROTO_REVERSE_LOOKUP CAPS_LOOKUP Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-96 OL-21483-02...
Page 107 ------------ ------------ ------------ IF_CREATE IF_READY IF_FIND 1578 IF_DELETE DEL_CONFIRM IF_UPDATE_PRIMARY IF_FLAGS CAPS_ADD CAPS_REMOVE CAPS_BASE_DEFINE ID_LOOKUP STATE_UPDATE MTU_UPDATE REG_WALK REG_CREATE IF_NAME STATE_QUERY MTU_QUERY PIC_UPDATE RSRC_ALLOC REPL_INTF_OPER REPL_INTF_OPER_ACK INTF_CONFIG_PARAM IF_REPLICATE ADOPT_CONFIRM CFG_NEW_BC Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-97 OL-21483-02...
Page 108 Initial Sz: Chunk Sz: #Chunks: Max Chunks: ----------------------------------------------------------------------- Physical interface handles Virtual interface handles Other physical handles Other virtual handles ----------------------------------------------------------------------- Initial Memory: 0 bytes Current Memory: 40960 bytes Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-98 OL-21483-02...
Page 109 DP Restart ----------------------------------------------------------------------- Total ----------------------------------------------------------------------- Client callback information #Active #Dead Total Memory (bytes) ----------------------------------------------------------------------- 19568 ----------------------------------------------------------------------- Requested Client Connections Requested Disconnections Unrequested Disconnections IM server (re)start count ----------------------------------------------------------------------- Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-99 OL-21483-02...
Page 110: Troubleshooting The Trace Logs For The Interface Manager
The debug im callbacks command is used to display when notifications are arriving at the clients from IM or Interface Manager Distributor (IMD). In addition, the command checks to see the number and types of messages that are in the set of notifications. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-100 OL-21483-02...
Page 111: Troubleshooting The Client For The Interface Manager
(SDR) unique identifiers to control the nodes. The interface handle is displayed in hexadecimal and contains platform-specific mapping for the node ID, virtual bit, and interface instance number for the node ID. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-101 OL-21483-02...
Page 112 If the rules for a specific capsulation, protocol, or interface are missing from the IM on a given node, you cannot create a control node for the type. If the rules are on the router, a process restart of ifmgr can resynchronize the rules. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-102 OL-21483-02...
Page 113: Troubleshooting The Control Chain And Interface Information
[pic:0x0, intf_flags:0x37] The following sample output from the show im children command with the ifhandle keyword shows when the subinterfaces or other control parent and child summaries are required: Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-103 OL-21483-02...
Page 114 Unknown Mg0/0/CPU0/0 ARPA 1514 100000 PO0/2/0/0 down HDLC 4474 155520 PO0/2/0/1 admin-down admin-down HDLC 4474 155520 PO0/2/0/2 admin-down admin-down HDLC 4474 155520 PO0/2/0/3 admin-down admin-down HDLC 4474 155520 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-104 OL-21483-02...
Page 115: Troubleshooting The Registrations For The Interface Manager
GigabitEthernet0_7_0_1 ether The registered client receives the changes for the IM notifications. If a client appears to be missing notifications, see the possible reasons in Table 3-1. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-105 OL-21483-02...
Page 116: Troubleshooting The Interface Manager Distributor
Discovers information about each control node in each interface manager service on each card • through the GSP. Contains the interface name, handle, state, MTU, and capsulation state. • Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-106 OL-21483-02...
Page 117: Troubleshooting The Trace Logs For The Interface Manager Distributor
State: 0x02 LC_req: LC_res: 12 ID: 0x00 Sync: Gather: Resync: Check: Clients Name Filter Ref Interface Ref ----------------------- ---------- ---------- ------------- ipv4_arm 0x00000064 1 ipv6_arm 0x00000065 1 0x30000009 4 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-107 OL-21483-02...
Page 118: Troubleshooting The Interface Information For The Interface Manager Distributor
0x00000000 protocol capsulation state --------------- -------------------- --------------- -------- 0 (Unknown) 17 (null) 3 (up) 1500 MgmtEth0_0_CPU0_0 (0x01000100) ----------------------- flags: 0x0000002f type: 8 (IFT_ETHERNET) encap: 30 (ether) Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-108 OL-21483-02...
Page 119: Troubleshooting The Global Registrations For The Interface Manager Distributor
The following sample output from the show imds registrations command with the all keyword shows all the IMD registrations: RP/0/RP0/CPU0:router# show imds registrations all location 0/0/CPU0 IMD REGISTRATIONS Reg: C - Create, E - Encap, M - MTU, S - State Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-109 OL-21483-02...
Page 120: Troubleshooting The Rules For The Interface Manager Distributor
The show imds rules command is used to display information for the IMD rules, which is a summary of basic information cached in the IMD. The show commands for the IM are used to display detailed data information on the rules. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 3-110 OL-21483-02...
Page 121: Crs-1 Switch Fabric Overview
(D)RP, the S1, S2, and S3 ASICS on the switch fabric planes and one of more fabric queue ASICs on the destination MSC or (D)RP. For Cisco CRS-1 Multishelf Systems, connect to the route processor (RP) for the owner logical router Note (LR) when troubleshooting the fabric plane.
Page 122 S2 fabric cards. Each S2 fabric card contains six S2 ASICs. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 4-112...
Page 123: C H A P T E R 4 Troubleshooting Router Switch Fabric
Cells with no data are termed as idle cells. Cell forwarding on the S3 ASIC on both the Cisco CRS-1 4-slot, 8-slot, and 16-slot systems involves examination of the cell header. Using the cell header, the S3 is able to determine which fabric queue ASIC the cells should be sent to.
Page 124 - plane multicast down— not all fabric queue ASICs present on the plane are reachable; multicast forwarding is not operational on this plane. Unicast traffic is unaffected. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 4-114 OL-21483-02...
Page 125: Using The Online Diagnostics Tools
| include UP.*DOWN (for MSC fabric interface ASICs) show controllers fabric sfe fabricq all | include UP.*DOWN (for MSC fabric interface ASICs) Contact Cisco Technical Support if the problem is not resolved Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router...
Page 126 If the CE, UCE or PE values are incrementing, contact Cisco Technical Support. For contact information for Cisco Technical Support, see the “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 4-116 OL-21483-02...
Page 127 LCC the system changes the status of the plane to RP/0/RP0/CPU0:router(admin)# show controllers fabric link health MCAST_DOWN. If connectivity is lost to all LCC the system changes the status of the plane to DOWN. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 4-117 OL-21483-02...
Page 128 - plane multicast down Plane Admin Oper Down Total Down State State Flags Bundles Bundles ------------------------------------------------------ The output above shows all eight fabric planes in the up state. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 4-118 OL-21483-02...
Page 129 - data down f - failed component downstream m - plane multicast down Plane Admin Oper Down Total Down State State Flags Bundles Bundles ------------------------------------------------------ DOWN DOWN Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 4-119 OL-21483-02...
Page 130 Total received multicast data cells: 177001 Total transmitted data cells: 749352358340 Total transmitted unicast data cells: 749351671469 Total transmitted multicast data cells: 686871 Total received correctable errored cells: 0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 4-120 OL-21483-02...
Page 131: Verifying And Troubleshooting Up Fabric Planes
To verify and troubleshoot the up fabric planes, perform the following procedure. For Cisco CRS-1 Multishelf Systems the S1 to S2 links and the S2 to S3 links are inter-chassis links. For Cisco CRS-1 single-shelf systems the links are internal chassis links. The commands used are the same.
Page 132 | exclude \ 0 +0 +0$ config controllers fabric link port {s1rx | s1tx | s2rx | s2tx | s3rx | s3tx} location link-id shutdown Reload the fabric card. Clean and reconnect the fiber-optic connectors (Cisco CRS-1 Multishelf Systems). DETAILED STEPS Command or Action Purpose Step 1 Enters administration executive (admin EXEC) mode.
Page 133: Verifying And Troubleshooting Up Fabric Planes
Submitting a Service Request” section on page viii in the Preface. If you have placed a link out of service on a Cisco CRS-1 single-shelf system, proceed to Step If you have placed a link out of service on a...
Page 134 You must clean the fabric fiber-optic connectors with the specially designed Cisco kit (CRS-FIBER-CLN-KIT=) and Review these reminders and warnings before you must use the procedure in the Cisco CRS-1 Carrier inspecting or handling your fiber-optic connection: Routing System Fiber-Optic Cleaning Guide. This...
Page 135 Cells Cells -------------------------------------------------------------------------------- The following example shows how to place a link out of service. Performing this operation should be conducted in conjunction with Cisco Technical Support engineers. Note RP/0/RP0/CPU0:router(admin-config)# controllers fabric link port s1rx all brief Wed May 2 16:31:45.560 EST EDT...
Page 136: Troubleshooting Down Fabric Planes
An individual link placed in a down state can be sustained without impact to the overall operation or forwarding capacity of the Cisco CRS-1 Multishelf System. At each stage of the fabric, there is a greater bandwidth capacity than can actually be generated by the modular services cards (MSCs). The egress side of the switch fabric has at least double the capacity of the ingress.
Page 137 For Cisco CRS-1 Multishelf Systems only. show running-config Displays the contents of the administrative running configuration. Example: Verify that the plane is set. In the Cisco CRS-1 Multishelf System, RP/0/RP0/CPU0:router(admin)# show the command output indicates which fiber module in the fabric card running-config chassis (FCC) is connected from the plane.
Page 138: Troubleshooting Down Fabric Planes
Step • RP/0/RP0/CPU0:router(admin)# show controllers fabric sfe s1 all If all the ASICs are not in the up state, contact Cisco Technical • Support. For Cisco Technical Support contact information, see “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface.
Page 139 Admin Oper R/S/M/P State State ------------------------------ 0/SM0/SP/0 0/SM0/SP/1 0/SM0/SP/2 0/SM1/SP/0 0/SM1/SP/1 0/SM1/SP/2 0/SM4/SP/0 0/SM4/SP/1 0/SM4/SP/2 0/SM5/SP/0 0/SM5/SP/1 0/SM5/SP/2 1/SM0/SP/0 1/SM0/SP/1 1/SM0/SP/2 1/SM1/SP/0 1/SM1/SP/1 1/SM1/SP/2 1/SM4/SP/0 1/SM4/SP/1 1/SM4/SP/2 1/SM5/SP/0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 4-129 OL-21483-02...
Page 140: Guidelines For Maintenance Of Fabric Links
See the other sections in this chapter for additional information on CLI commands and fiberoptic cleaning procedures. If you are unsure whether you need to perform maintenance procedures on your system, contact Cisco Technical Support first to discuss your options. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request”...
Page 141: Chapter 5 Troubleshooting Interfaces
• Troubleshooting Common Issues with Bundle Interfaces, page 5-144 • Verifying and Troubleshooting Configured Interfaces To troubleshoot the configured interfaces on Cisco IOS XR software, perform the following procedure. SUMMARY STEPS show interfaces type instance show controllers interface-type interface-instance stats...
Page 142: Verifying And Troubleshooting Configured Interfaces
POS0/0/1/0 is up, line protocol is up Hardware is Packet over SONET/SDH Internet address is 172.18.140.1/24 MTU 4474 bytes, BW 155520 Kbit reliability 255/255, txload 1/255, rxload 1/255 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-132 OL-21483-02...
Page 143 OUT multicast pkts: broadcast pkts: OUT broadcast pkts: drop pkts: OUT drop pkts: errors pkts: OUT errors pkts: Chains -------------------- Base decap chain: hdlc <14> <0xfd6a0a74, 0x00000000> < 0> Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-133 OL-21483-02...
Page 144 The following example shows counters implemented for Multiprotocol Label Switching (MPLS) packets. The following output under Protocol Chains in the show netio idb command shows the MPLS packets incrementing: mpls <25> <0xfcc7b2b8, 0x00000000> < 152, 17328> Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-134 OL-21483-02...
Page 145 <0xfd054bfc, 0x48319f04> < 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0> Decap: pcn_input <55> <0xfd054bfc, 0x4830ba8c> < 0> q_fq_input <96> <0xfd05f330, 0x48312c7c> < 0> Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-135 OL-21483-02...
Page 146 <60> <0xfcadba38, 0x4824c0fc> < 0> Protocol SAFI counts: -------------------- Protocol SAFI Pkts In Bytes In Pkts Out Bytes Out --------------- ---------- ---------- ---------- ---------- ---------- ipv4 Unicast Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-136 OL-21483-02...
Page 147 RP/0/RP0/CPU0:router# show controllers plim asic statistics interface pos 0/5/0/0 Node: 0/5/CPU0 ------------------------------------------------------------------------------- POS0/5/0/0 Tx Statistics ------------------------------------------------------------------------------- TotalOctets : 85578609 TotalPkts : 265727 Unicastpkts : 265727 MulticastPkts BroadcastPkts <64Octets : 70059 64Octets 65to127Octets : 145269 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-137 OL-21483-02...
Page 148 0 mac_transfer_error: 0 excessive_collision: 0 unrecog_mac_control_received: 0 fc_sent: 0 good_fc_received: 0 rx_over_flow_events: 0 undersize: 0 fragments: 0 oversize: 0 jabber: 0 mac_rcv_error: 71 bad_crc: 0 collisions: 0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-138 OL-21483-02...
Page 149: Verifying And Troubleshooting Pluggable Optical Interfaces
Contact Cisco Technical Support if the problem is not resolved. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-139 OL-21483-02...
Page 150 Xcvr Type: OC12 SR-1/STM4 MM (8) Connector: LC Vendor name: CISCO-OCP Vendor part number: TRPD12MM2EAS-CS State: Enabled SPA 0/6/4 device pluggable-optics 5/0 info: int sonet 6/4/5: ID: SFP Extended ID: 4 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-140 OL-21483-02...
Page 151 Rx power: -16 dBm (194 in units of 0.1 uW) Transceiver: Enabled SW TX Fault: unavailable SW LOS: unavailable No active alarms No active warnings Version Identifier (VID): V01 Product Identifier (PID): SFP-OC12-MM Part Number (PN): 10-2079-01 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-141 OL-21483-02...
Page 152 SPA 0/6/4 device pluggable-optics 6/0 info: int sonet 6/4/6: Phased Initialization Phase Reached: 4 Phase Exit Code: Success 0 Phase Read Offset: 256 Socket Verification Compatibility: Compatibility passed Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-142 OL-21483-02...
Page 153 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x20: 02 77 00 05 01 8E 00 08 00 00 00 00 00 00 00 00 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-143 OL-21483-02...
Page 154: Troubleshooting Common Issues With Bundle Interfaces
--More— Troubleshooting Common Issues with Bundle Interfaces Bundle interfaces utilize features and services provided by different components. To troubleshoot the common issues with bundle interface components on Cisco IOS XR software, perform the following procedure. SUMMARY STEPS show tech-support bundles file filename...
Page 155: Troubleshooting Common Issues With Bundle Interfaces
Step 7 Contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-145 OL-21483-02...
Page 156 Chapter 5 Troubleshooting Interfaces Troubleshooting Common Issues with Bundle Interfaces Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 5-146 OL-21483-02...
Page 157: Chapter 6 Troubleshooting The Control Plane Ethernet Network
All devices in a system using Cisco IOS XR software connect to the system control plane Ethernet network. There are two control planes in the control Ethernet topology: Intrachassis—Within a chassis, the control plane is provided using Fast Ethernet (FE) links between...
Page 158: Using The Online Diagnostics Tools
The online diagnostics tools can alert you to potential problems in the control plane connections between the Fabric Card Chassis and the Line Card Chassis. For information on using the online diagnostics, see “Using Diagnostic Commands” section on page 7-166. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-148 OL-21483-02...
Page 159: Troubleshooting Booting The System Control Plane Ethernet Network
Troubleshooting the Control Plane Ethernet Network Troubleshooting Booting the System Control Plane Ethernet Network Troubleshooting Booting the System Control Plane Ethernet Network To verify and troubleshoot booting the Cisco CRS-1 system control plane Ethernet network, perform the following procedures. SUMMARY STEPS show platform...
Page 160 RP/0/RP0/CPU0:router# show controllers switch 0 statistics location 0/RP1/CPU0 Step 6 Places the DSC in ROMMON mode. admin config-register 0x0 exit reload Example: RP/0/RP0/CPU0:router# admin RP/0/RP0/CPU0:router(admin)# config-register RP/0/RP0/CPU0:router(admin)# exit RP/0/RP0/CPU0:router# reload Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-150 OL-21483-02...
Page 161: Examples
“Obtaining Documentation and Submitting a Service Request” section on page viii in the Preface. Examples The following example shows that all expected nodes in the Cisco CRS-1 Multishelf System are in the run state: RP/0/RP0/CPU0:router# show platform Node Type...
Page 162 Card Configuration Protocol Test client out-of-band The following example shows that there are 18 nodes in the Cisco CRS-1 Multishelf System in the run state, which means that 18 boot requests have been received by eth_server and 18 replies have been sent:...
Page 163: Troubleshooting The Multishelf System Router Topology
You must reset or power cycle for new config to take effect rommon B2> reset Troubleshooting the Multishelf System Router Topology To verify and troubleshoot the Cisco CRS-1 Multishelf System router topology, perform the following procedures. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router...
Page 164 Place the DSC in ROMMON mode: admin config register 0x0 exit reload show_bcm_links Exit ROMMON mode: confreg 0x102 reset Contact Cisco Technical Support if the problem is not resolved Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-154 OL-21483-02...
Page 165 1 statistics location node-id Verify that the expected internal switch ports are carrying traffic (Tx and Rx frames). Example: RP/0/RP0/CPU0:router# show controllers switch 1 statistics location 0/RP1/CPU0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-155 OL-21483-02...
Page 166: Examples
This command is used to confirm connectivity from the RP show_bcm_links or SC to the Catalyst switch in ROM monitor (ROMMON) mode. See Cisco IOS XR ROM Monitor Guide for the Cisco Example: CRS-1 Router for information on entering ROMMON rommon B1 >...
Page 167 0/LC3 0/LC4 0/LC5 0/LC6 0/LC7 164799 0/LC8 10 : 0/LC9 11 : 0/LC10 12 : 0/LC11 13 : 0/LC12 14 : 0/LC13 15 : 0/LC14 16 : 0/LC15 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-157 OL-21483-02...
Page 168: Troubleshooting The Catalyst Switch
Restrictions The external Catalyst switches must operate with the same software version: Cisco IOS software release 12.2 with Sup720 engine. Please note that older hardware can also use the CatOS software, but Cisco strongly recommends using the Cisco IOS software.
Page 169: Troubleshooting The Crs-1, 4-Slot, 8-Slot, Or 16-Slot System Router Topology
Troubleshooting the CRS-1, 4-slot, 8-slot, or 16-slot System Router Topology To verify and troubleshoot the Cisco CRS-1 8-Slot Line Card Chassis, Cisco CRS-1 8-Slot Line Card Chassis, or Cisco CRS-1 16-Slot Line Card Chassis router topology, perform the following procedures.
Page 170 (intrachassis connections). Only the Cisco CRS-1 16-Slot Line Card Chassis contains switch 0 and switch 1. The Note Cisco CRS-1 8-Slot Line Card Chassis and Cisco CRS-1 4-Slot Line Card Chassis contain switch 0. SUMMARY STEPS show platform show controllers switch 0 ports location node-id...
Page 171: Examples
RP/0/RP0/CPU0:router# show controllers backplane ethernet detail location 0/RP1/CPU0 Step 7 Places the DSC in ROMMON mode. admin config-register 0x0 exit reload Example: RP/0/RP0/CPU0:router# admin RP/0/RP0/CPU0:router(admin)# config-register RP/0/RP0/CPU0:router(admin)# exit RP/0/RP0/CPU0:router# reload Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-161 OL-21483-02...
Page 172 GE Port 0 : Down (Connects to - GE_0) GE Port 1 : Down (Connects to - GE_1) The following example shows the statistics for Switch Port 0: Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-162 OL-21483-02...
Page 173 60733356 packets output, 1676989930 bytes, 0 total output drops Output 6904806 broadcast packets, 6904806 multicast packets Output errors: 0 underruns, 0 aborts, 0 loss of carrier Write rejects : 0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-163 OL-21483-02...
Page 174 Chapter 6 Troubleshooting the Control Plane Ethernet Network Troubleshooting the CRS-1, 4-slot, 8-slot, or 16-slot System Router Topology Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 6-164 OL-21483-02...
Page 175: Chapter 7 Collecting System Information
C H A P T E R Collecting System Information This chapter describes techniques that you can use to collect system information for troubleshooting routers using Cisco IOS XR software. It includes the following sections: Capturing Logs, page 7-165 •...
Page 176: Using Ping And Traceroute
The Cisco IOS XR diagnostic tests verify control Ethernet and fabric data paths between nodes in a system using Cisco IOS XR software. If a diagnostic tests fails, it indicates a bad data path. The integrity of the covered data paths is verified when the diagnostic tests pass.
Page 177 0/RP0/CPU0: SelfPingOverFabric{ID=2} has completed successfully RP/0/RP0/CPU0:Sep 1 12:50:58.809 : online_diag_rp[351]: %DIAG-DIAG-6-TEST_RUNNING : RP 0/RP0/CPU0: Running FabricPingTest{ID=3} ... RP/0/RP0/CPU0:Sep 1 12:51:00.672 : online_diag_rp[351]: %DIAG-DIAG-6-TEST_OK : RP 0/RP0/CPU0: FabricPingTest{ID=3} has completed successfully Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 7-167 OL-21483-02...
Page 178: Transient Condition When Standby Rp Becomes Active
Therefore, you should perform these tests manually after the standby RP has been active for at least five minutes. To run a specified on-demand diagnostic test or series of tests, use the diagnostic start location command. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 7-168 OL-21483-02...
Page 179: Offline Diagnostics-Fdiag Running State
"FDIAG RUNNING" state until it is explicitly unloaded using the diagnostic unload location node-id command. Additional Reference for Diagnostic Commands For details on diagnostics commands and available tests, see Cisco IOS XR Diagnostics at the Configuration Guide site: http://www.cisco.com/en/US/products/ps5763/ products_installation_and_configuration_guides_list.html...
Page 180 Chapter 7 Collecting System Information Commands Used to Display Process and Thread Details Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 7-170 OL-21483-02...
Page 181 Troubleshooting a Process Restart, page 8-195 • The Cisco IOS XR software is built on a modular system of processes. A process is a group of threads that share virtual address (memory) space. Each process provides specific functionality for the system and runs in a protected memory space to ensure that problems with one process cannot impact the entire system.
Page 182: C H A P T E R 8 Process Monitoring And Troubleshooting
Wdsysmon expects to periodically hear a “pulse” from every registered event manager in the system. When an event manager is missing, wdsysmon runs a debug script that shows exactly what the thread that created the event manager is doing. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-172 OL-21483-02...
Page 183: Core Dumps
The follow command is used to unobtrusively debug a live process or live thread in a process. The follow command is particularly useful for: • process deadlock, livelock, or mutex conditions • high CPU use conditions Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-173 OL-21483-02...
Page 184 Therefore, for these software releases, you should use other available commands for troubleshooting and call Cisco Technical Support if the problem is not resolved. (This crash behavior does not occur for releases other than the ones listed.) The following example shows the live thread of process 929034375.
Page 185: Show Processes Commands
Was a process delayed indicating a boot failure or boot problems • Did the processes start within the time constraints set by the system • RP/0/RP0/CPU0:router# show processes boot location 0/rp1/cpu0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-175 OL-21483-02...
Page 186: Show Processes Startup Command
07/05/2006 14:46:37.514 Run mqueue(1) Not configured None cat(1) /etc/motd Not configured None platform_dllmap( 07/05/2006 14:46:37.514 Run shmwin_svr(1) 07/05/2006 14:46:37.514 Run devf-scrp(1) -e 0xf0000038 -m /bootflash: -s 0xfc000000,64m -r -t4 -b10 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-176 OL-21483-02...
Page 187: Show Processes Failover Command
23.00 FT_INFRA 0.000 syslogd 0.065s 23.00 FT_INFRA 0.000 syslogd_helper 0.066s 38.00 FT_INFRA 0.000 packet 0.067s 40.00 FT_INFRA 0.000 chkpt_proxy 0.070s 40.00 FT_INFRA 0.000 chkpt_proxy 0.072s 40.00 FT_INFRA 0.000 chkpt_proxy Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-177 OL-21483-02...
Page 188: Show Processes Blocked Command
The polling interval should not be too short (enough to show a sustained blocked state). For example, the Cisco CRS-1 8-Slot Line Card Chassis requires a minimum of 20 requests for each interval (2 RPs and 8 LCs) if fully equipped.
Page 189: Example
Therefore, for these software releases, you should use other available commands for troubleshooting and call Cisco Technical Support if the problem is not resolved. (This crash behavior does not occur for releases other than the ones listed.) •...
Page 190: Process States
Process States Within the Cisco IOS XR software there are servers that provide the services and clients that use the services. A specific process can have a number of threads that provide the same service. Another process can have a number of clients that may require a specific service at any point in time.
Page 191: Synchronous Message Passing
(what is not normal for your functioning router). This provides a baseline for you to use as a comparison when troubleshooting process life cycles. At any point in a time a thread can be in a particular state. Table 8-1 lists the thread states. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-181 OL-21483-02...
Page 192: Process Monitoring
RP/0/RP0/CPU0:router# show fault manager metric process ospf location ===================================== job id: 269, node name: 0/RP0/CPU0 process name: ospf, instance: 1 -------------------------------- last event type: process start Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-182 OL-21483-02...
Page 193: Process Monitoring Commands
CPU-hog, it is terminated and a coredump of the process is captured and stored on the configured device (exception choice) to aid debugging. For information on troubleshooting high CPU usage, see the “Troubleshooting High CPU Utilization and Process Timeouts” section on page 8-185. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-183 OL-21483-02...
Page 194 CPU hog (a hog that lasts for more than 30 seconds) the node will be reset. There will be a log such as the following just before the reset: RP/0/RP0/CPU0:Dec 20 10:36:08.990 : wdsysmon[367]: %HA-HA_WD-1-CURRENT_STATE : Persistent Hog detected for more than 30 seconds Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-184 OL-21483-02...
Page 195: Troubleshooting High Cpu Utilization And Process Timeouts
Process Monitoring and Troubleshooting Troubleshooting High CPU Utilization and Process Timeouts If the hog is persistent and the node is reset, contact Cisco Technical Support. For contact information for Cisco Technical Support, see the “Obtaining Documentation and Submitting a Service Request”...
Page 196: Using Show Process And Top Processes Commands
Therefore, for these software releases, you should use other available commands for troubleshooting and call Cisco Technical Support if the problem is not resolved. (This crash behavior does not occur for releases other than the ones listed.) The following example shows the processing using the CPU.
Page 197 0:36:37 0.16% gsp 0:10:33 0.12% sysdb_mc 0:06:49 0.04% mpls_ldp 756K 0:00:48 0.04% envmon_mon 0:01:39 0.02% top_procs 728K 0:01:02 0.02% tcp 548K 0:00:06 0.00% oir_daemon 204K 0:00:07 0.00% i2c_server Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-187 OL-21483-02...
Page 198: Troubleshooting A Process Block
Cisco Technical Support if the problem is not resolved. (This crash behavior does not occur for releases other than the ones listed.) Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-188 OL-21483-02...
Page 199 Reply 8199 devc-ser8250 20502 attachd Reply 20500 eth_server 20502 attachd Reply 8204 mqueue 20503 qnet Reply 20500 eth_server 20503 qnet Reply 20500 eth_server 20503 qnet Reply 20500 eth_server Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-189 OL-21483-02...
Page 200 #3 0xfc1e4408 [_resmgr_io_handler] trace_back: #4 0xfc1e40b0 [_resmgr_handler] trace_back: #5 0xfc15aa28 [_eventmgr_resmgr_handler] trace_back: #6 0xfc159e04 [_event_message_handler] trace_back: #7 0xfc159d54 [_event_message_handler] trace_back: #8 0xfc1575a4 [event_dispatch] trace_back: #9 0x482007bc [<N/A>] Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-190 OL-21483-02...
Page 201 #4 0xfc1e40b0 [_resmgr_handler] trace_back: #5 0xfc15aa28 [_eventmgr_resmgr_handler] trace_back: #6 0xfc159e04 [_event_message_handler] trace_back: #7 0xfc159d54 [_event_message_handler] trace_back: #8 0xfc1575a4 [event_dispatch] trace_back: #9 0x482007bc [<N/A>] ENDOFSTACKTRACE Iteration 5 of 5 ------------------------------ Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-191 OL-21483-02...
Page 202: Troubleshooting A Process Crash On Line Cards
/c12k-os-mbi-3.7.0.26I/mbiprp-rp.vm Active Packages: mem:c12k-mpls-3.7.0.26I mem:c12k-mini-3.7.0.26I Step 2 Gives the workspace (directory) and the build server where show version image was built. Example: RP/0/RP0/CPU0:router# show version | begin 0/0/CPU0 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-192 OL-21483-02...
Page 203: Troubleshooting A Memory Leak
--------------- ------------ pid: 1 pid: 1 syslog_dev dllmgr pid: 1 pid: 1 sysdb_svr_local sysdb_svr_local sysmgr sysdb_svr_local sysdb_svr_local sysdb_mc sysdb_svr_local sysdb_mc sysdb_svr_local sysdb_mc sysdb_mc sysdb_mc pid: 1 pid: 1 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-193 OL-21483-02...
Page 204: Troubleshooting A Hardware Failure
(and this is generally true for SNMP queries), the Cisco CRS-1 might experience very high CPU utilization as it searches for the data. The Cisco CRS-1 might not be able to complete the query and data transfer before the timer on the SNMP server expires.
Page 205: Troubleshooting A Process Restart
However, it can be helpful to troubleshoot the event, to find out why the process crashed, and whether you need to consider taking any action to locate or correct a problem. To troubleshoot a process restart, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request”...
Page 206 Chapter 8 Process Monitoring and Troubleshooting Troubleshooting a Process Restart Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 8-196 OL-21483-02...
Page 207: Chapter 9 Troubleshooting Memory
Watchdog system monitor (wdsysmon) is part of the high-availability (HA) infrastructure of the Cisco IOS XR software. Wdsysmon runs on the distributed route processor (DRP) and line cards (LCs) with the primary goal of monitoring the system for problem conditions and attempting to recover from them.
Page 208: Configuring And Displaying Memory Thresholds
25.6 12.8 102.4 51.2 25.6 1024 204.8 102.4 51.2 2048 409.6 204.8 102.4 4096 819.2 409.6 204.8 To identify, configure, and display memory thresholds, perform the following procedure. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 9-198 OL-21483-02...
Page 209 [location node-id] minor percentage-memory-available severe percentage-memory-available critical percentage-memory-available commit exit show watchdog [memory-state | threshold memory {configured | default}] [location node-id] Contact Cisco Technical Support if the problem is not resolved. DETAILED STEPS Command or Action Purpose Step 1 Enters global configuration mode.
Page 210 The show watchdog threshold memory default command enables you to display the default memory thresholds. RP/0/RP0/CPU0:router# show watchdog threshold memory defaults location all [K---- node0_RP1_CPU0 --- Default memory thresholds: Minor: Severe: Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 9-200 OL-21483-02...
Page 211: Setting Timeout For Persistent Cpu Hogs
If wdsysmon detects a CPU hog on the card, it resets the node after 30 seconds. This default timeout value can be reset if required using the watchdog monitor cpu-hog persistent timeout command. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 9-201...
Page 212: Memory Usage Analyzer
(total memory minus image size, reserved, and flashfsys), the image size, the reserved space for packet memory, and the I/O memory used as a backup for packet memory. If there is not sufficient memory, install more memory. See the Cisco CRS-1 Carrier Routing System documentation at the following URL: http://www.cisco.com/en/US/products/ps5763/tsd_products_support_series_home.html...
Page 213: Troubleshooting Process Memory
Troubleshooting Process Memory TheCisco IOS XR Process Placement feature balances application processes between the available route processors (RPs) and distributed route processors (DRPs) on a Cisco CRS-1 system, based on memory usage and other criteria. Under normal operating conditions, processes are managed automatically by the Cisco IOS XR software.
Page 214 RP/0/RP0/CPU0:router# show processes memory location 0/rp0/cpu0 Text Data Stack Dynamic Process 65536 32768 57344 38064128 eth_server 147456 4096 24576 13217792 fgid_server 90112 4096 94208 8437760 parser_server 65554 40960 32768 7430144 devb-ata Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 9-204 OL-21483-02...
Page 215 48204000 4096 Program Text or Data 48205000 4096 Program Text or Data 48206000 16384 Allocated Memory 4820a000 16384 Allocated Memory 4820e000 16384 Allocated Memory 48212000 16384 Allocated Memory Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 9-205 OL-21483-02...
Page 216 --------- ---------- ------- ------- top_procs 584300 587052 2752 334144 334920 sysdb_svr_local 1445844 1446004 i2c_server 14464 14624 You are now free to remove snapshot memcmp_start.out and memcmp_end.out under /p Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 9-206 OL-21483-02...
Page 217: Resolving Process Memory Problems
If it is not a real process or if it is a real process and it does not restart, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining...
Page 218 If memory fragmentation is detected, shut down some interfaces. This may free the fragmented blocks. If this works, the memory is behaving normally. (You should add more memory.) If shutting down interfaces does not resolve the problem, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request”...
Page 219: Validating And Troubleshooting Rom Monitor Software Installation
“Troubleshooting Upgrading and Downgrading ROM Monitor Software on Cisco CRS-1 Routers” section on page 10-213 A copy of the ROMMON software exists on each card. If a card fails to boot the Cisco IOS XR software, Note the ROMMON software takes control and places the card in ROMMON mode. Because a card in ROMMON mode is not running the Cisco IOS XR software, that card will be unavailable for normal router operations.
Page 220: C H A P T E R 10 Troubleshooting Upgrading And Downgrading Software
Although it is distributed with routers that use the Cisco IOS XR software, ROMMON is a separate program from the Cisco IOS XR software. During normal startup, the ROMMON initializes the cards, and then control passes to the Cisco IOS XR software. After the Cisco IOS XR software takes over, ROMMON is no longer in use.
Page 221 Verify that the expected ROMMON version is installed. If Example: the version is not as expected, see Cisco IOS XR ROM RP/0/RP0/CPU0:router# show version Monitor Guide for the Cisco CRS-1 Router for information on upgrading or downgrading the ROM Monitor version.
Page 222 To save the current environment Example: variable settings, enter the sync command. See rommon B2> IP_ADDRESS=1.1.1.1 Cisco IOS XR ROM Monitor Guide for the Cisco CRS-1 rommon B3> IP_SUBNET_MASK=255.255.254.0 Router for more information on changing the environment rommon B4> DEFAULT_GATEWAY=1.1.0.1 variable settings.
Page 223: Troubleshooting Upgrading And Downgrading Rom Monitor Software On Cisco Crs-1 Routers
Chapter 10 Troubleshooting Upgrading and Downgrading Software Troubleshooting Upgrading and Downgrading ROM Monitor Software on Cisco CRS-1 Routers NET_MASK = 255.255.255.0, DEFAULT_GATEWAY = 192.39.52.1, ? = 0, TURBOBOOT = , EA SYBAKE = 0x0, ReloadReason = 66, BSI = 0, BOOT = disk0:hfr-os-mbi-3.3.0.1I/mbihf r-rp.vm,1;, confreg = 0x102^@...
Page 224: Troubleshooting Upgrading And Downgrading Cisco Ios Xr Software Packages
See Cisco IOS XR ROM Monitor Guide for the Cisco CRS-1 Router for detailed information and procedures on upgrading and downgrading ROM Monitor software on the Cisco CRS-1...
Page 225: Troubleshooting The Statistics Infrastructure
Diagnosing Problems with Statistics Values, page 11-223 Note Most of the commands used in the chapter requires a Cisco-support task id to execute them. Debugging Statistics Infrastructure The following sections list the debug commands that are used to diagnose problems with the statistics infrastructure: •...
Page 226: C H A P T E R 11 Troubleshooting The Statistics Infrastructure
It is recommended to restrict this command to a single interface. debug statsd server errors Command The debug statsd server errors command prints any unexpected events that occurred in the statistics server. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-216 OL-21483-02...
Page 227: Trace Commands For The Statistics Infrastructure
Lists the entries that have been added into or removed from the datatree that indexes the checkpoint table. types Lists the mappings that have been made between the feature and item strings and statistics type IDs. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-217 OL-21483-02...
Page 228: Show Statsd Server Trace Command
Command, page 11-219 show statsd collectors Command, page 11-220 • • show statsd registrations Command, page 11-222 show statsd requests Command, page 11-222 • Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-218 OL-21483-02...
Page 229: Show Statsd Manager Info Command
The output divides the requests into an internal namespace and a new-feature namespace. The requests for the new-feature namespace are subdivided with the latest data requests from the collectors, data cache requests, and snapshot requests. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-219 OL-21483-02...
Page 230: Show Statsd Collectors Command
The show statsd collectors command lists all the processes that have registered as a statistics collector, and can be filtered to a single node or a single process. Table 11-7 lists the keywords/options for the show statsd collectors command. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-220 OL-21483-02...
Page 231 RP/0/RP0/CPU0:router# show statsd collectors detail location 0/0/CPU0 pid 131157 STATS COLLECTORS ---------------------------------- statsd_server node: 0x0 PID: 131157 Total bytes sent: 25700 Total messages sent: 96 Errors: 0 periodic updates: 91 on-demand replies: 0 nacks: other messages: Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-221 OL-21483-02...
Page 232: Show Statsd Registrations Command
118 wrapping entries (1024 possible, 0 filtered, 118 total) STATS REQUEST DATA ------------------ Displaying data about the last 12 requests (0 failed) min / avg / max Time taken (ms): 1 / 15 / 37 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-222 OL-21483-02...
Page 233: Diagnosing Problems With Statistics Values
Timeouts and Delays When Retrieving Data, page 11-224 • Displaying Incorrect Rates for the show interfaces Command, page 11-225 • Displaying Incorrect Data for the show Commands, page 11-226 • Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-223 OL-21483-02...
Page 234: Errors When Retrieving Data From The Statistics Manager Edm
( as it is blocked, or failed to join the group). Whereas, if there are many missing responses, the problem lies with the statistics GPS group, which is broken on one or more nodes. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-224 OL-21483-02...
Page 235: Displaying Incorrect Rates For The Show Interfaces Command
[interface type instance] Contact Cisco Technical Support if the problem is not resolved. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-225...
Page 236: Displaying Incorrect Data For The Show Commands
Perform this task to find out which collectors are returning incorrect data. SUMMARY STEPS show stats manager info show statsd registrations ifname interface name type stats ID show statsd collectors brief Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-226 OL-21483-02...
Page 237 0x9 (ALL|REG) RP/0/RP0/CPU0:altadena#show statsd collectors brief | inc 57416 0x21 57416 hfr_pm 16840 0x31 57416 hfr_pm 27604 RP/0/RP0/CPU0:altadena#show statsd collectors brief | inc 61558 0x31 61558 plim_8p_10ge 22688 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-227 OL-21483-02...
Page 238 Chapter 11 Troubleshooting the Statistics Infrastructure Diagnosing Problems with Statistics Values Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 11-228 OL-21483-02...
Page 239: Chapter 12 Multiprotocol Label Switching Checklist
• show tech-support routing isis For more detailed information about the show tech-support commands, see Cisco IOS XR Advanced System Command Reference for the Cisco CRS-1 Router. Multiprotocol Label Switching Recommendations Follow these recommendations to troubleshoot Multiprotocol Label Switching (MPLS): Verify the show commands for the MPLS control plane to ensure that there is no problem.
Page 240: Troubleshooting L3Vpn Mpls Traffic Loss
Cisco CRS-1 Router. Troubleshooting L3VPN MPLS Traffic Loss To troubleshoot the traffic loss of MPLS packets on L3VPN layer in Cisco CRS-1, perform the following procedure. The general flow of packets in a Cisco CRS-1 goes through the following sequence: Incoming interface =>...
Page 241: Chapter 13 Troubleshooting Load Balancing
Load Balancing Function Load balancing is a Cisco IOS XR software feature that improves the utilization of parallel links by distributing traffic flows among them. A traffic flow consists of packets that have common Layer 3 and Layer 4 characteristics, such as the same source and destination IP addresses. The concept of a flow also extends to Layer 2 bundles and to virtual links such as MPLS traffic-engineering tunnels.
Page 242: Layer 2 Load Balancing
Layer 2 Load Balancing Load balancing at Layer 2 is also done by flow. Cisco IOS XR software provides load balancing for Layer 2 bundles only. Data is distributed to a link in proportion to the bandwidth of the link in relation to its bundle.
Page 243: Terminology
Path Describes a valid route to reach a destination prefix. Cisco IOS XR assigns a cost to each path. A set of active paths to a destination prefix may have equal or unequal costs.
Page 244: Verifying The Routing Table Entries For Parallel Links
Troubleshooting Layer 3 or Layer 4 Load Balancing Verifying the Routing Table Entries for Parallel Links Cisco Express Forwarding uses the path information in the IP routing table to balance traffic over multiple links. For this reason, verifying correct load balancing with Cisco Express Forwarding begins with confirming the contents of the IP routing table.
Page 245 Displays the routes to a destination address. Verify that number of routes in the routing table equals the number of parallel links. Example: RP/0/RP0/CPU0:Router# show route 10.1.2.1 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-235 OL-21483-02...
Page 246: Configuring Layer 4 Load Balancing
Verifying the CEF Database and Measuring Flows Cisco Express Forwarding uses the path information in the Cisco Express Forwarding (CEF) database to balance traffic over multiple links. For this reason, confirming proper CEF load balancing begins with confirming the contents of the CEF database.
Page 247: Troubleshooting Layer 2 Load Balancing
Viewing the Expected Paths and Measuring the Flows Verifying the Bundle Status, IGP Route, and CEF Database Cisco Express Forwarding uses the path information in the IP routing table to balance traffic over multiple links. For this reason, confirming proper Cisco Express Forwarding load balancing begins with confirming the contents of the IP routing table.
Page 248: Configuring Layer 4 Load Balancing
Viewing the Expected Paths and Measuring the Flows Cisco IOS XR provides a bundle utility that predicts how Layer 2 loads are balanced across member links. This is an interactive tool prompts for the information that the load balancing algorithm uses to allocate flows to member links.
Page 249: Configuration Examples For Troubleshooting Load Balancing
This section shows a practical example of troubleshooting load balancing problems, in the following sections: Troubleshooting Layer 3 or Layer 4 Load Balancing Example • Troubleshooting Layer 2 Load Balancing Example • Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-239 OL-21483-02...
Page 250: Troubleshooting Layer 3 Or Layer 4 Load Balancing Example
Known via "ospf 200", distance 110, metric 2, type intra area Installed Sep 19 06:49:40.555 for 04:12:20 Routing Descriptor Blocks 10.12.48.1, from 10.1.2.1, via GigabitEthernet0/6/5/5 Route metric is 2 No advertising protos. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-240 OL-21483-02...
Page 251 OSPF allows equal cost load balancing only, so one of the three paths must have a higher cost. To balance loads among links with unequal cost metrics, use Enhanced Interior Gateway Routing Protocol or Interior Gateway Routing Protocol (EIGRP/IGRP) as your Interior Gateway Protocol (IGP). Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-241 OL-21483-02...
Page 252 10.12.40.1, from 10.1.2.1, via GigabitEthernet0/6/5/7 Route metric is 2 No advertising protos. The next step is to look at how Cisco Express Forwarding uses the information in the routing table to forward packets. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router...
Page 253: Verifying Ipv4/Ipv6 Unicast Load Balancing
Checking the CEF Database • Verifying Load Balancing Configuring Layer 4 Load Balancing The default load balancing algorithm for Cisco IOS XR is Layer 3, so first enable Layer 4 (7-tuple algorithm): RP/0/RP0/CPU0:P2_CRS-8# configure RP/0/RP0/CPU0:P2_CRS-8(config)# cef load-balancing fields L4 RP/0/RP0/CPU0:P2_CRS-8 (config)# end Checking the CEF Database Verify that the CEF database has the expected interfaces to the test destination, 10.1.2.1:...
Page 254 IPV4_MULTICAST This shows that the ingress data rate on GigabitEthernet0/6/5/4 is 43000 bps and that the egress data rate on interface GigabitEthernet0/6/5/5 is also 43000 bps, as expected. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-244 OL-21483-02...
Page 255 TCP flows, but you can obtain similar results using the ping command. Just make sure that the ping packets pass through the router with load balancing interfaces to reach the destination IP address. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-245...
Page 256: Troubleshooting Layer 2 Load Balancing Example
This verifies that the bundle is active and is using the three expected interfaces (ports). If the bundle on your system is not active, troubleshoot that problem first. It might not have the minimum number of active links or minimum bandwidth to come up. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-246 OL-21483-02...
Page 257: Checking The Igp Routing Table And Cef Database
Verifying Bundle Load Balancing Cisco IOS XR software has a bundle utility that predicts how load balancing will distribute flows among the bundle links. This is an interactive utility that prompts for the information that the load balancing algorithm uses to assign flows to links. For more information, see Cisco IOS XR Interface and Hardware Component Command Reference.
Page 258 RP/0/RP0/CPU0:RouterB# bundle-hash bundle-ether 12 Tue Oct 28 03:56:48.786 PST DST Specify load-balance configuration (L3/3-tuple or L4/7-tuple) (L3,L4): L4 Single SA:SP/DA:SP pair (IPv4,IPv6) or range (IPv4 only): S/R [S]: S Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-248 OL-21483-02...
Page 259 0 output errors, 0 underruns, 0 applique, 0 resets 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions This verifies that the flow to port 23 is using GigabitEthernet0/6/5/7, as expected. Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 13-249 OL-21483-02...
Page 260 IOS XR IP Addresses and Services Configuration Guide for the Cisco CRS-1 Router. For more information on Cisco Express Forwarding commands on Cisco IOS XR Software, see Cisco IOS XR IP Addresses and Services Command Reference for the Cisco CRS-1 Router.
Page 261: I N D E X
6-149 monitor interface command 1-67 describe hostname command 1-39 monitor processes command 1-69 diagnostic commands 7-166 monitor threads command 1-70 dumpcore command 8-173 more nvram command 10-211 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router IN-251 OL-21483-02...
Page 262 4-128 show controllers fabric connectivity all command 4-128 show controllers fabric link port command 4-117, 4-122, show_bcm_links command 6-151, 6-156 4-123, 8-188, 8-192 show adjacency command 3-87 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router IN-252 OL-21483-02...
Page 263 8-182 1-10, 8-183 show processes all command top processes command 1-15 8-180 show processes blocked command trace commands 1-56 8-178, 8-181, 8-183, 8-188, 8-192 traceroute commands 7-166 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router IN-253 OL-21483-02...
Page 264 Index transport input telnet command watchdog monitor cpu-hog persistent timeout command 9-201 watchdog threshold memory command 9-199 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router IN-254 OL-21483-02...

Cisco CRS-1 - Carrier Routing System Router Troubleshooting Manual

Chapter 1 General Troubleshooting

Chapter 2 Troubleshooting Booting

CHAPTER 3 Troubleshooting Forwarding

CHAPTER 4 Troubleshooting Router Switch Fabric

Chapter 5 Troubleshooting Interfaces

Chapter 6 Troubleshooting the Control Plane Ethernet Network

Chapter 7 Collecting System Information

CHAPTER 8 Process Monitoring and Troubleshooting

Chapter 9 Troubleshooting Memory

CHAPTER 10 Troubleshooting Upgrading and Downgrading Software

Troubleshooting the Statistics Infrastructure

Chapter 12 Multiprotocol Label Switching Checklist

Chapter 13 Troubleshooting Load Balancing

I N D E X

Quick Links

Troubleshooting

Related Manuals for Cisco CRS-1 - Carrier Routing System Router

Summary of Contents for Cisco CRS-1 - Carrier Routing System Router