Cisco Nexus 9000 Series Troubleshooting Manual
  1. Manuals
  2. Brands
  3. Cisco Manuals
  4. Network Router
  5. Nexus 9000 Series
  6. Troubleshooting manual

Cisco Nexus 9000 Series Troubleshooting Manual

Nx-os release 7.x
Hide thumbs Also See for Nexus 9000 Series:
Table of Contents

Advertisement

Quick Links

See also: Configuration Manual , Manual
Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x
First Published: 2015-02-01
Last Modified: 2015-12-14
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883

Advertisement

Table of Contents
loading

Related Manuals for Cisco Nexus 9000 Series

Summary of Contents for Cisco Nexus 9000 Series

  • Page 1 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x First Published: 2015-02-01 Last Modified: 2015-12-14 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883...
  • Page 2 This product includes software written by Tim Hudson (tjh@cryptsoft.com). http:// Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks .

  • Page 3: Table Of Contents

    Troubleshooting with Logs Troubleshooting Modules Viewing NVRAM Logs Contacting Customer Support Troubleshooting Installations, Upgrades, and Reboots C H A P T E R 3 About Upgrades and Reboots Upgrade and Reboot Checklist Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 4 Troubleshooting Ports C H A P T E R 5 About Troubleshooting Ports Guidelines and Limitations for Troubleshooting Ports Initial Port Troubleshooting Checklist Viewing Port Information Troubleshooting Port Statistics from the CLI Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 5 C H A P T E R 8 About Troubleshooting STP Initial Troubleshooting STP Checklist Troubleshooting STP Data Loops Troubleshooting Excessive Packet Flooding Troubleshooting Convergence Time Issues Securing the Network Against Forwarding Loops Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 6 C H A P T E R 1 4 NX-API Guidelines NX-API Is Not Responding Configuration Fails Permission Is Denied for Bash Output Cannot Be Retrieved from the Browser Sandbox CLI Command Errors Are Appearing Error Messages Are Appearing Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 7 Using the show processes cpu Command Using the show system resources Command Using Onboard Failure Logging Using Diagnostics Using Embedded Event Manager Using Ethanalyzer SNMP and RMON Support Using the PCAP SNMP Parser Using RADIUS Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 8 Contents Using syslog Logging Levels Enabling Logging for Telnet or SSH Using SPAN Using the Blue Beacon Feature Using the watch Command Additional References for Troubleshooting Tools and Methodology Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x viii...

  • Page 9: Preface

    • Documentation Feedback, page x • Obtaining Documentation and Submitting a Service Request, page xi Audience This publication is for network administrators who install, configure, and maintain Cisco Nexus switches. Document Conventions Command descriptions use the following conventions: Convention Description...

  • Page 10: Related Documentation For Cisco Nexus 9000 Series Switches

    An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line. Related Documentation for Cisco Nexus 9000 Series Switches The entire Cisco Nexus 9000 Series switch documentation set is available at the following URL: http://www.cisco.com/en/US/products/ps13386/tsd_products_support_series_home.html Documentation Feedback To provide technical feedback on this document, or to report an error or omission, please send your comments to nexus9k-docfeedback@cisco.com.

  • Page 11: Obtaining Documentation And Submitting A Service Request

    Subscribe to What's New in Cisco Product Documentation, which lists all new and revised Cisco technical documentation as an RSS feed and delivers content directly to your desktop using a reader application. The RSS feeds are a free service.
  • Page 12 Preface Obtaining Documentation and Submitting a Service Request Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 13: New And Changed Information

    New and Changed Information, page 1 New and Changed Information This table summarizes the new and changed features for the Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x and tells you where they are documented. Table 1: New and Changed Features for Cisco NX-OS Release 7.x...
  • Page 14 New and Changed Information New and Changed Information Feature Description Changed in Where Documented Release Traceroute Removed source interface 7.0(3)I1(1) support, which is supported in Cisco NX-OS Release 6.1(2)I3(3). Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 15: Overview

    To troubleshoot your network, follow these general guidelines: • Maintain a consistent Cisco NX-OS release across all your devices. • See the Cisco NX-OS release notes for you Cisco NX-OS release for the latest features, limitations, and caveats. • Enable system message logging.

  • Page 16: Verifying Ports

    • Use the show port-channel compatibility-parameters command to verify that all of the ports in a port channel are configured the same for the speed, the duplex, and the trunk mode. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 17: Verifying Layer 3 Connectivity

    • Diagnose and correct Layer 3 issues. • Recover from switch upgrade failures. • Obtain core dumps and other diagnostic data for use by Cisco TAC or your customer support representative. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 18: System Messages

    They could be logged to different files or e-mailed to a particular user. Specifying a severity level on the syslog server determines that all messages of that level and greater severity (lower number) will be acted upon as you configure the syslog server. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 19 -ef |grep syslogd Test the syslog server by creating an event in Cisco NX-OS. In this case, port e1/2 was shut down and reenabled, and the following was listed on the syslog server. The IP address of the device is listed in brackets.

  • Page 20: Troubleshooting With Logs

    Troubleshooting with Logs Cisco NX-OS generates many types of system messages on the device and sends them to a syslog server. You can view these messages to determine what events might have led up to the current condition that you are facing.

  • Page 21: Contacting Customer Support

    • Brief explanation of the steps that you have already taken to isolate and resolve the problem For more information on steps to take before calling Technical Support, see Before Contacting Technical Support, on page Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 22 Overview Contacting Customer Support Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 23: Troubleshooting Installations, Upgrades, And Reboots

    This publication uses the term upgrade to refer to both Cisco NX-OS upgrades and downgrades. Note Upgrade and Reboot Checklist...

  • Page 24: Verifying Software Upgrades

    Verifying a Nondisruptive Upgrade When you initiate a nondisruptive upgrade, Cisco NX-OS notifies all services that an upgrade is about to start and determines whether the upgrade can proceed. If a service cannot allow the upgrade to proceed, the service aborts the upgrade, and you are prompted to enter the show install all failure-reason command to determine the reason why the upgrade cannot proceed.

  • Page 25: Troubleshooting Software Upgrades And Downgrades

    Upgrading the Cisco NX-OS Software You can perform an automated software upgrade on any system from the CLI. The image filename begins with "nxos" [beginning with Cisco NX-OS Release 7.0(3)I2(1)] or "n9000" (for example, nxos.7.0.3.I2.1.bin or n9000-dk9.7.0.3.I1.1.bin). Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 26: Troubleshooting Software System Reboots

    "Loading Boot Loader" message to interrupt the boot process at the >loader prompt. Boot parameters are Verify and correct the boot parameters and reboot. incorrect. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 27: Corrupted Bootflash Recovery

    The image is made available through bootflash. When the memory test is over, press Esc to enter the boot loader prompt. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 28: Recovery From The Loader> Prompt

    If you boot over TFTP from the loader> prompt, you must supply the full path to the image on the remote server. SUMMARY STEPS 1. loader> set ip ip-address 2. loader> set gw gw-address 3. loader> cmdline recoverymode=1 4. loader> boot tftp: tftp-path 5. switch(boot)# init system 6. switch(boot)# load-nxos Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 29 Address: 172.21.55.213 Netmask: 255.255.255.224 Server: 0.0.0.0 Gateway: 172.21.55.193 This example shows how to configure the IP address of the default gateway: loader> set gw 172.21.55.193 Correct gateway addr 172.21.55.193 Address: 172.21.55.213 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 30: System Or Process Restarts

    See Recovering System Restarts, on page 19 to determine the cause. A clock module failed. Verify that a clock module failed. Replace the failed clock module during the next maintenance window. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 31: Recovering System Restarts

    4. switch# show process log pid pid 5. switch# show system uptime 6. switch# show cores 7. switch# copy core: core path 8. switch# show processes log pid pid 9. switch# system cores tftp: tftp-path Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 32 • R = runnable (on run queue) kupdated kjournald • S = sleeping kjournald kjournald • T = traced or stopped 2abe333e httpd 2abfd33e xinetd 2ac1e33e sysmgr • Z = defunct (zombie) process 2abe91a2 httpd 2abe91a2 httpd Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 33 0x7FFFFCBC: 7FFFFD1C 0804C470 00000000 7FFFFCE8 ..p... 0x7FFFFCCC: 2AB4F7E9 2AAC1F00 00000001 08048A2C ...*...*..,... PID: 898 SAP: 0 UUID: 0 switch# Step 5 switch# show system uptime Displays if the restart recently occurred. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 34 ESI 2AD701A8 EDI 08109308 EBP 7FFFF2EC EAX 00000000 XDS 0000002B XES 0000002B EAX 00000025 (orig) EIP 2AC8CC71 XCS 00000023 EFL 00000207 ESP 7FFFF2C0 XSS 0000002B Stack: 2608 bytes. ESP 7FFFF2C0, TOP 7FFFFCF0 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 35: Unrecoverable System Restarts

    27 ----- reset reason for Supervisor-module 27 (from Supervisor in slot 27) --- 1) At 281000 usecs after Wed Jun 26 20:16:34 2013 Reason: Reset Requested by CLI command reload Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 36: Standby Supervisor Fails To Boot

    You can recover the network administrator password using one of these methods: • From the CLI with a username that has network-admin privileges • By power cycling the device • By reloading the device Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 37: Using The Cli With Network-Admin Privileges To Recover The Administrator Password

    Power Cycling the Device to Recover the Administrator Password If you cannot start a session on the device that has network-admin privileges, you can recover the network administrator password by power cycling the device. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 38 U.S. keymap. You can enter Ctrl-C instead of Ctrl-] (right square bracket) due to keyboard mapping. Step 2 If you use SSH or a terminal emulator to access the console — port, go to Step Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 39 Example: the Telnet prompt does not appear, go to Step telnet> set escape ^\ Escape Character is 'CTRL+\' Step 5 Press Enter one or more times to return to the Cisco NX-OS — login prompt. Example: telnet> <Enter> switch login: Step 6 Power cycle the device.
  • Page 40 Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php switch(boot)# Step 10 Press Enter one or more times to return to the Cisco NX-OS — login prompt. Example: telnet> <Enter> switch login:...
  • Page 41 Step 17 switch(config)# username admin password new-password Resets the new password to ensure that it is also the Simple Network Management Protocol (SNMP) password. Example: switch(config)# username admin password egBdf Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 42: Reloading The Device To Recover The Administrator Password

    2. switch# reload 3. loader> boot n9000-dk9.x.x.x.bin 4. Reset the network administrator password by following Steps 6 through 20 in Power Cycling the Device to Recover the Administrator Password, on page Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 43 A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php switch(boot)# Step 4 Reset the network administrator password by following Steps 6 through 20 in — Power Cycling the Device to Recover the Administrator Password, on page Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 44 Troubleshooting Installations, Upgrades, and Reboots Recovering the Administrator Password Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 45: Troubleshooting Licensing Issues

    Licensing Installation Issues, page 35 About Troubleshooting Licensing Issues Cisco NX-OS requires licenses for select features. The licenses enable those features on your system. You must purchase a license for each system on which you want to enable the licensed features.

  • Page 46: Initial Troubleshooting Checklist For Licensing

    ID for the license. Use the usage keyword to display a list of all licensed features or a list of features in a specified Example: package. switch# show license usage LAN_ENTERPRISE_SERVICES_PKG Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 47: Licensing Installation Issues

    [chars]. Explanation: The feature has a license with an invalid license Host ID. This can happen if a supervisor module with licensed features for one system is installed on another system. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 48: Rma Chassis Errors Or License Transfers Between Systems

    The supervisor module was replaced after Corrupted Bootflash Recovery, on missing. the license was installed. page 15 to recover from the corrupted bootflash:. Reinstall the license. The supervisor bootflash: is corrupted. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 49: Troubleshooting Ports

    This status cannot be changed and is read-only. Some values may not be valid when the interface is down (such as the operation speed). For a complete description of port modes, administrative states, and operational states, see the Cisco Nexus 9000 Series NX-OS Interfaces Configuration Guide.

  • Page 50: Initial Port Troubleshooting Checklist

    Verify that you have enabled the port by using the no shutdown command. Use the show interface command to verify the state of the interface. See the Cisco Nexus 9000 Series NX-OS Interfaces Configuration Guide for reasons why a port might be in a down operational state.

  • Page 51: Troubleshooting Port Statistics From The Cli

    0 Rx pause 0 Tx pause 0 reset Receive data field Size is 2112 Troubleshooting Port-Interface Issues The Interface Configuration Has Disappeared You may have a problem where your interface configuration disappears. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 52: You Cannot Enable An Interface

    Replace with a Cisco-supported SFP. You Cannot Configure a Dedicated Port You may have a problem when trying to configure a port as dedicated. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 53: A Port Remains In A Link Failure Or Not Connected State

    An Unexpected Link Flapping Occurs When a port is flapping, it cycles through the following states, in this order, and then starts over again: Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 54: A Port Is In The Errdisabled State

    CLI, on page 43 to verify the SFP, The device detected a high amount of cable, and connections. bad frames (CRC errors), which might indicate a problem with the media. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 55: Verifying The Errdisable State Using The Cli

    4 06:54:40 switch %PORT_CHANNEL-5-PORT_ADDED: e1/8 added to port-channel 7 4 06:54:56 switch %PORT-5-IF_DOWN_ADMIN_DOWN: Interface e1/7 is down (Admnistratively down) 4 06:54:59 switch %PORT_CHANNEL-3-COMPAT_CHECK_FAILURE: speed is not compatible Jan 4 06:55:56 switch%PORT_CHANNEL-5-PORT_ADDED: e1/7 added to port-channel 7 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 56 Troubleshooting Ports Verifying the ErrDisable State Using the CLI Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 57: Troubleshooting Vpcs

    Verify that both the source and destination IP addresses used for the peer-keepalive messages are reachable from the VRF associated with the vPC peer-keepalive link. Verify that the peer-keepalive link is up. Otherwise, the vPC peer link will not come up. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 58: Verifying Vpcs Using The Cli

    Verifies that the following STP parameters are identical when STP is enabled: • BPDU filter • BPDU guard • Cost • Link type • Priority • VLANs (PVRST+) Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 59: Received Type 1 Configuration Element Mismatch

    Type 1 : vPC will be suspended in case of mismatch Name Type Local Value Peer Value ------------- ---- ---------------------- ----------------------- STP Mode Rapid-PVST Rapid-PVST STP Disabled None None STP MST Region Name "" "" STP MST Region Revision Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 60: Cannot Enable The Vpc Feature

    If a BA dispute is detected, the entire vPC will be in the blocking state. VLANs on a vPC Moved to Suspend State VLANs on a vPC might move to the suspend state. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 61: Hosts With An Hsrp Gateway Cannot Access Beyond Their Vlan

    The peer gateway can loop prevention mechanism in the vPC. be a workaround for this scenario. Read the configuration guide for more information about the peer gateway before you implement it. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 62 Troubleshooting vPCs Hosts with an HSRP Gateway Cannot Access Beyond Their VLAN Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 63: Troubleshooting Vlans

    VLANs. • If you do not map the secondary VLAN to the Layer 3 VLAN interface of the primary VLAN, you can have different VACLs for primary and secondary VLANs. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 64: Initial Troubleshooting Vlans Checklist

    • A PVLAN host or promiscuous port cannot be a SPAN destination port. • TFTP download to Cisco Nexus 9000 Series switches is not supported when the transfer is done when you are using In-band Management, for example, VLAN SVI that is in default or custom VRF. The TFTP transfer times out and fails.

  • Page 65: Troubleshooting Vlan Issues

    The PVLAN feature is not enabled. Use the feature private-vlan command to PVLAN. enable the PVLAN feature. The VLAN Interface is Down You might have a problem when configuring VLAN interfaces. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 66 VLAN exists. Use the vlan command to create the VLAN. The interface is in the wrong VRF. Use the show vrf interface command to determine the interface to which the VLAN interface is assigned. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 67: Troubleshooting Stp

    STP provides a loop-free network at the Layer 2 level. Layer 2 LAN ports send and receive STP frames at regular intervals. Network devices do not forward these frames but use the frames to construct a loop-free path. For more information on Layer 2, see the Cisco Nexus 9000 Series Layer 2 Configuration Guide. Initial Troubleshooting STP Checklist Troubleshooting an STP problem involves gathering information about the configuration and connectivity of individual devices and the entire network.

  • Page 68: Troubleshooting Stp Data Loops

    Use the show spanning-tree summary totals command to verify that the total number of logical interfaces in the Active state are less than the maximum allowed. For information on these limits, see the Cisco Nexus 9000 Series NX-OS Layer 2 Switching Configuration Guide.
  • Page 69 Port 385 (Ethernet3/1) of VLAN0001 is root forwarding Port path cost 4, Port priority 128, Port Identifier 128.385 Designated root has priority 32769, address 0018.bad7.db15 Designated bridge has priority 32769, address 0018.bad7.db15 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 70 This example shows how to check the hardware packet statistic counters for a possible BPDU error drop: switch# show interface counters errors -------------------------------------------------------------- Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards -------------------------------------------------------------- mgmt0 Eth1/1 Eth1/2 Eth1/3 Eth1/4 Eth1/5 Eth1/6 Eth1/7 Eth1/8 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 71: Troubleshooting Excessive Packet Flooding

    Root port is 385 (Ethernet3/1), cost of root path is 4 Topology change flag not set, detected flag not set Number of topology changes 8 last change occurred 1:32:11 ago '' from Ethernet3/1'' Times: hold 1, topology change 35, notification 2 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 72: Troubleshooting Convergence Time Issues

    • Software defects that affect STP. Securing the Network Against Forwarding Loops To handle the inability of STP to deal correctly with certain failures, Cisco has developed a number of features and enhancements to protect the networks against forwarding loops.
  • Page 73 You must configure the STP root and backup STP root so that convergence occurs in a predictable way and builds optimal topology in every scenario. Do not leave the STP priority at the default value. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 74 Troubleshooting STP Securing the Network Against Forwarding Loops Command or Action Purpose Example: switch(config)# spanning-tree vlan 12 root secondary Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 75: Chapter 9 Troubleshooting Routing

    Cisco NX-OS supports multiple virtual routing and forwarding (VRF) instances and multiple routing information bases (RIBs) to support multiple address domains. Each VRF is associated with a RIB, and this information is collected by the Forwarding Information Base (FIB).

  • Page 76: Troubleshooting Routing

    % invalid command detected at '^' marker. Step 2 switch# show running-config eigrp all Verifies the configuration for this routing protocol. Example: switch# show running-config eigrp all Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 77 Verifies that the RIB is interacting with the forwarding plane. Example: switch# show forwarding distribution multicast client Number of Clients Registered: 3 Client-name Client-id Shared Memory Name Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 78 Messages sent: CLIENT: eigrp-99 index mask: 0x00002000 epid: 3148 MTS SAP: 63775 MRU cache hits/misses: Routing Instances: VRF: default table: base notifiers: self Messages received: Register Delete-all-routes : 1 Messages sent: Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 79: Troubleshooting Policy-Based Routing

    • Make sure the policy is active in the system (using the show ip policy command). • Check the statistics for the entry (using the show route-map map-name pbr-statistics command). Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 80 Troubleshooting Routing Troubleshooting Policy-Based Routing Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 81: C H A P T E

    An example of kernel memory usage is when an inband port driver allocates memory to receive packets. • User processes—This memory is used by Cisco NX-OS or Linux processes that are not integrated in the kernel (such as text, stack, heap, and so on).

  • Page 82: General/High Level Assessment Of Platform Memory Utilization

    If page cache and kernel issues have been ruled out, utilization might be high as a result of some user processes taking up too much memory or a high number of running processes (due to the number of features enabled). Cisco NX-OS defines memory limits for most processes (rlimit). If this rlimit is exceeded, sysmgr will Note crash the process, and a core file is usually generated.

  • Page 83: Determining Which Process Is Using A Lot Of Memory

    Built-in Platform Memory Monitoring Cisco NX-OS has built-in kernel monitoring of memory usage to help avoid system hangs, process crashes, and other undesirable behavior. The platform manager periodically checks the memory utilization (relative to the total RAM present) and automatically generates an alert event if the utilization passes the configured threshold values.
  • Page 84 Troubleshooting Memory Memory Thresholds Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 85: Troubleshooting Packet Flow Issues

    Use the show policy-map interface control-plane command to determine if packets are being dropped because of CoPP. switch# show policy-map interface control-plane class-map copp-system-p-class-exception (match-any) match exception ip option match exception ip icmp unreachable match exception ttl-failure Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 86 1 police cir 200 pps , bc 32 packets module 27 : transmitted 0 packets; dropped 0 packets; module 28 : transmitted 0 packets; dropped 0 packets; Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 87: C H A P T E

    2013 Oct 29 22:25:09 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script is being downloaded from [copy tftp://172.23.40.6//pxelinux.0 bootflash:scripts/script.sh vrf management ] 2013 Oct 29 22:25:10 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP boot file download failed. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 88 CLI : show file volatile:poap.cfg md5sum INFO: md5sum 46684d8f8b7c5ffac3b37ac8560928e5 (recalculated) CLI : config terminal ; boot nxos bootflash:poap/system.img CLI : copy running-config startup-config CLI : copy volatile:poap.cfg scheduled-config INFO: Configuration successful Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 89: Troubleshooting The Python Api

    Troubleshooting the Python API This chapter contains the following sections: • Receiving Python API Errors, page 77 Receiving Python API Errors Take these actions if any of the following Python API errors appear: Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 90 >>> import os >>> os.system('whoami') test The urllib2 or socket connection Make sure you is not processed. are using the correct virtual routing context. If not, switch to the correct one. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 91 1214, in http_open return self.do_open(httplib.HTTPConnection, req) File "/isan/python/python2.7/urllib2.py", line 1184, in do_open raise URLError(err) urllib2.URLError: <urlopen error [Errno 113] No route to host> >>> from cisco.vrf import * >>> VRF.get_vrf_name_by_id(get_global_vrf()) 'default' Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 92 Troubleshooting the Python API Receiving Python API Errors Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 93: C H A P T E

    • Check for a long running command. Currently NX-API runs on a single worker process and is single threaded. If one command takes a long time to complete, it will block other commands. NX-API caches the request. When the current request completes, the others will be served. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 94: Configuration Fails

    • Make sure that Bash is enabled by using the show feature | grep bash command. • Make sure that the current user has the correct privileges to access Bash. • For more information on Bash, see the Cisco Nexus 9000 Series NX-OS Programmability Guide. Output Cannot Be Retrieved from the Browser Sandbox Take these actions if you cannot retrieve the output from the browser sandbox: •...

  • Page 95: Temporary Files Are Disappearing

    Temporary Files Are Disappearing • If the Bash commands do not go through, make sure that Bash is enabled by using the show feature | grep bash command. For more information on Bash, see the Cisco Nexus 9000 Series NX-OS Programmability Guide.
  • Page 96 Troubleshooting NX-API Chunks of the Command Output Are Not Being Delivered Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 97: C H A P T E

    ---- -------- -------- ------- ----------------- ----------- 159744 2027520 ff808d30/ffffffff init kthreadd migration/0 ksoftirqd/0 watchdog/0 migration/1 ksoftirqd/1 watchdog/1 migration/2 ksoftirqd/2 watchdog/2 migration/3 ksoftirqd/3 watchdog/3 migration/4 ksoftirqd/4 watchdog/4 migration/5 ksoftirqd/5 watchdog/5 migration/6 ksoftirqd/6 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 98: Identifying Cpu Utilization For Processes

    1334683 0.00% 0.00% 0.00% migration/6 1084448 2520990 0.00% 0.00% 0.00% ksoftirqd/6 0.00% 0.00% 0.00% watchdog/6 46490 1306203 0.00% 0.00% 0.00% migration/7 1187547 2867126 0.00% 0.00% 0.00% ksoftirqd/7 0.00% 0.00% 0.00% watchdog/7 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 99: Monitoring Process Core Files

    Log-create-time ---------------- ------ ----------- ----------- ------- --------------- Jun 27 04:08 snsm Jun 24 20:50 Clearing the Core You can clear the core by using the clear cores command. switch# clear cores Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 100: Enabling Auto-Copy For Core Files

    Enabling Auto-Copy for Core Files You can enter the system cores command to enable the automatic copy of core files to a TFTP server, the flash drive, or a file. switch(config)# system cores tftp://10.1.1.1/cores Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 101: C H A P T E

    Using Core Dumps, page 92 Steps to Perform Before Calling TAC At some point, you might need to contact your technical support representative or Cisco TAC for some additional assistance. This section outlines the steps that you should perform before you contact your next level of support in order to reduce the amount of time spent resolving the issue.
  • Page 102 4 Answer the following questions before you contact your technical support representative: • On which device or port is the problem occurring? • Which Cisco NX-OS software, driver versions, operating systems versions, and storage device firmware are in your network? •...

  • Page 103: Copying Files To Or From Cisco Nx-Os

    Cisco NX-OS offers protocols to use for copying to or from the device. The device always acts as a client, so that an FTP, SCP, or TFTP session always originates from Cisco NX-OS and either pushes files to an external system or pulls files from an external system.

  • Page 104: Using Core Dumps

    Use the system cores command to set up core dumps on your system as follows: switch# system cores tftp://10.91.51.200/jsmith_cores switch# show system cores Cores are transferred to tftp://10.91.51.200/jsmith_cores The filename (indicated by jsmith_cores) must exist in the TFTP server directory. Note Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 105: Troubleshooting Tools And Methodology

    Additional References for Troubleshooting Tools and Methodology, page 109 Command-Line Interface Troubleshooting Commands The command-line interface (CLI) allows you to configure and monitor Cisco NX-OS using a local console or remotely using a Telnet or Secure Shell (SSH) session. The CLI provides a command structure similar to Cisco IOS software, with context-sensitive help, show commands, multiuser support, and roles-based access control.

  • Page 106: Consistency Checker Commands

    Consistency Checker Commands Cisco NX-OS provides consistency checker commands to validate the software state with the hardware state. The result of the consistency checker is logged as either PASSED or FAILED. 2013 Nov 1 16:31:39 switch vshd: CC_LINK_STATE:...

  • Page 107: Configuration Files

    Cisco Nexus 9000 Series NX-OS System Management Configuration Guide. Cisco NX-OS features can create internal locks on the startup configuration file. In rare cases, these locks might not be removed by the features. Use the system startup-config unlock command to remove these locks.

  • Page 108: Debug Filters

    We do not recommend using the Ping utility to test network performance with the IP address configured Note on the system. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 109: Using Traceroute

    Specifies the source IPv6 address of the traceroute6 {dest-ipv6-addr | hostname | interface}] [vrf packets from the specified IP address, hostname, or interface. vrf-name] Example: switch# traceroute6 2010:11:22:0:1000::1 source ethernet 2/2 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 110: Monitoring Processes And Cpus

    • TTY = terminal that controls the process. A - (hyphen) usually means a daemon not running on any particular TTY. • Process = name of the process. Process states are as follows: • D = uninterruptible sleep (usually I/O). • R = runnable (on run queue). Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 111: Using The Show Processes Cpu Command

    • 1Sec = Percentage of CPU utilization for the last 1 second. switch# show processes cpu Runtime(ms) Invoked uSecs 1Sec Process ----- ----------- -------- ----- ----- ----------- 2264 108252 init 211341 migration/0 1154 32833341 ksoftirqd/0 419568 desched/0 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 112: Using The Show System Resources Command

    Current memory status: OK Using Onboard Failure Logging Cisco NX-OS provides the facility to log failure data to the persistent storage, which can be retrieved and displayed for analysis. This onboard failure logging (OBFL) feature stores failure and environmental information in nonvolatile memory on the module.

  • Page 113: Using Diagnostics

    • OBFL specific history information • ASIC interrupt and error statistics history • ASIC register dumps For more information about configuring OBFL, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide. Using Diagnostics Generic online diagnostics (GOLD) define a common framework for diagnostic operations across Cisco platforms.

  • Page 114: Using Embedded Event Manager

    The script can generate actions, including, but not limited to, generating custom syslog or SNMP traps, invoking CLI commands, forcing a failover, and much more. For more information about configuring EEM, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide. Using Ethanalyzer Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark (formerly Ethereal) open source code.
  • Page 115 Protocol: UDP (0x11) Header checksum: 0x46db [correct] [Good: True] [Bad : False] Source: 172.28.230.3 (172.28.230.3) Destination: 224.0.0.2 (224.0.0.2) User Datagram Protocol, Src Port: 1985 (1985), Dst Port: 1985 (1985) Source port: 1985 (1985) Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 116: Snmp And Rmon Support

    The types of events that are supported include logging, SNMP traps, and log-and-trap. For more information about configuring SNMP and RMON, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide.
  • Page 117 Total SET: 0 Total TRAP: 0 Total INFORM: 0 Hosts GETNEXT WALK(NEXT) GETBULK BULKWALK(BULK) SET TRAP INFORM RESPONSE ------------------------------------------------------------------------------------------ 10.22.27.244 1(18) 0(0) Sessions -------- MIB Objects GET GETNEXT WALK(NEXT) GETBULK(Non_rep/Max_rep) BULKWALK(BULK, Non_rep/Max_rep) --------------------------------------------------------------------------------------------- Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 118: Using Radius

    Authentication refers to the authentication of users for access to a specific device. You can use RADIUS to manage user accounts for access to a Cisco NX-OS device. When you try to log into a device, Cisco NX-OS validates you with information from a central RADIUS server.

  • Page 119: Logging Levels

    Telnet or SSH session is enabled or disabled, that state is applied only to that session. The state is not preserved after the user exits the session. The no logging console command disables console logging and is enabled by default. switch(config)# no logging console Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...

  • Page 120: Using Span

    Flashes one of the power supply LEDs. Using the watch Command The watch command allows you to refresh and monitor Cisco NX-OS CLI command output or Unix command output (through the run bash command command). Use the command as follows:...

  • Page 121: Additional References For Troubleshooting Tools And Methodology

    Methodology Related Documents Related Topic Document Title System management tools Cisco Nexus 9000 Series NX-OS System Management Configuration Guide MIBs Cisco Nexus 7000 Series and 9000 Series NX-OS MIB Quick Reference Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 122 Troubleshooting Tools and Methodology Additional References for Troubleshooting Tools and Methodology Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x...
  • Page 123 {inband | mgmt} no feature nxapi ethanalyzer local interface {inband | mgmt} capture-filter no logging console ethanalyzer local interface {inband | mgmt} display-filter no shutdown 38, 41, 42 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x IN-1...
  • Page 124 64, 65 show tech-support vpc show ip policy show udld show ip process show user-account show ip route 64, 67 show version show ip routing show vlan 4, 53, 54 Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x IN-2...
  • Page 125 25, 29 system cores 88, 92 system cores tftp 19, 23 system memory-thresholds minor system startup-config unlock vlan Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x IN-3...
  • Page 126 Index Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 7.x IN-4...

Table of Contents