Page 1 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) May 16, 2016 Cisco Systems, Inc. www.cisco.com Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at www.cisco.com/go/offices. Text Part Number: OL-31393-01...
Page 2 OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks.
Page 3: Table Of Contents
Cisco Support Communities Contacting Cisco or VMware Customer Support Troubleshooting Tools C H A P T E R Commands Ping Traceroute Monitoring Processes and CPUs Identifying the Running Processes and their States Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 4 Recovering the Network Administrator Password Managing Extension Keys Known Extension Problems and Resolutions Resolving a Plug-In Conflict Finding the Extension Key on the Cisco Nexus 1000V Finding the Extension Key Tied to a Specific DVS Verifying Extension Keys Recreating the Cisco Nexus 1000V Installation...
Page 5 C H A P T E R Troubleshooting L3Sec Ports C H A P T E R Information About Ports Information About Interface Characteristics Information About Interface Counters Information About Link Flapping Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 6 C H A P T E R Information About Layer 2 Ethernet Switching 12-1 Port Model 12-1 Viewing Ports from the VEM 12-2 Viewing Ports from the VSM 12-3 Port Types 12-4 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 7 Common NetFlow Problems 15-2 Debugging a Policy Verification Error 15-3 Debugging Statistics Export 15-3 ACLs 16-1 C H A P T E R Information About Access Control Lists 16-1 ACL Configuration Limits 16-1 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 8 20-1 C H A P T E R Information About DHCP Snooping 20-1 Information About Dynamic ARP Inspection 20-2 Information About IP Source Guard 20-2 Guidelines and Limitations for Troubleshooting 20-2 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) viii OL-31593-01...
Page 9 Problems with VM Traffic 22-10 VEM Troubleshooting Commands 22-11 VEM Log Commands 22-12 Error Messages 22-12 Before Contacting Technical Support 23-1 C H A P T E R Cisco Support Communities 23-1 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 10 25-20 VSI Discovery and Configuration Protocol 26-1 C H A P T E R Information About VDP 26-1 Problems with VDP 26-2 VDP Troubleshooting Commands 26-2 VSM Commands 26-2 VEM Commands 26-4 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 11 C H A P T E R Information About vCenter Plug-in 28-1 Prerequisites for VMware vSphere Web Client 28-1 Generating a Log Bundle 28-2 Ethanalyzer 29-1 C H A P T E R Using Ethanalyzer 29-1 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 12 Contents Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 13: New And Changed Information
Updated the high availability section. 4.2(1)SV2(1.1) High Availability Added a command output for the new show system internal active-active remote accounting logs command and updated the output for the show system redundancy status command. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) xiii OL-31593-01...
Page 14 4.2(1)SV1(4) Upgrades upgrade problems. VEM health check Added information about the VEM health 4.0(4)SV1(3) Checking Network Connectivity check that shows the cause of a Between the VSM and the VEM connectivity problem. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 15 5.2(1)SV3(1.1) L3Sec the internal control plane communications (Control and Packet traffic) of Cisco Nexus 1000V in a more robust way than in previous releases. It operates only in Layer 3 control mode. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 16 New and Changed Information Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 17: Preface
Related Documentation, page xviii • Obtaining Documentation and Submitting a Service Request, page xx • Audience This publication is for experienced network administrators who configure and maintain a Cisco Nexus 1000V. Document Conventions Command descriptions use these conventions: Convention Description boldface font Commands and keywords are in boldface.
Page 18: Related Documentation
Means reader be careful. In this situation, you might do something that could result in equipment Caution damage or loss of data. Related Documentation This section lists the documents used with the Cisco Nexus 1000 and available on Cisco.com at the following URL: http://www.cisco.com/en/US/products/ps9902/tsd_products_support_series_home.html...
Page 19 Cisco Nexus 1000V Password Recovery Guide Cisco NX-OS System Messages Reference Virtual Services Appliance Documentation The Cisco Nexus Virtual Services Appliance (VSA) documentation is available at http://www.cisco.com/en/US/products/ps9902/tsd_products_support_series_home.html Virtual Security Gateway Documentation The Cisco Virtual Security Gateway documentation is available at http://www.cisco.com/en/US/products/ps13095/tsd_products_support_series_home.html...
Page 20: Obtaining Documentation And Submitting A Service Request
What’s New in Cisco Product Documentation at: http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html. Subscribe to What’s New in Cisco Product Documentation, which lists all new and revised Cisco technical documentation, as an RSS feed and deliver content directly to your desktop using a reader application. The RSS feeds are a free service.
Page 21: Chapter 1 Overview
We recommend that you do the following to ensure the proper operation of your networks: Maintain a consistent Cisco Nexus 1000V release across all network devices. • Refer to the release notes for your Cisco Nexus 1000V release for the latest features, limitations, • and caveats.
Page 22: Troubleshooting Basics
• Troubleshooting Basics This section introduces questions to ask when troubleshooting a problem with the Cisco Nexus 1000V or connected devices. Use the answers to these questions to identify the scope of the problem and to plan a course of action.
Page 23: Verifying Ports
Use the show interface-brief command to check the status of a virtual Ethernet port or a physical Ethernet port. Verifying Layer 3 Connectivity Answer the following questions to verify Layer 3 connectivity: Have you configured a gateway of last resort? • Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 24: Overview Of Symptoms
Those problems and corrective actions include the following: • Identify key Cisco Nexus 1000V troubleshooting tools. Obtain and analyze protocol traces using SPAN or Ethanalyzer on the CLI. • Identify or rule out physical port issues.
Page 25: Syslog Server Implementation
Server Implementation The syslog facility allows the Cisco Nexus 1000V to send a copy of the message log to a host for more permanent storage. This feature can be useful if the logs need to be examined over a long period of time or when the Cisco Nexus 1000V is not accessible.
Page 26: Troubleshooting With Logs
# ps -ef |grep syslogd root 23508 1 0 11:01:41 ? 0:00 /usr/sbin/syslogd Test the syslog server by creating an event in the Cisco Nexus 1000V. In this case, port e1/2 was bounced Step 3 and the following was listed on the syslog server. Notice that the IP address of the switch is listed in brackets.
Page 27: Cisco Support Communities
Brief explanation of the steps that you have already taken to isolate and resolve the problem • If you purchased the Cisco Nexus 1000V and support contract from Cisco, contact Cisco for Cisco Nexus 1000V support. Cisco provides Layer 1, Layer 2, and Layer 3 support.
Page 28 Chapter 1 Overview Contacting Cisco or VMware Customer Support Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 29: Commands
C H A P T E R Troubleshooting Tools This chapter describes the troubleshooting tools available for the Cisco Nexus 1000V and includes the following topics: Commands, page 2-1 • Ping, page 2-1 • Traceroute, page 2-2 • Monitoring Processes and CPUs, page 2-2 •...
Page 30: Traceroute
TTY—Terminal that controls the process. A “-” usually means a daemon is not running on any • particular TTY. Process—Name of the process. • Process states are as follows: D—Uninterruptible sleep (usually I/O). • R—Runnable (on run queue). • S—Sleeping. • • T—Traced or stopped. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 31: Displaying Cpu Utilization
CPU time in average for each process invocation. • 1Sec—CPU utilization in percentage for the last one second. Example 2-2 show processes cpu Command switch# show processes cpu Runtime(ms) Invoked uSecs 1Sec Process ----- ----------- -------- ----- ----- ----------- Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 32: Displaying Cpu And Memory Information
RADIUS is a protocol used for the exchange of attributes or credentials between a head-end RADIUS server and a client device. These attributes relate to three classes of services: Authentication • Authorization • Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 33: Syslog
Authentication refers to the authentication of users for access to a specific device. You can use RADIUS to manage user accounts for access to a Cisco Nexus 1000V. When you try to log into a device, the Cisco Nexus 1000V validates you with information from a central RADIUS server.
Page 34 Enables logging for Telnet or SSH • • Disabled by default Example 2-5 terminal monitor Command switch# terminal monitor For more information about configuring syslog, see the Cisco Nexus 1000V System Management Configuration Guide. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 35: Chapter 3 Installation
You are logged in to the vSphere client on the ESX server. • You are logged in to the Cisco Nexus 1000V CLI in EXEC mode. • This procedure verifies that your vSphere ESX server uses the VMware Enterprise Plus license. This license includes the Distributed Virtual Switch feature, which allows visibility to the Cisco Nexus 1000V.
Page 36: Host Is Not Visible From The Distributed Virtual Switch
If your vSphere ESX server does not have an Enterprise Plus license, you must upgrade your • VMware License to an Enterprise Plus license to have visibility to the Cisco Nexus 1000V. Host is Not Visible from the Distributed Virtual Switch...
Page 37: Refreshing The Vcenter Server Connection
Host is Not Visible from the Distributed Virtual Switch Refreshing the vCenter Server Connection You can refresh the connection between the Cisco Nexus 1000V and vCenter Server. From the Cisco Nexus 1000V Connection Configuration mode on the Virtual Supervisor Module (VSM), Step 1 enter the following command sequence:...
Page 38: Improving Performance On The Esx And Vm
Layer 2 domain. Following the installation of the Cisco Nexus 1000V, make certain that you configure a domain ID. Without a domain ID, the VSM cannot connect to the vCenter Server. Follow these guidelines: The domain ID should be a value within the range of 1 to 4095.
Page 39: Verifying Vsm And Vcenter Server Connectivity
Ensure that the VMware VirtualCenter Server service is running. Step 4 Troubleshooting Connections to vCenter Server You can troubleshoot connections between a Cisco Nexus 1000V VSM and a vCenter Server. In a web browser, enter the path: http://<VSM-IP> Step 1 Download the cisco_nexus_1000v_extension.xml file to your desktop.
Page 40: Recovering The Network Administrator Password
The actual value of “Cisco_Nexus_1000V_584325821” will vary. It should match the extension key Note from the cisco_nexus_1000v_extension.xml file. Recovering the Network Administrator Password For information about recovering the network administrator password, see the Cisco Nexus 1000V Password Recovery Guide. Managing Extension Keys This section includes the following topics: Known Extension Problems and Resolutions, page 3-7 •...
Page 41: Known Extension Problems And Resolutions
“Unregistering the Extension Key in the vCenter • Server” section on page 3-12. DETAILED STEPS From the Cisco Nexus 1000V for the VSM whose extension key you want to view, enter the following Step 1 command: show vmware vc extension-key Example: switch# show vmware vc extension-key Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
Page 42: Finding The Extension Key Tied To A Specific Dvs
The Summary tab opens with the extension key displayed in the Notes section of the Annotations block. Verifying Extension Keys You can verify that the Cisco Nexus 1000V and vCenter Server are using the same extension key. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
Page 43 Chapter 3 Installation Managing Extension Keys DETAILED STEPS Find the extension key used on the Cisco Nexus 1000V using the “Finding the Extension Key on the Step 1 Cisco Nexus 1000V” section on page 3-7. Find the extension key used on the vCenter Server using the “Finding the Extension Key Tied to a...
Page 44: Recreating The Cisco Nexus 1000V Installation
Recreating the Cisco Nexus 1000V Installation Recreating the Cisco Nexus 1000V Installation You can re-create the complete Cisco Nexus 1000V configuration in the event of a persistent problem that cannot be resolved using any other workaround. Flowchart: Re-creating the Cisco Nexus 1000V Installation...
Page 45: Removing Hosts From The Cisco Nexus 1000V Dvs
Log in to the VSM CLI in EXEC mode. • DETAILED STEPS Step 1 From the Cisco Nexus 1000V VSM, use the following commands to remove the DVS from the vCenter Server. config t svs connection vc no vmware dvs...
Page 46: Unregistering The Extension Key In The Vcenter Server
“Finding the Extension Key on the Cisco Nexus 1000V” section on page 3-7. After unregistering the extension key in vCenter Server, you can start a new installation of the Cisco • Nexus 1000V VSM software. DETAILED STEPS...
Page 47 “Finding the Extension Key on the Cisco Nexus 1000V” section on page 3-7, and then click Invoke Method. The extension key is unregistered in vCenter Server so that you can start a new installation of the Cisco Nexus 1000V VSM software. Step 4 You have completed this procedure.
Page 48: Problems With The Cisco Nexus 1000V Installation Management Center
Chapter 3 Installation Problems with the Cisco Nexus 1000V Installation Management Center Problems with the Cisco Nexus 1000V Installation Management Center The following are possible problems and their solutions. Symptom Problem Recommended Action Port migration fails. The VSM to VEM migration fails in Check if there is any VM running on the •...
Page 49: Chapter 4 Licenses
• Information About Licenses The name for the Cisco Nexus 1000V license package is NEXUS1000V_LAN_SERVICES_PKG and the version is 3.0. By default, 1024 licenses are installed with the Virtual Supervisor Module (VSM). These default licenses are valid for 60 days. You can purchase permanent licenses that do not expire.
Page 50: Contents Of The License File
4-5. If there is a license file with the same name, rename your new license file to something else. Do not edit the contents of the license file. If you have already done so, contact your Cisco Customer • Support Account Team.
Page 51: Problems With Licenses
1 days! The VEMs' VNICS will be brought down if license is allowed to expire. Please contact your Cisco account team or partner to purchase Licenses. To activate your purchased licenses, click on www.cisco.com/go/license. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 52: License Troubleshooting Commands
Example 4-8 on page 4-7. show license usage [license_name] Displays information about the licenses and where they are used. If displayed for a specific license, indicates VEM and socket information. Example 4-1 on page 4-5. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 53 Ethernet scale. Example 4-7 on page 4-7. For detailed information about show command output, see the Cisco Nexus 1000V Command Reference. EXAMPLES Example 4-1 show license usage license_name Command switch# show license usage NEXUS1000V_LAN_SERVICES_PKG...
Page 54 File Name Feature Name Version Count Expiry ------------------------------------------------------------------------------------------ eval.lic NEXUS1000V_LAN_SERVICES_PKG 1.0 17 3-nov-2014 eval0715.lic NEXUS1000V_LAN_SERVICES_PKG 3.0 17 15-jul-2015 show switch edition (purpose: Displays the switch edition, advanced feature status, license expiry and module and veth scale) Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 55 Licenses are Sticky Mod Socket Count License Usage Count License Version License Status --- ------------ ------------------- --------------- -------------- 103 2 2 3.0 licensed 104 2 2 3.0 licensed Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 56 Chapter 4 Licenses License Troubleshooting Commands Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 57: Chapter 5 Upgrades
Upgrade Troubleshooting Commands, page 5-16 Information About Upgrades The upgrade for the Cisco Nexus 1000V involves upgrading software on both the VSM and the Virtual Ethernet Module (VEM). An in service software upgrade (ISSU) is available for a stateful upgrade of the Cisco Nexus 1000V image(s) running on the VSM.
Page 58 Make sure that the module removal is complete. the upgrade. Restart the software upgrade using the instructions in Pre-Upgrade check failed. Return code 0x4093000A the Cisco Nexus 1000V Installation and Upgrade Guide. (SRG collection failed) Error message: The standby VSM is not Verify the HA synchronization state.
Page 59 When the correct software images are in the bootflash: repository, restart the software upgrade using the instructions in the Cisco Nexus 1000V Installation and Upgrade Guide. Error message: You might have used an Restart the software upgrade using the correct filenames for incorrect filename when the new software images.
Page 60 • Stop the upgrade and restart one session only using the (0x401E0007) instructions in the Cisco Nexus 1000V Installation and Upgrade Guide. The install command fails The standby VSM fails to Do one of the following: with following error boot with the new image.
Page 61: Problems With The Vem Upgrade
• cluster. hosts with the new VEM. availability (HA) Restart the VEM software upgrade using the instructions VMware fault tolerance • in the Cisco Nexus 1000V Installation and Upgrade (FT) Guide. Vmware Distributed • Power Management (DPM) VEM upgrade fails.
Page 62: Problems With The Gui Upgrade
Restart the software upgrade using the instructions in unreachable. The other VSM the Cisco Nexus 1000V Installation and Upgrade Guide. has the original pre-upgrade software version installed and is reachable. The upgrade GUI stops and...
Page 63: Recovering A Secondary Vsm With Active Primary
The same domain ID and password as that of the primary VSM. • For a detailed procedure, see the Cisco Nexus 1000V Installation and Upgrade Guide. The VSM comes up and forms an HA pair with the newly created standalone VSM. The VSMs have the previous version of the software installed.
Page 64 You have completed this procedure. Return to one of these sections: Step 4 “Recovering a Secondary VSM with Active Primary” section on page 5-7 • • “Recovering a Primary VSM with Active Secondary” section on page 5-12 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 65 Copy the running configuration to the startup configuration. Step 4 copy run start Example: switch#(config)# copy run start [########################################] 100%e switch#(config)# Verify the change in the system and kickstart boot variables. Step 5 show boot Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 66 From the vCenter Server left-hand panel, right-click the VSM and then choose Power > Power On. Step 1 The VSM starts. You have completed this procedure. Return to the “Recovering a Primary VSM with Active Secondary” Step 2 section on page 5-12. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-10 OL-31593-01...
Page 67 Example: switch#(config-svs-domain)# copy run start [########################################] 100%e switch#(config-svs-domain)# You have completed this procedure. Return to the “Recovering a Primary VSM with Active Secondary” Step 4 section on page 5-12. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-11 OL-31593-01...
Page 68: Recovering A Primary Vsm With Active Secondary
The host or cluster of the existing secondary VSM. • For detailed installation procedures, see the Cisco Nexus 1000V Installation and Upgrade Guide. Make sure that the port groups between the host server and VSM are not connected when the new VSM...
Page 69 Connected Connect at Power On • The connection from the VSM to the host server through the management port is dropped and is not restored when you power on the VSM. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-13 OL-31593-01...
Page 70 In vCenter Server, select the VSM and then choose Edit > Settings. Step 1 The Virtual Machine Properties dialog box opens. Select the Control port group and check the following Device Settings: Step 2 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-14 OL-31593-01...
Page 71 Select the Management port group and check the following Device Setting: Step 3 Connect at Power On • When you power on the VSM, it will connect to the host server through the management port. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-15 OL-31593-01...
Page 72: Problems With Vsm-Vem Layer 2 To 3 Conversion Tool
Displays the boot variables currently in the startup configuration. Example 5-6 on page 5-18. show svs connections Displays the current connections between the VSM and the VMware host server. Example 5-7 on page 5-18. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-16 OL-31593-01...
Page 73: Upgrade Troubleshooting Commands
Command (VEM and VSM upgraded) switch# show module Ports Module-Type Model Status ----- -------------------------------- ------------------ ------------ Virtual Supervisor Module Nexus1000V ha-standby Virtual Supervisor Module Nexus1000V active * Virtual Ethernet Module --------------- ------ Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-17 OL-31593-01...
Page 74 - switch# Example 5-8 show svs upgrade status Command switch# show svs upgrade status Upgrade State: Start Upgrade mgmt0 ipv4 addr: 1.1.1.1 Upgrade mgmt0 ipv6 addr: Upgrade control0 ipv4 addr: switch# Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-18 OL-31593-01...
Page 75 Upgrade VIBs: System VEM Image Upgrade Status: Upgrade Notification Sent Time: Upgrade Status Time(vCenter): Upgrade Start Time: Upgrade End Time(vCenter): Upgrade Error: Upgrade Bundle ID: VSM: VEM400-201007101-BG DVS: VEM400-201007101-BG switch# Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-19 OL-31593-01...
Page 76 Chapter 5 Upgrades Upgrade Troubleshooting Commands Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 5-20 OL-31593-01...
Page 77: Chapter 6 High Availability
Information About High Availability The purpose of high availability (HA) is to limit the impact of failures—both hardware and software— within a system. The Cisco NX-OS operating system is designed for high availability at the network, system, and service levels.
Page 78: System-Level High Availability
VSM fails. Network-Level High Availability The Cisco Nexus 1000V HA at the network level includes port channels and Link Aggregation Control Protocol (LACP). A port channel bundles physical links into a channel group to create a single logical link that provides the aggregate bandwidth of up to eight physical links.
Page 79 VSMs. interface. Check the output of the • show system internal redundancy info command and verify if the degraded_mode flag is set to true. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 80 MAC addresses Move the identified VSM(s) out of of the VSM(s) that collide the system to stop role collision. with the working VSM. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 81: High Availability Troubleshooting Commands
Example 6-1show cores Command, page 6-6 show processes [pid pid] Example 6-2show processes log [pid pid] Command, page 6-6 show system internal active-active Example 6-7show system internal active-active remote accounting logs Command, page 6-10 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 82 CWD: /var/sysmgr/work To check redundancy status, use the following commands: Example 6-3 show system redundancy status Command switch# show system redundancy status Redundancy role --------------- administrative: primary <-- Configured redundancy role Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 83 6 rx_set_ver_req_pkts: rx_set_ver_rsp_pkts: rx_heartbeat_req_pkts: 6 rx_heartbeat_rsp_pkts: 442546 <-- Counter should be increasing, as this indicates that communication between VSM is working properly. rx_drops_wrong_domain: 0 rx_drops_wrong_slot: rx_drops_short_pkt: rx_drops_queue_full: rx_drops_inactive_cp: rx_drops_bad_src: rx_drops_not_ready: rx_unknown_pkts: Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 84 Configured to use the real platform manager. Configured to use the real redundancy driver. Redundancy register: this_sup = RDN_ST_AC, other_sup = RDN_ST_SB. EOBC device name: eth0. Remote addresses: MTS - 0x00000201/3 IP - 127.1.1.2 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 85 NOTE: Please run the same command on sup-1 to check for conflicting(if any) sup-1(s) in the same domain. If no collisions are detected, the highlighted output is not displayed. Use the following command to display the accounting logs that are stored on a remote VSM. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 86 The standby VSM console is not accessible externally, but can be accessed from the active VSM through the attach module module-number command. switch# attach module 2 This command attaches to the console of the secondary VSM. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 6-10 OL-31593-01...
Page 87: Troubleshooting A Module Not Coming Up On The Vsm
The Cisco Nexus 1000V manages a data center defined by a VirtualCenter. Each server in the data center is represented as a module in the Cisco Nexus 1000V and can be managed as if it were a module in a physical Cisco switch.
Page 88: Guidelines For Troubleshooting Modules
“Checking the vCenter Server Configuration” section on page 7-10. – ERROR: Datacenter not found • For a list of terms used with the Cisco Nexus 1000V, see the Cisco Nexus 1000V Getting Started Guide. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 89: Flowchart For Troubleshooting Modules
Verifying the VSM Is Configured Correctly, page 7-7 Checking the vCenter Server Configuration, page 7-10 Checking Network Connectivity Between the VSM and the VEM, page 7-10 Checking the VEM Configuration, page 7-14 Collecting Logs, page 7-16 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 90: Problems With The Vsm
If the plug-in is not found, create one using the following procedure in the Cisco Nexus 1000V Getting Started Guide: Creating a Cisco Nexus 1000V Plug-In on the vCenter Server Following a reboot of the VSM, the system stops functioning in one of the following states and does not recover on its own.
Page 91 VSM. copy source filesystem: filename system:running-config If not, reconfigure the VSM using the • following section in the Cisco Nexus 1000V Getting Started Guide: Setting Up the Software After boot, VSM is stopped at Corrupt boot menu file.
Page 92: Verifying The Vsm Is Connected To Vcenter Server
“Unregistering the Extension Key in the vCenter Server” section on page 3-12. Install a new extension key using the “Creating a Cisco Nexus 1000V Plug-In on the vCenter • Server” procedure in the Cisco Nexus 1000V Getting Started Guide. Verify the connection between the VSM and vCenter Server.
Page 93: Verifying The Vsm Is Configured Correctly
Example: switch# show svs domain SVS domain config: Domain id: Control vlan: 3002 Packet vlan: 3003 L2/L3 Control VLAN mode: L2 L2/L3 Control VLAN interface: mgmt0 Status: Config push to VC successful Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 94 BEFORE YOU BEGIN Log in to the CLI in EXEC mode. • Check that the output of the show running-config command shows control and packet VLAN ID • numbers among the VLANs configured, Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 95 Card control VLAN: 168 Card packet VLAN: 168 Control type multicast: No Card Headless Mode : No Processors: 16 Processor Cores: 8 Processor Sockets: 2 Kernel Memory: 25102148 Port link-up delay: 5s Global UUFB: DISABLED Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 96: Checking The Vcenter Server Configuration
Checking the vCenter Server Configuration You can verify the configuration on vCenter Server. Confirm that the host is added to the data center and the Cisco Nexus 1000V DVS in that data center. Step 1 Confirm that at least one pnic of the host is added to the DVS, and that pnic is assigned to the Step 2 system-uplink profile.
Page 97 MAC address of the VSM, then there is a problem with connectivity between the server hosting the VSM and the upstream switch. Recheck the VSM configuration and vCenter Server configuration. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-11 OL-31593-01...
Page 98: Recovering Management And Control Connectivity Of A Host When A Vsm Is Running On A Vem
VLAN. If no such uplink exists, it reports this as an error. You need to specify the -p parameter and rerun the script. You can recover management and control connectivity of a host when a VSM is running on a VEM. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-12 OL-31593-01...
Page 99 VLANs in the trunk port profile. If the lists match, all of the expected VLANs are forwarding and the Cisco Nexus 1000V is blocking nonallowed VLANs.
Page 100: Checking The Vem Configuration
~ # vemcmd show port 48 IfIndex Vlan Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name . . . 1a030100 PHYS Trunk vmnic1 ~# vemcmd set mtu 9000 ltl 17 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-14 OL-31593-01...
Page 101 Verify that the vmnic port that is supposed to carry the control VLAN and packet VLAN is present. vemcmd show bd control_vlan vemcmd show bd packet_vlan Example: ~ # vemcmd show bd 3002 BD 3002, vdc 1, vlan 3002, 2 ports Portlist: Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-15 OL-31593-01...
Page 102: Collecting Logs
Card UUID type 0: 4908a717-7d86-d28b-7d69-001a64635d18 Card name: sfish-srvr-7 Switch name: switch Switch uuid: 50 84 06 50 81 36 4c 22-9b 4e c5 3e 1f 67 e5 ff Card domain: 11 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-16 OL-31593-01...
Page 103 13 • show system internal ethpm event-history module 13 • If you need to contact Cisco TAC for assistance in resolving an issue, you will need the output of the Note commands listed in Step VSM and VEM Troubleshooting Commands You can use the commands in this section to troubleshoot problems related to VSM.
Page 104 VM NIC is used. Example 7-16 on page 7-22. vem-connect -i ip_address -v vlan [-pnic Recovers management and control connectivity of vmnicN] a host when a VSM is running on a VEM. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-18 OL-31593-01...
Page 105: Vsm And Vem Troubleshooting Commands
DVS uuid: 92 7a 14 50 05 11 15 9c-1a b0 f2 d4 8a d7 6e 6c config status: Disabled operational status: Disconnected Example 7-3 show svs domain Command switch# show svs domain SVS domain config: Domain id: Control vlan: 3002 Packet vlan: 3003 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-19 OL-31593-01...
Page 106 Command switch# show mac address-table interface Gi3/1 vlan 3002 Legend: * - primary entry age - seconds since last seen n/a - not available vlan mac address type learn ports ------+----------------+--------+-----+----------+-------------------------- Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-20 OL-31593-01...
Page 107 1 Access l20 3969 VIRT 1 Access l21 3002 VIRT 1 Access l22 3968 VIRT 1 Access l23 3003 VIRT 1 Access l24 VIRT 0 Access l25 3967 VIRT 1 Access l26 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-21 OL-31593-01...
Page 108 VLANs. This might be a normal situation depending on the port profile allowed VLAN list. Compare the output of the vemcmd show port vlans command against the port profile trunk allowed VLANs. If the lists match, all of the expected VLANs are forwarding and the Cisco Nexus 1000V is blocking nonallowed VLANs.
Page 109 Chapter 7 VSM and VEM Modules VSM and VEM Troubleshooting Commands ----------- ------------------------------------ -------------- absent 33393935-3234-5553-4538-35314e355400 unlicensed powered-up 33393935-3234-5553-4538-35314e35545a licensed switch# Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-23 OL-31593-01...
Page 110 Chapter 7 VSM and VEM Modules VSM and VEM Troubleshooting Commands Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 7-24 OL-31593-01...
Page 111: Chapter 8 L3Sec
L3Sec This chapter describes how to secure the internal control plane communications (Control and Packet traffic) of Nexus 1000V in a more robust way than in previous releases. It operates only in Layer 3 Control mode. Troubleshooting L3Sec, page 8-1 •...
Page 112 Chapter 8 L3Sec Troubleshooting L3Sec Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 113: Ports
Some values might not be valid when the interface is down (such as the operation speed). For a complete description of port modes, administrative states, and operational states, see the Cisco Nexus 1000V Interface Configuration Guide.
Page 114: Information About Interface Counters
The port security feature allows you to secure a port by limiting and identifying the MAC addresses that can access the port. Secure MAC addresses can be manually configured or dynamically learned. For detailed information about port security, see the Cisco Nexus 1000V Security Configuration Guide. Is Port Security...
Page 115: Port Diagnostic Checklist
Chapter 9 Ports Port Diagnostic Checklist Port Diagnostic Checklist Use the following checklist to diagnose port interface activity. For more information about port states, see the Cisco Nexus 1000V Interface Configuration Guide. Table 9-1 Port Diagnostic Checklist Checklist Example Verify that the module is active.
Page 116: Problems With Ports
Disable and then enable the port. shut no shut Move the connection to a different port on the same module or a different module. Collect the ESX-side NIC configuration. vss-support Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 117: Link Flapping
A control frame is erroneously sent to the device. ESX errors, or link flapping, Use the troubleshooting guidelines in the documentation for your occurs on the upstream switch. ESX or upstream switch. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 118: Port Errdisabled
Open a support case and submit the output of the above commands. For more information see the “Contacting Cisco or VMware Customer Support” section on page 1-7. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 119: Vm Cannot Ping A Secured Port
Log in to the upstream switch and verify that the packet VLAN is allowed on the port. show running-config interface gigabitEthernet slot/port If the packet VLAN is not allowed on the port, add it to the allowed VLAN list. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 120: Port Security Violations
Chapter 9 Ports Problems with Ports Port Security Violations For detailed information about port security, see the Cisco Nexus 1000V Security Configuration Guide. Possible Cause Solution The configured maximum Display the secure addresses. number of secured show port -security address vethernet number...
Page 121: Port State Is Blocked On A Vem
Install the necessary licenses or move the switch to essential mode. svs switch edition essential Port Troubleshooting Commands You can use the commands in this section to troubleshoot problems related to ports. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) OL-31593-01...
Page 122 • Trunk VLAN status • Number of frames sent and received • Transmission errors, including discards, errors, CRCs, and invalid frames Example 9-9 on page 9-13. Example 9-10 on page 9-13. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 9-10 OL-31593-01...
Page 123 -security address interface vethernet Displays information about secure addresses on an interface. Example 9-17 on page 9-16. For detailed information about show command output, see the Cisco Nexus 1000V Command Reference. EXAMPLES Example 9-1 show module Command switch# show mod 3...
Page 124 1 22:43.. Previous state: [PI_FSM_ST_IF_INIT_EVAL] Triggered event: [PI_FSM_EV_IE_ERR_DISABLED_CAP_MISMATCH] Next state: [PI_FSM_ST_IF_DOWN_STATE] Example 9-6 show logging logfile Command switch# show logging logfile . . . 4 06:54:04 switch %PORT_CHANNEL-5-CREATED: port-channel 7 created Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 9-12 OL-31593-01...
Page 125 Auto-Negotiation is turned off Input flow-control is off, output flow-control is off Auto-mdix is turned on Switchport monitor is off 18775 Input Packets 10910 Unicast Packets 862 Multicast Packets 7003 Broadcast Packets 2165184 Bytes Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 9-13 OL-31593-01...
Page 126 Speed: 10,100,1000,auto Duplex: half/full/auto Trunk encap. type: 802.1Q Channel: Broadcast suppression: none Flowcontrol: rx-(none),tx-(none) Rate mode: none QOS scheduling: rx-(none),tx-(none) CoS rewrite: ToS rewrite: SPAN: UDLD: Link Debounce: Link Debounce Time: Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 9-14 OL-31593-01...
Page 127 Trunk encap. type: 802.1Q Channel: Broadcast suppression: percentage(0-100) Flowcontrol: rx-(off/on/desired),tx-(off/on/desired) Rate mode: none QOS scheduling: rx-(none),tx-(none) CoS rewrite: ToS rewrite: SPAN: UDLD: Link Debounce: Link Debounce Time: MDIX: Port Group Members: none control0 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 9-15 OL-31593-01...
Page 128 Vethernet1 Shutdown ========================================================================== Example 9-17 show port-security address interface vethernet Command switch# show port-security address interface vethernet 11 Secure Mac Address Table ---------------------------------------------------------------------- Vlan/Vxlan Mac Address Type Ports Configured Age (mins) Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 9-16 OL-31593-01...
Page 129 Chapter 9 Ports Port Troubleshooting Commands ---------- ----------- ------ ----- --------------- 50 0050.56a4.38ec STATIC Vethernet11 0 50 0000.0000.0011 DYNAMIC Vethernet11 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 9-17 OL-31593-01...
Page 130 Chapter 9 Ports Port Troubleshooting Commands Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 9-18 OL-31593-01...
Page 131: Chapter 10 Port Profiles
For more information about assigning port profiles to physical or virtual ports, see your VMware documentation. To verify that the profiles are assigned as expected to physical or virtual ports, use the following show commands: show port-profile virtual usage • show running-config interface interface-id • Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-1 OL-31593-01...
Page 132: Problems With Port Profiles
To verify port profile inheritance, use the following command: show running-config interface interface-id • Inherited port profiles cannot be changed or removed from an interface from the Cisco Nexus 1000V Note CLI. This action can only be done from vCenter Server.
Page 133 Overrun mapping is created. Fix the error in the port profile using the procedures in the Cisco Nexus 1000V Port Profile Configuration Guide. Bring the interface out of quarantine. no shutdown The interface comes back online.
Page 134: Recovering A Quarantined Offline Interface
Verify the port profile-to-interface mapping. show port-profile virtual usage Step 5 Verify the interface has come out of quarantine automatically. The interface should no longer appear in the show command output. show port-profile sync-status Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-4 OL-31593-01...
Page 135: Port Profile Logs
Displays the port profile role configuration, port-profile-role-name] including role names, descriptions, assigned users, and assigned groups. Example 10-7 on page 10-8. show running-config port-profile Displays the port profile configuration. [profile-name] Example 10-6 on page 10-8. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-5 OL-31593-01...
Page 136 Displays the messages logged about port profile msgs events within the Cisco Nexus 1000V. Example 10-13 on page 10-14. For detailed information about show command output, see the Cisco Nexus 1000V Command Reference. EXAMPLES Example 10-1 show port-profile Command switch# show port-profile port-profile 1...
Page 137 3549 1524 1524 Ethernet DAO-VSM# Vethernet Ethernet switch# Example 10-4 show port-profile expand-interface Command switch# show port-profile expand-interface port-profile 50 Vethernet6 switchport mode access switchport access vlan 50 no shutdown Vethernet27 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-7 OL-31593-01...
Page 138 112 channel-group auto mode on sub-group cdp no shutdown switch# Example 10-7 show port-profile-role Command switch# show port-profile-role name adminUser Name: adminUser Description: adminOnly Users: hdbaar (user) Assigned port-profiles: allaccess2 switch# Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-8 OL-31593-01...
Page 139 Net Adapter 3 bl-h-p switch# Example 10-11 show msp internal info Command switch# show msp internal info port-profile Access484 id: 5 capability: 0x0 state: 0x1 type: 0x1 system vlan mode: - Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-9 OL-31593-01...
Page 140 44 dc 3b 50 53 11 b7 ac-ef ed ef 46 ee df c2 d5 type: 2 port-profile eth-break-deinherit id: 10 capability: 0x1 state: 0x1 type: 0x1 system vlan mode: - system vlans: Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-10 OL-31593-01...
Page 141 44 dc 3b 50 53 11 b7 ac-ef ed ef 46 ee df c2 d5 type: 2 port-profile uplink-quar id: 12 capability: 0x1 state: 0x1 type: 0x1 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-11 OL-31593-01...
Page 142 44 dc 3b 50 53 11 b7 ac-ef ed ef 46 ee df c2 d5 type: 2 pg id: dvportgroup-3297 dvs uuid: 44 dc 3b 50 53 11 b7 ac-ef ed ef 46 ee df c2 d5 type: 2 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-12 OL-31593-01...
Page 143 Triggered event: [PPM_PROFILE_EV_EACKNOWLEDGE] Next state: [FSM_ST_NO_CHANGE] 4) FSM:<PROFILE_FSM:1> Transition at 873872 usecs after Tue Mar 8 19:12:04 2011 Previous state: [PPM_PROFILE_ST_SIF_CREATE] Triggered event: [PPM_PROFILE_EV_ESUCCESS] Next state: [PPM_PROFILE_ST_SIDLE] Curr state: [PPM_PROFILE_ST_SIDLE] switch# Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-13 OL-31593-01...
Page 144 8 19:12:05 2011 [NOT] Opc:MTS_OPC_PPM_INTERFACE_UPDATE(152601), Id:0X00003903, Ret:SUCCESS Src:0x00000101/489, Dst:0x00000101/0, Flags:None HA_SEQNO:0X00000000, RRtoken:0x00000000, Sync:UNKNOWN, Payloadsize:107 Payload: 0x0000: 00 00 00 02 00 00 00 02 00 00 00 0c 00 00 00 26 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 10-14 OL-31593-01...
Page 145: Information About Port Channels And Trunking
If a link goes down in a port channel, the upper protocol is not aware of it. To the upper protocol, the link is still there, although the bandwidth is diminished. The MAC address tables are not affected by link failures. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 11-1 OL-31593-01...
Page 146: Port Channel Restriction
Ensure that all interfaces in the port channel have the same destination device for Link Aggregation Control Protocol (LACP) channels. By using the Asymmetric Port Channel (APC) feature in the Cisco Nexus 1000V, ports in an ON mode channel can be connected to two different destination devices.
Page 147: Troubleshooting Asymmetric Port Channels
Use APC when you want to configure a port channel whose members are connected to two different • upstream switches. APC depends on Cisco Discovery Protocol (CDP). Make sure CDP is enabled on the VSM and • upstream switches. Physical ports within an APC get assigned subgroup IDs based on the CDP information received •...
Page 148: Cannot Create Port Channel
You can have a maximum of 256 port channels on the Cisco Nexus 1000V. Newly Added Interface Does Not Come Online In a Port Channel...
Page 149: Verifying A Port Channel Configuration
A VLAN is not in the allowed VLAN Add the VLAN to the allowed VLAN list. Use the does not traverse list. switchport trunk allowed vlan add vlan-id command in trunk. the profile used by the interface. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 11-5 OL-31593-01...
Page 150 Chapter 11 Port Channels and Trunking VLAN Traffic Does Not Traverse Trunk Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 11-6 OL-31593-01...
Page 151: Information About Layer 2 Ethernet Switching
Troubleshooting BPDU Guard, page 12-14 Information About Layer 2 Ethernet Switching The Cisco Nexus1000V is a distributed Layer 2 virtual switch that extends across many virtualized hosts. It consists of two components: The Virtual Supervisor Module (VSM), which is also known as the control plane (CP). The VSM •...
Page 152: Viewing Ports From The Vem
Cisco Nexus1000V. • Virtual Ethernet Ports (VEth)—A vEth port is a port on the Cisco Nexus 1000V. The Cisco Nexus 1000V has a flat space of vEth ports 0..N. The virtual cable plugs into these vEth ports that are moved to the host running the VM.
Page 153: Viewing Ports From The Vsm
Each uplink port on the host represents a physical interface. It acts like an lveth port, but because physical ports do not move between hosts, the mapping is 1:1 between an uplink port and a vmnic. Each physical port added to the Cisco Nexus1000V switch appears as a physical Ethernet port, just •...
Page 154: Port Types
Po (port channel interfaces)—The physical NICs of an ESX Host can be bundled into a logical • interface. This logical bundle is referred to as a port channel interface. For more information about Layer 2 switching, see the Cisco Nexus 1000V Layer 2 Switching Configuration Guide. Layer 2 Switching Problems This section describes how to troubleshoot Layer 2 problems and lists troubleshooting commands.
Page 155: Verifying A Connection Between Vems
Log in to the upstream switch and make sure that the port is configured to allow the VLAN that you are looking for. switch# show running-config interface gigabitEthernet 1/38 Building configuration... Current configuration : 161 bytes interface GigabitEthernet1/38 description Srvr-100:vmnic1 switchport switchport trunk allowed vlan 1,60-69,231-233 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-5 OL-31593-01...
Page 156: Isolating Traffic Interruptions
11w4d: RARP: Rcvd RARP req for 0050.56b7.52f4 10.78.1.123 0050.564f.3586 11w4d: IP ARP: rcvd req src , dst 10.78.1.24 Vlan3002 11w4d: RARP: Rcvd RARP req for 0050.56b7.3031 switch# Example: switch# show arp Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-6 OL-31593-01...
Page 157: Layer 2 Switching Troubleshooting Commands
[all-ports | brief | id vlan-id name Displays VLAN information as specified. See name | dot1q tag native] Example 12-4 on page 12-9. show vlan summary Displays a summary of VLAN information. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-7 OL-31593-01...
Page 158 Example 12-1 show mac address-table Command Note The Cisco Nexus 1000V MAC address table does not display multicast MAC addresses. The “Module” indicates the VEM on which this MAC address is seen. The “N1KV Internal Port” refers to an internal port created on the VEM. This port is used for control and management of the VEM and is not used for forwarding packets.
Page 159 VLAN0118 active VLAN0119 active VLAN0800 active VLAN0801 active VLAN0802 active VLAN0803 active VLAN0804 active VLAN0805 active VLAN0806 active VLAN0807 active VLAN0808 active VLAN0809 active VLAN0810 active VLAN0811 active VLAN0812 active Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-9 OL-31593-01...
Page 160 1 Access l22 3968 VIRT 1 Access l23 VIRT 1 Access l24 VIRT 0 Access l25 3967 VIRT 1 Access l26 1a030100 PHYS 1 Trunk vmnic1 1a030200 PHYS 1 Trunk vmnic2 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-10 OL-31593-01...
Page 161 ~ # module vem 5 execute vemcmd show l2 Bridge domain 115 brtmax 1024, brtcnt 2, timeout 300 Dynamic MAC 00:50:56:bb:49:d9 LTL 16 timeout 0 Dynamic MAC 00:02:3d:42:e3:03 LTL 10 timeout 0 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-11 OL-31593-01...
Page 162: Troubleshooting Microsoft Nlb Unicast Mode
Access to third-party websites identified in this document is provided solely as a courtesy to customers Note and others. Cisco Systems, Inc. and its affiliates are not in any way responsible or liable for the functioning of any third-party website, or the download, performance, quality, functioning, or support...
Page 163: Checking The Status On A Vem
When MS NLB VMs have more than one port on the same subnet, a request is flooded, which causes both ports to receive it. The server cannot manage this situation. As a workaround for this situation, enable Unknown Unicast Flood Blocking (UUFB). Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-13 OL-31593-01...
Page 164: Troubleshooting Bpdu Guard
Displays the switch edition and license information. Example 12-11 on page 12-15. show run interface name Displays the BPDU guard status on a port profile. Example 12-12 on page 12-15. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-14 OL-31593-01...
Page 165 Veth36 Enabled Veth68 Enabled Veth73 Enabled Veth77 Enabled name Example 12-14 show system internal cdm info port-profile Command switch(config-if)# show system internal cdm info port-profile name vm port-profile vm Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-15 OL-31593-01...
Page 166 ~ # echo "debug sfcdmagent all" > /tmp/dpafifo ~ # echo "debug sfportagent all" > /tmp/dpafifo Packet path: # vemlog debug sflayer2 all ~ # echo "debug sfportagent all" > /tmp/dpafifo Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 12-16 OL-31593-01...
Page 167: Chapter 13 Vlans
You can configure a private VLAN port as a SPAN source port. – You can use VLAN-based SPAN (VSPAN) on primary, isolated, and community VLANs or use – SPAN on only one VLAN to separately monitor egress or ingress traffic. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 13-1 OL-31593-01...
Page 168: Initial Troubleshooting Checklist
• show vlan internal bd-info vlan-to-bd 1 • show vlan internal errors • show vlan internal info • show vlan internal event-history errors • Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 13-2 OL-31593-01...
Page 169: Cannot Create A Vlan
Cannot Create a VLAN Symptom Possible Cause Solution Cannot create a Using a reserved VLAN ID VLANs 3968 to 4047 and 4094 are reserved for internal use VLAN. and cannot be changed. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 13-3 OL-31593-01...
Page 170 Chapter 13 VLANs Cannot Create a VLAN Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 13-4 OL-31593-01...
Page 171: Chapter 14 Private Vlans
The mechanism that restricts Layer 2 communication between two isolated ports in the same switch also restricts Layer 2 communication between two isolated ports in two different switches. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 14-1 OL-31593-01...
Page 172: Private Vlan Ports
• Isolated • Community • For additional information about private VLANs, see the Cisco Nexus 1000V Layer 2 Switching Configuration Guide. Troubleshooting Guidelines Follow these guidelines when troubleshooting private VLAN issues: • Use the show vlan private-vlan command to verify that a private VLAN is configured correctly.
Page 173 VIRT 4 Access l20 3969 VIRT 4 Access l21 VIRT 4 Access l22 3968 VIRT 4 Access l23 VIRT 4 Access l24 VIRT 0 Access l25 3967 VIRT 4 Access l26 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 14-3 OL-31593-01...
Page 174 1b020000 VIRT 4 Access fedora9.eth0 pvlan community 156 153 If additional information is required for Cisco Technical Support to troubleshoot a private VLAN issue, use the following commands: show system internal private-vlan info • show system internal private-vlan event-history traces •...
Page 175: Chapter 15 Netflow
• A flow record defines the information that NetFlow gathers, such as packets in the flow and the types of counters gathered per flow. You can define new flow records or use the predefined Cisco Nexus 1000V flow records. For detailed information about configuring NetFlow, see the Cisco Nexus 1000V System Management Configuration Guide.
Page 176: Common Netflow Problems
VEM command: vemcmd show netflow monitor show flow internal pdl detailed • Displays internal flow details. Common NetFlow Problems Common NetFlow configuration problems on the VSM can occur if you attempt to do the following: Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 15-2 OL-31593-01...
Page 177: Debugging A Policy Verification Error
Ensure that the UDP port configured on the exporter matches that used by the NetFlow Collector. • View statistics for the exporter and identify any drops by entering the show flow exporter • command. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 15-3 OL-31593-01...
Page 178 Chapter 15 NetFlow Common NetFlow Problems Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 15-4 OL-31593-01...
Page 179: Chapter 16 Acls
• IPv6—The device applies IPv6 ACLs only to IPv6 traffic • For detailed information about how ACL rules are used to configure network traffic, see the Cisco Nexus 1000V Security Configuration Guide. ACL Configuration Limits The following configuration limits apply to ACLs: •...
Page 180: Acl Restrictions
The commands listed in this section can be used to display configured ACL policies on the Virtual Ethernet Module (VEM). Use the following command to list the ACLs installed on that server switch(config-if)# module vem 3 execute vemcmd show acl Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 16-2 OL-31593-01...
Page 181: Debugging Policy Verification Issues
Save the Telnet or SSH session buffer to a file. Copy the logfile created in bootflash. Troubleshooting ACL Logging This section includes the following topics: Using the CLI to Troubleshoot ACL Logging on a VEM, page 16-4 • Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 16-3 OL-31593-01...
Page 182: Using The Cli To Troubleshoot Acl Logging On A Vem
You can use the vemcmd flush aclflows command to detect any new flows that affect the VEM. Clear all the existing flows, and then you can detect new flows that match any expected traffic. Syslog messages are not sent when you do this action. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 16-4 OL-31593-01...
Page 183: Acl Logging Troubleshooting Scenarios
If the ACL rule does not have a log keyword, any flow that matches the ACL is not reported although the ACL statistics continue to advance. You can verify a log keyword. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 16-5...
Page 184 Log in to the VSM and VEM CLI. PROCEDURE Command Description Step 1 Verifies that ACL logging is configured properly. show logging ip access-list status Example: switch# show logging ip access-list status switch # Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 16-6 OL-31593-01...
Page 185 Example: switch# show logging ip access-list status switch # Step 2 Verifies ACL logging on the VEM. vemcmd show acllog config Example: switch# vemcmd show acllog config switch # Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 16-7 OL-31593-01...
Page 186 Chapter 16 ACLs Troubleshooting ACL Logging Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 16-8 OL-31593-01...
Page 187: Chapter 17 Quality Of Service
Policing —Monitors data rates and burst sizes for a particular class of traffic. QoS policing on a • network determines whether network traffic is within a specified profile (contract). For detailed information about QoS, see the Cisco Nexus 1000V Quality of Service Configuration Guide. QoS Configuration Limits Table 17-1 Table 17-2 list the configuration limits for QoS.
Page 188: Qos Troubleshooting Commands
VSM to the connected modules. Example 17-1 on page 17-3 show resource-availability qos-queuing Checks whether the QoS configuration is not exceeding the recommended resource limits. show policy-map interface brief Displays the installed policies: Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 17-2 OL-31593-01...
Page 189: Troubleshooting The Vem
~ # module vem 3 execute vemcmd show qos node nodeid type details -------- -------- -------- policer cir:50 pir:50 bc:200000 be:200000 cir/pir units 1 bc/be units 3 flags 2 class op_AND DSCP Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 17-3 OL-31593-01...
Page 190: Debugging Policy Configuration Errors
Step 4 Enter the policy-map command which will execute the command once again with the DPA debug traces output to vemdpalog. Step 5 Enter module vem module-number execute vemdpalog stop command. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 17-4 OL-31593-01...
Page 191: Debugging Policy Verification Failures
- add_pinst gpolicy_id 72352041 verify - installing pinst type 0 49 for policy 0 verify - returned 0 commit - adding pinst ltl 49 use 2 to policy 0 Session commit complete and successful Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 17-5 OL-31593-01...
Page 192: Debugging Policing Configuration Errors
11 policy id 0 if_index 1a020200 --> Service-policy being applied installing pinst type 0 17 for policy 0 dpa_sf_qos_verify returned 0 … Session commit complete and successful --> Session ending Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 17-6 OL-31593-01...
Page 193: Chapter 18 Span
• Encapsulated remote SPAN (ERSPAN) that can send monitored traffic to an IP destination. • For detailed information about how to configure local SPAN or ERSPAN, see the Cisco Nexus 1000V System Management Configuration Guide. SPAN Session Guidelines The following are SPAN session guidelines: •...
Page 194: Problems With Span
Use the vempkt command to capture packets on the VMKernel NIC LTL and ensure ERSPAN packets are being sent. Use the vemlog debug sfspan d command so that the ERSPAN packets appear in the vempkt capture log. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 18-2 OL-31593-01...
Page 195: Span Troubleshooting Commands
: Eth3/3 source VLANs both filter VLANs : filter not specified destination IP : 10.54.54.1 ERSPAN ID : 999 ERSPAN TTL : 64 ERSPAN IP Prec. ERSPAN DSCP ERSPAN MTU : 1000 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 18-3 OL-31593-01...
Page 196 RX Vlan Sources : TX Vlan Sources : Source Filter : 2 local 50 RX Ltl Sources :51, TX Ltl Sources :51, RX Vlan Sources : TX Vlan Sources : Source Filter : Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 18-4 OL-31593-01...
Page 197: Chapter 19 Multicast Igmp
In general, IGMP snooping works as follows: • Ethernet switches, such as Cisco Catalyst 6000 Series switches, parse and intercept all IGMP packets and forward them to a CPU, such as a supervisor module, for protocol processing. Router ports are learned using IGMP queries. The switch returns IGMP queries, it remembers which •...
Page 198: Problems With Multicast Igmp Snooping
Make sure that the upstream switch has IGMP configured. • Use the show ip igmp snooping groups command to verify if the Cisco Nexus 1000V switch is • configured correctly and is ready to forward multicast traffic. In the displayed output of the command, look for the letter R under the port heading.
Page 199 Debug sf_igmp_snoop_thread: Check timed-out members in 224.6.7.8, BD: 59 Jul 15 18:19:34.418381 25 12 Debug IGMP pkt (snoop ON): orig_src_ltl 0x0, src_ltl 0x66 vlan 59 Jul 15 18:19:34.418385 26 12 Debug Notification size: 72 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 19-3 OL-31593-01...
Page 200 Jul 15 18:19:37.085345 Debug Forward report to router port: 10347 .Jul 15 18:19:37.085998 Debug IGMP pkt (snoop ON): orig_src_ltl 0x15, src_ltl 0x40f vlan 59 Jul 15 18:19:37.086002 Debug Notification size: 68 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 19-4 OL-31593-01...
Page 201 Debug sf_igmp_snoop_add_update_v4_grp: Existing Group 0.0.0.0 in BD 52. Jul 15 18:19:37.648439 Debug sf_igmp_snoop_add_update_v4_grp: Existing Member 1039 in Group 0.0.0.0 in BD 52. Jul 15 18:19:42.002071 Debug sf_igmp_snoop_thread: IGMP Snoop Thread waken up Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 19-5 OL-31593-01...
Page 202: Multicast Igmp Snooping Troubleshooting Commands
IGMPv3 Report suppression disabled Router port detection using PIM Hellos, IGMP Queries Number of router-ports: 0 Number of groups: 0 show ip igmp snooping groups • switch# show ip igmp snooping groups vlan 1784 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 19-6 OL-31593-01...
Page 203 11 2014 Jul 8 23:49:17.131693 igmp[3157]: SNOOP: Mode for if(Vethernet38): 0x80000 vlan: 11 2014 Jul 8 23:49:17.156004 igmp[3157]: SNOOP: <vlan 11> Added Veth47 to active ports for vlan 11 2014 J Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 19-7 OL-31593-01...
Page 204 The multicast group table for 224.1.2.3 shows the interfaces that the VEM forwards to when it receives multicast traffic for group 224.1.2.3. If fedora8 has multicast group 224.1.2.3 on its eth0 interface, LTL 47 should be in the multicast group table for 224.1.2.3. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 19-8 OL-31593-01...
Page 205: Problems With Multicast Igmp Snooping
Make sure that the table has the correct information in it. Make sure that the state of the trunk port and the access port is UP/UP. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 19-9 OL-31593-01...
Page 206 Chapter 19 Multicast IGMP Problems with Multicast IGMP Snooping Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 19-10 OL-31593-01...
Page 207: Information About Dhcp Snooping
Dynamic ARP inspection (DAI) and IP Source Guard also use information stored in the DHCP snooping binding database. For detailed information about configuring DHCP snooping, see the Cisco Nexus 1000V Security Configuration Guide. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
Page 208: Information About Dynamic Arp Inspection
• A maximum of 1000 static DHCP entries per interface can be configured. • For detailed guidelines and limitations used in configuring these features, see the Cisco Nexus 1000V Security Configuration Guide. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 20-2...
Page 209: Problems With Dhcp Snooping
On the host connected to the client, enable VEM • packet capture to verify incoming requests and acknowledgements in packets. The Cisco Nexus 1000V is dropping packets. On the VSM, verify DHCP statistics. show ip dhcp snooping statistics module vem mod# execute vemcmd show dhcps stats Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
Page 210: Troubleshooting Dropped Arp Responses
Configuration Guide. If all configurations are correct, make sure to turn on DHCP snooping before DAI or IPSG to make sure the Cisco Nexus 1000V has enough time to add the binding in the snooping database. For more information, see the Cisco Nexus 1000V Security Configuration Guide.
Page 211: Problems With Ip Source Guard
For detailed information about configuring IP Source Guard, see the Cisco Nexus 1000V Security Configuration Guide The IP address that corresponds to the On the VSM, display the binding table.
Page 212: Host Logging
Displays general information about DHCP snooping. Example 20-2 on page 20-7. show ip dhcp snooping binding Displays the contents of the DHCP snooping binding table. Example 20-3 on page 20-8. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 20-6 OL-31593-01...
Page 213 DHCP snooping is operational on the following VLANs: Insertion of Option 82 is disabled Verification of MAC address is enabled DHCP snooping trust is configured on the following interfaces: Interface Trusted ------------ ------- vEthernet 3 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 20-7 OL-31593-01...
Page 214 Configuration : Disabled Operation State : Inactive Example 20-6 show ip arp inspection interface vethernet Command switch# show ip arp inspection interface vethernet 6 Interface Trust State ------------- ----------- vEthernet 6 Trusted switch# Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 20-8 OL-31593-01...
Page 215 00 00 00 03 00 00 00 01 00 00 00 64 00 00 00 07 contd. Example 20-10 show system internal dhcp mem-stats detail Command VSM-N1k# show system internal dhcp mem-stats detail Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 20-9 OL-31593-01...
Page 216 [16843009] PPF goto setting state 1 4) Event:E_DEBUG, length:23, at 682346 usecs after Mon Oct 8 20:57:11 2012 [16843009] Processed log-mts contd Example 20-12 debug dhcp all Command switch# debug dhcp all Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 20-10 OL-31593-01...
Page 217: Chapter 21 Storm Control
Troubleshooting VEM Commands Displays all the statistics related to broadcast, multcast and unknown unicast traffic: • vemcmd show storm stats Displays the configured storm rate on a Virtual Ethernet Module (VEM): Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 21-1 OL-31593-01...
Page 218: Debugging Storm Control On A Vem
Debugging Storm Control on a VEM You can debug storm control on a VEM. vemlog clear. Step 1 vemlog start. Step 2 Step 3 vemlog debug sfstormcontrol all. Step 4 vemlog show all. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 21-2 OL-31593-01...
Page 219 The Nexus 1000V manages a data center defined by the vCenter Server. Each server in the Datacenter is represented as a linecard in Nexus 1000V and can be managed as if it were a line card in a physical Cisco switch. The Nexus 1000V implementation has two components: •...
Page 220: Chapter 22 System
Chapter 22 System General Restrictions for vCenter Server See the Cisco Nexus 1000V Getting Started Guide for a detailed overview of how the Nexus 1000V works with VMware ESX software. General Restrictions for vCenter Server When you are troubleshooting issues related to vCenter Server, make sure that you observe the following...
Page 221: Recovering A Dvs With A Saved Copy Of The Vsm
3-12. From the VC client, register the extension (plug-in) for the VSM. Step 4 For more information see the following procedure in the Cisco Nexus 1000V Getting Started Guide. Creating a Cisco Nexus 1000V Plug-In on the vCenter Server •...
Page 222: Recovering A Dvs Without A Saved Copy Of The Vsm
Step 3 For more information, see the “Unregistering the Extension Key in the vCenter Server” procedure on page 3-12. From the VC client, register the extension (plug-in) for the VSM. Step 4 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 22-4 OL-31593-01...
Page 223: Problems Related To Vsm And Vcenter Server Connectivity
Chapter 22 System Problems Related to VSM and vCenter Server Connectivity For more information see the following procedure in the Cisco Nexus 1000V Getting Started Guide. Creating a Cisco Nexus 1000V Plug-In on the vCenter Server • Manually recreate the old port profiles from your previous configuration.
Page 224: Connection Failure After Esx Reboot
If you use an MTU other than 1500 (the default) for a physical NIC attached to the Cisco Nexus 1000V, then reboots of the ESX can result in a mismatch with the VMware kernel NIC MTU and failure of the VSM and VEM.
Page 225: Setting The System Mtu
• jumbomtu configured on the interface. For more information about configuring MTU on the interface, see the Cisco Nexus 1000V Interface Configuration Guide. When you configure a system MTU on a system port profile, it takes precedence over an MTU you •...
Page 226: Recovering Lost Connectivity Due To Mtu Mismatch
Enters global configuration mode. config t Example: switch# config t switch(config)# Step 2 Displays the port configuration including the LTL number module vem module_number execute vemcmd show port port-LTL-number needed for Step Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 22-8 OL-31593-01...
Page 227: Vsm Creation
Profiles that have the system VLAN configuration allow the VEM to communicate with the VSM. Make sure that the system port-profile is defined with the right system VLANS. Use the show port-profile and show port-profile usage commands to collect basic required information. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 22-9 OL-31593-01...
Page 228: Problems With Port Profiles
The VSM may be overloaded. Make sure that you have 1 GB of memory and CPU shares for the VSM VM on the vCenter Server. Problems with VM Traffic When troubleshooting problems with intra-host VM traffic, follow these guidelines: Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 22-10 OL-31593-01...
Page 229: Vem Troubleshooting Commands
• Example 22-4 vemcmd help Command [root@ESX-cos1 ~]# vemcmd help show card Show the card's global info show vlan [vlan] Show the VLAN/BD table show bd [bd] Show the VLAN/BD table Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 22-11 OL-31593-01...
Page 230: Vem Log Commands
This error is displayed when the VSM tries to spec.extensionKey as create a different DVS after changing the switch Cisco_Nexus_1000V_2055343757 already name. exists, cannot create DVS new-switch. A specified parameter was not correct. spec.extensionKey Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 22-12 OL-31593-01...
Page 231 VSM is not aware of DVPortgroup test port 0 is in use. The resource the nics attached to the port groups. vim.dvs.DistributedVirtualPort 0 is in use. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 22-13 OL-31593-01...
Page 232 Chapter 22 System Error Messages Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 22-14 OL-31593-01...
Page 233: Chapter 23 Before Contacting Technical Support
• Gathering Information for Technical Support At some point, you may need to contact your customer support representative or Cisco TAC for some additional assistance. This section outlines the steps that the you should perform prior to contacting your next level of support, so you can reduce the amount of time that you spend resolving the issue.
Page 234: Obtaining A File Of Core Memory Information
Obtaining a File of Core Memory Information Cisco customer support engineers often use files from your system for analysis. One such file contains memory information and is referred to as a core dump. The file is sent to a TFTP server or to a flash card in slot0: of the local switch.
Page 235: Copying Files
Back up the startup configuration to a server daily before you make any changes. You can write a short script to be run on the Cisco Nexus 1000V to perform a save and then back up the configuration. The script only needs to contain two commands: copy running-configuration startup-configuration and copy startup-configuration tftp://server/name.
Page 236 Chapter 23 Before Contacting Technical Support Copying Files Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 23-4 OL-31593-01...
Page 237: Chapter 24 Network Segmentation Manager
Problems with Network Segmentation Manager, page 24-2 • Network Segmentation Manager Troubleshooting Commands, page 24-7 • Network Segmentation Manager Information About See the Cisco Nexus 1000V Network Segmentation Manager Configuration Guide for more information. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 24-1 OL-31593-01...
Page 238: Problems With Network Segmentation Manager
If not, replace the username and password on the in the networking configuration on vShield Manager. The NSM feature is not enabled on Verify if the NSM feature is enabled on the Cisco the Cisco Nexus 1000V. Nexus 1000V. show feature If not, enable the NSM feature.
Page 239 Director. no port-profile network name logged in vCloud Director: Delete the bridge domain with the same name if it exists. Network already exists no bridge-domain name Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 24-3 OL-31593-01...
Page 240 Check system logs for a port profile failure vCloud Director fails. A system to create the port profile required message reported by NSM. See the Cisco NX-OS message similar to the following is for the network. System Messages Reference for more logged in vCloud Director: information.
Page 241 Check system logs for a port group property vCloud Director fails. A system Vmware port group property on the failure message reported by NSM. See the Cisco message similar to the following is port profile. NX-OS System Messages Reference for more logged in vCloud Director: information.
Page 242 Check system logs for a port profile description vCloud Director fails. A system description for the port profile failure message reported by NSM. See the Cisco message similar to the following is associated with the network. NX-OS System Messages Reference for more logged in vCloud Director: information.
Page 243: Network Segmentation Manager Troubleshooting Commands
| grep NSMGR Displays the system logs from the network segmentation manager. For detailed information about show command output, see the Cisco Nexus 1000V Command Reference. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 24-7 OL-31593-01...
Page 244 Chapter 24 Network Segmentation Manager Network Segmentation Manager Troubleshooting Commands Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 24-8 OL-31593-01...
Page 245: Vxlans
MAC frames are sent over the network. You can have multiple VTEPs per VEM that are used as sources for this encapsulated traffic. The encapsulation carries the VXLAN identifier used to Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-1...
Page 246: Vxlan Tunnel Endpoint
VXLANs and routers or services that have traditional VLAN interfaces cannot be used by VXLAN networks. The only way that VXLANs can currently interconnect with traditional VLANs is through VM-based software routers. Starting with Release 5.2(1)SV3(1.15), Cisco Nexus 1000V for VMware vSphere does not support the Note VXLAN Gateway feature.
Page 247: Vxlan Trunks
BGP peering between 16 VSMs to allow VXLAN segments to reach across servers. BGP runs on the VSM and can exchange VXLAN information with the BGP on any other Cisco Nexus 1000V. The Cisco Nexus 1000V can also be used as a route reflector to exchange a VTEP list between VSMs.
Page 248 Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 172.23.181.67:5000 (EVI 5000) # RD = <Router-id>:<segment-id> *>l[3]:[5000]:[4]:[192.168.69.3]/88 #Local VTEP 192.168.69.3 0.0.0.0 100 32768 i Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-4 OL-31593-01...
Page 249 VTEP BGP routing table information for VRF default, address family L2VPN EVPN BGP table version is 17, local router ID is 192.168.66.10 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-5 OL-31593-01...
Page 250 Last written 00:00:59, keepalive timer expiry due 0.819374 Received 4006 messages, 0 notifications, 0 bytes in queue Sent 4008 messages, 0 notifications, 0 bytes in queue Connections established 1, dropped 0 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-6 OL-31593-01...
Page 251 Import pending : 0 Import in progress : 0 Export RTs : 1 Export RT list : 1:5000 Export RT chg/chg-pending : 0/0 Import RTs : 1 Import RT list : 1:5000 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-7 OL-31593-01...
Page 252: Multi-Mac Capability
VXLAN encapsulation and if the frame does not contain an IP packet. Scalability Maximum Number of VXLANs The Cisco Nexus 1000V supports a total of 4000 and 6144 bridge domains. VSM-DAOX(config-port-prof-srv)# show resource-availability vlan Maximum number of user VLANs supported: 4093...
Page 253: Supported Features
• Jumbo Frames Jumbo frames are supported by the Cisco Nexus 1000V if there is space on the frame to accommodate the VXLAN encapsulation overhead of at least 50 bytes, and the physical switch/router infrastructure has the capability to transport these jumbo-sized IP packets.
Page 254 0 port_count: 2 action: 4 hwbd: 28 pa_count: 0 Veth2, Veth5 switch(config)# Example 25-6 show system internal seg_bd info port switch# show system internal seg_bd info port if_index = <0x1c000010> Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-10 OL-31593-01...
Page 255: Vxlan Gateway Commands
= 4098 switch# VXLAN Gateway Commands Note Starting with Release 5.2(1)SV3(1.15), Cisco Nexus 1000V for VMware vSphere does not support the VXLAN Gateway feature. To display VXLAN Gateway information that is attached to the VSM: switch# show module vem...
Page 256 VTEP-Flags ------------------------------------------------------------------------------ Veth11 172.172.0.134 Veth66 172.172.0.145 Veth67 172.172.1.145 ------------------------------------------------------------------------------- Interface Module Serv Inst Vlan BD-Name ------------------------------------------------------------------------------- Ethernet8/1 1821 vxlan-7001 1822 vxlan-7002 1823 vxlan-7003 1824 vxlan-7004 1825 vxlan-7005 switch# sh module VTEPs Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-12 OL-31593-01...
Page 257 Bridge-domain: segment-cisco VTEP Table Version: 2 Note: You can compare the VTEP table version with the echo show vxlan version-table on VEM. Ifindex Module VTEP-IP Address ------------------------------------------------------------------------------ Veth4 10.106.199.116(D) Veth1 10.106.199.117(D) Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-13 OL-31593-01...
Page 258 Use “vemcmd show port vlans” to verify that the VTEPs are in the correct transport VLAN. To verify bridge domain creation on the VEM: ~ # vemcmd show bd bd-name vxlan-home Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-14 OL-31593-01...
Page 259 Portlist: RedHat_VM1_112.eth4 RedHat_VM1_112.eth5 To display the MAC address table that shows the MAC addresses delivered by the VSM: switch# vemcmd show l2 bd-name segment-cisco Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-15 OL-31593-01...
Page 260: Vem Packet Path Debugging
Note: You can compare the download sequence number against the VTEP download sequence number using the vemcmnd show bd bd-name. Displays if the MAC address table displays the remote IP learning in the segment-cisco bridge domain: switch# vemcmd show l2 bd-name segment-cisco Note - Use the module command to check the details of VEM and gateway on the VSM.
Page 261: Vem Multicast Debugging
You can view the output for all the above logs by using the module vem 4 execute vemlog show all command. VEM Multicast Debugging Use the following command to debug VEM multicast. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-17 OL-31593-01...
Page 262: Vxlan Data Path Debugging
"debug dpa_allplatform all" > /tmp/dpafifo To debug the bridge domain configuration, use the following command: echo “debug sfl2agent all” > /tmp/dpafifo To debug the port configuration, use the following command: Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-18 OL-31593-01...
Page 263: Vempkt
Vempkt has been enhanced to display the VLAN/SegmentID. Use vempkt to trace the packet path through the VEM. Encapsulated: Capture ingress on Seg-VEth LTL – Egress on uplink • Decapsulated: Capture ingress on uplink – Egress on Seg-VEth LTL • Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-19 OL-31593-01...
Page 264: Statistics
Displays the remote IP being learned. vemcmd show l2 bd-name bd-name-string Displays the Layer 2 table for one segment bridge domain. vemcmd show arp all Displays the IP-MAC mapping for the outer encapsulated header. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 25-20 OL-31593-01...
Page 265: Chapter 26 Vsi Discovery And Configuration Protocol
• Information About VDP VDP on the Cisco Nexus 1000V is an implementation of the IEEE standard 802.1Qbg/D2.2 (Edge Virtual Bridging). VDP can detect and signal the presence of end hosts and exchange capability with an adjacent VDP-capable bridge. VDP serves as a reliable first-hop protocol and communicates the presence of end-host Virtual Machines (VMs) to adjacent leaf nodes on the Cisco Dynamic Fabric Automation (DFA) architecture.
Page 266: Problems With Vdp
VM and use this command. A VSI state of 3 means that it is associated. Example 26-1 on page 26-3. show evb Displays configured information in the EVB process. Example 26-2 on page 26-3. Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 26-2 OL-31593-01...
Page 267 24 evb reinit-keep-alive 25 ecp retransmission-timer-exponent 15 ecp max-retries 6 Example 26-4 show ecp Command switch(config)# show ecp ECP Max ReTries : 3 ECP Retransmition Timer Exp : 14(163840 micro seconds) Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 26-3 OL-31593-01...
Page 268 Anycast,swbd 4098, VLAN 0, 28 ports, "BD-Mcast" Segment Mode: Multicast Portlist: VM-L-13-25-10.eth7 VM-L-13-25-2.eth7 VM-L-13-25-1.eth7 VM-L-13-25-3.eth7 VM-L-13-25-7.eth7 VM-L-13-25-5.eth7 VM-L-13-25-4.eth7 VM-L-13-25-6.eth7 VM-L-13-25-8.eth7 VM-L-14-25-1.eth7 VM-L-14-25-2.eth7 VM-L-14-25-10.eth7 VM-L-14-25-3.eth7 VM-L-13-25-9.eth7 VM-L-14-25-4.eth7 VM-L-14-25-8.eth7 VM-L-14-25-7.eth7 VM-L-14-25-6.eth7 VM-L-14-25-5.eth7 VM-L-14-25-9.eth7 VM-L-15-25-10.eth7 VM-L-15-25-3.eth7 VM-L-15-25-2.eth7 VM-L-15-25-1.eth7 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 26-4 OL-31593-01...
Page 269 BD 21, vdc 1, segment id 8000, segment group IP 224.9.19.10, encap NATIVE, vff_mode Anycast,swbd 4098, VLAN 0, 28 ports, "BD-Mcast" Segment Mode: Multicast Portlist: VM-L-13-25-10.eth7 VM-L-13-25-2.eth7 VM-L-13-25-1.eth7 VM-L-13-25-3.eth7 VM-L-13-25-7.eth7 VM-L-13-25-5.eth7 VM-L-13-25-4.eth7 VM-L-13-25-6.eth7 VM-L-13-25-8.eth7 VM-L-14-25-1.eth7 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 26-5 OL-31593-01...
Page 270 Chapter 26 VSI Discovery and Configuration Protocol VDP Troubleshooting Commands VM-L-14-25-2.eth7 VM-L-14-25-10.eth7 VM-L-14-25-3.eth7 VM-L-13-25-9.eth7 VM-L-14-25-4.eth7 VM-L-14-25-8.eth7 VM-L-14-25-7.eth7 VM-L-14-25-6.eth7 VM-L-14-25-5.eth7 VM-L-14-25-9.eth7 VM-L-15-25-10.eth7 VM-L-15-25-3.eth7 VM-L-15-25-2.eth7 VM-L-15-25-1.eth7 VM-L-15-25-7.eth7 VM-L-15-25-4.eth7 VM-L-15-25-5.eth7 VM-L-15-25-6.eth7 Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 26-6 OL-31593-01...
Page 271: Chapter 27 Cisco Trustsec
This packet classification is maintained by tagging packets on ingress to the Cisco TrustSec network so that they can be properly identified for the purpose of applying security and other policy criteria along the data path. The tag, also called the security group tag (SGT), allows the network to enforce the access control policy by enabling the endpoint device to act upon the SGT to filter traffic.
Page 272: Debugging Commands
Cisco TrustSec Cisco TrustSec Troubleshooting Commands Debugging Commands Command Purpose debug cts authentication Collects and views logs related to Cisco TrustSec authentication. debug cts authorization Collects and views logs related to Cisco TrustSec authorization. debug cts errors Collects and views logs related to Cisco TrustSec errors and warning messages.
Page 273 Cisco Nexus 1000V. See Example 27-1 on page 27-3 vemcmd show cts global Displays if Cisco TrustSec is enabled on the Cisco Nexus 1000V. See Example 27-2 on page 27-3 vemcmd show cts ipsgt Displays the Cisco TrustSec configuration on the Cisco Nexus 1000V.
Page 274: Show Commands
Chapter 27 Cisco TrustSec Cisco TrustSec Troubleshooting Commands show Commands See the Cisco Nexus 1000V Command Reference for more information on the show commands for Cisco TrustSec. Command Purpose show cts Displays the Cisco TrustSec configuration. show cts sxp Displays the SXP configuration for Cisco TrustSec.
Page 275: Problems With Cisco Trustsec
Chapter 27 Cisco TrustSec Problems with Cisco TrustSec Problems with Cisco TrustSec This section includes symptoms, possible causes and solutions for the following problems with Cisco TrustSec. Symptom Possible Causes Verification and Solution The Cisco Nexus 1000V is There is no connection between the...
Page 276 Chapter 27 Cisco TrustSec Problems with Cisco TrustSec Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 27-6 OL-31593-01...
Page 277: Chapter 28 Vcenter Plug-In
Web Clients only. The VMware vSphere Web Client enables you to connect to a VMware vCenter Server system to manage a Cisco Nexus 1000V through a browser. The vCenter Plug-in is installed as a new tab called Cisco Nexus 1000v as part of the user interface in the vSphere Web Client.
Page 278: Generating A Log Bundle
– vSphere Web Client requires the Adobe Flash Player version 11.1.0 or later to be installed. • Make sure that Cisco Nexus 1000V Release 4.2(1)SV2(1.1) is installed and configured to a vCenter. • Generating a Log Bundle You can collect the diagnostic information for VMware vCenter Server by collecting vSphere log files into a single location.
Page 279: Chapter 29 Ethanalyzer
C H A P T E R Ethanalyzer This chapter describes how to use Ethanalyzer as a Cisco NX-OS protocol analyzer tool and includes the following section: Using Ethanalyzer, page 29-1 • Using Ethanalyzer Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark (formerly Ethereal) open source code.
Page 280 2012-10-01 19:15:23.796608 10.78.110.241 -> 72.163.145.51 SSH Encrypted response packet len=144 2012-10-01 19:15:23.797060 10.78.110.241 -> 72.163.145.51 SSH Encrypted response packet len=144 4 packets captured switch# For more information about Wireshark, see the following URL: http://www.wireshark.org/docs/ Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1) 29-2 OL-31593-01...

Cisco Nexus 1000V Troubleshooting Manual

New and Changed Information

Preface

Audience

Chapter 1 Overview

Troubleshooting Tools

Chapter 2 Troubleshooting Tool

Chapter 3 Installation

Isolating Installation Problems

Improving Performance on the ESX and VM

Verifying the Domain Configuration

Verifying the Port Group Assignments for a VSM VM Virtual Interface

Verifying VSM and Vcenter Server Connectivity

Troubleshooting Connections to Vcenter Server

Recovering the Network Administrator Password

Managing Extension Keys

Recreating the Cisco Nexus 1000V Installation

Problems with the Cisco Nexus 1000V Installation Management Center

Chapter 4 Licenses

Chapter 5 Upgrades

Chapter 6 High Availability

Chapter 7 VSM and VEM Module

Problems with the VSM

Verifying the VSM Is Connected to Vcenter Server

Verifying the VSM Is Configured Correctly

Checking the Vcenter Server Configuration

Checking Network Connectivity between the VSM and the VEM

Chapter 8 L3Sec

Ports

Chapter 9 Port

Cannot Enable an Interface

Port Link Failure or Port Not Connected

Link Flapping

Port Errdisabled

VM Cannot Ping a Secured Port

Port Security Violations

Port State Is Blocked on a VEM

Chapter 10 Port Profiles

Information about Port Channels and Trunking

Chapter 11 Port Channel and Trunking

Initial Troubleshooting Checklist

Troubleshooting Asymmetric Port Channels

Cannot Create Port Channel

Newly Added Interface Does Not Come Online in a Port Channel

VLAN Traffic Does Not Traverse Trunk

Chapter 12 Layer 2 Switching

Chapter 13 Vlans

Chapter 14 Private Vlans

Chapter 15 Netflow

Chapter 16 Acls

Chapter 17 Quality of Service

Chapter 18 SPAN

Chapter 19 Multicast IGMP

Chapter 20 DHCP, DAI, and IPSG

Information about Dynamic ARP Inspection

Information about IP Source Guard

Guidelines and Limitations for Troubleshooting

Problems with DHCP Snooping

Troubleshooting Dropped ARP Responses

Problems with IP Source Guard

Collecting and Evaluating Logs

DHCP, DAI, and IPSG Troubleshooting Commands

Chapter 21 Storm Control

Chapter 22 System

General Restrictions for Vcenter Server

Recovering a DVS

Problems Related to VSM and Vcenter Server Connectivity

Connection Failure after ESX Reboot

VSM Creation

Port Profiles

Problems with Hosts

Problems with VM Traffic

VEM Troubleshooting Commands

VEM Log Commands

Error Messages

Chapter 23 Before Contacting Technical Support

Chapter 24 Network Segmentation Manager

Vxlans

Chapter 25 VXLAN

Vempkt