Page 1 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 2 OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks.
Page 3: Table Of Contents
Viewing Logs Cisco Support Communities Contacting Cisco or VMware Customer Support Tools Used in Troubleshooting Commands Ping Traceroute Monitoring Processes and CPUs Identifying the Processes Running and their States Displaying CPU Utilization Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 4 Recovering the Network Administrator Password Managing Extension Keys Known Extension Problems and Resolutions Resolving a Plug-In Conflict Finding the Extension Key on the Cisco Nexus 1000V Finding the Extension Key Tied to a Specific DVS Verifying Extension Keys Recreating the Cisco Nexus 1000V Installation...
Page 5 Recovering Management and Control Connectivity of a Host when a VSM is Running on a VEM 7-12 Using the VEM Connect Script 7-12 Checking the VEM Configuration 7-14 Collecting Logs 7-17 VSM and VEM Troubleshooting Commands 7-18 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 6 Forcing Port Channel Characteristics onto an Interface 10-4 Verifying a Port Channel Configuration 10-5 VLAN Traffic Does Not Traverse Trunk 10-5 Layer 2 Switching 11-1 Information About Layer 2 Ethernet Switching 11-1 Port Model 11-1 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 7 Information About NetFlow 14-1 NetFlow Troubleshooting Commands 14-2 Common NetFlow Problems 14-3 Debugging a Policy Verification Error 14-3 Debugging Statistics Export 14-3 ACLs 15-1 About Access Control Lists (ACLs) 15-1 ACL Configuration Limits 15-1 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 8 Problems with Multicast IGMP Snooping 18-2 Troubleshooting Guidelines 18-2 Troubleshooting Commands 18-2 Symptoms, Causes, and Solutions 18-4 DHCP, DAI, and IPSG 19-1 Information About DHCP Snooping 19-1 Information About Dynamic ARP Inspection 19-2 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) viii OL-28795-01...
Page 9 Problems with Port Profiles 21-10 Problems with Hosts 21-10 Problems with VM Traffic 21-10 VEM Troubleshooting Commands 21-11 VEM Log Commands 21-12 Error Messages 21-12 Network Segmentation Manager 22-1 Information About Network Segmentation Manager 22-1 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 10 Guidelines and Limitations for Troubleshooting Cisco TrustSec 24-1 Cisco TrustSec Troubleshooting Commands 24-2 Debugging Commands 24-2 Host Logging Commands 24-3 Example 24-3 Show Commands 24-4 Problems with Cisco TrustSec 24-4 vCenter Plug-in 25-1 vCenter Plug-in Overview 25-1 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 11 Using Ethanalyzer 26-1 Before Contacting Technical Support 27-1 Cisco Support Communities 27-1 Gathering Information for Technical Support 27-1 Obtaining a File of Core Memory Information 27-2 Copying Files 27-3 N D E X Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 12 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 13 Installation Nexus 1000V VC Plugin Installation. Nexus 1000V Installation Added new section to troubleshoot the 4.2(1)SV1(5.1) “Problems with the Nexus 1000V Management Center Nexus 1000V Installation Management Installation Management Center” Center. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) xiii OL-28795-01...
Page 14 VSM connection failure The section describes how to prevent loss 4.0(4)SV1(3) “Connection Failure After ESX after ESX reboot of connectivity related to an MTU Reboot” mismatch following a reboot of the ESX. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 15: Document Conventions
Related Documentation, page xvi • • Obtaining Documentation and Submitting a Service Request, page xviii Audience This publication is for experienced network administrators who configure and maintain a Cisco Nexus 1000V. Document Conventions Command descriptions use these conventions: Convention Description boldface font Commands and keywords are in boldface.
Page 16: Related Documentation
Means reader be careful. In this situation, you might do something that could result in equipment Caution damage or loss of data. Related Documentation This section lists the documents used with the Cisco Nexus 1000 and available on Cisco.com at the following URL: http://www.cisco.com/en/US/products/ps9902/tsd_products_support_series_home.html...
Page 17 Cisco Nexus 1000V Password Recovery Guide Cisco NX-OS System Messages Reference Virtual Services Appliance Documentation The Cisco Nexus Virtual Services Appliance (VSA) documentation is available at http://www.cisco.com/en/US/products/ps9902/tsd_products_support_series_home.html Virtual Security Gateway Documentation The Cisco Virtual Security Gateway documentation is available at http://www.cisco.com/en/US/products/ps13095/tsd_products_support_series_home.html...
Page 18: Obtaining Documentation And Submitting A Service Request
Obtaining Documentation and Submitting a Service Request For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at: http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html...
Page 19: Overview Of Troubleshooting
We recommend the following general best practices for most networks: • Maintain a consistent Cisco Nexus 1000V release across all network devices. Refer to the release notes for your Cisco Nexus 1000V release for the latest features, limitations, • and caveats.
Page 20: Troubleshooting Basics
• Troubleshooting Basics This section introduces questions to ask when troubleshooting a problem with Cisco Nexus 1000V or connected devices. Use the answers to these questions to identify the scope of the problem and to plan a course of action.
Page 21: Verifying Ports
Use the show interface-brief command to check the status of a virtual Ethernet port or a physical Ethernet port. Verifying Layer 3 Connectivity Answer the following questions to verify layer 3 connectivity: Have you configured a gateway of last resort? • Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-25387-01...
Page 22: Overview Of Symptoms
Those problems and corrective actions include the following: • Identify key Cisco Nexus 1000V troubleshooting tools. Obtain and analyze protocol traces using SPAN or Ethanalyzer on the CLI. • Identify or rule out physical port issues.
Page 23: Syslog Server Implementation
Syslog Server Implementation The syslog facility allows the Cisco Nexus 1000V device to send a copy of the message log to a host for more permanent storage. This can be useful if the logs need to be examined over a long period of time or when the Cisco Nexus 1000V device is not accessible.
Page 24: Troubleshooting With Logs
# ps -ef |grep syslogd root 23508 1 0 11:01:41 ? 0:00 /usr/sbin/syslogd Test the syslog server by creating an event in Cisco Nexus 1000V. In this case, port e1/2 was bounced Step 3 and the following was listed on the syslog server. Notice that the IP address of the switch is listed in brackets.
Page 25: Cisco Support Communities
If you purchased the Cisco Nexus 1000V and support contract from Cisco, contact Cisco for Nexus 1000V support. Cisco provides L1, L2, and L3 support. If you purchased the Cisco Nexus 1000V and an SNS through VMware, you should call VMware for Nexus 1000V support. VMware provides L1 and L2 support. Cisco provides L3 support.
Page 26 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 27: Tools Used In Troubleshooting
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . C H A P T E R Tools Used in Troubleshooting This chapter describes the troubleshooting tools available for the Cisco Nexus 1000V and includes the following topics: •...
Page 28: Traceroute
Process = name of the process. • Process states are: D = uninterruptible sleep (usually I/O). • R = runnable (on run queue). • S = sleeping. • • T = traced or stopped. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 29: Displaying Cpu Utilization
1Sec = CPU utilization in percentage for the last one second. Example 2-2 show processes cpu Command n1000v# show processes cpu Runtime(ms) Invoked uSecs 1Sec Process ----- ----------- -------- ----- ----- ----------- 4294967295 init 377810 migration/0 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 30: Displaying Cpu And Memory Information
RADIUS is a protocol used for the exchange of attributes or credentials between a head-end RADIUS server and a client device. These attributes relate to three classes of services: Authentication • Authorization • Accounting • Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 31: Syslog
Authentication refers to the authentication of users for access to a specific device. You can use RADIUS to manage user accounts for access to an Cisco Nexus 1000V device. When you try to log into a device, Cisco Nexus 1000V validates you with information from a central RADIUS server.
Page 32: Enabling Logging For Telnet Or Ssh
Enables logging for telnet or SSH • Disabled by default Example 2-5 terminal monitor Command n1000v# terminal monitor For more information about configuring syslog, see the Cisco Nexus 1000V System Management Configuration Guide, Release 4.2(1)SV2(1.1). Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 33: Installation
Before beginning this procedure, you must know or do the following: • You are logged in to the vSphere client on the ESX server. You are logged in to the Cisco Nexus 1000V CLI in EXEC mode. • Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 34: Host Is Not Visible From Distributed Virtual Switch
If your ESX server does not have an Enterprise Plus license, then you must upgrade your VMware • License to an Enterprise Plus license in order to have visibility to the Cisco Nexus 1000V. Host is Not Visible from Distributed Virtual Switch...
Page 35: Refreshing The Vcenter Server Connection
Host is Not Visible from the Distributed Virtual Switch Refreshing the vCenter Server Connection Use this procedure to refresh the connection between the Cisco Nexus 1000V and vCenter Server. From the Cisco Nexus 1000V Connection Configuration mode on the VSM, enter the following...
Page 36: Improving Performance
Layer 2 domain. Following the installation of the Cisco Nexus 1000V, make certain that you configure a domain ID. Without a domain ID, the VSM will not be able to connect to the vCenter Server. Follow these guidelines: •...
Page 37: Verifying Vsm And Vcenter Server Connectivity
Step 4 Ensure the VMware VirtualCenter Server service is running. Troubleshooting Connections to a vCenter Server Use this procedure to troubleshoot connections between a Cisco Nexus 1000V VSM and a vCenter Server: In a web browser, enter the path: http://<VSM-IP>...
Page 38: Recovering The Network Administrator Password
The actual value of “Cisco_Nexus_1000V_584325821” will vary. It should match the extension key Note from the cisco_nexus_1000v_extension.xml file. Recovering the Network Administrator Password For information about recovering the network administrator password, see the Cisco Nexus 1000V Password Recovery Guide. Managing Extension Keys This section includes the following topics: Known Extension Problems and Resolutions, page 3-7 •...
Page 39: Known Extension Problems And Resolutions
Resolving a Plug-In Conflict If you see the error, “The specified parameter was not correct,” when Creating a Nexus 1000V Plug-In on the vCenter Server, then you have tried to register a plugin that is already registered.
Page 40: Finding The Extension Key Tied To A Specific Dvs
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . DETAILED STEPS From the Cisco Nexus 1000V for the VSM whose extension key you want to view, enter the following Step 1...
Page 41: Verifying Extension Keys
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Verifying Extension Keys You can use this procedure to verify that the Cisco Nexus 1000V and vCenter Server are using the same extension key.
Page 42: Recreating The Cisco Nexus 1000V Installation
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Recreating the Cisco Nexus 1000V Installation Use this section to recreate the complete Cisco Nexus 1000V configuration in the event of a persistent problem that cannot be resolved using any other workaround.
Page 43: Removing The Hosts From The Cisco Nexus 1000V Dvs
Before beginning this procedure, you must know or do the following: You are logged in to the VSM CLI in EXEC mode. • DETAILED STEPS From the Cisco Nexus 1000V VSM, use the following commands to remove the DVS from the vCenter Step 1 Server. config t...
Page 44: Unregister The Extension Key In The Vcenter Server
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Unregister the Extension Key in the vCenter Server You can use this procedure to unregister the Cisco Nexus 1000V extension key in vCenter Server. BEFORE YOU BEGIN Before beginning this procedure, you must know or do the following: You have a browser window open.
Page 45: Problems With The Nexus 1000V Installation Management Center
“Finding the Extension Key on the Cisco Nexus 1000V” procedure on page 3-7, and then click Invoke Method. The extension key is unregistered in vCenter Server so that you can start a new installation of the Cisco Nexus 1000V VSM software. You have completed this procedure.
Page 46 Installation Problems with the Nexus 1000V Installation Management Center S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m .
Page 47: Licenses
For additional information about licensing, including how to purchase or install a license, or how to remove an installed license, see the Cisco Nexus 1000V License Configuration Guide, Release 4.2(1)SV2(1.1). Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 48: Contents Of The License File
Example 4-3 on page 4-5. • Do not edit the contents of the license file. If you have already done so, please contact your Cisco Customer Support Account Team. • Make sure the host-ID in the license file is the same as the host-ID on the switch, using the following...
Page 49: Problems With Licenses
%LICMGR-2-LOG_LIC_USAGE: Feature Example 4-4 on page 4-5 NEXUS1000V_LAN_SERVICES_PKG is using 17 licenses, only 16 Contact your Cisco Customer Support licenses are installed. Account Team to acquire additional licenses. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 50: License Troubleshooting Commands
Transfers the licenses from a VEM to the license license_pool pool. Example 4-9 on page 4-6. For detailed information about show command output, see the Cisco Nexus 1000V Command Reference, Release 4.2(1)SV2(1.1). EXAMPLES Example 4-1 show module n1000v# show module...
Page 51 Vethernet1 is down (VEM Unlicensed) Port description is VM-Pri, Network Adapter 1 Hardware is Virtual, address is 0050.56b7.1c7b Owner is VM "VM-Pri", adapter is Network Adapter 1 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 52 NOTICE="<LicFileID>sample.lic</LicFileID><LicLineID>0</LicLineID> \ <PAK>dummyPak</PAK>" SIGN=34FCB2B24AE8 n1000v# Example 4-8 show license brief n1000v# show license brief license_file.lic n1000v# Example 4-9 svs license transfer src-vem <vem no> license_pool n1000v# svs license transfer src-vem 3 license_pool n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 53: Upgrade
• Information about Upgrades The upgrade for the Cisco Nexus 1000V involves upgrading software on both the VSM and the VEM. An in service software upgrade (ISSU) is available for a stateful upgrade of the Cisco Nexus 1000V image(s) running on the VSM. A stateful upgrade is one without noticeable interruption of data plane services provided by the switch.
Page 54 Make sure the module removal is complete. the upgrade. Restart the software upgrade using the detailed Pre-Upgrade check failed. Return code 0x4093000A instructions in the Cisco Nexus 1000V Installation and (SRG collection failed) Upgrade Guide, Release 4.2(1)SV2(1.1). Error message: The standby VSM is not Verify the HA synchronization state.
Page 55 Verify that the correct images are in the bootflash: repository. show boot Restart the software upgrade using the detailed instructions in the Cisco Nexus 1000V Installation and Upgrade Guide, Release 4.2(1)SV2(1.1). If the problem persists, collect details of the upgrade and open a support case.
Page 56 Do one of the following: following error message: boot with the new image. Restart the software upgrade using the detailed • instructions in the Cisco Nexus 1000V Installation and -- FAIL. Return code 0x4093001E (Standby Upgrade Guide, Release 4.2(1)SV2(1.1). failed to come online) Postpone the upgrade and reset the boot variables to the •...
Page 57: Problems With The Vem Upgrade
Restart the software upgrade using the detailed instructions in the Cisco Nexus 1000V Installation and Upgrade Guide, Release 4.2(1)SV2(1.1). Pre-Upgrade check failed: Return code 0x40930062 (free space in the filesystem is below threshold).
Page 58: Problems With The Gui Upgrade
(all Proceed with the upgrade using the detailed instructions VMkernel logs notifies you VMs vmotioned over) before in the Cisco Nexus 1000V Installation and Upgrade that the loading and starting the upgrade. Guide, Release 4.2(1)SV2(1.1). unloading of modules failed.
Page 59 Please check the connection. software installed and is Restart the software upgrade using the detailed unreachable; while the other instructions in the Cisco Nexus 1000V Installation and VSM has the original Upgrade Guide, Release 4.2(1)SV2(1.1). pre-upgrade original pre-upgrade software version installed and is reachable.
Page 60: Recovering A Secondary Vsm With Active Primary
The same domain ID and password as that of the primary VSM. • For a detailed procedure, see the Cisco Nexus 1000V Installation and Upgrade Guide, Release 4.2(1)SV2(1.1). The VSM comes up and forms an HA pair with the newly-created standalone VSM. The VSMs have the previous version of the software installed.
Page 61: Stopping A Vsm Upgrade
You have completed this procedure. Return to the process that pointed you here: Step 4 “Recovering a Secondary VSM with Active Primary” section on page 5-8 • • “Recovering a Primary VSM with Active Secondary” section on page 5-13 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 62: Changing Boot Variables
Copy the running configuration to the startup configuration. Step 4 copy run start Example: n1000v#(config)# copy run start [########################################] 100%e n1000v#(config)# Verify the change in the system and kickstart boot variables. Step 5 show boot Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-10 OL-28795-01...
Page 63: Powering On The Vsm
From the vCenter Server left-hand panel, right-click the VSM and then choose Power > Power On. Step 1 The VSM starts. You have completed this procedure. Return to the “Recovering a Primary VSM with Active Secondary” Step 2 section on page 5-13. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-11 OL-28795-01...
Page 64: Changing The Ha Role
Example: n1000v#(config-svs-domain)# copy run start [########################################] 100%e n1000v#(config-svs-domain)# You have completed this procedure. Return to the “Recovering a Primary VSM with Active Secondary” Step 4 section on page 5-13. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-12 OL-28795-01...
Page 65: Recovering A Primary Vsm With Active Secondary
The host or cluster of the existing secondary VSM • For a detailed installation procedure, see the Cisco Nexus 1000V Installation and Upgrade Guide, Release 4.2(1)SV2(1.1). Make sure the port groups between the host server and VSM are not connected when the new VSM is...
Page 66 Connected Connect at Power On • The connection from the VSM to the host server through the management port is dropped and is not restored when you power on the VSM. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-14 OL-28795-01...
Page 67: Powering Off The Vsm
In vCenter Server, select the VSM and then choose Edit > Settings. Step 1 The Virtual Machine Properties dialog box opens. Select the Control port group and check the following Device Settings: Step 2 Connect at Power On • Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-15 OL-28795-01...
Page 68 Select the Management port group and check the following Device Setting: • Connect at Power On When you power on the VSM, it will connect to the host server through the management port. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-16 OL-28795-01...
Page 69: Problems With Vsm-Vem Layer 2 To 3 Conversion Tool
Displays the boot variables currently in the running configuration. Example 5-5 on page 5-19. show startup-config | in boot Displays the boot variables currently in the startup configuration. Example 5-6 on page 5-19. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-17 OL-28795-01...
Page 70: Upgrade Troubleshooting Commands
--- --------------- ------------------------------------ -------------------- 10.78.109.43 10.78.109.43 10.78.109.51 4220900d-76d3-89c5-17d7-b5a7d1a2487f 10.78.109.51 n1000v# Example 5-4 show module (VEM and VSM upgraded) n1000v# show module Ports Module-Type Model Status ----- -------------------------------- ------------------ ------------ Virtual Supervisor Module Nexus1000V ha-standby Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-18 OL-28795-01...
Page 71 DVS uuid: 9b dd 36 50 2e 27 27 8b-07 ed 81 89 ef 43 31 17 config status: Enabled operational status: Connected sync status: - version: - n1000v# Example 5-8 show svs upgrade status n1000v# show svs upgrade status Upgrade State: Start Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-19 OL-28795-01...
Page 72 Upgrade VIBs: System VEM Image Upgrade Status: Upgrade Notification Sent Time: Upgrade Status Time(vCenter): Upgrade Start Time: Upgrade End Time(vCenter): Upgrade Error: Upgrade Bundle ID: VSM: VEM400-201007101-BG DVS: VEM400-201007101-BG n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 5-20 OL-28795-01...
Page 73: High Availability
Information About High Availability The purpose of High Availability (HA) is to limit the impact of failures—both hardware and software— within a system. The Cisco NX-OS operating system is designed for high availability at the network, system, and service levels.
Page 74: System-Level High Availability
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . System-Level High Availability The Cisco Nexus 1000V supports redundant VSM virtual machines — a primary and a secondary — running as an HA pair. Dual VSMs operate in an active/standby capacity in which only one of the VSMs is active at any given time, while the other acts as a standby backup.
Page 75 Install the image matching the Active entry that indicates a version VSM on the standby. mismatch, as the following output shows: 2009 May 08:34:15.721920 sysmgr: active_verctrl: Stdby running diff version- force download the standby Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 76 MAC addresses Move the identified VSM(s) out of of the VSM(s) that collide the system to stop role collision. with the working VSM. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 77: High Availability Troubleshooting Commands
VDC Process Normal-exit Stack Core Log-create-time --- --------------- ------ ----------- ----- ----- --------------- 1 private-vlan 3207 Tue Apr 28 13:29:48 2009 n1000V# show processes log pid 3207 ====================================================== Service: private-vlan Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 78 RDN_ST_AC <-- Indicates redundancy state (RDN_ST) of the this VSM is Active (AC) state: RDN_DRV_ST_AC_SB intr: enabled power_off_reqs: 0 reset_reqs: Other CP: slot: 1 status: RDN_ST_SB <-- Indicates redundancy state (RDN_ST) of the other VSM is Standby (SB) Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 79 Last time System Manager was gracefully shutdown. The state is SRV_STATE_MASTER_ACTIVE_HOTSTDBY entered at time Tue Apr 28 13:09:13 2009. The '-b' option (disable heartbeat) is currently disabled. The '-n' (don't use rlimit) option is currently disabled. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 80 Redundancy mode --------------- administrative: HA operational: HA This supervisor (sup-2) ----------------------- Redundancy state: Active Supervisor state: Active Internal state: Active with HA standby Other supervisor (sup-1) ------------------------ Redundancy state: Standby Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 81 The standby VSM console is not accessible externally, but can be accessed from the active VSM through the attach module module-number command. n1000V# attach module 2 This command attaches to the console of the secondary VSM. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 82 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 83: Vsm And Vem Modules
Cisco Nexus 1000V manages a data center defined by a VirtualCenter. Each server in the data center is represented as a module in Nexus 1000V and can be managed as if it were a module in a physical Cisco switch.
Page 84: Guidelines For Troubleshooting Modules
“Checking the vCenter Server Configuration” procedure on • page 7-10. ERROR: Datacenter not found – For a list of terms used with Cisco Nexus 1000V, see the Cisco Nexus 1000V Getting Started Guide, • Release 4.2(1)SV1(5.1). Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 85: Flow Chart For Troubleshooting Modules
Verifying the VSM Is Configured Correctly, page 7-7 Checking the vCenter Server Configuration, page 7-10 Checking Network Connectivity Between the VSM and the VEM, page 7-10 Checking the VEM Configuration, page 7-14 Collecting Logs, page 7-17 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 86: Problems With The Vsm
If the plug-in is not found, then create one using the following procedure in the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(5.1): Creating a Cisco Nexus 1000V Plug-In on the vCenter Server Following a reboot of the VSM, the system stops functioning in one of the following states and does not recover on its own.
Page 87 VSM. copy source filesystem: filename system:running-config If not, reconfigure the VSM using the • following section in the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(5.1): Setting Up the Software After boot, VSM stopped at “Loader Corrupt boot menu file.
Page 88: Verifying The Vsm Is Connected To The Vcenter Server
“Unregister the Extension Key in the vCenter Server” procedure on page 3-12. Install a new extension key using the following procedure in the Cisco Nexus 1000V Getting Started • Guide, Release 4.2(1)SV1(5.1). Creating a Cisco Nexus 1000V Plug-In on the vCenter Server –...
Page 89: Verifying The Vsm Is Configured Correctly
Example: n1000v# show svs domain SVS domain config: Domain id: Control vlan: 3002 Packet vlan: 3003 L2/L3 Control VLAN mode: L2 L2/L3 Control VLAN interface: mgmt0 Status: Config push to VC successful Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 90: Verifying The System Port Profile Configuration
You can use the following procedure to verify that the control and packet VLANs are configured on the VSM. The procedure documented is for troubleshooting VSM and VEM connectivity with layer 2 mode. Note Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 91 Primary L3 Control IPv4 address: 0.0.0.0 Secondary VSM MAC : 00:00:00:00:00:00 Secondary L3 Control IPv4 address: 0.0.0.0 Upgrade : Default Max physical ports: 32 Max virtual ports: 216 Card control VLAN: 168 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 92: Checking The Vcenter Server Configuration
You can use the following procedure from vSphere client to verify the configuration on the vCenter Server. Confirm that the host is added to the data center and the Cisco Nexus 1000V DVS in that data center. Step 1 Confirm that at least one pnic of the host is added to the DVS, and that pnic is assigned to the Step 2 system-uplink profile.
Page 93 ------+----------------+--------+-----+----------+-------------------------- Active Supervisor: * 3002 0050.56be.7ca7 dynamic Gi3/1 switch# show mac address-table interface Gi3/2 vlan 3002 Legend: * - primary entry age - seconds since last seen n/a - not available Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-11 OL-28795-01...
Page 94: Recovering Management And Control Connectivity Of A Host When A Vsm Is Running On A Vem
The VEM connect script sets a given VLAN as a system VLAN on the vmknic that has the given IP address, and also sets the VLAN on all the required uplinks. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-12 OL-28795-01...
Page 95 VLANs in the trunk port profile. If the lists vemcmd show port vlans match, then all of the expected VLANs are forwarding and the Cisco Nexus 1000V is blocking non-allowed VLANs. Enter the following command to display details about the system VLANS:...
Page 96: Checking The Vem Configuration
DVS. vem status Example: ~ # vem status VEM modules are loaded Switch Name Num Ports Used Ports Configured Ports Uplinks vSwitch0 1500 vmnic0 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-14 OL-28795-01...
Page 97 CBL Mode Name 3969 VIRT 1 Access l20 3969 VIRT 1 Access l21 3002 VIRT 1 Access l22 3968 VIRT 1 Access l23 3003 VIRT 1 Access l24 VIRT 0 Access l25 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-15 OL-28795-01...
Page 98 0 Access vmk1 ~ # vemcmd set system-vlan 99 ltl 48 Note Use these vemcmds only as a recovery measure and then update the port profile configuration with correct system VLANs. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-16 OL-28795-01...
Page 99: Collecting Logs
13 • Note If you need to contact Cisco TAC for assistance in resolving an issue, you will need the output of the commands listed in Step Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 100: Vsm And Vem Troubleshooting Commands
DVS are listed, and that the ports are correctly configured as access or trunk on the host. Example 7-11 on page 7-22. Example 7-12 on page 7-22. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-18 OL-28795-01...
Page 101 Active Domain ID: 113 AIPC Interface MAC: 0050-56b6-2bd3 Inband Interface MAC: 0050-56b6-4f2d Src MAC Type Domain-id Node-id Last learnt (Sec. ago) ------------------------------------------------------------------------ 0002-3d40-7102 0302 71441.12 0002-3d40-7103 0402 390.77 n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-19 OL-28795-01...
Page 102 Example 7-5 show running-configuration vlan n1000v# show running-config vlan 260-261 version 4.0(4)SV1(3) vlan 260 name cp_control vlan 261 name cp_packet n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-20 OL-28795-01...
Page 103 Switch uuid: 56 e0 36 50 91 1c 32 7a-e9 9f 31 59 88 0c 7f 76 Card domain: 1024 Card slot: 4 VEM Control (Control VLAN) MAC: 00:02:3d:14:00:03 VEM Packet (Inband) MAC: 00:02:3d:24:00:03 VEM Control Agent (DPA) MAC: 00:02:3d:44:00:03 VEM SPAN MAC: 00:02:3d:34:00:03 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-21 OL-28795-01...
Page 104 VLANs. If the lists match, then all vemcmd show port vlans of the expected VLANs are forwarding and the Cisco Nexus 1000V is blocking non-allowed VLANs. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-22...
Page 105 Trunk port 16 native_vlan 1 CBL 1vlan(1) cbl 1, vlan(3002) cbl 1, vlan(3003) cbl 1, Example 7-16 show module vem mapping n1000v# show module vem mapping Status UUID License Status ----------- ------------------------------------ -------------- absent 33393935-3234-5553-4538-35314e355400 unlicensed powered-up 33393935-3234-5553-4538-35314e35545a licensed n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 7-23 OL-28795-01...
Page 106 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 107: Ports
Some values may not be valid when the interface is down (such as the operation speed). For a complete description of port modes, administrative states, and operational states, see the Cisco Nexus 1000V Interface Configuration Guide, Release 4.2(1)SV2(1.1).
Page 108: Information About Interface Counters
The port security feature allows you to secure a port by limiting and identifying the MAC addresses that can access the port. Secure MACs can be manually configured or dynamically learned. For detailed information about port security, see the Cisco Nexus 1000V Security Configuration Guide, Release 4.2(1)SV2(1.1)).
Page 109: Port Diagnostic Checklist
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Port Diagnostic Checklist Use the following checklist to diagnose port interface activity. For more information about port states, see the Cisco Nexus 1000V Interface Configuration Guide, Release 4.2(1)SV2(1.1). Table 8-1...
Page 110: Problems With Ports
Disable and then enable the port. shut no shut Move the connection to a different port on the same module or a different module. Collect the ESX side NIC configuration. vss-support Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 111: Link Flapping
Use the guidelines in this section to troubleshoot ports that are error disabled. Table 8-4 Troubleshooting error disabled ports Possible Cause Solution Defective or damaged Verify the physical cabling. cable. Replace or repair defective cables. Re-enable the port. shut no shut Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 112: Vm Cannot Ping A Secured Port
3 execute vemcmd show portsec stats New MAC addresses If DSM is set, clear the DSM bit on the VSM. cannot be learned by this no port-security stop learning port. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 113: Port Security Violations
VLAN list. Port Security Violations Use these troubleshooting guidelines when a vEthernet port is disabled because of a security violation. For detailed information about port security, see the Cisco Nexus 1000V Security Configuration Guide, Release 4.2(1)SV2(1.1)). Table 8-6 Troubleshooting Port Security Violations...
Page 114: Port Troubleshooting Commands
Example 8-10 on page 8-12. show interface ethernet counters Displays port counters for identifying synchronization problems. For information about counters, see “Information About Interface Counters” section on page 8-2. Example 8-11 on page 8-12. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 115 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 116 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . For detailed information about show command output, see the Cisco Nexus 1000V Command Reference, Release 4.2(1)SV2(1.1).
Page 117 -------------------------------------------------------------------------------- Eth3/2 1 eth trunk up none 1000(D) -- Eth3/3 1 eth access up none 1000(D) -- n1000v# Example 8-9 show interface ethernet n1000v# show interface e1/14 e1/7 is down (errDisabled) Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 8-11 OL-28795-01...
Page 118 7 Multicast Packets 55 Broadcast Packets 29488527 Bytes 504958 Output Packets 491181 Unicast Packets 1 Multicast Packets 13776 Broadcast Packets 941 Flood Packets 714925076 Bytes 11 Input Packet Drops 0 Output Packet Drops Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 8-12 OL-28795-01...
Page 119 10,100,1000,10000,auto Duplex: half/full/auto Trunk encap. type: 802.1Q Channel: Broadcast suppression: percentage(0-100) Flowcontrol: rx-(off/on/desired),tx-(off/on/desired) Rate mode: none QOS scheduling: rx-(none),tx-(none) CoS rewrite: ToS rewrite: SPAN: UDLD: Link Debounce: Link Debounce Time: MDIX: Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 8-13 OL-28795-01...
Page 120 Example 8-15 module vem execute vemcmd show portsec status n1000V# module vem 3 execute vemcmd show portsec stats if_index cp-cnt Aging Aging Sticky Secure Time Type Enabled Name Addresses 1b020000 Absolute VM-Pri.eth1 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 8-14 OL-28795-01...
Page 121 Max Addresses limit in System (excluding one mac per port) : 8192 ---------------------------------------------------------------------- Secure Mac Address Table ---------------------------------------------------------------------- Vlan Mac Address Type Ports Remaining age (mins) ---- ----------- ------ ----- --------------- 0050.56B7.7DE2 DYNAMIC Vethernet1 ====================================================================== Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 8-15 OL-28795-01...
Page 122 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 123: Port Profiles
For more information about assigning port profiles to physical or virtual ports, see your VMware documentation. To verify that the profiles are assigned as expected to physical or virtual ports, use the following show commands: show port-profile virtual usage • show running-config interface interface-id • Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 124: Problems With Port Profiles
To verify port profile inheritance, use the following command: show running-config interface interface-id • Inherited port profiles cannot be changed or removed from an interface from the Cisco Nexus 1000V Note CLI. This can only be done from vCenter Server.
Page 125 Fix any problems with the domain configuration. For information about configuring the domain, see the Cisco Nexus 1000V System Management Configuration Guide, Release 4.2(1)SV2(1.1). The port profile is configured Verify that the vmware port-group is incorrectly.
Page 126 Cache show port-profile virtual usage Overrun mapping is created. Fix the error in the port profile using the procedures in the Cisco Nexus 1000V Port Profile Configuration Guide, Release 4.2(1)SV2(1.1). Bring the interface out of quarantine. no shutdown The interface comes back online.
Page 127: Recovering A Quarantined Offline Interface
The interface comes back online. Verify that the interface has come back online. Step 3 show interface brief Verify the port profile-to-interface mapping. Step 4 show port-profile virtual usage Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 128: Port Profile Logs
Example 9-4 on page 9-9. show port-profile expand-interface name name Displays a named port profile expanded to include the interfaces assigned to it. Example 9-5 on page 9-11. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 129 Displays the messages logged about port profile event-history msgs events within the Cisco Nexus 1000V. Example 9-13 on page 9-17 For detailed information about show command output, see the Cisco Nexus 1000V Command Reference, Release 4.2(1)SV2(1.1). EXAMPLES Example 9-1 show port-profile...
Page 130 - pinning packet-vlan: - system vlans: none port-group: max ports: 32 inherit: config attributes: channel-group auto mode on sub-group manual evaluated config attributes: channel-group auto mode on sub-group manual assigned interfaces: n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 131 1 max ports: 64 min ports: 32 used ports: 0 vmware config information pg name: PP1027 dvs: (ignore) reserved ports: 16 port-profile role: alias information: pg id: PP1027 dvs uuid: Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01...
Page 132 2 port-profile Unused_Or_Quarantine_Veth id: 7 capability: 0x0 state: 0x1 type: 0x1 system vlan mode: - system vlans: port-binding: dynamic bind_opts: 1 max ports: 32 min ports: 32 used ports: 0 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-10 OL-28795-01...
Page 133 Example 9-5 show port-profile expand-interface name UplinkProfile1 n1000v# show port-profile expand-interface name UplinkProfile1 port-profile EthProfile1 Ethernet2/2 switchport mode trunk switchport trunk allowed vlan 110-119 no shutdown n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-11 OL-28795-01...
Page 134 3/2 Ethernet3/2 port-profile: uplink interface status: quarantine sync status: out of sync cached commands: errors: command cache overrun recovery steps: bring interface online n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-12 OL-28795-01...
Page 135 44 dc 3b 50 53 11 b7 ac-ef ed ef 46 ee df c2 d5 type: 2 port-profile Unused_Or_Quarantine_Uplink id: 1 capability: 0x1 state: 0x1 type: 0x1 system vlan mode: - system vlans: port-binding: static max ports: 32 vmware config information pg name: Unused_Or_Quarantine_Uplink Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-13 OL-28795-01...
Page 136 2 port-profile eth-break-inherit id: 9 capability: 0x1 state: 0x1 type: 0x1 system vlan mode: - system vlans: port-binding: static max ports: 32 vmware config information pg name: eth-break-inherit dvs: (ignore) Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-14 OL-28795-01...
Page 137 44 dc 3b 50 53 11 b7 ac-ef ed ef 46 ee df c2 d5 type: 2 port-profile veth-break-deinherit id: 8 capability: 0x0 state: 0x1 type: 0x1 system vlan mode: - system vlans: port-binding: static max ports: 256 vmware config information Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-15 OL-28795-01...
Page 138 44 dc 3b 50 53 11 b7 ac-ef ed ef 46 ee df c2 d5 type: 2 pending binds: port-profile-role adfd id: 0 desc: num users: 1 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-16 OL-28795-01...
Page 139 8 19:12:05 2011 [NOT] Opc:MTS_OPC_PPM_INTERFACE_UPDATE(152601), Id:0X00003905, Ret:SUCCESS Src:0x00000101/489, Dst:0x00000101/0, Flags:None HA_SEQNO:0X00000000, RRtoken:0x00000000, Sync:UNKNOWN, Payloadsize:107 Payload: 0x0000: 00 00 00 02 00 00 00 02 00 00 00 0c 00 00 00 26 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-17 OL-28795-01...
Page 140 8 19:12:05 2011 [NOT] Opc:MTS_OPC_PPM_INTERFACE_UPDATE(152601), Id:0X00003903, Ret:SUCCESS Src:0x00000101/489, Dst:0x00000101/0, Flags:None HA_SEQNO:0X00000000, RRtoken:0x00000000, Sync:UNKNOWN, Payloadsize:107 Payload: 0x0000: 00 00 00 02 00 00 00 02 00 00 00 0c 00 00 00 26 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 9-18 OL-28795-01...
Page 141: Port Channels And Trunking
If a link goes down in a port channel, the upper protocol is not aware of it. To the upper protocol, the link is still there, although the bandwidth is diminished. The MAC address tables are not affected by link failure. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01 10-1...
Page 142: Port Channel Restriction
Ensure that all interfaces in the port channel have the same destination device for LACP channels. By using Asymmetric Port Channel (APC) feature in Nexus 1000V, ports in a ON mode channel can be connected to two different destination devices.
Page 143: Troubleshooting Asymmetric Port Channels
Use APC when you want to configure a port channel whose members are connected to two different • upstream switches. APC depends on Cisco Discovery Protocol (CDP). Make sure CDP is enabled on VSM and upstream • switches. Physical ports within an APC get assigned subgroup IDs based on the CDP information received •...
Page 144: Cannot Create Port Channel
Use the command, show port-channel summary, to verify channel. reached for system. the number of port-channels already configured. You can have a maximum of 256 port channels on the Cisco Nexus 1000V. Newly Added Interface Does Not Come Online In a Port Channel Symptom...
Page 145: Verifying A Port Channel Configuration
VLAN not in allowed VLAN list. Add the VLAN to allowed VLAN list. Use the switchport not traverse trunk. trunk allowed vlan add vlan-id command in the profile used by the interface. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 10-5 OL-28795-01...
Page 146 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 147: Layer 2 Switching
Virtual Supervisor Module (VSM), which is also known as the Control Plane (CP), acts as the • Supervisor and contains the Cisco CLI, configuration, and high-level features. Virtual Ethernet Module (VEM), which is also known as the Data Plane (DP), acts as a line card and •...
Page 148: Viewing Ports From The Vem
Virtual Ethernet Ports (VEth): A VEth port is a port on the Cisco Nexus 1000V Distributed Virtual Switch. Cisco Nexus 1000V has a flat space of VEth ports 0..N. The virtual cable plugs into these VEth ports that are moved to the host running the VM.
Page 149: Viewing Ports From The Vsm
Figure 11-2 shows the VSM view ports. Figure 11-2 VSM View of Ports L2 Cloud VEM 3 VEM 4 VEM 5 eth4/2 eth4/3 eth3/3 eth3/2 eth5/2 Upstream eth5/3 Switch VM-A VM-B Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-3 OL-28795-01...
Page 150: Port Types
Po (Port Channel Interfaces) – The physical NICs of an ESX Host can be bundled into a logical • interface. This logical bundle is referred to as a port channel interface. For more information about Layer 2 switching, see the Cisco Nexus 1000V Layer 2 Switching Configuration Guide, Release 4.2(1)SV2(1.1). Layer 2 Switching Problems This section describes how to troubleshoot Layer 2 problems and lists troubleshooting commands.
Page 151: Verifying A Connection Between Vems
Log in to the upstream switch and make sure the port is configured to allow the VLAN you are looking for. n1000v#show running-config interface gigabitEthernet 1/38 Building configuration... Current configuration : 161 bytes interface GigabitEthernet1/38 description Srvr-100:vmnic1 switchport switchport trunk allowed vlan 1,60-69,231-233 switchport mode trunk Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-5 OL-28795-01...
Page 152: Isolating Traffic Interruptions
11w4d: RARP: Rcvd RARP req for 0050.56b7.52f4 10.78.1.123 0050.564f.3586 11w4d: IP ARP: rcvd req src , dst 10.78.1.24 Vlan3002 11w4d: RARP: Rcvd RARP req for 0050.56b7.3031 n1000v_CAT6K# Example: n1000v_CAT6K# sh arp Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-6 OL-28795-01...
Page 153: Verifying Layer 2 Switching
[all-ports | brief | id <vlan-id> Displays VLAN information as specified. See name <name> | dot1q tag native] Example 11-4 on page 11-9. show vlan summary Displays a summary of VLAN information. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-7 OL-28795-01...
Page 154 Example 11-1 show mac address-table Note The Cisco Nexus 1000VMAC address table does not display multicast MAC addresses. Module indicates the VEM on which this MAC is seen. N1KV Internal Port refers to an internal port created on the VEM. This port is used for control and management of the VEM and is not used for forwarding packets.
Page 155 VLAN0118 active VLAN0119 active VLAN0800 active VLAN0801 active VLAN0802 active VLAN0803 active VLAN0804 active VLAN0805 active VLAN0806 active VLAN0807 active VLAN0808 active VLAN0809 active VLAN0810 active VLAN0811 active VLAN0812 active Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-9 OL-28795-01...
Page 156 1 Access l22 3968 VIRT 1 Access l23 VIRT 1 Access l24 VIRT 0 Access l25 3967 VIRT 1 Access l26 1a030100 PHYS 1 Trunk vmnic1 1a030200 PHYS 1 Trunk vmnic2 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-10 OL-28795-01...
Page 157 Example 11-9 module vem execute vemcmd show l2 Bridge domain 115 brtmax 1024, brtcnt 2, timeout 300 Dynamic MAC 00:50:56:bb:49:d9 LTL 16 timeout 0 Dynamic MAC 00:02:3d:42:e3:03 LTL 10 timeout 0 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-11 OL-28795-01...
Page 158: Troubleshooting Microsoft Nlb Unicast Mode
Access to third-party websites identified in this document is provided solely as a courtesy to customers Note and others. Cisco Systems, Inc. and its affiliates are not in any way responsible or liable for the functioning of any third-party website, or the download, performance, quality, functioning or support of...
Page 159: Checking Status On A Vem
VLAN 59, swbd 59, "" Flags: P - PVLAN S - Secure D - Drop Type MAC Address timeout Flags PVLAN Dynamic 00:15:5d:b4:d7:02 Dynamic 00:15:5d:b4:d7:04 Dynamic 00:50:56:b3:00:96 Dynamic 00:50:56:b3:00:94 Dynamic 00:0b:45:b6:e4:00 Dynamic 00:00:5e:00:01:0a Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-13 OL-28795-01...
Page 160: Configuring Ms Nlb For Multiple Vm Nics In The Same Subnet
Issues might occur for VMs that use dynamic MAC addresses, other than those assigned by VMware. For ports hosting these types of VMs, disable UUFB. To disable UUFB, enter the following commands: n1000v(config)# int veth3 n1000v(config-if)# switchport uufb disable n1000v(config-if)# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 11-14 OL-28795-01...
Page 161: Vlans
You can configure a private VLAN port as a SPAN source port. – You can use VLAN-based SPAN (VSPAN) on primary, isolated, and community VLANs or use SPAN on only one VLAN to separately monitor egress or ingress traffic. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01 12-1...
Page 162: Initial Troubleshooting Checklist
• show vlan internal bd-info vlan-to-bd 1 • show vlan internal errors • show vlan internal info • show vlan internal event-history errors • Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 12-2 OL-28795-01...
Page 163: Cannot Create A Vlan
Cannot Create a VLAN Symptom Possible Cause Solution Cannot create a Using a reserved VLAN ID VLANs 3968 to 4047 and 4094 are reserved for internal use VLAN. and cannot be changed. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 12-3 OL-28795-01...
Page 164 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 165: Private Vlans
Therefore, the mechanism which restricts Layer 2 communication between two isolated ports in the same switch, also restricts Layer 2 communication between two isolated ports in two different switches. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01 13-1...
Page 166: Private Vlan Ports
• isolated • community • For additional information about private VLANs, see the Cisco Nexus 1000V Layer 2 Switching Configuration Guide, Release 4.2(1)SV2(1.1). Troubleshooting Guidelines Follow these guidelines when troubleshooting private VLAN issues: • Use the show vlan private-vlan command to verify that a private VLAN is configured correctly.
Page 167 155 --> 156 157 --> 152 158 --> 152 1a020400 PHYS Trunk vmnic4 pvlan promiscuous trunk port 153 --> 156 154 --> 156 155 --> 156 157 --> 152 158 --> 152 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 13-3 OL-28795-01...
Page 168 1b020000 VIRT 4 Access fedora9.eth0 pvlan community 156 153 If additional information is required for Cisco Technical Support to troubleshoot a private VLAN issue, use the following commands: • show system internal private-vlan info show system internal private-vlan event-history traces •...
Page 169: Netflow
Tallied packets and bytes show the amount of traffic. A flow record defines the information that NetFlow gathers, such as packets in the flow and the types of counters gathered per flow. You can define new flow records or use the pre-defined Nexus 1000V flow record.
Page 170: Netflow Troubleshooting Commands
High traffic could result in lot of debug messages. Use the following commands to collect information about NFM process run-time configuration errors: • show flow internal event-history errors • show flow internal event-history msgs Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 14-2 OL-28795-01...
Page 171: Common Netflow Problems
When debugging a NetFlow statistics export problem, follow these guidelines: • Ensure the destination IP address is reachable from the VSM. Ensure the UDP port configured on the exporter matches that used by the NetFlow Collector. • Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 14-3 OL-28795-01...
Page 172 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Issue the show flow exporter command to view statistics for the exporter and identify any drops on • the VSM. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 14-4 OL-28795-01...
Page 173: Acls
• • MAC ACLs—The device applies MAC ACLs only to non-IP traffic. For detailed information about how ACL rules are used to configure network traffic, see the Cisco Nexus 1000V Security Configuration Guide, Release 4.2(1)SV2(1.1)). ACL Configuration Limits The following configuration limits apply to ACLs: You cannot have more that 128 rules in an ACL.
Page 174: Acl Restrictions
The commands listed in this section can be used to display configured ACL policies on the VEM. Use the following command to list the ACLs installed on that server ~ # module vem 3 execute vemcmd show acl Acl-id Ref-cnt Type Numrules Stats IPv4 disabled Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 15-2 OL-28795-01...
Page 175: Debugging Policy Verification Issues
Save the Telnet or SSH session buffer to a file. Copy the logfile created in bootflash. Troubleshooting ACL Logging This section includes the following topics: • Using the CLI to Troubleshoot ACL Logging on a VEM, page 15-4 ACL Logging Troubleshooting Scenarios, page 15-5 • Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 15-3 OL-28795-01...
Page 176: Using The Cli To Troubleshoot Acl Logging On A Vem
You can use this command to detect any new flows affecting the VEM. Clear all the existing flows, then you can detect new flows that match any expected traffic. Syslog messages are not sent when you do this. Enter the following command: vemcmd flush aclflows Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 15-4 OL-28795-01...
Page 177: Showing Flow Debug Statistics
Example: n1000v# vemcmd show acllog config n1000v # Step 3 Checks to see if any errors occurred. vemcmd show aclflows dbgstats Example: n1000v# vemcmd show aclflows dbgstats n1000v # Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 15-5 OL-28795-01...
Page 178: Troubleshooting An Acl Rule That Does Not Have A Log Keyword
BEFORE YOU BEGIN Before beginning this procedure, you must know or do the following: You are logged in to the VSM and VEM CLI. • SUMMARY STEPS show logging ip access-list status Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 15-6 OL-28795-01...
Page 179: Troubleshooting A Mismatched Configuration Between A Vsm And A Vem
Before beginning this procedure, you must know or do the following: • You are logged in to the CLI in EXEC mode. SUMMARY STEPS show logging ip access-list status vemcmd show acllog config Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 15-7 OL-28795-01...
Page 180 Example: n1000v # show logging ip access-list status n1000v # Step 2 Verifies ACL logging on the VEM. vemcmd show acllog config Example: n1000v # vemcmd show acllog config n1000v # Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 15-8 OL-28795-01...
Page 181: Quality Of Service
Policing —Monitors data rates and burst sizes for a particular class of traffic. QoS policing on a • network determines whether network traffic is within a specified profile (contract). For detailed information about QoS, refer to the Cisco Nexus 1000V Quality of Service Configuration Guide, Release 4.2(1)SV2(1.1). QoS Configuration Limits...
Page 182: Qos Troubleshooting Commands
Use the following command to list all class maps and polices in use on the server: module vem module-number execute vemcmd show qos node • ~ # module vem 3 execute vemcmd show qos node nodeid type details -------- -------- -------- Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 16-2 OL-28795-01...
Page 183: Debugging Policing Verification Errors
Save the Telnet SSH session buffer to a file. Step 5 If you are debugging a policy on a port profile, it may be easier to first install it directly on an interface. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 16-3 OL-28795-01...
Page 184 11 policy id 0 if_index 1a020200 --> Service-policy being applied installing pinst type 0 17 for policy 0 dpa_sf_qos_verify returned 0 … Session commit complete and successful --> Session ending Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 16-4 OL-28795-01...
Page 185: Span
• Encapsulated remote SPAN (ERSPAN) that can send monitored traffic to an IP destination. • For detailed information about how to configure local SPAN or ERSPAN, see the Cisco Nexus 1000V System Management Configuration Guide, Release 4.2(1)SV2(1.1). SPAN Session Guidelines...
Page 186: Problems With Span
ERSPAN. VEM. The ERSPAN enabled Ping the ERSPAN IP destination from VMKernel NIC is not the host VMKernel NIC. configured with a proper IP, vmkping dest-id gateway, or both. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 17-2 OL-28795-01...
Page 187: Span Troubleshooting Commands
: Eth3/3 source VLANs both filter VLANs : filter not specified destination IP : 10.54.54.1 ERSPAN ID : 999 ERSPAN TTL : 64 ERSPAN IP Prec. ERSPAN DSCP ERSPAN MTU : 1000 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 17-3 OL-28795-01...
Page 188 Example 17-3 module vem execute vemcmd show span n1000v# module vem 3 execute vemcmd show span VEM SOURCE IP: 10.54.54.10 HW SSN ID DST LTL/IP ERSPAN ID 0 10.54.54.1 999 1 48 local Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 17-4 OL-28795-01...
Page 189: Multicast Igmp
IGMP membership reports. The IGMP snooping software responds to topology change notifications. In general, IGMP snooping works as follows: Ethernet switches, like Cisco Catalyst 6000 switches, parse and intercept all IGMP packets and • forward them to a CPU, such as a Supervisor module, for protocol processing.
Page 190: Problems With Multicast Igmp Snooping
R under the port heading. The R indicates that the VSM has learned the uplink router port from the IGMP query that was sent by the upstream switch, and means that the Nexus 1000V is ready to forward multicast traffic.
Page 191 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 18-3...
Page 192: Symptoms, Causes, And Solutions
Make sure the table has the correct information in it. Make sure that the state of the trunk port and the access port is UP/UP. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 18-4 OL-28795-01...
Page 193: Dhcp, Dai, And Ipsg
Dynamic ARP inspection (DAI) and IP Source Guard also use information stored in the DHCP snooping binding database. For detailed information about configuring DHCP snooping, see the Cisco Nexus 1000V Security Configuration Guide, Release 4.2(1)SV2(1.1)). Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 194: Information About Dynamic Arp Inspection
Rate limits on interfaces must be set to high values for trusted interfaces such as VSD SVM ports • or vEthernet ports connecting to DHCP servers. For detailed guidelines and limitations used in configuring these features, see the Cisco Nexus 1000V Security Configuration Guide, Release 4.2(1)SV2(1.1)). Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 195: Problems With Dhcp Snooping
On the host connected to the client, enable VEM • packet capture to verify incoming requests and acknowledgements in packets. The Cisco Nexus 1000V is dropping packets. On the VSM, verify DHCP statistics. show ip dhcp snooping statistics module vem mod# execute vemcmd show dhcps stats Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 196: Troubleshooting Dropped Arp Responses
Configuration Guide, Release 4.2(1)SV2(1.1)). If all configurations are correct, make sure to turn on DHCP snooping before DAI or IPSG. This is to make sure the Cisco Nexus 1000V has enough time to add the binding in the snooping database.
Page 197: Problems With Ip Source Guard
For detailed information about configuring IP Source Guard, see the Cisco Nexus 1000V Security Configuration Guide, Release 4.2(1)SV2(1.1)) The IP address corresponding to the vEthernet On the VSM, display the binding table.
Page 198: Host Logging
Displays the status of DAI. Example 19-5 on page 19-8. show ip arp inspection interface vethernet Displays the trust state and ARP packet rate for a interface-number specific interface. Example 19-6 on page 19-8. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 19-6 OL-28795-01...
Page 199 MacAddress IpAddress LeaseSec Type VLAN Interface ----------------- --------------- -------- ---------- ---- ------------- 0f:00:60:b3:23:33 10.3.2.2 infinite static vEthernet 6 0f:00:60:b3:23:35 10.2.2.2 infinite static vEthernet 10 n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 19-7 OL-28795-01...
Page 200 6 Interface Trust State ------------- ----------- vEthernet 6 Trusted n1000v# Example 19-7 show ip arp inspection vlan n1000v# show ip arp inspection vlan 13 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 19-8 OL-28795-01...
Page 201 Private Mem stats for UUID : Malloc track Library(103) Max types: 5 -------------------------------------------------------------------------------- TYPE NAME ALLOCS BYTES CURR CURR 2 MT_MEM_mtrack_hdl 19236 19384 3 MT_MEM_mtrack_info 9408 14080 4 MT_MEM_mtrack_lib_name 1174 42246 56230 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 19-9 OL-28795-01...
Page 202 [16843009] PPF goto setting state 1 4) Event:E_DEBUG, length:23, at 682346 usecs after Mon Oct 8 20:57:11 2012 [16843009] Processed log-mts contd Example 19-12 debug dhcp all n1000v# debug dhcp all Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 19-10 OL-28795-01...
Page 203: Virtual Service Domain
A Virtual Service Domain (VSD) is a logical group of interfaces that is serviced by a common Service VM (SVM). With VSD the Cisco Nexus 1000V can support third party appliances such as vShield. VSD lets you classify and separate traffic for network services such as firewalls and traffic monitoring.
Page 204: Collecting And Evaluating Logs
2011 Feb 17 10:14:01 vsm vsim: <{vsim}> [DBG]==============ZONES=============== 2011 Feb 17 10:14:01 vsm vsim: <{vsim}>[DBG]Zone_id: 1, name: vsd1, is_in_use? 1, default_action: (DROP), member_cnt: 5 2011 Feb 17 10:14:01 vsm vsim: <{vsim}> [DBG]==============INTFS=============== Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 20-2 OL-28795-01...
Page 205: Virtual Service Domain Troubleshooting Commands
Displays the port state on the VEM. Useful for port debugging traffic flow on interfaces. Example 20-6 on page 20-5. show virtual-service-domain name vsd-name Displays a specific VSD configuration. Example 20-7 on page 20-5. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 20-3 OL-28795-01...
Page 206 Example 20-9 on page 20-6. module vem module_number execute vemcmd Displays the VEM VSD configuration by sending show vsd the command to the VEM from the remote Cisco Nexus 1000V. Example 20-10 on page 20-6. module vem module_number execute vemcmd...
Page 207 Member Vethernet6 Member Vethernet7 Inside Vethernet8 Outside n1000v# Example 20-8 show virtual-service-domain brief n1000v# show virtual-service-domain brief Name vsd-id default action in-ports out-ports mem-ports Modules with VSD Enabled zone forward n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 20-5 OL-28795-01...
Page 208 - system vlans: none port-group: max ports: 32 inherit: config attributes: channel-group auto mode on sub-group manual evaluated config attributes: channel-group auto mode on sub-group manual assigned interfaces: n1000v# Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 20-6 OL-28795-01...
Page 209: System
The Nexus 1000V manages a data center defined by the vCenter Server. Each server in the Datacenter is represented as a linecard in Nexus 1000V and can be managed as if it were a line card in a physical Cisco switch. The Nexus 1000V implementation has two components: Virtual supervisor module (VSM) –...
Page 210: General Restrictions For Vcenter Server
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . See the Cisco Nexus 1000V Getting Started Guide for a detailed overview of how the Nexus 1000V works with VMware ESX software.
Page 211: Recovering A Dvs With A Saved Copy Of The Vsm
From the VC client, register the extension (plug-in) for the VSM. Step 4 For more information see the following procedure in the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(5.1). Creating a Cisco Nexus 1000V Plug-In on the vCenter Server •...
Page 212: Recovering A Dvs Without A Saved Copy Of The Vsm
Step 3 For more information, see the “Unregister the Extension Key in the vCenter Server” procedure on page 3-12. From the VC client, register the extension (plug-in) for the VSM. Step 4 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 21-4 OL-28795-01...
Page 213: Problems Related To Vsm And Vcenter Server Connectivity
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . For more information see the following procedure in the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(5.1).
Page 214: Connection Failure After Esx Reboot
If you use an MTU other than 1500 (the default) for a physical NIC attached to the Cisco Nexus 1000V, then reboots of the ESX can result in a mismatch with the VMware kernel NIC MTU and failure of the VSM and VEM.
Page 215: Setting The System Mtu
• jumbomtu configured on the interface. For more information about configuring MTU on the interface, see the Cisco Nexus 1000V Interface Configuration Guide, Release 4.2(1)SV2(1.1). When you configure a system MTU on a system port profile, it takes precedence over an MTU you •...
Page 216: Recovering Lost Connectivity Due To Mtu Mismatch
Enters global configuration mode. config t Example: n1000v# config t n1000v(config)# Step 2 Displays the port configuration including the LTL number module vem module_number execute vemcmd show port port-LTL-number needed for Step Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 21-8 OL-28795-01...
Page 217: Vsm Creation
Profiles that have the system VLAN configuration allow the VEM to communicate with the VSM. Make sure that the system port-profile is defined with the right system VLANS. Use the show port-profile and show port-profile usage commands to collect basic required information. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 21-9 OL-28795-01...
Page 218: Problems With Port Profiles
The VSM may be overloaded. Make sure that you have 1 GB of memory and CPU shares for the VSM VM on the vCenter Server. Problems with VM Traffic When troubleshooting problems with intra-host VM traffic, follow these guidelines: Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 21-10 OL-28795-01...
Page 219: Vem Troubleshooting Commands
• Example 21-4 vemcmd help Command [root@esx-cos1 ~]# vemcmd help show card Show the card's global info show vlan [vlan] Show the VLAN/BD table show bd [bd] Show the VLAN/BD table Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 21-11 OL-28795-01...
Page 220: Vem Log Commands
This error is displayed when the VSM tries to spec.extensionKey as create a different DVS after changing the switch Cisco_Nexus_1000V_2055343757 already name. exists, cannot create DVS new-n1000v. A specified parameter was not correct. spec.extensionKey Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 21-12 OL-28795-01...
Page 221 VSM is not aware of DVPortgroup test port 0 is in use. The resource the nics attached to the port groups. vim.dvs.DistributedVirtualPort 0 is in use. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 21-13 OL-28795-01...
Page 222 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 223: Network Segmentation Manager
Problems with Network Segmentation Manager, page 22-2 • Network Segmentation Manager Troubleshooting Commands, page 22-7 Network Segmentation Manager Information About See the Cisco Nexus 1000V Network Segmentation Manager Configuration Guide, Release 4.2(1)SV2(1.1)) for more information. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01 22-1...
Page 224: Problems With Network Segmentation Manager
If not, replace the username and password on the in the networking configuration on the vShield Manager. The NSM feature is not enabled on Verify if the NSM feature is enabled on the Cisco the Cisco Nexus 1000V. Nexus 1000V. show feature If not, enable the NSM feature.
Page 225 Director. no port-profile network name logged in the vCloud Director: Delete the bridge domain with the same name if it exists. Network already exists no bridge-domain name Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 22-3 OL-28795-01...
Page 226 See the Cisco NX-OS System Messages logged in the vCloud Director: Reference for more information. Failed to set vlan Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 22-4 OL-28795-01...
Page 227 Directors fails. A system associated with the network. show feature message similar to the following is If not, enable the segmentation feature. logged in the vCloud Director: feature segmentation Failed to create bridge domain Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 22-5 OL-28795-01...
Page 228 NoPortProfile state. Bring the interface out of quarantine. no shutdown The interface comes back online. Verify if the interface is online. show interface vethernet Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 22-6 OL-28795-01...
Page 229: Network Segmentation Manager Troubleshooting Commands
| grep NSMGR Displays the system logs from the network segmentation manager. For detailed information about show command output, see the Cisco Nexus 1000V Command Reference, Release 4.2(1)SV2(1.1). Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 22-7...
Page 230 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 231: Information About Vxlans
VM connects to the network. A VXLAN supports three different modes for broadcast, multicast, and MAC distribution mode transport: Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) OL-28795-01 23-1...
Page 232: Vxlan Tunnel Endpoint
A VXLAN trunk allows you to trunk multiple VXLANs on a single virtual Ethernet interface. In order to achieve this configuration, you must encapsulate a VXLAN-VLAN mapping on the virtual Ethernet interface. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-2 OL-28795-01...
Page 233: Multi-Mac Capability
VXLAN encapsulation and if the frame does not contain an IP packet. Scalability Maximum Number of VXLANs The Cisco Nexus 1000V supports a total of 4096 VLANs or VXLANs (or a maximum of 2048 VLANs or 2048 VXLANs in any combination that totals 4096). Supported Features...
Page 234: Disabling The Vxlan Feature Globally
Bridge-domain vxlan-home (2 ports in all) Segment ID: 5555 (Manual/Active) Group IP: 235.5.5.5 State: UP Mac learning: Enabled is_bd_created: Yes current state: SEG_BD_FSM_ST_READY pending_delete: 0 port_count: 2 action: 4 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-4 OL-28795-01...
Page 235: Vxlan Gateway Commands
Ucast Mcast/Repl Ucast Mcast Total Encaps Encaps Decaps Decaps Drops 8717 8334 switch(vem-attach)# switch(vem-attach)# vemcmd show vxlan-stats ltl 17 VXLAN Port Stats for LTL 17 Unicast Encapsulations: 8756 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-5 OL-28795-01...
Page 236 10.106.199.116 To verify the port configuration on VSM: switch# show int switchport | begin Vethernet2 Name: Vethernet2 Switchport: Enabled Switchport Monitor: Not enabled Operational Mode: access Access Mode VLAN: 0 (none) Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-6 OL-28795-01...
Page 237: Vem Commands
To verify bridge domain creation on the VEM: ~ # vemcmd show bd bd-name vxlan-home BD 31, vdc 1, segment id 5555, segment group IP 235.5.5.5, swbd 4098, 1 ports, "vxlan-home" Portlist: RedHat_VM1.eth0 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-7 OL-28795-01...
Page 238 26 brtmax 4096, brtcnt 3, timeout 300 Segment ID 9001, swbd 4102, "segment-cisco" Flags: P - PVLAN S - Secure D - Drop Type MAC Address timeout Flags PVLAN Remote IP SwInsta 00:50:56:83:01:4e 10.106.199.117 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-8 OL-28795-01...
Page 239: Vem Packet Path Debugging
Note: You can compare the download sequence number against the VTEP download sequence number using the vemcmnd show bd bd-name. To verify if the MAC address table displays the remote IP learning in the segment-cisco bridge domain: switch# vemcmd show l2 bd-name segment-cisco Note - Use the module command to check the details of VEM and gateway on the VSM.
Page 240: Vem Multicast Debugging
You can view the output for all the above logs by using the module vem 4 execute vemlog show all command. VEM Multicast Debugging Use the following command to debug VEM multicast. IGMP state on the VEM: • vemcmd show igmp vxlan_transport_vlan detail Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-10 OL-28795-01...
Page 241: Vxlan Datapath Debugging
"debug dpa_allplatform all" > /tmp/dpafifo To debug the bridge domain configuration, use the following command: echo “debug sfl2agent all” > /tmp/dpafifo To debug port configuration, use the following command: echo “debug sfportagent all” > /tmp/dpafifo Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-11 OL-28795-01...
Page 242: Vempkt
To display detailed per-port statistics for VXLAN vmknic, use the following command: vxlan_vmknic_ltl vemcmd show vxlan-stats ltl To display detailed per-port statistics for vEthernet in a VXLAN, use the following command: Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-12 OL-28795-01...
Page 243: Show Commands
Displays the remote IP being learned. vemcmd show l2 bd-name bd-name-string Displays the Layer 2 table for one segment bridge domain. vemcmd show arp all Displays the IP-MAC mapping for the outer encapsulated header. Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) 23-13 OL-28795-01...
Page 244 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 245: Cisco Trustsec
This packet classification is maintained by tagging packets on ingress to the Cisco TrustSec network so that they can be properly identified for the purpose of applying security and other policy criteria along the data path. The tag, also called the security group tag (SGT), allows the network to enforce the access control policy by enabling the endpoint device to act upon the SGT to filter traffic.
Page 246: Cisco Trustsec Troubleshooting Commands
Table 24-1 Cisco TrustSec Debugging Commands Command Purpose debug cts authentication Collect and view logs related to Cisco TrustSec authentication. debug cts authorization Collect and view logs related to Cisco TrustSec authorization. debug cts errors Collect and view logs related to Cisco TrustSec errors and warning messages.
Page 247: Host Logging Commands
S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Host Logging Commands Table 24-2 lists the commands from the ESX host to collect and view logs related to Cisco TrustSec. Table 24-2 ESX Host Commands...
Page 248: Show Commands
10.78.1.76 49 353 7 6766 Device Tracking switch# Show Commands Table 24-3 lists available Cisco TrustSec show commands. See the Cisco Nexus 1000V Command Reference, Release 4.2(1)SV2(1.1) for more information on the show commands for Cisco TrustSec. Table 24-3 Cisco TrustSec Show Commands...
Page 249 Possible Causes Verification and Solution The Cisco Nexus 1000V is There is no connection between Cisco Verify if the Cisco Nexus 1000V is connected to unable to form a SXP session Nexus 1000V and its peer. its peer. with Cisco TrustSec.
Page 250 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
Page 251: Vcenter Plug-In
Server system to manage a Cisco Nexus 1000V through a browser. The vCenter Plug-in is installed as a new tab called Cisco Nexus 1000v as part of the user interface in the vSphere Web client. With the vCenter Plug-in, the server administrators can export the necessary networking details from the vCenter server, investigate the root cause of and prevent the networking issues, and deploy the virtual machines with the suitable policies.
Page 252: Generating A Log Bundle
– The vSphere Web Client requires the Adobe Flash Player version 11.1.0 or later to be installed. • Make sure that Cisco Nexus 1000V Release 4.2(1)SV2(1.1) is installed and configured to a vCenter. • Generating a Log Bundle You can collect the diagnostic information for VMware vCenter Server by collecting vSphere log files into a single location.
Page 253: Ethanalyzer
Using Ethanalyzer, page 26-1 Using Ethanalyzer Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark (formerly Ethereal) open source code. Ethanalyzer is a command-line version of Wireshark that captures and decodes packets. You can use Ethanalyzer to troubleshoot your network and analyze the control-plane traffic.
Page 254 Opens a captured data file and analyzes it. Ethanalyzer does not capture data traffic that Cisco NX-OS forwards in the hardware. Ethanalyzer uses the same capture filter syntax as tcpdump. For more information, see the following URL: http://www.tcpdump.org/tcpdump_man.html...
Page 255: Before Contacting Technical Support
• Gathering Information for Technical Support At some point, you may need to contact your customer support representative or Cisco TAC for some additional assistance. This section outlines the steps that the you should perform prior to contacting your next level of support, as this will reduce the amount of time spent resolving the issue.
Page 256: Obtaining A File Of Core Memory Information
Obtaining a File of Core Memory Information Cisco customer support engineers often use files from your system for analysis. One of these is a file containing memory information, and is referred to as a core dump. The file is sent to a TFTP server or to a Flash card in slot0: of the local switch.
Page 257: Copying Files
It may be required to move files to or from the switch. These files may include log, configuration, or firmware files. Cisco Nexus 1000V always acts as a client, such that an ftp/scp/tftp session will always originate from the switch and either push files to an external system or pull files from an external system.
Page 258 Backing up the startup-configuration to a server should be done on a daily basis and prior to any changes. A short script could be written to be run on Cisco Nexus 1000V to perform a save and then backup of the configuration. The script only needs to contain two commands: copy running-configuration startup-configuration and then copy startup-configuration tftp://server/name.
Page 259 VSM and vCenter Server problem symptoms and finding for specific DVS solutions 21-5 unregister in vCenter Server 3-12 core dumps 27-2 CPU status monitoring customer support fragmentation collecting information 27-1 VXLANs 23-3 contacting Cisco or VMware Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) IN-I OL-28795-01...
Page 260 ISSU upgrade unlicensed problem symptoms and solutions virtual Ethernet module (VEM) virtual supervisor module (VSM) MS-NLB 11-12 MTU settings, with jumbo frames 21-6 jumbo frames and MTU 21-6 multicast description 18-1 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) IN-II OL-28795-01...
Page 261 QoS policies on the VEM 16-2 10-5 port groups Quality of Service. See QoS assigning to VSM VM virtual interfaces port profiles commands to troubleshoot 4-4, 9-6 RADIUS Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) IN-III OL-28795-01...
Page 262 21-12 using CLI commands to troubleshoot 21-11 system processes domain parameters monitoring physical ports 11-2 status 11-13 verifying correct configuration 7-14 view of ports 11-2 virtual ports 11-2 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) IN-IV OL-28795-01...
Page 263 3-12 removing hosts from 3-11 status 11-12 verifying correct configuration view of ports 11-3 vSphere Client error messages 21-12 VXLANs fragmentation 23-3 maximum number 23-3 overview 23-1 scalability 23-3 Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1) IN-V OL-28795-01...
Page 264 S e n d d o c u m e n t c o m m e n t s t o n e x u s 1 k - d o c f e e d b a c k @ c i s c o . c o m . Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)

Cisco Nexus 1000V Troubleshooting Manual

Overview of Troubleshooting

Tools Used in Troubleshooting

Installation

Licenses

Upgrade

High Availability

VSM and VEM Modules

Ports

Port Profiles

Port Channels and Trunking

Layer 2 Switching

Vlans

Private Vlans

Netflow

Acls

Quality of Service

Span

Multicast IGMP

DHCP, DAI, and IPSG

Virtual Service Domain

System

Network Segmentation Manager

Cisco Trustsec

Vcenter Plug-In

Ethanalyzer

Before Contacting Technical Support

Quick Links

Troubleshooting

Related Manuals for Cisco Nexus 1000V

Summary of Contents for Cisco Nexus 1000V