For configuration recovery, regularly back up the configuration to a remote server. When you troubleshoot HP FlexFabric 12900 switches, follow these general guidelines: To help identify the cause of the problem, collect system and configuration information, including: •...
# Save log messages to the log file on each non-default MDC. <Sysname> system-view [Sysname] switchto mdc mdc2 ****************************************************************************** * Copyright (c) 2010-2014 Hewlett-Packard Development Company, L.P. * Without the owner's prior written consent, * no decompiling or reverse-engineering shall be allowed. ****************************************************************************** <HP> logfile save...
Page 7
* Copyright (c) 2010-2014 Hewlett-Packard Development Company, L.P. * Without the owner's prior written consent, * no decompiling or reverse-engineering shall be allowed. ****************************************************************************** <HP> logfile save The contents in the log file buffer have been saved to the file flash:/logfile/logfile.log <HP> switchback...
Page 8
<Sysname> dir flash:/mdc/mdc3/logfile/ Directory of flash:/mdc/mdc3/logfile 0 -rw- 465 Jul 11 2013 16:10:39 logfile.log 1021104 KB total (421476 KB free) Transfer the files to the desired destination by using FTP or TFTP. (Details not shown.) Release 11xx Save common log messages from the log buffer to a log file. By default, the log file is saved in the logfile directory of the flash memory on the active MPU (in standalone mode) or global active MPU (in IRF mode).
Page 9
* Copyright (c) 2010-2014 Hewlett-Packard Development Company, L.P. * Without the owner's prior written consent, * no decompiling or reverse-engineering shall be allowed. ****************************************************************************** <HP> diagnostic-logfile save The contents in the diagnostic log file buffer have been saved to the file flash:/diagfile/diagfile.log <HP> switchback...
Page 10
In IRF mode, display the diagnostic log file on each standby MPU. <Sysname> dir chassis2#slot17#flash:/diagfile/ Directory of chassis2#slot17#flash:/diagfile 0 -rw- 161321 Jul 11 2013 16:16:00 diagfile.log 1021104 KB total (421416 KB free) NOTE: If a subordinate chassis has two MPUs, make sure you identify and export the diagnostic log files on both MPUs.
Network and management ports are faster than the console port. • Do not execute commands while operating statistics are being collected. HP recommends saving operating statistics to a file to retain the information. • NOTE: The amount of time to collect statistics increases along with the number of cards.
None Reboot Cause UserReboot …… Contacting technical support If you cannot resolve a problem after using the troubleshooting procedures in this document, contact HP Support. When you contact an authorized HP support representative, be prepared to provide the following information:...
Technical support registration numbers (if applicable). • This information will help the support engineer assist you as quickly as possible. The following is the contact of HP Support for networking products: http://www.hp.com/networking/support Removing deployment errors Use the deployment checklist in...
Page 14
Question Command or method Result Remarks Active and standby MPUs □OK Are the software versions of Make sure the software the active and standby display boot-loader □Not OK versions of the active and MPUs the same? standby MPUs are the same. □Not related If any standby MPUs do not have a startup configuration...
Page 15
• Port setting inconsistencies with the peer port. Fiber ports To connect an HP device to a device from another □OK Do the ports at the two ends display current-configuration vendor, HP recommends □Not OK...
Page 16
Remove ports from VLAN 1 □Not OK enable vlan 1 as needed. □Not related Spanning tree feature HP recommends that you set □OK a timeout factor in the range Is the timeout factor □Not OK display current-configuration of 5 to 7 on a stable network...
Page 17
IEEE spanning tree protocols. • Set up a Layer 3 connection to the Cisco PVST+ device if the HP device is operating in MSTP, RSTP, or STP mode. If the topologies deviate from the design, reassign ports to VLANs and revise...
Page 18
Question Command or method Result Remarks Examine the following items in the command output for TC attacks: • Incoming and outgoing TC/TCN BPDU statistics. • Historical port role calculation information. There is a risk of TC attack if frequent STP status changes occur on a stable network.
Page 19
Question Command or method Result Remarks Change the router ID if it is not unique on the network. □OK To restart route learning Is the router ID of the device □Not OK display ospf peer after you remove the router unique on the network? ID conflict, you must execute □Not related...
Records in the local log buffer Execute the local logbuffer display command in probe view. If the local log buffer contains exception records, contact HP Support to • troubleshoot the exceptions. In standalone mode: local logbuffer slot Use the following □OK...
Troubleshooting flowchart Figure 1 Troubleshooting MPU reboot failure MPU reboot failure System software Reload the system Resolved? image correct software image MPU memory Replace the MPU Resolved? runs correctly? Error continues Reinstall the MPU Resolved? to be reported? securely Contact the support Solution To resolve the problem: Verify that the system software image on the MPU is correct.
If the MPU is installed securely, replace the MPU. If the problem persists, contact HP Support. Newly installed standby MPU boot failure Symptom An MPU designated as the standby MPU fails to boot after installation. Troubleshooting flowchart Figure 2 Troubleshooting newly installed standby MPU boot failure...
LPU reboot failure Symptom An LPU fails to reboot. Troubleshooting flowchart Figure 3 Troubleshooting LPU reboot failure LPU reboot failure Reload the system System software Resolved? software image image correct? Another LPU boots Replace the MPU Resolved? successfully in the slot? Install the LPU into Loading records Resolved?
If the new LPU fails to boot, replace the MPU. Verify that the LPU has loaded files. <Sysname>display logbuffer %May 3 13:27:17:086 2013 HP DEVM/4/BOARD_LOADING: Board is loading file on Chassis 1 Slot 7. %May 3 13:27:17:647 2013 HP DEVM/5/LOAD_FINISHED: Board has finished loading file on Chassis 1 Slot 7.
Troubleshooting flowchart Figure 4 Troubleshooting newly installed LPU boot failure Newly installed LPU boot failure System software Reload the system Resolved? image correct? software image System software image Upgrade the system Resolved? supports the LPU? software image Another LPU boots Replace the MPU Resolved? successfully in the slot?
If the LPU fails to boot, replace the MPU. Verify that the LPU has loaded files. [Sysname]display logbuffer %May 3 13:27:17:086 2013 HP DEVM/4/BOARD_LOADING: Board is loading file on Chassis 1 Slot 7. %May 3 13:27:17:647 2013 HP DEVM/5/LOAD_FINISHED: Board has finished loading file on Chassis 1 Slot 7.
Troubleshooting flowchart Figure 5 Troubleshooting operating power supply failure Operating power supply failure The power supply Remove and reinstall Resolved? in Absent state? the power supply Power cord Connect the power Resolved? connected securely? cord securely Make sure the The temperature temperature is in the Resolved? too high?
Normal state in the new slot. If the power supply remains in Fault state, replace the power supply. If the problem persists, contact HP Support. Newly installed power supply failure Symptom A newly installed power supply fails.
Troubleshooting flowchart Figure 6 Troubleshooting newly-installed power supply failure Newly installed power supply failure The power supply Remove and reinstall Resolved? in Absent state? the power supply Power cord Connect the power Resolved? connected securely? cord securely Contact the support Solution To resolve the problem: Identify the operating state of the power supply.
Normal state in the new slot. If the power supply remains in Fault state, go to step 4. If the problem persists, contact HP Support. Fan tray failure Symptom An operating fan tray or a newly installed fan tray fails.
Page 31
You must make sure the switching operating temperature is below 60°C (140°F) while you replace the fan tray. If a new fan tray is not readily available, power off the switch to avoid damage caused by high temperature. If the problem persists, contact HP Support.
Related commands This section lists the commands that you might use for troubleshooting the hardware. Command Description Displays information about files and directories. display alarm Displays alarm information. Displays current configuration files and system software display boot-loader images. display environment Displays temperature information.
Contact the support Solution To resolve the problem: Identify the job that has a high CPU utilization. For example: [HP-probe] display process cpu slot 1 CPU utilization in 5 secs: 2.4%; 1 min: 2.5%; 5 mins: 2.4% 5Sec 1Min 5Min Name 0.0%...
[<ffffffff80260730>] kthread+0x130/0x140 [<ffffffff8021aec0>] kernel_thread_helper+0x10/0x20 Save the information displayed in the previous steps. Contact HP Support. High memory utilization on a card Symptom The memory utilization on a card is over 60% for more than 30 minutes. Troubleshooting flowchart Figure 9 Troubleshooting high memory utilization...
Page 36
16777160 0 4096 kmalloc-16777160 2048 sgpool-64 inotify_event_cache 16328 kmalloc_dma-16328 LFIB_IlmEntryCache 1080 LFIB_IlmEntryCache 1464 MFW_FsCache L2VFIB_Ac_cache CCF_JOBDESC NS4_Aggre_TosSrcPre IPFS_CacheHash_cachep ---- More ---- Each value line shows the memory information for a slab. The Number field shows the number of objects (memory blocks) allocated to the module. The Active field shows the number of objects used.
Save the information displayed in the previous steps. Contact HP Support. IMPORTANT: HP recommends not rebooting the device before you contact HP Support. You might lose critical diagnostic information if you reboot the device. Temperature alarms Symptom Temperature alarms occur.
Check the device temperature. If the device temperature is higher than the high-temperature warning threshold, use the display fan command to verify that the fan tray is operating correctly. If the system displays Fault, see HP FlexFabric 12900 Switch Series Installation Guide to resolve the problem.
Replace the local port with a new one (on a card of the same type, if possible) to verify that the local port is operating correctly. Replace the peer port with a new one (on a card of the same type, if possible) to verify that the peer port is operating correctly. If the problem persists, contact HP Support.
1000-Mbps SFP fiber port fails to come up Symptom A 1000-Mbps SFP fiber port fails to come up. Troubleshooting flowchart Figure 12 Troubleshooting link up failure on a port A port failed to come up Speed/duplex match Configure matching Resolved? on local and peer port? speed/duplex Speed/duplex...
Execute the display transceiver interface command to verify that the local transceiver module has the same wavelength and transmission distance as the peer transceiver module. If the transceiver module is not operating correctly, replace it with a new HP transceiver module that matches the fiber port.
Troubleshooting flowchart Figure 13 Troubleshooting link up failure on a port A port failed to come up Speed/duplex match Configure matching Resolved? on local and peer port? speed/duplex Speed/duplex Configure matching match on transceiver module and Resolved? speed/duplex port? Local/peer port Replace local/peer port Resolved? operating correctly?
Execute the display transceiver interface command to verify that the local transceiver module has the same wavelength and transmission distance as the peer transceiver module. If the transceiver module or cable is not operating correctly, replace it with a new HP transceiver module or cable that matches the fiber port.
Troubleshooting flowchart Figure 14 Troubleshooting link up failure on a port A port failed to come up Local/peer port Replace local/peer port Resolved? operating correctly? Transceiver Replace transceiver Resolved? module/cable correct? module/cable Fiber correct? Replace fiber Resolved? Contact the support Solution To resolve the problem: Verify that the local and peer ports have not failed:...
If the transceiver module or cable is not operating correctly, replace it with a new HP transceiver module or cable that matches the fiber port. For more information about transceiver modules and cables, see the installation guide for the switch.
Execute the display transceiver interface command to verify that the local transceiver module has the same wavelength and transmission distance as the peer transceiver module. If the transceiver module is not operating correctly, replace it with a new HP transceiver module that matches the fiber port.
Verify that the transceiver module is an HP transceiver module: Execute the display transceiver interface command to view the vendor name of the transceiver module. If the vendor name field does not display HP, replace the transceiver module with an HP transceiver module.
Verify that the transceiver module is an HP transceiver module: Execute the display transceiver interface command to view the vendor name of the transceiver module. If the vendor name field does not display HP, replace the transceiver module with an HP transceiver module.
Troubleshooting flowchart Figure 18 Troubleshooting error frames (for example, CRC errors) on a port Error frames on the port Determine the type of error frames accumulating Optical power Replace it with a transceiver of transceiver module with correct optical Resolved? module correct? power Speed and...
Page 50
If the port is a fiber port, verify that the optical power of the transceiver module is operating correctly: Use the display transceiver diagnosis interface command in probe view to view the present measured values of the digital diagnosis parameters for the transceiver module. [Sysname-probe] display transceiver diagnosis interface ten-gigabitethernet 1/0/1 Ten-GigabitEthernet1/0/1 transceiver diagnostic information: Current diagnostic parameters:...
Verify that the incoming traffic and outgoing traffic have not exceeded the maximum traffic processing capability of the local device and the peer device. If the problem persists, contact HP Support. Failure to receive packets Symptom A port is up, but it cannot receive packets.
If the port is set to the discarding state by the spanning tree feature, examine and modify the spanning tree feature configurations to resolve the problem. HP recommends that you configure the port as an edge port or disable the spanning tree feature on the port if it is directly connected to a terminal.
Troubleshooting flowchart Figure 20 Troubleshooting failure to send packets A port failed to send packets View packet statistics of the port Port Modify port configurations affect packet Resolved? configurations sending? Port and link Use correct port and Resolved? medium correct? link medium Contact the support Solution...
If the port is set to the discarding state by the spanning tree feature, examine and modify the spanning tree feature configurations to resolve the problem. HP recommends that you configure the port as an edge port or disable the spanning tree feature on the port if it is directly connected to a terminal.
Troubleshooting IRF This section provides troubleshooting information for common IRF problems. IRF fabric setup failure Symptom A chassis cannot be added to an IRF fabric.
Troubleshooting flowchart Figure 21 Troubleshooting IRF fabric setup failure IRF setup failure Number of chassis in the You cannot add the chassis to fabric fewer than the upper limit? the fabric Replace with a chassis that is Same chassis model? the same model as existing Resolved? IRF member chassis...
Page 57
Verify that the chassis is the same model as the chassis in the IRF fabric. The HP FlexFabric 12900 switches must be the same model to form an IRF fabric. Verify that the chassis is operating in IRF mode: Execute the display interface brief command to verify that the physical interface numbers on the chassis have changed from three segments to four segments.
IPv6 Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 0023-8912-3d07 If the chassis has the same bridge MAC address as the IRF fabric, remove the bridge MAC conflict. If the problem persists, contact HP Support. IRF split Symptom An IRF fabric splits.
Troubleshooting flowchart Figure 22 Troubleshooting IRF split IRF fabric splits IRF physical Identify the cause and Resolved? interfaces down? resolve the problem LPU that has IRF Identify the cause and Resolved? links rebooted? resolve the problem Member chassis Identify the cause and Resolved? rebooted? resolve the problem...
"Operating power − supply failure" to resolve the problem. If the problem persists, contact HP Support. BFD MAD failure Symptom BFD MAD fails to detect an IRF split event. Two IRF fabrics are operating with the same Layer 3 configurations, including the same IP address.
Troubleshooting flowchart Figure 23 Troubleshooting BFD MAD failure BFD MAD failure BFD MAD link Correct the connections Resolved? connections correct? All BFD MAD Bring up the ports Resolved? ports up? BFD MAD VLAN interface Correct the settings Resolved? configured correctly? BFD MAD VLAN and ports Correct the settings Resolved?
Page 62
If you do not use an intermediate device, verify that each pair of member chassis has a dedicated BFD MAD link. If you use an intermediate device, verify that each member chassis has a dedicated BFD MAD link with the intermediate device. Verify that all physical ports used for BFD MAD are up: Execute the display interface command.
Identify the cause of the events, and remove the issue. If the problem persists, contact HP Support. LACP MAD failure Symptom LACP MAD fails to detect an IRF split event. Two IRF fabrics are operating with the same Layer 3 configurations, including the same IP address.
Solution To resolve the problem: Verify that the intermediate device is an HP device that supports extended LACPDUs for MAD. If the intermediate device does not support extended LACPDUs for MAD, replace the intermediate device, or use BFD MAD for split detection.
Command Description display interface Displays interface information. display interface brief Displays brief interface information. Displays IRF fabric information, including the member ID, role, display irf priority, bridge MAC address, and description of each IRF member. display irf configuration Displays the IRF configuration on each member chassis. display irf topology Displays the IRF topology.
Apply the packet filter or QoS policy again. Identify the unsupported criterion according to the new error message. If the rule contains only one criterion, the criterion is not supported. If the problem persists, contact HP Support. ACL application failure for insufficient resources Symptom The system fails to apply a packet filter or an ACL-based QoS policy to the hardware.
Delete unnecessary ACLs that use Meter or Counter resources if the failure is caused by Meter or Counter resource insufficiency. If the ACL resources are sufficient, go to step 4. Apply the packet filter or QoS policy again. If the problem persists, contact HP Support. ACL application failure without an error message Symptom The system applies a packet filter or an ACL-based QoS policy to the hardware.
Troubleshooting flowchart Figure 27 Troubleshooting ACL application failure Failure to apply an ACL Overlapping ACLs exist? Conflicting behaviors Revise ACLs/packet for packets matching Resolved? filters/QoS policies overlapping ACLs? Contact the support Solution To resolve the problem: Check the ACLs used by QoS policies and packet filters for overlapping rules: Use the following commands to display ACLs that are used by QoS policies and packet filters: display packet-filter −...
The behavior configured first. • filter deny Revise ACLs, packet filters, or QoS policies to remove the behavior conflict. If the problem persists, contact HP Support. Related commands This section lists the commands that you might use for troubleshooting QoS and ACLs. Command...
Troubleshooting flowchart Figure 28 Troubleshooting LPU authorization failure LPU authorization failure Resources sufficient? Re-assign resources Resolved? Reclaim the LPU from the LPU belongs to the default default MDC, and assign the Resolved? MDC? LPU to the non-default MDC Contact the support Solution Typical reasons for LPU authorization failures include: The available hardware resources are insufficient.
Page 71
Name Quota(MB) Used(MB) Available(MB) ---------------------------------------------------------------- Admin 680.7 16.9 663.8 Chassis 3 Slot 5: Used 16.9MB, Free 663.8MB, Total 680.7MB: Name Quota(MB) Used(MB) Available(MB) ---------------------------------------------------------------- Admin 680.7 16.9 663.8 Chassis 3 Slot 9: Used 16.9MB, Free 663.9MB, Total 680.8MB: Name Quota(MB) Used(MB) Available(MB) ----------------------------------------------------------------...
3 location chassis 3 slot 3 mdc start Return If the problem persists, contact HP Support. Interface assignment failure Symptom After an LPU is assigned to an MDC, the allocate command fails to assign interfaces on the LPU or the...
Troubleshooting flowchart Figure 29 Troubleshooting IRF fabric establishment failure Interface assignment failure Display the interfaces of MDCs Reclaim the LPU and its Interfaces interface and re-assign Resolved? assigned to another MDC? them All interfaces on Assign all interfaces on the LPU assigned to the same MDC the LPU to the same MDC Resolved? at the same time?
Page 74
GigabitEthernet3/3/0/10 GigabitEthernet3/3/0/11 GigabitEthernet3/3/0/12 GigabitEthernet3/3/0/13 GigabitEthernet3/3/0/14 GigabitEthernet3/3/0/15 GigabitEthernet3/3/0/16 GigabitEthernet3/3/0/17 GigabitEthernet3/3/0/18 GigabitEthernet3/3/0/19 GigabitEthernet3/3/0/20 GigabitEthernet3/3/0/21 GigabitEthernet3/3/0/22 GigabitEthernet3/3/0/23 GigabitEthernet3/3/0/24 Ten-GigabitEthernet3/3/0/25 Ten-GigabitEthernet3/3/0/26 MDC mdcB's interface(s): M-GigabitEthernet1/0/0/0 [Sysname-mdc-2-mdcA]display this mdc mdcA id 2 location chassis 3 slot 3 mdc start allocate interface GigabitEthernet3/3/0/1 to GigabitEthernet3/3/0/24 Ten-GigabitEthernet3/3/0/25 Ten-GigabitEthernet3/3/0/26 The output shows that the LPU in slot 3 on member device 3 and its interfaces belong to mdcA.
GigabitEthernet3/3/0/20 GigabitEthernet3/3/0/21 GigabitEthernet3/3/0/22 GigabitEthernet3/3/0/23 GigabitEthernet3/3/0/24 Ten-GigabitEthernet3/3/0/25 Ten-GigabitEthernet3/3/0/26 If the problem persists, contact HP Support. Related commands This section lists the commands that you might use for troubleshooting MDC. Command Description allocate interface Assigns a physical interface to an MDC. Displays operating statistics for multiple feature modules in display diagnostic-information the system.