Page 1
T R O U B L E S H O O T I N G G U I D E Alcatel-Lucent Proprietary This document contains proprietary information of Alcatel-Lucent and is not to be disclosed or used except in accordance with applicable agreements.
Page 2
The customer hereby agrees that the use, sale, license or other distribution of the products for any such application without the prior written consent of Alcatel-Lucent, shall be at the customer's sole risk. The customer hereby agrees to defend and hold Alcatel-Lucent harmless from any claims for loss, cost, damage, expense or liability that may arise out of or in connection with the use, sale, license or other distribution of the products in such applications.
Page 3
Alcatel-Lucent License Agreement SAMPLE END USER LICENSE AGREEMENT 1. LICENSE Subject to the terms and conditions of this Agreement, Alcatel-Lucent grants to Customer and Customer accepts a nonexclusive, nontransferable license to use any software and related documentation provided by Alcatel-Lucent pursuant to this Agreement ("Licensed Program") for Customer's own internal...
Page 4
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 Alcatel-Lucent may terminate this Agreement: (a) upon notice to Customer if any amount payable to Alcatel-Lucent is not paid within thirty (30) days of the date on which payment is due; (b) if Customer becomes bankrupt, makes an assignment for the benefit of its creditors, or if its assets vest or become subject to the rights of any trustee, receiver or other administrator;...
Page 5
Alcatel-Lucent infringes any patent, copyright, trade secret or other intellectual property right, provided that Customer notifies Alcatel-Lucent within ten (10) days of the existence of the claim, gives Alcatel-Lucent sole control of the litigation or settlement of the claim, and provides all such assistance as Alcatel-Lucent may reasonably require.
Page 6
The Licensed Program may contain freeware or shareware obtained by Alcatel-Lucent from a third party source. No license fee has been paid by Alcatel-Lucent for the inclusion of any such freeware or shareware, and no license fee is charged to Customer for its use.
Page 7
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 No term or provision of this Agreement shall be deemed waived and no breach excused unless such waiver or consent is in writing and signed by the party claimed to have waived or consented.
Page 9
Preface The Preface provides general information about the 5620 Service Aware Manager documentation suite, including this guide. Prerequisites Readers of the 5620 SAM documentation suite are assumed to be familiar with the following: • 5620 SAM software structure and components •...
Page 10
The 5620 SAM online help system includes a Parameter Search Tool that allows you to look up parameters or 5620 SAM and 5650 CPAM forms. See the 5620 SAM User Guide for more information about using the Parameter Search Tool.
Page 11
The 5620 SAM MPR User Guide describes how to discover, configure, and manage 9500 MPR and 9500 MPRe devices using the 5620 SAM. The guide is intended for network planners, administrators, and operators and is to be used in conjunction with other guides in the 5620 SAM documentation suite where management of 9500 MPR and 9500 MPRe devices does not differ from other network elements.
Page 12
5620 SAM-O schema in the context of LTE, compliance with 3GPP standards, and typical operational scenarios. 5620 SAM LTE Parameter The 5620 SAM LTE Parameter Reference provides a list of all LTE ePC and LTE RAN Reference parameters supported in the 5620 SAM.
Page 13
5620 SAM product alerts Product technical alerts are used to communicate important information to customers about released loads. You can view and subscribe to product alerts for the 5620 SAM from the Alcatel-Lucent Customer Support Center. Alcatel-Lucent 5620 Service Aware Manager...
Page 14
You can use the Search function of Acrobat Reader (File→Search) to find a term in a PDF of any 5620 SAM document. To refine your search, use appropriate search options (for example, search for whole words only or enable case-sensitive searching).
Page 15
Table lists the measurement symbols used in this document. Table 3 Bits and bytes conventions Measurement Symbol byte byte kilobits per second kb/s Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Caution — Caution indicates that the described activity or situation may, or will, cause service interruption. Note — Notes provide information that is, or may be, of special interest. Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Page 17
5620 SAM troubleshooting overview..........1-2 Troubleshooting the managed-NE network........1-2 Troubleshooting the network management domain ......1-3 Troubleshooting the 5620 SAM platform, database, server, or clients ..1-3 Troubleshooting process ............... 1-4 Network maintenance ............... 1-4 Troubleshooting problem-solving model.......... 1-4 Troubleshooting guidelines ............
Page 18
Contents 5620 SAM log files ..............1-8 User activity log ..............1-8 Before you call support ..............1-9 Workflow to troubleshoot a problem in the 5620 SAM ......1-9 Troubleshooting the managed network 2 — Troubleshooting using network alarms Troubleshooting using network alarms ..........2-2 Workflow to troubleshoot using network alarms ........
Page 19
Procedure 6-2 Problem: Lost connectivity to one or more network management domain stations ..........6-3 Procedure 6-3 Problem: Another station can be pinged, but some functions are unavailable ............. 6-4 Procedure 6-4 Problem: Packet size and fragmentation issues ....6-5 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Page 20
Procedure 9-1 To identify the user activity for a network object ... 9-3 Procedure 9-2 To identify the user activity for a 5620 SAM object ..9-3 Procedure 9-3 To navigate to the object of a user action ....9-4 Procedure 9-4 To view the user activity records of an object ....
Page 21
5620 SAM server status ........... 13-3 Procedure 13-2 Problem: 5620 SAM server and database not communicating ..............13-7 Procedure 13-3 Problem: A 5620 SAM server starts up, and then quickly shuts down ............13-7 Alcatel-Lucent 5620 Service Aware Manager...
Page 22
5620 SAM, or alarm performance is degraded ......13-10 Procedure 13-8 Problem: All SNMP traps from managed devices are arriving at one 5620 SAM server, or no SNMP traps are arriving ..13-11 Procedure 13-9 Problem: Cannot manage new devices ....13-11 Procedure 13-10 Problem: Cannot discover more than one device, or device resynchronization fails .......
Page 23
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 Troubleshooting overview 1 — 5620 SAM troubleshooting Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Page 24
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
1.1 5620 SAM troubleshooting overview 1.2 Troubleshooting process 1.3 Troubleshooting guidelines 1.4 5620 SAM troubleshooting tools 1.5 Before you call support 1.6 Workflow to troubleshoot a problem in the 5620 SAM Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
The 5620 SAM raises alarms against network objects in response to received SNMP traps from managed NEs. You can then use the 5620 SAM to correlate the events and alarms to the managed object, configured services and policies. A correlated event or alarm can cause fault conditions on multiple network objects and services.
1 — 5620 SAM troubleshooting Troubleshooting the network management domain The 5620 SAM has a number of powerful troubleshooting tools that help to quickly pinpoint the root cause of network and service management problems to speed resolution. Troubleshooting the client, server, or database component in a 5620 SAM network management domain requires familiarity with the following: •...
The maintenance of the baseline information is critical because a network is not a static environment. See the 5620 SAM System Administrator Guide for more information on how to generate baseline information for 5620 SAM applications.
Page 29
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 1 — 5620 SAM troubleshooting Categorize the problem When you categorize a problem, you must differentiate between total failures and problems that result in a degradation in performance. For example, the failure of an access switch results in a total failure for a customer who has one DS3 link into a network.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 1 — 5620 SAM troubleshooting The following 5620 SAM features can help you identify the root cause of a problem: • alarms with vendor-specific and X.733 standardized probable causes •...
• Perform troubleshooting based on your network requirements. 5620 SAM troubleshooting tools The 5620 SAM supports a number of troubleshooting tools and event logs to help identify the root cause of a network or network management problem. OAM diagnostics The 5620 SAM supports configurable in-band and out-of-band, packet-based OAM diagnostic tools for network troubleshooting and for verifying compliance with SLAs.
See the 5620 SAM User Guide for more information about the RCA audit tool. 5620 SAM log files You can use 5620 SAM log files to help troubleshoot your network. The log files can consume a large amount of disk space during a long period of significant activity.
The following is the high-level sequence of actions to perform with respect to the problem-solving model described in section 1.2. Establish an operational baseline for your network. See the 5620 SAM System Administrator Guide for more information. When a problem occurs, identify the type of problem. Table lists the general 5620 SAM troubleshooting types.
Page 34
Use Table to identify the appropriate 5620 SAM platform, database, server, or client troubleshooting procedure for the problem. Table 1-3 5620 SAM Managed NE network problems or tasks Problem or tasks Solution Troubleshooting with alarms...
Page 35
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 1 — 5620 SAM troubleshooting Problem or tasks Solution To verify connectivity for all egress points in a service using MAC Ping and MAC Procedure Trace To verify connectivity for all egress points in a service using MEF MAC Ping...
Page 36
To navigate to the object of a user action Procedure To view the user activity records of an object Procedure Table 1-5 5620 SAM platform, database, server, or client problems or tasks Problem or task Solution Troubleshooting 5620 SAM platform problems...
Page 37
13-6 Problem: Unable to receive alarms on the 5620 SAM, or alarm performance is degraded Procedure 13-7 Problem: All SNMP traps from managed devices are arriving at one 5620 SAM server, or no Procedure 13-8 SNMP traps are arriving Problem: Cannot manage new devices...
Page 38
Problem or task Solution Problem: Cannot clear alarms using 5620 SAM client GUI Procedure 14-13 Problem: The 5620 SAM client GUI does not display NE user accounts created, modified, or Procedure 14-14 deleted using the CLI (3 of 3) Verify the solution.
Page 39
Troubleshooting the managed network 2 — Troubleshooting using network alarms 3 — Troubleshooting services and connectivity 4 — Troubleshooting using topology maps 5 — Troubleshooting using the NE resync audit function Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Page 40
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
See the 5620 SAM Alarm Reference for information about a specific alarm. See the 5620 SAM User Guide for information about 5620 SAM alarm management using the Alarm Info and Alarm History forms.
Use the following procedures to troubleshoot network problems using alarms. Procedure 2-1 To view and sort alarms in the dynamic alarm list Monitor the dynamic alarm list in the 5620 SAM alarm window and attempt to address alarms in the order that they are raised.
Procedure 2-2 To view object alarms and aggregated object alarms You can use the navigation tree to view object alarm status, and aggregated alarm status for parent objects. See the 5620 SAM User Guide for more information about the relationship between objects, related alarms, and aggregated alarms.
Page 45
Therefore, the First Time Detected stamp is not a reliable indication of the exact time an event occurred and should be used only as an aid in troubleshooting. If you need more information about an alarm, see the 5620 SAM Alarm Reference. Alcatel-Lucent 5620 Service Aware Manager 5620 SAM...
Page 46
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 2 — Troubleshooting using network alarms Table 2-1 Hierarchy of 5620 SAM-managed objects Level Managed object Domain (class) — General network management or 5620 SAM objects Accounting (accounting)
Page 47
Service tunnel management (tunnelmgmt) Tunnel Ethernet tunnel (ethernettunnel) L2TP (l2tp) MPLS (mpls) Rules (rules) Service tunnel (svt) LSP binding MPLS (mpls) Session RSVP (rsvp) LDP interface or targeted peer LDP (ldp) (2 of 3) Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Right-click on the selected alarm in the dynamic alarm list and choose Acknowledge Alarm(s) from the contextual menu. The Alarm Acknowledgement form opens. If required, add text in the Acknowledgement Text box. Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Page 49
Click on the OK button. A dialog box appears. Click on the OK button to continue. A check mark appears for each of the selected alarms under the Ack. column in the dynamic alarm list. Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
The affected object is the root cause of the problem. The alarm probable • cause is the root cause. See the 5620 SAM Alarm Reference for additional information about the alarm, which may help to correct the problem. When the problem is fixed, all correlated alarms are cleared. See section for a sample equipment problem.
1/1/4 1/1/4 10.10.11.1/24 10.10.11.2/24 Customer Customer access access 7750 SR 7750 SR ports ports site ID site ID 10.1.200.52/32 10.1.200.53/32 BGP, OSPF, and MPLS are on each network interface. 17558 Alcatel-Lucent 5620 Service Aware Manager 2-11 5620 SAM Troubleshooting Guide...
The alarm cause is inoperable equipment. • Check the port states. Click on the Affected Objects tab, then click on the Properties button to view state and other information about the object in the alarm. 2-12 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Approximately one half hour later, a second Lsp Down alarm and a second Lsp Path Down alarm were raised for a total of 18 alarms. Click on the Object Type column to sort the alarms alphabetically according to object type. Alcatel-Lucent 5620 Service Aware Manager 2-13 5620 SAM Troubleshooting Guide...
Page 54
Open the equipment view of the navigation tree. It indicates that there are alarms related to both existing LAGs (Site Id 10.1.200.52 and Site Id 10.1.200.53). However, there is no LAG alarm in the dynamic alarm list and the LAG State is Up. 2-14 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
16 automatically clear. The remaining two alarms are Session alarms. Choose one of the remaining alarms in the dynamic alarm list and choose Show Affected Object from the contextual menu. The affected object properties form opens. Alcatel-Lucent 5620 Service Aware Manager 2-15 5620 SAM Troubleshooting Guide...
Administrative State is Up. • Operational State is MTU Mismatch. • MTU Mismatch is the root cause of the Frame Size Problem alarm. You do not need to investigate the related objects. 2-16 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
The maximum frame size configured exceeds the maximum frame size • supported for the service ingress and service egress termination points, which are also called the MTU. See the 5620 SAM Alarm Reference for additional information about the Frame Size Problem alarm. Perform procedure 2-12 to clear the Frame Size Problem alarm.
Page 58
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 2 — Troubleshooting using network alarms 2-18 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 3 — Troubleshooting services and connectivity 3.1 Troubleshooting services and connectivity 3.2 Workflow to troubleshoot a service or connectivity problem 3.3 Service and connectivity troubleshooting procedures Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
5620 SAM alarms. STM OAM diagnostics for troubleshooting You can use the 5620 SAM Service Test Manager, or STM, OAM diagnostic tools for network troubleshooting and for verifying compliance with SLAs The STM provides the ability to group OAM diagnostic tests into test suites for more comprehensive fault monitoring and troubleshooting.
Procedure for more information. Verify that the 5620 SAM service configuration aligns with the customer requirements. For example, ensure that 5620 SAM configuration uses the correct service type and SAP configuration, and that the circuit and site are included in the service.
Retrieve MIB information from a GNE using the snmpDump utility. See Procedure 3-13 for more information. Contact your Alcatel-Lucent technical support representative if the problem persists. See chapter for more information. Service and connectivity troubleshooting procedures Use the following procedures to perform service and connectivity troubleshooting.
3.2. If the operational state is Down and the administrative state is Up for one or more service components, the 5620 SAM generates an alarm. You must investigate the root problem on the underlying object. See chapter more information.
If there are no FIB entries, there is a configuration problem with the customer equipment or the connection from the equipment to the service tunnel. Confirm that the 5620 SAM service configuration aligns with the customer requirements. Confirm that there are no problems with the customer equipment and associated configuration.
3 — Troubleshooting services and connectivity Procedure 3-4 To verify connectivity for all egress points in a service using MAC Ping and MAC Trace Choose Tools→Service Test Manager (STM) from the 5620 SAM main menu. The Manage Tests form appears. Click on the Create button.
Page 66
MAC Ping diagnostic returned the expected result for the configuration of • your network MAC Trace diagnostic did not return the expected result for the • configuration of your network Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
3 — Troubleshooting services and connectivity Procedure 3-5 To verify connectivity for all egress points in a service using MEF MAC Ping Choose Tools→Service Test Manager (STM) from the 5620 SAM main menu. The Manage Tests form appears. Click on the Create button.
MTU Ping Record the maximum frame transmission size for the service. Choose Manage→Service Tunnels from the 5620 SAM main menu. The Manage Service Tunnels form appears. Filter to list only the source and destination routers of the service tunnel and click on the Search button.
Procedure 3-7 To verify the end-to-end connectivity of a service using Service Site Ping Choose Tools→Service Test Manager (STM) from the 5620 SAM main menu. The Manage Tests form appears. Click on the Create button. Choose Service→Create Service Site Ping from the Create contextual menu. The Service site ping create form appears with the General tab selected.
Page 70
Use the CLI to verify that the IP address of the destination site is in the routing table for the originating site by entering: show router route-table ↵ 3-12 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Procedure 3-8 To verify the end-to-end connectivity of a service tunnel using Tunnel Ping Choose Manage→Service Tunnels from the 5620 SAM main menu. The Manage Service Tunnels form appears. Filter to list only the source and destination routers of the service tunnel and click on the Search button.
Page 72
If the Tunnel Ping fails, go to step 7 b iii of the section troubleshooting workflow to verify the end-to-end connectivity of services using MPLS LSP paths, if required. 3-14 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Procedure 3-9 To verify end-to-end connectivity of an MPLS LSP using LSP Ping Choose Tools→Service Test Manager (STM) from the 5620 SAM main menu. The Manage Tests form appears. Click on the Create button. Choose MPLS→Create LSP Ping from the Create contextual menu. The LSP Ping (Create) form appears with the General tab selected.
Alcatel-Lucent technical support representative. See chapter more information. Procedure 3-10 To review the route for an MPLS LSP using LSP Trace Choose Tools→Service Test Manager (STM) from the 5620 SAM main menu. The Manage Tests form appears. Click on the Create button.
Perform the steps of the section troubleshooting workflow. If the troubleshooting workflow does not identify the problem with your service, contact your Alcatel-Lucent technical support representative. See chapter more information. Procedure 3-11 To review ACL filter properties Click on the L2 Access Interfaces or L3 Access Interfaces tabs on the Services (Edit) form.
If a host is having a problem connecting to the network, one possibility for the problem is dropped packets as a result of anti-spoofing filters on the SAP. The 5620 SAM allows you to view the anti-spoof filters currently in effect on a SAP.
Procedure 3-13 To retrieve MIB information from a GNE using the snmpDump utility Perform this procedure to export all object values from the 5620 SAM-supported SNMP MIBs on a GNE. The exported information may help with troubleshooting the GNE configuration on the device or in the 5620 SAM.
Page 78
MPLS-LDP-STD-MIB.mplsLdpEntityStatsEntry : 3 MPLS-LDP-STD-MIB.mplsLdpPeerEntry : 3 The utility is finished when the command prompt is displayed. To view the utility output, open the file using a MIB browser or a text editor. 3-20 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
4 — Troubleshooting using topology maps Network topology map overview Several network topology maps are available on the 5620 SAM. The maps display network objects. You can open contextual menus and submenus to open forms with additional information. For more information about topology maps, see the 5620 SAM User Guide.
Yellow indicates that the device is being synchronized. Green indicates that the device is SNMP reachable. For a service view, red indicates that the service on the device is down. (1 of 2) Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Page 82
Purple indicates that a physical link is being diagnosed. Red indicates that the link is out of service or failed. (2 of 2) Table lists icon symbols and colors for 5620 SAM alarms. Table 4-5 Map alarm status indicators Map icon Alarm...
Troubleshooting alarms using topology maps Use the following procedures to perform network monitoring and troubleshooting activities using the 5620 SAM maps. Procedure 4-1 To monitor alarm status on maps Use this procedure to view alarm information for network elements on a map.
Page 84
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 4 — Troubleshooting using topology maps Click on the Faults tab. The Faults tab form opens. View alarm status and diagnose the problem, as described in chapter 2. Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 5 — Troubleshooting using the NE resync audit function 5.1 NE resync audit overview 5.2 Workflow for NE resync auditing 5.3 NE resync auditing procedures Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
5 — Troubleshooting using the NE resync audit function NE resync audit overview You can use the NE resync audit function of the 5620 SAM to detect and report differences between the 5620 SAM and the NE configuration. This function is helpful for troubleshooting configuration misalignment.
• NE resync audits only compare parameters that are synchronized with the 5620 SAM database. Parameters that are stored in the NE database only and are not managed by the 5620 SAM are not included in the audit report. •...
Page 88
Click on the Resync button. A dialog box appears. Click on the Yes button and wait for the object to resync with the 5620 SAM. The value of the misaligned parameter changes if the resync operation is successful.
You can use the NE audit manager to view the results of previous NE audits and delete audit results. Choose Administration→NE Maintenance→NE Audit Results from the 5620 SAM main menu. The NE Audit Manager list form opens. Configure the filter criteria, if required, and click on the Search button. A list of NE resync audit results is displayed.
Page 90
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 5 — Troubleshooting using the NE resync audit function Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Page 91
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 Network management troubleshooting 6 — Troubleshooting network management LAN issues 7 — Troubleshooting using 5620 SAM client GUI warning messages 8 — Troubleshooting with Problems Encountered forms 9 — Troubleshooting using the 5620 SAM user activity log...
Page 92
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
6 — Troubleshooting network management LAN issues Troubleshooting network management domain LAN issues Table describes the problems associated with troubleshooting network management domain LAN issues. Table 6-1 5620 SAM Network management domain LAN problems Problem Solution Problem: All network management domain stations experience performance Procedure...
For information about network planning expertise, contact your Alcatel-Lucent technical support representative. See the 5620 SAM Planning Guide for more information about the bandwidth requirements. When you are using in-band management, ensure that the network devices used to transport the management traffic are up.
• routing configuration • If the 5620 SAM deployment includes a firewall, view the firewall log entries in the appropriate file: on a RHEL station—/var/logs/messages • on a Solaris station—/var/adm/messages •...
Log in as the root user on a station in the network management domain. Verify that the required ports are open or protected by a firewall. See the 5620 SAM Planning Guide for a complete list of the ports that the 5620 SAM requires and the purpose of each port.
Page 98
Verify that large packets can travel from the managed devices to the 5620 SAM • by using CLI to ping the IP address of the 5620 SAM server, with a large packet. Ensure that the firewalls between the managed devices and the 5620 SAM •...
When an error condition is encountered that the 5620 SAM client has not anticipated, a Problems Encountered form is displayed. See section for more information.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 7 — Troubleshooting using 5620 SAM client GUI warning messages Figure 7-2 Password missing warning dialog box The warning message indicates the information that is required. In this case, click on the OK button to close the dialog box, and configure the New Authentication Password and Confirm New Auth Password parameters.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 7 — Troubleshooting using 5620 SAM client GUI warning messages Figure 7-4 Committing changes to resources warning dialog box When you click on the Yes button for the last confirmation the changes to the parent or sub-forms are committed.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 7 — Troubleshooting using 5620 SAM client GUI warning messages Duplicate configuration form conflicts There are multiple ways to access a configuration form for the same object. For example, you can view the configuration form for a port by choosing Manage→Equipment, or you can access the port by clicking on the port object in the...
Page 104
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 7 — Troubleshooting using 5620 SAM client GUI warning messages Correct the problem based on the information provided. For the example in Figure 7-8, configure the authentication order parameters.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 8 — Troubleshooting with Problems Encountered forms 8.1 Problems Encountered form overview 8.2 Using Problems Encountered forms Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Figure shows the Problems Encountered form. Figure 8-1 Problems Encountered form Table describes the tasks associated with troubleshooting the 5620 SAM using the Problems Encountered form. Table 8-1 Problems Encountered form tasks Problem Solution...
Try to correct the problem based on the information provided. If you cannot correct the problem, complete the procedure and perform Procedure 8-2. Click on the Close button to close the details form. Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
For example, if you were trying to create a VLL service, record the details about the service that you were trying to create. Record the appropriate problem information, as described in chapter 1. Collect logs for your Alcatel-Lucent support representative, as described in Procedure 10-1. Alcatel-Lucent 5620 Service Aware Manager...
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 9 — Troubleshooting using the 5620 SAM user activity log 9.1 Troubleshooting using the 5620 SAM user activity log overview 9.2 Troubleshooting using the 5620 SAM User Activity procedures...
9 — Troubleshooting using the 5620 SAM user activity log Troubleshooting using the 5620 SAM user activity log overview The 5620 SAM user activity log allows an operator to view information about the actions performed by each 5620 SAM GUI and OSS user. Note —...
Troubleshooting using the 5620 SAM User Activity procedures The following procedures describe how to use the 5620 SAM user activity log functions to determine whether a user action is the cause of a problem. Procedure 9-1 To identify the user activity for a network object Open the User Activity form.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 9 — Troubleshooting using the 5620 SAM user activity log Use the activity information from one or more entries to determine whether a sequence of user actions is the source of the problem.
Page 113
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 9 — Troubleshooting using the 5620 SAM user activity log Specify the filter criteria, if required. Click on the Search button. A list of the actions performed by the user during the session is displayed.
Page 114
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 9 — Troubleshooting using the 5620 SAM user activity log Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Troubleshooting the 5620 SAM platform, database, server, or clients 10 — Troubleshooting the 5620 SAM platform 11 — Troubleshooting with the 5620 SAM LogViewer 12 — Troubleshooting the 5620 SAM database 13 — Troubleshooting 5620 SAM server issues 14 — Troubleshooting 5620 SAM clients...
Page 116
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 10 — Troubleshooting the 5620 SAM platform 10.1 Troubleshooting the 5620 SAM platform 10-2 10.2 Troubleshooting the 5620 SAM platform procedures 10-3 Alcatel-Lucent 5620 Service Aware Manager 10-1 5620 SAM...
Page 118
10.1 Troubleshooting the 5620 SAM platform Table 10-1 describes the problems or tasks associated with troubleshooting 5620 SAM platform issues. Table 10-1 5620 SAM platform problems or tasks Problem Solution To collect the 5620 SAM log files Procedure 10-1 Problem: Poor performance on a RHEL or Solaris station...
The following procedures describe how to troubleshoot the 5620 SAM platform. Procedure 10-1 To collect the 5620 SAM log files Perform this procedure to collect the relevant log files for troubleshooting a 5620 SAM platform, database, server, or client problem.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 10 — Troubleshooting the 5620 SAM platform For a database problem that is not related to installation, perform the following steps: Log on to the station as the Oracle management user.
Page 121
If required, you can run the getDebugFiles.bash utility to collect a comprehensive group of troubleshooting log files for use by Alcatel-Lucent technical support. Log in to the 5620 SAM main or auxiliary server station as the root user. Open a console window.
Page 122
Store the files in a secure location to ensure that the files are not overwritten. For example, if there are two 5620 SAM clients that experience problems, rename the files to identify each 5620 SAM client and to prevent the overwriting of one file with another of the same name.
Page 123
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 10 — Troubleshooting the 5620 SAM platform Review the command output. Code 10-1 is an example of RHEL mpstat output; Table 10-2 describes each output field. Code 10-1: RHEL mpstat output example...
Page 124
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 10 — Troubleshooting the 5620 SAM platform The top 5620 SAM process in the CPU column should be the Java process. However, the Java process should not be consuming too much CPU. Some Oracle processes may also consume CPU, depending on the database load.
Procedure 10-3 Problem: Device discovery fails because of exceeded RHEL ARP cache When a 5620 SAM system on RHEL manages a large number of NEs in a broadcast domain the ARP cache may fill and prevent the discovery of additional devices. When this...
Page 126
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 10 — Troubleshooting the 5620 SAM platform Perform this procedure when one of the following occurs: The /var/log/messages file contains more than 1024 entries. • You need to increase the ARP cache size to accommodate the network.
11.1 5620 SAM LogViewer overview The 5620 SAM LogViewer is a system monitoring and troubleshooting utility that parses, formats, and displays the contents of 5620 SAM log files. You can use LogViewer to perform the following: • View and filter real-time log updates.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer You can customize LogViewer by creating and saving log filters and log profiles that are available to all GUI and CLI users, and can save the GUI application configuration, or workspace, to have LogViewer display the currently open logs the next time it starts.
11.3 LogViewer CLI The CLI-based 5620 SAM LogViewer works like the UNIX tail command when in display mode. The command mode has a multiple-level menu that you can display at any time. You can specify a command or log file using the minimum number of unique characters in the name, and can quickly toggle between the command and display modes.
Page 133
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer To open a log file, perform one of the following: If the Quick Links panel is displayed, click on a link to view the associated log file.
Page 134
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer A log tab opens to display the most recent entries in a log. If the log is active and the Auto-Tail parameter is enabled, the list scrolls upward to display new log entries as they are generated.
Page 135
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer To remove one or more log entries from the current view, perform one of the following. To clear all listed log entries, choose Log→Clear All Events from the LogViewer main menu, or click on the Clear all button in the main tool bar.
Page 136
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer To pause the display of log-file updates, choose Log→Pause from the LogViewer main menu, or click on the Pause log updates button in the log tab tool bar.
Page 137
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer A new tab opens to display the captured log entries in a static view. Static view operations To sort a list of log entries in a static view, right-click on a column header and choose Sort Ascending, Sort Descending, or No Sort from the contextual menu.
Page 138
Base File Messages Directory—Click in the parameter field and use the browser • form that opens to specify the base 5620 SAM log directory. Default Character Set—Edit this parameter to specify the character set that •...
Page 139
• Filter Display—for filtered log entries • Configure the Always Use ANSI Display parameter, as required. Click on the 5620 SAM tab to configure the parameters that are specific to the 5620 SAM. Alcatel-Lucent 5620 Service Aware Manager 11-13 5620 SAM...
• directory Oracle Location—specifies the base 5620 SAM Oracle installation directory • NMS Root—specifies the nms directory under the base 5620 SAM server • installation directory Note — Your configuration of LogViewer is stored in the user directory. Any filters, fonts, colors, or other preferences you have set are preserved when you install a newer version of the application.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer To restrict the search to logs with certain filenames, enter a regular expression in the File Mask parameter. To search all logs in the specified path, leave this parameter blank.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer For either foreground or background items, set the color as required. You can choose a color from the samples shown on the Swatch tab, or you can specify a color by entering its red, green, and blue values in the RGB tab.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer Procedure 11-7 To manage filters using the GUI Filter Manager Perform this procedure to create, modify, assign or delete a LogViewer filter.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer To create a saved filter based on the current quick filter, perform the following steps. Choose Log→Create from Quick Filter from the LogViewer main menu, or click on the Create from quick button in the main tool bar.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer Select a log file and click on the Add object... icon button between the form panels. The log is listed in the panel on the right.
Procedure 11-9 To display logs using the LogViewer CLI Perform this procedure to start the LogViewer CLI application and view one or more logs. Log in to the 5620 SAM server as the samadmin user. Open a console window. Enter the following at the prompt: bash$ path/nms/bin/logviewer.bash argument options parameter ↵...
Page 147
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer The LogViewer CLI opens in display mode. If a log file is specified as a startup parameter, the most recent entries in the log file are displayed as they are written to the log file.
Page 148
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer File Name (full path)? Enter the absolute or relative path of the log file that you want to open and press ↵.
Page 149
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer At each prompt, enter a regular expression to use as a match criterion, if required, and press ↵. The following prompt is displayed: Display Filter? (Y/N): Enter y ↵...
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer LogViewer resets the counters. Enter the following at the prompt to return to display mode: return ↵ Enter the following at the prompt to close LogViewer: quit ↵...
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer Enter the absolute or relative path of the log file that you want to add and press ↵. LogViewer adds the file to the list in the open menu.
Page 152
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 11 — Troubleshooting with the 5620 SAM LogViewer The following prompt is displayed: log-plugin> Enter the following at the prompt: add ↵ LogViewer displays a list of the available plug-ins and the following prompt: Which plugin would you like to specify? (name) Enter the name of a plug-in from the list and press ↵.
5620 SAM system. Contact Alcatel-Lucent technical support before you attempt to perform a database restore. In a redundant 5620 SAM system, you must perform one or both of the following to regain database function and redundancy: Restore the primary database.
Both operations are required after a primary database failure. After a standby database failure, no restore operation is required, but you must reinstantiate the standby database to restore redundancy. You can use the 5620 SAM client GUI or a CLI script to reinstantiate a database.
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 12 — Troubleshooting the 5620 SAM database Perform a database backup using the 5620 SAM client GUI or a C LI, as described in the 5620 SAM User Guide.
See chapter for more information. Procedure 12-6 Problem: Primary or standby database is down The status bar of the 5620 SAM client GUI indicates that the primary or standby database is down. Warning — Performing database modifications using Oracle database...
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 12 — Troubleshooting the 5620 SAM database Procedure 12-7 Problem: Need to verify that Oracle database and listener services are started Perform the following procedure to determine the status of the Oracle database and listener services, each of which starts automatically during system initialization.
Page 159
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 12 — Troubleshooting the 5620 SAM database 5620 SAM Version 11.0 R1 - Built on Wed Mar 27 03:14:15 EST 2013 Close the console window. Alcatel-Lucent 5620 Service Aware Manager...
Page 160
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 12 — Troubleshooting the 5620 SAM database 12-8 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
5620 SAM server issues. Table 13-1 5620 SAM server problems Problem Solution Problem: Cannot start a 5620 SAM server, or unsure of 5620 SAM server status Procedure 13-1 Problem: 5620 SAM server and database not communicating Procedure...
• Log in to the 5620 SAM server as the samadmin user. Open a console window. To check the status of a 5620 SAM main server, perform the following steps. Enter the following at the CLI prompt: path/nms/bin/nmsserver.bash appserver_status ↵...
Page 164
Stops the 5620 SAM main server debug Starts the 5620 SAM server in an interactive mode appserver_status Returns information about the status of the 5620 SAM main server (both active and standby servers when the 5620 SAM is configured for redundancy) appserver_version...
Page 165
(2 of 2) Note The server shuts down if the console is closed or if CTRL-C is pressed. To check the status of a 5620 SAM auxiliary server, perform the following steps. Enter the following at the CLI prompt: path/nms/bin/auxnmsserver.bash aux_status ↵...
Page 166
Enter the following at the CLI prompt: path/nms/bin/auxnmsserver.bash auxappserver_status ↵ where path is the 5620 SAM server installation location, typically opt/5620sam/auxserver Detailed 5620 SAM server information is displayed. To obtain more specific server status information, run the nmsserver script using the appropriate option from Table...
Verify that the database is running in the correct mode. • See the 5620 SAM Planning Guide for more information about the ports that must be available for the 5620 SAM to function. If the problem persists, collect the logs...
Perform a ping test to measure reachability, as described in Procedure 6-2. On a RHEL or Solaris station, if you cannot ping the 5620 SAM server, make sure that the hostname of the server is in the /etc/hosts file. Perform the following steps.
Page 169
Open a 5620 SAM client GUI. Choose Administration→System Information. The System Information form opens. Click on the Faults tab to view auxiliary server and general 5620 SAM system alarm information, if required. If your 5620 SAM deployment includes one or more auxiliary servers, perform the following steps to check the status of each auxiliary server.
Caution — Exceeding the alarm limit configured in the nms-server.xml file may cause system performance problems. Check the status bar of the 5620 SAM client GUI status bar for indications that the maximum number of alarms for the system is reached.
5620 SAM server, or no SNMP traps are arriving When you install the 5620 SAM server, you specify the port on which SNMP traps arrive. In addition, two sets of configurations must be completed for SNMP trap notifications to work: Enable key SNMP parameters on the devices before managing them.
• polling fails completely • When 5620 SAM resynchronizes some functions on an NE, for example, BGP • configurations for the 7750 SR, the SNMP packets may be broken into two or more smaller packets, when the maximum PDU size of 9216 bytes is exceeded.
When 5620 SAM performance is slow, especially when performing network device resynchronizations, SNMP and IP performance along the in-band or out-of-band interfaces between the network device and the 5620 SAM server may be the problem. Check the following: configuration of the LAN switch port and the 5620 SAM station port match •...
• For collision errors, consider increasing the SNMP retry value • Check the mediation policy for the device using the 5620 SAM client GUI. Check the SNMP timeout and retry value for the mediation policy. If the output of step...
OSS application requests data from the statistics tables less frequently • than the configured roll over interval FTP must be enabled on the managed device in order for the 5620 SAM to • retrieve statistics. Alcatel-Lucent recommends that statistics collection planning includes the following considerations to prevent the loss of statistics data.
Page 176
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 13 — Troubleshooting 5620 SAM server issues 13-16 Alcatel-Lucent 5620 Service Aware Manager 5620 SAM Troubleshooting Guide...
Procedure 14-12 Problem: Cannot clear alarms using 5620 SAM client GUI Procedure 14-13 Problem: The 5620 SAM client GUI does not display NE user accounts created, Procedure 14-14 modified, or deleted using the CLI 14.2 Troubleshooting common client application problem...
Page 179
Perform the following steps to correct this. Log in as root, or as the user that installed the 5620 SAM client, on the client station. Open a console window.
Firewalls between the 5620 SAM clients and the server are correctly configured • To check that the 5620 SAM client points to the correct IP address and port of the server, open the nms-client.xml file using a text editor. The default file location is installation_directory/nms/config.
# netstat -i s ↵ where s is the time, in seconds, over which you want to collect data. Alcatel-Lucent recommends that you start with 50 s Review the output. The following is sample netstat output:...
To stop the command, press CTRL-C. Check that the client platform is appropriately sized. See the 5620 SAM Planning Guide for more information. Procedure 14-4 Problem: Cannot view 5620 SAM alarms using 5620 NM client Possible causes include incorrectly configured param.cfg parameters on the 5620 NM to...
Release 12.0 R4 | August 2014 | 3HE 08864 AAAD TQZZA Edition 01 14 — Troubleshooting 5620 SAM clients Procedure 14-5 Problem: Unable to print from RHEL or Solaris client Printers are connected to clients to provide a printed record of alarms, the GUI, or text files.
↵ Procedure 14-6 Problem: Cannot place newly discovered device in managed state Possible causes are: the number of managed cards (MDAs) exceeds the 5620 SAM server license • another application using a port required by the 5620 SAM server •...
Suspend Retries or Resume Retries button. You can clear a deployment by clicking on the Clear button. When you clear a deployer, no further attempt is made to reconcile the network device status with the 5620 SAM database. Affected objects should be resynchronized.
Page 186
Attribute: alarmClassTag Value: generic.DeploymentFailure The alarm also contains additional information, including the object affected by the alarm and the severity of the alarm. See the 5620 SAM XML OSS Interface Developer Guide for more information. Find the following text in the alarm: Attribute: requestID=requestID The parameter specifies the request id sent with the original request.
14 — Troubleshooting 5620 SAM clients Procedure 14-8 Problem: Device configuration backup not occurring Use the 5620 SAM client to check the device database backup settings. Choose Administration→NE Maintenance→Backup/Restore from the 5620 SAM main menu. The Backup/Restore form opens with the Backup/Restore Policy tab displayed.
The following procedures describe how to troubleshoot client GUI-specific issues. Procedure 14-9 Problem: 5620 SAM client GUI shuts down regularly The 5620 SAM client GUI automatically shuts down under the following conditions: no activity on the GUI for a specified amount of time •...
Alcatel-Lucent recommends that you use filters to restrict the number of items in a list or search operation to 10 000 or fewer. See the 5620 SAM User Guide for information about the 5620 SAM client GUI list and search functionality. See the 5620 SAM Planning Guide for information about 5620 SAM scalability and system capacity guidelines.
Page 190
NE user accounts created, modified, or deleted using the CLI When an NE user account is created, modified, or deleted using the CLI, the 5620 SAM client GUI does not update the user list in the NE User Profiles form. For increased security, the NE does not send a trap for changes made to NE user accounts.
Customer documentation and product support Customer documentation http://www.alcatel-lucent.com/myaccess Product manuals and documentation updates are available at alcatel-lucent.com. If you are a new user and require access to this service, please contact your Alcatel-Lucent sales representative. Technical Support http://support.alcatel-lucent.com Documentation feedback...