Print Page
ITIL: Service Support
Incident Management
An incident is any event that is not part of normal operation of a service and that causes an interruption or a reduction in the quality of the service. Incidents can be reported by users or can be discovered by a detection system. The goal of Incident Management is to restore IT service operations to their normal state as quickly as possible in order to minimize the adverse effect on business operations.
Examples of incidents users might report include poor performance of an important business application, a down link, or an application that is unavailable.
In the Incident Management discipline, PacketShaper can help detect, investigate, diagnose, and resolve incidents.
Detect and Record Incidents
Use Adaptive Response for incident detection and notification
PacketShaper's adaptive response feature can monitor various aspects of your network and then respond by notifying support staff if an incident occurs. For example, PacketShaper can monitor bandwidth usage of hosts on the network and if a single host's traffic exceeds a certain percentage of the link size, the adaptive response feature will send an alert that a threshold has been exceeded. Or, you can have PacketShaper monitor retransmissions on a link and send an alert if the network efficiency drops below a certain level. PacketShaper includes a variety of agents you can customize for your network's needs. You can choose the application you want to monitor as well as the metric (application availability, average round trip time, network delay, packet exchange time, to name just a few). For examples of real-world ways you can use the agents, see Adaptive Response Agent Examples and Detect Subscriber Access Failures. See also Monitor a Condition of Interest.
The alert can be in any of the following forms:
SNMP Traps — PacketWise can be configured to work with any standard SNMP program, such as HP OpenView or Packeteer ReportCenter. To configure PacketWise for SNMP support, you need to specify the trap destination (the IP address of an SNMP program). Incidents will then be recorded in the SNMP program — the trap listener. See SNMP Overview for more information.
Syslog Messages — Syslog gives administrators a way to centrally log and analyze incidents. See Setup Syslog for details on defining syslog servers and enabling the logging feature.
Email Messages — An email message sent to a cell phone or pager can provide instant notification that an incident has occurred. The sooner IT knows about an incident, the sooner the problem can be solved.
Review PacketShaper reports to uncover performance problems
PacketShaper offers extensive reporting capabilities. By regularly reviewing a carefully selected group of reports, you may be able to uncover performance problems before an incident is even reported.
Define and review reports (using the PacketShaper's individual reports). This is not applicable to the PacketShaper 1400 Lite since this model has limited reporting capability.
Define and review reports (using ReportCenter's reports) that incorporate multiple PacketShapers. ReportCenter's reports are especially helpful for PacketShaper 1400 Lites because of their limited number of built-in reports.
Detect new applications that impact performance
New types of traffic or new applications on your network can impact performance of other applications, especially if they involve great volumes of traffic. PacketShaper's superior classification capabilities can automatically detect and classify hundreds of business and recreational applications. If the Monitor screen or the Top Ten window shows that there is a new aggressive application, you can act to identify it and contain its impact. Such an application will appear in the Top Ten pie chart when it has not done so previously, or will move up into occupying one of the largest slices of pie.
PacketShaper 1400 Lite Users: Use ReportCenter to analyze traffic and generate reports for your PacketShaper 1400 Lite. See Using Packeteer ReportCenter Reports.
If your newly aggressive traffic lands in a Default traffic class (when it's not classified), you can use the PacketShaper to identify the mystery traffic and create a class. Sometimes, you can get more granular information about the components of a single traffic class if you enable traffic discovery for just that one class.
Investigate and Diagnose Incidents
When incidents occur, you can use PacketShaper's forensic tools to investigate and diagnose the incidents.
Graphing Tools
Packeteer measures hundreds of characteristics about traffic as it passes, creating an extensive collection of measurement data. Packeteer stores measurement data on appliances for up to two months and for months or years on a ReportCenter server. Packeteer’s metrics can also be incorporated into SNMP management platforms, NetFlow v5 collectors, and third-party reporting tools, such as Concorde, InfoVista , and Microsoft Excel.
Packeteer uses this measurement data to generate a variety of graphs, several of which can assist in the diagnostic process. The TCP Health graph shown above gives you a comprehensive picture of TCP connections for a link, partition, or traffic class. It compares the number of TCP connections that were started, aborted, ignored, or refused by the server. This can help you identify where problems occur. For example, if you see a large number of Server Ignores, you know that you have an overloaded or malfunctioning server that is ignoring new connection requests.
PacketShaper's graphing tools are also useful for investigating an incident related to an application's behavior. You can:
- analyze how bandwidth is used by an application
- evaluate how much of an application's bandwidth is due to retransmissions, and
- assess network, server, and overall response times for an application.
Flow Detail Records
Packeteer can provide drill-down metrics on a per-flow basis that include items such as flow origin and destination, flow size (in packets and bytes), when the flow was sent, the flow’s application or service, the flow’s Layer-4 protocol and IP ToS/Diffserv bits, the type of controls that were applied to the flow, response times, and more.
This granular level of detail opens up a wealth of opportunity for enhanced troubleshooting and forensic help. For example, you can:
- Examine the “chattiest” host IP pairs for traffic from a specific application, location, or combination of the two.
- Split the traffic from one branch office into its different application, service, or DSCP types, even if you didn’t sub-classify traffic into its services as it passed.
- List traffic’s busiest ports; list the ports a specific application or host used; list which applications used a specific port; spot potential portscans.
- Expose the top current or historical traffic contributors or recipients for a location or application
Packeteer puts these more detailed, per-flow metrics into flow detail records (FDRs). If you specify Packeteer ReportCenter as the collector, you can use a variety of reports to aid in troubleshooting network problems. For example, ReportCenter’s flow detail reports let you see the busiest hosts on your network and drill down to see the applications used and destination addresses contacted.
Host Analysis
Packeteer maintains a database of all hosts that have active connections through the PacketShaper. Once a host closes its connection, the host will be purged from the database. In addition, the PacketShaper will clear host entries if they aren’t active for approximately ten minutes. Thus, the host database is a real-time list of hosts.
Host Analysis is a reporting tool that allows you to view, sort, and drill-down on hosts in the host database. If you spot a suspicious host (for instance, one that is using excessive bandwidth or creating an inordinate number of connections or failed connections), you can drill down to find out more detailed information. What other hosts is the suspicious host communicating with? What are the host’s current and recent traffic flows? What type of traffic is it (HTTP, SSL, etc.)? The Host Analysis tool can provide the answers.
Packet Capture
For a more in-depth analysis of your flows, Packeteer offers the packet capture feature. This feature captures packets for future analysis, allowing you to analyze detailed information about the packets in a traffic class, such as the source and destination IP addresses and protocols used. You can even examine the packet down to the byte level, if you are so inclined. With the packet capture tool, you can capture a small sample of traffic from an unknown or troublesome flow so that you can investigate it in detail: view packet headers, real-time display of top users, or content at specific offsets into packets.
First, you need to decide which packets you would like to collect. A major advantage of using the PacketShaper as a collector is that you define precisely which traffic to capture. You don’t have to collect huge log files with mostly irrelevant traffic. Any traffic you can identify with matching rules in a traffic class can be captured independently. For example, if you want to capture all Telnet packets to a certain IP address — you can. Or if you want to capture only Oracle traffic for one particular database — you can.
The basic process is to specify which traffic class(es) from which to capture packets, enable packet logging for a period of time, and then pass the resulting log file to a third-party analysis tool, such as EtherPeek, Ethereal, or a Sniffer. See Sniff without a Sniffer for details.
Resolve Incidents
The resolution of incidents will frequently involve the use of the Shaping Module. Through the judicious use of policies and partitions, this module can solve a variety of network and application problems. If an important application, such as SAP or Oracle, runs too slowly, you can put controls in place to protect its performance. If unsanctioned recreational traffic, such as KaZaa and Audiogalaxy, is monopolizing your link, you can block or limit the amount of this traffic that can be used on your network. If users complain about jittery voice or video traffic, you can provision steady streams for this traffic to ensure smooth performance. If a single user is consuming a high proportion of bandwidth, you can apply policies to restrict this user's usage.
For procedures on implementing appropriate policies and partitions to resolve incidents, see the following recommendations:
- Protect Critical Application Performance
- Block Unwanted Traffic
- Contain a Greedy Business Application
- Insulate Users of the Same Application
- Quarantine Bandwidth Abusers
To resolve incidents related to specific applications, see Per Application Strategies. Here you will find strategies for managing the performance of Citrix, ERP, streaming media, and so forth.
View the other disciplines in the ITIL Service Support area:
Please see bluecoat.com/support/packeteer
for more detailed information.
Resource Library