分析WebSphere产生的javacore文件判断问题

Problem determination for javacore files from WebSphere Application Server

Technote (FAQ)
  
Problem
Some error conditions in an IBM® Java™ virtual machine (JVM™) running under WebSphere® Application Server result in a crash. The output of a crash is a javacore file. This document describes how to read and interpret the data in a javacore file.

Performing problem determination by analyzing the javacore file is an effective means of determining root cause; taking preventative action helps to avoid future crashes of the JVM.

These problem determination techniques are applicable to the Java Software Developer Kit (Java SDK) delivered by IBM® on Microsoft® Windows®, Linux® and AIX®.
  
Cause
A javacore file is a snapshot of a running Java process. The IBM Java SDK produces a javacore in response to specific operating system signals. Javacore files show the state of every thread in the Java process, as well as the monitor information identifying Java synchronization locks.

You can use Javacore files to resolve the problems listed below. For more information on capturing data for problem determination, see the MustGather documents for these problems:

100% CPU Usage Crash Hang/Performance Degradation

Notes:

Application Server restarts often indicate a JVM crash. Sometimes a JVM crash does not produce a javacore, in which case it might be necessary to disable the javacore processing to allow a core file to be created on UNIX® or user.dmp file to be created on Microsoft Windows.
Solution
Table of contents for problem determination:  

1. Locating and identifying javacore files
1.1 Locating where a javacore file is written is determined by the following
1.2 Javacore naming
2. How javacore files (thread dumps) are created
2.1 Operating system signals
2.1.1 UNIX signals
2.1.2 Windows signals
2.2 Thread dump via an explicit action by the customer
3. Reading javacore files
3.1 Signal information
3.1.1 Time and date of the javacore
3.1.2 Type of signal and location of the JVM process at the time of the javacore
3.1.3 Java fullversion
3.1.4 Current thread details
3.2 Operating environment
3.2.1 Signal handlers
3.2.2 Loaded libraries
3.2.3 Java command line
3.2.3.1 Determining which WebSphere Application Server process a javacore is for

3.3 JVM monitor information
3.4 The JVM thread dump
4. How to proceed if the library that caused the crash is not identified
5. What can prevent a javacore file from being produced?

1. Locating and identifying javacore files


1.1. Locating where a javacore file is written is determined by the following
The setting of the IBM_JAVACOREDIR environment variable.If this variable is not set, the javacore is written to the working directory of the JVM; use the -DWORKING_DIR parameter on the JVM command line to set this variable.Otherwise, the javacore is written to the directory from which the Java process was started.V4.0 release Java processes
For WebSphere Application Server Java processes, this is usually the install_root/bin directory.V5.0 release Base Edition Java processes
For WebSphere Application Server Java processes, this is usually the install_root/bin directory.
1.2 Javacore naming
The name of the javacore file (for example, javacore24802.1026159146.txt) is derived from a couple of system values. Each javacore has a unique name. The following table indicates the name format for each operating system.

Operating System	Javacore file name format	Meaning
Windows and Linux	javacore.YYYYMMDD.HHMMSS.PID.txt	YYYY=year, MM=month, DD=day, SS=second, PID=processID
AIX	javacorePID.TIME.txt	PID=processID, TIME=time since 1/1/1970
z/OS	Javadump.YYYYMMDD.HHMMSS.PID.txt	YYYY=year, MM=month, DD=day, SS=second, PID=processID

The PID is the process ID of the JVM that dumped the javacore. The PID can be used in conjunction with the WebSphere Application Server logs to determine which WebSphere Application Server process the javacore is for. For Linux, each thread is its own process, so any JVM appears to have several running processes on Linux. The PID is the one with which the JVM started.

2. How javacore files (thread dumps) are created
The IBM Java SDKs produce a javacore file in response to an operating system signal. The JVM has signal handlers that handle operating system signals. An operating system signal can be raised because of a run-time exception, or by an explicit action by the user.

2.1 Operating system signals
Operating system signals are raised in native code. Java itself will not issue a signal to end the JVM process. An operating system signal that causes the JVM process to end is caused by native code that the JVM calls. Native code libraries are used for a variety of different functions; for example, database calls.

2.1.1 UNIX Signals
The javacore displays the operating system signal that caused the javacore to be written. Some signals also produce a core file in UNIX operating systems. The JVM signal handler library is libhpi.a.

Following are some of the most commonly seen signals in javacore files:

SIGQUIT
This signal is sent when a kill -3 command is issued against the JVM process. This signal typically does not end the JVM process. It generates a thread dump (javacore) for diagnosing a potentially hung JVM process.SIGILL
This is the equivalent of a kill -4 command. This means an illegal instruction was executed. This can mean a corruption of the code segment or an branch that is not valid within the native code. This signal often indicates a problem caused by JIT-compiled code.SIGSEGV
This signal is equivalent to a kill -11 command. It indicates an operation that is not valid in a program, such as accessing an illegal memory address. This is typically indicative of a programming problem in one of the native libraries.
For more information on signals in WebSphere Application Server, refer to the technote What is a signal and why does this matter for WebSphere Application Server.

2.1.2 Windows Signals
The signals that Windows uses are entirely different than those used by UNIX. Here is a list of some of the operating system signals produced on Windows. Some signals also produce a user.dmp file. The JVM signal handler library is hpi.dll.

Signal Name	Description	JVM action
Memory access error	Invalid memory address	javacore and abort
Illegal access error		javacore and abort

2.2 Thread Dump via an Explicit Action by the Customer
A customer can explicitly cause a thread dump of the JVM process through an operating system command or the Task Manager in Windows.

Triggering a javacore using an operating system commandIssuing a kill statement against the JVM process on a Unix system typically causes a thread dump; however, that depends on the signal that is specified on the command, and if there is a signal handler that intercepts that signal before the JVM gets it. The JVM process is not always ended by an operating system signal; that depends on the signal that is sent to the JVM process. A kill -3 (SIGQUIT) is not a terminating signal. A kill -11 (SIGSEGV) is a terminating signal. The kill command on AIX, by default, sends a SIGTERM signal.Triggering a javacore using a WebSphere Application Server script or bat fileFor V3.5 and V4.0 releases on Windows, the DrAdmin.bat utility also causes a thread dump. See How to use DrAdmin with WebSphere Application Server 4.0 for more details.

For V5.0 releases, wsadmin can be used to create a Java thread dump. For additional information, see technote MustGather: No response (hang) or performance degradation on Windows for v5.0 releases.

Note: Using these utilities to dump threads does not end the JVM process.
3. Reading javacore files
Javacore files are text files, so you can read them with a text editor. There are two different formats of javacore files. The older format is simpler. This document provides examples and discusses both formats. The examples refer to javacores on AIX systems; however, the principles are applicable to Linux and Windows IBM Java SDKs.

3.1 Signal information
This section tells you details about the operating system signal that produced the javacore. It identifies when the signal was issued and where the signal was issued.


3.1.1 Time and Date of the javacore
Every javacore file has a line that indicates the time and date of when the javacore file was produced. In the older javacore format, the first line of a javacore file identifies when the javacore was taken. In the newer javacore format, there is a line that has the tag, 1TIDATETIME and the tag Date: that indicates the date and time when the javacore was written.

Diagnostic Step
Verify when the javacore file was written to determine if it is relevant to problem being analyzed.

3.1.2 Type of Signal and Location of the JVM process at the time of the javacore
Every javacore has a line that indicates the signal that caused the javacore to be produced. In the older javacore format, this is the next line after the date in the javacore file. In the older javacore file format, the line that identifies the signal often identifies the library in which the signal was generated (if it is possible for the javacore processing to identify it). In the sample javacore output shown, the signal that generated the thread dump is a SIGSEGV, which means that the JVM process ended abnormally. The library in which the crash occurred is libdb2jdbc.so.