关于Dark matter问题

Cause

The allocation and creation of an object on the Java heap will often leave space between objects that is less than 512 bytes. This small portion of the heap is removed from the free space list, and to the garbage collector, appears like a small object. This dark matter is not collected during the normal mark/sweep phase of the IBM Java SDK garbage collection (GC) cycle. Dark matter will remain in the heap until a compaction phase occurs or the adjacent objects are collected. Compaction cycles can be lengthy and can consume a large amount of the system resources, so they are not initiated by the JVM unless it is necessary. A compaction cycle will occur if any of the following criteria are met:

1.        -Xcompactgc has been specified.

2.        Following the mark/sweep phase, not enough contiguous free space is available to satisfy the allocation request.

3.        A System.gc() has been requested and the last allocation failure garbage collection did not perform a compaction.

4.        At least half the previously available memory has been consumed by thread local heap (TLH), cache, allocations (ensuring an accurate sample) and the average TLH size falls below 1000 bytes.

5.        Less than 5% of the active heap is free.

6.        Less than 128 KB of the active heap is free.

If the heap does not meet the above criteria and the amount of Dark Matter continues to increase, performance issues associated with excessive or long running GC cycles can be experienced. The JVM may begin to experience frequent GC cycles which consume a majority of the processing cycles resulting in severe performance degradation on the application server. It may also result in a very long running GC cycle where a compaction is performed causing intermittent pauses in application responsiveness.

To determine if dark matter is the source of a GC issue, enable verbose GC and examine the GC patterns during a performance issue. If dark matter is involved, you will should see a series of GC cycles where no compaction takes place and a large amount of heap space is not freed. This is followed by a GC cycle where a compaction takes place and a large amount of heap space is reclaimed.

For example:
<AF[92]: Allocation Failure. need 300016 bytes, 3907 ms since last AF>  
<AF[92]: managing allocation failure, action=2 (20023632/1073740288)>  
<GC(110): GC cycle started Thu Apr 07 08:17:20 2005                    
<GC(110): freed 24254168 bytes, 4% free (44277800/1073740288), in 4173 ms>                                                                    
  <GC(110): mark: 4055 ms, sweep: 118 ms, compact: 0 ms>                
  <GC(110): refs: soft 0 (age >= 32), weak 5769, final 293, phantom 0>  
<AF[92]: managing allocation failure, action=3 (44277800/1073740288)>  
<AF[92]: managing allocation failure, action=4 (44277800/1073740288)>  
<AF[92]: clearing all remaining soft refs>                              
<GC(111): GC cycle started Thu Apr 07 08:17:24 2005                    
<GC(111): freed 196248 bytes, 4% free (44474048/1073740288), in 4106 ms>
  <GC(111): mark: 3987 ms, sweep: 119 ms, compact: 0 ms>                
  <GC(111): refs: soft 13 (age >= 32), weak 1, final 0, phantom 0>      
<GC(112): GC cycle started Thu Apr 07 08:17:29 2005                    
<GC(112): freed 624 bytes, 4% free (44474672/1073740288), in 4851 ms>  
  <GC(112): mark: 4733 ms, sweep: 118 ms, compact: 0 ms>                
  <GC(112): refs: soft 0 (age >= 32), weak 0, final 0, phantom 0>      
<AF[92]: completed in 13153 ms>                                        
                                                                       
<AF[93]: Allocation Failure. need 300016 bytes, 13911 ms since last AF>
<AF[93]: managing allocation failure, action=2 (6488344/1073740288)>    
<GC(113): GC cycle started Thu Apr 07 08:19:47 2005                    
<GC(113): freed 644892792 bytes, 60% free (651381136/1073740288), in 20605 ms>                                                              
<GC(113): mark: 4735 ms, sweep: 140 ms, compact: 15730 ms>            
<GC(113): refs: soft 0 (age >= 32), weak 28771, final 864, phantom 0>
<GC(113): moved 7268044 objects, 270012456 bytes, reason=5, used 5272 more bytes>                                                            
<AF[93]: completed in 20619 ms>

From this data, we see that Allocation Failure (AF) 92 is able to free around 25M of heap with no compaction attempted. Fourteen seconds later, AF 93 performs a compaction and frees 644 MB of memory taking 20.6 seconds to complete. The collecting of a relatively large amount of space during a GC cycle that invokes a compaction is a clear symptom that dark matter is involved.

Resolving the problem

Beginning with the 1.4.0 release of the IBM Java SDK, a new feature was added to deal with the problem of long pauses caused by the compaction phase. The 1.4 Java SDK will perform incremental compactions by default, spreading the compaction process across the GC cycles. The -Xpartialcompactgc parameter, which can be added as a Generic JVM Argument for the 1.4 Java SDK, causes an incremental compaction to occur on every GC cycle. Incremental compaction works by dividing the heap into sections and running the compaction on one given section at a time. The 1.4 release of the Java SDK is used beginning with the WebSphere® Application Server V5.1 release.