Slowness in Java Application due to increased FullGC events — G1GC

In this Blog, we will see one of the issues and solutions which I found when one one of our production servers , our java application is becoming slow due to more gc pauses .

I will explain this particular approach which can be one of the reasons for initiating more gc pauses .

To understand this article , one must have basic knowledge of G1GC algorithm of java for garbage collection .

Don’t worry if you don’t have knowledge of G1GC , I will make articles on basics of G1GC later and then you can read this article again .

So, Let’s start from what issue we are facing

Issue : applications become unresponsive in between very frequently .

Analysis :

• after debugging from jvm level stats from JMX bean dumping it was clear that GC collection time was increased so much in between
• Heap Also increasing

After that we enabled gc log by using-Xlog:gc=debug:file=/tmp/gc.log in jvm arguments while starting application .

Analyzing gc.log , we found Full GC is triggering many times and whenever FullGC triggers , it generally stop the application for sometime , in java language we call it STW (Stop the World) .

Generally there are following type of events in G1GC :

• Minor: Eden + Survivor From -> Survivor To
• Mixed: Minor + (# reclaimable Tenured regions / -XX:G1MixedGCCountTarget) regions of Tenured
• Full GC: All regions evacuated
• Minor/Mixed + To-space exhaustion: Minor/Mixed + rollback + Full GC

In a smoothly running application, batches of Minor events alternate with batches of Mixed events should be there only . Full GC events and To-space exhaustion are things you absolutely don’t want to see when running G1GC, they need to be detected and eliminated and if they are running they should be run by some external events like (jstack,jmap etc …) .

For in depth details of these events , as already stated I will make a blog series on explaining G1GC concepts , for now you can search on the net .

Now, coming back to our debugging ,

We checked that no external command for taking thread dump or heap dump or histogram was made that can possibly initiate Full GC event .

So , the question now was why this full GC is Triggering .

On Further Researching we found that Humongous objects can be one of the reasons for triggering the Full GC event .

Now what is Humongous objects ? ? ?

A Brief Definition is : Any single data allocation ≥ G1HeapRegionSize/2 is considered a Humongous object, which is allocated out of contiguous regions of Free space, which are then added to Tenured. As Humongous objects are allocated out of Free space. Allocation failures trigger GC events. If an allocation failure from Free space triggers GC, the GC event will be a Full GC, which is very undesirable in most circumstances. To avoid Full GC events in an application with lots of Humongous objects, one must ensure the Free space pool is large enough as compared to Eden that Eden will always fill up first .

So , We started checking if our application is generating Humongous objects .

And from gc.log we found that lots of Humongous objects are created which were the reasons for triggering Full GC events .

I made following commands to check the Humongous objects specially in linux :

Step 1. : run following command on your gc.log

Command 1 :

grep "source: concurrent humongous allocation" /tmp/gc.log | sed 's/.*allocation request: \([0-9]*\) bytes.*/\1/' > humoungous_humongoud_size.txt

Command 2 :

awk -F',' '{sum+=$1} END{print sum;}' humoungous_humongoud_size.txt

It will give you the size of humongous objects generated in my application.

We have java less than Oracle JDK 8u45 version and for java greater than this , it is written in release notes that these Humongous objects also get collected in Minor events also .

Search for “ G1 now collects unreachable Humongous objects during young collections” in the Release Notes JDK

So then we upgraded our jdk and issue frequency was minimized too much as these objects are now not triggering any major event like FullGC .

But one should also care about generating these large object

So , we also checked and analyzed one of the heaps and corrected the code not to generate these big objects if not needed .

I hope this blog will be helpful . Please comment , share and subscribe .

Originally published at http://hello-worlds.in on April 4, 2022.