Call the police one day : A service deployed by a machine suddenly cannot be accessed . Remember the first reaction, log in to the machine and view the log , Because the service hung up , Probably because OOM. At this time, the following information is found in the machine log :
nio handle failed java.lang.OutOfMemoryError: Direct buffer memory at at at
Indicates that it is indeed OOM, Which district caused the problem ? It can be seen that :Direct buffer memory
, And see a lot jetty Related method call stack , Just these logs , Can analyze OOM reason .

<>Direct buffer memory

Out of heap memory ,JVM A piece of memory outside the heap , Not by JVM Administration , but Java Code can be in JVM Some memory space is used outside the heap . These spaces are Direct buffer
memory, Direct memory , This memory consists of os Direct management . But it's strange to call it direct memory , I don't like to call it “ Out of heap memory ”.

Jetty As JVM The process runs the process of the system we have written :

this time OOM yes Jetty Caused when using off heap memory . Can be calculated ,Jetty Off heap memory may be in constant use , Then out of heap memory space is insufficient , Unable to use more off heap memory , Just OOM Yes .

Jetty Non stop use of off heap memory :

<> solve OOM Underlying technology of

Jetty Since it is used Java Written , How did he pass Java Code to request out of heap memory ? Then, how can the out of heap memory space be released ? This involves Java of NIO bottom .

JVM The performance optimization of is relatively easy , But if it's solved OOM, Except for some mentally retarded and simple , If someone keeps creating objects in the code . Many other products OOM problem , There are some technical difficulties , Need solid technology .

<> How is out of heap memory applied , How is it released ?

If in Java A block of off heap memory space should be applied for in the code , Is to use DirectByteBuffer This class , You can build one through this class DirectByteBuffer Object of , The object itself is JVM In heap memory .

But while you're building this object , A memory space will be set aside in the off heap memory to associate with this object , Let's look at the picture below , You know their relationship very well .

Therefore, when allocating off heap memory , That's the basic idea .

<> How to free off heap memory

When you DirectByteBuffer Object is not referenced , After garbage , At some point YGC or Full GC Recycled when .

Just recycle one DirectByteBuffer object , It frees its associated off heap memory :

<> Then why is there an out of heap memory overflow ?

If you create many DirectByteBuffer object , Takes up a large amount of off heap memory , Then these DirectByteBuffer Object not yet GC Thread to recycle , Then it won't be released !

When out of heap memory is heavily DirectByteBuffer Object association usage , If you need to use additional off heap memory , Report memory overflow ! When will there be a large number of DirectByteBuffer The object is always alive , As a result, a large amount of off heap memory cannot be released ?

It is also possible that the system is highly concurrent , Create too many DirectByteBuffer, Use a lot of off heap memory , Continue to use off heap memory at this time , Will OOM! But this is clearly not the case with the system .

<> The real reason for out of heap memory overflow

Can use jstat Observe the online system operation , At the same time, check the processing time of some requests according to the log , Analyze the past gc journal , After taking a look at the time-consuming call of each interface of the system , The analysis idea is as follows .

First, look at the time-consuming interface call , System concurrency is not high , But it takes more time to process each request , Average per request 1s.

then jstat find , Continuously called with the system , Various objects are always created , include Jetty It keeps creating itself DirectByteBuffer Object to apply for off heap memory space , Then until Eden full , Will trigger YGC:

But often in progress GC For a moment , Maybe some requests haven't been processed yet , There are many at this time DirectByteBuffer The object is alive , It hasn't been recycled yet , Of course, many before DirectByteBuffer The request corresponding to the object may have been processed , They can be recycled .

There must be some at this time DirectByteBuffer Objects and some other objects are alive , You need to transfer in Survivor area . Remember when the system went online , Memory allocation is extremely unreasonable , Gave the young generation one or two hundred M, The old generation gave seven or eight hundred M, Leading to Survivor only 10M. So often in YGC after , Some surviving objects ( Including some DirectByteBuffer) Will exceed 10M, Can't put it in Survivor, Direct entry Old:

So the process is repeated , Lead to some DirectByteBuffer Objects enter slowly Old,Old of DirectByteBuffer
More and more objects , And these DirectByteBuffer Are associated with a lot of off heap memory :

In these older generations DirectByteBuffer In fact, many are recyclable , But because the older generation hasn't been full , So it didn't trigger full
gc, It will naturally not recycle these from the elderly generation DirectByteBuffer Yes ! Of course, these are not recycled in the elderly generation DirectByteBuffer The association has occupied a large amount of off heap memory space !

Until the end , When you want to continue using off heap memory , All out of heap memory is heavily used by older generations DirectByteBuffer It's occupied , Although they can be recycled , But helpless, because it has not triggered the old age full
gc, Therefore, out of heap memory can never be recycled . Finally lead OOM!

<> this Java NIO Why does it look so sand sculpture ?

Java NIO Didn't you think this would happen ?

Considered ! He knows a lot DirectByteBuffer The object may not be used , But it was not triggered gc As a result, they always occupy off heap memory .Java
NIO The following treatment has been done , Every time a new out of heap memory is allocated , All call System.gc(), remind JVM Proactively perform the following GC, To recycle some garbage that no one cites DirectByteBuffer object , Free out of heap memory space .

As long as it can trigger GC To recycle something that no one quoted DirectByteBuffer, Some off heap memory will be released , Naturally, more objects can be allocated to off heap memory . But because we're here again JVM Set :
Cause this System.gc() Not effective , As a result OOM.

<> Ultimate optimization

The project has the following problems :

* Unreasonable memory setting , cause DirectByteBuffer Objects have been slowly entering the elderly generation , Out of heap memory has been unable to be freed
* Set -XX:+DisableExplicitGC, cause Java
NIO Some garbage can't be recycled actively DIrectByteBuffer object , It also leads to the failure to free off heap memory
This should be :

* Allocate memory reasonably , Give the younger generation more memory , Give Way Survivor There is more space in the area
* let go -XX:+DisableExplicitGC This restriction , Give Way System.gc() take effect
After optimization ,DirectByteBuffer Generally, they will not continue to enter the elderly generation . As long as he stays in the younger generation , along with young gc The memory outside the heap will be recycled normally .

Just let go -XX:+DisableExplicitGC limit ,Java
NIO Out of heap memory was found to be low , It will pass naturally System.gc() remind JVM Active garbage collection , Recycle some DirectByteBuffer, This frees up off heap memory .