JVM hangs during the serialization

Question

I am debugging our server which hangs every couple of days.

java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

Out of nowhere, everything freezes and the server stops processing/receiving any requests (or any work to be precise). After capturing the thread dump, I noticed a couple of things:

• No deadlocks, according to jstack
• 199 threads in BLOCKED state, 5 in IN_NATIVE state and 1 in IN_VM state
• All of the blocking threads are blocked at wait/sleep/receive/etc methods

Only the thread that is in a IN_VM actually shows some significant stack:

Thread 10453: (state = IN_VM)
 - java.lang.System.identityHashCode(java.lang.Object) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.ObjectOutputStream$HandleTable.hash(java.lang.Object) @bci=1, line=2360 (Compiled frame)
 - java.io.ObjectOutputStream$HandleTable.insert(java.lang.Object, int) @bci=2, line=2323 (Compiled frame)
 - java.io.ObjectOutputStream$HandleTable.growSpine() @bci=57, line=2338 (Compiled frame)
 - java.io.ObjectOutputStream$HandleTable.assign(java.lang.Object) @bci=28, line=2279 (Compiled frame)
 - java.io.ObjectOutputStream.writeOrdinaryObject(java.lang.Object, java.io.ObjectStreamClass, boolean) @bci=105, line=1428 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject0(java.lang.Object, boolean) @bci=506, line=1178 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject(java.lang.Object) @bci=16, line=348 (Compiled frame)
 - java.util.HashMap.internalWriteEntries(java.io.ObjectOutputStream) @bci=49, line=1793 (Compiled frame)
 - java.util.HashMap.writeObject(java.io.ObjectOutputStream) @bci=24, line=1363 (Compiled frame)
 - sun.reflect.GeneratedMethodAccessor13.invoke(java.lang.Object, java.lang.Object[]) @bci=40 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)
 - java.io.ObjectStreamClass.invokeWriteObject(java.lang.Object, java.io.ObjectOutputStream) @bci=24, line=1140 (Compiled frame)
 - java.io.ObjectOutputStream.writeSerialData(java.lang.Object, java.io.ObjectStreamClass) @bci=125, line=1497 (Compiled frame)
 - java.io.ObjectOutputStream.writeOrdinaryObject(java.lang.Object, java.io.ObjectStreamClass, boolean) @bci=137, line=1432 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject0(java.lang.Object, boolean) @bci=506, line=1178 (Compiled frame)
 - java.io.ObjectOutputStream.defaultWriteFields(java.lang.Object, java.io.ObjectStreamClass) @bci=18, line=1523 (Compiled frame)
 - java.io.ObjectOutputStream.defaultWriteObject() @bci=41, line=441 (Compiled frame)
 - java.util.Collections$SynchronizedMap.writeObject(java.io.ObjectOutputStream) @bci=8, line=2691 (Compiled frame)
 - sun.reflect.GeneratedMethodAccessor12.invoke(java.lang.Object, java.lang.Object[]) @bci=40 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)
 - java.io.ObjectStreamClass.invokeWriteObject(java.lang.Object, java.io.ObjectOutputStream) @bci=24, line=1140 (Compiled frame)
 - java.io.ObjectOutputStream.writeSerialData(java.lang.Object, java.io.ObjectStreamClass) @bci=125, line=1497 (Compiled frame)
 - java.io.ObjectOutputStream.writeOrdinaryObject(java.lang.Object, java.io.ObjectStreamClass, boolean) @bci=137, line=1432 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject0(java.lang.Object, boolean) @bci=506, line=1178 (Compiled frame)
 - java.io.ObjectStreamClass$FieldReflector.getPrimFieldValues(java.lang.Object, byte[]) @bci=313, line=2135 (Compiled frame)
 - java.io.ObjectStreamClass.getPrimFieldValues(java.lang.Object, byte[]) @bci=6, line=1389 (Compiled frame)
 - java.io.ObjectOutputStream.defaultWriteFields(java.lang.Object, java.io.ObjectStreamClass) @bci=70, line=1533 (Compiled frame)
 - java.io.ObjectOutputStream.writeSerialData(java.lang.Object, java.io.ObjectStreamClass) @bci=211, line=1509 (Compiled frame)
 - java.io.ObjectOutputStream.writeOrdinaryObject(java.lang.Object, java.io.ObjectStreamClass, boolean) @bci=137, line=1432 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject0(java.lang.Object, boolean) @bci=506, line=1178 (Compiled frame)
 - java.io.ObjectOutputStream$BlockDataOutputStream.flush() @bci=1, line=1822 (Compiled frame)
 - java.io.ObjectOutputStream.flush() @bci=4, line=719 (Compiled frame)
 ....
 ....
 - few more frames where we invoke `flush()`

The object that we are attempting to serialize looks like this:

public class DomainObjectCollection implements Serializable
{
    private static final long serialVersionUID = -2050273337993827124L;

    private Map<String, DomainObject> objects;
    private Map<String, List<DomainObject>> objectsForUsername;

    public DomainObjectCollection()
    {
        this.objects = Collections.synchronizedMap(new HashMap<>());
        this.objectsForUsername = Collections.synchronizedMap(new HashMap<>());
    }
}

And we serialize the object rather simply:

os = new ObjectOutputStream(new BufferedOutputStream(new FileOutputStream(file), 128 * 1024));
                os.writeObject(object);
                os.flush();

So, apart from synchronization, nothing fancy.

Has anyone encountered something similar? If I missed providing some details please feel free to ask.

Update

About an hour ago from above thread dump, I captured the mixed stack trace:

----------------- 10453 -----------------
0x00007f0e5b65d709  __pthread_cond_timedwait + 0x129
0x00007f0e5a8bcbb7  _ZN18ObjectSynchronizer12FastHashCodeEP6ThreadP7oopDesc + 0x377
0x00007f0e5a5746df  JVM_IHashCode + 0x4f
0x00007f0e4577d13f  <Unknown compiled code>
Locked ownable synchronizers:
    - None