Hazelcast caller member throws timeout exception when a serialization exception happens on callee member

Question

I'm using Hazelcast (JAVA, version 3.7.5) in a scenario with 2 members. The first member delegates a task to the second member through an IExecutorService. After some processing, the second member tries to send back an unserializable response.

As it's not possible to send back a response, the second member prints the stacktrace related to the HazelcastSerializationException.

As no response arrives, the first member throws an OperationTimeoutException when the operation-heartbeat-timeout is reached.

Currently, when the IExecutorService fails at parsing the Callable result, it prints a stacktrace (on callee side). Let say I have a simple caller :

private Future<Object> startFlow() {
    //This throws an OperationTimeoutException
    return hazelcastInstance.getExecutorService("myExecutor").submit(myRunnable);
}

Which calls a simple callee :

@Override
public Object call() throws Exception {
    //The object returned is not serializable, therefore an HazelcastSerializationException is thrown
    return service.execute();
}

The callee prints a stacktrace after it failed to parse the response (see end of post).

In my case, it's not possible to know what kind of object the service might return, and it's not possible to trust the service to send back serializable objects.

I would like to be able to know the reason of the timeout on the caller-side.

After some search, I found that no configuration/API is available to intercept exceptions thrown by an IExecutorService when it fails to serialize a response.

So I tried to see if it would be possible to check if an object was parseable by Hazelcast, again without success.

Any ideas ?

Thank you

---

The stacktrace printed by the callee will be looking like this :

Exception in thread "hz._hzInstance_1_dev.cached.thread-1" com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.spi.impl.operationservice.impl.responses.NormalResponse'
    at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:73)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:143)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:124)
    at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.send(OperationServiceImpl.java:427)
    at com.hazelcast.spi.impl.operationservice.impl.RemoteInvocationResponseHandler.sendResponse(RemoteInvocationResponseHandler.java:51)
    at com.hazelcast.spi.Operation.sendResponse(Operation.java:291)
    at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.sendResponse(DistributedExecutorService.java:269)
    at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:253)
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:212)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.myomain.UnserialiableObject'
    at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:73)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:236)
    at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:371)
    at com.hazelcast.spi.impl.operationservice.impl.responses.NormalResponse.writeData(NormalResponse.java:91)
    at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.write(DataSerializableSerializer.java:189)
    at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.write(DataSerializableSerializer.java:54)
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:140)
    ... 12 more
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable serializer for class com.myomain.UnserialiableObject
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.serializerFor(AbstractSerializationService.java:469)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:232)
    ... 18 more

EDIT (SOLUTION)

So I ended up registering a global serializer, that would simply send an Exception whenever it's called. Something like this :

public class GlobalSerializerException  implements StreamSerializer<Object> {

    @Override
    public void write(ObjectDataOutput out, Object object) throws IOException {
        String objectInfo;
        if(object == null){
            objectInfo = "Object was null.";
        }else{
            objectInfo = String.format("Object of class %s and printed as String gives %s", object.getClass().getCanonicalName(), object.toString());
        }
        objectInfo = "Hazelcast was unable to serialize an object. " + objectInfo;
        out.writeUTF(objectInfo);
    }

    @Override
    public Object read(ObjectDataInput in) throws IOException {
        String message = in.readUTF();
        HazelcastSerializationException hazelcastSerializationException = new HazelcastSerializationException(message);
        return hazelcastSerializationException;
    }

    @Override
    public int getTypeId() {
        return 63426;
    }

    @Override
    public void destroy() {

    }
} 

Answer 1

Caller fails with a timeout because it doesn't get a response from the target. Target node fails to serialize the response, as a reason fails to send a response. This is current behaviour but I think it's also possible to send a special exception to denote that response failure too.

Hazelcast, by default, is able to serialize classes implementing java.io.Serializable, java.io.Externalizable and some Hazelcast specific interfaces, such as DataSerializable, Portable. It's also possible to define custom serializers or delegate to another serialization library. See Hazelcast Reference Manual - Serialization section for more info.

In a distributed system, messages exchanged between nodes must be serializable to a binary form to transmit them through network. So, an entity/service participating in a distributed system must ensure its messages are serializable in some form.

If you don't know the type of the messages, then you can register Hazelcast a global serializer, which first tries to serialize using known formats (Serializable, Externalizable etc), if type is not known then it writes a custom error message instead. Alternatively, you can wrap result of the service's execution with a custom serializable wrapper object. During serialization, if original wrapped result fails to serialize, then you again write a custom error message.

For example;

class NonSerializableResponseException extends Exception {}

class ServiceResponseWrapper implements DataSerializable {

    private Object response;

    @Override
    public void writeData(ObjectDataOutput out) throws IOException {
        try {
            out.writeObject(response);
        } catch (HazelcastSerializationException e) {
            out.writeObject(new NonSerializableResponseException());
        }
    }

    @Override
    public void readData(ObjectDataInput in) throws IOException {
        response = in.readObject();
    }
}