How to broadcast cache invalidate messages to all servers running a web app?

Question

I have a Java based web app hosted on AWS. It is read-mostly so it makes a lot of sense to cache objects retrieved from the database for performance.

When I do update an object, I would like to be able to broadcast to all the servers that the object was saved and it should be invalidated from all local caches.

The does not need to be real time. Stale objects are annoying and need to be flushed within about 20 seconds. Users notice if they stick around for minutes. Cache invalidation does not have to happen the millisecond that objects get saved.

---

What I've thought about

• I've looked into broadcast technologies just as jGroups, but jGroups isn't supported on AWS.
• I don't think that Amazon's SQS messaging service can be made into a broadcast service.
• I'm considering using the database for this purpose: I'd write events to a database table and have each server poll this table every few seconds to get a new list items.

Answer 1

Two options come to mind. The first is to use Amazon SNS, which can use SQS as a delivery backend. This might be overkill, though, since it's designed as a frontend to lots of delivery types, including e-mail and SMS.

The approach I'd try is something along the lines of Comet-style push notifications. Have each machine with a cache open a long-lived TCP connection to the server who's responsible for handling updates, and send a compact "invalidate" message from that server to everyone who's listening. As a special-purpose protocol, this could be done with minimal overhead, perhaps just by sending the object ID (and class if necessary).

Answer 2

Redis is handy solution for broadcasting a message to all subscribers on a topic. It is convenient because it can be used as a docker container for rapid prototyping, but is also offered by AWS as a managed service for multi-node clusters.

Setting up a ReactiveRedisOperations bean:

@Bean
public ReactiveRedisOperations<String, Notification> notificationTemplate(LettuceConnectionFactory lettuceConnectionFactory){
    RedisSerializer<Notification> valueSerializer = new Jackson2JsonRedisSerializer<>(Notification.class);
    RedisSerializationContext<String, Notification> serializationContext = RedisSerializationContext.<String, Notification>newSerializationContext(RedisSerializer.string())
            .value(valueSerializer)
            .build();
    return new ReactiveRedisTemplate<>(lettuceConnectionFactory, serializationContext);
}

Subscribing on a topic:

@Autowired
private ReactiveRedisOperations<String, Notification> reactiveRedisTemplate;

@Value("${example.topic}")
private String topic;

@PostConstruct
private void init() {
    this.reactiveRedisTemplate
            .listenTo(ChannelTopic.of(topic))
            .map(ReactiveSubscription.Message::getMessage)
            .subscribe(this::processNotification);
}

Publishing a message on a topic:

@Autowired
private ReactiveRedisOperations<String, Notification> redisTemplate;

@Value("${example.topic}")
private String topic;

public void publish(Notification notification) {
    this.redisTemplate.convertAndSend(topic, notification).subscribe();
}

RedisInsight is a GUI that can be used for interacting with redis.

Here is a complete sample implementation using spring-data-redis.