Cleanest Way to Get a Set in Redis where Individual Elements Expire in 30 Days?

Question

I need a data structure similar to a Python set in redis, with the additional capability of individual elements automatically expiring (getting popped) from the set 30 days after insertion. Basically, these are the abstract behaviours I want out of the class.

from abc import abstractmethod, ABC
from typing import Iterable 

class RedisSet(ABC):
    """Implement a set of strings with a remote redis host."""
    @abstractmethod
    def __init__(self, url:str):
        """
           Initialise the class at the remote redis url.
           The set of strings should be empty initially.            
        """
        raise NotImplementedError    
    @abstractmethod
    def add(self, new_elements:Iterable[str])->None:
        """Insert all the elements into the set."""
        raise NotImplementedError
    @abstractmethod
    def __contains__(self, elem:str)->bool:
        """Check membership."""
        raise NotImplementedError

So what would be the cleanest way of achieving it? I do not need the entire class implemented, but asking what would be the correct data types and APIs to use in redis, as I am not thoroughly familiar with full capabilities of redis.

I noted redis has a set datatype, but seems (happy to be corrected if I am wrong) it does not support any time to live (TTL). Au contraire, the dictionary supports TTL, but I have to use a placeholder value for each key (unnecessary overhead) and I am not sure whether the membership check will be constant time.

N. B.

• I do not foresee any need to iterate through the elements, so optimising with respect to that operation is superfluous. All I need is membership check.
• I intend to use aioredis as python client. It should support the necessary redis operations.

Any idea of the correct redis data types whose documentations I should look up will be greatly appreciated.

Answer 1

To achieve a similar functionality you want in Redis you should consider that there is a limitation with Redis and the final solution may seem complex.

What is the limitation?

As you mentioned correctly, Redis doesn't provide a TTL feature with HASH, SETS, or any other data types. Just support the simple Key/Value.

What is the solution?

You can use SortedSet. This data type gets a score for each element you add to it which in this case will be the expiration time. You need to maintain the sorted set by the code to remove expired elements from it. You can implement this part with a scheduler.

Add new Elements

ZADD my-set 1699631227694 elm1
(integer) 1
ZADD my-set 1699631228234 elm2 1699631228234 elm3
(integer) 2
ZADD my-set 1699631229123 elm1
(integer) 0

(integer) ${number} - ${number} determine the number of element you added, if you get 0 or less than your input elements means you've just updated the score of the element.

Check Existance

Now you can check the existance of a key by this command, Note that you need to verify that the elememnt is not expired, so in you code need to check the time with the score.

ZSCORE my-set elm1 // which retuen the score of elm1
"1699631227694"
ZSCORE my-set elm4
(nil)

(nil) means the element does not exist

Removing outdated elements

This is run by the your schaduler to remove element that are expired from the sorted set

ZREMRANGEBYSCORE my-set -inf 1699631228234
(integer) 2

Unfortunately, I'm not a Python developer and can't help you with the code. But I think this code help you to get insight:

from abc import abstractmethod, ABC
from typing import Iterable 

class RedisSet(ABC):
    """Implement a set of strings with a remote redis host."""
    @abstractmethod
    def __init__(self, url:str):
        """
           Initialise the class at the remote redis url.
           The set of strings should be empty initially.            
        """
        self.redis = // init the redis connection
        self.set_key = set_key
        
    @abstractmethod
    def add(self, new_elements:Iterable[str])->None:
        """Insert all the elements into the set."""
        const startTime = Date.now();
        const offset = 1000*60*60*24*30; // 30 days
        
        for elem in new_elements:
            await self.redis..zadd(self.set_key, Date.now() + offset, elem)

    @abstractmethod
    def __contains__(self, elem:str)->bool:
        """Check membership."""
        exists = await self.redis.zscore(self.set_key, elem)

        if score is None or score < time.time():
            return False
        
        return True