I have around 500k unique keys and billions of transactions per day and each transaction can hold a different key. With each transaction, we would like to get a boolean of whether the key already exists in the Redis list. The unique key size can grow and depends on the day of the week.
I learned about the Redis stack and bloom filtering which can help to efficiently return the key that does not exist or the probability of the key not existing when a collision happens. This looks like a good fit for us. However, we would like to know what size of memory should be prepared for this Redis stack and what is the acceptable probability for ensuring the key does not already exist before? How is the calculation to determine the probability and memory size?
Other than Redis stack and Bloom filtering. Are there other alternatives (e.g. Redis search) that will fit my question? Please advise.