Optimising feedback forms backend design for gaming applications

Question

I have to design and develop a feature in the backend for feedback forms in a gaming application.

So the use case is that admin will create multiple feedback forms which will be shown to the user in the gaming application. This form will consist of some questions related to feedback for the application.

The form will have a start time and end time and will only be displayed to user when a form is active i.e. its current time is between the start time and end time and when a certain set of triggers are obtained for a user. Here trigger could be game won or XP Increased. These are predefined triggers that will be associated with each form created by the admin.

At the end of the game, an API will be called which will check triggers registered with a user and will fetch relevant feedback from the database according to the triggers registered and the current time. My design was to store this data in some document database (NOSQL) as there is no need for transactions and the format of forms can vary For forms, I would create a collection form_collection with a document structure as below

{
  
  _id: ObjectId,                 // Unique identifier for the form
  title: String,                  // Title or name of the feedback form
  description: String,            // Description or additional information about the form
  created_at: Date,               // Timestamp indicating when the form was created
  status : bool, 
  start_time : Date, //Timestamp when the form will be active
  end_time : Date, //Timestamp when the form will be inactive
  questions: [
    {
      question_id: ObjectId,      // Unique identifier for the question
      question_text: String,       // The actual text of the question
      question_type: String,       // Type of question (e.g., open-ended, multiple-choice, rating-based)
      options: [String]            // If applicable, an array of available options for multiple-choice questions
    },
    // Other questions...
  ],
  triggers: [ObjectId]            // Array of trigger IDs associated with the form 
}

And the user collection will have document structure as

{
  _id: ObjectId,                 // Unique identifier for the user
  username: String,               // User's username or identifier
  email: String,                  // User's email address
  // Other user profile fields...

  triggers: [
    {
      trigger_id: ObjectId,       // Unique identifier for the trigger
      triggered_at: Date          // Timestamp indicating when the trigger was triggered for the user
    },
    // Other triggers...
  ]
}

Now there will be two flows

Flow one: Storing trigger info with the user

Whenever a trigger happens Game won that trigger will be stored inside user collection

Flow two: Fetching feedback form for a user

We will fetch a current set of triggers for a user and then query document collection where the current time is between the start_time and end_time of forms and then from that list we will filter those documents which have all the triggers mapped that were obtained by the user This will involve searching two times and does not seem optimized. I can add indexes over start time and triggers array in form_collection however this does not seems optimized for high traffic.

Another approach I can think of is

Precompute trigger list

Whenever the admin creates a form we will have a key in some Key-value kind of nosql database for a set of triggers and a list of documents will be added there. This will shift some complexity to form creation from admin.

<TriggerOne-TriggerTwo-TriggerThree> : [Forms]

Is there any other approach that could reduce the complexity of fetching forms each time and also is the pre-computation approach correct as it will hit a limit for the list of forms (no of forms that can be added in a list) Any suggestion would be highly appreciated. Also, I am a fresher so sorry if I was not able to explain correctly/sufficiently, and is open to discussion in comments for any rectification of requirements. I can use any database for backend. MongoDB is not neccesary. Thank you again, guys.

Answer 1

Your current design and approach are good in terms of using a NoSQL database to store flexible forms and using triggers to dynamically fetch forms. However, as you noted, the process of fetching forms can become inefficient in high-traffic scenarios.

The alternate approach you suggested, precomputing the form list for a set of triggers, is a good one, but like you mentioned, it can hit a limit on the number of forms that can be added in a list.

I would suggest a hybrid approach, based on ideas like "Caching a MongoDB Database with Redis", or "Database Cashing in-memory with Redis NoSQL Databases":

1. Keep your current design: Your current database design is good as it allows flexible form structures and user-trigger associations. It is a good practice to keep a normalized form of data in your database for integrity and flexibility.

2. Add a caching layer: When an admin creates a new form, you can precompute the mapping between triggers and forms and store it in a cache such as Redis. This cache can be a hashmap where the key is the set of triggers and the value is the list of form ids. This will speed up the form fetching process as you will not have to query the database each time.

3. Limit the cache size: To prevent the cache from becoming too large, you can set a limit on the size of the form list for each trigger set. If the list exceeds this limit, you can either remove the oldest forms (based on the end_time) or not add the new form to the cache. You can decide this based on your specific use case.

4. Fall back to the database: If a form is not found in the cache, you can fall back to querying the database. This can happen if the cache limit is exceeded or if the cache is not available for some reason. This ensures that your system is still functional even if the cache is not available.

5. Update the cache periodically: You can update the cache periodically to remove forms that are no longer active (based on the end_time) or to add new forms that were not added to the cache because of the limit. This can be done using a background job that runs at regular intervals.

This approach allows you to maintain the flexibility of your current design while reducing the complexity and time of fetching forms for each user. The caching layer provides a speedup for the common case, while the database provides a fallback for edge cases. This approach is also scalable as it can handle a large number of forms and users.

---

The caching layer can be implemented using a cache server such as Redis. Here are the general steps:

1. Install Redis: Redis is a popular open-source, in-memory data structure store that can be used as a database, cache, and message broker. It is known for its high performance and support for diverse data structures such as strings, hashes, lists, sets, and more.

2. Create a Cache Service: This is a service in your backend code that interacts with Redis. The service should have methods for adding, fetching, and deleting data from Redis.

3. Pre-compute and Add Data to Redis: When an admin creates a new form, precompute the set of triggers as a single string (for example, "TriggerOne-TriggerTwo-TriggerThree"). Use this string as the key and the form ID as the value and store it in Redis. If the key already exists, append the new form ID to the existing list.

Here is a pseudocode example of adding data to Redis:

key = "TriggerOne-TriggerTwo-TriggerThree"
value = form_id
if redis.exists(key):
    redis.append(key, value)
else:
    redis.set(key, [value])

4. Fetch Data from Redis: When you need to fetch forms for a user, compute the set of triggers as a string and fetch the corresponding form IDs from Redis. If the data is not found in Redis, fall back to querying the database.

Here is a pseudocode example of fetching data from Redis:

key = "TriggerOne-TriggerTwo-TriggerThree"
form_ids = redis.get(key)
if form_ids is None:
    // Fetch from database
else:
    // Use form_ids

5. Update Data in Redis: Create a background job that runs at regular intervals to update the data in Redis. This job should remove forms that are no longer active and add new forms that were not added to Redis because of the cache size limit.

Remember, while Redis is an in-memory store, it also offers persistence features, so you can configure it according to your needs.

Do note that the above examples are oversimplified and in practice, you would need to handle edge cases, errors, and maintain a connection pool to Redis. The exact implementation will also depend on the programming language and framework you are using.

---

Just want to confirm that, in case when we want to show forms to the user, we will fetch a list of all triggers that the user has, and then fetch the form accordingly from the Redis, right?

Yes, that is correct. When you want to show forms to the user, you would first fetch the triggers associated with the user.
For each of these triggers, you would then look up the corresponding forms in your Redis cache.

Here is a more detailed step-by-step process:

1. Fetch User Triggers: Retrieve the user's triggers from your main database (MongoDB in your original design).

2. Generate Key Strings: For each trigger or combination of triggers (depending on how you have designed your trigger-to-form mapping), generate the key string that you have used in your Redis cache.

3. Fetch Forms from Redis: For each of these key strings, fetch the corresponding form IDs from your Redis cache.

4. Fallback to Database: If for any reason a key is not present in Redis, you would fall back to querying your main database for the forms. This could happen if your Redis cache is cold, or if there are more forms for a specific set of triggers than can be stored in your Redis cache.

5. Compile List of Forms: Combine all the form IDs you have fetched from Redis (and potentially your main database) into a single list, ensuring to remove any duplicates. This is the list of forms that you would then present to the user.

Again, do note that the actual implementation might be a bit more complex, depending on your specific requirements and constraints.
For example, you might want to consider sorting the forms by some priority or another attribute, or you might need to filter out forms that are no longer active based on their start and end times.