I'm trying to create a backend in Spring Data Mongodb. I have the following code which works and I have used the built in methods by extending my repo with the MongoRepository class:
@RequestMapping(value="/nextpost", method=RequestMethod.GET)
public List getNextPosts(@RequestParam int next) {
Pageable pageable = new PageRequest(next, 5, new Sort(new Sort.Order(Direction.DESC, "id")));
page = repo.findAll(pageable);
return page.getContent();
}
The above code will return the page as per the page number inserted into the "next" variable.
My android frontend however allows for things to be added and deleted from the database and this causes problems with this pagination method. Lets take an example:
- When my android frontend starts up, it loads the first 5 items by calling "getNextPosts" with next = 0.
- My android frontend also keeps track of the page it is on and increments it when the user wants to see more items.
Now, we immediately add 5 more items.
When I swipe up to fetch the next 5 items, it calls the "getNextPosts" method passing the the "next page" value = 1. The app will load the same 5 items originally displayed when the app was started as the 5 "NEW" items I have added just pushed the 5 "OLD" items down in the database.
Therefore on the app, we see 15 items comprising of:
5 "NEW" + 5 "OLD" + 5 "OLD"
So if I gave numbers to all my items on my android ListView, I would see:
15
14
13
12
11
// the above would be the new items added
10
9
8
7
6
//the above would be the original items on page 0
10
9
8
7
6
//the above would be still be the original items but now we are on page 1
Does anyone know how one can solve this issue so that when I swipe up, the items would be:
15
14
13
12
11
// the above would be the new items added
10
9
8
7
6
5
//the above would be the original items on page 0
4
3
2
1
0
//the above would be on page 1
tl;dr
That's the nature of the beast. Pagination in Spring Data is defined as retrieving a part of the result set at the time of querying. Especially for remote communication, that kind of statelessness is usually the best tradeoff between keeping state, keeping connections open, scalability etc.
Details
The only way to avoid this would be to capture the state of the database at the time of the first access and only work on that. You can actually build this by retrieving all items and page through the data locally.
Of course hardly anyone does this as it easily gets out of hand for larger data volumes. Also, this would bring up other problems like: when do you actually want to see the items introduced in the meantime? So the definition of "correct content" when paginating a list is not distinct.
Mitigation strategies
If applicable to your scenario you could try to apply a sorting that guarantees new items to be added at the very end and thus basically making this an append-only list. This would naturally sort the most recent items last though, which is contrary to what's needed often times.
If you use the pagination to work down a list of items and process all of them, another approach is to keep track of the identifiers of the items you already have processed. In your particular scenario, you'd be able to detect that the items have already been processed and go on with the next page. This of course only makes sense if you read and process faster than someone else manipulates the list in the backend.