Django - Dojo/Dgrid - how to manage LARGE data sets

Question

6.30.15 - HOW CAN I MAKE THIS QUESTION BETTER AND MORE HELPFUL TO OTHERS? FEEDBACK WOULD BE HELPFUL. THANKS!

I am developing a web application that will handle/manage a VERY LARGE data set - Currently any kind of heavy load causes the browser to lock up - whether I'm in the Django Rest Framework API or in the Dojo/Dgrid. This is kind of a dual question.

I've researched and can't find a clear way to do this on either side.

How do I limit how much the database sends at one time to the Django Rest Framework and/ or The Dojo Dgrid. The Dgrid pulls the data from the Django Rest API. The DRF pulls data directly from the MySQL database.

If I can control how much data is sent at one time, then hopefully it won't lock up the browser as much. ANY suggestions, advice, help, examples would be very helpful. Thanks in advance!

UPDATED 6.22.15 -

Alright, I FINALLY Got the pagination to work and it display the limit/offset in the headers. :) YAY!!!! I can also see the data in the Response headers. HOWEVER... the grid won't populate and I keep getting this odd error: TypeError: transform(...) is null

return transform(value, key).toString();

instrum...tion.js (line 20)

I've gotten this error before, but I've never been able to find a solution to it. After researching, there's not much I can find on HOW to fix or really even what it is. Any help with this would be greatly appreciated!! I'm SO CLOSE to getting this thing to work correctly after WEEKS and WEEKS of beating my head against a wall. Please help! :) Thanks in advance!!!

2nd Update - This was an answer from a previous post - but I'm still not sure how to fix it. When I addressed another issue - it went away for awhile, but I still have no idea how to correct the issue.

Problem 3: "transform(...) is null return transform(value, key).toString();"

This sounds largely tangential to the original issue, but the most common cause is a widget template that is referencing a property via ${...} that doesn't actually exist in the widget.

Answer 1

I don't know how to answer this on the layer between DRF and the database, but as discussed in other SO questions like this one, DRF allows you to limit the amount of data sent with requests via page or offset/limit parameters.

Based on the phrasing of your question, it sounds like the client side is actually requesting too much data. I'll outline how the flow should work, so hopefully you can spot what you've missed:

1. A dgrid instance is set up with a collection referencing a dstore/Rest instance
2. The dstore/Rest instance is created with appropriate properties set. In this case, based on the DRF Documentation:
   • useRangeHeaders: false (this is already the default)
   • rangeStartParam: 'offset'
   • rangeCountParam: 'limit'
3. As a result, when the grid renders, you should see requests sent to your server e.g. endpoint?offset=0&limit=25 - if you don't see those two parameters, that would be why you're getting too much data
4. The server will need to query the database with the respective OFFSET and LIMIT
5. The server must provide a response with the expected number of items (except if it reaches the end of the data set first, which should be reflected by the total property in the response, presuming the customization in the previous SO answer I linked is used)

Ultimately, if the service is working as expected, the grid should only be requesting a handful of items at a time, and should only be firing one or two requests at any given time.

Answer 2

Would add as a comment, but not enough reputation at the moment ....

Your question is pretty general, but one strategy would be to allow the user to select the number of items they wish to view at a time and then allow the user to page through the data with 'next x items' and 'prev x items' buttons. Your data object query would then use the current position +/- 'x' as the index range to reduce the number of data objects sent to the browser. This is the basic flow for Ebay, Amazon, Google, or any site with thousands of items to display. The 'next' and 'prev' actions could be wired as POST requests.