Right now, I'm using SuperObject to parse my JSON data (when receiving it from the server) and then dump it into a client dataset. There can be very large amounts of data coming from the server. Currently, I have to first parse that JSON data into an ISuperObject, and then iterate through that to populate the client dataset.
What I'm wondering is whether there's a way to bypass the double-loading, triggering events as the raw data is being parsed. For example, as soon as the parser detects the beginning of a new object, it would trigger an event that I would then prepare that corresponding object on the spot. Or when it parses an array, I can prepare a new dataset record.
The reason is because with very large datasets, it takes a few seconds some times, and I have to wait for it to finish being parsed before I can make use of it. If I get the data while it's being parsed, I can immediately make use of that data on the spot.
I'm already implementing pagination at between 200-500 records per page (on millions of records) with many columns. Pagination still doesn't solve the full needs when it comes to responsiveness.
How can I go about this, either using SuperObject, or any other known mechanism, without having to write my own complete parser?
Writing your own parser is not that hard. I've written one for the mongodb connector I've written: https://github.com/stijnsanders/TMongoWire/blob/master/bsonUtils.pas You could easily adapt that one to fire the events you describe.