pagination node.js mongoose express, am I doing it wrong?

Question

Afternoon all. I have code that "works" but I fear it might be "dumb".
I've recently added a route to handle a regex search against my mongodb. The client sending the request is very limited where it can only handle < 2K body size. So I needed to add pagination to break up a large return into multiple http requests...

Importantly, I really don't know what I don't know and my solution is likely built on ignorance. I have concerns about scalability and redundancy. To get a count on returned query, I am running the query twice. Once to get a count of all the records, and again to paginate it. The client can then loop through other calls to get all the data it requires. that'll still require two runs per hit... dumb.

My searching has danced around these issues with lots of arguments that are frankly over my head. Aggregation? buckets? caching a query?

In writing the route below it felt like I should be able to paginate the initial totalData .find but couldn't figure out how to paginate those results. So I just ran it again with pagination.

Anyway, let me know where I went astray:

//Regex info is passed in the body
//mmValue string containing the fields of a document I want to retrieve ex: -_id mmKey mmValue updateAt 
//mmPage return this page, defaults to 1
//mmLimit limit documents returned
//mmField what field in the document to apply the regex against ex: updatedAt or mmKey  or _id 
//mmRegex the regular expression to applied to mmField ex: Findme

router.post('/mmRegex', async (req, res) => {
    try {
        const returnData = req.body.mmValue 
        const page = parseInt(req.body.mmPage)
        const limit = parseInt(req.body.mmLimit)

        const map = new Map([[ req.body.mmField , new RegExp(req.body.mmRegex,req.body.mmRegexOp)]])
        const rquery = Object.fromEntries(map)
        const totalData = await Model.find(rquery).select(returnData)
        const pageData = await Model.find(rquery).skip((page -1)*limit).limit(limit).select(returnData)
        const totalPages = Math.ceil(totalData.length / limit)
        console.log(" RegEx: ",rquery, " count: ",totalData.length);
        const results = {
            count: totalData.length,
            cPage: page,
            tPage: totalPages,
            pageData
        }
        res.send(results)
    }
    catch (error){
        console.log(" RegEx: ",rquery," msg: ", error.message);
        res.status(500).json({ status: "failed", message: error.message })  
    }
})

Again this does return the data I need, but likely in a foolish way Thanks for time!

Edit!

I took the example and flushed it out. So far it's working about twice as fast as the original route above. 220ms vs 94ms against the same dataset respectively. Also added a jsonCheck as passing invalid json to the route tended to crash the app.

router.post('/mmRegex', async (req, res) => {
    if (req.body.mmValue) {
        let jsonCheck;
        try {
            jsonCheck=JSON.parse(req.body.mmValue);
        }
        catch (error) {
            res.status(500).json({ status: "failed", message: error.message }) 
            return console.log("Error invalid json in mmValue" , req.body.mmValue)
        }
    }
    const returnData=JSON.parse(req.body.mmValue)
    const page = parseInt(req.body.mmPage)
    const limit = parseInt(req.body.mmLimit)
    const map = new Map([[ req.body.mmField , new RegExp(req.body.mmRegex,req.body.mmRegexOp)]])
    const rquery = Object.fromEntries(map)

    try {
        const pageData = await Model.aggregate([
            {
              $facet: {
                total: [
                  { $match: rquery,},
                  { $count: "count",},
                ],
        
                docs: [
                  {$match: rquery,},
                  {$project: returnData, },
                  {$skip: ((page -1) *limit),},
                  {$limit: limit,},
                ],
              },
            },
          ])

        console.log("Ok RegEx: ",rquery );
        res.send(pageData)
    }
    catch (error){
        console.log("Error RegEx: ",rquery," msg: ", error.message);
        res.status(500).json({ status: "failed", message: error.message })  
    }
})

Answer 1

You can retrieve the documents count by using the countDocuments function.

const count = await Model.countDocuments(filter)

Also the same can be done in single query using facet. Your query would be

const data = await Model.aggregate([
    {
      $facet: {
        total: [
          {
            $match: filter,
          },
          {
            $count: "count",
          },
        ],

        docs: [
          {
            $match: filter,
          },
          {
            $skip: skipValue,
          },
          {
            $limit: limitValue,
          },
        ],
      },
    },
  ])

$facet mongo

countDocuments