Query documents based on sum of nested fields - elasticsearch

Question

I'm fairly new to Elastic and I've been trying to figure out if it is possible to query documents based on the sum or count of nested fields within them. All the research I've done so far would say that it's not, but maybe there's a way around it?

I know about aggregations but these output a total instead of changing the search results based on the total.

For example, I have a set of data which looks like this:

{
  "records": [
    {
      "JhseTzYuOF_first": "Liam",
      "JhseTzYuOF_last": "McAneny",
      "record_activity": [
        {
          "category": "YfV068",
          "comment": "Appeal",
          "created": "2015-06-14T21:41:22+10:00",
          "value": 15
        }
      ],
      "$id": "-Jr8G3YEMjafAvRP1a8I"
    },
    {
      "JhseTzYuOF_first": "Pha-Hai",
      "JhseTzYuOF_last": "Nguyen",
      "record_activity": [
        {
          "category": "YfV068",
          "comment": "Donated",
          "created": "2015-06-14T13:49:23+10:00",
          "value": 10
        },
        {
          "category": "YfV068",
          "comment": "Donated",
          "created": "2015-06-14T13:49:55+10:00",
          "value": 15
        }
      ],
      "$id": "-Jr8G3Y7JQrBTb-cpFsT"
    }
  ]
}

record_activity is mapped as nested and I can query on it, but I was hoping I could do something like:

show all where sum of record_activity.value > 20

Any ideas how to achieve this through ElasticSearch?