elasticsearch - get top n records group by sort by value

Question

The top level doc has a field categories which is an array of {"categoryId": xyz, "sortOrder": 123}.

I want to group by categoryId, and get top 10 records sorted by sortOrder.

I tried

  "aggs": {
    "categories": {
      "nested": {
        "path": "classCategories"
      }, 
      "aggs": {
        "group-by-category": {
          "terms": {
            "field": "classCategories.categoryId",
            "order": {
              "_key": "asc"
            }
          },
          "aggs": {
            "top-classes-by-category": {
              "reverse_nested": {}, 
              "aggs": {
                "top-classes": {
                  "top_hits": {
                    "size": 10,
                    "sort": [
                      {
                        "classCategories.sortOrder": {
                          "order": "asc",
                          "nested": {
                            "path": "classCategories"
                          }
                        }
                      }
                    ]
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "size": 0
}

At first glance, the output seems to be what is expected.

But then for the buckets starting from second, the sort is still based on sortOrder of the previous bucket.

I also tried to use bucket_sort, but all the examples I found were based on sum aggregation. In this case, I don't want any sum or other aggregation, but just use the value as is.

Hope the question is clear.