Elasticsearch - Term Aggregation In Theory

Question

imagine a random term aggregation on a specific field:

"aggs":{
"top_terms": {
  "terms": {
    "field": "any.specific.field"
    }
  }
}

My question here is: how does ES aggregate terms?

If the inverted & fielddate index looks like this: ES: inverted index & fielddata index and the fact that unique terms are stored per document, not per field, how does ES aggregate terms per field? What is happening behind the scenes to aggregate them?

Can somebody shed some light to me/us? Thanks in advance