How to improve performance of grouped DataGrid?

Question

I am having some trouble with WPF when using GroupStyle in a DataGrid.

Loading the data itself is fine (about 2,500 rows). Also having GroupDescriptions on the CollectionView or calculating over the elements of the groups is absolutely fine in respect to performance.

But the visualisation is a complete nightmare.

I whittled it down to this. Once I define a GroupStyle like such in the DataGrid:

<DataGrid.GroupStyle>
    <GroupStyle>
        <GroupStyle.HeaderTemplate>
            <DataTemplate>
                <Grid>
                </Grid>
            </DataTemplate>
        </GroupStyle.HeaderTemplate>
    </GroupStyle>
</DataGrid.GroupStyle>

It takes like 10 times as long to load the data and also uses 20 times the amount of RAM although there is actually no visual difference (Grid is yet empty)

This cannot be right. I must be doing something wrong. What am I doing that is causing the DataGrid to be so slow using the above GroupStyle?