This picture should show it all:
As you can see the minimum instance is set on three, but autoscale decides to first scale up, and than scale back down again - below the minimum.
I guess even if the CPU is far below the target I assume it should never go below the minimum. Why is this happening? Do I need to set somewhere else a setting?
Oops. Apparently I had set up a day / night time schedule, and at this certain point it went to the night time schedule where it was min/max 1 instance.