I have a power bi dataset with aprox. 300MB in pbix file size and 1.2GB RAM memory size (according to Vertipaq analyser in dax studio).
I've always been able to refresh dataset directly from power bi website using its own servers without using premium capacity ( and still can update the same dataset)
Most of the dataset inputs comes from Athena AWS as input mode, not direct query and we dont have any incremental refresh so far.
I's been a few months that we started to deploy some reports using Power BI Capacities. Most of our reports are not heavy ram/cpu usage and we use a A1 capacity without problems.
The issue starts when we try to refresh our dataset that is deployed by the capacity in times where our users are requesting data from it. We trigger the refresh from microsoft API. At first, we started to get some memory errors so we changed our script to first change capacity from A1 to A3 or A4, wait 60 seconds to be sure that new memory is available ( is this really necessary?) and then trigger the dataset refresh.
Even with A4 capacity, i'm unable to refresh the dataset with the following error after some minutes (usually takes 20mins to refresh, errors appears around 10-15 minutes running):
"This operation was canceled because there wasn’t enough memory to finish running it. Either increase the memory of the Premium capacity where this dataset is hosted or reduce the memory footprint of your dataset by doing things like limiting the amount of imported data. More details: consumed memory 1764 MB, memory limit 1764 MB, database size before command execution 1307 MB. Learn more, see https://go.microsoft.com/fwlink/?linkid=2159753."
What sees strange is the total memory description in the error, aprox. 3GB that would be the A1 capacity, but i'm 100% sure that i've changed the capacity to A4 before the refresh started using microsoft API and even double checked on Azure website.
Also, it's worth mentioning that I'm able to refresh this dataset using the same strategy when we first turn the capacity on (its early in the day about 06:30am and no one is actually consuming resources from capacity).
I'm aware that capacity needs twice the size of aprox. ram memory size + incoming querys when the dataset refresh is requested. But in my calculations, that would be 1.2GB * 2 = 2.4GB + incoming querys. Since A1 capacity can run perfect to users demand, I can't understand why i'm having trouble updating a dataset with A4 capacity (25gb ram available).
Am I doing something wrong changing the capacity before trying to update? It seems like it has no effects on the total RAM available to refresh.
In the scenario that a incoming query is using GBs of ram and reaching the 25gb ram ( i dont thing this is the case), how can I be sure that this is not happening or find out who is requesting such query?
This is my first time writting a question here, I apologize in advance for any mistakes and would be very glad for any help.
Tried to refresh dataset that is served by Power BI Capacity after upgrading capacity to A4. I expected to refresh the dataset without problemns and even without needing to upgrade to A4 (this sounds to me really unnecessary). The problem appears after some minutes only when the same capacity is receiving query from users reports.