Our sql warehouse in databricks is shared among many teams. Some have few queries to run and some teams have large queries on large tables. How can I attribute cost on this shared resource. What all we know of a query DAtabricks provide few metrics around query runs like
- Query runtime
- Query memory usage
- Rows scanned
Also we know the DBUs a sql warehouse is consuming. We were thinking of using the memory consumed by the query to calculate the cost by getting the cost of running per mb/ per sec
Let me know if there are better ways to doing it