I run a query from AWS Athena console and takes 10s. The same query run from Sagemaker using PyAthena takes 155s. Is PyAthena slowing it down or is the data transfer from Athena to sagemaker so time consuming?
What could I do to speed this up?
I run a query from AWS Athena console and takes 10s. The same query run from Sagemaker using PyAthena takes 155s. Is PyAthena slowing it down or is the data transfer from Athena to sagemaker so time consuming?
What could I do to speed this up?
Just figure out a way of boosting the queries:
Before I was trying:
Figured out that using a PandasCursor instead of a connection is way faster
Ref: https://github.com/laughingman7743/PyAthena/issues/46