Databricks odbc overhead and latency

Question

I'm executing a sql query against Databricks serverless Sql warehouse size Large running against Unity catalog. I'm running the code from my own machine and my latency to Azure West Europe is about 50ms.

The code looks like this

from databricks import sql
import datetime
from tabulate import tabulate
import msvcrt
from datetime import datetime
import dateutil.tz

connection = sql.connect(
                        server_hostname = "adb-abc.12.azuredatabricks.net",
                        http_path = "/sql/1.0/warehouses/def",
                        access_token = "dapighi-2")

cursor = connection.cursor()

key_pressed = False
while not key_pressed:

    current_time_with_timezone = datetime.now(dateutil.tz.tzlocal()).isoformat()

    cursor.execute(f"select to_timestamp('{current_time_with_timezone}') as client_time, current_timestamp() as server_time")# from prod.myschema.mytable limit 1")
    all_data = cursor.fetchall()

    client_time = all_data[0][0]
    server_time = all_data[0][1]
    difference = server_time - client_time
    print("Client Time: ", client_time)
    print("Server Time: ", server_time)
    print("Difference: ", difference)
    
    print("-----------------------------------------")

    # Check if any key in input buffer but don't wait for input
    if msvcrt.kbhit():
        key_pressed = True
        msvcrt.getch()
        print("Key Pressed")

In the output I'm constantly seeing that the datetime added to the query string is ~1 second before the datetime set on the server.

If I remove the table from the select I still see a ~0.4s difference

cursor.execute(f"select to_timestamp('{current_time_with_timezone}') as client_time, current_timestamp() as server_time")

Is there any way to reduce this latency or overhead?

Update: When running a simple query in Databricks web ui I can see the below query information. Additionally is says that the data was fetched from cache. How can I reduce the "Optimizing query & pruning files" phase?