Problems with running duckdb (from ibis-framework) script

60 views Asked by At

I have this issue when using duckdb with ibis in iPython:

So I use duckdb with ibis and try to access parquet data on s3.

I basically do:

import boto3
import ibis 

def get_duck_db_client() -> BaseBackend:
    boto3.resource("s3")
    boto3.client("s3")
    con = ibis.duckdb.connect(database=":memory:", read_only=False)

    con.raw_sql(
        f"""
    INSTALL httpfs;
    LOAD httpfs;
    SET s3_region='{AWS_REGION}';
    SET s3_access_key_id='{AWS_ACCESS_KEY}';
    SET s3_secret_access_key='{AWS_SECRET_KEY}';
    """
    )
    return con

if __name__ == "__main__":
    con = get_duck_db_client()
    data = con.read_parquet(S3_PATH)

I want to use this in an ipython shell. But, when I do %run script.py I get:

OperationalError: (duckdb.duckdb.TransactionException) TransactionContext Error: cannot start a transaction within a transaction
(Background on this error at: https://sqlalche.me/e/20/e3q8)

Things work fine when I just execute the code in the ipython shell directly (i.e. without running the script).

Is there a way to prevent this in this setting?

0

There are 0 answers