Airflow DBT TypeError: SQLServerConnectionManager.execute() got an unexpected keyword argument 'limit'

210 views Asked by At

I am running the airflow DBT with ODBC connection in docker container. I run airflow using docker-compose.yml file.

The docker-compose file is same as this curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.7.3/docker-compose.yaml'

Additionally i have added the volume for DBT in docker-container.yaml file.

  ...
  volumes:
    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
    - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
    - ${AIRFLOW_PROJ_DIR:-.}/dbt_lindace_airflow:/opt/airflow/dbt_airflow
    - ${AIRFLOW_PROJ_DIR:-.}/ssl:/etc/airflow/ssl
    ...

When i run the DBT using the airflow DAG

 dbt_test_raw = DbtTestOperator(
     dag=dag,
     task_id="dbt_test_raw",
     dir="./dbt_airflow",
     target="dev",
     vars=dict(
        customer_name=self.customer_name,
     ),
     select="raw.fb",
     profiles_dir="./"
 )

or using the following command from docker exec

dbt test --profiles-dir ./ --target dev --vars '{"customer_name": "abc"}' --select raw.fb

I am getting error.

module 'dbt.exceptions' has no attribute 'RuntimeException'
15:32:12  Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 302, in exception_handler
    yield
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 52, in macro
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/adapters/base/impl.py", line 310, in execute
    return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)
TypeError: SQLServerConnectionManager.execute() got an unexpected keyword argument 'limit'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/adapters/sqlserver/sql_server_connection_manager.py", line 242, in exception_handler
    yield
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/adapters/base/impl.py", line 1112, in execute_macro
    result = macro_function(**kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 21, in macro
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 35, in macro
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 255, in call_macro
    with self.exception_handler():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/clients/jinja.py", line 304, in exception_handler
    raise CaughtMacroErrorWithNodeError(exc=e, node=self.macro)
dbt.exceptions.CaughtMacroErrorWithNodeError: Compilation Error
  SQLServerConnectionManager.execute() got an unexpected keyword argument 'limit'
  
  > in macro sqlserver__list_relations_without_caching (macros/adapters/metadata.sql)
  > called by macro list_relations_without_caching (macros/adapters/metadata.sql)
  > called by <Unknown>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/cli/requires.py", line 90, in wrapper
    result, success = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/cli/requires.py", line 75, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/cli/requires.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/cli/requires.py", line 197, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/cli/requires.py", line 244, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/cli/requires.py", line 284, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/cli/main.py", line 894, in test
    results = task.run()
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/task/runnable.py", line 474, in run
    result = self.execute_with_hooks(selected_uids)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/task/runnable.py", line 434, in execute_with_hooks
    self.before_run(adapter, selected_uids)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/task/run.py", line 447, in before_run
    self.populate_adapter_cache(adapter, required_schemas)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/task/runnable.py", line 412, in populate_adapter_cache
    adapter.set_relations_cache(self.manifest)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/adapters/base/impl.py", line 519, in set_relations_cache
    self._relations_cache_for_schemas(manifest, required_schemas)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/adapters/base/impl.py", line 495, in _relations_cache_for_schemas
    for relation in future.result():
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/utils.py", line 471, in connected
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/adapters/sql/impl.py", line 185, in list_relations_without_caching
    results = self.execute_macro(LIST_RELATIONS_MACRO_NAME, kwargs=kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/adapters/base/impl.py", line 1111, in execute_macro
    with self.connections.exception_handler(f"macro {macro_name}"):
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/airflow/.local/lib/python3.10/site-packages/dbt/adapters/sqlserver/sql_server_connection_manager.py", line 259, in exception_handler
    if isinstance(e, dbt.exceptions.RuntimeException):
AttributeError: module 'dbt.exceptions' has no attribute 'RuntimeException'

How can I resolve this?

0

There are 0 answers