datahub connects to hive, uses kerberos, occurred an error in web-ui

Question

When I was using the datahub ui for hive data ingestion, there was an exception that Could not connect to it

info: [2024-02-04 08:11:42,352] INFO {thrift.transport.TSocket:118} - Could not connect to ('69.167.164.199', 10000)

Here is my recipe.

source: type: hive config: host_port: 'xxxxx.test.com:10000' database: dm_prod username: hive include_views: false stateful_ingestion: ignore_old_state: true enabled: true database_pattern: allow: - dm_prod table_pattern: allow: - 'dm_prod.*' options: connect_args: auth: KERBEROS kerberos_service_name: hive scheme: hive+https pipeline_name: ingest-scheduler-test

When I run this recipe on the cli, I can normally ingest hive data. However, when I ingested the data through the web-ui, it could not be executed properly. I'm curious, what's the problem with that, thank you

I try to find a solution, but so far, there is still no good solution

Answer 1

source:
type: hive
config:
    host_port: 'xxxx.test.com:10000'
    database: dm_prod
    username: hive
    include_views: false
    stateful_ingestion:
        ignore_old_state: true
        enabled: true
    database_pattern:
        allow:
            - dm_prod
    table_pattern:
        allow:
            - dm_prod.*
    options:
        connect_args:
            auth: KERBEROS
            kerberos_service_name: hive
            scheme: hive+https

pipeline_name: 'ingest-test'