Way to get current editor id (doc id) in HiveServerClient in Hue source code

Question

When having multiple hue pages run tez applications at the same time, it, sometimes, will apply the same session to two different tasks, which will cause of of them receiving KILL signal and the other one complains that current app master is being used and retrying. I looked into the code of HiveServerClient._get_tez_session and I think the problem lies in the way busy_sessions is retrieved, which is not thread-safe. So there's chance that two query will be allocated to the same session when submitted virtually the same time.

I'd like to know is there any way to get current editor id (doc_id) from HiveServerClient._get_tez_session method, so I could do some hacking for a quick solution now. Thanks.

Answer 1

You can solve this by disabling Tez session mode

set tez.am.mode.session=false;

Session mode is more aggressive in reserving execution resources and is typically used for interactive applications where multiple DAGs are submitted in quick succession by the same user. For long running applications, one-off executions, batch jobs etc non-session mode is recommended. If session mode is enabled then container reuse is recommended.

Also try to disable container reuse:

set tez.am.container.reuse.enabled=false;

See all Tez configuration settings here.

Also read this thread about Tez session naming. I did not test it myself, maybe you can use hive.session.id property for getting/setting session id's.