U-SQL - Execution related queries

Question

I wrote multiple U-SQL scripts and its output got stored in ADLA, based on this I have few question.

• How we can run dependent jobs in U-SQL?
• How to execute statement based on some condition like

If RecordCount > 0 then
     insert into table1
endif

• How we can schedule U-SQL jobs?
• Can we write multiple scripts and call them from main script?
• During script execution, compiler prepare and compiles the code. It took almost 30-40 secs. How we can bundle the compiled code and create the ADF pipeline?

Answer 1

You can schedule and orchestrate U-SQL jobs with Azure Data Factory or by writing your own scheduler with one of the SDKs (Powershell, C#, Java, node.js, Python).

U-SQL supports two ways for conditional execution:

1. If your conditional can be evaluated at compile time, e.g., when you pass a parameter value or check for the existence of a file, you can use the IF statement.
2. If your conditional can only be determined during the execution of the script, then you can use the WHERE clause as wBob outlines in his comment.

As wBob mentions, you can encapsulate most of the U-SQL statements in procedures and then call them from other scripts/procedures, or you can write your own way of inclusion/orchestration if you need script file reuse.

There is currently no ability to reuse and submit just compiled code, since the compilation depends on the exact information such as what files are present and the statistics of the accessed data.