Get source port and packets/bytes counters from Azure AzureNetworkAnalytics_CL

Question

I am trying to get flow data on an Azure application using KQL AzureNetworkAnalytics_CL.

1. In the documentation there is no field for the source port, it is a mandatory field for my solution (5-touple based), is there anyway to get it?

2. Kusto query failed to return PacketsDestToSrc, PacketsSrcToDest, BytesDestToSrc and BytesSrcToDest.

I tried the following Java code:

    public static void main(String[] args) throws ParseException {
        String kustoQuery = "AzureNetworkAnalytics_CL | project TimeGenerated, SrcIP_s, DestIP_s, DestPort_d, L4Protocol_s, PacketsDestToSrc";
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'");
        sdf.setTimeZone(TimeZone.getTimeZone("UTC"));

        TokenCredential tokenCredential = new ClientSecretCredentialBuilder()
                .clientId("my-client-id")
                .clientSecret("my-secret")
                .tenantId("my-tenant")
                .build();

        LogsQueryClient logsQueryClient = new LogsQueryClientBuilder()
                .credential(tokenCredential)
                .buildClient();

        LogsQueryResult queryResults = logsQueryClient.queryWorkspace("my-workspace-id", kustoQuery,
                new QueryTimeInterval(Duration.ofDays(1)));

        for (LogsTableRow row : queryResults.getTable().getRows()) {
            Date timeGenerated = sdf.parse(row.getColumnValue("TimeGenerated").get().getValueAsString());
            System.out.println(
                    new StringBuilder().
                            append("Time: ").
                            append(timeGenerated.toString()).append(", source ip: ")
                            .append(row.getColumnValue("SrcIP_s").get().getValueAsString())
                            .append(", dest ip: ")
                            .append(row.getColumnValue("DestIP_s").get().getValueAsString())
                            .append(", dest port: ")
                            .append(row.getColumnValue("DestPort_d").get().getValueAsString())
                            .append(", protocol: ")
                            .append(row.getColumnValue("L4Protocol_s").get().getValueAsString())
                            .toString());
        }
}

I am getting the following error:

Caused by: com.azure.monitor.query.implementation.logs.models.ErrorResponseException: Status code 400, "{"error":{"message":"The request had some invalid properties","code":"BadArgumentError","correlationId":"some-id","innererror":{"code":"SemanticError","message":"A semantic error occurred.","innererror":{"code":"SEM0100","message":"'project' operator: Failed to resolve scalar expression named 'PacketsDestToSrc'"}}}}"

Same for PacketsSrcToDest, BytesDestToSrc and BytesSrcToDest.

Update: I am using log schema version 2:

any Idea why?

Answer 1

After a workaround on it, I found that the table AzureNetworkAnalytics_CL only contains the log information about the Azure Network security groups and firewalls. It doesn't contain information about the virtual network flow logs. And PacketsSrcToDest is a field which comes under virtual network flow logs but not NSG flow logs.

Refer MS Doc for VNet flow logs fields.

In order to retrieve the PacketsSrcToDest/BytesDestToSrc related data, you need to use Azure Diagnostics with a suitable category type (Eg: Virtual Network) and for retrieving the port details use clientPort_d or port_d in general purpose. Refer the provided Diagnostics MS Doc and obtain the log data according to your requirement.

To do so, firstly enable the virtual network flow logs by visiting the VM/NSG linked VNet as shown below.

I have also verified the AzureNetworkAnalytics_CL table after enabling NSG Flow logs by referring to the @TheSleepyAdmins article and was able to obtain only the NSG flow log columns as shown.

AzureNetworkAnalytics_CL
| Project TimeGenerated, FlowLogStorageAccount_s,VirtualMachine_s,PublicIPAddresses_s,VirtualSubnetwork_s