Azure Application Insights - Not recording all requests on high traffic situations

Question

It seems "Azure Application Insights" not recording all the requests on high traffic environment.

For example when we testing the .Net Core2.1 Web API app which is deployed on "Azure Service Fabric" with 10,000 requests for the period of 30 mins, can able to retrieve details of all the requests from "Azure Application Insights" via KQL using datetime stamp as filter, no problem.

When we increase the load to 100,000 requests for the period of 30 mins, only around 5-10% of the requests get recorded on "Azure Application Insights".

Why "Azure Application Insights" misses ingest/recording on high traffic environment which serves approximately 60 requests/second ?

Any extra configuration required? (or) Below one line code to ingest details of requests served by Azure Service Fabric service, is not enough? please clarify

SDK used,

Code used on Azure Service Fabric for ingestion

  return new WebHostBuilder().UseHttpSys()
                                .ConfigureServices(
                                    services => services
                                        .AddSingleton<StatelessServiceContext>(serviceContext)
                                        .AddSingleton<ServiceFabricAppContext>(new ServiceFabricAppContext(){
                                                NodeName = serviceContext.NodeContext.NodeName,
                                                ServiceHostIP=serviceContext.NodeContext.IPAddressOrFQDN,
                                                ServiceHostPort=FabricRuntime.GetActivationContext().GetEndpoints()[0].Port
                                        } )
                                            .AddSingleton<ITelemetryInitializer>((serviceProvider) => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext))) // Azure Service Fabric Telemetry Initializer
                                .UseContentRoot(Directory.GetCurrentDirectory())
                                   .UseApplicationInsights()
                                .UseStartup<Startup>()
                                .UseEnvironment(environment)
                                .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                .UseUrls(url)
                                .Build();

Sample query with item count

Sampling not enabled from Azure portal

Answer 1

You can disable adaptive sampling from code. Note instead of using deprecated .UseApplicationInsights() (removed that below), I am using .AddApplicationInsightsTelemetry below.

  return new WebHostBuilder().UseHttpSys()
                                .ConfigureServices(
                                    services => services
                                        .AddSingleton<StatelessServiceContext>(serviceContext)
                                        .AddSingleton<ServiceFabricAppContext>(new ServiceFabricAppContext(){
                                                NodeName = serviceContext.NodeContext.NodeName,
                                                ServiceHostIP=serviceContext.NodeContext.IPAddressOrFQDN,
                                                ServiceHostPort=FabricRuntime.GetActivationContext().GetEndpoints()[0].Port
                                        } )
                                       .AddSingleton<ITelemetryInitializer>((serviceProvider) => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext)) // Azure Service Fabric Telemetry Initializer
                                       .AddApplicationInsightsTelemetry(o => 
                                            {
                                                o.EnableAdaptiveSampling = false; // disabling adaptive sampling
                                            }))
                                .UseContentRoot(Directory.GetCurrentDirectory())
                                .UseStartup<Startup>()
                                .UseEnvironment(environment)
                                .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                .UseUrls(url)
                                .Build();

NOTE: I had to add nuget Microsoft.ApplicationInsights.AspNetCore

CAUTION: Disabling sampling might result throttling and too much network traffic in case of very high volume scenario. So you might want to look at Fixed rate sampling instead of adaptive. Refer Sampling in Application Insights.