C# Selenium 4: Setup request interception

Question

I'm trying to use Selenium 4 to log requests during manual usage of Chrome browser.

The issue is that request interception stops after around 40 seconds of usage (approximately).

I've tried to change commandTimeout but it didn't change anything. Also I've tried to look into chromedriver logs but I didn't find anithing there.

Here's my code:

static void Main(string[] args)
    {
        // Enable chromedriver logging
        var service = ChromeDriverService.CreateDefaultService();
        service.LogPath = AppDomain.CurrentDomain.BaseDirectory + "chromedriver.log";
        service.EnableVerboseLogging = true;
        var options = new ChromeOptions();

        var webDriver = new ChromeDriver(service, options);
        var devToolsSession = webDriver.CreateDevToolsSession();
        devToolsSession.Network.Enable(new EnableCommandSettings());

        EventHandler<RequestInterceptedEventArgs> requestIntercepted = (sender, e) =>
        {
            Console.WriteLine(e.Request.Url);
        };

        RequestPattern requestPattern = new RequestPattern();
        requestPattern.InterceptionStage = InterceptionStage.Request;
        requestPattern.ResourceType = ResourceType.Image;
        var setRequestInterceptionCommandSettings = new SetRequestInterceptionCommandSettings();
        setRequestInterceptionCommandSettings.Patterns = new RequestPattern[] { requestPattern };
        devToolsSession.Network.SetRequestInterception(setRequestInterceptionCommandSettings);
        devToolsSession.Network.RequestIntercepted += requestIntercepted;

        while (true)
        {
            webDriver.Url = "https://translate.google.com/";
            Thread.Sleep(5000);
            webDriver.Navigate().Refresh();
        }
    }

Answer 1

As of Selenium 4 beta-1, you can use the Fetch API and retrieve the same resluts (SetRequestIntercepted is deprecated in ChromeTools as late as December 29, 2020).

In either case, the key point for both RequestIntercepted (prior to deprecation) and Fetch to be able to continue after intercepting the request is to capture the RequestId.

With Fetch this can be done in ContinueRequest():

fetch.ContinueRequest(new Fetch.ContinueRequestCommandSettings()
{
   RequestId = e.RequestId
});

Below is your code updated to use Fetch which allowed the query to run for 2+ minutes and 5+ minutes before I manually stopped it:

(Note: ensure all of your DevTools as using the same version (i.e. 89 in my case) or you will get errors):

using V89 = OpenQA.Selenium.DevTools.V89;
using V89Net = OpenQA.Selenium.DevTools.V89.Network;
using OpenQA.Selenium.DevTools.V89.Log;

var service = ChromeDriverService.CreateDefaultService();
service.LogPath = AppDomain.CurrentDomain.BaseDirectory + "chromedriver.log";
service.EnableVerboseLogging = true;
var options = new ChromeOptions();

new DriverManager().SetUpDriver(new ChromeConfig(), VersionResolveStrategy.MatchingBrowser);

var driver = new ChromeDriver(service, options);

IDevTools devTools = driver as IDevTools;

var devToolsSession = devTools.GetDevToolsSession();

var fetch = devToolsSession.GetVersionSpecificDomains<V89.DevToolsSessionDomains>()
    .Fetch;
    
var enableCommandSettings = new V89.Fetch.EnableCommandSettings();

var requestPattern = new V89.Fetch.RequestPattern();
requestPattern.RequestStage = V89.Fetch.RequestStage.Response;
requestPattern.ResourceType = V89Net.ResourceType.Document;

enableCommandSettings.Patterns = new V89.Fetch.RequestPattern[] { requestPattern };

fetch.Enable(enableCommandSettings);

void RequestIntercepted(object sender, V89.Fetch.RequestPausedEventArgs e)
{

    e.Request.Url.Dump();

    fetch.ContinueRequest(new V89.Fetch.ContinueRequestCommandSettings()
    {
        RequestId = e.RequestId
    });
}

fetch.RequestPaused += RequestIntercepted;

while (true)
{
    driver.Url = "https://translate.google.com/";
    Thread.Sleep(5000);
    driver.Navigate().Refresh();
}

Screenshot (From left-to-right): LinqPad output Console.Write.Url equivalent, Above code in LINQPad, and view of Chrome log file)