Trying to scrape google with HtmlAgilityPack - stuck at cookies site

41 views Asked by At

this is my first problem on this site.. so please bear with me.

I'm trying to scrape google shopping by using HtmlAgilityPack.

I have established a connection, but I'm stuck at their cookies(Which is a site that redirects you to the google shopping search afterwards - when doing it manually). So they are essentially blocking me I guess I'd say.

I've tried to follow the step by step guide looking like this:

using HtmlAgilityPack;

HtmlWeb web = new HtmlWeb();


HtmlDocument doc = web.Load("https://www.google.com/search?q=bicycle&source=lnms&tbm=shop");



var headernames = doc.DocumentNode.Descendants();//.Where(n => n.GetAttributeValue("class", "").Equals("tAxDx"));
var titles = new List<Row>();
foreach (var headername in headernames)
{
    titles.Add(new Row
    {
        title = headername.InnerText
    });
}

foreach (var title in titles)
{
    Console.WriteLine(title.title);
}

Console.ReadLine();
public class Row
{
    public string title { get; set; }

}

I've outcommented my where statement - since it doesnt return anything, given we are on the cookies site.

Any ideas or thoughts on how to solve this question? I'm fairly new to scraping, and still a student developer.. So I might need some guidance. Thank you.

Note: I see alot of suggestions trying different stuff - nothing has led me further than to acces the cookies site of google. (including WebBrowser from windows)

I'm trying to scrape google shopping by using HtmlAgilityPack.

I have established a connection, but I'm stuck at their cookies(Which is a site that redirects you to the google shopping search afterwards - when doing it manually). So they are essentially blocking me I guess I'd say.

I'm trying to scrape the titles of products in the shopping view of google.

0

There are 0 answers