this is my first problem on this site.. so please bear with me.
I'm trying to scrape google shopping by using HtmlAgilityPack.
I have established a connection, but I'm stuck at their cookies(Which is a site that redirects you to the google shopping search afterwards - when doing it manually). So they are essentially blocking me I guess I'd say.
I've tried to follow the step by step guide looking like this:
using HtmlAgilityPack;
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("https://www.google.com/search?q=bicycle&source=lnms&tbm=shop");
var headernames = doc.DocumentNode.Descendants();//.Where(n => n.GetAttributeValue("class", "").Equals("tAxDx"));
var titles = new List<Row>();
foreach (var headername in headernames)
{
titles.Add(new Row
{
title = headername.InnerText
});
}
foreach (var title in titles)
{
Console.WriteLine(title.title);
}
Console.ReadLine();
public class Row
{
public string title { get; set; }
}
I've outcommented my where statement - since it doesnt return anything, given we are on the cookies site.
Any ideas or thoughts on how to solve this question? I'm fairly new to scraping, and still a student developer.. So I might need some guidance. Thank you.
Note: I see alot of suggestions trying different stuff - nothing has led me further than to acces the cookies site of google. (including WebBrowser from windows)
I'm trying to scrape google shopping by using HtmlAgilityPack.
I have established a connection, but I'm stuck at their cookies(Which is a site that redirects you to the google shopping search afterwards - when doing it manually). So they are essentially blocking me I guess I'd say.
I'm trying to scrape the titles of products in the shopping view of google.