I am trying to scrape the table from the URL. I have been using requests library for a while now, along with beautiful soup, but what I don't want to venture into is using a web driver, as I have been down that road before.
So I make a request with requests, and read the response. But I get the following in the header, and nothing else. Can someone explain to me what I need to do (spent all morning on it and beginning to lose the plot)?
<head>
<meta charset="utf-8">
<title>SoccerSTATS.com - cookie consent</title>
<style>
.button {
background-color: #4CAF50; /* Green */
border: none;
color: white;
text-align: center;
text-decoration: none;
display: inline-block;
font-size: 18px;
margin: 4px 2px;
cursor: pointer;
}
.button1 {padding: 10px 24px;}
.button2 {padding: 12px 28px;}
.button3 {padding: 14px 40px;}
.button4 {padding: 32px 16px;}
.button5 {padding: 16px;}
</style>
<script type="text/javascript">
function setCookielocal(cname, cvalue, exdays) {
var d = new Date();
d.setTime(d.getTime() + (exdays*24*60*60*1000));
var expires = "expires="+d.toUTCString();
var originpage = "/team.asp?league=england_2018&stats=20-bournemouth";
document.cookie = cname + "=" + cvalue + "; " + expires;
window.location = "//www.soccerstats.com" + originpage;
}
</script>
</head>
(Source: http://go-colly.org/articles/scraping_related_http_headers/)
The only thing you need to do is to set a legitimate user-agent. Therefore add headers to emulate a browser. :
Example:
Additionally, you can add another set of headers to pretend like a legitimate browser. Add some more headers like this: