I am working with a school project where I need to obtain the analyst price targets estimates of certain stocks from the Yahoo Finance Website (this is mandatory).
When I tried to use it by beautiful soup I couldn't scrape it (I believe JS is adjusting the page as it loads), so I turned to selenium to obtain such data. However when I'm trying to obtain the elements through the XPATH, it returns error as if it doesn't exist. I am using EC in case it needs to load, but its not working. I've tried modifying the wait times up until 2 minutes, so that is not the issue
Code below:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--headless")
chrome_options.add_argument(f'user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36')
chrome_options.add_argument("window-size=1920,1080")
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://finance.yahoo.com/quote/BBAJIOO.MX?.tsrc=fin-srch")
driver.delete_all_cookies()
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="Col2-11-QuoteModule-Proxy"]/div/section/div')))
Anyone have an idea as to why this is happening? and how can I obtain such ratings?
Image below of desired ratings
This is the example of the HTML Code:
<div aria-label="Low 60 Current 64.59 Average 69.25 High 76.8" class="Px(10px)">
<div class="Pos(r) Pb(30px) H(1em)">
<div class="Start(75%) W(25%) Pos(a) H(1em) Bdbc($seperatorColor) Bdbw(2px) Bdbs(d) T(30px)"></div>
<div class="Pos(a) H(1em) Bdbc($seperatorColor) Bdbw(2px) Bdbs(s) T(30px) W(100%)"></div>
<div class="Pos(a) D(ib) T(35px)" data-test="analyst-cur-tg" style="left: 27.3214%;">
<div class="W(7px) H(7px) Bgc($linkActiveColor) Bdrs(50%) Z(1) B(-5px) Translate3d($half3dTranslate) Pos(r)"></div>
<div class="Bgc($linkActiveColor) Start(0) T(5px) W(1px) H(17px) Z(0) Pos(r)"></div>
<div class="Miw(100px) T(6px) C($linkActiveColor) Pos(r) Fz(s) Fw(500) D(ib) Ta(c) Translate3d($half3dTranslate)"><span>Current</span> <span>64.59</span></div>
</div>
<div class="Pos(a) D(ib) T(-1px)" data-test="analyst-avg-tg" style="left: 55.0595%;">
<div class="Pos(r) T(5px) Miw(100px) Fz(s) Fw(500) D(ib) C($primaryColor)Ta(c) Translate3d($half3dTranslate)"><span>Average</span> <span>69.25</span></div>
<div class="Pos(r) Bgc($tertiaryColor) W(1px) H(17px) Z(0) T(6px) Start(-1px)"></div>
<div class="W(8px) H(8px) Bgc(t) Bd Bdc($seperatorColor) Bdrs(50%) Z(1) B(-6px) Pos(r) Translate3d($half3dTranslate)"></div>
</div><span class="W(6px) H(6px) Bgc($tertiaryColor) Bdrs(50%) Z(0) B(-5px) Start(0) Pos(a) Translate3d($half153dTranslate)"></span><span class="W(6px) H(6px) Bgc($tertiaryColor) Bdrs(50%) Z(0) B(-5px) Pos(a) Translate3d($zero153dTranslate) Start(100%)"></span></div>
<div class="Ov(a) Fz(xs) Mt(10px) C($tertiaryColor)">
<div class="Pos(r) Fl(start) Fz(xs) C($tertiaryColor) "><span>Low</span> <span>60.00</span></div>
<div class="Pos(r) Fl(end) Fz(xs) C($tertiaryColor) "><span>High</span> <span>76.80</span></div>
</div>
</div>
