Web scraping from multiple tables appearing on click

Question

Basically I would like to open this page, select "Rüzgar" from the last dropdown, run the query with "Sorgula" button and extract all the coordinates stored in the table appearing once clicked the first button of the first column in the main table. I want to do that for all the rows.

Unfortunately I don't have sufficient programming experience to carry out this task. However, since I am a little bit familiar with programming, I think if somebody would point out the correct source for me to learn how to do that (regarding the requirements of the web page from which I am trying to extract data) I can build a small script for this task, maybe with scrapy or some other tool.

P.S.:I tried to do it with scrapinghub's Portia, but that did not work either.

Answer 1

take a look at the Python module called selenium, namely the webdriver part of it. Some quick code that would do perform the search query you're after would be written as such:

from selenium import webdriver

driver = webdriver.Firefox()
search_link = 'http://lisans.epdk.org.tr/epvys-web/faces/pages/lisans/elektrikUretimOnLisans/elektrikUretimOnLisansOzetSorgula.xhtml?lisansDurumu=7'

driver.get(search_link)
last_dropdown_menu = driver.find_element_by_id('elektrikUretimOnLisansOzetForm:j_idt32')

last_dropdown_menu.click() # send a click to the element
last_dropdown_menu.send_keys('R') # scroll to Ruzgar
sorgula_button = driver.find_element_by_xpath('//*[@id="elektrikUretimOnLisansOzetForm:j_idt51"]/span[2]').click()

from there, you can figure out how to scrape the info you're after :-)

Answer 2

Selenium might be ok since there are only 3 pages when you set the pagination on the bottom to 500. Nevertheless I wouldn't go with selenium because it's ... there are better ways.

All you do when you click the "Rüzgar" button is a POST request with the following arguments:

Open the chrome debugger and see for yourself the type of requests you're doing. You can replicate the request. If you're interested in this method tell me to - maybe - write some more.