Unable to scrape data looks encrypted

Question

I am a beginner web scraping I am trying to get the phone number from this page. https://www.quickerala.com/listings?q=Healthcare&location=Alleppey

Under this element tag

<span data-qk-el-trackcontact="1" data-trackdata="{&quot;business&quot;:&quot;427151&quot;,&quot;address&quot;:&quot;430820&quot;,&quot;number&quot;:&quot;653252%6252%3252%553252%6252%3252%653252%6252%3252%843252%6252%3252%653252%6252%3252%553252%6252%3252%753252%6252%3252%843252%6252%3252%653252%6252%3252%753252%6252%3252%&quot;,&quot;type&quot;:&quot;mobile&quot;,&quot;page&quot;:&quot;businessListings&quot;}" data-qk-el-unobfuscate="1" data-unobfuscate-text="653252%6252%3252%553252%6252%3252%653252%6252%3252%843252%6252%3252%653252%6252%3252%553252%6252%3252%753252%6252%3252%843252%6252%3252%653252%6252%3252%753252%6252%3252%">9809780878</span>

I can view the phone number "9809780878" but I am not sure how to get it.

I have tried

response.xpath('//div[@class="listContacts brtop-10"]//data-trackdata') 

response.xpath('//span[@data-qk-el-trackcontact=1]/data-trackdata') 

response.xpath('//span[@data-qk-el-trackcontact=1]').extract()   

without any luck.

Answer 1

Mobile number is encrypted in the response in a specific area.

Though numbers are avialable in other area too. But in this solution I have focused on copy there(website) decryption JS function and execute in our python script

# Library Import
from bs4 import BeautifulSoup as Soup
import requests as rq
import js2py
import json

Find & copy their JS decryption function & excute using js4py

# JS function to decode mobile number
script = '''
function(e) {
    var t = e.split("").reverse().join(""),
        n = "",
        o = decodeURIComponent(decodeURIComponent(t)).split("#&#");
    for (i = 0; i < o.length; i++) "" != o[i] && (n += String.fromCharCode(o[i]));
    return n
}
'''
number_fun = js2py.eval_js(script)

Rest of the code is what as usual

# Process Data
res = rq.get(
    "https://www.quickerala.com/listings?q=Healthcare&location=Alleppey")
res_data = Soup(res.text, features="html.parser")
rows = res_data.findAll("button", {"data-el": "view-location"})
for row in rows:
    try:
        json_data = json.loads(row["data-map"])
        name = json_data["name"]
        location = json_data["location"]
        phone = number_fun(json_data["phone"][0])
        print(name, location, phone)
    except:
        pass

Output

Nediyathu Speech & Hearing Aid Centre Thrissur Mavelikara, Alleppey 9446477258
Mypharma Laboratories Alleppey, Alleppey 9809780878
Air Rescuers World Wide Pvt Ltd Chengannur, Alleppey 9870001118
Abhaya Ayurveda Hospital Ennakkad, Alleppey 9539297062
SOUPARNIKA AYURVEDA CLINIC Mavelikara, Alleppey 8075803773
Thottikuzhiyil Medicals Alappuzha, Alleppey 9447470538
Aawaaz Speech & Hearing Care Centre Cherthala, Alleppey 9995822386
DANA GYM Alleppey, Alleppey 9567448535
Upasana Yoga and Reiki Clinic Ambalapuzha, Alleppey 9947260352
SIDHA DEEPAM Cherthala, Alleppey 8714233349
SANTHISUKHAM Alleppey, Alleppey 8111928007
SAMANGA AYURVEDA & PHYSIOTHERAPY REHAB. Alleppey, Alleppey 90618 60702
KRV AYURVEDA KENDRAM Alleppey, Alleppey 9447145738
School of Life Skills Alappuzha, Alleppey 9895458500
Kripa Wellness Clinic & Diabetic Research Center Kayamkulam, Alleppey 0479 2446789
Dr. Oommens Eye Hospital & Microsurgery Center Chengannur, Alleppey 0479-2453416
Santhigiri Ayurveda & Siddha Vaidyasala, Chengannur, Alleppey 479-2452582
Santhigiri Ayurveda & Siddha Hospital Alleppey, Alleppey 0478-2879734
Santhigiri Ayurveda & Siddha Hospital Thiruvambady, Alleppey 0477-3200724
Tricare Diagnostics Chengannur, Alleppey 0479-2456664
Dhathri Ayurveda Hospital & Panchakarma Center Kayamkulam, Alleppey 0479-2431403
Pranala Diagnostics Haripad, Alleppey 9495603511
Sankar's Healthcare Diagnostics Pathirappally, Alleppey 9961234488

Answer 2

All the data about each posting is contained in a script tag just above the div of each post. As the op indicated the question was about scrapy here is a solution using scrapy. The data_json object contains a lot more info that you can scrape as well.

import scrapy
import json

class TestSpider(scrapy.Spider):
    name = 'test'

    start_urls = ['https://www.quickerala.com/listings?q=Healthcare&location=Alleppey']

    allowed_domains = ['quickerala.com']

    def parse(self, response):
        for data in response.xpath("//div[@class='listingWrap']/script[@type='application/ld+json']/text()").getall():
            data_json = json.loads(data)
            yield {
                "name": data_json.get('name'),
                "telephone": data_json.get('telephone')
            }

Running the spider, you obtain results such as the below:

2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Nediyathu Speech &amp; Hearing Aid Centre Thrissur', 'telephone': ['9446477258']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Mypharma Laboratories', 'telephone': ['9809780878']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Air Rescuers World Wide Pvt Ltd', 'telephone': ['9870001118']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Abhaya Ayurveda Hospital', 'telephone': ['9539297062', '0479 2466021']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'SOUPARNIKA AYURVEDA CLINIC', 'telephone': ['8075803773', '04792478536']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Thottikuzhiyil Medicals', 'telephone': ['9447470538']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Aawaaz Speech &amp; Hearing Care Centre', 'telephone': ['9995822386', '0478 3242386']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'DANA GYM', 'telephone': ['9567448535']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'DANA GYM', 'telephone': ['9567448535']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Upasana Yoga and Reiki Clinic', 'telephone': ['9947260352']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'SIDHA DEEPAM', 'telephone': ['8714233349']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'SANTHISUKHAM', 'telephone': ['8111928007']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'SAMANGA AYURVEDA &amp; PHYSIOTHERAPY REHAB.', 'telephone': ['90618 60702']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'KRV AYURVEDA KENDRAM', 'telephone': ['9447145738']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'School of Life Skills', 'telephone': ['9895458500']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'School of Life Skills', 'telephone': ['9895458500']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Kripa Wellness Clinic &amp; Diabetic Research Center', 'telephone': ['0479 2446789', '0479 2448118']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Ashta Ayurveda Vaidyalayam', 'telephone': ['9747172442', '8848224376']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Karingattil Medicals', 'telephone': ['8281123295', '9747112263']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Dr. Oommens Eye Hospital &amp; Microsurgery Center', 'telephone': ['0479-2453416']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Santhigiri Ayurveda &amp; Siddha Vaidyasala,', 'telephone': ['479-2452582']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Santhigiri Ayurveda &amp; Siddha Hospital', 'telephone': ['0478-2879734']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Santhigiri Ayurveda &amp; Siddha Hospital', 'telephone': ['0477-3200724']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Tricare Diagnostics', 'telephone': ['0479-2456664']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Dhathri Ayurveda Hospital &amp; Panchakarma Center', 'telephone': ['0479-2431403', '0479-2431535']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Pranala Diagnostics', 'telephone': ['9495603511', '0479-2443131']}
2022-02-03 05:34:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quickerala.com/listings?q=Healthcare&location=Alleppey>
{'name': 'Sankar&#039;s Healthcare Diagnostics', 'telephone': ['9961234488']}