How to scrape website which loads json content dynamically?

Question

I am working on a scraping project where I am trying to scrape NSE announcements page - https://www.nseindia.com/companies-listing/corporate-filings-announcements.

Now the table is static but the JSON content get loads dynamically.

I am able to hit the API endpoint of the website but after few times it says - “Resource not found”.

I am not able to find workaround for it. How can I extract the news from announcements page?

I tried using the headers in the code but even that is not helping.

Answer 1

To load the Json with the data first get the cookies from the main page:

import pandas as pd
import requests

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"
}

url = "https://www.nseindia.com/api/corporate-announcements?index=equities"

with requests.session() as s:
    s.headers.update(headers)

    s.get("https://www.nseindia.com/companies-listing/corporate-filings-announcements")
    data = s.get(url).json()
    df = pd.DataFrame(data)
    print(df)

Prints:

        symbol                                                                                              desc              dt                                                                                                                     attchmntFile                                  sm_name       sm_isin                 an_dt            sort_date     seq_id                smIndustry orgid                                                                                                                                                                                       attchmntText bflag old_new csvName          exchdisstime difference
0       SASKEN                                                                              Change in Management  31032024234101             https://nsearchives.nseindia.com/corporate/SASKEN_31032024234101_Ltr_to_SE_Retirement_B_Ramkumar_SMP_31_Mar_2024.pdf              Sasken Technologies Limited  INE231F01020  31-Mar-2024 23:41:01  2024-03-31 23:41:01  105801547      Computers - Software  None                                                                                                                   Sasken Technologies Limited has informed the Exchange about change in Management  None    None    None  31-Mar-2024 23:41:06   00:00:05
1   CEREBRAINT                                                                               Trading Window-XBRL  31032024234041  https://nsearchives.nseindia.com/corporate/xbrl/CEREBRAINT_31032024234042_CLOSURE_TRADING_WINDOW_1078060_31032024114041_WEB.xml  Cerebra Integrated Technologies Limited  INE345B01019  31-Mar-2024 23:40:41  2024-03-31 23:40:41  105801546                      None  None                                                                                                  CEREBRA INTEGRATED TECHNOLOGIES LIMITED has informed the Exchange about Closure of Trading Window  None    None    None  31-Mar-2024 23:40:46   00:00:05
2     UNIVASTU                                                                                           Updates  31032024233728                                           https://nsearchives.nseindia.com/corporate/UNIVASTU_31032024233728_Disposalofshare.pdf                   Univastu India Limited  INE562X01013  31-Mar-2024 23:37:28  2024-03-31 23:37:28  105801545                      None  None                                                                                                                   Univastu India Limited has informed the Exchange regarding 'Disposal of shares'.  None    None    None  31-Mar-2024 23:37:32   00:00:04
3   CEREBRAINT                                                                                    Trading Window  31032024233603                                            https://nsearchives.nseindia.com/corporate/CEREBRAINT_31032024233603_SEINTIMATION.pdf  Cerebra Integrated Technologies Limited  INE345B01019  31-Mar-2024 23:36:03  2024-03-31 23:36:03  105801544                      None  None                         Cerebra Integrated Technologies Limited has informed the Exchange regarding the Trading Window closure pursuant to SEBI (Prohibition of Insider Trading) Regulations, 2015  None    None    None  31-Mar-2024 23:36:07   00:00:04
4      CAREERP  Change in Directors/ Key Managerial Personnel/ Auditor/ Compliance Officer/ Share Transfer Agent  31032024232247                  https://nsearchives.nseindia.com/corporate/xbrl/CAREERP_31032024232247_CIM_21314_1078057_31032024112246_WEB.xml                     Career Point Limited  INE521J01018  31-Mar-2024 23:22:47  2024-03-31 23:22:47  105801542      Computers - Software  None                                              CAREER POINT LIMITED has informed the Exchange about Change in Directors/ Key Managerial Personnel/ Auditor/ Compliance Officer/ Share Transfer Agent  None    None    None  31-Mar-2024 23:22:53   00:00:06
5      WELCORP                                                                               Trading Window-XBRL  31032024232101     https://nsearchives.nseindia.com/corporate/xbrl/WELCORP_31032024232101_CLOSURE_TRADING_WINDOW_1078055_31032024112101_WEB.xml                     Welspun Corp Limited  INE191B01025  31-Mar-2024 23:21:01  2024-03-31 23:21:01  105801541  Steel And Steel Products  None                                                                                                                     Welspun Corp Limited has informed the Exchange about Closure of Trading Window  None    None    None  31-Mar-2024 23:21:10   00:00:09
6     PRESTIGE                                                                                       Acquisition  31032024232058                                                https://nsearchives.nseindia.com/corporate/PRESTIGE_31032024232058_Intimation.pdf        Prestige Estates Projects Limited  INE811K01011  31-Mar-2024 23:20:58  2024-03-31 23:20:58  105801540              Construction  None                                                              Prestige Estates Projects Limited has informed the Exchange about Acquisition of 50% partnership interest in Prestige Realty Ventures  None    None    None  31-Mar-2024 23:21:02   00:00:04
7      APTECHT                                                                                           Updates  31032024231519                                   https://nsearchives.nseindia.com/corporate/APTECHT_31032024231519_ReconsitutionofCommitees.pdf                           Aptech Limited  INE266F01018  31-Mar-2024 23:15:19  2024-03-31 23:15:19  105801537      Computers - Software  None                                                                                    Reconstitution of Audit Committee, Nomination and Remuneration Committee andStakeholders Relationship Committee  None    None    None  31-Mar-2024 23:15:25   00:00:06
8     RAMCOCEM  Change in Directors/ Key Managerial Personnel/ Auditor/ Compliance Officer/ Share Transfer Agent  31032024231041                 https://nsearchives.nseindia.com/corporate/xbrl/RAMCOCEM_31032024231041_CIM_21312_1078048_31032024111040_WEB.xml                The Ramco Cements Limited  INE331A01037  31-Mar-2024 23:10:41  2024-03-31 23:10:41  105801535                      None  None                                         THE RAMCO CEMENTS LIMITED has informed the Exchange about Change in Directors/ Key Managerial Personnel/ Auditor/ Compliance Officer/ Share Transfer Agent  None    None    None  31-Mar-2024 23:10:51   00:00:10
9   INDIAGLYCO                                                                                           Updates  31032024231038                   https://nsearchives.nseindia.com/corporate/INDIAGLYCO_31032024231038_Reg30GraincapacityandNSUAugustMarch24.pdf                    India Glycols Limited  INE560A01015  31-Mar-2024 23:10:38  2024-03-31 23:10:38  105801534       Chemicals - Organic  None                       India Glycols Limited has informed the Exchange regarding 'Update on enhancement of capacity of Grain Based Distillery and facilities/project for New Specialty Chemicals.'.  None    None    None  31-Mar-2024 23:10:44   00:00:06
10     CAREERP                                                                             Change in Director(s)  31032024230809                                  https://nsearchives.nseindia.com/corporate/CAREERP_31032024230809_CPL_Change_In_Directorate.pdf                     Career Point Limited  INE521J01018  31-Mar-2024 23:08:09  2024-03-31 23:08:09  105801533      Computers - Software  None                                                                                                     Career Point Limited has informed the Exchange regarding Change in Director(s) of the company.  None    None    None  31-Mar-2024 23:08:16   00:00:07
11    JUSTDIAL  Change in Directors/ Key Managerial Personnel/ Auditor/ Compliance Officer/ Share Transfer Agent  31032024230216                 https://nsearchives.nseindia.com/corporate/xbrl/JUSTDIAL_31032024230216_CIM_21311_1078045_31032024110215_WEB.xml                        Just Dial Limited  INE599M01018  31-Mar-2024 23:02:16  2024-03-31 23:02:16  105801532                      None  None                                                 JUST DIAL LIMITED has informed the Exchange about Change in Directors/ Key Managerial Personnel/ Auditor/ Compliance Officer/ Share Transfer Agent  None    None    None  31-Mar-2024 23:02:22   00:00:06
12    JUSTDIAL                                                                                       Appointment  31032024225920                                              https://nsearchives.nseindia.com/corporate/JUSTDIAL_31032024225920_regulation30.pdf                        Just Dial Limited  INE599M01018  31-Mar-2024 22:59:20  2024-03-31 22:59:20  105801531                      None  None         Completion of term of  Ms. Bhavna Thakur, Independent Director of the Company and appointment of  Ms. Bhama Krishnamurthy as an additional director designated as an Independent Director.  None    None    None  31-Mar-2024 22:59:27   00:00:07
13         ADL                                                                               Trading Window-XBRL  31032024225502         https://nsearchives.nseindia.com/corporate/xbrl/ADL_31032024225502_CLOSURE_TRADING_WINDOW_1078043_31032024105501_WEB.xml                  Archidply Decor Limited  INE0CHO01012  31-Mar-2024 22:55:02  2024-03-31 22:55:02  105801530                      None  None                                                                                                                  ARCHIDPLY DECOR LIMITED has informed the Exchange about Closure of Trading Window  None    None    None  31-Mar-2024 22:55:07   00:00:05
14    TITAGARH                                                                                         Cessation  31032024225203                           https://nsearchives.nseindia.com/corporate/TITAGARH_31032024225204_IDCESSATIONSEDISCLOSURE31032024.pdf            TITAGARH RAIL SYSTEMS LIMITED  INE615H01020  31-Mar-2024 22:52:03  2024-03-31 22:52:03  105801529                      None  None  TITAGARH RAIL SYSTEMS LIMITED has informed the Exchange regarding Cessation of Mr Sunirmal Talukdar (DIN: 00920608)  as Non- Executive Independent Director of the company w.e.f. March 31, 2024.  None    None    None  31-Mar-2024 22:52:17   00:00:14
15    TITAGARH                                                                                         Cessation  31032024225203                           https://nsearchives.nseindia.com/corporate/TITAGARH_31032024225203_IDCESSATIONSEDISCLOSURE31032024.pdf            TITAGARH RAIL SYSTEMS LIMITED  INE615H01020  31-Mar-2024 22:52:03  2024-03-31 22:52:03  105801528                      None  None                      TITAGARH RAIL SYSTEMS LIMITED has informed the Exchange regarding Cessation of Mr Manoj Mohanka  as Non- Executive Independent Director of the company w.e.f. March 31, 2024.  None    None    None  31-Mar-2024 22:52:10   00:00:07
16         ADL                                                                                    Trading Window  31032024224744                                            https://nsearchives.nseindia.com/corporate/ADL_31032024224744_tradingwindowsigned.pdf                  Archidply Decor Limited  INE0CHO01012  31-Mar-2024 22:47:44  2024-03-31 22:47:44  105801527                      None  None                                         Archidply Decor Limited has informed the Exchange regarding the Trading Window closure pursuant to SEBI (Prohibition of Insider Trading) Regulations, 2015  None    None    None  31-Mar-2024 22:47:50   00:00:06
17      MASTEK                                                                                           Updates  31032024224617                          https://nsearchives.nseindia.com/corporate/MASTEK_31032024224617_DubaiEntitiesPenalty31032024signed.pdf                           Mastek Limited  INE759A01021  31-Mar-2024 22:46:17  2024-03-31 22:46:17  105801526      Computers - Software  None                                                                    Mastek Limited has informed the Exchange regarding ' Intimation of payment  of penalty by UAE based group Subsidiary entities'.  None    None    None  31-Mar-2024 22:46:24   00:00:07
18    RAMCOCEM                                                                             Change in Director(s)  31032024224455                                            https://nsearchives.nseindia.com/corporate/RAMCOCEM_31032024224455_DIRSCESSSIGNED.pdf                The Ramco Cements Limited  INE331A01037  31-Mar-2024 22:44:55  2024-03-31 22:44:55  105801525                      None  None                                                                                                The Ramco Cements Limited has informed the Exchange regarding Change in Director(s) of the company.  None    None    None  31-Mar-2024 22:45:01   00:00:06
19  ASIANPAINT  Change in Directors/ Key Managerial Personnel/ Auditor/ Compliance Officer/ Share Transfer Agent  31032024223609               https://nsearchives.nseindia.com/corporate/xbrl/ASIANPAINT_31032024223609_CIM_21310_1078039_31032024103608_WEB.xml                     Asian Paints Limited  INE021A01018  31-Mar-2024 22:36:09  2024-03-31 22:36:09  105801523                    Paints  None                                              ASIAN PAINTS LIMITED has informed the Exchange about Change in Directors/ Key Managerial Personnel/ Auditor/ Compliance Officer/ Share Transfer Agent  None    None    None  31-Mar-2024 22:36:15   00:00:06