I'm trying interesting data inside the Web Data Commons dumps. It is taking day to grep across it on my machine (in parallel). Is there an index out there of what websites are covered and an ability to extract specifically from those sites?
Means of getting data for a given website from the Web Data Commons?
510 views Asked by user1556658 At
1
To get all of the pages from a particular domain -- one option is to query the common crawl api site:
http://index.commoncrawl.org
To list all of the pages from the specific domain wikipedia.org:
This shows you how many pages of blocks common crawl has from this domain (note you can use wildcards as in this example).
Then go into each page and ask common crawl to send you a json object of each file:
You can then parse the json and get each warc file through the field:
filename
This link will help you.