How i can integrate scrapy in django and get spider results in django views?

Question

i want to use scrapy spiders in django views, crawl that spider within django view and store scraped data in python list or dictionary. Is there any easy way to do this?

Answer 1

You are better off saving a scrapy file in your project where settings.py exists, and create a scrapy project with scrapy startproject some_project.

Then in your items.py import your model classes and djangoitem to create something like

from djangoitem import DjangoItem
from some_app.models import Practice_Model
class scraper(DjangoItem):
    django_model = Practice_Model

Then in your settings include:

import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'some_project.settings'
import django
django.setup()

where some_project is the name of your project when you did django-admin startproject some_project.

This should be all you need to connect your scraper to django.