I want to scrape a website daily by using AirFlow and Selenium Grid server. I have created a single DAG for this.
The website provide an endpoint (e.g. website.com/python, website.com/java, website.com/ruby) for each data in a same web user interface.
I've succesfully triggered The DAG by using AirFlow REST API with a conf variable. Now I want to schedule every run, which I have triggered, daily with the coressponding conf variable.
Is it possible to achieve this (maybe by creating a new DAG or dynamically set a cronjob) ?
Schedule a DAG at different time by using Airflow REST API with conf variables
126 views Asked by Jooo21 At
1
You can schedule an Airflow DAG by setting
schedule
on theDAG
, for example:From your question, it's unclear to me if your conf changes per DAG run. If not, you could hardcode all the settings, or set them through
params
on the DAG, see example here.