I began writing a program using the Druid crate and Crabler crate to make a webscraping application whose data I can explore. I only realized that merging synchronous and asynchronous programming was a bad idea long after I had spent a while building this program. What I am trying to do right now is have the scraper run while the application is open (preferably every hour).
Right now the scraper doesn't run until after the application is closed. I tried to use Tokio's spawn
to make a separate thread that starts before the application opens, but this doesn't work because the Crabler future doesn't have the "Send" trait.
I tried to make a minimal functional program as shown below. The title_handler
doesn't function as expected but otherwise it demonstrates the issue I'm having well.
Is it possible to allow the WebScraper to run while the application is open? If so, how?
EDIT: I tried using task::spawn_blocking()
to run the application and it threw out a ton of errors, including that druid doesn't implement the trait Send
.
use crabler::*;
use druid::widget::prelude::*;
use druid::widget::{Align, Flex, Label, TextBox};
use druid::{AppLauncher, Data, Lens, WindowDesc, WidgetExt};
const ENTRY_PREFIX: [&str; 1] = ["https://duckduckgo.com/?t=ffab&q=rust&ia=web"];
// Use WebScraper trait to get each item with the ".result__title" class
#[derive(WebScraper)]
#[on_response(response_handler)]
#[on_html(".result__title", title_handler)]
struct Scraper {}
impl Scraper {
// Print webpage status
async fn response_handler(&self, response: Response) -> Result<()> {
println!("Status {}", response.status);
Ok(())
}
async fn title_handler(&self, _: Response, el: Element) -> Result<()> {
// Get text of element
let title_data = el.children();
let title_text = title_data.first().unwrap().text().unwrap();
println!("Result is {}", title_text);
Ok(())
}
}
// Run scraper to get info from https://duckduckgo.com/?t=ffab&q=rust&ia=web
async fn one_scrape() -> Result<()> {
let scraper = Scraper {};
scraper.run(Opts::new().with_urls(ENTRY_PREFIX.to_vec()).with_threads(1)).await
}
#[derive(Clone, Data, Lens)]
struct Init {
tag: String,
}
fn build_ui() -> impl Widget<Init> {
// Search box
let l_search = Label::new("Search: ");
let tb_search = TextBox::new()
.with_placeholder("Enter tag to search")
.lens(Init::tag);
let search = Flex::row()
.with_child(l_search)
.with_child(tb_search);
// Describe layout of UI
let layout = Flex::column()
.with_child(search);
Align::centered(layout)
}
#[async_std::main]
async fn main() -> Result<()> {
// Describe the main window
let main_window = WindowDesc::new(build_ui())
.title("Title Tracker")
.window_size((400.0, 400.0));
// Create starting app state
let init_state = Init {
tag: String::from("#"),
};
// Start application
AppLauncher::with_window(main_window)
.launch(init_state)
.expect("Failed to launch application");
one_scrape().await
}