I'm scraping an online gaming calendar to be copied into my Google Calendar. Ultimately, I'm using a bot on Discord to read my Google Calendar and create Discord events that mirror the online gaming calendar.
I can successfully create a single event, but when I want to iterate the array and create an event in Google Calendar for each event scraped and in my array, I keep running into a reference error where "steps" is "not defined."
Since the calendar is dynamic and there can be any number of events in a given month, I don't want to just create 100 different "create_event" steps in the workflow. It's sloppy and I think it would time out the workflow.
The calendar I am scraping: https://www.tibia.com/news/?subtopic=eventcalendar
The sharable workflow: https://pipedream.com/new?h=tch_3M9fQo
In case the sharable workflow doesn't work, I'll explain it basically and paste the code here.
- There is a trigger that runs on a time interval.
- There is a scraper that pulls the data and formats it into an array.
- There is a google integration that creates an event based on which variable I select from the array.
The code for the scraper:
const axios = require('axios');
const cheerio = require('cheerio');
module.exports = {
name: 'tibia_calendar_scraper',
version: '0.0.1',
props: {
// Define any props you need
},
async run() {
try {
// Send a GET request to the Tibia Events Calendar page
const response = await axios.get('https://www.tibia.com/news/?subtopic=eventcalendar');
const html = response.data;
// Load the HTML into Cheerio
const $ = cheerio.load(html);
// Extract event information using the provided element code
const events = [];
// Specify the selector for the event you provided
const eventSelector = 'td[style="height:82px; background-clip: padding-box; overflow: hidden; vertical-align:top; background-color:#E7D1AF;"]';
// Find all elements matching the selector
const eventElements = $(eventSelector);
// Get the current date
const currentDate = new Date();
const currentYear = currentDate.getFullYear();
const currentMonth = currentDate.getMonth() + 1; // Months are zero-based, so add 1
// Loop through the event elements
eventElements.each((index, element) => {
const eventElement = $(element);
// Extract the day from the first span in the first div
const day = eventElement.find('div span:first-child').text().trim();
// Find all child div elements within the HelperDivIndicator span
const eventDivs = eventElement.find('.HelperDivIndicator > div');
// Loop through the child divs to extract event names
eventDivs.each((divIndex, divElement) => {
const eventName = $(divElement).text().trim().replace(/\*/g, ''); // Remove asterisks
// Create an event object with the date including month and year
const eventDate = `${currentYear}-${currentMonth}-${day}`;
// Add the event date to the events array
events.push({
date: eventDate,
name: eventName,
});
});
});
// Store the scraped events in a variable
const scrapedEvents = events;
// Return the scraped events as the result
return scrapedEvents;
} catch (error) {
console.error('Error scraping Tibia Events Calendar:', error);
throw error;
}
},
};
The code for the "iterater":
// Iterate through each event in the exported array
for (const event of steps.tibia_calendar_scraper) {
// Create Google Calendar event for each event
create_event({
summary: event.name,
description: event.description, // Update with your event data
start: event.startTime,
end: event.endTime,
timeZone: 'America/New_York', // Set the appropriate time zone
// ...other event fields
}).then(() => {
console.log(`Google Calendar event created for ${event.name}`);
}).catch((error) => {
console.error(`Error creating Google Calendar event for ${event.name}:`, error);
});
}
Once I get this working, I would like to also check if the event already exists on the Google Calendar so that I can trigger the workflow daily in case the online gaming calendar is updated with anything (which they usually do every month for surprise events) without duplicating all of the other events.
I've been at it for about 8 hours and kind of stuck with which angle to try next. Appreciate any guidance or suggestions. I am using ChatGPT for all of the coding. I'm not proficient. Thanks for reading, hopefully someone can point me in the right direction or offer a solution.