Save Google App Script state while parsing an object array and continue where left off later on

Question

I am using this simple google app script to parse through all available Google Sites and dump the html content of individual pages. There are quite many pages so the script will eventually run into 6 minute time limit.

Is it possible to somehow use the PropertiesService to save the current progress (especially in the array loops) and continue where left off later on?

    var sites = SitesApp.getAllSites("somedomain.com");
    var exportFolder = DriveApp.getFolderById("a4342asd1242424folderid-");
            
            // Cycle through all sites
        for (var i in sites){
              var SiteName = sites[i].getName();
              var pages = sites[i].getAllDescendants();
              // Create folder in Drive for each site name
              var siteFolder = exportFolder.createFolder(SiteName)
        
        for (var p in pages){
                // Get page name and url
                var PageUrl = pages[p].getUrl();
             
               
                //Dump the raw html content in the text file
                var htmlDump = pages[p].getHtmlContent();
                siteFolder.createFile(PageUrl+".html", htmlDump)
                
              }
        }

I can image how one can use the Properties Service to store current line number in the Spreadsheet, and continute where left off. But how can this be done with array containing objects like Sites or Pages?

Answer 1

Using Objects with Properties Service

According to the quotas the maximum size of something you can store in the properties service is 9kb. With a total of 500kb. So if your object is less than this size, it should be no problem. That said, you will need to convert the object to a string with JSON.stringify() and when you retrieve it, use JSON.parse.

Working around the run time limit

What is commonly done to work around the limit is to structure a process around the properties service and triggers. Essentially you make the script keep track of time, and if it starts to take a long time, you get it to save its position and then create a trigger so that the script runs again in 10 seconds (or however long you want), for example:

function mainJob(x) {
  
  let timeStart = new Date()
  console.log("Starting at ", timeStart)
  
  for (let i = x; i < 500000000; i++){ // NOTE THE i = x
    
    // MAIN JOB INSTRUCTIONS
    let j = i
    // ...
  
    // Check Time
    let timeCheck = new Date()
    if (timeCheck.getTime() - timeStart.getTime() > 30000) {
      console.log("Time limit reached, i = ", i)
      
      // Store iteration number
      PropertiesService
          .getScriptProperties()
          .setProperty('PROGRESS', i)
      
      console.log("stored value of i")
      
      // Create trigger to run in 10 seconds.
      ScriptApp.newTrigger("jobContinue")
          .timeBased()
          .after(10000)
          .create()
      
      console.log("Trigger created for 10 seconds from now")
      return 0
    }
  }
  
  // Reset progress counter
  PropertiesService
          .getScriptProperties()
          .setProperty('PROGRESS', 0)
  
  console.log("job complete")
}

function jobContinue() {
  
  console.log("Restarting job")
  
  previousTrigger = ScriptApp.getProjectTriggers()[0]
  ScriptApp.deleteTrigger(previousTrigger)
  console.log("Previous trigger deleted")
  
  triggersRemain = ScriptApp.getProjectTriggers()
  console.log("project triggers", triggersRemain)
  
  let progress = PropertiesService
                   .getScriptProperties()
                   .getProperty('PROGRESS')
  
  console.log("about to start main job again at i = ", progress)
  
  mainJob(progress)
  
}

function startJob() {
  mainJob(0)
}

Explanation

• This script only has a for loop with 500 million iterations in which it assigns i to j, it is just an example of a long job that potentially goes over the run time limit.
• The script is started by calling function startJob which calls mainJob(0).
• Within mainJob
  • It starts by creating a Date object to get the start time of the mainJob.
  • It takes the argument 0 and uses it to initialize the for loop to 0 as you would normally initialise a for loop.
  • At the end of every iteration, it creates a new Date object to compare with the one created at the beginning of mainJob. In the example, it is set to see if the script has been running for 30 seconds, this can obviously be extended but keep it well below the limit.
  • If it has taken more than 30 seconds, it stores the value of i in the properties service and then creates a trigger to run jobContinue in 10 seconds.
• After 10 seconds, the function jobContinue calls the properties service for the value for i, and calls mainJob with the value returned from the properties service.
• jobContinue also deletes the trigger it just created to keep things clean.
• This script should run as-is in a new project, try it out! When I run it, it takes around 80 seconds, so it runs the first time, creates a trigger, runs again, creates a trigger, runs again and then finally finishes the for loop.

References

• quotas
• JSON.stringify()
• JSON.parse.
• ScriptApp
• Triggers

Answer 2

If you are able to process all pages of 1 site under 6 minutes then you could try saving the site names first in a sheet or props depending on the number again. And keep processing n-sites per run. Can also try SitesApp.getAllSites(domain, start, max) and save start value in props after incrementing.

Can do something similar for pages if you cannot process them under 6 minutes. SitesApp.getAllDescendants(options)