Nestled Loop not Working to gather data from NOAA

Question

I'm using the R package rnoaa(along with it required other packages) to gather historical weather data. I wrote this nestled loop to gather all the data sets but I keep getting errors when I run it. It seems to run for a second fine

The loop:

require('triebeard')
require('bindr')
require('colorspace')
require('mime')
require('curl')
require('openssl')
require('R6')
require('urltools')
require('httpcode')
require('stringr')
require('assertthat')
require('bindrcpp')
require('glue')
require('magrittr')
require('pkgconfig')
require('rlang')
require('Rcpp')
require('BH')
require('plogr')
require('purrr')
require('stringi')
require('tidyselect')
require('digest')
require('gtable')
require('plyr')
require('reshape2')
require('lazyeval')
require('RColorBrewer')
require('dichromat')
require('munsell')
require('labeling')
require('viridisLite')
require('data.table')
require('rjson')
require('httr')
require('crul')
require('lubridate')
require('dplyr')
require('tidyr')
require('ggplot2')
require('scales')
require('XML')
require('xml2')
require('jsonlite')
require('rappdirs')
require('gridExtra')
require('tibble')
require('isdparser')
require('geonames')
require('hoardr')
require('rnoaa')
install.package('ncdf4')

install.packages("devtools")
library(devtools)
install_github("rnoaa", "ropensci")
library(rnoaa)


list <- buoys(dataset='wlevel')
lid <- data.frame(list$id)
foo <- for(range in 1990:2017){
for(bid in lid){
bid_range <- buoy(dataset = 'wlevel', buoyid = bid, year = range)
bid.year.data <- data.frame(bid.year$data)
write.csv(bid.year.data, file='cwind/bid_range.csv')
} 
}

The response:

Using c1990.nc
Using 
Error: length(url) == 1 is not TRUE

It saves the first data-set but it does not apply the for in the file name it just names it bid_range.csv.

Answer 1

This error message shows that there are no any data of a given station id in 1990. Because you were using for loop, once it gots an error, it stops.

Here I introduce the use of tidyverse to download the NOAA buoy data. A lot of the following functions are from the purrr package, which is part of the tidyverse.

# Load packages
library(tidyverse)
library(rnoaa)

Step 1: Create a "Grid" containing all combination of id and year

The expand function from tidyr can create the combination of different values.

data_list <- buoys(dataset = 'wlevel')

data_list2 <- data_list %>%
  select(id) %>%
  expand(id, year = 1990:2017)

Step 2: Create a "safe" version that does not break when there is no data. Also make this function suitable for the map2 function

Because we will use map2 to loop through all the combination of id and year using the map2 function by its .x and .y argument. We modified the sequence of argument to create buoy_modify. We also use the safely function to create a safe version of buoy_modify. Now when it meets error, it will store the error message and moves to the next one rather than breaks.

# Modify the buoy function
buoy_modify <- function(buoyid, year, dataset, ...){
  buoy(dataset, buoyid = buoyid, year = year, ...)
}

# Creare a safe version of buoy_modify
buoy_safe <- safely(buoy_modify)

Step 3: Apply the buoy_safe function

wlevel_data <- map2(data_list2$id, data_list2$year, buoy_safe, dataset = "wlevel")

# Assign name for the element in the list based on id and year
names(wlevel_data) <- paste(data_list2$id, data_list2$year, sep = "_")

After this step, all the data were downloaded in wlevel_data. Each element in wlevel_data has two parts. $result shows the data if the download is successful, otherwise, it shows NULL. $error shows NULL if the download is successful, otherwise, it shows the error message.

Step 4: Access the data

transpose can turn a list "inside out". So now wlevel_data2 has two elements: result and error. We can store these two and access the data.

# Turn the list "inside out"
wlevel_data2 <- transpose(wlevel_data)

# Get the error message
wlevel_error <- wlevel_data2$error

# Get he result
wlevel_result <-  wlevel_data2$result

# Remove NULL element in wlevel_result
wlevel_result2 <- wlevel_result[!map_lgl(wlevel_result, is.null)]