JSON data in R limited to first 100 rows

Question

I am working with API data from the US Treasury. I used the GET() function from the httr package and fromJSON() function from jsonlite package. Here is an example API:

library(httr)
library(jsonlite)
url <- https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v2/accounting/od/avg_interest_rates

data1 <- GET(url)
data2 <- fromJSON(rawToChar(data1$content))
data2 <- data.frame((data1$data))

The problem is I only see data for the first 100 observations. Output:

str(data3)
'data.frame':   100 obs. of  11 variables:
 $ record_date            : chr  "2001-01-31" "2001-01-31" "2001-01-31" "2001-01-31" ...
 $ security_type_desc     : chr  "Marketable" "Marketable" "Marketable" "Marketable" ...
 $ security_desc          : chr  "Treasury Notes" "Treasury Bonds" "Treasury Inflation-Indexed Notes" "Treasury Inflation-Indexed Bonds" ...
 $ avg_interest_rate_amt  : chr  "6.096" "8.450" "3.772" "3.866" ...
 $ src_line_nbr           : chr  "2" "3" "4" "5" ...
 $ record_fiscal_year     : chr  "2001" "2001" "2001" "2001" ...
 $ record_fiscal_quarter  : chr  "2" "2" "2" "2" ...
 $ record_calendar_year   : chr  "2001" "2001" "2001" "2001" ...
 $ record_calendar_quarter: chr  "1" "1" "1" "1" ...
 $ record_calendar_month  : chr  "01" "01" "01" "01" ...
 $ record_calendar_day    : chr  "31" "31" "31" "31" ...

I need the whole available data (not just the first 100 observations) and I am not able to figure out where the issue is. The API itself does not restrict or impose limits on data. Can you please help me out?

Answer 1

As hinted in my comment above, this is a server-side restriction. The API docs at https://fiscaldata.treasury.gov/datasets/average-interest-rates-treasury-securities/average-interest-rates-on-u-s-treasury-securities describe pagination and e.g. by requesting

url <- paste0("https://api.fiscaldata.treasury.gov/",
              "services/api/fiscal_service/v2/accounting/od/avg_interest_rates")
res1 <- RcppSimdJson::fload(paste0(url,"?page[number]=1&page[size]=1000"))
res2 <- RcppSimdJson::fload(paste0(url,"?page[number]=2&page[size]=1000"))

you request and get pages one and two of sets sized at a 1000 records.