API access, Hashing query, possibly related to character encoding

Question

Its going to be impossible to provide a reproducible example for this as it has to do with accessing a particular API - apologies in advance. I am trying to access the edinburgh fringe festivals API through R. The documentation says the following:

Authentication

An access key is required for all requests to the API and can be obtained by registration. This key, combined with the signature explained below, is required to get any access to the API. If you do not provide an access key, or provide an invalid key or signature, the server will return an HTTP 403 error.

Your secret token is used to calculate the correct signature for each API request and must never be disclosed. Once issued, it is never transmitted back or forward. The API server holds a copy of your secret token, which it uses to check that you have signed each request correctly.

You calculate the signature using the HMAC-SHA1 algorithm:

• Build the full API request URL, including your access key but excluding the server domain - eg /events?festival=book&key=12345. See the note below on URL encoding.

• Calculate the hmac hash of the url using the sha1 algorithm and your secret token as the key.

• Append the hex-encoded hash to your url as the signature parameter

URL encoding in queries

You should calculate the signature after URL-encoding any parameters - for example, to search for the title "Mrs Brown" you would first build the URL /events?title=Mrs%20Brown&key=12345 and then sign this string and append the signature.

Signature encoding

Some languages - notably C# - default to encoding hashes in UTF-16. Ensure your signature is encoded in plain ASCII hex or it will not be valid.

What I have tried so far is below:

    library(digest)
    library(jsonlite)
    source("authentication.R") # credentials stored here

create the query string

    query <- paste0('/events?festival=demofringe&size=20&from=1&key=', API_KEY)

create the hashed query

    sig <- hmac(SECRET_SIGNING_KEY, query, algo="sha1")

create the final url

    url <- paste0('https://api.edinburghfestivalcity.com', query, '&signature=', sig)

submit to the API

    results <- fromJSON(url)

and I get the error:

Error in open.connection(con, "rb") : HTTP error 417.

I'm not sure that the signature is ASCII encoded as per the documentation. Does anyone know how to debug this situation? I have tried iconv() to try and convert the encoding and when I call Encoding() on the character object it returns "unknown". I have also tried saving both files in RStudio with "save with encoding" set to ASCII and I have tried sourcing the authentications with encoding = "ASCII".

Incidentally when I paste the final url into a browser, I get the following error:

Invalid Accept header value. ['application/json', 'application/json;ver=2.0'] are supported

Answer 1

That's a server error. It should understand that

Accept: application/json, text/*, */*

Matches application/json. Note that instead of modifying jsonlite, it is beter to manually retrieve the response form the server, and then feed it to jsonlite.

library(httr)
library(jsonlite)
req <- GET(your_url, accept("application/json")
json <- content(req, as = 'text')
data <- fromJSON(json)

Answer 2

In the end, I solved this problem by editing the jsonlite::fromJSON() function. namely the line

curl::handle_setheaders(h, Accept = "application/json, text/*, */*")

I changed to

curl::handle_setheaders(h, Accept = "application/json")

I then also had to point to the internal functions with jsonlite:::. Now it works fine.