Stream data from API

Question

I am trying to pull data on mails coming into an API from an email testing tool mailhog. If I use a call to get a list of emails e.g GET /api/v1/messages I can load this data into a struct with no issues and print out values I need.

However if I use a different endpoint that is essentially a stream of new emails coming in, I have different behavior. When I run my go application I get no output whatsoever.

Do I need to do like a while loop to constantly listen to the endpoint to get the output? My end goal is to pull some information from emails as they come in and then pass them into a different function.

Here is me trying to access the streaming endpoint https://github.com/mailhog/MailHog/blob/master/docs/APIv1.md

res, err := http.Get("http://localhost:8084/api/v1/events")

if err != nil {
    panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)

if err != nil {
    panic(err.Error())
}

var data Email
json.Unmarshal(body, &data)
fmt.Printf("Email: %v\n", data)

If I do a curl request at the mailhog service with the same endpoint, I do get output as mails come in. However I cant seem to figure out why I am getting no output via my go app. The app does stay running just I dont get any output.

I am new to Go so apologies if this is a really simple question

Answer 1

From ioutil.ReadAll documentation:

ReadAll reads from r until an error or EOF and returns the data it read.

When you use to read the body of a regular endpoint, it works because the payload has an EOF: the server uses the header Content-Length to tell how many bytes the body response has, and once the client read that many bytes, it understands that it has read all of the body and can stop.

Your "streaming" endpoint doesn't use Content-Length though, because the body has an unknown size, it's supposed to write events as they come, so you can't use ReadAll in this case. Usually, in this case, you are supposed to read line-by-line, where each line represents an event. bufio.Scanner does exactly that:

res, err := http.Get("http://localhost:8084/api/v1/events")
if err != nil {
    panic(err.Error())
}
scanner := bufio.NewScanner(res.Body)
for e.scanner.Scan() {
    if err := e.scanner.Err(); err != nil {
        panic(err.Error())
    }
    event := e.scanner.Bytes()
    var data Email
    json.Unmarshal(event, &data)
    fmt.Printf("Email: %v\n", data)
}

curl can process the response as you expect because it checks that the endpoint will stream data, so it reacts accordinly. It may be helpful to add the response curl gets to the question.