Using reactiveValues() inside an observe function in Shiny Leaflet app causes infinite recursion

Question

I have a Shiny Leaflet app I'm working on that saves map click events with the use of reactiveValues() (that question was posted by me here). When I use observeEvent for each click event on the map, the reactiveValues work perfectly.

In my full app code, it has become necessary to use observe instead of observeEvent. When I make this change from observeEvent to observe, the reactiveValues seem to cause infinite recursion. Here's some reproducible example code:

library(raster)
library(shiny)
library(leaflet)

#load shapefile
rwa <- getData("GADM", country = "RWA", level = 1)

shinyApp(
  ui = fluidPage(
    leafletOutput("map")
  ), 

  server <- function(input, output, session){

    #create empty vector to hold all click ids
    clickedIds <- reactiveValues(ids = vector())

    #initial map output
    output$map <- renderLeaflet({
      leaflet() %>% 
        addTiles() %>% 
        addPolygons(data = rwa, 
                    fillColor = "white", 
                    fillOpacity = 1, 
                    color = "black", 
                    stroke = T, 
                    weight = 1, 
                    layerId = rwa@data$OBJECTID, 
                    group = "regions")
    }) #END RENDER LEAFLET

    observe({

      #create object for clicked polygon
      click <- input$map_shape_click

      #append all click ids in empty vector
      clickedIds$ids <- c(clickedIds$ids, click$id)

      print(clickedIds$ids)
    }) #END OBSERVE EVENT
  }) #END SHINYAPP

I'm able to stop this by using unique() or isolate() in the following line of code:

  #append all click ids in empty vector
  clickedIds$ids <- c(unique(clickedIds$ids), click$id)

BUT in my full app, this (for whatever reason) slows the entire app and makes some visualizations clunky. I'm looking for some alternatives and ideally an explanation why observe causes infinite recursion whereas observeEvent does not.