Order difference between 'shinytest' expected and current json files

Question

I have an ordinary shinytest script to test my Shiny app. When I run it with shinytest::testApp, it is successful: both the JSON files and the images are similar between current and expected.

But when the test is ran via testhat:

  expect_pass(
    testApp(
      system.file("app", package = "shinyCircularDichroism"),
      compareImages = TRUE
    )
  )

then the images are the same but the JSON files differ:

Name         Status      
001.json  != Files differ
001.png      No change   
002.json  != Files differ
002.png      No change   
003.json  != Files differ
003.png      No change   
004.json  != Files differ
004.png      No change   
005.json  != Files differ
005.png      No change  

I've inspected the differences between the JSON files. Actually they are the same up to the order (that is, one is [a, b, ... and the other is [b, a, ...]).

Is it a known issue? Why does it occur, only via testthat? Is there a way to compare the JSON files up to the order?

Answer 1

Solved thanks to the Github issue spotted by @bretauv. The problem is caused by the LC_COLLATE variable. Depending on the value of this variable, the results of sort can be different.

library(shinytest)
collation <- Sys.getlocale("LC_COLLATE")
Sys.setenv(LC_COLLATE = "en_US.UTF-8")
Sys.setlocale("LC_COLLATE", "en_US.UTF-8")
shinytest::testApp("../")
Sys.setlocale("LC_COLLATE", collation)

and:

test_that("Shiny test", {
  skip_if_not_installed("shinytest")
  library(shinytest)
  collation <- Sys.getlocale("LC_COLLATE")
  Sys.setenv(LC_COLLATE = "en_US.UTF-8")
  Sys.setlocale("LC_COLLATE", "en_US.UTF-8")
  expect_pass(
    testApp(
      system.file("app", package = "shinyCircularDichroism"),
      quiet = TRUE,
      compareImages = TRUE
    )
  )
  Sys.setlocale("LC_COLLATE", collation)
})