VGG16 pre-trained model is not giving accurate Image Similarity in scala

Question

I'm trying to calculate similarity between two Images to determine that whether they belong to same person or not, for that I'm using deeplearning4j, keras and VGG16 pre-trained model (vgg16_dl4j_inference.zip this model downloads automatically just after running the program, Its size is about 513.7MB).

I'm trying to implement it in scala (version 2.13.4) with sbt (version 1.9.6), here is my code.

build.sbt:

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.4"

lazy val root = (project in file("."))
  .settings(
    name := "AddExternalLib"
  )

libraryDependencies ++= Seq(
  "org.deeplearning4j" % "deeplearning4j-core" % "1.0.0-beta7"
  , "org.deeplearning4j" % "deeplearning4j-modelimport" % "1.0.0-beta7"
  , "org.deeplearning4j" % "deeplearning4j-nn" % "1.0.0-beta7"
  , "org.deeplearning4j" % "deeplearning4j-zoo" % "1.0.0-beta7"
  , "org.nd4j" % "nd4j-native-platform" % "1.0.0-beta7"
  , "ch.qos.logback" % "logback-classic" % "1.4.7"
)

here is my main class.

Deeplearning.scala

import org.deeplearning4j.nn.graph.ComputationGraph
import org.deeplearning4j.zoo.model.VGG16
import org.nd4j.linalg.api.ndarray.INDArray
import org.datavec.image.loader.NativeImageLoader
import org.nd4j.linalg.factory.Nd4j

object Deeplearning extends App {
  val vgg16 = VGG16.builder().build()
  val model: ComputationGraph = vgg16.initPretrained().asInstanceOf[ComputationGraph]

  val loader = new NativeImageLoader(224, 224, 3)
  val image1 = loader.asMatrix("image1.jpg")
  val image2 = loader.asMatrix("image2.jpeg")

  val input1 = Nd4j.expandDims(image1, 0).reshape(Array(1, 3, 224, 224))
  val input2 = Nd4j.expandDims(image2, 0).reshape(Array(1, 3, 224, 224))

  val embeddings1: INDArray = model.outputSingle(input1)
  val embeddings2: INDArray = model.outputSingle(input2)
  val cosineSimilarity: Double = cosineSimilarity(embeddings1, embeddings2)

  println(s"Cosine Similarity: $cosineSimilarity")

  def cosineSimilarity(vec1: INDArray, vec2: INDArray): Double = {
    val dotProduct: Double = vec1.mul(vec2).sumNumber().doubleValue()
    val magnitude1: Double = Math.sqrt(vec1.mul(vec1).sumNumber().doubleValue())
    val magnitude2: Double = Math.sqrt(vec2.mul(vec2).sumNumber().doubleValue())

    dotProduct / (magnitude1 * magnitude2)
  }
}

The problem is that it calculates similarity of identical images fine, but in case if I give my current and 2,3 years old image it considers them as Images of different person although they are same.

Output for identical Images:

Cosine Similarity: 1.0

Output for Current and 2,3 years old Images:

Cosine Similarity: 0.08223503419296793

although images quality is fine , even if I crop faces from images and calculate similarity, still it is not working accurately.

Guide me how can I increase accuracy?