What is the proper way of add babel core-js conditionally

Question

I was checking my site with google page speed insight and it showed "Avoid serving legacy JavaScript to modern browsers"

I have babel-env set up something like this, which seems not working properly.

 ["@babel/preset-env", {
  "useBuiltIns": "usage",
  "corejs": 3,
  "targets": {
    "browsers": ["last 2 version", "ie > 11"]
  }
}]

On my entry.js file, I used to have import "core-js/stable" However, I tried to change like this. I am not sure core-js will actually get added before legacy browser crashes.

if(!isModernBrowser(navigator.userAgent)) {
  import('core-js/stable')
}

How do you guys add core-js only for some browsers that need core-js? I am using a custom webpack and SSR

My full babelrc

{
  "presets": [
    "@babel/preset-react",
    ["@babel/preset-env", {
      "useBuiltIns": "usage",
      "corejs": 3,
      "targets": {
        "browsers": ["last 2 version", "ie > 11"]
      }
    }]
  ],
  "plugins": [
    "react-hot-loader/babel",
    "@babel/plugin-transform-arrow-functions",
    "@babel/plugin-transform-spread",
    "@babel/plugin-proposal-object-rest-spread",
    "@babel/plugin-syntax-dynamic-import",
    [
      "babel-plugin-webpack-alias",
      {
        "config": "${PWD}/config/aliases/aliases.config.js"
      }
    ],
    [
      "@babel/plugin-proposal-decorators",
      {
        "legacy": true
      }
    ],
    "@babel/plugin-transform-runtime",
    "react-imported-component/babel",
    "@babel/plugin-syntax-import-meta",
    "@babel/plugin-proposal-class-properties",
    "@babel/plugin-proposal-json-strings",
    "@babel/plugin-proposal-function-sent",
    "@babel/plugin-proposal-export-namespace-from",
    "@babel/plugin-proposal-numeric-separator",
    "@babel/plugin-proposal-throw-expressions"
  ]
}