Visual Studio Code: Starting in non debug mode stops in breakpoints and complains

Question

I am using Visual Studio Code with nestjs and nodemon and I have the following configuration:

package.json

"start:dev": "nodemon --config nodemon.json",
"start:debug": "nodemon --config nodemon-debug.json",

nodemon.json

{
    "watch": ["src"],
    "ext": "ts,gql",
    "ignore": ["src/**/*.spec.ts","src/testing/*"],
    "env": {}
}

nodemon-debug.json

{
    "watch": ["src"],
    "ext": "ts,gql",
    "ignore": ["src/**/*.spec.ts","src/testing/*"],
    "env": {},
    "exec": "node --inspect -r ts-node/register src/main.ts"
}

I also set Auto Attach toggle in VS Code with

Debug: Toggle Auto Attach

and now I can see and toggle auto attach from the status bar.

If I leave Auto Attach ON and run npm start:dev (non debug mode), every time it reaches an active breakpoint, it stops on it and complains Could not load source '/src/...': Unable to retrieve source content.. Also vs code shows the debug toolbar.

If I toggle auto attach to off, everything works ok.

My question is: Is this the expected behaviour and I need to toggle on or off if I will debug or not? Should it not to do that automatically, being smart enough to know if it is in debug mode?