How do I get SpecFlow.NUnit to suppress passing scenarios from console output?

Question

When I run my SpecFlow suite with dotnet test, I don't want to see all the step output output from the passing scenarios. I've got nearly 200 scenarios-- I only want to see what's broken!

This was working fine until I just upgraded all the test nugets in this 4-year-old project to the most recent (non-prerelease) versions. Before, if everything passed, I'd see not much more than the summary line. If something failed, I'd see only the steps for the broken tests. I want this behavior back!

These are the current nugets I'm using:

    <ItemGroup>
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
        <PackageReference Include="NUnit" Version="4.1.0" />
        <PackageReference Include="NUnit.ConsoleRunner" Version="3.17.0" />
        <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
        <PackageReference Include="SpecFlow" Version="3.9.74" />
        <PackageReference Include="SpecFlow.NUnit" Version="3.9.74" />
        <PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.9.74" />
    </ItemGroup>

NUnit note: If I downgrade all the SpecFlow projects to my previous version (3.3.57), I still see the problem. It appears that the NUnit upgrade was what actually introduced the output change.

Things I've tried:

1. "verbosity" parameter of dotnet test

dotnet test --filter TestCategory!=wip koala.IntegrationTests/ -l "console;verbosity=quiet"

The default is 'minimal', which prints the output I don't want. This is the second-quietest setting.

The quietest setting is quiet, but that hides all failure output too, so that's no good.

2. adding to the specflow.json file

Per this doc, I tried tweaking the specflow.json file in my SpecFlow project dir. I added the setting "traceSuccessfulSteps", so now the file looks like:

{
  "bindingCulture": { "language" :"en-us" },
  "language": { "feature": "en-us" },
  "runtime": { "missingOrPendingStepsOutcome": "Error" },
  "trace": { "traceSuccessfulSteps": false },
  "plugins": []
}

No luck, I still see the passing output. I don't see any other config settings that could apply.

3. created a .runsettings file

Per this doc, I created a .runsettings file next to the specflow csproj.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration/>
  <TestRunParameters/>
  <NUnit>
    <InternalTraceLevel>Error</InternalTraceLevel>
    <Verbosity>-1</Verbosity>
  </NUnit>
</RunSettings>

I see no difference in output.

How can I keep the passing-scenario steps out of my dotnet test console output?

Answer 1

I'll try my hand at this anyways. I know in MS Test each test shows "Passed" or "Failed" on the command line. Assuming NUnit does something similar, pipe the test output to the Select-String commandlet in PowerShell:

dotnet test --filter TestCategory!=wip koala.IntegrationTests/ -l "console;verbosity=quiet" | Select-String 'Failed'
#                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^

Just replace 'Failed' with a string that always occurs when a test fails, but not when a test passes. If you aren't familiar with PowerShell, the general pattern is:

dotnet test arguments | Select-String 'foo'

Where you replace arguments with whatever you normally pass to dotnet test. It should filter out all test output except for lines that match the pattern passed to Select-String.

---

I used to do this sort of thing all the time years ago developing a Ruby on Rails application. We had a Rake task that ran our tests similar to dotnet test. It showed a series of ******FFF*FFF*FFF*** characters where * meant a test passed, and F meant a test failed. Just a quick rake test | grep 'F' in Bash made for a lot less output to track visually.

Composable commands are a powerful tool I used all the time on Linux and MacOS. I frequently forget this on Windows, but PowerShell has some very powerful pipe commands as well.