Pysys - How to run only the validate portion of a test

Question

I am looking at a way to add a new "mode" in the Pysys Baserunner.

In particular I would like to add a validate mode, that just re-run the validation portion. Useful when you are writing your testcase and trying to tune the validation condition so to fit the current output without having to re-reun the complete testcase.

What is the best way to do this without having to change the original class?

Answer 1

This requires support from the framework unfortunately. The issue is that the BaseRunner class will always automatically purge the output directory, and there is no hook into the framework to allow you to avoid this. You can for instance move the output subdirectory manually you want to re-run the validation over to say 'repeat' (same directory level), and then use;

from pysys.constants import *
from pysys.basetest import BaseTest

class PySysTest(BaseTest):
    def execute(self):
        if self.mode=='repeat': pass

    def validate(self):
        if self.mode=='repeat':
            self.output=os.path.join(self.descriptor.output, 'repeat')

where I have ommitted the implementations of the execute and validate. You would need to add the mode into the descriptor for the test

  <classification>
    <groups>
      <group></group>
    </groups>
    <modes>
      <mode>repeat</mode>
    </modes>
  </classification>

and run using "pysys.py run -mrepeat". This would help with the debugging if your execute takes a long time, but probably not want you want out-of-the-box i.e. a top level option to the runner to just perform validation over a previously run test. I'll add a feature request for this.

Answer 2

Since the original discussion, a --validateOnly command line option was added to PySys (in v1.1.1) which does pretty much what you suggest - it skips the execute method and just runs validate.

Assumes you aren't running with --purge (which I imagine is a safe assumption for this use case), and that you don't have validation commands that try to read zero-byte files from the output dir (which always get deleted even if --purge is not specified). However assuming those conditions are met, your (non-empty) output files will still be there after you've completed the first run of the test and you can re-run just validation using the --validateOnly command.

To get this feature you can install the latest PySys version (1.4.0) - see https://pypi.org/project/PySys/