I have an application for which the testing is quite extensive. Essentially, we must run the application a few hundred thousand time on different input. So I have built a custom Gradle task which manages forking processes and reaping finished processes. Each of the thousands of test runs generate a file that goes in a results directory. Full testing can take about a week when distributed across 10 cluster nodes.
The issue is, if I need to stop the testing for whatever reason, then there is currently no way for me to start back up where I left off. Gradle's incremental build and caching features (to my understanding) really only work when tasks finish, and it will just rerun the entire task from scratch if the previous invocation was interrupted (ctrl-c).
I could build in some detection of the results files and only rerun segments for which there is no results file. However, this will not work properly when the application is rebuilt, and then testing legitimately must start from scratch.
So how can I reliably detect which testing segments are up to date when the previous task invocation was interrupted?
Annotated tasks
For any Gradle task, if its output files exist and its inputs (including predecessor tasks) are all up-to-date, Gradle will treat that task as up-to-date and not run it again. You tell Gradle about inputs and outputs by annotating properties of the class you write to define the task.
You can make use of this by breaking your custom Gradle testing task into a number of smaller test tasks and have each of those task definitions declare annotated outputs. The test reports are probably the most suitable for those outputs. Then the test tasks which have a report will not have to be re-run if you stop the build halfway.
A whole application rebuild will always need all tests to be re-run
However, if your whole application is rebuilt then those test tasks will no longer be up-to-date as their predecessor build tasks will not be up-to-date. This makes sense of course: a new application build needs its tests to be run again to check it still works as intended.
Multimodule builds may mean only part of an application needs rebuilding
It may be that there are parts of the application that are not being rebuilt, and test tasks that depend solely on those intact parts of the application. If the chain of predecessor tasks for any previously completed test task are all up-to-date, then Gradle will not re-run those tests again either.
This would be more likely to be the case for more test tasks if your application, if appropriate, is separated into different Gradle subprojects in a multimodule build. Each then would have its own chain of tasks which may not have to be re-run if only part of the application's code or other inputs is changed.