Linked Questions

Popular Questions

The TLDR/summary: Is there a way, given two executable files (both compiled from C++, with debug-info present, and from identical or near-identical source code) to list out only the functions that are different in one, relative to the other?

The background/motivation: At the end of every sprint, our engineering team hands a "hero" build over to our QA team as a release candidate. The SQA team then spends several man-weeks testing the release candidate to make sure it works correctly. (They've automated as much of the testing as they can, but there are some parts that aren't easy to automate, hence the laborious manual testing/verification). Inevitably, they come across one or more faults in the release candidate, at which point a bug report is filed, and engineering fixes the bugs and generates a new release-candidate.

At this point, the SQA team has to make a decision: do they restart all testing from scratch, or do they assume that the new release-candidate is identical to the previous release-candidate except for the requested fixes -- i.e. that no regressions were introduced in other parts of the program?

The first choice ("restart from scratch") is safer, but expensive, and causes significant schedule-slippage. The second choice ("just continue testing as before, but with the new build") is quicker, but risks having regressions slip through to the user, if any newly-introduced faults appear in parts of the program that had already been tested using the old build.

So when SQA asks me for advice on this decision, I currently do an "svn diff" between the two build-dates to check exactly what C++ code changed between the two builds, and use that to estimate the risk level of not doing a full-retest. However, some of our SQA people don't trust our build process as much as I do, and they are reluctant to make the assumption that just because the source code has barely changed, that means the executable has barely changed. (i.e. they say "but what if some setting has been changed on the build machine that we don't know about?"; to which I can only appeal to their faith in humanity that nobody would do such a thing)

Therefore, to assuage SQA's fears (and my own), it would be very useful if I could take the new release-candidate executable and do a "diff" on it (vs the old/partially-tested release-candidate executable) and see exactly which functions are different and which remain unchanged. (Obviously I could just use binary-diff or MD5 checksums to do this, but those will only tell me that the two executables are different; they won't tell me which functions are different, which is what I really want to know -- e.g. that after I fixed a typo in the About-box, the core-business-logic functions in the executable didn't change relative to was tested earlier)

I realize the answer to this is likely OS-specific; however we compile under Linux, MacOS/X, and Linux, so any insight for any of those three OS's would be welcome.

Related Questions