So I’m briefly going to talk about test plan numbers here, mainly in response to mst’s blog post, where he expresses his opinion that test plan numbers are bad. I happen to disagree with him in some respects and agree with him in others (we’ve also discussed on IRC briefly).
Matt’s verdict is that test plan counts don’t gain you anything as we don’t need to worry about arbitrary
`exit 0;`
showing up in CPAN code and the like. However, I think there’s a deeper problem with not using test plans as there is code that can happen in tests at runtime that wouldn’t show up as a failure in a test run.
Imagine the following scenario:
While this looks like it’ll run just fine, imagine that there was a bug in $obj->gain_results() that caused it to either return early, or something like an off-by-one error. Your test would run through, but you wouldn’t see the failure and ship buggy code to production. Now, the actual solution to this is to use the right tool for the job and use a test function like is_deeply() or something from Test::Deep or the like; but I’ve seen code like this in multitudes of test suites.
Using a test plan with your expected number of results would have thrown this test up as a failure, with perhaps some diving in to investigate the cause, but you’d at least know there was a regression. Personally I’ll use the right tool for the job and test plans as they give me something to fall back on.
Matt makes a very good point regarding merging of branches and test counts, but I think I’d rather take the hit at merge time, since I’d be dealing with merging issues anyway than have the prospect of a not-easily traceable regression on my hands. When I’m reviewing code changes, I also find it handy to see if tests have been added or removed to double check that I’m not re-covering the same code with the tests I’ve added.
Opinions?
I wholeheartedly agree. Perhaps resolving merge conflicts with test count numbers is a pain, but frankly just suck it up as part of having a test suite. If you don’t want to deal with the hassle of updating test counts, then why are you bothering with test cases? They’re there to catch mistakes, and by being explicit you’re being that much more careful.
And in addition to what Scott says above, I find test counts are especially useful *because* of merging branches of code. Test counts can’t be automatically merged by version control systems, whereas the body of a test can. I have run across many instances where the meaning of a test changes, but the auto-merge between branches of similar code don’t notice that, and instead blindly merge changes together, making the meaning of the test pointless. It’s only the unmergeable test counts that give you any hint that something in that test case has changed.
So when I see test counts that need to be merged, that’s a red flag to me telling me I need to scan through the test to see if it makes sense. Again, in the spirit of having test coverage at all, it covers your ass when something goes wrong.