tag:blogger.com,1999:blog-9620948.post6897511468346662616..comments2023-10-20T12:24:17.734-07:00Comments on Binstock on Software: The Fallacy of 100% Code CoverageAndrew Binstockhttp://www.blogger.com/profile/16321156191558412680noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-9620948.post-62517687364913372232013-12-16T01:04:28.285-08:002013-12-16T01:04:28.285-08:00Umm, what does "culling" mean?Umm, what does "culling" mean?<br />bloggisthttps://www.blogger.com/profile/03541997946420443116noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-42313111858576619182013-04-17T08:19:58.005-07:002013-04-17T08:19:58.005-07:00I'm getting annoyed with alot of developers wh...I'm getting annoyed with alot of developers who use this as an excuse not to improve or measure their unit tests and code coverage.<br /><br />100% coverage, will give you more confidence in your system that 30%, but it's important to understand that 100% code coverage, doesn't mean 100% of scenarios are tested i.e. a while loop that explodes after the nth execution wouldn't necessarily be caught by a test that achieved 100% code coverage.<br /><br />i.e. strive for the best code coverage you can get, but don't believe this makes your system bug free.<br /><br />There are of course reasons why you probably should never get 100% coverage as writing tests for getters/setters or some other generated code doesn't always make sense.<br /><br />CodeMonkeyhttps://www.blogger.com/profile/04591974913496200134noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-36105183157638319772008-03-08T23:53:00.000-08:002008-03-08T23:53:00.000-08:00stephan said: "Beside that I believe it's easier t...stephan said: "Beside that I believe it's easier to keep your coverage at 100% than to keep it at 80%."<BR/><BR/>I agree with that as an important reason to push for 100% coverage. If my setter / getter isn't working, i'd like my unit test to tell me that. <BR/><BR/>My approach is that I write my code test first, continue developing till all tests pass, then I check my coverage and it is interesting to me that often my code coverage tool picks up an edge case I had forgotten to test but had coded.<BR/><BR/>By having 100% coverage, these missing tests jump out much more than if they were just another untested line amongst a whole bunch of untested setters / getters.<BR/><BR/>I've written more here:<BR/><BR/>http://homepage.mac.com/hey.you/lessons.html<BR/><BR/>TonyTonyhttps://www.blogger.com/profile/16719976089855203933noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-59109330448614000902007-11-29T19:08:00.000-08:002007-11-29T19:08:00.000-08:00Though I totally agree that 100% code coverage is ...Though I totally agree that 100% code coverage is a fallacy, I think it is good to aim for it not for the sake of it but just as a driver to improve the code.<BR/><BR/>In that matter, let me quote you: "It appeals to me because almost invariably when I refactor code to make it more testable, I find that I've improved it".Anonymoushttps://www.blogger.com/profile/13987512594987806769noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-5204338305077050462007-11-28T19:40:00.000-08:002007-11-28T19:40:00.000-08:00@stephan: Sorry, in your original reply, I took yo...@stephan: Sorry, in your original reply, I took your "100% CC" to mean 100% code coverage, not as you meant it: "100% cyclomatic complexity," which is a term I'm not familiar with. Indeed, if you cover every logical branch, you're going to cover a lot of your code. But surely many paths will require more than one test, while some paths will hardly be worth the time to write a test--where you draw the line reflects the main point of my post; that getting every last line tested (such as methods where CC = 1) is not an inherently desirable goal. There are better things to do to improve your code than getting every bit tested.<BR/><BR/>By the way, <A HREF="http://binstock.blogspot.com/2007/04/how-many-unit-tests-per-method-see.html" REL="nofollow"> here</A> is a post on this blog relating to the minimum number of tests to use in relation to the cyclomatic complexity of a method. In retrospect, I think it presents too low a minimum.Andrew Binstockhttps://www.blogger.com/profile/16321156191558412680noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-80036725605212078652007-11-27T14:30:00.000-08:002007-11-27T14:30:00.000-08:00That's not true, the code snippet:public void setH...That's not true, the code snippet:<BR/><BR/>public void setHeight( float newHeight ) <BR/>{ height = newHeight; }<BR/><BR/>The behaviour is that height is being set to newHeight. I don't know if the Java compiler would detect this (I work exclusively in dynamic languages):<BR/><BR/>public void setHeight( float newHeight ) <BR/>{ height = newHeightLOLCATS; }<BR/><BR/>If the compiler doesn't pick that up, then you have a broken test case that you haven't tested for.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9620948.post-39332432488155521652007-11-27T08:11:00.000-08:002007-11-27T08:11:00.000-08:00Personally I believe the greatest value of using a...Personally I believe the greatest value of using a Code Coverage tool is not to determine if your code is 100% tested (in fact on larger, older projects I rarely see 100%) but to determine what areas of code are consistantly not covered.<BR/><BR/>For example, the greatest area of untested code I see when introducing a code coverage tool on a project is Exception Handling. If the developer can see this and learn omissions in their testing then their skill set increases.<BR/><BR/>Also, I think another related conversation/argument is choosing between writing your own tests and using a tool that generates them for you....oops! Sorry Andrew, looks like another can of worms opening.. ;-)<BR/>-RichUnknownhttps://www.blogger.com/profile/11774975286066909299noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-13902903815209633902007-11-27T02:19:00.000-08:002007-11-27T02:19:00.000-08:00@Andrew: I'm confused because I thought cyclomatic...@Andrew: I'm confused because I thought cyclomatic complexity (CC) does represent all possible paths through the code. So with 100% CC coverage you have 100% path coverage.<BR/><BR/>Peace<BR/>-stephanStephan.Schmidthttps://www.blogger.com/profile/03845125686370893937noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-52860247495654642132007-11-27T00:56:00.000-08:002007-11-27T00:56:00.000-08:00@stephan: You wrote: "With 100% code coverage all ...@stephan: You wrote: "With 100% code coverage all your decisions are covered." No, this is not correct, and is another fallacy I'll have to blog on. See David's post (right above yours) for a slightly different sentiment that clarifies the problems with your assertion.<BR/><BR/>@mr. neighborly: Agreed that TDD is a different matter. Although 100% code coverage is not a goal of TDD that I know of, rather it's a side effect. <BR/><BR/>@larry: not sure I agree. If you see unit tests as a way of preventing future stupid errors, then sure I'm cool with what you propose. For myself, I find it hard enough to write code that works correctly and is clear. Worrying about building in safeguards for possible errors that might or might not occur down the road is a near-endless task, and investing in 100% CC seems like an extravagant price to pay a priori. I'd vote for spending the time refactoring and documenting before writing unit tests for getters/setters. <BR/><BR/>@dre, @ryan: thanks for thoughtful responses with different takes.Andrew Binstockhttps://www.blogger.com/profile/16321156191558412680noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-86992784386307440072007-11-26T15:31:00.000-08:002007-11-26T15:31:00.000-08:00I agree with reasons others have given for 100% co...I agree with reasons others have given for 100% coverage (hitting every line, not just calling every method): TDD, documentation, safety net when refactoring, and (especially important in my case) preventing stupid errors.<BR/><BR/>There is a problem with focusing on 100% coverage, though: thinking that you're done just because you've got it. I think that's a much bigger problem than wasted time unit testing getters and setters. <BR/><BR/>You've got to also test the edge cases, cases that intuition and experience tell you are likely to cause errors, etc.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9620948.post-1009914777688275512007-11-26T14:51:00.000-08:002007-11-26T14:51:00.000-08:00public void setHeight( float newHeight ) { height ...public void setHeight( float newHeight ) <BR/>{ height = newHeight; }<BR/><BR/>It can't go wrong and can't change. Hmmm.<BR/><BR/>%s/height/something_else/<BR/><BR/>Oops.<BR/><BR/>"Constants aren't, variables won't." Things that "can't" go wrong often do, and things that "can't" change often do. When you assert that some code doesn't need unit tests because it can't go wrong and can't change, you also assert that no one that ever works on the project will ever make a simple typo, will never hit the delete key in the wrong place, and (in the parlance of Common Lisp) will never, say, put an "around" method on the setter and forget to call call-next-method.Larry Clapphttps://www.blogger.com/profile/09056791619989601236noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-72098266720973092552007-11-26T14:11:00.000-08:002007-11-26T14:11:00.000-08:00If you feel that the problem is only with getters ...If you feel that the problem is only with getters and setters, then why not just ignore them in the code coverage report?<BR/><BR/><A HREF="http://www.jroller.com/MikeSlattery/entry/effective_code_coverage" REL="nofollow">Mike Slattery</A> posted about re-targeting code coverage, and provided a patch for Cobertura to ignore simplistic getters and setters.<BR/><BR/>I do think that code coverage allows you to learn how to write better unit tests. Code coverage therefore isn't a means to an ends, or any end goal. It's a tool, just like unit testing, to further code quality and improve the results of testing efficiently.<BR/><BR/>For example, if you're writing unit tests, find a huge cluster of bugs (no matter the size of your codebase), but have only received 10% code coverage (decision-condition coverage, not just statement coverage) - then you have done a really good job writing your unit tests and may actually be done. If a new person starts writing unit tests everyday throughout the project, and continually gets 80-99% code coverage with very few bugs - then there is either a problem with the developer-tester or you have perfect coders (less likely).<BR/><BR/>Code coverage is simply a tool - you can use it in your IDE or you can use it at build time, comparing the results of yourself (and your unit tests) with that of the rest of the team.<BR/><BR/>This whole notion of having a green bar at all times is out of control with some developers. These are the same people who never promote warnings to errors. They will write comments to prevent certain lines from being checked by their static analyzer just to get "clean code". The sad part is when managers start wanting to hire/promote these guys, while punishing the developers who show how shoddy their code is.drehttps://www.blogger.com/profile/17414510788948258195noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-47648806814601793562007-11-26T13:46:00.000-08:002007-11-26T13:46:00.000-08:00Writing tests for setters and getters can be neede...Writing tests for setters and getters can be needed, but not in compiled languages. Dynamic languages like python cannot always verify that simply stuff like a setter or getter does what you want without a unit test.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9620948.post-20942936687316368782007-11-26T12:50:00.000-08:002007-11-26T12:50:00.000-08:00This is preposterous. It's as if you're suggestin...This is preposterous. It's as if you're suggesting that the only way to write good code is to write the code well. Next you'll be saying that the only way for management to ensure their programmers are putting out quality is to hire good programmers.<BR/><BR/>Clearly an insupportable position.Unknownhttps://www.blogger.com/profile/13878944123727679139noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-36966033584081372822007-11-26T09:40:00.000-08:002007-11-26T09:40:00.000-08:00I think you have a poor view of what testing is. ...I think you have a poor view of what testing is. Testing (if you're doing the test-first/driven type at least) is supposed to be used to specify behavior, not to verify it.<BR/><BR/>So, if I'm creating a getter or setter, I want to specify that my class should have that before I write the class. I specify it, run the test, it fails. I write the getter/setter, re-run the test, all tests pass, the class's behavior is complete. <BR/><BR/>Now with these tests I can hand them to someone who doesn't necessarily know the system through and through or possibly even know how to write code and they can basically look over the function names (or if you're doing BDD, the spec names) and tell me if I'm on the right track. I can also 100% verify that I haven't broken something on a dependent code path by overriding a method, etc.<BR/><BR/>100% code coverage is possible and is something to strive for, but I think making it a requirement (as some shops do these days) is silly.Mr. Neighborlyhttps://www.blogger.com/profile/02063939073935274480noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-31654439820826238312007-11-26T08:48:00.000-08:002007-11-26T08:48:00.000-08:00"By pushing for an artificial goal of 100% a devel..."By pushing for an artificial goal of 100% a developer is incentivized against writing multiple tests for complex code, in order to have the time to write tests for getters and setters. That surely can't be right."<BR/><BR/>It's not artifical to request 100% CC coverage. With 100% CC coverage all your decisions are covered.<BR/><BR/>Beside that, most people I know use reflection to test setters and getters automatically. So no need to write any code to test basic setters and getters. And when they evolve and contain new business logic, you detect that by failing automatic tests and you can write a custom unit test (which you should as it is no longer a simple getter/setter).<BR/><BR/>Beside that I believe it's easier to keep your coverage at 100% than to keep it at 80%.<BR/><BR/>Peace<BR/>-stephan<BR/><BR/>-- <BR/>Stephan Schmidt :: stephan@reposita.org<BR/>Reposita Open Source - Monitor your software development<BR/>http://www.reposita.org <BR/>Blog at http://stephan.reposita.org - No signal. No noise.Stephan.Schmidthttps://www.blogger.com/profile/03845125686370893937noreply@blogger.comtag:blogger.com,1999:blog-9620948.post-16512853216262928582007-11-26T03:15:00.000-08:002007-11-26T03:15:00.000-08:00via dzoneas you hint, code coverage is not the sam...via dzone<BR/>as you hint, code coverage is not the same as code pathways, but while 100% coverage is generally achievable although of dubious use, testing all pathways is not typically feasible.David O'Mearahttps://www.blogger.com/profile/18270557377107624642noreply@blogger.com