Sunday, November 22, 2009

The Limitations of TDD

During the last 12-18 months, TDD has broken into the mainstream, it seems. And now, we're starting to see some backlash, as its limitations become better understood. Here is a sample discussion from Cédric Beust, who wrote the commentary, is not some unknown guy with a weird name. He wrote the TestNG unit testing framework, which is second only to JUnit in popularity. He also wrote the book, Next Generation Java Testing, which is probably the best book on pragmatic software testing that I've read in a long time. Here goes...

> That's an interesting point. Are you, in effect, saying
> that unit testing is overly emphasized, and at the expense
> of other forms of testing?

This has also been my experience, although to be honest, I see this problem more in agile/XP literature than in the real world.

This is the reason why I claim that:

- TDD encourages micro-design over macro-design
- TDD generates code churn

If you obsessively do TDD, you write tests for code that you are pretty much guaranteed to throw away. And when you do that, you will have to refactor your tests or rewrite them completely. Whether this refactoring can be done automatically or not is beside the point: you are in effect creating more work for yourself.

When I start solving a problem, I like to iterate two or three times on my code before I'm comfortable enough to write a test.

Another important point is that unit tests are a convenience for *you*, the developer, while functional tests are important for your *users*. When I have limited time, I always give priority to writing functional tests. Your duty is to your users, not to your test coverage tools.

You also bring up another interesting point: overtesting can lead to paralysis. I can imagine reaching a point where you don't want to modify your code because you will have too many tests to update (especially in dynamically typed languages, where you can't use tools that will automate this refactoring for you). The lesson here is to do your best so that your tests don't overlap.

--Cedric Beust


Thomas Eyde said...

In my opinion, unit tests and functional tests has nothing to do with TDD. They are a part of our toolbox, yes, but TDD is about using our tests to drive our code.

The focus on unit tests vs functional test vs purity is misguided.

I always start out with my functional tests, and add my detailed unit tests when the need arrives.

I still call it TDD

Andrew Binstock said...

Understood. As you know, what you describe is not "standard TDD". (Write a test, write the smallest amount of code to make the test pass. Repeat many times, refactor. Start again.)

But I think it is the direction TDD will eventually go in. That is, functional tests will be the principal tests, and unit tests will simply be for purposes of capturing functionality ("characterization tests") and for testing things that are tedious to test via functional tests (edge-conditions, etc.)

Anonymous said...

Your description of the "standard TDD" cycle is incorrect:

From Test-Driven Development by Kent Beck:

1 . Quickly add a test.
2. Run all tests and see the new one fail.
3. Make a little change.
4. Run all tests and see them all succeed.
5. Refactor to remove duplication.

Your summary of the cycle is 1-4, 1-4, 1-4... then 5. Refactoring occurs more frequently than this in TDD.

This seems to be a common misunderstanding about TDD: refactoring is an essential, and core part of the cycle. Of course, people can do 1-4 N times then 5 -but to borrow from Ron Jefferies "If you don’t do what we suggest, then don’t call what you do by the name of what we suggest": don't call it TDD if refactoring isn't part of every cycle.

Anonymous said...

Many people seem to coin their own definition for a word or phrase. You've assigned your own definition to TDD by having refactoring occur less frequently than the actual definition by Beck. Whereas you state: test, code; test, code; test, code; refactor; Beck's definition states:

Red -
Write a little test that doesn't work, and perhaps doesn't even compile at first.

Green -
Make the test work quickly, committing whatever sins necessary in the process.

Refactor -
Eliminate all of the duplication created in merely getting the test to work.

Eg, test, code, refactor; test, code, refactor; et cetera. And since Beck coined the term and defined the practice, he gets to define what it means. The cycle you define is not wrong, but it's not right to call it TDD. I don't say this to be pedantic, but to communicate: if you say "Decorator pattern" and you mean the GoF pattern but I have my own definition for "Decorator" then we're going to get really confused - and we may not even realize it for some time.

Anonymous said...

I think there's confusion over what TDD is, and what it isn't. It is a practice espoused by XP. It isn't the only testing practice espoused by XP.

The government health agencies emphasize people should eat more fruit and vegetables, but it's incorrect to state they say you should only eat fruit and vegetables. If people started eating enough fruit at the expense of grains, their emphasis would shift. Just like TDD - TDD is only one part of XP, and certainly not the only testing strategy discussed in the canonical XP literature. I wonder what "agile/XP literature" Cédric Beust is referring to.

It's easy to see why automated unit testing has been focussed on so much by people at large in the real world - it wasn't really being practiced at all. Prior to JUnit, "testing" usually meant functional testing, and usually through the UI; perhaps it was even automated.

However, whatever the emphasis by people in the real world, the XP literature *does* stress functional testing (as for "agile" literature, who knows? Scrum, for example, doesn't discuss any actual software practices and certainly unit testing).

These are canonical XP literature references from original XPers: Kent Beck, Martin Fowler, Ward Cunningham:


In Extreme Programming Explained (1st Edition, to demonstrate it's always been at the heart of XP) Beck writes:

"Who Writes Tests? ... the tests come from two sources: - Programmers - Customers... The programmers write tests method-by-method.... The customers write tests story-by-story. The question they need to ask themselves is, "What would have to be checked before I would be confident that this story was done?" Each scenario they come up with turns into a test, in this case a functional test."

"[T]he unit and functional tests are the heart of the XP testing strategy..."

"Here's what XP testing is like. Every time a programmer writes some code, they think it is going to work. So every time they think some code is going to work, they take that confidence out of the ether and turn it into an artifact that goes into the program..... The same story works for the customer. Every time they think of something concrete the program should do, they turn it into another piece of confidence that goes into the program. Now their confidence is in there with the programmers' confidence."


From Planning Extreme Programming (Beck, Fowler):

"[T]the customer will have to specify acceptance tests whose execution will determine whether the user stories have been successfully implemented. Thus all user stories must be testable. The programmers and the customer alike must agree that there are concrete tests that will demonstrate that the user story has been successfully implemented."


Ward Cunningham wrote a whole framework - Fit - and a whole book on Fit, which is a framework for functional tests that has been around in some form or other since 1989 (


So, perhaps the public perception is XP has given functional testing the short straw, but the founders' XP literature has acknowledged the importance of functional testing from day zero. One of them has even written a framework around functional testing, just like another one wrote a framework for unit testing. That the "real world" happened to adopt JUnit and unit testing more than Fit and functional testing is not something the XP founders can control. If they had that kind of control, I'm sure the "real world" would be very different ;)

The actual issue seems to be that people are looking for one magic pill to cure all their problems - but no one above said TDD or unit testing was that cure-all pill or that you only need unit testing. It would be more accurate to say most people, including Cédric, ignored the parts about functional testing in the XP literature.

Perryn Fowler said...

I see Cedric still doesn't get it.

A very large part of TDD done properly is about informing your design. That is why you write the test *first*. If you 'iterate' over your code several times before you feel comfortable in writing a test, then you are clearly designing first and then trying to figure out how to test it. Down this path lies high coupling in code and in tests.

Given that, it comes as no surpise that he comes up with the old BS about not wanting to change code because it would break too many tests. If you do TDD properly your tests will *enable* change not act against it - that is the whole point. If this is not happening for you, you are doing it wrong. Try again before dismissing the technique.

Anonymous said...

The article and comments are thought provoking and informative. I also like your recent article "Seriously pragmatic agility" on SD.

What's your opinion on the Model-Based Testing (MBT) and how do you think MBT can be adopted into Agile process? The fact that MBT process starts at functional requirements making it ideal for Agile process as it allows test design and development to go in tandem with the software design and development.

Andrew Binstock said...

@Perryn Fowler: Unit tests do enable refactoring, but they do not to make large changes easier.

@Anonymous Unfortunately, I don't know enough about MBT to be able to answer your question.

Anonymous said...

I don't think Perryn Fowler gets it.

His comments are prescriptive but not particularly instructive or enlightening (e.g., "If you do TDD properly your tests will *enable* change not act against it")

Unit testing is not a substitute for other forms of testing, and is certainly a substantial added expense and overhead. And TDD, much like Communism, is better in theory than in actual practice.

If I could increase my team's development cycle times by 25%-30% so that they could write unit tests, would it improve product quality? Yes it likely would. But if I had an additional 25%-30% time in a project cycle (and what projects ever do?) I would rather put that time into functional QA and/or developing functional code. That's the better bang for the buck.

Cedric Beust says it best:

"...unit tests are a convenience for *you*, the developer, while functional tests are important for your *users*. When I have limited time, I always give priority to writing functional tests. Your duty is to your users, not to your test coverage tools."

Thomas Eyde said...

@anonymous, your argument for better bang for the buck is flawed. If I understand you correctly, you say that you need more time to write unit tests? That is not true, as TDD enables you to write more code faster.

The time savings comes from fewer mistakes, less broken code, and writing only the code you need.

I am not saying we don't need the other things, I'm with you there. But I think writing more functional code without automatic unit tests will take longer and with less quality.

The net gain will be less than you think.

guajira said...

I hope TDD abolish these limitations.