This ultimately means, what most programmers intuitively know, that it's impossible to write adequate test coverage up front (since we don't even really know how we want the program to behave) or worse, test coverage gets in the way of the iterative design process. In theory TDD should work as part of that iterative design, but in practice it means a growing collection of broken tests and tests for parts of the program that end up being completely irrelevant.
The obvious exception to this, where I still use TDD, is when implementing a well defined spec. Anytime you need to build a library to match an existing protocol, well documented api, or even an non-trivial mathematical function, TDD is a tremendous boon. But this is only because the program behavior is well defined.
The times where I've used TDD and it makes sense it's be a tremendous productivity increase. If you're implementing some standard you can basically write the tests to confirm you understand how the protocol/api/function works.
Unfortunately most software is just not well defined up front.
This is exactly how I feel about TDD, but it always feels like you're not supposed to say it. Even in environments where features are described, planned, designed, refined, written as ACs, and then developed, there are still almost always pivots made or holes filled in mid-implementation. I feel like TDD is not for the vast majority of software in practice - it seems more like something useful for highly specialist contexts with extremely well defined objective requirements that are made by, for, and under engineers, not business partners or consumers.
That rings true to my experience, and TDD doesn't add much to that process.
An implementation without definition, and a whole host of assumptions gets delivered as v1.
Product expectations get lowered, bugs and defects raised, implementation is monkey patched as v2.
devs quit, managers get promoted, new managers hire new devs, they ask for the definition and they're asked to follow some flavor of the year process (TDD, Agile, whatever).... rinse and repeat v3.
Sad. True. Helpless. Hopeless.
It’s always possible to write a test case that covers a new high-level functional requirement as you understand it; part of the skill of test-first (disclaimer - I use this approach sometimes but not religiously and don’t consider myself a master at this) is identifying the best next test to write.
But a lot of people cast “unit test” as “test for each method on a class” which is too low-level and coupled to the implementation; if you are writing those sort of UTs then in some sense you are doing gradient descent with a too-small step size. There is no appreciable gradient to move down; adding a new test for a small method doesn’t always get you closer to adding the next meaningful bit of functionality.
When I have done best with TDD is when I start with what most would call “functional tests” and test the behaviors, which is isomorphic to the design process of working with stakeholders to think through all the ways the product should react to inputs.
I think the early TDD guys like Kent Beck probably assumed you are sitting next to a stakeholder so that you can rapidly iterate on those business/product/domain questions as you proceed. There is no “upfront spec” in agile, the process of growing an implementation leads you to the next product question to ask.
In my experience, the best time to do "test for each method on a class" or "test for each function in a module" is when the component in question is a low level component in the system that must be relied upon for correctness by higher level parts of the system.
Similarly, in my experience, it is often a waste of effort and time to do such thorough low level unit testing on higher level components composed of multiple lower level components. In those cases, I find it's much better to write unit tests at the highest level possible (i.e. checking `module.top_level_super_function()` inputs produce expected outputs or side effects)
Definitely agree with you here - I've seen people dogmatically write unit tests for getter and setter methods at which point I have a hard time believing they're not just fucking with me. However, there's a "sweet spot" in between writing unit tests on every single function and writing "unit tests" that don't run without a live database and a few configuration files in specific locations, which (in my experience) is more common when you ask a mediocre programmer to try to write some tests.
Those tests suit a project that applies the open-closed principle strictly, such as libraries / packages that will rarely be modified directly and will mostly be used by "clients" as their building blocks.
They don't suit a spaghetti monolith with dozens of leaky APIs that change on every sprint.
The harsh truth is that in the industry you are more likely to work with spaghetti code than with stable packages. "TDD done right" is a pipe dream for the average engineer.
I always suspect that many people who have a hard time relating to TDD already have experience writing these class & method oriented tests. So they understandably struggle with trying to figure out how to write them before writing the code.
Thinking about tests in terms of product features is how it clicked for me.
That being said, as another poster above mentioned, using TDD for unstable or exploratory features is often unproductive. But that’s because tests for uncertain features are often unproductive, regardless if you wrote them before or after.
I once spent months trying to invent a new product using TDD. I was constantly deleting tests because I was constantly changing features. Even worse, I found myself resisting changing features that needed changing because I was attached to the work I had done to test them. I eventually gave up.
I still use TDD all the time, but not when I’m exploring new ideas.
You can (and often should!) have a suite of unit tests, but you can choose to write them after the fact, and after the fact means after most of the exploration is done.
I think if most people stopped thinking of unit tests as a correctness mechanism and instead thought of them as a regression mechanism unit tests as a whole would be a lot better off.
But TDD is fantastic for growing software as well! I managed to save an otherwise doomed project by rigorously sticking to TDD (and its close cousin Behavior Driven Development.)
It sounds like you're expecting that the entire test suite ought to be written up front? The way I've had success is to write a single test, watch it fail, fix the failure as quickly as possible, repeat, and then once the test passes fix up whatever junk I wrote so I don't hate it in a month. Red, Green, Refactor.
If you combine that with frequent stakeholder review, you're golden. This way you're never sitting on a huge pile of unimplemented tests; nor are you writing tests for parts of the software you don't need. For example from that project: week one was the core business logic setup. Normally I'd have dove into users/permissions, soft deletes, auditing, all that as part of basic setup. But this way, I started with basic tests: "If I go to this page I should see these details;" "If I click this button the status should update to Complete." Nowhere do those tests ask about users, so we don't have them. Focus remains on what we told people we'd have done.
I know not everyone works that way, but damn if the results didn't make me a firm believer.
Unit tests are still easy to write but most complex software have many parts that combine combinatorially and writing integration tests requires lots of mocking. This investment pays off when the design is stable but when business requirements are not that stable this becomes very expensive.
Some tests are actually very hard to write — I once led a project that where the code had both cloud and on-prem API calls (and called Twilio). Some of those environments were outside our control but we still had to make sure they we handled their failure modes. The testing code was very difficult to write and I wished we’d waited until we stabilized the code before attempting to test. There were too many rabbit holes that we naturally got rid of as we iterated and testing was like a ball and chain that made everything super laborious.
TDD also represents a kind of first order thinking that assumes that if the individual parts are correct, the whole will likely be correct. It’s not wrong but it’s also very expensive to achieve. Software does have higher order effects.
It’s like the old car analogy. American car makers used to believe that if you QC every part and make unit tolerances tight, you’ll get a good car on final assembly (unit tests). This is true if you can get it right all the time but it made US car manufacturing very expensive because it required perfection at every step.
Ironically Japanese carmakers eschewed this and allowed loose unit tolerances, but made sure the final build tolerance worked even when the individual unit tolerances had variation. They found this made manufacturing less expensive and still produced very high quality (arguably higher quality since the assembly was rigid where it had to be, and flexible where it had to be). This is craftsman thinking vs strict precision thinking.
This method is called “functional build” and Ford was the first US carmaker to adopt it. It eventually came to be adopted by all car makers.
https://www.gardnerweb.com/articles/building-better-vehicles...
Say, initially you were told "if I click this button the status should update to complete", you write the test, you implement the code, rinse and repeat until a demo. During the demo, you discover that actually they'd rather the button become a slider, and it shouldn't say Complete when it's pressed, it should show a percent as you pull it more and more. Now, all the extra care you did to make sure the initial implementation was correct turns out to be useless. It would have been better to have spent half the time on a buggy version of the initial feature, and found out sooner that you need to fundamentally change the code by showing your clients what it looks like.
Of course, if the feature doesn't turn out to be wrong, then TDD was great - not only is your code working, you probably even finished faster than if you had started with a first pass + bug fixing later.
But I agree with the GP: unclear and changing requirements + TDD is a recipe for wasted time polishing throw-away code.
Edit: the second problem is well addressed by a sibling comment, related to complex interactions.
Writing tests as you write the code is just regular and proper software development.
You think of a small chunk of functionality you are comfident about, write the tests for that (some people say just one test, i am happy with up to three or so), then write the implementation that makes those tests pass. Then you refactor. Then you pick off another chunk and 20 GOTO 10.
If at some point it turns out your belief about the functionality was wrong, fine. Delete the tests for that bit, delete the code for it, make sure no other tests are broken, refactor, and 20 GOTO 10 again.
The process of TDD is precisely about writing code when you don't know how the program is going to work upfront!
On the other hand, implementing a well-defined spec is when TDD is much less useful, because you have a rigid structure to work to in both implementation and testing.
I think the biggest problem with TDD is that completely mistaken ideas about it are so widespread that comments like this get upvoted to the top even on HN.
Is general understanding of TDD really that far off the mark? I had no idea, and I've been doing this for essentially 2 decades now.
If you're thinking of unit tests as the thing that catches bugs before going to production and proves your code is correct, and want to write a suite of tests before writing code, that is far beyond the capabilities of most software engineers in most orgs, including my own. Some folks can do it, good for them.
But if you think of unit tests as a way to make sure individual little bits of your code work as you're writing them (that is, you're testing "the screws" and "the legs" of the tables, not the whole table), then it's quite simple and really does save time, and you certainly do not need full specs or even know what you're doing.
Write 2-3 simple tests, write a function, write a few more tests, write another function, realize the first function was wrong, replace the tests, write the next function.
You need to test your code anyway and type systems only catch so much, so even if you're the most agile place ever and have no idea how the code will work, that approach will work fine.
If you do it right, the tests are trivial to write and are very short and disposable (so you don't feel bad when you have to delete them in the next refactor).
Do you have a useful test suite to do regression testing at the end? Absolutely not! In the analogy, if you have tests for a screw attaching the leg of a table, and you change the type of legs and the screws to hook them up, of course the tests won't work anymore. What you have is a set of disposable but useful specs for every piece of the code though.
You'll still need to write tests to handle regressions and integration, but that's okay.
But the end result of writing tests is often that you create a lot of testing tied to what should be implementation details of the code.
E.g. to write "more testable" code, some people advocate making very small functions. But the public API doesn't change. So if you test only the internal functions, you're just making it harder to refractor.
You're not supposed to write every single test upfront, you write a tiny test first. Then you add more and refactor your code, repeat until there is nothing left of that large complicated thing you were working on.
There are also people who test stupid things and 3rd party code in their tests and either they get a fatigue from it and/or think their tests are well written.
The raison d'etre of TDD is that developers can't be trusted to write tests that pass for the right reason - that they can't be trusted to write code that isn't buggy. Yet it depends on them being able to write tests with enough velocity that they're cheap enough to dispose?
But yeah, trying to write all the tests for a whole big component up front, unless it's for something with a stable spec (eg. I once implemented some portions of the websockets spec in servo, and it was awesome to have an executable spec as the tests), is usually an exercise in frustration.
People blame engineers for not writing tests or doing TDD when, if they did, they would likely be replaced with someone who can churn out code faster. It is rare, IME, to have culture where the measured and slow progress of TDD is an acceptable trade off.
I would however be more amenable to e.g. Prototyping first, and then using that as a guide for TDD. Not sure if there is a name for that approach though. "spike" maybe?
[1] https://www.machow.ski/posts/galls-law-and-prototype-driven-...
Interesting - for me, that's the only time I truly practice TDD, when I don't know how the code is going to work. It allows me to start with describing the ideal use case - call the API / function I would like to have, describe the response I would expect. Then work on making those expectations a reality. Add more examples. When I run into a non-trivial function deeper down, repeat - write the ideal interface to call, describe the expected response, make it happen.
For me, TDD is the software definition process itself. And if you start with the ideal interface, chances are you will end up with something above average, instead of whatever happened to fall in place while arranging code blocks.
So the TDD OP describes here is not an Agile TDD.
The normal TDD process is:
1. add one test
2. make it (and all others) pass
3. maybe refactor so code is sane
4. back to 1, unless you're done.
Even if you know exactly how the software is going to work, how would you know if your test cases are written correctly without having the software to run them against? For that reason alone, the whole idea of TDD doesn't even make sense to me.
The reason is, the tests and the code are symbiotic, your tests prove the code works and your code proves the tests are correct. TDD guarantees you always have both of those parts. But granted it is not the only way to get those 2 parts.
You can still throw into the mix times when a bug is present, and it is this symbiotic relationship that helps you find the bug fast, change the test to exercise the newly discovered desired behaviour, see the test go red for the correct reason and then make the code tweak to pass the test (and see all the other tests still pass).
What I find is much much better approach is what I call "detached test development" (DTD). The idea is: 2 separate teams get the requirements; one team writes code, the other write tests. They do not talk to each other! Fist when a test is not passed, they have to discuss: is the requirement not clear enough? What is the part that A thought about, but not B? Assignment of tests and code can be mixed, so a team makes code for requirements 1 through 100, and tests for 101 to 200, or something like that. I had very very good results with such approach.
TDD is a feedback cycle, you write small increments of tests before writing a small bit of a code. You don't write a bunch of tests upfront, that'd be silly. The whole point is to integrate small amounts of learning as you go, which help guide the follow-on tests, as well as the actual implementation, not to mention questions to need to ask the broader business.
Your DTD idea has been tried a lot in prior decades. In fact, as a student I was on one of those testing teams. It's a terrible idea, throwing code over a wall like that is a great way to radically increase the latency of communication, and to have a raft of things get missed.
I have no idea why there's such common misconceptions of what TDD is. Maybe folks are being taught some really bad ideas here?
100%. Metrics of quality are really really hard to define in a way that are both productive and not gamified by engineers.
> What I find is much much better approach is what I call "detached test development" (DTD)
I'm a test engineer and some companies do 'embed' an SDET like the way you mention within a team - it's not quite that clear cut, they can discuss, but it's still one person implementing and another testing.
I'm always happy to see people with thoughts on testing as a core part of good engineering rather than an afterthought/annoyance :)
This feels a bit like when you write a layer of encapsulation to try to make a problem easier only to discover that all of the complexity is now in the interface. Isn't converting the PO's requirements into good, testable requirements the hard technical bit?
Therefore, TDD's secret sauce is in concretely forcing developers to think through requirements, mental models etc. and quantify them in some way. When you hit a block, you need to ask yourself whats missing, then figure out, and continue onward, making adjustments along the way.
This is quite malleable to unknown unknowns etc.
I think the problem is most people just aren't chunking down the steps of creating a solution enough. I'd argue that the core way of approaching TDD fights most human behavioral traits. It forces a sort of abstract level of reasoning about something that lets you break things down into reasonable chunks.
TDD is not a silver bullet, it's one tool among many.
So much of this is because TDD has become synonymous with unit testing, and specifically solitary unit testing of minimally sized units, even though that was often not the original intent of the ideators of unit testing. These tests are tightly coupled to your unit decomposition. Not the unit implementation (unless they're just bad UTs), but the decomposition of the software into which units/interfaces. Then the decomposition becomes very hard to change because the tests are exactly coupled to them.
If you take a higher view of unit testing, such as what is suggested by Martin Fowler, a lot of these problems go away. Tests can be medium level and that's fine. You don't waste a bunch of time building mocks for abstractions you ultimately don't need. Decompositions are easier to change. Tests may be more flaky, but you can always improve that later once you've understood your requirements better. Tests are quicker to write, and they're more easily aligned with actual user requirements rather than made up unit boundaries. When those requirements change, it's obvious which tests are now useless. Since tests are decoupled from the lowest level implementation details, it's cheap to evolve those details to optimize implementation details when your performance needs change.
This is a trouble I often see expressed about static types. And it’s an intuition I shared before embracing both. Thing is, embracing both helped me overcome the trouble in most cases.
- If I have a type interface, there I have the shape of the definition up front. It’s already beginning to help verify the approach that’ll form within that shape.
- Each time I write a failing test, there I have begun to define the expected behavior. Combined with types, this also helps verify that the interface is appropriate, as the article discusses, though not in terms of types. My point is that it’s also verifying the initial definition.
Combined, types and tests are (at least a substantial part of) the definition. Writing them up front is an act of defining the software up front.
I’m not saying this works for everyone or for every use case. I find it works well for me in the majority of cases, and that the exception tends to be when integrating with systems I don’t fully understand and which subset of their APIs are appropriate for my solution. Even so writing tests (and even sometimes types for those systems, though this is mostly a thing in gradually typed languages) often helps lead me to that clarity. Again, it helps me define up front.
All of this, for what it’s worth, is why I also find the semantics of BDD helpful: they’re explicit about tests being a spec.
This is true, and I think that's why TDD is a valuable exercise to disambiguate requirements.
You don't need to take an all/nothing approach. Even if you clarify 15-20% of the requirements enough to write tests before code, that's a great place to begin iterating on the murky 80%.
Because for years people have practice with defining software iteratively, whether by choice or being forced by deadlines and agile.
That doesn't inherently make one or the other harder, it's just another familiarity problem.
TDD goes nicely with top-down design using something like Haskell's undefined to stub out functionality that typechecks and it's where clauses.
myFunction = haveAParty . worldPeace . fixPoverty $ world
where worldPeace = undefined
haveAParty = undefined
fixPoverty = undefined
By the time I've reached a point where the feature can actually be tested, I end up with a pretty good skeleton of what tests should be written.
There's a hidden benefit to doing this, actually. It frees up your brain from keeping that running tally of "the feature should do X" and "the feature should guard against Y", etc. (the very items that go poof when you get distracted, mind you)
Try it. Write a test for 1, and an implementation which passes that test then for 2, and so on.
Bellow is something written without any TDD (in Java)
private static String convert(int digit, String one, String half, String ten) {
switch(digit) {
case 0: return "";
case 1: return one;
case 2: return one + one;
case 3: return one + one + one;
case 4: return one + half;
case 5: return half;
case 6: return half + one;
case 7: return half + one + one;
case 8: return half + one + one + one;
case 9: return one + ten;
default:
throw new IllegalArgumentException("Digit out of range 0-9: " + digit);
}
}
public static String convert(int n) {
if (n > 3000) {
throw new IllegalArgumentException("Number out of range 0-3000: " + n);
}
return convert(n / 1000, "M", "", "") +
convert((n / 100) % 10, "C", "D", "M") +
convert((n / 10) % 10, "X", "L", "C") +
convert(n % 10, "I", "V", "X");
}If you have "a growing collection of broken tests", that's not TDD. That's "they told us we have to have tests, so we wrote some, but we don't actually want them enough to maintain them, so instead we ignore them".
Tests help massively with iterating a design on a partly-implemented code base. I start with the existing tests running. I iterate by changing some parts. Did that break anything else? How do I know? Well, I run the tests. Oh, those four tests broke. That one is no longer relevant; I delete it. That other one is testing behavior that changed; I fix it for the new reality. Those other two... why are they breaking? Those are showing me unintended consequences of my change. I think very carefully about what they're showing me, and decide if I want the code to do that. If yes, I fix the test; if not, I fix the code. At the end, I've got working tests again, and I've got a solid basis for believing that the code does what I think it does.
> The obvious exception to this, where I still use TDD, is when implementing a well defined spec.
From my understanding (and experience), TDD is quite the opposite. It's most useful when you don't have the spec, don't have clue how software will work in the end. TDD creates the spec, iteratively.
1. Hack in what I want in some exploratory way
2. Write good tests
3. Delete my hacks from step 1, and ensure all my new tests now fail
4. Re-implement what I hacked together in step 1
5. Ensure all tests pass
We typically write acceptance tests, and they have been helpful either early on or later in our product development lifecycle.
Even if software isn't defined upfront, the end goal is likely defined upfront, isn't it? "User X should be able to get data about a car," or "User Y should be able to add a star ratings to this review," etc.
If you're building a product where you're regularly throwing out large parts of the UI / functionality, though, I suppose it could be bad. But as a small startup, we have almost never been in that situation over the many years we've been in business.
In many ways I guess I lean maximalist in my practices, and find it helpful, but I'd readily concede that the maximalist advocates are annoying and off-putting. I once had the opportunity to program with Ward Cunningham for a weekend, and it was a completely easygoing and pragmatic experience.
But spikes are written to be thrown away. You never put them into production. Production code is always written against some preexisting test, otherwise it is by definition broken.
I'm not sure what you mean by this. Why are the tests you're writing not "adequate" for the code you're testing?
If I read into this that you're using code coverage as a metric -- and perhaps even striving for as close to 100% as possible -- I'd argue that's not useful. Code coverage, as a goal, is perhaps even harmful. You can have 100% code coverage and still miss important scenarios -- this means the software can still be wrong, despite the huge effort put into getting 100% coverage and having all tests both correct and passing.
Functionality based on a set of initial specs and a hazy understanding of the actual problem you are trying to solve might on the other hand might not be worth investing in protecting.
It might work if you are starting on some new, relatively self-contained feature...
And that is, write code, chuck it away, start again.
Prototype your feature without TDD. Then chuck it away and build it again with TDD.
My guess is by doing so code quality and reduced technical debt pay more than what is lost in time.
Very few companies work like this I imagine: None that I have worked for.
Since keyboard typing is a short part of software development it is probably a great use of time and could catch more bugs and design quirks early on when they cost $200/h instead of $2000/h.
Another partly orthogonal issue is that design is important for some problems, and you don't usually reach a good design by chipping away at a problem in tiny pieces.
TDD fanatics insist that it works for everything. Do I believe them that it improved the quality of their code? Absolutely; I've seen tons of crap code that would have benefited from any improvement to the design, and forcing it to be testable is one way to coerce better design decisions.
But it really only forces the first-order design at the lowest level to be decent. It doesn't help at all, or at least not much, with the data architecture or the overall data flow through the application.
And sometimes the only sane way to achieve a solid result is to sit down and design a clean architecture for the problem you're trying to solve.
I'm thinking of one solution I came up with for a problem that really wasn't amenable to the "write one test and get a positive result" approach of TDD. I built up a full tree data structure that was linked horizontally to "past" trees in the same hierarchy (each node was linked to its historical equivalent node). This data structure was really, really needed to handle the complex data constraints the client was requesting. As yes, we pushed the client to try to simplify those constraints, but they insisted.
The absolute spaghetti mess that would have resulted from TDD wouldn't have been possible to refactor into what I came up with. There's just no evolutionary path between points A and B. And after it was implemented and it functioned correctly--they changed the constraints. About a hundred times. I'm not even exaggerating.
Each new constraint required about 15 minutes of tweaking to the structure I'd created. And yes, I piled on tests to ensure it was working correctly--but the tests were all after the fact, and they weren't micro-unit tests but more of a broad system test that covered far more functionality than you'd normally put in a unit test. Some of the tests even needed to be serialized so that earlier tests could set up complex data and states for the later tests to exercise, which I understand is also a huge No No in TDD, but short of creating 10x as much testing code, much of it being completely redundant, I didn't really have a choice.
So your point about the design changing as you go is important, but sometimes even the initial design is complex enough that you don't want to just sit down and start coding without thinking about how the whole design should work. And no methodology will magically grant good design sense; that's just something that needs to be learned. There Is No Silver Bullet, after all.
True, but… you can still design the architecture, outlining the solution for the entire problem, and then apply TDD. In this case your architectural solution will be an input for low level design created in TDD.
Regulations are always ambiguous, standards are never followed, and widely implemented standards are never implemented the way the document tells.
You will probably still gain productivity by following TDD for those, but your process must not penalize too much changes in spec, because it doesn't matter if it's written in Law, what you read is not exactly what you will create.
- The Tao of Programming (1987)
I created dozens of “edge case” sample spreadsheets with horrible things in them like Bad Strings in every property and field. Think control characters in the tab names, RTL Unicode in the file description, etc…
I found several bugs… in Excel.
As a made up example. The "what" of the program is to take in a bunch of transactions and emit daily summaries. That's a straight forward "what". It however leaves tons of questions unanswered. Where does the data come from and in what format? Is it ASCII or Unicode? Do we control the source or is it from a third party? How do we want to emit the summaries? Printed to a text console? Saved to an Excel spreadsheet? What version of Excel? Serialized to XML or JSON? Do we have a spec for that serialized form? What precision do we need to calculate vs what we emit?
So the real "what" is: take in transaction data encoded as UTF-8 from a third party provider which lives in log files on the file system without inline metadata then translate the weird date format with only minute precision and lacking an explicit time zone and summarize daily stats to four decimal places but round to two decimal places for reporting and emit the summaries as JSON with dates as ISO ordinal dates and values at two decimal places saved to an FTP server we don't control.
While waiting for all that necessarily but often elided detail you can either start writing some code with unit test or wait and do no work until you get a fully fleshed out spec that can serve as the basis for writing tests. Most organizations want to start work even while the final specs of the work are being worked on.
Often times you write to find what you want to accomplish. It sounds backwards, perhaps it is backwards, but it's also very human. Without something to show the user, they often have no idea what they want. In fact, people are far better at telling you what's wrong with what's presented to them then enumerating everything they want ahead of time.
TDD is great but also completely useless for sussing requirements out of users.
Maybe we need to learn how to delete stuff that doesn't make sense.
Get rid of broken test. Get rid of incorrect documentation.
Don't be afraid to delete stuff to improve the overall program.
The result was that 1) the devs were exceptionally happy, 2) the TL was mostly happy, except with some of the extra forced work he created for himself as the bottleneck, 3) the project took longer than expected, and 4) the code was SOOOOO readable but also very inefficient. We realized during the project that forcing unit tests for literally everything was also forcing a breaking up of methods & functions into much smaller discrete pieces than would have been optimal from both performance & extensibility perspectives.
It wasn't the last TDD project we ran, but we were far more flexible after that.
I had one other "science project" while managing that team, too. It was one where we decided to create an architect role (it was the hotness at that time), and let them design everything from the beginning, after which the dev team would run with it using their typical agile/sprint methodology. We ended up with the most spaghetti code of abstraction upon abstraction, factories for all sorts of things, and a codebase that became almost unsupportable from the time it was launched, necessitating v2.0 be a near complete rewrite of the business logic and a lot of the data interfaces.
The lessons I learned from those projects was that it's important to have experienced folks on every dev team, and that creating a general standard that allows for flexibility in specific architectural/technical decisions will result in higher quality software, faster, than if one is too prescriptive (either in process or in architecture/design patterns). I also learned that there's no such thing as too much SQA, but that's for a different story.
I'm not quite sure what is right or wrong about my approach, but I do find the code tends to work and work reliably once it compiles and the tests pass.
Very few critics notice this.
I write tdd when doing advent of code. And it’s not that I set out to do it or to practice it or anything. It just comes very natural to small, well defined problems.
Nobody out there is writing all their tests up front.
TDD is an iterative process, RED GREEN REFACTOR.
- You write one test.
- Write JUST enough code to make it pass.
- Refactor while maintaining green.
- Write a new test.
- Repeat.
I don't want this to come off the wrong way but what you're describing shows you are severely misinformed about what TDD actually is or you're just making assumptions about something based on its name and nothing else.
Everyone understands the idea, it's just a massive time sink for no more benefit than a test-after methodology provides.
> - Write JUST enough code to make it pass.
Those two steps aren't really trivial. Even just writing the single test might require making a lot of design decisions that you can't really make up-front without the code.
> - You write one test
Easier said than done. Say your task is to create a low level audio mixer which is something you've never done before. Where do you even begin? That's the hard part.
Some other commenters here have pointed out that exploratory code is different from TDD code, which is a much better argument then what you made here imo.
> I don't want this to come off the wrong way but what you're describing shows you are severely misinformed about what TDD actually is or you're just making assumptions about something based on its name and nothing else.
Instead of questioning the OP's qualifications, perhaps you should hold a slightly less dogmatic opinion. Perhaps OP is familiar with this style of development, and they've run into problem firsthand when they've tried to write tests for an unknown problem domain.
I've seen loads of examples where the tests haven't been updated in years to take account of new functionality. When that happens you aren't really doing TDD anymore.
> This has happened to me several times while writing this book. I would get the code a bit twisted. “But I have to finish the book. The children are starving, and the bill collectors are pounding on the door.”
Instead of realizing that Kent Beck stretched out an article-sized idea into an entire book, because he makes his money writing vague books on vague "methodology" that are really advertising brochures for his corporate training seminars, people actually took the thing seriously and legitimately believed that you (yes, you) should write all code that way.
So a technique that is sometimes useful for refactoring and sometimes useful for writing new code got cargo-culted into a no-exceptions-this-is-how-you-must-do-all-your-work Law by people that don't really understand what they are doing anymore or why. Don't let the TDD zealots ruin TDD.
A simple idea ("hey, I was facing a tricky problem and this new way of approaching it worked for me. Maybe it will help you too?") mutates into a blanket law ("this is the only way to solve all the problems") and then pointy-haired folks notice the trend and enshrine it into corporate policy.
But Fred Brooks was right: there are no silver bullets. Do what works best for you/your team.
There are really two questions lurking here:
How much ground should each test cover?
How many intermediate stages should you go through as you refactor?
You could write the tests so they each encouraged the addition of a single line of logic and a handful of refactorings. You could write the tests so they each encouraged the addition of hundreds of lines of logic and hours of refactoring. Which should you do?
Part of the answer is that you should be able to do either. The tendency of Test-Driven Developers over time is clear, though - smaller steps. However, folks are experimenting with driving development from application-level tests, either alone or in conjunction with the programmer-level tests we've been writing.One thing I liked specifically was his emphasis on the idea that you can use TDD to adjust the size of your steps to match the complexity of the code. Very complex? Small steps with many tests, maybe using the minimal code-approach to get things going. Simple/trivial? A single test and the solution immediately with no awkward step in between.
I wonder how much methodologies, books are written with the same banal driver. It is somebody's livelihood and they don't pay writers to stop middle of it because they realize its flawed.
I once found a book on triangular currency arbitrage or something like that at my library. It was 4000 pages long and the book was heavy. The book would ramble on in languages that made it difficult to follow and would be filled with mathmetical notations to the brim which really offered no value because the book was written in the 70s and it no longer offered any executable knowledge. But finance schools swear by it and speaking out would trigger a lot of people.
TDD is a cult. Science is also a cult in that manner, it rejects the existence of what it cannot measure and it gangs up on those that go against it.
Short of looking over every developer's shoulder, how do you actually know the extent to which TDD is being practiced as prescribed? (red, green, refactor) Code review? How do you validate your code reviewer's ability to identify TDD code? What if someone submits working tested code; but, you smell it's not TDD, what then? Tell them to pretend they didn't write it and start over with the correct process? What part of the development process to you start to practice it? Do you make the R&D people do it? Do you make the prototypers do it? What if the prototype got shipped into production?
Because of all this, even if the programmers really do write good TDD code, the business people still can't trust you, they still have to QA test all your stuff. Because they can't measure TDD, they have no idea when you are doing it. Maybe you did TDD for the last release; but, are starting to slip? Who knows, just QA the product anyways.
I like his characterization of TDD as a technique. That's exactly what it is, a tool you use when the situation calls for it. It's a fantastic technique when you need it.
In theory, if TDD really reduces the number of bugs and speeds up development you would see if reflected in those higher level metrics that impact the customer.
The issue is that many TDD diehards believe that bugs and delays are made by coders who did not properly qualify their code before they wrote it.
In reality, bugs and delays are a product of an organization. Bad coders can write bad tests that pass bad code just fine. Overly short deadlines will cause poor tests. Furthermore, many coders reply that they have trouble with the task-switching nature of TDD. To write a complex function, I will probably break it out into a bunch of smaller pure functions. In TDD that may require you to either: 1. Write a larger function that passes the test and break it down. 2. Write a test that validates that the larger function calls other functions and then write tests that define each smaller function.
The problem with these flows is that 1: Causes rework and 2 ends up being like reading a book out of order, you may get to function 3 and realize that function 2 needed additional data and now you have to rewrite your test for 2. Once again rework. I'm sure there are some gains in some spaces but overall it seems that the rework burns those gains off.
I don't know that Pivotal (in particular) does pair programming so that TDD is followed, I do know that they (did) follow TDD and do everything via pair programming. I'm agnostic as to whether it's a good idea generally, it's not how I want to live but I've had a few associates who really liked it.
This kind of situation matches my experience. It was cemented when I worked with a guy who was a zealot about TDD and the whole Clean Code cabal around Uncle Bob. He was also one of the worst programmers I have worked with.
I don't mean to say that whole mindset is necessarily bad. I just found that becoming obsessed with it isn't sufficient. I've worked with guys who have never written a single test yet ship code that does the job, meets performance specs, and runs in production environments with no issues. And I've worked with guys who get on their high horse about TDD but can't ship code on time, or it is too slow, and it has constant issues in production.
No amount of rationalizing about the theoretical benefits can match my experience. I do not believe you can take a bad programmer and make them good by forcing them to adhere to TDD.
But that's because he deliberately does it in a stupid way to make TDD look bad, just like the linked article does with his "quicksort test". But that's beside the point - of course a stupid person would write a stupid test, but that same stupid person would write a stupid implementation, too... but at least there would be a test for it.
I'm curious to unpack this a bit. I'm curious what other tools people use other than testing programatic testing; programatic testing seems to be the most efficient, especially for a programmer. I'm also maybe a bit stuck on the binary nature of your statement. You know developers who've never let a bug or performance issue enter production(with or without testing)?
Peter Norvig's solution has one central precept that is not something that you would arrive at by an incremental approach.
But I wonder if this incrementalism is essential for TDD.
I think TDD has a lot to offer, but don't go in for the purist approach. I like Free Software but don't agree with Stallman. It's the same thing.
The author takes a well reasoned, mature, productive, engineering focused approach, like the majority of people should be doing. We shouldn't be applying the pure views directly, we should be informed by them and figure out what we can learn for our own work.
This took off like wildfire probably for the same reason that we see extreme social movements/politics take off. People love purity because it's so clean and tidy. Nice easy answers. If I write a test for everything something good will emerge. No need for judgement and hand wringing.
But the thing is that I think Kent Beck got caught up in this himself and forgot the original intention. I could be wrong but it seems like that.
[0] https://blog.cleancoder.com/uncle-bob/2016/10/26/DijkstrasAl...
So thanks for the link, I guess. I’ll keep this as ammunition for the next time someone quotes Uncle Bob.
The code is definitely a horrow show.
I would just like to compare them. I too find Uncle Bobs “clean code” book very much overrated.
My understanding of the “design” aspect of TDD is, that you start from client code and create the code that conforms to your tests. Too often I worked in a team with other developers and I wanted to use what they wrote, and they somehow coded what was part of the spec, but it was unusable from my code. Only because I was able to change their code (most often the public API) I was able to use it.
The whole interface hazard evaporates if you write the tests in the same scope as the implementation, so the tests can access internals directly without changing the interface. E.g. put them in the same translation unit for C++. Have separate source files only containing API tests as well if you like. Weird that's so unpopular.
There's also a strong synergy with design by contract, especially for data structures. Put (expensive) pre/post and invariants on the methods, then hit the edge cases from unit tests, and fuzz the thing for good measure. You get exactly the public API you want plus great assurance that the structure works, provided you don't change semantics when disabling the contract checks.
The post is weird, I agree with almost everything in the first half and disagreed with most of the second part.
What makes TDD hard for integration testing is that there are no simple readymade tools similar to XUnit frameworks and people need to build their own tools and make them fast.
Lots of shops claim to do TDD, but in practice what they mean is that they sometimes write unit tests. I've literally never encountered it outside of toy examples and small academic exercises.
Where is the software successfully developed according to TDD principles? Surely a superior method of software development should produce abundant examples of superior software? TDD has been around for a pretty long time.
1. No bug is ever fixed before we have at least one failing test. Test needs to fail, and then turn green after bugfix. [1]
2. No new code ever committed without a test specifically testing the behavior expected from the new code. Test needs to fail, and then turn green after the new code.
3. If we're writing a brand new service/product/program etc, we first create a spec in human language. Turn the spec into tests. This doesn't mean, formally speaking "write tests first, code later" because we do write tests and code at the same. It's just that everything in the spec has to have an accompanying test, and every behavior in the code needs to have a test. This is checked informally.
As they say, unittests are also code, and all code has bugs. In particular, tests have bugs too. So, this framework is not bullet-proof either, but I've personally been enjoying working in this flow.
[1] The only exception is if there is a serious prod incident. Then we fix the bug first. When this happens, I, personally, remove the fix, make sure a test fails, then add the fix back.
The times I actually use TDD are basically limited to really tricky problems I don't know how to solve or break down or when I have a problem with some rough ideas for domain boundaries but I don't quite know where I should draw the lines around things. TDD pulls these out of thin air like magic and they consistently take less time to reach than if I just sit there and think about it for a week by trying different approaches out.
Then we have TDD itself, there are at least two different schools of TDD. What the author calls “maximal TDD” sounds like the mockist school to me. Would his criticism also apply to the classical school? I’m sincerely curious.
If we don’t have a common ground, communication becomes really difficult. Discussion and criticism becomes unfruitful.
If you are lucky enough to be writing code in a way that each unit is absolutely clear before you start working, awesome you got it. But in business-logic-land things rarely end up this clean.
Personally, I program the happy path then write tests and use them to help uncover edge cases.
This approach resonates with me as well. I would add that writing tests when investigating bugs or deviations from expected behavior is also useful.
In this day and age of vastly distributed systems where distribution and re-distribution is relatively cheap we can afford to be a little less obsessive. Many exceptions still exist, of course, I would think that the teams developing my car's control system might warm up to TDD a bit more than someone putting together a quickie web app.
The funny thing is I end spending just as much time trying to debug or figure out the other layers, looking at you AWS IAM, that I don't feel I am that much more productive, I've just taken what my code needed to do and scatterred it to the 4 winds. Now instead of dealing with an OS and the code I'm fighting with docker, and a cloud service, and permissions and network and a dozen other things.
Honestly this feels like the OOP hype era of Object Database and Java EE all over again, just this time substitute OOP for tooling.
My view is that TDD is great for non-explorative coding. So data science -> way less TDD. Web APIs -> almost always TDD.
That said, one of the things I think the vast majority of the leans anti-TDD crowd misses is that someone else on the team is picking up the slack for you and you never really appreciated it. I've joined too many teams, even great ones, where I needed to make a change to an endpoint and there were no functional or integration tests against it. So now I'm the one that is writing the tests you should have written. I'm the one that has to figure out how the code should work, and I'm the one that puts it all together in a new test for all of your existing functionality before I can even get started.
Had you written them in the first place I would have had a nice integration test that documents the intended behaviour and guards against regressions.
Basically I'm carrying water for you and the rest of the team that has little to do with my feature.
Now there are some devs out there that don't need TDD to remember to write tests, but I don't know many of them and they're usually writing really weird stuff (high performance or video or whatever).
But I have stopped concerning myself with changing other peoples minds on this. Some people have just naturally reactive minds and TDD isn't what they like so they don't do it.
If you know exactly what you need upfront, you can simply start coding, adding a sprinkling of acceptance tests to help catch mistakes. No need for TDD in that case.
We have so many things on our tech belt, like clean architecture or x pattern. This is just another tool, and I think it helps especially in building complex software.
Just be practical and don’t try to be “the 100%er” who is super rigid about things. Go into everything with a 80/20 mindset. If this is something mission critical and needs to be as dependable as possible, then use the tools best suited for it. If you’re literally putting buttons on the screen which Product is going to scrap in two weeks, maybe use TDD for the code responsible for dynamically switching code based on Product mindset that week.
Pluses were refactoring was easy and I had confidence that the system would work well at the end.
Minuses were it took a lot longer to write and I had to throw away a lot of code and tests as my understanding increased. It slowed down exploration immensely. Also, factoring the code to be completely testable led to some dubious design decisions that I wouldn't make if I wasn't following a pure TDD approach.
On balance I decided it wasn't a generally good way to write code, although I guess there may be some circumstances it works well for.
Tests in general aren't something I regularly use and a lot of TDD feels somewhat insane to me. You can write all the tests you want ahead of time but until the rubber meet the road it's a lot of wishful thinking in my experience. Also it makes refactoring hell since you often have to rewrite all the tests except the ones at the top level and sometimes even those if you change enough.
I believe tests can work, I've just never really seen them work well expect for very well defined sets of functionality that are core to a product. For example I worked at a company that had tests around their geofencing code. Due to backfilling data, zones being turned on/off by time, exception zones within zones, and locations not always being super accurate, the test suite was impressive. Something like 16 different use cases it tested for (to determine if a person was in violation for a given set of locations, for a given time). However, at the same company, there was a huge push to get 80%+ code coverage. So many of our tests were brittle that we ended up shipping code regularly with broken tests because we knew they couldn't be trusted. The tests that were less brittle often had complicated code to generate the test data and the test expectations (who tests the tests?). In my entire time at that company we very rarely (I want to say "never" but my memory could be wrong) had a test break that actually was pointing at a real issue, instead the test was just brittle or the function changed and someone forgot to update the test. If you have to update the test every time you touch the code it's testing... well I don't find that super useful, especially coupled with it never catching real bugs.
In a lot of TDD/Tests in general tutorials I've seen they make it seem all roses and sunshine but their examples are simple and look nothing like code I've seen in the wild. I'd be interested in some real-world code and the tests as the evolved over time.
All that said, I continue to be at least interested in tests/TDD in the hopes one day it will "click" for me and not see just like a huge waste of time.
TDD is miserable for code that is dependent on data or external resources (especially stateful resources). In most cases, writing "integration" tests feels like its not worth the effort given all the code that goes into managing those external resources. Yes, I know about mocking. But mocking frameworks are: 1 - not trivial to use correctly, and 2 - often don't implement all the functionality you may need to mock.
When debugging, I'll also turn failure cases into unit tests and add them to the CI. The cost to write the test has already been paid in this case, so using them to catch regressions is all-upside.
System tests are harder to do (since they require reasoning about the entire program rather than single functions) but in my experience are the most productive, in terms of catching the most bugs in least time. Certainly every minute spent writing a framework for mocking inputs into unit tests should probably have been spent on system testing instead.
IMO functional and integration testing should be where effort is spent first and foremost. If there are resources beyond that, then do finer grained unit testing, but even then closely tracking the amount of time spent on all the scaffolding necessary for it vs the benefits for what gets tested.
I find that TDD is very well fit to fix the expectations from the external dependencies.
Of course, when such dependency is extensive, like an API wrapper, then writing equally extensive tests would be redundant. Even then, the core aspects of the external dependencies should be fixed testably.
Testing is a balance game, even with TDD. The goal is to increase certainty under dynamic changes and increasing complexity.
High-discipline meaning, it entirely depends on highly competent developers (able to produce clean code, deep understanding of programming), rigorously disciplined out of pure intrinsic motivation, and even able to do this under peak pressure.
Which is not at all how most software is built today. Specs are shit so you gradually find out what it needs to do. Most coders are bread programmers and I don't mean that in any insulting way. They barely get by getting anything to work. Most projects are under very high time pressure, shit needs to get delivered and as fast as possible. Code being written in such a way that it's not really testable. We think in 2 week sprints which means anything long term is pretty much ignored.
In such an environment, the shortest path is taken. And since updating your tests is also something you can skip, coverage will sink. Bugs escape the test suite and the belief in the point of TDD crumbles. Like a broken window effect.
My point is not against TDD. It's against ivory tower thinking that does not take into account a typical messy real world situation.
I've noticed a major shift in the last decade. We used to think like this, in TDD, in documenting things with UML, in reasoning about design patterns. It feels like we lost it all, as if it's all totally irrelevant now. The paradigm is now hyper speed. Deliver. Fast. In any way you can.
This short-sighted approach leading to long term catastrophe? Not even that seems to matter anymore, as the thing you're working on has the shelf life of fish. It seems to be business as usual to replace everything in about 3-5 years.
The world is really, really fast now.
I see TDD is REPL driven development for languages without a REPL. It allows you to play with your code in a tighter feed back loop, than you generally have without it.
I see unit tests as a tool to be used where it makes sense and I use it a lot. It is true that testable code is better. Testability should be a factor when selecting tech.
Unit tests serve multiple purposes. Number 1 is to have a way for you to play around with your design. Then it also can document requirements. I then lastly serves as a vehicle for you to prove something about your design, typically the fulfillment of a requirement, or the handling of some edge case, etc. This last part is what people unfortunately mostly refer to as a test.
TDD says that you should write your tests before you even have a design, one-by-one typically, adding in more functionality and design as you go. You will end up with crap code. If you do not throw the first iteration away then you will commit crappy code.
Most people naturally find that an iterative cycle of design and test code works the best and trying to sell TDD to them is a harmful activity, because it yields no benefits and might actually be a big step backwards.
Some hidden/unexpected side effects of TDD include the often extremely high cost of maintaining the tests once you get past the simple cases, the subtle incentive to not think too holistically about certain things, and the progression as a developer in which you naturally improve and stop writing the types of bugs that basic tests are good at catching but which you continue to write anyway (a real benefit, sure, but one that further devalues the tests). The cost of creating a test that would have caught the really "interesting" bugs is often exorbitant, both up front and to maintain.
The closest thing I've encountered to a reliable exception is that having e.g. a comprehensive suite of regression tests is really great when you are doing a total rewrite of a library or critical routine. But even that doesn't necessarily mean that the cost of creating and maintaining that test suite was worth it, and so far every time I've encountered this situation, it's always been relatively easy to amass a huge collection of real world test data, which not only exercises the code to be replaced but also provides you a high degree of confidence that the rewrite is correct.
What it helps with a lot is improving your individual programming skills. So I recommend TDD to everyone, who never did it in practise (best case on a legacy code base) - if not to improve the code itself, then just to LEARN how it could improve your code.
It helped me to understand, why IoC and Dependency Injection are a thing and when to use it. Writing "testable" code is important, while writing real tests may be not as important, as long as you do not plan to have along running project or do a major refactoring. If you ARE planning a major refactoring, you should first write the tests to ensure you don't break anything, though ;)
What I also would recommend is having a CI / Build-Environment supporting TDD, SonarQube and CodeCoverage - not trying to establish that afterwards... Being able to switch to TDD is also a very neat way to get a nice CI setup.
My feeling is, that my programming and deployment skills improved best, when I did one of my personal pet projects strictly test driven with automated CI and found out about the things in TDD and CI, I really need to care about.
During class, the teacher taught us TDD, using the Test-Code-Refactor loop. Then he wanted us to write an implementation of Conway's Game of Life using TDD. As the students were doing it, he was doing it as well.
After the lesson but before the exercise, I thought "This looks tedious and looks like it would make coding take far longer than necessary" and just wrote the Game first, then wrote a couple dozen tests. Took me probably about 45 minutes.
At that point, I looked up on the projector and saw the teacher had barely done much more than having a window, a couple buttons, and some squares drawn on it, and a dozen tests making sure the window was created, buttons were created, clicking the button called the function, and that the calls to draw squares succeeded.
What really bothers me about "true" TDD (and TFA points this out), is that if you're writing bare minimum code to make a unit test pass, then it will likely be incorrect. Imagine writing an abs() function, and your first test is "assert (abs(-1) == 1)". So you write in your function "if (i == -1) return 1". Congrats, you wrote the bare minimum code. Tadaa! TDD!
Define the systems that make up your software, the classes that make up the systems, and then the functions they'll use to talk to each other. Once it works on paper, start coding. If the design is good, the code should be so brain dead simple to write that a monkey could write it.
I find doing this I do end up with long call stacks, because each module will do something, then pass the data on like an assembly line. In each step my functions are super short and I don't find it worth writing a test for 3 lines of code that I can tell is correct at a glance. For those few meaty functions that do more heavy logic, I will write tests for, though.
The blurb from the latter:
> In this talk we will look at the key Fallacies of Test-Driven Development, such as 'Developers write Unit Tests', or 'Test After is as effective as Test First' and explore a set of Principles that let us write good unit tests instead. Attendees should be able to take away a clear set of guidelines as to how they should be approaching TDD to be successful. The session is intended to be pragmatic advice on how to follow the ideas outlined in my 2013 talk "TDD Where Did it All Go Wrong"
The talks focus on reasons to avoid slavish TDD and advocate for the benefits of judiciously applying TDD's originating principles.
It does for nice conference talks though.
It is also no surprise these terms have mostly to do with IT or related consulting and not really about engineering endeavor. In my first hand experience when I worked at engineering department there was whole lot of work done with almost non-existent buzzword bullshit. And later on with merger etc it is now an IT department so there is endless money on process training, resources, scrum masters and so on but little money is left for half decent computer setup.
Outside work I have seen this in my cooking, first time new dish is a hustle but in future iterations I would create little in-brain task list tickets for my own processing. Doing this in jackasstic consulting framework way would turn 1 hr worth of butter chicken recipe into 1 month work of taste feature implementation sprints.
Basically, we know that we're going to have to write tests when we're done, so we are sure to develop it in a way that is going to be (relatively) easy to write accurate and comprehensive tests for.
1. Many companies are agile, and the requirements constantly change, which makes implementing TDD even harder.
2. TDD does not bring enough value to justify the investment of time (for writing & maintaining the test suites), the benefits are negligible, and the changes are often.
3. Everything is subjective [1], and there's no reason to have such strongly held opinions about the "only right way to write code" when people write software in a way that is efficient for their companies.
[1] https://vadimkravcenko.com/shorts/software-development-subje...
If it's C libraries that fiddle that do lots of munging of variables, like positioning UI or fiddling with data structures... then yes, totally, it has well-defined requirements that you can assert in tests before you write it.
If it's like React UI code, though, get out of here. You shouldn't even really be writing unit tests for most of that (IMO), much less blocking on writing it first. It'll probably change 20 times before it's done anyway; writing the tests up from is going to just be annoying.
I smell a circular argument but can’t quite put my finger on it.
> If it doesn’t work for you, you’re doing it wrong
Ah. Start with a nigh unattainable and self-justifying moral standard, sell services to get people there, and treat any deviation as heresy. How convenient. Reminds me of Scrum evangelists. Or a cult.
TDD is a great tool for library-level code and in cases where the details are known upfront and stable.
But I can’t get it to work for exploratory work or anything directly UI related. It traps me in a local optimum and has draws my focus to the wrong places.
I have a feeling this is where TDD loses out the most
Most of the early learning was by pairing with people who already knew how to do it, working on a codebase using it. People learn it easily and fast that way.
But that doesn't scale, and at some point people started trying to do it having only read about it. It doesn't surprise me at all that that has often been unsuccessful.
In some of my projects.
Like any technique, it's not dogma; just another tool.
One of my biggest issues with "pure" TDD, is the requirement to have a very well-developed upfront spec; which is actually a good thing.
sometimes.
I like to take an "evolutionary" approach to design and implementation[0], and "pure" TDD isn't particularly helpful, here.
[0] https://littlegreenviper.com/miscellany/evolutionary-design-...
Also, I do a lot of GUI and device interface stuff. Unit tests tend to be a problem, in these types of scenarios (no, "UI unit testing" is not a solution I like). That's why I often prefer test harnesses[1]. My testing code generally dwarfs my implementation code.
[1] https://littlegreenviper.com/miscellany/testing-harness-vs-u...
Here's a story on how I ran into an issue, early on[2].
[2] https://littlegreenviper.com/miscellany/concrete-galoshes/#s...
Can you expand on what you mean by "a very well-developed upfront spec"? Because that doesn't sound at all like TDD as i know it.
I work on software that takes in prices for financial instruments and does calcualtions with them. initially there was one input price for everything. A while ago, a requirement came up to take in price quotes from multiple authorities, create a consensus, and use that. I had a chat with some expert colleagues about how we could do that, so i had a rough idea of what we needed. Nothing written down.
I created an empty PriceQuoteCombiner class. Then an empty PriceQuoteCombinerTest class. Then i thought, "well, what is the first thing it needs to do?". And decided "if we get a price from one authority, we should just use that". So i wrote a test that expressed that. Then made it pass. Then thought "well, what is the next thing it should do?". And so on and so forth. And today, it has tests for one authority, multiple authorities, no authorities, multiple authorities but then one sends bad data, multiple authorities and one has a suspicious jump in its price which might be correct, might not, and many more cases.
The only point at which i had anything resembling a well-developed upfront spec was when i had written a test, and that was only an upfront spec for the 1-100 lines of implementation code i was about to write.
So your mention of "a very well-developed upfront spec" makes me wonder if you weren't actually doing TDD.
No argument about testing user interfaces, though. There is no really good solution to that, as far as i know.
But here's an anecdote that explains why you'd always want integration tests (other anecdotes for other test paradigms probably also exist): imagine a modern subway train. That train is highly automated but, for safety reasons, still requires a driver. The train has two important safety features:
1. The train won't leave a station unless the driver gives their OK. 2. The train won't leave the station unless all doors are closed.
The following happened during testing: The driver gives the OK to leave the station. The train doesn't start because a door is still open. The driver leaves the train and finds one door blocked. After the driver removes the blockage the door closes and the train departs. Now driverless.
I think it's crucial to view integration tests as unit tests on a different level: You need to test services, programs, and subsystems as well as your classes, methods, or modules.
Write code first. If that code is the v0.1 of the protocol between two blog systems great ! you can do that on a whiteboard and it looks like design when actually it's writing code on a whiteboard.
Now you know what to test so write the test, after writing the code.
Now write the next piece of code.
Do not at any time let a project manager in the room
As a result, I often try to fit my tests into these existing systems rather than starting with the test and refactor the code under test to fit that shape. The only resource I've seen for dealing with this issue is the advice in the book "Working Effectively with Legacy Code", to write larger system tests first so you can safely refactor the code at a lower level. Still, that's a daunting amount of work when it's ultimately much easier for me to just make the change and move on.
However, getting the client to agree that their design feature is satisfied by a specific set of steps, then writing software that satisfies that request, is a form of test-driven-development and I support it.
1. Writing frontend code -- I've left testing all together. I'd never hope to keep up with the pace
2, Writing APIs -- rudimentary testing that at least catches when I introduce regressions
3. Writing smart contracts -- magnitudes more test than actual code.
1. TDD isn't faster than simply writing test cases down, and executing them manually, especially when UI is involved.
2. TDD quadruples the amount of work needed for any given change.
3. TDD is the opposite of agile where you try to ship some product to the user ASAP to get feedback before spending time to refactor and clear technical debt. Write tests only for features that are confirmed so you don't spend time and effort on stuff people don't want.
4. Similar point as 1, but you need to evaluate if making something easy to test is worth the time savings than just running the test manually.
Yes, when you start writing code, it may not be well defined, or you (the coder) may not understand the requirement as intended. That's OK.
Write your test. Make clear your assumptions and write the code against that. Now your code is easier to refactor and acts as living documentation of how you understood the requirement. It also acts to help other engineers not break your code when they "improve" it.
If QA, the client or God himself decides the code needs to change later, for whatever reason, well that's OK too.
Unless you need to change the design of the interface in any way -- then it's harder. Tests lock a particular interface in place -- which is great if you have a well defined interface. But if you're trying to figure out that interface then you've prematurely locked yourself in.
* Using tests as a (or even the primary) design tool (strong TDD)
* Test-first development (weak TDD)
* Integration vs. unit testing
* What is a unit test?
* Should one use mocks and if so, when and how?
I think each of these topics merits a separate discussion and you can be e.g. in favour of at least weak TDD while maintaining that unit tests have little value, or you can be in favour of unit tests but disagree that "unit test" means "unit = class/method/function". You can have differing opinions on the value of mocks even if you subscribe to strong TDD (that's essentially the classicist vs. mockist divide in the TDD scene - for example, Bob Martin is more skeptical of mocking than, say, the "Growing Object-Oriented Software, Guided by Tests" crowd is).
IMHO, the biggest problem with testing is that most developers are not very good at it. I routinely see tests that are so complicated that it becomes very hard to understand, let alone debug them. In my experience, a lot of people also skip tests when reviewing code.
Well-written tests make a code base a joy to work with. Bad tests make everything painful. I don't know how to fix this, but we should pay more attention to it. If we had better tests, it would be easier to argue about the merits of TDD, unit testing, mocking etc. With badly written tests, everything devolves into a "why even test [this specific thing]?" kind of discussion.
It has a good idea but as many others have said, you need a pretty well spec'ed software beforehand for it to work. When you code a certain piece you might change it, top to bottom, several times -- we aren't perfectly thinking machines and we need to iterate. Having to re-prototype tests every time hurts productivity not only in terms of hours -- an obstacle that can be overcame in a positive environment and is rarely a true problem. It hurts in terms of it demotivating you and you losing the creative energy you wanted to devote to solving the problem.
The "it depends" thing will always be true. When you gather enough experience you will intuitively know the right approach to prototyping + testing an idea.
TDD is but one tool in a huge toolbox. Don't become religious over it.
I liked part of Kent Beck's writings back in the day but I am inclined to agree with other posters that he mostly wrote books to sell courses. I mean the book had good content, don't get me wrong, but they also didn't teach you much except "don't do waterfall".
Martin Fowler also wrote some gems, especially "Refactoring", but in the end he too just tried to enrich your toolbox -- for which I am grateful.
Ultimately, do just that: enrich your toolbox. Don't over-fixate on one solution. There is not one universal solution, at least we don't know it yet. Probably one day a mix of mathematical notation + a programming language will converge into one and we won't ever need another notation again but sadly none of us will live to see it.
It’s pretty clear at this point that testing is one of the most valuable tools in the box for getting sufficiently “correct” software in most domains.
But it’s only one tool. Some people would call property checkers like Hypothesis or QuickCheck “testing”, some people wouldn’t. Either way they are awesome.
Formal methods are also known to be critical in extreme low-defect settings, and seem to be gaining ground more generally, which is a good thing. Richer and richer type systems are going mainstream with like Rust and other things heavily influenced by Haskell and Idris et al.
And then there’s good old: “shipping is a feature, and sometimes a more important feature than a low defect count”. This is also true in some settings. jwz talk very compellingly about this.
I think it’s fine to be religious about certain kinds of correctness-preserving, defect-preventing processes in domains that call for an extreme posture on defects. Maybe you work on avionics software or something.
But in general? This “Minimal test case! Red light! Green light! Cast out the unbelievers!” is woo-woo stuff. I had no idea people took this shit seriously.
But fundamentally no one should ever be trying to merge code that hasn’t been unit tested. If they are, that is a huge problem because it shows arrogance, ignorance, willingness to kick-the-can-down-the-road, etc.
From an engineering perspective the problem is simple: if you’re not willing to test your solution then you have failed to demonstrate that you understand the problem.
If you’re willing to subsidize poor engineering then you’re going to have to come to terms with adopting TDD eventually, at some stage of the project’s lifecycle, because, you have created an environment where people have merged untested code and you have no way to guarantee to stakeholders that you’re not blowing smoke. More importantly, your users care. Because your users are trusting you. And you should care most of all about your users. They are the ones paying your bills. Be good to them.
Here you are asserting that unit testing is fundamental, and that not believing this is arrogance and ignorance.
I'd suggest your view that your way is "the" way, is an ironic display of arrogance, and perhaps ignorance.
And this perhaps I think is the core of much of the anti-TDD sentiment. It's not that we don't think TDD and unit tests are without their positives, it's that we don't like being told this is the one true way to write software, and if we don't do it your way we are engaging in poor engineering.
You set your standards as a team of what you will consider acceptable tests and as long as the dev submitting the PR meets that standard why does it matter if they did TDD or not? TDD is a means to end, it's not a religion. As long as you write tests that meet the standard it doesn't matter when you write those tests.
The level of micromanaging that TDD evangelists seem to want in people's workflows is infuriating. It's literally cultish.
Edit: I realize this came across more negative and abrasive than I intended. I think TDD has some good parts (primarily around gamification of writing tests and giving a serotonin hit whenever a test goes from red to green). I practice TDD around 50% of the time when appropriate, but most people who have worked in the industry that TDD as sold by purists is impractical and adds negative value.
Refactoring becomes then easy although sometimes tedious, but provides a degree of confidence that your change works with all posible ways to use the code.
Its also healthy to break the rules if time pressure, but keeping a registry of technical debt, helps keeping the morale up. There is nothing more demoralizing that working on an environment where you can’t estimate because who knows what will be broken, it’s hard to make improvements because everything is entangled, and there is no time to invest in a rewrite. This usually ends up with very talent people quitting if they are unable to fix it.
1. We want a useful target for our software. You could design a graphical mock-up of software and design your software to fit it. Or you could create a diagram (or several). Or you could create a piece of software (a test) which explains how the software is supposed to work and demonstrates it.
2. When we modify software over time, the software eventually has regressions, bugs, design changes, etc. These problems are natural and unavoidable. If we write tests before merging code, we catch these problems quickly and early. Catching problems early reduces cost and time and increases quality. (This concept has been studied thoroughly, is at the root of practices such as Toyota Production System, and is now called Shift Left)
3. It's easy to over-design something, and hard to design it "only as much as needed". By writing a simple test, and then writing only enough code to pass the test, we can force ourselves to write simpler code in smaller deliverable units. This helps deliver value quicker by only providing what is needed and no more.
4. Other reasons that are "in the weeds" of software design, and can be carefully avoided or left alone if desired. Depends on if you're building a bicycle, a car, or a spaceship. :-)
But as in all things, the devil's in the details. It's easy to run into problems following this method. It's also easy to run into problems not following this method. If you use it, you will probably screw up for a while, until you find your own way of making it work. You shouldn't use it for everything, and you should use good judgement in how to do it.
This is an example of software being more craft than science. Not every craftsperson develops the same object with the same methods, and that's fine. Just because you use ceramic to make a mug, and another person uses glass, doesn't mean one or the other method is bad. And you can even make something with both. Try to keep an open mind; even if you don't find them productive, others do.
The problem I have with TDD is the concept of writing tests first. Tests are not specifications (in TDD world the line is blurred.) Tests are confirmation.
I develop my code (I write back end plumbing code for iOS currently) from a test frame work.
My flow:
* Specify. A weak and short specification. putting too much work into the specification is a waste. "The Gizmo record must be imported and decoded from the WHIZZBAZ encoding into a Gizmo object" is plenty of specification.
* Write code for the basic function.
* Write a test for validity of the code (the validity of the record once loaded in the Gizmo/WHIZBAZ case)
But the most important tests are small micro tests (usually asserts) before and after every major section (a tight loop, a network operation, system calls etcetera). More than half my code is that sort of test.
In my experience TDD uptake and understanding suffers because a lot of developers are in a context of using an existing framework, and that framework sort of fights against the TDD concepts sometimes. Getting around that with things like dependency injections, reversals etc then gets into the weeds and all sorts of 'Why am I doing this' pain.
Put another way, a lot of commercial development isn't the nice green-field coding katas freedom, it's spelunking through 'Why did ActiveRecord give me that?' or 'Why isn't the DOM refreshing now?'. Any friction then gets interpreted as something wrong about TDD and the flow gets stopped.
For more information on studies covering the topic (and much more), I highly recommend watching Greg Wilson's Software Engineering's Greatest Hits[1].
[0]: https://neverworkintheory.org/2016/10/05/test-driven-develop... [1]: https://youtu.be/HrVtA-ue-x0?t=448
The author quotes a tweet expressing amazement that any company might not use TDD, 20 years after it was first popularised - and then writes
"I’d equate it to shell scripting. I spent a lot of time this spring learning shell scripting"
Wow! I feel like the person in the tweet. It's amazing to me that someone could be in a position to write an article with such solid development background without having had shell scripting in their everyday toolbox.
(I use TDD some of the time - I was slow to pick it up and a lot of my older code would have been much better if I had appreciated it back then. I like it very much when I don't really know how the algorithm is going to work yet, or what a good API looks like.)
I get the feeling that we are all inclined to think that we have the entire story of development in our own personal heads and can therefore lay down laws.
...and yet most of these disciplines need everyone's co-operation and one can feel that if you don't treat it as dogma then you're never going to make everyone "comply"...
I think the fact that TDD (and other popularly debated methodologies) haven't taken over just by being obviously easier and better is a sign that they aren't really suitable for being made into dogmas. They're tools and we should have the choice like any workman to choose them or not.
For a brand new product, I'm almost never using TDD. I'm building out a solution in the pattern I want, getting some minimal feature or features up, and then I write tests appropriate to that pattern. Later on I might use TDD to keep working on it, but it can be a burden at the start of projects.
1) you likely aren’t going to get the job without some absurd degree of unit testing
2) most of your code should be pure functions making them trivial to unit test
3) writing pure functions and making your code testable makes your system less coupled which is a good thing
4) designing the tests is designing the software which is often helpful
That’s it I don’t have a fifth point. Actually I do, writing software is a form of art and it cannot be summed up as simply as unit testing everything always bad or always good. There might be somethings like converting an engine performance simulation to Golang from Excel that might be fantastic to unit test the shit out of, testing if onPress works on your button component is basically pointless.
It also makes cutting corners more difficult, because it is possible to have (sort of) working software without testing, but you can't have working software if the only thing you have are failing tests (the important first step in TDD). Most TDD people probably thing of that as a positive, I don't. Sometimes, cutting corners is the right thing to do, sometimes, you actually need to write the code to see if it is viable, and if it is not, well, you wasted both the tests and the code, not just the code.
But I don't think it is the only problem with TDD. The main problem, I think, is right there in the name "test driven". With a few exceptions, tests shouldn't drive development, the user needs should. Test driven development essentially means: write tests based on the users need, and then write code based on the tests. It means that if your tests are wrong and your code passes the tests, the code will be wrong, 100% chance, and you won't notice because by focusing on the tests, you lost track of the user needs. It is an extra level of indirection, and things get lost in translation.
Another issue I have noticed personally: it can make you write code no one understands, not even yourself. For example, your function is supposed to returned a number, but after testing, you notice that are always off by +1, the solution: easy, subtract 1 to the final value. Why? dunno, it passes the tests, it may even work, but no one understands, and it may bite you later. Should I work like that? Of course not, but this is a behavior that is encouraged by the rapid feedback loop that TDD permits. I speak from experience, I wrote some of my worst code using that method.
If you want an analogy of why I am not a fan of TDD: if you are a teacher and give your students the test answers before you start your lesson, most will probably just study the test and not the lesson, and as a consequence they will most likely end up with good grades but poor understanding of the subject.
TDD is great because it forces you to concretize and challenge assumptions, and provides a library of examples for new devs to a codebase.
It seems that TDD can forever evade actually solving a problem in its general form, always just extending the number of concrete cases that work.
For instance, a function to measure the length of a string first works only for the case len("") == 0; the result is wrong for all else. TDD allows this to be extended into "working" in idiotic steps like len("a") == 1, but len("b") returns 0, and so on.
Also, how, in TDD, can we write a test which says "for any input not handled by the tests developed so far, I want an exception". That is to say, when I write the first test len("") == 0 and get it to pass, I don't want len("a") to return 0; I want it to throw. I could write a test for that: throws(len("a")), and it would initially fail. But I want the behavior to be entirely general, and I don't want to maintain the test when len("a") changes to returning 1.
These problems make TDD just look like a way to get useful work out of complete morons, in problem areas involving calculating functions that have finite domains that can be exhaustively tested.
As soon as you write a code that is more general: which makes more than the new test case to pass, you're leaving TDD. For instance, the test case wants len("a") == 1, but you write code such that len("b") == 1 would also pass, and len("abc") == 3 would also pass and so on. You now have a lot of useful and good behavior that is not tested. You've not had a len("abc") == 3 which went from red to green.
Once other code starts relying on len being a reliable length function, you must have left TDD behind. Code is calling len("foo.txt"), which has not been tested! How realistic is to prevent that?
At some point, a supposedly TDD-developed program must handle real-world inputs, and they could not all have been tested, because that's the reality. Only simple functions with small finite input spaces can be exhaustively tested. TDD must necessarily allow a lack of testing to creep in, and the rules for that, if any, are ad hoc.
I deviate from some proponents in that I think this kind of TDD can be done while writing zero unit tests, but in practice they keep you sensitized to good design techniques. Plus the tests do occasionally catch bugs, and are otherwise a good forum to exercise your types and illustrate your code in action.
But then there are tests that make sense but are hard to write.
And then there are tests that require infrastucture.
I don't write tests for every little thing. But I do write them if I actually do want to test the functionality of what I just wrote. But stuff like
s := new (Service)
if s == nil {
t.Fail()
}
When business people don't know what they want, do not try TDD. It will be a waste of time. When people do KNOW, or you have a RELIABLE subject matter expert (at a big company you might have one of these), TDD is a lot safer and easier to do.
Maybe it's an OCD thing, but I don't like seeing compiler errors of unimplemented pseudo-code and mock placeholders. It breaks my flow.
But 2 files open at all times, writing tests as the main class is being developed? And no compiler errors? Love it.
> 1. Write a minimal failing test.
> 2. Write the minimum code possible to pass the test.
> 3. Refactor everything without introducing new behavior.
> The emphasis is on minimality. In its purest form we have Kent Beck’s test && commit || reset (TCR): if the minimal code doesn’t pass, erase all changes and start over.
An example would be helpful here. In fact, there's only a single example in the entire article. That's part of the problem with TDD and criticisms of it. General discussions leave too much to the imagination and biases from past experience.
Give me an example (pick any language - it doesn't matter), and now we can talk about something interesting. You have a much better chance of changing my mind and I have a much better chance of changing yours.
The example in the article (quick sort) is interesting, but it's not clear how it would apply to different kinds of functions. The author uses "property testing" to assert that a sorted list's members are of ascending value. The author contrasts this with the alleged TDD approach of picking specific lists with specific features. It's not clear how this approach would translate to a different kind of function (say, a boolean result). Nor is it clear what the actual difference is because in both cases specific lists are being chosen.
It's definitely useful, but those strongly opposed often won't use it at all unless it mandated which tends to lead to strict adherence policies at a lot of companies.
I have seen only a handful of projects, show as examples of TDD, that actually were projects I liked.
Its a variation of the "don't worry they'll tell you" joke. How do you know a project was TDD?
Early stage programmers and all stages of project manager.
Don't have any expectations and are exploring? Don't do any of this.
If you are picking up some code to work on, the nicest to work with is the one that was done with TDD.
You rarely see someone who writes production code preach TDD.
Something fishy there...
Said teams absorb a lot of the pain of Soft Devs who don't bother even running their own code.
I have similar feelings about maximalism in a lot of areas.
Many organizations producing software today don't share many values with me as an engineer. Startups aren't going to value correctness, reliability, and performance nearly as much as an established hardware company. A startup is stumbling around trying to find a niche in a market to exploit. They will value time to market above most anything else: quick, fast solutions with minimal effort. Almost all code written in this context is going to be sloppy balls of mud. The goal of the organization is to cash out as fast as possible; the code only has to be sufficient to find product-market fit and everyone riding the coat-tails of this effort will tolerate a huge number of software errors, performance issues, etc.
In my experience practicing TDD in the context of a startup is a coping mechanism to keep the ball of mud going long enough that we don't drown in errors and defects. It's the least amount of effort to maintain machine-checked specifications that our software does what we think it does. It's not great. In other contexts it's not even sufficient. But it's often the only form of verification you can get away with.
Often startups will combine testing strategies and that's usually, "good enough." This tends to result in the testing pyramid some might be familiar with: many unit tests at the bottom, a good amount of integration tests in the middle, and some end-to-end tests and acceptance tests at the top.
However the problem with TDD is that I often find it insufficient. As the article alludes to there are plenty of cases where property based testing has stronger guarantees towards correctness and can prevent a great deal more errors by stating properties about our software that must be true in order for our system to be correct: queued items must always be ordered appropriately, state must be fully re-entrant, algorithms must be lock-free: these things are extremely hard to prove with examples: you need property tests at a minimum.
The difficulty with this is that the skills used to think about correctness, reliability, and performance require a certain level of mathematical sophistication that is not introduced into most commercial/industrial programming pedagogy. Teaching programmers how to think about what it would mean to specify that a program is correct is a broad and deep topic that isn't very popular. Most people are satisfied with "works for me."
In the end I tend to agree that it takes a portfolio of techniques and the wisdom to see the context you're working in to choose the appropriate techniques that are sufficient for your goals. If you're working at a startup where the consequences are pretty low it's unlikely you're going to be using proof repair techniques. However if you're working at a security company and are providing a verified computing base: this will be your bread-and-butter. Unit tests alone would be insufficient.
TDD is the wind, it cannot be captured by your net.
The article is really quite good. Much, much better than the discussion here prepared me for!
The way I practice "pragmatic TDD" is to construct my code in a way that allows it to be tested. I use dependency injection. I prefer small, static methods when possible. I try not to add interfaces unless actually needed, and I also try to avoid requiring mocks in my unit tests (because I find those tests harder to write, understand, and maintain).
Notably: I explicitly don't test "glue code". This includes stuff in startup -- initializing DI and wiring up config -- and things like MVC controllers. That code just doesn't have the cost-benefit to writing tests: it's often insanely difficult to test (requiring lots of mocks or way over-complicated design) and it's obvious when broken as the app just won't work at all. Integration or UI automation tests are a better way to check this if you want to automate it.
I strive to just test algorithm code. Stuff with math, if/else logic, and parsing. I typically write the code and tests in parallel. Sometimes I start writing what I think is a simple glue method before realizing it has logic, so I'll refactor it to be easy to test: move the logic out to its own method, make it static with a couple extra parameters (rather than accessing instance properties), move it to its own class, etc.
Sometimes I write tests first, sometimes last, but most often I write a few lines of code before I write the first tests. As I continue writing the code I think up a new edge case and go add it as a test, and then usually that triggers me to think of a dozen more variations which I add even if I don't implement them immediately. I try not to have broken commits though, so I'll sometimes comment out the broken ones with a `TODO`, or interactive rebase my branch and squash some stuff together. By the time anyone sees my PR everything is passing.
I think the important thing is: if you look at my PR you can't tell what TDD method I used. All you see is I have a bunch of code that is (hopefully) easy to understand and has a lot of unit tests. If you want to argue some (non-tested) code I added should have tests, I'm happy to discuss and/or add tests, but your argument had better be stronger than "to get our code coverage metric higher".
Whether I did "strong red-green-refactor TDD" or "weak TDD" or "pragmatic TDD" the result is the same. I'd argue caring about how I got there is as relevant as caring about what model of keyboard I used to type it.