Automated vs manual testing

Yesterday I had an interesting tweetversation (it actually looks like this word exists) with @lunivore which started with this sentence:

My #1 suggestion for legacy code: If it works, and you change it, check that it still works. If there aren't any tests, run the app already.

I replied something about writing tests, and the discussion went on with several interesting remarks that made me think. As on many others, here's one thing on which I agree with Liz: automated tests do not imply that your app is working as it should. I have seen it too many times to be so naive. It could be (almost) true if our code coverage was perfect, but let's face it, it is not. Please note that I'm not talking about having tests on getters. We should have unit tests, integration tests, end-to-end tests, there-is-no-try-yoda-tests and everything you can think of as long as it ends with "tests" for almost everything. And more often than not we don't.

And even if we did, tests are not users. Well they are, but they are not the users we're most interested in: those who receive value from our app.

Automated tests are very important, but in the end they only are our safety net (even if they also guarantee our customers), but in the end it all comes up to real users.

Automated tests never sign checks, users do. Or at least the bosses of users do. Then, if we belong to that overrated category of people that still insist on eating every day (possibly more than once), we shoud strive to have happy users more than happy tests.

Automated tests are one of our means, not our ends. Let's not forget it.

PS don't forget to follow Liz on Twitter and on her blog.

Startup without falldown: the slides

Here are the slides of my talk "Startup without falldown: strategic planning beyond wishful thinking", held at the 8th Italian Agile Day in Rome:

Startup senza falldown

View more presentations from Andrea Maietta

I'd like to thank everyone who has attended, I hope I haven't wasted their time. If you are one of the lucky (???) ones, please help me to improve and leave a feedback here.

If you want an exhaustive bibliography please register here and you'll receive it in a few days, with a bonus mind map of the talk.

On "Working Effectively with Legacy Code"

In the last few weeks I have spent most of my time working on a legacy system which is going to be partly replaced by a new one which greatly broadens its scope. But what exactly is legacy code? Michael Feathers says that

To me, legacy code is simply code without tests.

It is as simple as that:

Code without tests is bad code. (...) With tests, we can change the behavior of our code quickly and verifiably. Without them, we really don't know if our code is getting better or worse.

Alas, too often the answer is not the one we'd like to hear.

We started out so well

Almost every system, if you have at least decent programmers working on it, starts out as a wonderful green field: defined architecture, code conventions, solid principles, patterns and practices.

After a while the gardeners get busy with other gardens, and a couple of weeds start to poke out. Then some more, then more. And more. Meanwhile seasons pass and rain falls, so our green field gets muddier and muddier. Actually it looks more like it has rained cats and dogs, day after day after day. Mud becomes quicksand, and nobody dares to even get close to it.

The (in)famous spaghetti incident

Too soon the project reaches the point in which whenever you pull a string you have an undesired and often undetected change in a completely different part of the system. When I want to sound important I refer to this as the butterfly effect, a metaphore used in chaos theory. Yes, I know that your projects are different and you have never seen it happen, so try to use your imagination and stay with me a little more, will you?

That was exactly where I was, all tangled up in obscure dependencies: the perfect spaghetti code.


As I tweeted, I love spaghetti when they are in my dish, not in my code, so I had to do something (even because deadlines are always waiting in ambush). To sharpen my tools I retrieved my copy of "Working Effectively with Legacy Code". The book covers many aspects of what we developers have to face daily: instead of the "write once, run anywhere" mantra, we have to deal with "write once, read it at least ten times", but too often our code is obscure. Yes, our code. And we don't have tests, or at least we don't have enough. Do you have a complete coverage? If you do, you have all my respect, otherwise welcome in the family.

How do we get out of this?

First of all, we have to get our code into a test harness, but it is not as easy as it sounds. The book contains a series of 24 different dependency breaking techniques, each of which presented in several almost-real situations, to minimize the impacts of changes and ensure that we are not breaking anything. It sounds like the classic catch-22 thing:

When we change code, we should have tests in place. To put tests in place, we often have to change code.

This is where the techniques come in handy. I was pleasantly surprised when I discovered that, after some years of TDD, almost every problem and solution described in the book sounded familiar, so I skimmed it more than actually reading it. I still remember when I first read it: man it was hard, not the writer's fault, but because of the reader.
One of the most important things is that you have to preserve the behavior of your system when you refactor it (and preserve the rest of the system when you introduce new features or change existing ones), so you should at least know what it does. What is better than some characterisation tests?

Cleaning it up

I love NetBeans. I always have, since it was Forte4Java. Now I love it even more because it has a heap of automated refactorings that make "Refactoring" needless (ehm... that's not true, NetBeans deals with the mechanics, but you should know what you're doing, so stop reading and go buy your copy if you still don't have it. Done? OK, let's get on).

Do you need to create a new test class? nothing simpler, just a simple SHIFT + CTRL + U and a template with all the skeleton methods is ready for you. Just punch in the starting conditions and the expected results and you're done. Feathers describes all refactorings steps by steps, but a few keystrokes are all NetBeans needs to extract superclasses, methods, interfaces, pull up or push down members and methods, delegate, and so on and so forth. Nevertheless, you should know what you're doing.

While working your code could get uglier in some places. It could be temporary, or temporary in the Italian way (which means definitive). Even in the latter case, at least you would have tests in place so that you know you're not breaking anything.

In several places Feathers suggests to reduce incapsulation to put code into a test harness. Somebody might find it strange, extremists might find it insane. Yet...

Encapsulation isn't an end in itself: it is a tool for understanding.

Remember when I wrote that you write code once and read the very same code over and over? What 's the point of an encapsulated design if I spend hours for the most trivial task? Mnd you, I'm not suggesting every field must be public and collections should be directly exposed, but nobody is going die if a private method becomes protected for the sake of testing.

Where have my spaghetti gone?

After a while, one after the other, your pull spaghetti out of your dish, slowly replacing them with neat and tidy sushi.

Before you realize it, you will look at your codebase with different eyes, and chances are that you will enjoy dealing with legacy code. Don't enjoy too much though, even if you have a lot of technical debt to pay: you don't have to pay it for the sake of doing it, but because it allows you to move faster when you need it (and you know you will need it).

Let me stress this again: if I didn't have some hundreds of tests already in place, these weeks would have been months. Baby steps, nip and tuck, red and green bar. But I only introduced tests and refactored legacy code when I needed it, otherwise I would spend too much time on activities that were not a priority. See something rotten? pen it down, fix what you're doing and if you still have time you can deal with it. Look for decisions that could change, but change your code just in time (the YAGNI rule rules).

The importance of unit tests II

Yesterday I had some fun spending some hours to introduce ajax in a legacy application (I call it legacy because it has a very small amount of automated tests). During the afternoon I was quite pleased with myself, partly due to the satisfaction associated with the work done, partly due to the "Steppenwolf" novel, partly due to the Beethoven Sonatas I was listening to, partly due to my full stomach. Being a very wise and intelllectual person, I am in favour of the latter.

All this abrupltly ended when I got an unexpected error from a web service called from a part of the codebase that I had modified in the morning, which seemed to go wrong when I submitted a foreign address.

First I checked the test that exercised the web service client, where I verified that all the parameters where correctly passed as expected. Then, I wrote some other tests trying to exercise the particular feature, narrowing the scope of my private investigations. Btw, this reminds me that I have not listened to the Dire Straits in a long time, which is a shame.

Everything seemed fine, so I spent some hours with the developers of the service trying to figure out what was going wrong with the service (if anything). After some head banging it emerged that he problem was that we were focusing on a particular parameter, while the error sneaked in another one. I hope they will not make me pay for the dents in the desk.

That would have been clear from the beginning if we checked all the parameters in the XML stream instead of focusing only on the ones that we thought were important for the specific call (the port in the service is only one, even if there should be many, and the behaviour is determined by which of the several zillions parameters are set and what their values are; I don't think this is very brilliant, but it cannot be changed, so complaining is useless... or at least it brings no business value).

So, everything went back to a wrong value, due to a "simple" setter method.

Now, you normally never write a test for a setter method. This is perfectly acceptable, as you should write tests for "everything that could possibily go wrong" and a normal setter is not included in the list.

Pity this setter was not a plain one, but contained an if:

public void setMyProperty(final String value) {
  if (value != null || StringUtils.isEmpty(value.trim())) {
      this.myProperty = "FIXED";
  } else {
      this.myProperty = value
  }
}

This looks like a blunder... why that not-null check? is it to be able to trim the value or should it actually be a check the value is NOT null? If the former is true, why hasn't the author simply used the isBlank method instead? maybe because she was thinking of saving the trimmed property (which in the case she forgot to do) or because she didn't know the existence of the method?

Though strange this might sound, all this is not really important as I have access to the product owner and I could get all the answers I needed.

The point is: that method required a test, and it was nowhere to be found. You might argue that it's simple enough to avoid writing one, but all the time I wasted says something different. Still, you might add that it was my fault because I didn't check the setter in the first place, but this is only another arrow in my quiver: this is exactly the reason for which we need automated tests, as people do make mistakes and forget to check simple methods.

I'd like to think that if I were pair programming with the author of this code I'd never let her skip writing the failing tests first, at least not without fighting.

Note that I'm not pointing a blaming finger, I could have written that code myself (and sometimes I did, and I'm happy to say that I always regret it when I realize it).

I know I am fighting - and so far losing - a running battle, but tests are necessary, even if someone tells you they represent a cost. Actually they are not even a cost, they are an investment, they are one of the most important risk management tools software developers have.

It only took me some fifteen minutes to write four different tests that assessed the desired behaviour and rewrite the method from scratch. These fifteen minutes would spare three people some wasted hours. This also demonstrates how the cost of bug fixing dramatically increases with time.

Yet, for some unknown reasons and against all evidence, too many managers AND developers refuse to believe in practices like TDD or pair programming. Is it to mantain the illusion of control? is it fear of changes? is it lack of trust? is this evidence not so evident? let's try some math:

• cost of writing the setter method: 5 minutes
• perceived cost (usually coincides with the former): 5 minutes
• cost of writing tests and the setter method: 15 minutes
• perceived waste: cost of writing tests minus cost of writing the setter method = 10 minutes
• cost of writing tests and the setter method, pair programming: 30 minutes
• perceived as almost blasphemous waste: cost of writing tests pair programming minus cost of writing the setter method = 25 minutes

And this is where analysis normally end. On the other hand...

• cost of finding the bug: 6 hours
• actual cost: 6 hours plus 15 minutes PLUS 5 minutes = 6 hours and 20 minutes
• actual waste: 6 hours PLUS 5 minutes

Also note that the time needed to write the code in the first place was completely wasted.

Still think that writing tests is too expensive? Well, the curious thing is that at this point everyone seem to agree on the reason behind the added cost: the code was sloppy. As a corollary, the blame is on the developer. Well, THIS IS NOT THE PROBLEM. The problem is that the way used to write the code was sloppy. And this does not depend entirely on the developer.

Think about it.

What about my failing test?

After all, it turned out that my modified interceptor was not behaving correctly. Woah, wait a minute... didn't we have a test for that? Yes we did.

Yet, the user test failed.

What did I write the tests for? was it only an addictional burden, like someone suggests here? On the contrary. Well, when you write a test you actually are stating that "given X, when Y then Z". Unit tests are about "when Y, then Z", as when you write them you are supposed to know what X is (otherwise it would not be given). As I positively knew that the "when Y, then Z" part was working, the diagnosis was easy: my assumptions were wrong. A couple of clickety-clackety-clack and I collected the necessary data to build up the fixture. Ta-daaa! (yes, another red-green-refactor cycle followed).

TDD is a time saver, not a burden. And you don't need to be a master of design, like Brett suggests, as TDD forces you to concentrate on desing, even at an unconscious level. Whenever you think "how the hell am I going to test that?" the code is suggesting that you're taking a wrong way. Your tests are the first clients of your classes. Treat them well, or else they will stop buying. And finding a new customer is much more expensive than keeping the ones you've got, which means that you won't write tests. It's a descending spiral: keep out of it.

Write a failing test first

Checking a functionality of a Struts2 interceptor I realized that we forgot to check a condition. Adding the correction was trivial, but I resisted. I first added a failing test, even though the class had no tests at all (this was justified by the fact that most methods are actually pretty simple one-liners).

Yet, an error lurched its way from behind.

The answer is NOT the code fix. At least, not only the code fix. Because other errors are lurking, and I don't want to be in the balance: I want to avoid regression(s). There are no excuses, as testing with Struts2 is much easier than it was with Struts.

That said, I wrote two test cases to avoid regressions on what I knew was working. Green. Then I added another test case that checked the failing condition. Red, as supposed. Only at this point I could add the magical line of code I needed. Back to green. Now we're going somewhere. And I left my camp (slightly) better than how I found it.

Then it was time for some riskless refactoring (well maybe not riskless, but close). Following Steve Freeman suggestions, I changed the names of a couple of methods to make them more expressive. Of course I could have done this before, but having a test harness makes you feel less... vulnerable. Once you get the habit, it is hard to get rid of it. And I don't want to get rid of it :-) on the contrary, I want to keep the ball rolling.

A better programmer?

After writing about it a couple of years ago recently I took the BetterProgrammer test.

It turned out I am not so bad, even if I'm sure there's plenty of room to improve my skills, mostly because the tests also track the time it takes you to submit the solutions; I'm pretty sure my answers were correct, as for each task I produced almost all the unit tests I could think of (before I actually produced my code, that's obvious).

The tests were very interesting and can be used as katas; some of them could also be resolved by brute force, but some knowledge of maths and combinatorics surely helped me a lot in writing faster and more elegant solutions (it hasn't given me a penny yet, but it turns out that getting a University degree with full marks has yielded some results after all).

You can check your results against mine here.

The importance of unit tests

Today I made a small change in a class in the application I'm working on. Being a diligent monkey, before adding the new feature I wrote a small dedicated test. The test failed, I added the feature, the test passed, I cleaned up a little bit (not too much, having just added three SLOCs) and finally committed the class in our SCM before hopping on to the next feature.

After a while our Continuous Integration Server pitilessly sent me a mail in which it informed me that the build was broken. No, better yet, that I broke the build, causing not one but 17 tests to fail. O shame and amazement, how was it possible? I had even written a test...

Despite all apparences, this is very good news. Why, you ask, grasshopper? Let me ask you a question: when do you want to know you have a problem? talking about software development, as soon as possible. Those 17 failures have saved me from delivering buggy code that would upset the user and set the stage for some annoying debugging sessions.

This has only been possible because of the test harness: all the time we spent writing tests has, once more, paid off well. So, without forgetting to add a test for the previously unchecked condition, I fixed the code in double-quick time.

I wish all managers understood this: writing tests is not a cost, but a tremendously effective cost saving tool.

First Iteration in itinere

For the second time in the sprint - but the first one does not count because we merely cut funcionalities that were not needed anymore - our Burndown chart clearly express what everybody has felt in the last weeks, even if it was unsupported by facts (at least until yesterday): we will deliver.


Once more I'd like to stress the importance of such a tool: we always know what is going to happen, and we can take informed decisions.

As I said, the first big slope is a cut due to a modification of the contract; the second came after most architectural decisions had been tested and taken, just as expected. For this first iteration we choose the part of the system we knew best and that - not by chance - is the basis upon which we will build the whole system. Nevertheless, we mostly focused on architecture, even if we will deliver actual funcionality. The project follows a test driven approach and a domain driven design, and we just anticipated a little more than we needed at the time because we know we will need it within a couple of months (so the YAGNI thing does not apply here).

So the bar is green, our Hudson happily smiles and so do I; first demo next week, maybe more on this after the retrospective.

The birthday greetings kata

On February the 14th the milano-xpug organized a kata whose subject was proposed by Matteo on his blog to introduce the exagonal architecture.

Unluckily I couldn't attend the meeting in the flesh, so I just decided to try the kata out here on my own. I first started from scratch, thinking it would prove more fun; then, as I was taking quite a different approach - you could also say that a whole different design was emerging - I decided to get back and start from the original code provided and refactor it to try and focus on the architectural issues which were the aim of the exercise.

Even if I think this exercise to be quite simple, I decided to post a trace anyway because someone might still find it useful. You can never say!

The heart of the application is the BirthdayService class, whose responsibilities consist in identifying employees whose birthday falls on a given date and sending them a greeting message.

This is the public method, called by the acceptance test:


I know it looks odd, but I've had enough of escaping characters and trying to format code to make it look nice, so I hope you will forgive me.

Anyway, the sendGreetings method parses an input file, creates employees, checks if they are eligible for a greeting message and delegate the actual sending of the message to a private method:


Seems a great bunch of things for a single class... let's try to sort it out. First I decoupled the service from the file with the employees' names introducing the interface EmployeeRepository, with the method getEmployees, then I wrote a FileSystemEmployeeRepository that implemented it. The repository was then injected into the BirthdayService via constructor by
the acceptance test. At this point the code looked like this:

...
List<Employee> employees = repository.getEmployees();
for (Employee employee : employees) {
 if (employee.isBirthday) {
    //compose message and call method that sends it
 }
}
...

Still I thought that the check on the date of birth could be done by the repository itself, so I changed the interface adding the new findEmployeesBornOn(OurDate) method and changed the code so that it looked like this:

...
List<Employee> employees = 
      repository.findEmployeesBornOn(ourDate);
for (Employee employee : employees) {
 //compose message and call method that sends it
}
...

Maybe not a very big change but I think it was worth it, as now the code is cleaner. Then there was the messaging part. First of all I decided to introduce a new BirthdayGreeting class, that produces a message body, a fixed subject and a recipient starting from an employee; at this point I extracted the interface SimpleMessage that exposes these three strings. The code had become

...
if (employee.isBirthday) {
  SimpleMessage greeting = new BirthdayGreeting(employee);
  sendMessage(smptHost, smptPort, "sender@here.com", greeting);
}
...

Still I had to resolve the dependency from the SMTP, so just like I did for the repository I introduced the MessagingService interface with the sendMessage(String sender, SimpleMessage message) method and injected it into the BirthdayService via constructor from the acceptance test. Some nip and tuck to move the body of the sendMessage method from BirthdayService into a new EmailService, implementer of MessagingService, and voila, the new BirthdayService code:

public void sendGreetings(int month, int day) {
  List<Employee> employees = 
      repository.findEmployeesBornOn(month, day);
  for (Employee employee : employees) {
     BirthdayGreeting greeting = new BirthdayGreeting(employee);
     messagingService.sendMessage("sender@here.com", greeting);
  }
}

I also changed the method parameter from OurDate to month, day. Now we have a simple and decoupled service that can focus on its true responsibilities: ask for all people who should receive a greeting, create a new message and delegate its sending.

One might wonder whether the sender belongs where I put it, or if the service should directly instantiate a BirthdayGreeting with a new rather than getting it in some other way. But maybe that's going too far.

BTW I also decided not to use the Employee.isBirthday() method and I created a value object called IsBirthday, created with two parameters representing month and day, with the boolean method forEmployee(Employee). The code in the repository now looks like this:

public List findEmployeesBornOn(int month, int day) {
   List<Employee> list = new ArrayList<Employee>();
   IsBirthday isBirthday = new IsBirthday(month, day);

   for (Employee employee : employees) {
      if (isBirthday.forEmployee(employee)) {
         list.add(employee);
      }
   }
   return Collections.unmodifiableList(list);
}

That's all folks... maybe I'll manage to post the complete code here, where you can find some other (and more accredited) solutions produced by people who actually were there.

Collective Code Ownership Part 2

When I write "everyone is responsible for all the code" I also mean that everyone is responsible for all the tests, so if you only have unit tests and you need an integration test... write it! It does not matter if you didn't write the original code or the original unit tests, it does not matter even if you didn't talk to the original developer. It is your responsibility to write it: navigating through an application just to click a link is (almost) completely pointless, and it sure is a waste of time. That does not mean that end user testing must not take place (how could you assert that "a normal user should be able to foobar the barfoo without previous training" otherwise?), but we should strive to automate as much as possible.

This is even harder to assimilate, because not only you have to touch code written by others, but (heresy!) write tests for code you didn't write. It's already hard to persuade people of how much TDD pays off the initial investment and the mental shift, but it's even harder to push them to use tests as a primary way to discover how a particular piece of code works (I often write tests for third part libraries or also for checking the behaviour of java.lang or java.util classes I'm not sure about). I can understand that people fear changes, and that one could be unwilling to change the way he's always worked, but here we're talking about being afraid of one's own shadow. Try it, and try it seriously, and after you did if you still think it won't work (and you won't) you'll still be able to give it up. And... no, "I have no time" is not a valid answer, because not having tests only leads to wasting more time.

Alas, too often I perfectly agree with Gabriele...

Growing Object-Oriented Software

Steve Freeman and Nat Pryce are posting online parts of a book they're working on, called Growing Object-Oriented Software, Guided by Tests. Should anyone be interested in giving feedback, they provide a Yahoo discussion group.

TDD being imposed in every quality discussion?

Following this discussion I posted my opinion on the matter.

When you have a "simplest thing that could possibly work" the first fit can be a good answer, but it could also be a horrible one. That is where refactoring comes in, and I must admit I'm pretty astounded that nobody mentioned it so far.

TDD means "let me think what this should do... ok I'll write a test that uses this class... there you have the class... ok it works... LET'S ELIMINATE ALL DUPLICATES AND SEE IF WE CAN SIMPLIFY IT FURTHER".

TDD helps quality not only because your software works, but also because it is simple and easy to understand (then to modify). TDD should not be imposed in every quality discussion, but it surely is a very good means to reach quality; writing tests surely is time consuming, but it is well invested time: you're paying for quality. As Philip Crosby said, quality is free but only to those who are willing to pay heavily for it. That's why also Clean Room works, because you're willing to pay.

To quote,

code quality is a function of thought and reflection, not bug prevention.

Sadek Drobi replied that

Code quality is not only about code simplification, but also and most importantly about reflecting correctly the domain (see DDD). TDD is applied to units, you can not build a coherent quality code while thinking locally. That is why TDD is not the "Tool" to quality code. Quality code is a result of good design skills (thorough domain analyses), a lot of discipline, and Functional tests.

TDD is not necessarily applied to units (I know I wrote on InfoQ that I agreed, but I was thinking of Unit Testing and not about TDD), but it's true you cannot build a coherent quality code while ONLY thinking locally. But it surely helps, and this is particularly true if when Sadek talks about functional tests he is referring to automated functional tests.

Sadek then pointed out how TDD is about short cycles and thinking locally, supporting this idea quoting Beck, and stressing the point that TDD does not produce quality code per se and reiterating the importance of design skills, domain analysis, discipline and functional tests.

Again, I agree with him: TDD surely helps to produce good code but is not "the" tool. I mean, if you have a perfectly analyzed class which perfectly represents a domain object (I'm fond on DDD too) but you find it difficult to be tested, probably there is a slightly different and slightly better metaphor to describe it (and maybe an SME can help in defining it). Being able to write a test for a class before actually writing it kind of "forces" you to have an attitute just like the one Sadek describes: you have to analyze the domain with a lot of discipline, because with TDD you are actually saying "what am I expecting from this piece of code? what is the expected behavior of the real object it represents?". Of course, you must also have funcional tests, and having good design skills is almost mandatory, otherwise one would blatantly miss the point of the whole question.