About planning and emergency

The title was inspired by Davide Bianchi, that posted out of his office a sign that reads "Pianificazione schifosa dalla tua parte non è emergenza dalla mia parte", which more or less means "if you don't know what planning is about, don't come here complaining, because I have my own priorities". Or, to quote something a little farther on the timeline, "frankly, my dear, I don't give a damn".


It is true that no plan survives the contact with the enemy, but planning is essential. It is one of the core practices of risk management, which, as Tom DeMarco and Timothy Lister write in Waltzing with Bears, is "Project Management for Adults". What are the risks? You have to look ahead, and decide how you want to deal with them. Of course hoping for luck is an option... but how good is it? Sometimes you can count on it, maybe because even if your luck abandons you it'll affect but a small portion of the whole project. But, honestly, do you really think that none of those twenty or thirty bad events is actually going to happen? Do you know what the odds are for a chance like this? I was pretty good at statistics, and as far as I can remember the answer is "very, very, very low".

And this is when emergency comes into play. And, obviously, as there were no risk mitigation plans, there are no contingency plans, and your luck is having a holiday, there is a problem. As strange as it may seem, this comes as a complete surprise to many.

When planning fails (if ever there was any) the hunting season for scapegoats open. Little matters the fact that the scapegoats themselves have spent a good part of the previous months asking for a plan, or at least for directions, and trying to raise alarms of all kinds. Don't think scapegoating is something you can ad-lib, it is an art: there is also an article about the art of scapegoating in IT projects. Wikipedia reports that "Scapegoating is a known practice in management where a lower staff employee is blamed for the mistakes of senior executives. This is often due to lack of accountability in upper management."

Unlike as in Davide's office, in my experience bad planning by someone else does transform into an emergency happily hopping and bouncing on your desk. At this point, at least in the eyes of the stakeholder, quality becomes absolutely tradable, because the deadline has stopped soaring and is swooping down on them. Too bad, as what seems a saving today will cost blood and sweat tomorrow (not in a month, but within a week). Ok, maybe not blood. But I can guarantee for sweat - and money. Moreover

poor Martin Fowler
will feel so sad for all that
which is way too bad

featuring both a haiku and a link to an interesting article at the same time.

Some hints from a Scrum Master

We've seen and read things like this a thousand times, but it is always worth repeating them: take a look at this post and you'll get some valuable suggestions. While you're there, you'll also want to check other Alex's posts.

Embrace Change

Not only the subtitle for Extreme Programming Explained. Changes happen, that's the only thing you can be sure of. Things change, people change, hair styles change (actually I'm quoting here). So we must be open to change.

I've seen POs swap projects, new projects created, weeks of planned work cut, people added to workforce, people cut from workforce, and much, much more in less that a fortnight. Is that bad? it is, if you're lost with no hints. On the other hand, it is also an opportunity.

If you spent some time in serious planning you have a good basis on which you can build your ability to steer the change. That does not mean that you need a good plan (which could now be completely worthless), but that you have a solid knowledge of what's on the table, and that means more information, which translates into a conscious decision instead of a jump in the void (or at least it mitigates the risks).

And let me say that again:

The waterfall model is wrong!

Agile and procedural

Today I partecipated in a discussion started by Dennis Morton who asked if anyone had succesfully adopted Scrum, XP or RUP on non-OO procedural based applications. A little rephrased, these are my thoughts.

Scrum is relatively easy. Implementing Scrum could not be that easy because you have to face several impediments: one of the biggest ones is that it clashes with the existing culture, but that does not depend on the particular language you use.

As Keith and Charlie pointed out, life is easier for OO programmers, as there's plenty of relatively inexpensive tools and technologies (if not inexpensive at all) that can help them: Junit, Cobertura, JMock, EasyMock, Hudson, CruiseControl, Eclipse, NetBeans, and so on and so forth in a sparse order, just to talk about Java.

I don't know of similar tools to be used in RPGLE ("inexpensive" and "IBM" cannot share the same sentence) but that might just be my ignorance, so it is up to the team to find a suitable solution (that could also be an expensive but affordable tool).

Anyway, I have to point out that Scrum is just (?) a very good set of techniques for project management, but it is not enough, as you must have in place the proper engineering practices to benefit from the advantages that Scrum offers.

It is useless to give a product owner the possibility to steer the project at every sprint planning meeting if a simple change requires tons of programming hours, as she always has to weight benefits against cost. You can be agile because you have test harnesses supporting your changes. You can be agile because you continuously refactor. You can be agile because all the team members own all the code. You can NOT be agile just because you use Scrum, as you're just exposing problems - problems that most of the times existed well before Scrum was implemented, so don't shoot the messenger. You also have to master the tools to resolve problems. And the first and most important tool is people, so I'm completely with Keith and Charlie who put them in the heart of the process.

We have to undergo a similar challenge as we'll embark on a very big project next year, almost completely based on IBM technologies. As we're starting from about 650 pages of detailed requirements, aged about one year, a remote customer, a distributed team with several new members (not to mention the rest), we'll surely have to cope with changing requirements (no, not the band in which Craig Larman plays in his free time) and lots of other variables and issues.

It is likely we'll have some answers within the next few months; up to now, as it is well understood that we'll have an application layer exposing services, the only thing that I could think of is the use JMeter as an acceptance testing tool. As always, inspect and adapt, rinse and repeat :-)

We're open to suggestions!

First draft requirements: lessons learned

I recently talked to a friend of mine that had just terminated a preliminary collection of requirements for a couple of new customers; I found his insights very interesting for a bunch of reasons, and I'd like to report some lessons he learned and was so kind to share. Having seen the result of his efforts I'd like to add a couple of thoughts of my own.

• Having just a person conducting the interviews and another one working on the resulting minutes is not a good idea (we're not talking about polishing them up), as it is very time consuming. Better to spread the work among two persons and interact frequently between sessions: even if it might seem more expensive, as it doubles the resources involved, it greatly reduces further analysis and review time and leads to better results (likewise pair programming). This practice also reduces bottlenecks.
  
• A versioning system is very useful. Use it.
• Time spent on "aesthetics", like formatting, logos and such, would be better spent at the pub. Ops, I mean concentrating on the core stuff. A good look should not be important to you. If you really need it (you normally do when you approach a new prospective customer) leave it for someone else when the process is (almost) terminated. Better yet, separate presentation from content.
  
• Don't forget nonfunctional requirements. I am always astonished when I see that nobody cares (or remembers) about security, scalability, reliability, configurability, performance, and so on. I can understand a customer might not think of maintenability at first, but how is it possibile that nobody ever says things like "the system should support no less than 2000 concurrent users" or "the system should adapt to many different situations"?
• Inspect and adapt. Rinse and repeat.
  

Note that the purpose of these interviews was the production of a first draft for a possible collaboration, not working software; nevertheless the same rules hold true also in production time.

Scrum vs Lean?

Following this discussion I posted my thoughts on the subject. Even if I have just become a CSM, I have to agree about the importance of the certification itself; as our instructor Joseph Pelrine said, "I can only certify that we breathed the same air for a couple of days". This feeling seems to be shared by most hirers in the market. Nevertheless, I learned a lot in those two days, and I also had the opportunity to meet many skilled guys who share my very same interests.

That said, Scrum is about addressing the chaos you tipically get in a complex and fast changing environment, not about developing software: e.g. I always use it with my wife whenever we have to “refactor” our garden (ok, that’s not too complex an environment, but I’m sure you get the point). If you’re looking for practices to improve the quality of software, you’re looking in the wrong place. One word of caution: Scrum really fosters software quality improvement, but it does not provoke it. People improve software quality.

Scrum is not bad because you see many bad Scrum implementations; I’m sure you can find bad implementations of almost everything (OO is not bad because many programmers disguise their procedural code - which is not bad per se, but the mix of the two is often weird). I want to stress that a Scrum Master does not tell the team what to do, as he is not the leader; he is a servant to the team: sort of an istance of the “sacrifice one person” strategy by Cockburn, except maybe for the fact that in the latter you normally assign the sacrificed person a particular task.

I also agree with the fact that a team needs good leadership thus a good leader, even If I'm aware that many will not agree with that. I also think that the leader should emerge naturally in the team, and should not be superimposed. A team without a good leadership (I'm talking about an "internal" one) could easily end up out of action, especially an inexperienced one.

Transition from Waterfall to Agile

Jatin Leuva asked what is required when an organization is transiting from waterfall to agile: this is an interesting, and very common, question.

First of all you should have good reasons for the transition: if waterfall works for you, you have no reasons to change it. Unluckily this is almost never the case.

So your process does not work, and you have to change it. I'd start with retrospectives: what went well, what you'd do differently, what did you learn, what still bothers you. This can give you a start.

Meanwhile, you have to work at higher levels, as agile methods require a shift of paradigm with respect to standard management techniques: managers have to trust people and must resist the impulse to tell them what to do, and this generates fear: fear of losing power, fear of being useless, fear of loss of visibility, and so on and so forth. Unless you dispel this fear, your organization will never get agile, as managers won't allow it.

That said, a very good way to start is to hire an agile expert to help your team (and your managers) get confident with the new process - I should say with the new attitude: your team must face a shift of paradigm too, as team members will not be told what to do anymore, but they shall commit to deliver useful software. They'll have to earn the trust that managers should have in them, as it won't come for free. It's not about skill, it's about committment.

Should IT projects be insured?

I recently posted a comment on a discussion whose topic was similar to the above; the the initial question was related to the very high failure rate of IT projects.

Most of my experience refers to IT product development, so I'm not talking about other things such as insuring an IT project against political instability. As other contributors have pointed out, insurance companies will consider all the risks involved, which means that with such a low success rate software projects are not so likely to be insured, at least as a rule of thumb.

Most of the times the problem lies in the process used to manage the projects, so a better process - which can result in a more motivated team - is often the best insurance you can get.

More on this on this great book by Craig Larman.

Sustainable pace

I read a discussion about Sustainable Pace, one of XP's practices, in which the contributors try to relate the subject with a fixed number of hours per week. Literature (based on extensive studies) shows that there is no linear relationship between productivity and hours worked, and I quite agree with that; moreover, there's a law (you will forgive me, but I don't have the reference at hand) that states that if you have a certain amount of time to do a job this will expand to fill the whole time slot available. Sistematically working overtime does not increase productivity in the long run (not even in the medium run); one of the (many) reasons is that software production has nothing to do with traditional industrial processes. My personal opinion on that is that more hours do not guarantee an extra productivity; nonetheless under exceptional circumstances (which is to say not every week) extra hours can be invaluable.

Here in Italy the "normal" working week is based on forty hours over five days. A contributor enthusiastically sustains that one extra hour a day guarantees "more than one month of extra development time during a year!"... or does it? I strongly disagree.

It's not unwillingness to work, or (better) unwillingness to get the job done; it's about commitments and balance between work and private life. In an agile enviromnemt you only plan for the next (very) few weeks, so the estimates are much more accurate and, as the project goes on, you have a pretty good knowledge of your productivity: that allows you to commit to a fair amount of work without steadily subtracting time to your private life. An extra hour in the office is an hour in which I can't play with my children, talk to my wife, read a good book, oble a new biscum or burble the tramling (Cockburn readers might get the reference) or whatever. That can lead to a burnout, thus resulting in a great loss of productivity. I already spend more time with my colleagues than with my family, and I think there's really no need to make things worse.

Besides... how many people on their death bed say "I regret I have not spent enough time working extra hours..."?

Top Ten Actions for IT PMs

I happened to see a very interesting question on LinkedIn: "What top ten actions can IT project managers take to increase the likelihood of implementation success?"

I think that each of these actions could easily fill more than a book, but I'll try to share some of my thoughts, in no particular order.

• Choose the right people. As I already quoted elsewhere, blockheads will never produce good code. I know it might sound nasty, but that's how it works.
• Build a team with the people you chose. That means you must end up with something which is better than the sum of the parts.
• Respect your team, don't enslave them. Trust them to be able to deliver if they commit to. Happy and motivated people always perform better.
• Go agile. Waterfall is one of the worst causes of software project failures.
• Protect your team from external influences. Once the goal for the iteration is fixed, stick to it (hey, have not you gone agile yet? didn't you read the previous item?).
• Deal collaboratively with the customer, do not drive the hard bargain when you negotiate but look for a win-win situation. Involve him, involve him deeply, involve him early.
• Often show your team's progress to the customer, and involve him in acceptance testing.
• Avoid WBSs, plan by features. Let the team manage the WBS internally and only discuss the features with the customer.
• Always communicate clearly the status of the project. Bad and early informations can become opportunities, bad and late informations only reveal a death march. And it will be too late to avoid it.
• Plans are nothing, but planning is essential. Adapting to new informations is much better than sticking to a plan which, in the end, will satisfy nobody.

I'm sure there are lots of other tips, but these will do for now.

Estimating blasphemy?

I am working with my team on a new project; we still are in the Inception phase, so even if our ideas are quite clear there are also many foggy aspects. Anyway, we've identified almost all the stories we shall need (the domain is well known) and we are estimating them with story points.

This project will be quite different from all the others we've worked on for many different reasons, so there will be a lot of uncertainity; I then reported to the management that our first estimate will actually be a guesstimate, thus possibly ranging from 60% to 160%. The Project Management Institute (PMI) has an equally broad, but more pessimistic, range for this stage of the project (75% to 175%).

As expected, the management reaction was less than cold, and I was told we had to narrow the gap between the worst and best estimates - a perfectly legitimate request. The problem is that we do not have enough informations at the moment, and only when we'll have an approved product definition we will be able to get to an 80% to 125% range (close enough to the PMI budgetary estimate), but luckily that should happen in a few days.

I'm afraid that will not be enough, because the second question after "how big will it be?" is always "how much will it take?", which obviously leads to "how much will it cost?". Estimates in story points are only a pure measure value, and a duration must still be derived. We cannot use our historical velocity values because, as said, this project is very different from the ones which generated them, so we'll have to run at least a couple of small iterations, observe the values and make a more or less reliable forecast.

Luckily our managers deserve their jobs and they understood our motivations, so I've been able to reach an agreement upon a spike in which we will very lightly touch all the layers of the application and the new technologies we've identified in some previous pilots.

This was another example of how an honest conversation can lead to a win-win situation: the team will not have the pressure of an irrealistic plan and the management will be able to take decisions and actions on the basis of sufficiently reliable informations.

Agile Estimating and Planning

Yesterday I finished reading Agile Estimating and Planning by Mike Cohn. I found it quite interesting, even if id didn't add very much to what I had found in other readings; the good thing is that the whole process is explained, delving into different aspects (some of which are too lightly touched, in my opinion).

Estimating in Ideal days

When it comes to estimating you have different options to pick from; two of the most common are ideal days and story points, each of which has its own strenghts and weaknesses. One of the worst weakness of the ideal days method is that an "ideal day" means very different things to different people; moreover, different team members give very different estimates depending on their relative skills, so you can have a very wide range of estimates for a single item to deliver (call it use case, user story, feature, you name it).

If you're used to ideal days estimating and decide to stick with it, there are some tips that can help you to reduce the risk associated with the method.

The first (and obvious? unluckily not so much...) tip is to let who will actually develop the software do the estimate. Managers do not have enough informations, so they must trust the team. After all, a great slice of management work is to select collaborators, so if a manager trusts himself (and you bet she does) she must trust the people she has chosen as well.

Moreover, more than a single member should estimate the same item; actually the whole team should do it, thus exposing - early - different points of view; more often than not, if somebody estimated an item much less than other members she forgot something or did not considerate all the implications of the item: the key to effectiveness of this practice is the constructive dialogue between developers. A particular way to do it is planning poker: all developers have a deck of cards, each of which indicates a valid estimate; a moderator reads an item from the list to be estimated and each developer choses a card, then everyone turns up their card at the same time. If the estimates are more or less alike there is a short conversation and an agreement is reached, otherwise there can be a deeper discussion, after which there is a new turn.

A deeper problem emerges when developers are differently skilled: they might agree on the relative size of the items, but their estimates will differ greatly; an item which would take an experienced developer two days could require ten from an inexperienced one. On the contrary, if all developers in a team have more or less the same skills they can ignore the problem. If that is not your case (and surely - and unluckily - it is not mine) you have to deal with the problem of scheduling also: if you estimate on the experienced developer's words you have to assume that she will do the work, but you cannot have the guarantee that when the time comes she will be available, so you'll have to spend ten days but you can bill for only two of them; if you choose to take the estimate of the inexperienced developer the customer could complain because your team costs too much. In this cases I normally estimate somewhere in the middle and do all I can to have the two developers pair on the item to leverage knowledge and experience.

Last but not least, ideal days are different for different people: someone consider ideal days as "eight hours focused on a single task", someone else as "six hours focused on a single task, plus a couple of hours on different things". Obviously that brings to different results, and the bigger the project the worse the outcome. That also clashes with scheduling, because people should manage their time but that is not always easy, particularly regarding how to deal with (internal and external) interruptions. To deal with this risk you have to know the velocity of your team members and of your team as a whole, so that you can translate their estimates into sufficiently reliable schedules.

Most of the advices are also useful for estimating with story points.

A legacy application: the story of the fridge and the oven

Today I'll tell a story which inspiration comes from an overheard conversation.

The situation: you have a legacy application which you use in your organization to provide services to your customers. You have no tests. It uses proprietary libraries. Layers are nowhere to be seen. A simple change requires a gargantuan amount of code. And sweat. And chills down your back. You know it will not be able to provide the new services the business needs without a huge architectural and model change.

A possible solution: start a new project adopting an agile and iterative method, focusing on the most important architectural aspects (an UP practice) and on the most valuable features (a practice you find in many methods). Incrementally substitute the old system with the new one (an EVO practice).

The solution from the management: learn everything there is to learn about this application: it is one of our assets, and as such it must be considered. When your experience is consolidated, we'll reuse the same framework for all our new projects.

You can consider this story from two different points of view. The manager, who has no development experience or knowledge, thinks that the problem lies in the team, and fails to acknowledge all the signals the team is giving him. He is afraid and wants to keep things under control. After all, if we have provided the service so far we can go on just as well. Why venture into a new project, with all the risks that new technologies imply, first of all the team's inexperience with them? Another perspective: the management might have dealt a sale of the product within a very short time frame - this might be a confidential information - and he knows - or he's afraid that - the team will not be able to deliver on time.

In each case, most of the times an honest conversation between the management and the team can lead to find a good, if not the best, compromise: a classic win-win strategy.

One last (funny? tragic?) note: the project manager, trying to explain the problem to the management, provided this metaphore :

"Look, we have this huge refrigerator. You need an oven. You're asking me and the team to learn how this refrigerator works, so that we can embed the oven within it. What we really need is a power plug which can work for the fridge and for the oven. Why would you need a fridge to roast a turkey?"

The answer:

"Your oven will have frosting features, which no other oven has. That will be a significant competitive advantage."

Agile budgeting

How do you build a plan for a new software project? There are different approaches, each of which has its peculiarities; I shall discuss them shortly.

Agile teams are very well aware of the fact that requirements do change, so they don't try to prevent it but ride the wave, focusing on delivering, iteration after iteration, the highest priority requirements: they are happy because they will deliver high quality software, they know their customers will be happy because they will steer the project according to their evolving needs... and they lived happily ever after. Until they talked to the management:

WHAT? NO FIXED PRICE? NO FIXED SCOPE? POSSIBILITY FOR THE CLIENT TO CUT THE BUDGET? AAAAARGH! YOU'RE FIRED! YOU'RE ALL FIRED!

I'm exaggerating of course, but I'm sure you get the point. Well... OK, that's what we'll do: we shall meet the customers, explain to them how they will get only the functionality they really need, how they will always be in control, how they will decide how their money will be spent, and all the other advantages of an agile incremental and iterative approach. So they will talk to our management, persuade them, and tell them they are really enthusiastic with our proposal and that we should get a considerable pay raise. Actually they go more or less like that:

WHAT? NO FIXED PRICE? HOW MUCH WILL IT COST? HOW LONG WILL IT TAKE? AAAAARGH! GET OUTTA HERE! GET OUTTA HERE RIGHT NOW!

Well, we have to reconsider... Agile approaches are really great, but the difficulty lies in persuading a traditional management. Let's take it from their own point of view.

In a traditional approach you spend a lot of time to write a long (and when I say long I mean very, very long) and comprehensive requirements document - which probably no one will ever really read - which in turn is, more or less, accepted by the contractor. At this point you have already invested a considerable amount of resources, but project managers have enough informations to precisely schedule everyone's work for the next months, and that allows them to estimate a fixed price. That makes the management happy because they think they are dealing with the risk of scope creep in an effective way, and customers happy because they know in advance the amount of money to allocate for the project.

That sounds great. But let's face the facts: studies show how 19% of the initial requirements are only rarely used, while 45% are not used at all. And the requirements have changed. They have changed a lot. But the plan did not. So, when the project comes to an end, customers, which have paid a lot of money, will not be happy because they won't have what they really need, managers will not be happy because customers complain, and the blame will be "on those weirdos who developed all this crap".

How can we have happy developers, happy managers, happy clients? a succesful approach is to divide the project in different phases, just like in UP, and contract for the elaboration phase at a fixed price, than use an agile Just-In-Time (JIT) approach for the construction and transition phases. During the inception you only get enough informations to estimate the scope of the project (yes, with this intermediate approach we still want to avoid big creeps) and to do enough architectural modeling to have a sufficient basis on which to produce an initial schedule and budget. In the next phases you let customers steer the project with an agile approach: they'll be more willing to do it because they'll already have the core of the new working, although incomplete, system, and the risks will be smaller. The management will have a reasonable estimate, which will improve iteration after iteration.

Still too extreme? You can follow UP and have two fixed price contracts, the first one covering the elaboration phases, and the second one covering the construction and the transition phases.

And you would get that pay raise, after all.

Forget that. "Money, so they say, is the root of all evil today. But if you ask for a rise it's no surprise that they're giving none away". Thanks to Pink Floyd for the tip (pun intended).