Process improvement is a lot like software development. Making incremental, iterative changes works well.
Methodologies are aggregations of assumptions, prescriptive heuristics, and entertaining narratives. There are many, I love them all, they make great bedtime reading. But as a project manager you need to choose a process that is appropriate to the situation. That requires a deep understanding of the assumptions and differences between methods, a deep understanding of your project's characteristics, and a willingness to adjust the prepackaged prescriptions to your situation.
Adopting, or even changing, a methodology is very expensive. Doing so without proper reflection, without making sure that you're addressing the problems that need fixing, is a wasted opportunity (though it's done all the time).
Rapidly making modest improvements (ie evolutionary design and staged delivery) is the core maneuver at the heart of the agile methods. But why just use it for software development? The same precepts and problems (eg limited visibility) apply to the organization's methods. Instead of radical change, I suggest we're better off solving one problem at a time. Identify a process flaw, fix it, iterate. Stop when none seem worth fixing. Adopting a method "freeze-dried, out of the box" is likely to cause a lot of bewilderment, because half of the processes won't address real problems in your organization.
Process definition is great if you do the same thing over and over again. But what if this is your first time, or your world is constantly changing? Then perhaps what you need is process flexibility. You need to quickly recognize when and how processes need to change.
One way to assess processes is to constantly vary them - try doing things several different ways - and select based on the experience. This clearly works for nature; it might work for you, too.
Defining processes is time consuming and expensive, so once it is accomplished people are loathe to revisit and revise them. Ironically, then, defining processes often leads to a stagnation in the improvement of those processes. This is by far the greater evil.
A straight reading of the CMMI model suggests that process maturity is hierarchical and sequential - that a software development organization goes through stages of maturity like a developing insect, ultimately reaching the master-of-the-universe "process optimizing" stage, having previously gone through the less capable juvenile ("defined processes") and larvae ("individual heroics") stages. That's not in accord with the facts. Many small organizations (eg, startups) have rather poorly defined processes. Yet they optimize their processes very quickly and effectively, and are unquestionably much more productive than other organizations with impeccably defined processes.
A team with everybody in the same room is different from 3,000 people in 4 countries. Don't pretend the same process is going to work in both situations. Introduce processes to solve the specific problems that you need to solve. Meaningless process wastes everybody's time.
Too little bureaucracy leads to things falling between the cracks, confusion, miscommunication - things that are clearly not good. Too much bureaucracy leads to a stultifying death of initiative and responsibility. Decide which is worse and aim accordingly.
Processes need to make sure they don't take the fun out of writing software. Really. I suppose you could replace the word "fun" with engaging, or rewarding, or something similar, but "fun" works for me.
Making software is very hard. A programmer needs to concentrate on a monitor all day, thinking very carefully about something very small and at the same time being open to all sorts of weird things happening to his code. It isn't socially satisfying. The pay is mediocre. Why does anyone do it at all? Because it can be fun to make things work, and there aren't all that many opportunities in contemporary life to make things work.
Programmers become programmers because at some point in their life they discovered that programming could be fun.
On occasion I stumble upon a neo-Calvinist who deeply believes that work should not be fun, that expecting fun is for whiners, that having fun is somehow immoral (cf Dilbert for additional analysis). Fine; I don't get it but so what. However, I can point out with some certainty that if a programmer is not having fun his productivity will decline, the bug rate will rise, and not only he but also you, Mr Manager, will come to lead distinctly less privileged lives.
So ixnay on the ocesspray that takes the fun out of programming.
By the way, fun is not beer parties on Friday and going to the movies with your teammates, regardless of what the team management books tell you. To every techie I know, fun is seeing a creation come alive. Fun is solving a tough problem. Fun is getting something done that no one has ever done before.
How can process kill fun? It isn't inevitable. But here is a starter list of some very common, big league fun killers:
• A process that forces you to do lousy work. This is a tough one to deal with, because sometimes very real schedule or resource constraints demand that you do lousy work, and there's just no getting around it. But it isn't fun.
• Creating design artifacts and documentation which take hours and days to do that no one will ever look at. Designing and documenting are not inherently good; like every other element of a project, they are beneficial only insofar as they move the project forward. Unfortunately, much design and documentation work fails to meet that criterion.
• Whimsical decision making. Few things are more irritating than having a carefully considered decision overruled by someone in the hierarchy who feels a need to put their personal stamp on something, logic be damned. Managers have an absolute obligation to be rational, and explain their decisions, else their subordinates will have no fun (and soon, no initiative, either).
• A lack of control. If everything is specified to the nth degree, rightly or wrongly, then what I do will not be fun. Hint: so don't over specify things that the programmer can probably figure out himself, as he'll probably do a better job of it and it will definitely make his job more fun.
I suspect the original creators of C and Unix at Bell Labs back in the seventies had fun. It didn't seem to keep them from doing good work. I think a lot of open source people are having fun now. It's not keeping them from doing good work, either.
I'm astonished at how rarely a process prescription will make explicit its assumptions about scale. Is it right for 10 people or 1000? Has anyone tried it with 10 people? With 1000? It's ridiculous to assume that projects of such different scope require the same processes, yet how often are the prescriptions tuned to the scale? How often are the limits made explicit? Whenever I pick up a book about or listen to someone's opinions about process, the first thing I do is figure out the size of project the author is assuming.
Don't pretend that the two developers on your team who can't even maintain a civil discourse in a status meeting can become a productive team of pair programmers. Don't pretend sending everyone off to training class for a few days is going to solve your management problems. Don't pretend that the nice guy in the corner with a room temperature IQ has what it takes to be your next architect. Instead, start with a close look at the reality of your organization, its assets and problems, and go from there.