If all (or even most) systems projects went well, we wouldn't need to talk about this subject …. But they don’t, and those failures compromise the progress and integrity of the information world.

The selection and application of formal methodologies is an important ingredient in the success or failure of most projects. In today’s automation project world, choosing the right project methodology sometimes seems as important as the project itself. Choose the right methodology from among the top contenders -- Six SigmaAgileLean and SDLC -- and, we are told, you are on the road to success. Choose poorly and you are alone on the windswept plain of project failure or mediocrity at best.

But even projects that claim application of rigorous methodology often fall short of their hoped for outcome. How can that be?

Most of today’s most popular methodologies, at least under their current names, have been around for a while -- Six Sigma since the 1980s, Agile since 2001, SDLC since the 60s, Lean since 2003 -- each seeing major growth as a required ingredient for management of systems projects.

The question for many people is how well we have done selecting and using these methodologies.

Outside each methodology’s cheering section, the answer appears to be mixed, with greater success in more detailed, technology focused projects, but less in the growing percentage of more lifecycle focused efforts involving groups and considerations throughout and external to the organization.

Getting Down to Basics: The Underlying Principles

A close look at the philosophies underlying most methodologies uncovers an interesting fact: all are based on a set of time tested precepts, expressed in differing ways, but the same nonetheless:

  • Include all relevant groups (customers, developers, managers, etc.) and encourage bonding and team spirit among them.
  • Establish and maintain strong communication, collaboration and information sharing among all participants.
  • Distribute authority for important decisions where and when appropriate.
  • Use rapid and iterative prototyping to identify, clarify and respond to user needs.
  • Use rigorous control methods where appropriate.
  • Keep entire picture in mind as project proceeds.

If this is true, then we might agree that a successful project must include these. The question then is how to make sure they are properly included. Experience with use of the current methodologies tells us that how things are done today isn’t the answer.

The problem is partly due to the fact that starting from the core principles, each methodology develops and applies increasingly detailed tools and techniques -- some say to claim bragging rights -- for implementation in specific projects. The problem with layering of specifics on general principles is that given the choice between the two, many projects find themselves applying the specifics, but ignoring the foundation principles that are the real objectives.

Moreover, while failures in projects based on these approaches are often chalked up to failures in application and lack of management support within the organization, some also blame the fact that most of the methodologies, developed in and for a manufacturing environment, were designed to improve repetitive processes -- Six Sigma from manufacturing at Allied Signal and Motorola, Agile from widespread frustration with the ponderous methods left over from early software development and Lean from automobile manufacturing -- and may actually not be useful in the one-off world of systems development projects.

It appears that checking the right boxes, assigning the participants catchy names and holding the appropriately titled meetings do not a successful project make.

Concurrent Engineering, Antidote for Misplaced Detail?

So what should we do when faced with a complex, lifecycle oriented systems project -- the type increasingly encountered in today’s connected world?

The underlying principles described above are sound and should be part of any project. So the question is “what approach will best enable us to apply and profit from those principles with a minimum of overhead and distraction?” Many suggest that the methodologies mentioned above are not the best paths to apply these key principles to a systems development project.

One valuable alternative may be right in front of us, embodied in a methodology that, although around since the 1980s, has largely disappeared from the marquee of leading approaches. That methodology is Concurrent Engineering (CE). This product development approach was created in manufacturing but adapted -- and useful -- for a wide range of systems projects in many settings. It is founded on the core principles described above, but stays with them, organizing projects to apply and profit from the principles that other approaches give only lip service to.

Concurrent Engineering grew from the frustrations and failures of traditional product design and manufacturing efforts and from the often tortured relationships among marketing, product design, manufacturing engineering and executive groups, none of whom were particularly interested in technical detail for its own sake. With these differing attitudes at the table, CE had to deal with and include, sometimes painfully, considerations that other methodologies added later but in many cases still do not fully understand or support.

The result was a methodology that encouraged the communication and information sharing needed with a minimum of overhead: something that other approaches identified as vital but have failed to achieve. Six Sigma suggests that customer-driven requirements should be replaced by requirements analysis from the development team itself. Agile encourages developers to focus on “emergent design,” bypassing up front analysis and design and letting requirements emerge as the project progresses. This sounds great and may work … unless a project has multiple phases and components, all of which must be kept congruent and mutually interactive. In such cases, up front planning is key to getting -- and keeping -- everyone on the same page.

Concurrent Engineering’s genesis in the world of product conception, design and engineering, confronts head-on the complexities associated with getting everyone involved in a product development on the same team and working together to the team’s benefit.

While some say that CE, developed in manufacturing, isn't appropriate for software engineering based projects, Six Sigma and Lean share these manufacturing roots. CE’s application to software-based efforts is discussed in some detail in scholarly publications [1] [2] [3], and receives generally positive evaluations.

What Are We to Do?

In a world that demands ever more complex systems, doing nothing is probably not a good option, especially when project failures often render the resulting systems, and their users, vulnerable to attack by hackers and cybercriminals.

But it appears that more rigorous application of today’s favorite methodologies won’t solve the problem either, and the chaos of no management approach isn’t the answer. So what are we to do?

If virtually all of today’s methodologies are based on a similar set of foundation principles then maybe we should focus on the principles instead of the methodologies. Look closely at the way many projects use methodologies like Six Sigma, Agile or Lean and it becomes clear that once the focus turns to the artifacts of the methodology, the foundation principles are lost. Even the certification programs associated with these methodologies focus more on how to apply the details than on how to observe the foundation principles, with sadly predictable results.

If we agree that projects need some unifying structure, the methodology with the fewest layers of detail seems likely to best keep us on track. Concurrent Engineering fares well here because it has stuck closely to its underlying principles, teaching us how to apply them effectively even as it asks us to change the way we organize and manage projects, working together in a mature and professional manner.

Rekindling interest in CE won’t be easy. It isn't easily bolted onto marginal situations, and there are major and passionate constituencies around today’s methodologies -- but the effort, even if only partially successful, will be worth the trouble.