Today, IT managers have a bewildering array of quality disciplines to choose from. Some, such as Six Sigma, ISO 9000 and the Malcolm Baldrige program, may be dictated to you by your CEO. Others, such as Control Objectives for Information and Related Technology (CobiT), may be imposed by your auditors. And IT-focused disciplines may originate in your own shop, such as CMM for software development and the Information Technology Infrastructure Library (ITIL) for IT operations and services.
While there is some overlap among these quality frameworks, in most cases, they don't conflict. Indeed, most large companies use two or three of them. For example, IBM uses ISO 9000, CMM, ITIL, Six Sigma and several homegrown quality programs.
Meanwhile, other equally sophisticated companies don't use any of them, preferring to roll their own. For instance, MasterCard International Inc. has adapted parts of a number of programs to its own way of doing business. It underwent an external assessment for CMM a year ago and implemented some ideas from that, but it hasn't adopted the framework formally.
"We have a hybrid of quality programs," says Sheryl Andrasko, vice president for systems development at MasterCard. The program has reduced the development time for new software releases from 18 months to 12 and has reduced the number of software defects as well, she says.
Other companies, such as Nortel Networks Ltd., say the choice should be driven by customers and partners. Nortel uses a telecommunications-oriented version of ISO 9000 because that's what its customers use.
|
|
| Credit: Maria Rendon | |
But a company can overspend on any of these programs, says Matt Light, an analyst at Gartner Inc. "We have a philosophy called 'just enough process,' " he says. "So to roll your own and apply it just where it makes sense is often the best choice for organizations that don't have certification requirements."
Nevertheless, you should do something on the quality front, urges Michael J. Ashworth, CIO of the investment banking unit at J.P. Morgan Chase & Co. "All of these things are just better ways of doing the things that people are trying to do on an ad hoc basis," he says. "They are not mumbo jumbo; they are codified common sense."
Capability Maturity Model Integration (CMMI)
Sponsor: Software Engineering Institute, Carnegie Mellon University
What it is: The CMMI extends and combines the Capability Maturity Model for Software (SW-CMM), the Systems Engineering Capability Model and the Integrated Product Development Capability Maturity Model. SW-CMM is a collection of best practices for software development and maintenance. It allows companies to assess their practices and compare them to those of other companies. The SW-CMM measures process maturity, which progresses through five levels: Level 1 (initial), 2 (managed), 3 (defined), 4 (predictable) and 5 (optimizing).
Strengths: Very detailed. Geared specifically to software development organizations. Focuses on continuous improvement, not just on maintaining a certification. Can be used for self-assessment.
Limitations: Doesn't address IT operations issues, such as security, change and configuration management, capacity planning, troubleshooting and help desk functions. Sets goals, but doesn't say how to meet them. (For example, CMMI says to do requirements analysis but doesn't say how to do requirements analysis.)
For 15 years, companies that wanted to significantly improve their software development practices—and earn a merit badge for all the world to see—embarked on a long, hard road called CMM for Software, a road map that can lead companies from a state of semichaos, where most are today, to one marked by the precision, repeatability and low error rates normally associated with a manufacturing assembly line.
CMMI, recently unveiled by the Software Engineering Institute, is a more comprehensive process-maturity framework that combines SW-CMM with broader disciplines in systems engineering and product development. The institute says it will eventually stop supporting SW-CMM in favor of CMMI.
The IT shop at J.P. Morgan Chase uses SW-CMM, while the company overall works under Six Sigma. "We've got our development teams up to CMM Level 2 and are pushing toward Level 3 in some cases," Ashworth says.
Ashworth says the move from Level 1 to Level 2 brought with it more reliable planning, so application features are more likely to be right the first time, reducing costly rework. The investment bank has seen the following additional benefits, he says:
Nevertheless, Ashworth cautions against "analysis paralysis" when it comes to evaluating the results of CMM. "We found it not useful to spend too much time trying to measure things, rather than just doing it," he says.
Motorola Inc. has software development units at all five SW-CMM levels, but most are at Levels 3 or 4, according to Anthony Carter, director of the Digital Six Sigma program at Schaumburg, Ill.-based Motorola. He says that as groups reach Level 5, they'll migrate to CMMI. The product development framework in CMMI makes it an attractive choice for a company that makes products such as cell phones that contain software, he says.
The IT organization at Capital One Financial Corp. in McLean, Va., is at Level 1 and plans to reach Level 2 by the end of this year and Level 3 by the end of 2005, says Ray Frigo, vice president of IT management services. But unlike, say, a defense contractor that wants to become certified at a high CMM level in order to sell to the Pentagon, Capital One doesn't feel compelled to follow CMM disciplines to the letter.
"We developed a process framework to provide repeatable, consistent delivery," Frigo says. "We are picking and choosing elements of CMM and using CMM scoring to assess where we need to develop processes."
Moving from one maturity level to the next can entail two years or more of hard work, and in some cases, it's not worth the effort, users say. For example, Allstate Insurance Co. wants to move from Level 1 to Level 3 and stop there. "We really don't see the need to go to Level 4 or 5," says Robin Richmond, an assistant vice president at Allstate Protection Technology. "We can see payback from getting to Level 2 and 3. We are hoping for speed to market, efficiencies and improved quality."
And Richmond says she won't migrate to CMMI anytime soon. "It's very difficult to find people with experience in it as assessors or as implementers," she says.
Control Objectives for Information and Related Technology (CobiT)
Sponsor: Information Systems Audit and Control Association and the IT Governance Institute
What it is: An audit-oriented set of guidelines for IT processes, practices and controls. Geared to risk reduction, focusing on integrity, reliability and security. Addresses four domains: planning and organization, acquisition and implementation, delivery and support, and monitoring. Has six maturity levels, similar to CMM's.
Strengths: Good checklists for IT. Enables IT to address risks not explicitly addressed by other frameworks and to pass audits. Can work well with other quality frameworks, especially ITIL.
Limitations: Says what to do but not how to do it. Doesn't deal directly with software development or IT services. Doesn't provide road map for continuous process improvement.
Lance Turcato, managing director for technology infrastructure and security oversight at Charles Schwab & Co., calls CobiT "an IT governance tool" to help IT managers understand what controls are needed and how to measure the effectiveness of those controls. "There's an audit tool that's part of it, so that auditors can audit against those same criteria," he adds.
CobiT takes considerable effort to integrate into a company's processes. "The statements in CobiT are very generic, so we had to turn it into 'Schwab-speak' so people could understand it," Turcato says. "The biggest challenge was getting everyone to buy into it. What we had to do is determine who are the appropriate people throughout the technology group that own these controls and educate them in CobiT."
Lockheed Martin Corp. has four units at CMMI Level 5. It also uses Six Sigma and ISO 9000 disciplines in various parts of its IT organization, but CobiT is the "umbrella quality framework," says CIO Joseph R. Cleveland. He says it provides useful checklists in each of its four domains.
For example, he says, for something as simple as adding the BlackBerry PDA to the company's catalog of approved devices, CobiT will ask whether there's help desk support for it, whether security has been addressed, whether procedures are in place to acquire and maintain the device and so on.
Cleveland says CobiT fits in nicely with CMMI, with CobiT pinpointing the need for certain controls and CMMI putting them into place. Auditors' questions can often be satisfied by pointing to aspects of CMMI, he says.
IT Infrastructure Library (ITIL)
Sponsor: The U.K. Office of Government Commerce, Pink Elephant Inc. and others.
What it is: Best practices for IT service management and operations (such as service-desk, incident, change, capacity, service-level and security management). Especially popular in Europe.
Strengths: Well established, mature, detailed and focused on IT production and operational quality issues. Can combine with CMMI to cover all of IT.
Limitations: Doesn't address the development of quality management systems. Not geared to software development processes. Use is highly dependent on interpretation.
While CMM is the de facto quality standard for software development processes, ITIL for many is the tool of choice for the operations and infrastructure side of IT, particularly for IT services.
Capital One rolled out an ITIL program for internal and external customers in 2001 in the wake of very rapid growth accompanied by an increasing number of "service interruptions," says Gregory Gannon, vice president of technology delivery. By 2003, Capital One had reduced "production incidents"—such as system crashes and software-distribution errors—by 30% and had reduced "business-critical" or "Severity 1" incidents by 92%, he says.
ITIL tracks problems in IT service areas such as help desk, applications support, software distribution and customer-contact system support, and it overlaps CMM in certain areas such as configuration management. For example, Gannon says, ITIL tracks the changes made to operational systems, but the quality of those changes—in terms of the number and severity of problems resulting from them—is more a CMM metric.
ITIL facilitates root-cause analysis of problems, Gannon says. "We used to be pretty good at service restoration, but the reason we had to do so much service restoration was because we were restoring service, but not fixing the problem," he adds.
ITIL isn't a substitute for ISO 9000, Gannon says, because ISO 9000 is more relevant to certification of processes. Capital One has some Six Sigma efforts under way, but they're more on the business side of the house than on the IT side, he adds.
Six Sigma
Sponsor: Developed by Motorola Inc.
What it is: A statistical process-improvement method focusing on quality from a customer's or user's point of view. Defines service levels and measures variances from those levels. Projects go through five phases: define, measure, analyze, improve and control. The Design for Six Sigma variant applies this method's principles to the creation of defect-free products or services, rather than the improvement of existing ones.
Strengths: A data-driven approach to finding the root causes of business problems and solving them. Takes into account the cost of quality. In IT, best applied for relatively homogeneous, repeatable activities such as call center or help desk operations. Design for Six Sigma can help develop good software specifications.
Limitations: Originally designed for manufacturing environments; may be difficult to apply to processes that aren't already well defined and measurable. Can improve a process but doesn't tell you if you have the right process to begin with.
LSI Logic Corp. has been applying Six Sigma for about three years and this year will begin using Design for Six Sigma, a variant it feels is a better fit for IT environments. "Traditional Six Sigma does apply to some areas of software development, like testing. It was developed in a manufacturing environment, where there's a high volume of product," says Terry Gowin, director of quality at Milpitas, Calif.-based LSI Logic Storage Systems. "But software development varies with each project and has much longer cycle times."
Design for Six Sigma is especially powerful early in projects, Gowin says. "A lot of its focus is getting the requirements correct upfront. It helps to really tighten down the specifications, so there aren't surprises later on."
Design for Six Sigma and CMM could complement each other nicely, says Ron Engelbrecht, an operations general manager at LSI Logic. "CMM is more of an assessment tool and an assessment guide, whereas Design for Six Sigma is a set of tools designed to help you improve your scores, improve your assessments."
At J.P. Morgan Chase, Six Sigma isn't applied directly to IT processes, but it is an essential starting point for most IT projects, Ashworth says. "We look at business processes we wish to improve and do the various steps in Six Sigma to come up with a new business process model. Once you know what it is you are trying to do, that's when CMM comes into play."
Six Sigma could be applied to IT operations and services, he says. The bank is using a homegrown quality framework in that area but is considering using ITIL. "Just as we brought the naming conventions and the assets that are created in Six Sigma and CMM together into a single list that everyone can understand, we'd add ITIL onto that," Ashworth says.
ISO 9000
Sponsor: International Standards Organization
What it is: A set of high-level, customer-oriented, auditable standards (ISO 9000, 9001 and 9004) for quality management systems. Intended to ensure control, repeatability and good documentation of processes (not products).
Strengths: Well established, mature. Enjoys global prestige. Can be applied enterprisewide. Can cover software development and IT operations and services.
Limitations: Requires considerable adaptation when used in IT organizations. Focuses on repeatability and consistency of processes, not directly on the quality of those processes. Not good for analyzing a process and finding root causes of problems.
LSI Logic has been certified in ISO 9000 since 1992. It also uses Six Sigma and Design for Six Sigma. "But ISO is the broadest quality system that we use," Engelbrecht says. "It applies to manufacturing, engineering, marketing, sales and IT."
Design for Six Sigma focuses on individual projects and tries to fix the problems it spotlights, and it can "make breakthrough improvements," Engelbrecht says. ISO 9000, on the other hand, aims to make broad, incremental, year-to-year quality improvements across IT, he says. These improvements come via annual ISO 9000 audits by both internal and external auditors, he adds.
"ISO 9000 requires you to define and document your processes, get them measurable and monitor them for compliance to a quality standard," says LSI's Gowin. "Six Sigma gives you the tools, once you have a process defined, to go in and remove the variation in the process to make the output very consistent."
Nortel Networks Ltd. adheres to TL 9000, a version of ISO 9000 tailored to the telecommunications industry. Its TL 9000 certification applies to the company as a whole, but quality initiatives within IT support the certification, says Chris Ashwood, vice president for product development solutions. "TL 9000 has taken ISO 9000 a step further in really recognizing the importance of IT to the development of products," he says.
The Brampton, Ontario-based company's IT shop has a well-defined set of priorities that's updated every six months, a scorecard for every project and a strict management process for tracking accountability, says Nortel CIO Albert Hitchcock. "That very clearly aligns with the ISO approach—doing what you say you are going to do, tracking accountability and documenting the process," he says.
Malcolm Baldrige National Quality Program
Sponsor: National Institute of Standards and Technology, U.S. Department of Commerce
What it is: A high-level framework for quality in seven areas: company leadership, strategic planning, customer and market focus, information and analysis, human resources, process management and business results. Rates each of these, in terms of approach, execution and results, on a scale from 0 to 100.
Strengths: Very broad, holistic scope. Can be used by any organization. Can sit on top of other, more focused IT quality programs.
Limitations: Doesn't address process details; doesn't say how to achieve quality. Doesn't directly address IT processes and issues.
Motorola is a big user of CMM, and it invented Six Sigma 20 years ago. But more recently, it has embraced the Baldrige quality program. The company won a Baldrige award in 1988, and in 2002, its Commercial, Government and Industrial Solutions Sector (CGISS) unit won the award in the manufacturing category.
In 1999, CGISS did a self-assessment against the Baldrige criteria and scored just 399 out of 1,000 possible points. "It was a huge opportunity," says Mark Hurlbert, director of business processes in CGISS's Office of Business Excellence. "We established this office to really tie what are the right things to do [in the Baldrige program] with doing them the right way [Six Sigma]."
The company assigned each of the Baldrige domains to a senior manager. For example, process management went to a supply chain manager, customer and market focus went to a sales and marketing manager, and information and analysis went to the CGISS division's CIO. Each of these managers has his own "balanced scorecard" with strategic objectives and annual initiatives to support those objectives. For example, the CIO this year has a strategic objective, "to serve customers better," and a specific project aimed at that: to standardize the tools and databases in call centers.
Having chartered a course via Baldrige, CGISS is using Six Sigma to drive the ship, Hurlbert says. In 2002, CGISS boosted its Baldrige score from 399 to between 650 and 750, more than enough to win the prize.
|
Process Model Selection Framework
|
|
|
 In 2002, Motorola’s Commercial, Government and Industrial Solutions Sector won the Malcolm Baldrige National Quality Award in the manufacturing category. |
