NIST, The Economis Impacts of Inadequate Infrasturcture for Software Testing, 200205

 

NIST, The Economis Impacts of Inadequate Infrasturcture for Software Testing, 200205

http://www.nist.gov/director/planning/upload/report02-3.pdf

 

직접 다운받기

NIST_TheEconomisImpactsofInadequateInfrasturctureforSoftwareTesting_(200205).pdf

 

목차

1. Introduction to Software Quality and Testing 1-1

1.1 Software Quality Attributes................................................ 1-3

1.2 Software Quality Metrics ................................................... 1-6

1.2.1 What Makes a Good Metric ................................... 1-7

1.2.2 What Can be Measured ......................................... 1-8

1.2.3 Choosing Among Metrics....................................... 1-8

1.3 Software Testing.............................................................. 1-10

1.4 The Impact of Inadequate Testing .................................... 1-11

1.4.1 Failures due to Poor Quality................................. 1-11

1.4.2 Increased Software Development Costs ................ 1-12

1.4.3 Increased Time to Market..................................... 1-12

1.4.4 Increased Market Transaction Costs...................... 1-13

2. Software Testing Methods and Tools 2-1

2.1 Historical Approach to Software Development................... 2-1

2.2 Software Testing Infrastructure ........................................... 2-4

2.2.1 Software Testing Stages .......................................... 2-4

2.2.2 Commercial Software Testing Tools........................ 2-7

2.3 Software Testing Types ...................................................... 2-9

2.3.1 Conformance Testing............................................. 2-9

2.3.2 Interoperability Testing ........................................ 2-10

2.3.4 Relationship between Software Stages, Testing

Types, and Testing Tools...................................... 2-13

2.3.5 Standardized Software Testing Technologies ........ 2-15

3. Inadequate Infrastructure for Software Testing:

Overview and Conceptual Model 3-1

3.1 Software Testing Inadequacies........................................... 3-2

3.1.1 Integration and Interoperability Testing Issues......... 3-2

3.1.2 Automated Generation of Test Code....................... 3-3

3.1.3 Lack of a Rigorous Method for Determining

When a Product Is Good Enough to Release........... 3-3

3.1.4 Lack of Readily Available Performance Metrics

and Testing Procedures .......................................... 3-4

3.1.5 Approaches for Improving Software Testing

Infrastructure ......................................................... 3-5

3.2 Conceptual Economic Model............................................. 3-6

3.3 Software Developers ......................................................... 3-7

3.3.1 Cost Framework..................................................... 3-7

3.3.2 Factors Influencing the Profit-Maximizing Level

of R&D Expenditures.............................................. 3-9

3.4 End Users........................................................................ 3-12

3.4.1 Cost Framework................................................... 3-12

3.5 The Market for Software Products .................................... 3-14

3.5.1 Quality's Impact on Market Prices........................ 3-14

3.6 Modeling an Inadequate Software Testing

Infrastructure................................................................... 3-15

3.6.1 Inadequate Infrastructure's Impact on the Cost

of Quality............................................................ 3-17

3.6.2 Inadequate Infrastructure's Impact on the Cost

of After-Sales Service ........................................... 3-18

3.6.3 Inadequate Infrastructure's Impact on End-

Users' Demand.................................................... 3-19

3.6.4 Aggregate Impact................................................. 3-19

3.7 The Time Dimension....................................................... 3-20

3.8 Conclusion...................................................................... 3-21

4. Taxonomy for Software Testing Costs 4-1

4.1 Principles that Drive Software Testing Objectives............... 4-1

4.1.1 Testing Activities.................................................... 4-2

4.1.2 Detecting Bugs Sooner........................................... 4-3

4.1.3 Locating the Source of Bugs Faster and with

More Precision ...................................................... 4-3

4.2 Software Developers' Cost Taxonomy................................ 4-3

4.2.1 Resource Categories .............................................. 4-4

4.2.2 Summary of Developer Technical and

Economic Metrics .................................................. 4-6

4.3 Software Users' Cost Taxonomy......................................... 4-7

4.3.1 Pre-purchase Costs ................................................ 4-8

4.3.2 Installation Costs.................................................... 4-9

4.3.3 Post-purchase Costs ............................................. 4-11

5. Measuring the Economic Impacts of an

Inadequate Infrastructure for Software Testing 5-1

5.1 Defining the Counterfactual World .................................... 5-1

5.1.1 Developers' Costs of Identifying and Correcting

Errors..................................................................... 5-3

5.1.2 Counterfactual Scenario for Developers ................. 5-8

5.1.3 Counterfactual Scenario for Users .......................... 5-9

5.2 Custom Versus Commercial Software Products .................. 5-9

5.3 Estimating Software Developer Costs ............................... 5-11

5.4 Estimating Software User Costs ........................................ 5-13

5.5 Period of Analysis............................................................ 5-17

5.6 Industry-Specific User Costs ............................................ 5-19

6. Transportation Manufacturing Sector 6-1

6.1 Overview of CAD/CAM/CAE and PDM Software in the

Transportation Manufacturing Sector ................................. 6-2

6.1.1 Use of CAD/CAM/CAE and PDM Software ............. 6-3

6.1.2 Development of CAD/CAM/CAE and PDM

Software ................................................................ 6-5

6.2 Software Developer Costs in the Transportation

Manufacturing Sector ........................................................ 6-6

6.2.1 Estimation Approach.............................................. 6-8

6.2.2 Survey Findings ..................................................... 6-9

6.2.3 Cost Impacts Per Employee for Software

Developers .......................................................... 6-13

6.2.4 Industry-Level Impact........................................... 6-14

6.3 End-User Costs in the Transportation Manufacturing

Sector ............................................................................. 6-15

6.3.1 Survey Method .................................................... 6-15

6.3.2 Survey Response Rates and Industry Coverage...... 6-16

6.3.3 Survey Findings ................................................... 6-18

6.3.4 Costs of Bugs and Errors Per Employee ................. 6-23

6.3.5 Partial Reduction of Software Errors...................... 6-26

6.4 Users' Industry-Level Impact Estimates............................. 6-27

7. Financial Services Sector 7-1

7.1 Overview of the Use of Clearinghouse Software and

Routers and Switches in the Financial Services Sector ........ 7-2

7.1.1 Overview of Electronic Transactions in the

Financial Services Sector........................................ 7-3

7.1.2 Software Used by Financial Services Providers ....... 7-5

7.1.3 Software Embedded in Hardware Used to

Support Financial Transactions............................... 7-6

7.2 Software Developer Costs in the Financial Services

Sector ............................................................................... 7-8

7.2.1 Industry Surveys................................................... 7-10

7.2.2 Survey Findings ................................................... 7-10

7.2.3 Cost Impacts Per Employee for Software

Developers .......................................................... 7-14

7.2.4 Industry-Level Impacts ......................................... 7-15

7.3 Software User Costs in the Financial Services Sector ........ 7-16

7.3.1 Survey Method .................................................... 7-16

7.3.2 Survey Response Rates and Industry Coverage...... 7-17

7.3.3 Survey Findings .................................................. 7-19

7.3.4 Software User Costs Per Transaction..................... 7-24

7.3.5 Partial Reduction of Software Errors...................... 7-26

7.3.6 Users' Industry-Level Impact Estimates ................. 7-28

8. National Impact Estimates 8-1

8.1 Per-Employee Testing Costs: Software Developers............. 8-2

8.2 Per-Employee Costs: Software Users ................................. 8-4

8.4 National Impact Estimates ................................................. 8-5

8.5 Limitations and Caveats..................................................... 8-6

References R-1

Appendixes

A: Glossary of Testing Stages and Tools..................................A-1

B: CAD/CAM/CAE/PDM Use and Development in the

Transportation Sector ........................................................ B-1

C: CAD/CAM/CAE/PDM Developers and Users Survey

Instruments .......................................................................C-1

D: Financial Services Software Use and Development ............D-1

E: Financial Services Survey Instruments .................................E-1

 

Figure 2-1 Waterfall Model......................................................................... 2-3

Figure 2-2 Commercial Software Testing Infrastructure Hierarchy................ 2-5

Figure 3-1 Software Quality's Role in Profit Maximization......................... 3-10

Figure 3-2 Minimize Joint Costs of Pre-sales Testing and After-Sales

Service (Holding Price and Quantity Constant) ......................... 3-11

Figure 3-3 Change in Quality's Impact on Price, Quantity, and Net

Revenue .................................................................................. 3-16

Figure 3-4 Enhanced Testing Tool's Impact on the Marginal Cost of

Quality .................................................................................... 3-18

Figure 5-1 The Waterfall Process................................................................. 5-3

Figure 5-2 Typical Cumulative Distribution of Error Detection..................... 5-6

Figure 5-3 Software Testing Costs Shown by Where Bugs Are Detected

(Example Only) .......................................................................... 5-7

Figure 5-4 Cost Reductions of Detecting Bugs and Fixing Them Faster

(Example Only) .......................................................................... 5-8

Figure 5-5 Custom vs. Commercial Development Cost Allocation ............. 5-10

Figure 5-6 Relationship between Users Costs and Percentage Reduction

in Bugs .................................................................................... 5-18

Figure 6-1 Economic Relationship Among CAD/CAM/CAE Producers

and Consumers .......................................................................... 6-3

Figure 6-2 CAD/CAE/CAM and PDM in the Product Development

Cycle......................................................................................... 6-4

 

Table 1-1 McCall, Richards, and Walters's Software Quality Attributes....... 1-4

Table 1-2 ISO Software Quality Attributes.................................................. 1-5

Table 1-3 List of Metrics Available............................................................. 1-7

Table 1-4 Recent Aerospace Losses due to Software Failures .................... 1-11

Table 1-5 Relative Costs to Repair Defects when Found at Different

Stages of the Life-Cycle ............................................................ 1-13

Table 2-1 Allocation of Effort ..................................................................... 2-4

Table 2-2 The Degree of Usage of the Different Testing Stages with the

Various Testing Types .............................................................. 2-13

Table 2-3 Software Testing Types Associated with the Life Cycle .............. 2-14

Table 2-4 Tools Used by Type of Testing.................................................. 2-16

Table 2-5 Tools Used by Testing Stage..................................................... 2-17

Table 4-1 Labor Taxonomy........................................................................ 4-4

Table 4-2 Software Testing Capital Taxonomy............................................ 4-5

Table 4-3 Impact Cost Metrics for Software Developers.............................. 4-7

Table 4-4 Users' Pre-Purchase Costs Associated with Bugs ......................... 4-9

Table 4-5 Users' Implementation Costs Associated with Bugs ................... 4-10

Table 4-6 Users' Post-purchase Costs Associated with Bugs...................... 4-11

Table 5-1 Relative Cost to Repair Defects When Found at Different

Stages of Software Development (Example Only) ........................ 5-4

Table 5-2 Preliminary Estimates of Relative Cost Factors of Correcting

Errors as a Function of Where Errors Are Introduced and

Found (Example Only) ............................................................... 5-4

Table 5-3 Example of the Frequency (%) of Where Errors Are Found, in

Relationship to Where They Were Introduced ............................ 5-5

Table 5-4 Impact Cost Metrics for Software Developers............................ 5-12

Table 5-5 Cost Metrics for Users .............................................................. 5-16

Table 5-6 Importance of Quality Attributes in the Transportation

Equipment and Financial Services Industries............................. 5-20

Table 6-1 Cost Impacts on U.S. Software Developers and Users in the

Transportation Manufacturing Sector Due to an Inadequate

Testing Infrastructure ($ millions)................................................ 6-2

Table 6-2 Distribution of Bugs Found Based on Introduction Point ........... 6-10

Table 6-3 Hours to Fix Bug Based on Introduction Point........................... 6-10

Table 6-4 Time to Fix a Bug Based on Discovery Point............................. 6-11

Table 6-5 Distribution of Bugs Based on Infrastructure ............................. 6-12

Table 6-6 Developer Testing Costs for a Typical Company of 10,000

Employees ............................................................................... 6-13

Table 6-7 Annual Impact on U.S. Software Developers of

CAD/CAM/CAE/PDM Software................................................. 6-14

Table 6-8 Transportation Equipment Industry Survey Completion Rates

............................................................................................... 6-17

Table 6-9 Industry Coverage by Employment ........................................... 6-17

Table 6-10 Reported Software Products...................................................... 6-19

Table 6-11 Incidence and Costs of Software Bugs....................................... 6-21

Table 6-12 Average Company-Level Costs of Search, Installation, and

Maintenance (Life-Cycle Costs) ................................................ 6-22

Table 6-13 Costs Per Employee.................................................................. 6-24

Table 6-14 Company-Level Costs Associated with Bugs for Hypothetical

Transportation Company at Different Employment Levels ......... 6-25

Table 6-15 Cost Reductions as a Function of Bug Reductions ..................... 6-27

Table 6-16 Annual Impacts' Weighted Cost Per Deposits and Loans ........... 6-28

Table 7-1 Cost Impacts on U.S. Software Developers and Users in the

Financial Services Sector Due to an Inadequate Testing

Infrastructure ($ millions) ........................................................... 7-2

Table 7-2 Characteristics of Firms in the Financial Services Sector,

1997.......................................................................................... 7-4

Table 7-3 Router Market Shares of Major Firms .......................................... 7-6

Table 7-4 Distribution of Bugs Found Based on Introduction Point ........... 7-11

Table 7-5 Hours to Fix Bug based on Introduction Point........................... 7-12

Table 7-6 Time to Fix a Bug Based on Discovery Point............................. 7-13

Table 7-7 Shift in the Distribution of Where Bugs are Found Based on

Infrastructure ........................................................................... 7-13

Table 7-8 Developer Testing Costs for a Typical Company of 10,000

Employees ............................................................................... 7-15

Table 7-9 Annual Impact on U.S. Software Developers Supporting the

Financial Services Sector.......................................................... 7-16

Table 7-10 Financial Industry Survey Completion Rates ............................. 7-18

Table 7-11 Industry Coverage .................................................................... 7-19

Table 7-12 Reported Software Products...................................................... 7-20

Table 7-13 Incidence and Costs of Software Errors ..................................... 7-21

Table 7-14 Total Costs of Search, Installation, and Maintenance (Life-

Cycle Costs)............................................................................. 7-23

Table 7-15 Software Bug and Error Costs Per Million Dollars of Deposits

and Loans ................................................................................ 7-25

Table 7-16 Company Costs Associated with Bugs for Hypothetical

Company Sizes ........................................................................ 7-26

Table 7-17 Cost Reductions as a Function of Error Reductions.................... 7-27

Table 7-18 Annual Impacts' Weighted Cost Per Deposits and Loans ........... 7-28

Table 8-1 National Economic Impact Estimates .......................................... 8-1

Table 8-2 FTEs Engaged in Software Testing (2000) .................................... 8-3

Table 8-3 Software Developer Costs Per Tester........................................... 8-4

Table 8-4 National Employment in the Service and Manufacturing

Sectors....................................................................................... 8-5

Table 8-5 Per-Employee Cost Metrics......................................................... 8-5

Table 8-6 National Impact Estimates .......................................................... 8-6