ROI Fundamentals: Why and When to Measure ROI » (1st Edition)

Jack J. Phillips is chairman of the ROI Institute, Inc. A world-renowned expert on measurement and evaluation, Phillips provides consulting services to Fortune 500 companies and workshops for major conference providers throughout the world. Phillips is the author or editor of more than thirty books and more than one hundred articles.

Patricia Pulliam Phillips is an internationally recognized author, consultant, and president and CEO of the ROI Institute, Inc. Phillips provides consulting services to organizations worldwide. She helps organizations build capacity in the ROI Methodology by facilitating the ROI certification process and teaching the ROI Methodology through workshops and graduate-level courses.

The ROI Institute, Inc., is a benchmarking, research, and information sharing organization that provides consulting services, workshops, and certification in the ROI Methodology. Widely considered the leading authority on evaluation and measurement of learning and development in organizations, the ROI Institute conducts workshops and offers certification for thousands of practitioners through a variety of strategic partners.

ROI Fundamentals

ROI Fundamentals is the first of six books in the Measurement and Evaluation Series from Pfeiffer. The proven ROI Methodology—developed by the ROI Institute—provides a practical system for evaluation planning, data collection, data analysis, and reporting. All six books in the series offer the latest tools, most current research, and practical advice for measuring ROI in a variety of settings.

ROI Fundamentals outlines the basic ROI Methodology and shows why measuring return on investment is such a valuable process. The book highlights the benefits of implementing an effective ROI model and also reveals the challenges organizations face when incorporating the ROI Methodology. Using real-world examples from international companies, the authors explore the types of organizations that are best suited for the ROI Methodology and suggest the optimal time to implement it. The book also offers step-by-step information for planning an effective ROI evaluation.

Acknowledgments from the Editors     xix
Preface: The Realities of ROI     xxi
A Brief Description of the ROI Methodology     1
Evaluation Levels: A Beginning Point     1
Evaluation Planning     9
Data Collection     14
Isolation of Program Effects     17
Data Conversion     19
Intangible Benefits     22
Program Costs     22
Return on Investment Calculation     23
Reporting     24
Case Study     24
Final Thoughts     26
Why ROI?     27
Progress and Status of ROI     28
Global Trends in Measurement     29
ROI Applications Across Fields and Sectors     30
Growth of ROI as a Conference Topic     32
Global Expansion of ROI Applications     32
The Move from Activity to Results     33
ROI Is Here to Stay     34
Why ROI?     34
Increased Budgets     35
The Ultimate Level of Evaluation     36
Change, Quality, and Reengineering     36
Business Mindset of Support Managers     37
The Trend Toward Accountability     38
Top ExecutiveRequirements     39
Concerns About Using ROI     39
Practitioners     40
Senior Managers, Sponsors, and Clients     40
Researchers     41
Criteria for an Effective ROI Process     41
Barriers to ROI Implementation     43
Financial and Time Costs     44
Lack of Staff Skills and Orientation     44
Faulty Needs Assessment     44
Fear     44
Lack of Discipline or Planning     45
False Assumptions     45
Benefits of Using ROI     46
Measurement of a Program's Contribution     46
Clear Priorities     46
Focus on Results     46
Respect from Senior Executives and Program Sponsors     47
Positive Changes in Management Perceptions     47
ROI Best Practices     47
Evaluation Targets     48
Microlevel Evaluation     48
A Variety of Data Collection Methods     49
Isolation of the Program     49
Sampling for ROI Calculations     49
Conversion of Program Results to Monetary Values     51
Final Thoughts     51
Who Should Use the ROI Methodology?      53
The Typical Organization     54
The Typical Program     55
Signs That an Organization Is Ready for the ROI Methodology     57
Taking a Reactive Versus a Proactive Approach     61
Reactive Approach     62
Proactive Approach     62
Final Thoughts     63
How to Build a Credible Process     65
The Evaluation Framework     67
The Process Model     72
The Operating Standards: Guiding Principles     72
Report the Complete Story     73
Conserve Important Resources     73
Enhance Credibility     73
Be Conservative     73
Account for Other Factors     74
Account for Missing Data     74
Adjust Estimates for Error     74
Omit the Extremes     74
Capture Annual Benefits for Short-Term Programs     75
Account for All Program Costs     75
Report Intangible Benefits     75
Communicate Results     75
Case Applications and Practice     76
Implementation     77
Assign Responsibilities     77
Develop Skills     77
Develop an Implementation Plan      78
Prepare or Revise Evaluation Guidelines     81
Brief Managers on the Evaluation Process     81
Final Thoughts     81
Inhibitors of Implementation     83
Barriers to Implementation     83
Costs and Time     83
Lack of Skills     84
Faulty or Inadequate Initial Analysis     85
Fear     85
Discipline and Planning     86
ROI Myths     86
ROI Is Too Complex for Most Users     87
ROI Is Expensive, Consuming Too Many Critical Resources     87
If Senior Management Does Not Require ROI, There Is No Need to Pursue It     87
ROI Is a Passing Fad     88
ROI Is Only One Type of Data     88
ROI Is Not Future-Oriented; It Reflects Only Past Performance     89
ROI Is Rarely Used by Organizations     89
The ROI Methodology Cannot Be Easily Replicated     89
ROI Is Not a Credible Process; It Is Too Subjective     90
ROI Is Not Credible When Evaluating Soft-Skill Programs     90
ROI Is Only for Manufacturing and Service Organizations     90
Isolation of the Influence of Factors Other Than the Program Is Not Always Possible     91
Measurement of On-the-Job Activities Is Impossible Because Post-Program Control of Application Is Impossible     91
ROI Is Appropriate Only for Large Organizations     92
The ROI Methodology Has No Standards     92
It's All About Change Management     92
Next Steps     94
Final Thoughts     95
ROI Quiz     96
Quiz Answers     98
Planning for Evaluation     99
Establishing Purpose and Feasibility     99
Purpose     99
Make Decisions About Programs     100
Improve Programs and Processes     100
Demonstrate Program Value     101
Feasibility     103
Validation of Program Objectives     103
Availability of Data     104
Appropriateness for ROI Measurement     104
Defining Program Objectives     104
Reaction Objectives     105
Learning Objectives     108
Application Objectives     111
Impact Objectives     112
ROI Objectives     115
Developing the Planning Documents     117
Data Collection Plan     117
What Do You Ask?     117
How Do You Ask?     117
Whom Do You Ask?      121
When Do You Ask?     121
Who Asks?     121
ROI Analysis Plan     121
Methods for Isolating the Effects of the Program     121
Methods for Converting Data to Monetary Values     123
Cost Categories     123
Intangible Benefits     123
Communication Targets for the Final Report     123
Other Application Influences and Issues     123
Comments     124
Project Plan     124
Conducting the Planning Meeting     125
Who Should Be Involved     125
Agenda     126
Factors for Success     126
Identifying Data Sources     127
Organizational Performance Records     127
Participants     127
Participants' Managers     128
Participants' Direct Reports     128
Team or Peer Group     128
Internal and External Groups     129
Final Thoughts     129
Appendix     131
Index     133
About the Authors     143