Data Collection: Planning for and Collecting All Types of Data » (1st Edition)

Patricia Pulliam Phillips is an internationally recognized author, consultant, and president and CEO of the ROI Institute, Inc. Phillips provides consulting services to organizations worldwide. She helps organizations build capacity in the ROI Methodology by facilitating the ROI certification process and teaching the ROI Methodology through workshops and graduate-level courses.

Cathy A. Stawarski is program manager of the Strategic Performance Improvement and Evaluation program at the Human Resources Research Organization (HumRRO) in Alexandria, Virginia. She has more than twenty-five years of experience in research, training and development, and program evaluation. Throughout her nearly twenty years at HumRRO, she has worked primarily with clients in the federal sector. Her work includes leading and conducting the evaluation of leadership and human capital initiatives as well as assisting organizations in developing comprehensive evaluation strategies.

The ROI Institute, Inc., is a benchmarking, research, and information sharing organization that provides consulting services, workshops, and certification in the ROI Methodology. Widely considered the leading authority on evaluation and measurement of learning and development in organizations, the ROI Institute conducts workshops and offers certification for thousands of practitioners through a variety of strategic partners.

Data Collection

Data Collection is the second of six books in the Measurement and Evaluation Series from Pfeiffer. The proven ROI Methodology—developed by the ROI Institute—provides a practical system for evaluation planning, data collection, data analysis, and reporting. All six books in the series offer the latest tools, most current research, and practical advice for measuring ROI in a variety of settings.

Data Collection offers an effective process for collecting data that is essential to the implementation of the ROI Methodology. The authors outline the techniques, processes, and critical issues involved in successful data collection. The book examines the various methods of data collection, including questionnaires, interviews, focus groups, observation, action plans, performance contracts, and monitoring records. Written for evaluators, facilitators, analysts, designers, coordinators, and managers, Data Collection is a valuable guide for collecting data that are adequate in quantity and quality to produce a complete and credible analysis.

Acknowledgments from the Editors     xxi
Principles of the ROI Methodology     xxiii
Using Questionnaires and Surveys     1
Types of Questions     1
Questionnaire Design Steps     2
Determine the Specific Information Needed     2
Involve Stakeholders in the Process     3
Select the Types of Questions     3
Develop the Questions     3
Check the Reading Level     3
Test the Questions     4
Address the Anonymity Issue     4
Design for Ease of Tabulation and Analysis     4
Develop the Completed Questionnaire and Prepare a Data Summary     5
Improving the Response Rate for Questionnaires and Surveys     5
Provide Advance Communication     5
Communicate the Purpose     6
Describe the Data Integration Process     6
Keep the Questionnaire as Simple as Possible     6
Simplify the Response Process     6
Use Local Manager Support     7
Let the Participants Know That They Are Part of a Sample     7
Consider Incentives     7
Have an Executive Sign the Introductory Letter     8
Use Follow-Up Reminders     8
Providea Copy of the Results to the Participants     8
Review the Questionnaire with Participants     9
Consider a Captive Audience     9
Communicate the Timing of Data Flow     9
Select the Appropriate Medium     10
Consider Anonymous or Confidential Input     10
Pilot Test the Questionnaire     10
Explain How Long Completing the Questionnaire Will Take     11
Personalize the Process     11
Provide an Update     11
Final Thoughts     12
Using Tests     13
Types of Tests     13
Norm-Referenced Tests     13
Criterion-Referenced Tests     14
Performance Tests     14
Simulations     16
Electromechanical Simulation     17
Task Simulation     17
Business Games     17
In-Basket Simulation     17
Case Study     18
Role-Playing     18
Informal Tests     19
Exercises, Problems, or Activities     19
Self-Assessment     20
Facilitator Assessment     20
Final Thoughts     21
Using Interviews, Focus Groups, and Observation      23
Interviews     23
Types of Interviews     24
Interview Guidelines     24
Develop the Questions to Be Asked     24
Test the Interview     24
Prepare the Interviewers     25
Provide Clear Instructions to the Participants     25
Schedule the Interviews     25
Focus Groups     25
Applications of Focus Groups     26
Guidelines     27
Plan Topics, Questions, and Strategy Carefully     27
Keep the Group Size Small     27
Use a Representative Sample     27
Use Experienced Facilitators     28
Observations     28
Guidelines for Effective Observation     28
Observations Should Be Systematic     29
Observers Should Be Knowledgeable     29
The Observer's Influence Should Be Minimized     29
Observers Should Be Selected Carefully     30
Observers Must Be Fully Prepared     30
Observation Methods     30
Behavior Checklist     30
Delayed Report     31
Video Recording     31
Audio Monitoring     32
Computer Monitoring     32
Final Thoughts     32
Using Other Data Collection Methods     35
Business Performance Monitoring     35
Using Current Measures     36
Identify Appropriate Measures     36
Convert Current Measures to Usable Ones     36
Developing New Measures     37
Action Planning     38
Developing an Action Plan     40
Using Action Plans Successfully     42
Communicate the Action Plan Requirement Early     42
Describe the Action Planning Process at the Beginning of the Program     42
Teach the Action Planning Process     42
Allow Time to Develop the Plan     43
Have the Facilitator Approve Action Plans     43
Require Participants to Assign a Monetary Value to Each Improvement     43
Ask Participants to Isolate the Effects of the Program     44
Ask Participants to Provide a Confidence Level for Estimates     44
Require That Action Plans Be Presented to the Group     45
Explain the Follow-Up Process     45
Collect Action Plans at the Stated Follow-Up Time     46
Summarize the Data and Calculate the ROI     46
Applying Action Plans     48
Identifying Advantages and Disadvantages of Action Plans     51
Performance Contracts     51
Final Thoughts     54
Measuring Reaction and Planned Action     55
Why Measure Reaction and Planned Action?     55
Customer Satisfaction     55
Immediate Adjustments     56
Team Evaluation     56
Predictive Capability     56
Importance of Other Levels of Evaluation     58
Areas of Feedback     58
Data Collection Issues     63
Timing     63
Methods     64
Administrative Guidelines     65
Uses of Reaction Data     67
Final Thoughts     69
Measuring Learning and Confidence     71
Why Measure Learning and Confidence?     71
The Learning Organization     71
Compliance Issues     72
Development of Competencies     73
Certification     73
Consequences of an Unprepared Workforce     73
The Role of Learning in Programs     74
Measurement Issues     75
Challenges     75
Program Objectives     75
Typical Measures     76
Timing     77
Data Collection Methods     79
Administrative Issues     81
Validity and Reliability     81
Consistency     82
Pilot Testing     83
Scoring and Reporting     83
Confronting Failure     84
Uses of Learning Data     84
Final Thoughts     85
Measuring Application and Implementation     87
Why Measure Application and Implementation?     87
Obtain Essential Information     88
Track Program Focus     88
Discover Problems and Opportunities     89
Reward Effectiveness     90
Challenges     90
Linking Application with Learning     90
Building Data Collection into the Program     90
Ensuring a Sufficient Amount of Data     91
Addressing Application Needs at the Outset     91
Measurement Issues     92
Methods     92
Objectives     92
Areas of Coverage     93
Data Sources     93
Timing     95
Responsibilities     96
Data Collection Methods     96
Questionnaires     96
Progress with Objectives      97
Use of Program Materials and Handouts     97
Application of Knowledge and Skills     97
Changes in Work Activities     104
Improvements or Accomplishments     105
Definition of the Measure     105
Amount of Change     105
Unit Value     105
Basis for Value     106
Total Annual Impact     106
Other Factors     106
Improvements Linked with the Program     107
Confidence Level     107
Perception of Investment in the Program     107
Link with Output Measures     107
Other Benefits     108
Barriers     108
Enablers     108
Management Support     108
Other Solutions     109
Target Audience Recommendations     109
Suggestions for Improvement     109
Interviews, Focus Groups, and Observation     110
Action Plans     110
Barriers to Application     111
Uses of Application Data     112
Final Thoughts     112
Measuring Impact and Consequences     115
Why Measure Business Impact?     115
Impact Data Provide Higher-Level Information on Performance     115
Impact Data Represent the Business Driver of a Program     116
Impact Data Provide Value for Sponsors     117
Impact Data Are Easy to Measure     117
Effective Impact Measures     117
Hard Data Measures     118
Soft Data Measures     120
Tangible Versus Intangible Measures     121
Impact Objectives     122
Linking Specific Measures to Programs     123
Sources of Impact Data     126
Data Collection Methods     127
Monitoring Business Performance Data     127
Identify Appropriate Measures     128
Convert Current Measures to Usable Ones     128
Develop New Measures     129
Action Plans     129
Set Goals and Targets     130
Define the Unit of Measure     130
Place a Monetary Value on Each Improvement     131
Implement the Action Plan     131
Document Specific Improvements     132
Isolate the Effects of the Program     132
Provide a Confidence Level for Estimates     133
Collect Action Plans at Specified Time Intervals     133
Summarize the Data and Calculate the ROI      133
Performance Contracts     134
Questionnaires     134
Final Thoughts     138
Selecting the Proper Data Collection Method     139
Matching Exercise     139
Selecting the Appropriate Method for Each Level     143
Type of Data     143
Investment of Participants' Time     143
Investment of Managers' Time     144
Cost     144
Disruption of Normal Work Activities     144
Accuracy     145
Built-In Design Possibility     145
Utility of an Additional Method     146
Cultural Bias of Data Collection Method     146
Final Thoughts     146
Index     147
About the Authors     153