A One-Day Workshop Describing How to Calculate theROI in Government Training Programs
Jack J. Phillips, Ph.D.
Patti P. Phillips, Ph.D.
Copyright © 2009 ROI Institute, Inc.
No part of this may be reproduced, stored in a retrieval system, or transmitted in any form or by a means without written permission.
P.O. Box 380637
Birmingham, AL 35238
Phone: 205-678-8101 Fax: 205-678-8102
Email: info@roiinstitute.net
Web site: www.roiinstitute.net
LEARNING OBJECTIVES
After completing this workshop, you should be able to:
Identify the drivers for ROI in the public sector
Make the business case for ROI in the public sector
Develop program objectives at multiple levels
Explain to clients how ROI works
Identify 7 of the 12 guiding principles
Describe the 10 steps in the ROI Methodology
Plan next steps
2
3
THE BOTTOM LINE ON ROI
The materials in this workbook are taken from the following publications:
Return on Investment Basics. Patti P. Phillips and Jack J. Phillips. Alexandria: ASTD, 2006
Return in Investment in Training and Performance Improvement Programs (2nd Edition). Jack J. Phillips.
Boston: Butterworth-Heinemann, 2003.
The Bottomline on ROI. Patti P. Phillips. Atlanta: CEP Press, 2002.
Measuring Return on Investment, Volume 3. In Action series, Patti P. Phillips (editor) Jack J. Phillips
(series editor). Alexandria: ASTD, 2001.
ROI at Work: Best Practice Case Studies from the Real World. Jack J. Phillips and Patti P. Phillips,
Editors. American Society for Training and Development, Alexandria, VA, 2005.
CONTENT
The need for ROI
When ROI is appropriate
ROI application
ROI objectives
ROI model
ROI standards
ROI implementation
ROI benefits
4
1
Objectives
After completing this workshop, you should be able to:
Identify the drivers for ROI
Make the business case for ROI
Develop program objectives at multiple levels
Explain to clients how ROI works
Identify 7 of the 12 guiding principles
Describe the 10 steps in the ROI Methodology
Plan next steps
Shifting Paradigms
Activity Based Results Based
No business need for the
program
Program linked to specific business
needs
No assessment of
performance issues
Assessment of performance
effectiveness
No specific measurable
objectives
Specific objectives for behavior and
business impact
No effort to prepare program
participants to achieve results
Results expectations communicated
to participants
No effort to prepare the work
environment to support
transfer
Environment prepared to support
transfer
No efforts to build
partnerships with key
managers
Partnerships established with key
managers and clients
No measurement of results or
benefit-cost analysis
Measurement of results and benefitcost
analysis
Planning and reporting is input
focused
Planning and reporting is outcome
focused
Why does ROI work?
Provides a balanced set of measures
Offers a step-by-step process
Bridges business evaluation and program evaluation
Balances research/statistical methods with practical application
Flexible for all types of programs
Credible with managers and administrators
2
Got Results?
Deborah Stewart is the new director of Intelligence Academy (IA), the learning
organization for one of the top secret intelligence communities within the Department of
Defense (DOD). Matt Mathews is Chief, Strategy and Analytics.
Deborah has just returned from a meeting with the Deputy Director. There is growing
pressure to show results for funding flowing to IA programs. The Deputy Director has
reinforced with Deborah the need for IA to start showing results. IA programs are costly
and there is ongoing discussion among department leaders that IA may be a candidate
for resource reallocation.
Deborah has called a meeting with all of her department heads; this meeting includes
Matt Mathews.
Deborah: Thank you for joining me this morning. The intent of the meeting, as I
mentioned in my recent e-mail, is for each of you to brief me on your recent assessment
of your programs.
As I mentioned, IA funding is being scrutinized throughout the agency. Resources are
tight and the leadership wants to ensure that the right programs are receiving an
appropriate allocation of funding.
Each of you was asked to determine how your programs are adding value to the
mission of this department. Matt, you’re the chief of analytics; let’s begin with you. How
is IA doing as far as contributing to the mission?
Matt: Thanks Deborah. As you can see by the slide presented here and in your hand
out, IA is quite busy.
We are developing a variety of new programs; many of these programs are required to
quickly develop new hires so they can be valuable contributors. We are also about to
deploy a new professional health program that offers staff members throughout the
agency the opportunity to take unique courses they feel will help them professionally
and personally. We believe that personal development is a contributing factor to mission
success.
In the past nine months we have developed ten new programs, offered 1,200 hours of
training, taken 2,000 employees through some sort of training experience, and on
average our instructors receive a rating of 4.5 out of 5 on the program satisfaction rating
that is developed through the end-of-course evaluations.
We piloted the Human Performance Improvement (HPI) Certificate program with 18
participants. Though the end-of-course evaluations indicated minimal enthusiasm about
the course from participants, we thought we’d give it another try by offering it to a
second cohort. As far as we can tell people are somewhat excited about what we are
doing.
3
Got results? (cont.)
We are also deploying on-the-job training and one-on-one coaching. The idea is that the
people closest to the job can train others in the job. It takes up a lot of time, especially
from supervisors, but we think reducing training costs is worthy. The only uncertainty is
whether or not people are actually performing at a higher rate than when they
participate in group training efforts.
Deborah interrupts: Excuse me Matt. This does sound like you are quite busy; but,
what about results. Are we receiving any value from IA efforts?
Matt: Value? Well, yes. Participants like the programs. In most cases, supervisors like
the programs as well – especially, when they can send some of their weaker
performers. This helps with team productivity.
Deborah: Do you have any data that show what people are doing or how they are
becoming more productive in their work as a result of the IA programs?
Matt: Nothing concrete. We do hear from time to time how a tool from one of the
language courses has assisted them with translation. Also, the information assurance
team does well with the Black Hawk simulation. Through this activity, we can determine
whether or not participants know how to ward off unwanted intrusions into the computer
network.
Deborah: But do you know if those unwanted intrusions in our systems are being
detected and prevented?
Matt: Actually, I did hear there is a downward trend in intrusions. According to some of
the IA staff, we have had less downtime due to unwanted intrusions than any time in the
past.
Deborah: That’s great. But is that due to IA programs or is that due to the new system
the agency just installed?
Looking disgusted, Deborah asks for a recess and motions Matt over for a five minute
sidebar.
-------
You are Matt. What dialogue do you believe is about to occur?
You are Deborah. What are you about to say to Matt?
Would you say that Intelligence Academy is results based?
4
Case Application: United Way Agency
At the end of a monthly staff meeting, Philip Harrah, CEO of United Way Agency (UWA), asked
Gina Woolson, Manager of Learning and Development, about the Communications Workshops
that had been conducted with all supervisors and managers throughout the agency. The
workshop featured the Myers-Briggs Type Indicator (MBTI) and showed participants how to
interact with, and understand, each other in their routine activities. The MBTI classifies people
into one of 16 personality types.
Philip continued, ―I found the workshop very interesting and intriguing. I can certainly identify
with my particular personality type, but I am curious what specific value these workshops have
brought to the organization. Do you have any way of showing the results of all 25 workshops?‖
Gina quickly replied, ―We certainly have improved teamwork and communications throughout
the agency. I hear people make comments about how useful the process has been to them
personally.‖ Philip added, ―Do we have anything more precise? Also, do you know how much
money we have spent on these workshops?‖ Gina quickly responded by saying, ―I am not sure
that we have any precise data and I am not sure exactly how much money we spent, but I can
certainly find out.‖ Philip concluded with some encouragement, ―Any specifics would be helpful.
Please understand that I am not opposing this training effort. However, when we initiate these
types of programs, we need to make sure that they are adding value to the organization. Will
you let me know your thoughts on this issue in about two weeks?‖
Gina was a little concerned about this CEO’s comments, particularly since the CEO enjoyed
the workshop and made several positive comments about it. Why was he questioning the
effectiveness of it? Why was he concerned about the costs?
These questions began to frustrate Gina as she reflected over the workshop. She recalled how
she was first introduced to the MBTI. She attended the workshop when it was conducted by a
friend, was impressed with the instrument, and found it to be helpful as she learned more about
her own personality type.
Gina thought the process would be useful to UWA managers and asked the consultant to
conduct a session internally with a group of middle-level managers. With favorable reaction,
she decided to try a second group with the top executives, including Philip Harrah. Their
reaction was favorable. Then she launched it with the entire staff. The feedback was excellent,
including feedback from the CEO.
She realized that the workshops were expensive. Over 600 managers had attended. She felt
teamwork had improved, but there was no way of knowing for sure. With some types of training
you never know if it works, she thought. Still, Gina was facing a dilemma. Should she respond
to the CEO or just ignore the issue?
Continued on the next page.
5
Questions for Discussion
1. Is this situation typical? Explain.
______________________________________________________________________________
______________________________________________________________________________
2. What are the basic issues in this case?
______________________________________________________________________________
______________________________________________________________________________
3. How could this situation be avoided in the future?
______________________________________________________________________________
______________________________________________________________________________
4. What should Gina do? Be specific.
______________________________________________________________________________
______________________________________________________________________________
5. What will Philip be expecting in the future?
______________________________________________________________________________
______________________________________________________________________________
6
Public Sector Issues
(Real or Imagined)
• No profits – no ROI?
• Lack of business alignment
• Absence of hard data?
• Too much politics
• Programs are necessary
• Multiple ROI perspectives
The Money Issue in the Public Sector
• No profits
• Payoff in cost savings or cost reduction
• OR
• Payoff in cost avoidance
• Costs of program are fully loaded
• Impact on budget?
Who is Using the Methodology?
• US Department of Defense
• National Security Agency
• Department of Veterans Affairs
• Central Intelligence Agency
• US Office of Personnel Management
• NASA
• National Science Foundation
• City of New York
• City of San Diego
• City of Richmond
• City of Denver
• State of Texas
• State of New York
• Government of New Zealand
• Government of Singapore
• Government of Poland
• Government of Australia
• Government of Canada
• British Government
• Government of Ireland
Over 200 public sector organizations
Applications
Learning and Development Organization Development
Career Development Orientation Systems
Competency Systems Recruiting Strategies
Diversity Programs Safety & Health Programs
E-Learning Self-Directed Teams
Executive Coaching
Gainsharing
Skill-Based/Knowledge-Based
Compensation
Meetings and Events
Leadership Development
Green Initiatives
Technology Implementation
Quality Management
Wellness/Fitness Initiatives
Drivers for Increased Public Sector Accountability
Regulations are requiring more accountability
Increase cost of programs
Budget shortfalls
Taxpayer pressure
Consistent lack of results
New breed of government managers
7
Is Your Organization A Candidate for ROI Implementation?
Check the most appropriate level of agreement for each statement:
1 = Strongly Disagree; 5 = Strongly Agree
Disagree Agree
1 2 3 4 5
1. My organization is considered a large organization with a wide
variety of programs.
2. We have a large budget that attracts the interest of senior management.
3. Our organization has a culture of measurement and is focused on
establishing a variety of measures in all functions and departments.
4. My organization is undergoing significant change.
5. There is pressure from senior management to measure results of our
programs.
6. My function currently has a very low investment in measurement and
evaluation.
7. My organization has experienced more than one program disaster in
the past.
8. My department has a new leader.
9. My team would like to be the leaders in our field.
10. The image of our department is less than satisfactory.
11. My clients are demanding that our processes show bottom-line results.
12. My function competes with other functions with our organization for
resources.
13. There is increased focus on linking our process to the strategic
direction of the organization.
14. My function is a key player in change initiatives currently taking place
in the organization.
15. Our overall budget is growing and we are required to prove the bottom
line of value of our processes.
8
Scoring
If you scored:
15 – 30 You are not yet a candidate for ROI.
31 – 45 You are not a strong candidate for ROI, however, it is time to start
pursuing some type of measurement process.
46 – 60 You are a candidate for building skills to implement the ROI process. At
this point there is no real pressure to show the ROI, which is the perfect
opportunity to perfect the process within the organization.
61 – 75 You should already be implementing a comprehensive measurement and
evaluation process, including ROI.
Are you a candidate?
The Results
Consider your most important program (i.e., a strategic, expensive, high-profile project
that attracts management attention). Suppose you conducted an impact study to
measure the success of the program. You discover that three months after completing
the program, participants have:
reacted positively to the program and found it to be relevant to their work;
learned new skills and gained new information;
applied the skills and information routinely on the job, although they had some
difficulty in a few areas;
improved several important work unit measures, with some measures improving
as much as 30%;
achieved an impressive 105% return on investment; and
reported an increase in job satisfaction in the work unit.
Questions for discussion:
1. What issues or concerns do you have about the data?
2. What would these types of data mean for the program?
3. What would the results mean to you personally?
4. If the above items were negative, what would it mean for the program?
5. What would negative results mean to you personally?
6. How should the results be used?
9
The ROI Methodology
Generates Six Types of Measures
Reaction and Planned Action
Learning
Application and Implementation
Business Impact
Return on Investment
Intangible Measures
….and includes a technique to isolate the effects of the program or solution
Grounded in Research and Application
Approximately 5,000 impact studies are conducted each year
Over 20,000 individuals have attended a two-day ROI workshop
Over 4,000 individuals are certified to implement the ROI methodology
The ROI methodology has been adopted by hundreds or organizations
in manufacturing, service, non-profit, and government settings in 44
countries
The process has been refined over a 20-year period
30 books have been developed to support the process
ROI Network has 5,000 members
Several ROI Network conferences are conducted annually
Ongoing research on the ROI methodology is conducted by the ROI
Institute to support practitioners
The ROI Calculation
Benefit Cost Ratio (BCR) =
Program Benefits
Program Costs
ROI =
Net Program Benefits
X 100
Program Costs
Notes
10
ROI Framework
Level Measurement Focus
1. Reaction and Planned Action Measures participant satisfaction with the
program/project and captures planned
action.
2. Learning and Confidence Measure changes in knowledge, skills,
and attitudes.
3. Application and Implementation Measures changes in on-the-job
behavior or actions.
4. Impact and Consequences Captures changes in business impact
measures.
5. Return on Investment Compares benefits to the costs.
Questions answered at Level 1:
Questions answered at Level 2:
Questions answered at Level 3:
Questions answered at Level 4:
Questions answered at Level 5:
11
Five Levels of Measurement - Examples
Level 0 Input and Indicators
Number of projects
Audiences
Web site hits
Request
Attendance
Costs
Time to Deliver
Level 1 Reaction and Planned Action
Relevance
Importance
Usefulness
Appeal
Emotion
Brevity
Uniqueness
Concreteness
New Information
Motivation
Appropriateness
Intent to Use
Level 2 Learning and Confidence
Information
Knowledge
Understanding
Capability
Contacts
Confidence
Perceptions
Skills
Level 3 Application and Implementation
Use of Information
Use of Knowledge
Use of Skill
Completion of Actions
Completion of Tasks
Implementation of Ideas
Following the Policy
Use of Procedure
Use of Regulation
Success with Application
Barriers
Enablers
Level 4 Impact
Productivity
Quality
Errors
Incidents
Re-Work
Efficiency
Compliance Discrepancies
Citizen Complaints
Costs
Employee Engagement
Employee Retention
Service Delivery
Cycle Time
Customer Satisfaction
Intangible Measures
…. includes a technique to isolate the effects of
the communication project.
Level 5 Return on Investment
ROI (%)
Benefit Cost Ratio
Payback Period
12
Evaluation is like a Puzzle
An Evaluation
Framework
Case Applications
and Practice
A Process
Model
Operating
Standards and
Philosophy
Implementation
The purpose of each piece of the evaluation puzzle:
Evaluation Framework
Process Model
Operating Standards
Case Application
Implementation
13
Measurement in the Learning and Development Field
Level Measurement Category Current
Status
Goal in
5 Years
Comments About
Status
Coverage*
(Now)(%)
Coverage*
(Goal)(%)
O Inputs/Indicators
Measures inputs into learning and
development including the number of
programs, attendees, audience, costs,
and efficiencies
100% 100% This is being
accomplished
now
1 Reaction and Perceived Value
Measures reaction to, and satisfaction
with, the experience, contents, and value
of program
100% 100% Need more focus
on content and
perceived value
2 Learning
Measures what participants learned in the
program – information, knowledge, skills,
and contacts (takes-away from the
program)
30 – 40% 80 – 90% Must use simple
learning
measures
3 Application and Implementation
Measures progress after the program –
the use of information, knowledge, skills,
and contacts
10% 30% Need more
follow-up
4 Impact and Consequences
Measures changes in business impact
variables such as output, quality, time,
and cost-linked to the program
5% 10% This is the
connection to
business impact
5 ROI
Compares the monetary benefits of the
business impact measures to the costs of
the program.
1% 5% The ultimate level
of evaluation
*Percent of Programs Evaluated At This Level
1. Add your current status in the status column.
2. Add your goal in the goal column.
14
The ROI Methodology
Reporting Data Analysis
LEVEL 5: ROI
Data Collection
LEVEL 4:
Business Impact
LEVEL 2:
Learning and
Confidence
LEVEL 3:
Application and
Implementation
LEVEL 1:
Reaction and
Planned Action
Evaluation
Planning
Develop
Evaluation
Plans and
Baseline Data
Collect Data
During
Program
Implementation
Collect Data
After
Program
Implementation
Isolate the
Effects of
Program
Convert Data
to Monetary
Value
Calculate
the Return On
Investment
Generate
Impact
Study
Capture Costs
Of Program
Identify
Intangibles
Intangible Benefits
Develop/
Review
Objectives of
Program
ROI Calculation
ROI =
Net Project Benefits
Project Costs
Cost of project $230,000
Benefits of project (1st year) $430,000
ROI =
$430,000-$230,000
= 087 x 100 = 87%
$230,000
Evaluation Framework
Level Measurement Focus
1. Reaction & Planned
Action
Measure participant reaction to the program and
captures planned action.
2. Learning &
Confidence
Measures changes in knowledge and skills.
3. Application &
Implementation
Measures implementation, actions, and changes in
behavior on the job.
4. Business Impact Measures changes in business impact variables.
5. Return on
Investment
Compares monetary benefits of the impact of the
program.
15
Guiding Principles
1. When conducting a higher-level evaluation, collect
data at lower levels.
2. When planning a higher level evaluation, the
previous level of evaluation is not required to be
comprehensive.
3. When collecting and analyzing data, use only the most
credible sources.
4. When analyzing data, select the most conservative
alternatives for calculations.
5. Use at least one method to isolate the effects of the
program or project.
6. If no improvement data are available for a population or
from a specific source, assume that no improvement has
occurred.
7. Adjust estimates of improvements for the potential error
of the estimates.
8. Avoid use of extreme data items and unsupported claims
when calculating ROI calculations.
9. Use only the first year of annual benefits in the ROI
analysis of short-term solutions.
10. Fully load all costs of the solution, project, or program
when analyzing ROI.
11. Intangible measures are defined as measures that are
purposely not converted to monetary values.
12. Communicate the results of the ROI Methodology to all
key stakeholders.
The ROI Process
A comprehensive measurement and evaluation process
that generates six types of measures:
Reaction and Perceived Value
Learning and Confidence
Application and Implementation
Business Impact
Return on Investment
Intangible Measures
This balanced approach to measurement includes a
technique to isolate the effect of the program or solution.
Results-based Solutions
Performance solutions/projects are initiated,
developed and delivered with the end in mind.
Participants understand their responsibility to
obtain results with programs/solutions.
Support groups (management, supervisors, coworkers,
etc.) help to achieve results from
performance solutions.
A comprehensive measurement and evaluation
system is in place for each program/project.
Variety of approaches utilized to measure
contribution, representing a balanced viewpoint.
Follow-up evaluations (Application, Impact, and
ROI) are developed for targeted solutions/projects
and results are reported to a variety of
stakeholders.
16
Chain of Impact
Reaction & Planned Action
Learning
Application & Implementation
Impact
ROI
Isolate the Effects of the Program
Intangible Benefits
Notes
Multiple Stakeholders
Reaction & Planned Action
Learning
Application & Implementation
Organization
Impact
Stakeholder 2
Impact
Stakeholder 3
Impact
ROI
Intangibles
ROI
Intangibles
ROI
Intangibles
Characteristics of Evaluation Levels
Power to
Chain of Value of Show Frequency Difficulty of
Impact Information Focus Results of Use Assessment
Lowest Consumer Lowest Frequent Easy
Reaction
Learning
Application
Impact
ROI
Highest Client Highest Infrequent Difficult
Consumers: The customers who are actively involved in the process.
Clients: The customers who fund, support, and approve the project.
17
Guiding Principle #1
Guiding Principle #2
When Selecting Programs for Level 4 and Level 5 Evaluation,
Consider the Following
Benchmarking*
• Life cycle of the program 14%
• Linkage of program to operational goals and issues 29%
• Importance of program to strategic objectives 50%
• Executive interest in the evaluation 48%
• Cost of the program 52%
• Visibility of the program 45%
• Size of target audience 6%
• Investment of time required 7%
*2007 Survey of Users, N = 235
Level 5
Level 4
Level 2
Level 1
Cost-benefit
comparison
Changes in
outcome measures
Changes in
performance
Need for skills or
knowledge
Preferences
Evaluation
Purpose
Program
Need
Program
Profile
Stakeholder
Needs
Level 3
18
Evaluation Targets
Percent of Projects
Level Current Target Suggested Benchmarking*
0 Input 100% 100%
1 Reaction 90 – 100% 79%
2 Learning 40 – 60% 54%
3 Application 30 % 31%
4 Impact 10 – 20% 14.4%
5 ROI 5 – 10% 4.3%
The Evaluation Framework Serves Three Purposes
The evaluation framework serves roles beyond categorizing evaluation data. The
framework also guides the development of program objectives and ensures stakeholder
needs are clearly identified.
By beginning the needs assessment at the highest level of need, an appropriate
solution can be identified. This is the first step toward linking your programs with
business results.
By developing program objectives at each level, the program can be positioned
for success. Program objectives represent the measures to be taken during the
evaluation process, thereby, eliminating the guesswork when a senior executive
says ―Show Me the Money!‖ This is your second step toward linking your
programs with business results.
By evaluating at each of the five levels, results important to all stakeholders are
reported and business impact and ROI can be connected to what people do with
what they learned. Isolating program effect on results during the evaluation
process is the third step toward linking your programs with business results.
19
Matching Evaluation Levels with Objectives
Instructions: For each objective listed below, indicate the level of evaluation at which
the objective is aimed.
1. Reaction 4. Business Impact
2. Learning 5. Return on Investment
3. Application
Objective
Evaluation
Level
After completing this program or project, participants should:
1. Improve work group productivity by 20%
2. Initiate at least three cost reduction projects in 15 days.
3. Achieve an average cost reduction of $20,000 per project.
4. Increase the use of counseling discussion skills in 90% of situations where work
habits are unacceptable.
5. Achieve a 2:1 benefit to cost ratio one year after the new performance-based
program is implemented.
6. Develop an understanding of how the employee assistance program works.
7. Increase the job satisfaction index by 25% in 3 months.
8. Integrate patient records into new database following the 5-step process.
9. Increase research grant dollars 20%.
10. Achieve a leadership simulation score average of 75 out of a possible 100.
11. Conduct a performance review meeting with direct reports to establish
performance improvement goals.
12. Receive a 4 out of 5 rating on appropriateness of new ethics policy.
13. Decrease the time to recruit new professional staff from 35 days to 20 days.
14. Complete all steps on their action plan in 60 days.
15. Achieve a positive reaction to flextime work schedule system.
16. Be involved in career enhancement program at a participation rate of 15%.
17. Decrease the number of security breaches of patient records.
18. Achieve a post-test score increase of 30% over pre-test.
19. Utilize new software daily as reflected by an 80% score on an unscheduled audit
of use.
20. Submit ideas or suggestions for improvement in the first year (10% target).
20
The levels serve three purposes!
Evaluation
Objectives
Needs
Level 3 and 4 Objectives Provide
Direction to designers and developers
Guidance to instructors and facilitators
Goals for participants
Satisfaction for program sponsors
A framework for evaluators
How will you build your team’s capability to write Level 3 and Level 4 objectives?
21
Program Alignment
V Model
Start Here End Here
Payoff Needs 5 ROI Objectives 5 ROI
Business Needs 4 Impact Objectives 4 Impact
Performance Needs 3 Application Objectives 3 Application
Learning Needs 2 Learning Objectives 2 Learning
Preference Needs 1 Reaction Objectives 1 Reaction
Project
Initial
Analysis
Measurement
and Evaluation
Business Alignment and Forecasting
The ROI Process Model
22
Program Alignment
V Model
Start Here End Here
Payoff Needs 5 ROI Objectives 5 ROI
Business Needs 4 Impact Objectives 4 Impact
Performance Needs 3 Application Objectives 3 Application
Learning Needs 2 Learning Objectives 2 Learning
Preference Needs 1 Reaction Objectives 1 Reaction
_____________________
_____________________
___________________
Project
Initial
Analysis
Measurement
and Evaluation
Discussions between
team member and
supervisor are not
occurring when there is
an unplanned absence
Deficiency in counseling/
discussion skills
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_____________________
_______________
_______________
_______________
_______________
___
_____________________
_____________________
_____________________
_____________________
_____________________
Objectives Evaluation
Business Alignment and Forecasting The ROI Process Model
One-day counseling skills
workshop must provide usable,
necessary and relevant skills;
facilitator-led; participants are
supervisors
Unplanned absenteeism is
9% and increasing; greater
than benchmarking of 5%
Absenteeism is
costing $100,000
monthly
Needs
EXERCISE: Complete
Objectives and Evaluation
23
The Alignment Process
Start Here V Model End Here
Payoff Needs 5 ROI Objectives 5 ROI
Business Needs 4 Impact Objectives 4 Impact
Performance Needs 3 Application Objectives 3 Application
Learning Needs 2 Learning Objectives 2 Learning
Preference Needs 1 Reaction Objectives 1 Reaction
Project
Absenteeism is
costing $100,000
monthly
Unexpected absenteeism is
9 % and growing;
benchmark data is at 5%
Discussions between team
member and supervisor are
not occurring when there is
an unplanned absence
Deficiency in counseling/
discussion skills
One-day counseling skills
workshop must provide usable,
necessary and relevant skills;
facilitator-led; participants are
supervisors
_____________________
_____________________
ROI of 25%
Reduce absenteeism to
5% six months after
course
Counseling discussions
conducted in 95% of
situations when an
unexpected absence occurs
Acquisition of
counseling skills are
demonstrated
Program receives favorable rating
of 4 out of 5 on need for program,
relevance of the program and the
practicality of program
Calculate ROI;
Monitor absenteeism
records for six months
Follow-up questionnaire
to participants to check
frequency of discussions
– three months
Skills practice session
during program;
Reaction questionnaire at the
end of program
Business Alignment and Forecasting The ROI Process
Model
24
Create a Green Organization
OPCW
-Sample-
Level Needs Objectives Evaluation Level
5
Help protect the environment
Save costs
ROI target of 10%
Project benefits compared to costs
5
4
Reduce carbon emissions
High energy costs
Rising costs of operations
Reduce carbon emission by ___
Reduce energy costs by ___
Reduce materials/supplies by ____
Organization records
4
3
Not recycling materials
Need to change consumption habits
Need to use less materials and
supplies
Not making environmentally friendly
choices
Six months after the project begins, employees will
o Recycle in eight categories
o Alter consumption patterns
o Reduce usage, conserve
o Use environmentally friendly supplies
Self-assessment questionnaire
Recycle records
Records of purchasing eco-friendly products
3
2
How actions effect the environment
Specific green methods
Environmental issues
All employees will learn
o Environmental issues
o Specific green actions they can take
o How to make eco-friendly choices
Self assessment questionnaire
Environment Quiz
2
1
Employees must see project as
necessary, important, relevant
feasible
Program receives favorable rating of 4 out of 5 on:
o Necessary to OPCW
o Relevance to OPCW
o Importance of adhering concepts in support of
public good
Reaction questionnaire administered to all project
participants
1
25
Police Project (Malaysia)
UN Security
-Sample-
Level Objectives Evaluation Level
5
Police officers are causing
problems
Break-even (BCR = 1:1)
Program benefits compared to program costs
5
4
Crime is too high in four
categories
Citizen complaints about police
is excessive
In one year:
o Crime in four categories will be reduced by
___
o Citizen complaints about police officers with
be reduced by ____
City records
Country records
4
3
Not following procedures
Rule of law not followed
Conflicts not resolved properly
Actions inconsistent
Police officers will:
o Follow procedures
o Enforce laws consistently
o Resolve conflicts
Observation
Questionnaire
3
2
Legal procedures
Rule of law
Conflict resolution
Communication
Police officers will demonstrate knowledge of:
o Legal procedures
o Rule of law
o Conflict resolution
o Communication
Role plays
Demonstrations
Simple quiz
Self assessment
2
1
Police officers must see this
program as:
Necessary
Helpful
Relevant
Important to their success
Program receives favorable rating of 4 out of 5 on the
following measures:
o Necessary
o Helpful
o Relevant
o Important to their success
Reaction questionnaire administered to police officers
1
26
Recidivism Program
-Sample-
Level Needs Objectives Evaluation Level
5
Cost of housing repeat offenders is
$102,306,520.
Of that amount $17,179,441
is dedicated to repeat drug related
offensives.1
1:1 BCR or 0% ROI
Program costs compared to
monetary benefits of program.
Monetary benefit determined by
costs savings of housing repeat
offenders.
5
4
Reduce recidivism of drug related
offenders
Reduce # of rearrests
Reduce # of convictions
Increase time period of rearrests
or reconviction
Monitor performance of measures
Compare to group not
participating in program
4
3
Offenders continue to engage in drugrelated
crimes after incarceration.
Engage in treatment early in
process
Attend treatment sessions
Appear in status hearings
Eliminate drug use
Identify barriers to participating in
program
Monitor performance records
Self report via questionnaire
3
2
Offenders do not take seriously the
need to change
Offenders do not fully understand the
continued consequence of their
ongoing behavior
Offenders do not know of a support
system that will assist them in
modifying their behavior
Upon introduction to the drug court
program:
Offenders understand costs and
benefits of participation in the
program
Offenders understand program
process and the requirements
they must meet
Offenders understand the
consequences of their not
participating in the program
Verbal acknowledgement
obtained by judge and parole
officer
2
1
Drug Court Program
Offenders recognize the program
as relevant and important to their
future success
Offenders commit to participation
Verbal acknowledgement
obtained by judge and parole
officer
1
According to the State of Alabama Department of Corrections October 2005 status report, there are currently 8,504 habitual offenders housed in State facilities; there are 1,428 habitual offenders
associated with drug crimes. The cost of housing inmates is $12,030.40 per year. http://www.communitycorrectionsworks.org.
26
27
Key Alignment Questions
Is this a problem worth
solving?
Is there a potential pay
off?
How should the solution
be structured?
What is the specific
measure?
What happens if we do
nothing?
What is occurring or not
occurring on the job that
influences the business
measure?
What skills or
knowledge is needed to
support the job
performance need?
Which business
measure improved?
How much is related to
the program?
What has changed?
Which skills/knowledge
have been applied?
What did they learn?
Who did they meet?
What was the reaction
to the program?
Do we intend to
implement the program?
What is the actual ROI?
What is the BCR?
Needs Program
Assessment Objectives Evaluation
Business Impact Business
Needs Objectives Impact
Job Performance Application Application &
Needs Objectives Implementation
Skills/Knowledge Learning Learning &
Needs Objectives Confidence
Satisfaction Reaction &
Preferences Objectives Planned Action
4 4
3 3
2 2
1 1
Potential ROI ROI
Payoffs Objectives
5 5
28
Developing Reaction Objectives
Measuring Reaction and Satisfaction
Reaction objectives are critical in this measurement chain because they:
• Describe expected immediate and long term satisfaction
• Describe issues that are important to the success of the program
• Provide basis for evaluating the beginning of the measurement chain of impact
• Place emphasis on planned action, if feasible
The best reaction objectives:
• Identify issues that are important and measurable
• Are attitude-based, clearly worded, and specific
• Specify the participant has changed in thinking or perception as a result of the
program
• Underscores the linkage between attitude and the success of the program
• Represent a satisfaction index from key stakeholders
• Have the capability to predict program success
Key questions are:
• How relevant is this program?
• How important is this program?
• Are the facilitators effective?
• How appropriate is this program?
• Is this new information?
• Is this program rewarding?
• Will you implement this program?
• Will you use the concepts/advice?
• What would keep you from implementing objectives from this program?
• Would you recommend the program to others?
Examples of Level 1 Objectives
1.
2.
3.
29
Developing Learning Objectives
Measuring Skills and Knowledge Enhancement
Learning objectives are critical to measuring learning because they:
• Communicate expected outcomes from instruction
• Describe competent performance that should be the result of training or learning
• Provide basis for evaluating learning
• Focus learning for participants
The best learning objectives:
• Describe behaviors that are observable and measurable
• Are outcome-based, clearly worded and specific
• Specify what the learner must do (not know or understand) as a result of the
training
• Have three components:
1. Performance—what the learner will be able to do at the end of the training
2. Condition—circumstances under which the learner will perform the task
3. Criteria—degree or level of proficiency that is necessary to perform the job
Three types of learning objectives are:
• Awareness — familiarity with terms, concepts, processes
• Knowledge — general understanding of concepts, processes, etc.
• Performance — able to demonstrate the skill (at least at a basic level)
Examples of Level 2 Objectives
1. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
2. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
3. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
30
Developing Application Objectives
Measuring on the Job Application and Implementation
Application objectives are critical to measuring application of skills and knowledge
because they:
• Describe expected intermediate outcomes
• Describe competent performance that should be the result of the program
• Provide basis for evaluation of on the job performance changes
• Place emphasis on applying what was learned
The best application objectives:
• Identify behaviors that are observable and measurable
• Are outcome-based, clearly worded and specific
• Specify what the participant will change or has changed as a result of the
program
• May have three components:
1. Performance – what the participant has changed/accomplished at a specified
follow-up time after the program
2. Condition – circumstances under which the participant performed the task
3. Criteria – degree or level of proficiency under which the task or job was
performed
Two types of application objectives are:
• Knowledge based – general use of concepts, processes, etc.
• Behavior based – able to demonstrate the use of the skill (at least at a basic
level)
Key questions are:
• What new or improve knowledge will be applied on the job?
• What is the frequency of skill application?
• What new tasks will be performed?
• What new steps will be implemented?
• What new action items will be implemented?
• What new procedures will be implemented or changed?
• What new guidelines will be implemented or changed?
• What new processes will be implemented or changed?
Examples of Level 3 Objectives
1. ___________________________________________________________________
___________________________________________________________________
2. ___________________________________________________________________
___________________________________________________________________
3. ___________________________________________________________________
___________________________________________________________________
31
Developing Impact Objectives
Measuring Business Impact
Impact objectives are critical to measuring business performance because they:
• Describe expected outcomes
• Describe business unit performance that should be the result of training or
instruction
• Provide basis for measuring the consequences of application of skills and
knowledge
• Place emphasis on achieving bottom line results
The best impact objectives:
• Must contain measures that are linked to the skills and knowledge taught in the
program
• Describe measures that are easily collected
• Are results-based, clearly worded and specific
• Specify what the participant has accomplished in the business unit as a result of
the training
Four types of impact objectives involving hard data are:
• Output focused
• Quality focused
• Cost focused
• Time focused
Three common types of impact objectives involving soft data are:
• Customer service focused
• Work climate focused
• Job satisfaction focused
Examples of Level 4 Objectives
1. ___________________________________________________________________
___________________________________________________________________
2. ___________________________________________________________________
___________________________________________________________________
3. ___________________________________________________________________
___________________________________________________________________
32
Developing Level 3 and 4 Objectives
Think of a program that is linked to important organizational goals. Develop at least two
Level 3 and Level 4 objectives for the program. Make any assumptions you need to
complete the objectives.
Program Title:
Target Audience: Duration:
Level 3 Objectives:
After completing this program, participants will:
1. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
2. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
Level 4 Objectives:
After participants apply learned skills/behavior, their performance will impact:
1. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
2. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
33
Plan Your Project Evaluation
Program: _______________________________________________________
Evaluation Team: ________________________________________________
Expected Date of Completion: ______________________________________
1. What is your purpose in conducting an evaluation on this program?
2. What are the program objectives at each level of evaluation?
Level 1 ______________________________________________________
Level 2 ______________________________________________________
Level 3 ______________________________________________________
Level 4 ______________________________________________________
Level 5 ______________________________________________________
3. What are your measures of success for each objective?
Level 1 ______________________________________________________
______________________________________________________
Level 2 ______________________________________________________
______________________________________________________
Level 3 ______________________________________________________
______________________________________________________
Level 4 ______________________________________________________
______________________________________________________
Level 5 ______________________________________________________
______________________________________________________
34
Data Collection Plan
Program: ______________________________ Responsibility: _________________________ Date: ____________
Level Broad Program Objective(s) Measures
Data Collection
Method/Instruments Data Sources Timing Responsibilities
1
REACTION AND
PLANNED ACTION
2
LEARNING AND
CONFIDENCE
3
APPLICATION AND
IMPLEMENTATION
4
BUSINESS IMPACT
5 ROI Comments:__________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________
35
ROI Analysis Plan
Program:______________________________ Responsibility:_________________________ Date:____________
Data Items
(Usually
Level 4)
Methods for
Isolating the
Effects of the
Program/
Process
Methods of
Converting Data
to Monetary
Values
Cost
Categories
Intangible
Benefits
Communication
Targets for Final
Report
Other
Influences/
Issues During
Application Comments
36
Collecting Post Program
Data
The Methods: Level 3 Level 4
Follow-up Surveys
Follow-up Questionnaires
Observation On the Job
Interviews with Participants
Follow-up Focus Groups
Program Assignments
Action Planning
Performance Contracting
Program Follow-up Sessions
Performance Monitoring
Factors to Consider
When selecting data collection methods
Type of data
Time – Participant / Supervisor
Costs
Accuracy – Validity / Reliability
Utility
Culture / Philosophy
When determining timing of follow-up
Availability of data
Ideal time for behavior change (Level 3)
Ideal time for business impact (Level 4)
Convenience of collection
Constraints on collection
Option 1, When You Don’t Have a Clue
1. How did you use the material from this project or program?
2. What influence did it have in your work? Team?
3. What specific measure was influenced? Define it.
4. What is the unit value of the measure? (Profit or Cost)
5. What is the basis of this value?
6. How much did the measure change since the project was implemented?
7. What is the frequency of the measure? Daily, weekly, monthly, etc.
8. What is the total annual value of the improvement?
9. List the other factors that could have caused this total improvement?
10. What percent of the total improvement can be attributed to this project?
11. What is your confidence estimate, expressed as a percent, for the above data?
0% = no confidence; 100% = certainty
37
Option 2, When the Measure Is in a Defined Set
1. To what extent did this project or program positively influence the following measures:
Significant
Influence
No
Influence
5 4 3 2 1 n/a
productivity
sales
quality
cost
efficiency
time
employee satisfaction
customer satisfaction
other
2. What other measures were positively influenced by this project?
3. Of the measures listed above, which one is most directly linked to the project? (check
only one)
productivity sales quality
cost efficiency time
employee satisfaction customer satisfaction other
4. Please define the measure above.
5. Indicate the specific unit of measurement.
6. How much did this measure improve since you began this project?
7. What is the frequency of the measure? daily weekly monthly annually
8. For this measure, what is the monetary value of improvement for one unit of this
measure? Although this is difficult, please make every effort to provide the value.
9. Please state your basis for the estimated value of one unit of improvement you
indicated above.
10. What is the total annual value of improvement in the measure you selected above?
38
Option 2 (cont.)
11. List the other factors that have caused this total annual improvement.
12. Recognizing that other factors may have caused this improvement, estimate the
percent of improvement related directly to this project of program?
%
13. What confidence do you place in the estimates you have provided in the prior
questions? (0% is no confidence, 100% is certainty.)
%
Option 3, When the Measure Is Known
1. Please define the first measure connected to your project.
2. Define the unit of measure.
3. For this measure, what is the monetary value of improvement for one unit of this
measure?
4. Please state your basis for the value of the unit of improvement you indicated above.
5. For the measure listed as most directly linked to the program, how much has this
measure improved in performance?
6. Indicate the frequency base for the measure. Daily, weekly, monthly, quarterly.
7. What is the annual value of improvement in the measure you selected above?
Multiply the increase (Question 5) by the frequency (Question 6) times the unit of
value (Question 4).
8. List the other factors that could have influenced these results.
9. Recognizing that the other factors could have influenced this annual value of
improvement, please estimate the percent of improvement that is attributable (or
isolated) to the program. Express as a percentage out of 100%.
10. What confidence do you place in the estimates you have provided in the questions
above? A 0% is no confidence, a 100% is certainty.
Increasing Questionnaire Response Rates
Provide advance communication about the questionnaire
Clearly communicate the reason for the questionnaire
Indicate who will see the results of the questionnaire
Show how the data will be integrated with other data
Keep the questionnaire simple and as brief as possible
Keep questionnaire responses anonymous – or at least confidential
Make it easy to respond; include a self-addressed, stamped envelope/e-mail
Use the local manager to distribute the questionnaires, show support, and
encourage response
If appropriate, let the target audience know that they are part of a carefully
selected sample
Use one or two follow-up reminders
39
Increasing Questionnaire Response Rates (cont.)
Have the introduction letter signed by a top executive
Enclose a giveaway item with the questionnaire (pen, money, etc.)
Provide an incentive (or chance of incentive) for quick response
Send a summary of results to target audience
Distribute questionnaire to a captive audience
Consider an alternative distribution channel, such as e-mail
Have a third party gather and analyze data
Communicate the time limit for submitting responses
Consider paying for the time it takes to complete the questionnaire
Review the questionnaire at the end of the formal session
Carefully select the survey sample
Allow completion of the survey during work hours
Add emotional appeal
Design questionnaire to attract attention, with a professional format
Let participants know what actions will be taken with the data
Provide options to respond (paper, email, web-site)
Use a local coordinator to help distribute and collect questionnaires
Frame questions so participants can respond appropriately and make the
questions relevant
Methods to Isolate
Program Effects
Use of a control group arrangement
Trend line analysis of performance data
Use of forecasting methods of performance data
Participant’s estimate of program impact (percent)
Supervisor’s estimate of program impact (percent)
Manager’s estimate of program impact
Use of expert/previous studies
Calculate/estimate the impact of other factors
Customer input
40
Use of Control Groups Example
Retention Solution at the Federal Information Agency
An opportunity to participate in a master’s degree program at agency expense and
agency time.
One hundred high-potential employees chosen for program for a three year
masters program in information science
Experimental group of one hundred were involved, another one hundred in
control group were not involved
Observed employee turnover performance of both groups during the same time
Collected evaluation data for both groups at the same time
Neither group is aware of the control group arrangement
How would you select the control group?
Criteria for Selecting the Control Group:
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
Control Group Method
What is the difference in improvement?
M1 M2
Control
Group
M1 Program M2 Experimental
Group
41
Example of Trend Line Analysis
Example of Estimation
Factor that
Influenced
Improvement
Percent of
Improvement
Caused By
Confidence
Expressed as a
Percent
Adjusted
Percent of
Improvement
Caused By
Program 60% 80% 48%
System Changes 15% 70% 10.5%
Environmental
Changes 5% 60% 3%
Compensation
Changes 20% 80% 16%
Other _______% _______% _______%
Total 100%
18.5% Pre Program Six-Month
Average
20%
10%
Projected Average —
Using Pre Data as a
Base 14.5%
7% Post
Program Six-
Month Average
Fraud Program Conducted
Fraud
Incidents
Rates
J F M A M J J A S O N D J
MONTHS
42
The Wisdom of Crowds*
One day in the fall of 1906, British scientist Francis Galton left his home in the town of
Plymouth and headed for a country fair. Galton was eighty-five years old and beginning
to feel his age, but he was still brimming with the curiosity that had won him renown—
and notoriety—for his work on statistics and the science of heredity. On that particular
day, what Galton was curious about was livestock.
Galton’s destination was the annual West of England Fat Stock and Poultry Exhibition, a
regional fair where the local farmers and townspeople gathered to appraise the quality
of each other’s cattle, sheep, chickens, horses, and pigs. Wandering through rows of
stalls examining workhorses and prize hogs may have seemed a strange way for a
scientist to spend an afternoon, but there was certain logic to it. Galton was a man
obsessed with two things: the measurement of physical and mental qualities and
breeding. And what, after all, is a livestock show but a big showcase for the effects of
good and bad breeding?
Breeding mattered to Galton because he believed that only a very few people had the
characteristics necessary to keep societies healthy. He had devoted much of his career
to measuring those characteristics, in fact, in order to prove that the vast majority of
people did not have them. His experiments left him with little faith in the intelligence of
the average person, ―the stupidity and wrong-headedness of many men and women
being so great as to be scarcely credible.‖ Galton believed, ―Only if power and control
stayed in the hands of the select, well-bred few, could a society remain healthy and
strong.‖
As he walked through the exhibition that day, Galton came across a weight-judging
competition. A fat ox had been selected and placed on display, and members of a
gathering crowd were lining up to place wagers on what the weight of the ox would be
after it had been slaughtered and dressed. For sixpence, an individual could buy a
stamped and numbered ticket; fill in their name, occupation, address, and estimate.
The best guesses would receive prizes.
Eight hundred people tried their luck. They were a diverse lot. Many of them were
butchers and farmers, who were presumably expert at judging the weight of livestock,
but there were also quite a few people who had no insider knowledge of cattle. ―Many
non-experts competed,‖ Galton wrote later in the scientific journal Nature. ―The average
competitor was probably as well fitted for making a just estimate of the dressed weight
of the ox, as an average voter is of judging the merits of most political issues on which
he votes.‖
* Taken from The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective
Wisdom Shapes Business, Economics, Societies and Nations. James Surowicki. New York.
Doubleday, 2004
43
Galton was interested in figuring out what the ―average voter‖ was capable of because
he wanted to prove that the average voter was capable of very little. So he turned the
competition into an impromptu experiment. When the contest was over and the prizes
had been awarded, Galton borrowed the tickets from the organizers and ran a series of
statistical tests on them. Galton arranged the guesses (totaling 787 – thirteen were
discarded because they were illegible) in order from highest to lowest and graphed
them to see if they would form a bell curve. Then, among other things, he added all the
contestants’ estimates, and calculated the mean of the group’s guesses. That number
represented, you could say, the collective wisdom of the Plymouth crowd. If the crowd
were a single person, that was how much it would have guessed the ox weighed.
Galton undoubtedly thought that the average guess of the group would be way off the
mark. After all, mix a few very smart people with some mediocre people and a lot of
dumb people, and it seems likely you’d end up with a dumb answer. But Galton was
wrong. The crowd had guessed that the ox, after it had been slaughtered and dressed,
would weigh 1,197 pounds. After it had been slaughtered and dressed, the ox weighed
1,198 pounds. In other words, the crowd’s judgment was essentially perfect. The
―experts‖ were not close. Perhaps breeding didn’t mean so much after all. Galton wrote
later: ―The result seems more creditable to the trustworthiness of a democratic
judgment than it might have been expected.‖ That was, to say the least, an
understatement.
What Francis Galton stumbled on that day in Plymouth was the simple, but powerful,
truth: under the right circumstances, groups are remarkably intelligent, and are often
smarter than the smartest people in them. Groups do not need to be dominated by
exceptionally intelligent people in order to be smart. Even if most of the people within a
group are not especially well-informed or rational, they can still reach a collectively wise
decision.
Questions for discussion:
1. What implications does this concept have in evaluation?
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
2. Can you cite other examples?
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
44
Credibility of Data is
Influenced by
Reputation of the source
o Source of data
o Source of the study
Biases
o Motives of the researcher
o Personal bias of the audience
Methodology Used
o Assumptions made in the analysis
o Realism of the outcome data
o Type of data
Scope of analysis
Data are Converted by
Converting output to contribution – standard value
Converting the cost of quality – standard value
Converting employee’s time – standard value
Using historical costs
Using internal and external experts
Using data from external databases
Linking with other measures
Using participants’ estimates
Using supervisors’ and managers’ estimates
Using staff estimates
Example
Cost of One Turnover from External Database
Salary of Middle Manager $70,000/annually
Value of Turnover* 150% of annual salary
Cost of Turnover $105,000
*Value obtained from industry-related study (external data)
45
Cost of a Sexual Harassment Complaint using Historical Costs and
Expert Input
Cost per complaint
$852,000
= $24,343
35
To Convert or Not Convert
Is there a standard value?
Is there a method to get there?
Can we get there with minimum resources?
Can we convince our executive in two minutes that the value is credible?
5-Step Data
Conversion
Step 1: Focus on a unit of measure
Step 2: Determine the value (V) of each unit
Step 3: Calculate the change in performance (ΔP)
Step 4: Determine the annual amount of change (AΔP)
Step 5: Calculate the total annual value of the improvement (AΔP x V)
35 Complaints
Actual Costs from
Records
Additional Estimated Costs
from Staff
Legal Fees, Settlements,
Losses, Material, Direct
Expenses
EEO/AA Staff Time,
Management Time
$852,000 Annually
46
Example using
Internal Experts
Step 1: One grievance
Step 2: V = $6,500 (from Director of Nursing and HR experts)
Step 3: ΔP = average of 7 out of 10 grievances prevented per month
due to program
Step 4: Annual ΔP =
Step 5: AΔP x V =
Which cost category is appropriate for ROI?
A B
Operating Costs
Support Costs
Administrative Costs
Participant Compensation and
Facility Costs
Classroom Costs
C D
Program Development Costs
Administrative Costs
Classroom Costs
Participant Costs
Analysis Costs
Development Costs
Delivery Costs
Overhead/Administrative Costs
Evaluation Costs
Notes
47
Fully-Loaded Cost
Profile
Assessment costs (prorated)
Development costs (prorated)
Program materials
Instructor/Facilitator costs
Facilities costs
Travel/Lodging/Meals
Participant salaries and benefits
Administrative/Overhead costs
Evaluation costs
Potential
Intangible
Benefits
Increased job satisfaction
Increased organizational commitment
Improved teamwork
Improved customer service
Reduced complaints
Reduced conflicts
Reduced stress
ROI is reported in two ways
Benefit Cost Ratio (BCR) =
Program Benefits
Program Costs
ROI (%) =
Net Program Benefits
X 100
Program Costs
Notes
48
Calculate the ROI
Costs per program (25 participants) – $80,000
Benefits per program (1st year) – $240,000
Benefit Cost Ratio (BCR) =
ROI =
X 100 =
What is an
acceptable
ROI?
Set the value as with other investments – 15%
Set the value slightly above other investments – 25%
Set at break even – 0%
Set at client expectations
When properly implemented, high ROI values can be achieved with
programs on:
Leadership
Team Building
Management Development
Supervisor Training
Sales Training
100% to 700% ROI is not
uncommon
49
Healthcare Organization
Sexual Harassment Prevention Workshop
Target Group: All supervisors and managers (655) with subsequent meetings with all
employees (6,844)
Data Collection
(3) Self Assessment Questionnaire – 6 months after program
(3) Employee Survey (25% sample) – 6 months after program
(4) Complaint and Turnover Records – 12 months after program
Isolating the Effects of the Program
Complaints – Trendline Analysis
Turnover – Forecasting
Converting Data to Monetary Values
Complaints – Historical costs and input from experts (internal EEO/AA staff)
Turnover – External studies within industry
Program Costs
Fully loaded to include needs assessment, development, coordination, participant
salaries and benefits, and evaluation
Total Costs = $277,987
Intangible Benefits
Job Satisfaction, absenteeism, stress reduction, community image, and recruiting
ROI Calculation
Monetary benefits from complaint reduction
Value of one internal complaint = $24,343
Annual improvement related to program = 14.8 complaints (prevented)
_______________________________________________
Monetary benefits from turnover reduction
Value of one turnover statistic = $20,887
Annual improvement related to program = 136 turnovers (prevented)
________________________________________________
Calculate the following:
BCR =
Total Benefits
=
Program Costs
ROI =
Total Benefits – Program Costs
X 100 =
Program Costs
50
Communication
Why the concern?
Measurement and evaluation are meaningless without communication
Communication is necessary for making improvement
Communication is a sensitive issue
Different audiences need different information
Principles
Keep communication timely
Target communication to specific audiences
Carefully select communication media
Keep communication consistent with past practices
Incorporate testimonials from influential individuals
Consider the training function’s reputation when developing the overall
strategy
Reports
There are four types
1. Complete report
2. Executive summary
3. General audience summary
4. Streamlined report
The complete report includes the details
General information
Methodology for impact study
Data analysis
Costs
Results
Barriers and enablers
Conclusions and recommendations
Exhibits
51
Sample Table of Contents for an ROI Impact Study
Table of Contents
List of Tables
List of Figures
List of Exhibits
Part I The Challenge and The Approach
Section 1: Introduction
Section 2: The Program
Section 3: Model for Impact Study
Section 4: Data Collection Strategy
Part II The Results
Section 5: Reaction and Satisfaction
Section 6: Learning
Section 7: Application and Implementation
Section 8: Business Impact
Section 9: Program Costs
Section 10: ROI and Its Meaning
Section 11: Intangible Benefits
Part III Recommendations
Section 12: Barriers and Enablers
Section 13: Suggestions for Improvement
Section 14: Conclusions
Section 15: Recommendations
52
ROI is Credible for Public Sector
• Common categories of data
• Systematic, step-by-step process
• Conservative standards
• Results-based approach
• High level of use
• Client focused
Satisfies all stakeholders
ROI is Feasible for Public Sector
• Not very expensive
• Many shortcut methods
• Time requirement can be managed
• Fits all types of programs
• Technology helps with costs/time
• Implementation is planned/systematic
Have No Fear
• ROI is a process improvement tool – designed to improve projects and programs
• ROI is not designed for performance review for individuals
• Every study reveals opportunities for changes
• Negative results represent the best opportunity to learn
• Negative results have a positive story
• Don’t wait for a sponsor to ask for Impact and ROI
Implementation Strategies in the Public Sector
• Brief, train, educate
• Involve the staff – early and often
• Emphasize process improvement
• Explain why – routinely
• Build it into programs – not add it on
• Provide resources
• Use the results appropriately
• Celebrate and recognize
ROI Best Practices
1. The ROI methodology is implemented as a process improvement tool and not a
performance evaluation tool for the staff.
2. ROI impact studies are conducted very selectively, usually involving 5-10% of
projects and programs.
3. A variety of data collection methods are used in ROI analysis.
4. For a specific ROI evaluation, the effects of the program are isolated from other
influences.
5. Business impact data are converted to monetary values.
6. ROI evaluation targets are developed, showing the percent of programs evaluated at
each level.
7. The ROI methodology generates a micro level scorecard.
53
8. ROI methodology data are being integrated to create a macro scorecard for the
learning/ development function
9. The ROI methodology is being implemented for about 3-5% of the budget.
10. ROI forecasting is being implemented routinely.
11. The ROI methodology is used as a tool to strengthen/improve the programs and
processes.
*Based on benchmarking with over 200 organizations using ROI routinely
Cost-Saving Approaches to ROI
• lan for evaluation early in the
process
• Build evaluation into the process
• Share the responsibilities for
evaluation
• Require participants to conduct
major steps
• Use short-cut methods for major
steps
• Use sampling to select the most
appropriate programs for ROI analysis
• Use estimates in the collection and
analysis of data
• Develop internal capability to
implement the ROI process
• Streamline the reporting process
• Utilize web-based software to reduce
time
Status of ROI Use*
Is your leadership and/or clients asking about ROI?
1. Yes
2. No
Does your organization have a measurement/evaluation strategy?
1. Yes
2. No and no plans for it in the
near future
3. No, but is it in process of being
developed
54
Does your organization have a measurement or evaluation function?
1. Yes
2. No and no plans for it in the near
future
3. No, but being developed
How many ROI studies has your organization completed to date?
1. 1
2. 2 – 3
3. 4 – 8
4. 9 – 15
5. 16 or more
*2007 Survey of Users, N = 235
What happens if we do nothing?
ROI Reality
• Impact/ROI information is desired by clients/ executives
• The impact/ROI process provides a balanced, credible approach with six types of
data
• All types of organizations are routinely using impact/ROI
• The impact/ROI process can be implemented without draining resources
• The impact/ROI process is a long-term goal for many organizations.
• Budget?
• Influence?
• Support?
• Other Issues?
55
Impact/ROI Standards*
1. When a higher-level evaluation is conducted, data must be collected at lower levels.
2. When an evaluation is planned for a higher level, the previous level of evaluation
does not have to be comprehensive.
3. When collecting and analyzing data, use only the most credible source.
4. When analyzing data, choose the most conservative among the alternatives.
5. At least one method must be used to isolate the effects of the solution.
6. If no improvement data are available for a population or from a specific source, it is
assumed that little or no improvement has occurred.
7. Estimates of improvements should be adjusted for the potential error of the
estimate.
8. Extreme data items and unsupported claims should not be used in ROI calculations.
9. Only the first year of monetary benefits should be used in the ROI analysis for shortterm
solutions.
10. Costs of the solution should be fully loaded for ROI analysis.
11. Intangible measures are defined as measures that are purposely not converted to
monetary values.
12. The results from the ROI methodology must be communicated to all key
stakeholders.
* Developed by the ROI Institute with input from users.
The standards are used by more than 5,000 organizations in 52 countries.
56
ROI Quiz
True or False? Please choose the answer you feel is most correct
T F
1. The ROI Methodology generates just one data item, expressed as a percentage.
2. A program with monetary benefits of $200,000 and costs of $100,000 translates
into a 200% ROI.
3. The ROI Methodology is a tool to improve process and projects,
learning/development process.
4. After reviewing a detailed ROI impact study, senior executives will usually require
ROI studies on all programs.
5. ROI studies should be conducted very selectively, usually involving 5-10% of
programs.
6. While it may be a rough estimate, it is always possible to isolate the effects of a
program on impact data
7. A program costing $100 per participant, designed to teach basic skills with job
related software, is an ideal program for an ROI impact study.
8. Data can always be converted to monetary value, credibly.
9. The ROI Methodology contains too many complicated formulas.
10. The ROI Methodology can be implemented for about 3-5% of my budget.
11. ROI is not future oriented; it only reflects past performance.
12. ROI is not possible for soft skills programs.
13. If an ROI impact study, conducted on an existing program, shows a negative ROI,
the client is usually already aware of the program’s weaknesses.
14. The best time to consider an ROI evaluation is three months after the program is
completed.
15. In the early stages of implementation, the ROI Methodology is a process
improvement tool and not performance evaluation for the team.
16. If senior executives are not asking for ROI, there is no need to pursue the ROI
Methodology.
57
So, how did you do?
Now that the answers to the quiz have been explained, see how you fared. Tally
your scores. Based on the interpretations below, what is your ROI acumen?
No. of Correct
Responses Interpretation
14-16 You could be an ROI consultant
10-13 You could be a speaker at the next ROI Conference
7-9 You need a copy of a thick ROI book
4-6 You need to attend a two-day ROI workshop
1-3 You need to attend the ROI certification
58
Sample of Published ROI Studies
Measuring the ROI: Key Impact Measures: ROI
Performance Management
(Restaurant Chain)
A variety of measures, such as productivity, quality,
time, costs, turnover, and absenteeism
298%1
Process Improvement Team
(Apple Computer)
Productivity and labor efficiency 182%1
Skill-Based Pay
(Construction Materials Firm)
Labor costs, turnover, absenteeism 805%2
Sexual Harassment
Prevention (Health Care
Chain)
Complaints, turnover, absenteeism, job satisfaction 1052%2
Safety Incentive Plan (Steel
Company)
Accident frequency rate, accident severity rates 379%2
Diversity (Nextel
Communications)
Retention, employee satisfaction 163%6
Retention Improvement
(Financial Services)
Turnover, staffing levels, employee satisfaction 258%3
Absenteeism
Control/Reduction Program
Absenteeism, customer satisfaction 882%2
Stress Management Program
(Electric Utility)
Medical costs, turnover, absenteeism 320%2
Executive Leadership
Development (Financial)
Team projects, individual projects, retention 62%2
E-Learning (Petroleum) Sales 206%2
Internal Graduate Degree
Program (Federal Agency)
Retention, individual graduate projects 153%4
Executive Coaching (Nortel
Networks)
Several measures, including productivity, quality, cost
control, and product development time
788%5
Competency Development
(Veteran’s Health
Administration)
Time savings, work quality, faster response 159%4
First Level Leadership
Development (Auto Rental
Company)
Various measures – at least two per manager 105%7
59
References for Published Studies
1. In Action: Measuring Return on Investment, Volume 3. Patricia P. Phillips, Editor; Jack J. Phillips,
Series Editor. Alexandria: ASTD, 2001.
2. The Human Resources Scorecard: Measuring the Return on Investment. Jack Phillips, Ron D. Stone,
Patricia P. Phillips. Woburn: Butterworth-Heinemann, 2001.
3. In Action: Retaining Your Best Employees. Patricia P. Phillips, Editor; Jack J. Phillips, Series Editor.
Alexandria: ASTD and the Society for Human Resource Management, 2002.
4. In Action: Measuring ROI in the Public Sector. Patricia P. Phillips, Editor. Alexandria: ASTD, 2002.
5. In Action: Coaching for Extraordinary Results. Darelyn J. Mitch, Editor; Jack J. Phillips, Series Editor.
Alexandria: ASTD, 2002.
6. In Action: Implementing Training Scorecards. Lynn Schmidt, Editor; Jack J. Phillips, Series Editor.
Alexandria: ASTD, 2003.
7. The Leadership Scorecard, Jack J. Phillips and Lynn Schmidt, Woburn: Butterworth-Heinemann,
2004.
Additional Resources
Measuring for Success: What CEOs Really Think About Learning Investments. Jack J. Phillips and Patti
P. Phillips. ASTD, 2010.
Beyond Learning Objectives: Develop Measurable Objectives That Link To The Bottom Line. Jack J.
Phillips and Patti P. Phillips. ASTD, 2008.
The Measurement and Evaluation Series. ROI Fundamentals: Why and When to Measure Return on
Investment; Data Collection: Planning For and Collecting All Types of Data; Isolation of Results:
Defining the Impact of the Program; Data Conversion: Calculating the Monetary Benefits, Costs and
ROI: Evaluating at the Ultimate Level; Communication and Implementation: Sustaining the Practice.
Patricia P. Phillips and Jack J. Phillips. Pfeiffer, 2008
ROI In Action Casebook. Patricia P. Phillips and Jack J. Phillips. Pfeiffer, 2008.
The Value of Learning: How Organizations Capture Value and ROI and Translate It into Support,
Improvement, and Funds. Jack J. Phillips and Patricia P. Phillips. Pfeiffer, 2007.
Show Me the Money. Jack J. Phillips and Patricia P. Phillips. Berrett-Koehler, San Francisco, CA, 2007
Return on Investment in Training and Performance Improvement Programs,2nd Edition, Jack J. Phillips.
Woburn: Butterworth Heinemann, 2003.
How to Measure Training Results: A Practical Guide to Tracking the Six Key Indicators. Jack J. Phillips
and Ron D. Stone. New York: McGraw-Hill Publishing, 2002.
In Action: Measuring Intellectual Capital. Patricia P. Phillips, Editor; Jack J. Phillips, Series Editor.
Alexandria: ASTD, 2002.
The Bottomline on ROI. Patricia P. Phillips, Atlanta: CEP Press, 2002.
The Consultant’s Scorecard. Jack J. Phillips, New York: McGraw-Hill Publishing, 2000.
In Action: Performance Analysis and Consulting. Jack J. Phillips, Editor and Series Editor, Alexandria:
ASTD, 1999.
60
Origin/Development/History
The ROI Methodology™ was developed by Dr. Jack J. Phillips in the 1970s, refined through application and use in the 1980s, and
implemented globally during the 1990s.
First impact study – 1973, Measuring the ROI in a Cooperative Education Program, for Lockheed-Martin
First public presentation on the methodology – 1978, ASTD Annual Conference
First book published to include methodology – 1983, Handbook of Training Evaluation and Measurement Methods, Gulf Publishing (this was
the first USA book on training evaluation)
First one-day public workshop –1991, Birmingham, Alabama
First two-day public workshop –1992, Johannesburg, South Africa
First case study book published – 1994, Measuring Return on Investment, ASTD
First international partnership established – 1994, Indonesia
First public certification workshop – 1995, Nashville, Tennessee
ROI Network organized - 1996
First ROI Network Conference –1997, New Orleans, Louisiana
First international ROI Network Conference – 2002, Toronto, Canada
First ROI in Government Conference – 2003, Gulfport, Mississippi, Co-sponsored by the University of Southern Mississippi
First ROI software release – 2003, Knowledge Advisors
Distinguished contribution to workplace learning and performance awarded by ASTD to Jack Phillips for the work on ROI - 2005
On-line ROI Certification launched – 2006, University Alliance-Villanova University
ROI Certification offered as part of Master’s and Ph.D. degree – Capella University, 2006
ROI Methodology adopted by the United Nations for system implementation- 2008
Use
More than 4,000 organizations are using the ROI Methodology, through planned implementation
3,000 organizations have formally implemented the methodology through ROI Certification™ conducted by the ROI Institute
Approximately 5,000 impact studies are conducted annually in learning and development and human resources
At least 300 public sector governmental units are using the methodology
ROI implementation was first pursued in manufacturing, then moved to service, healthcare, non-profits, governments, and is now in
educational systems
Applications
Typical applications include:
Leadership Development
Career Management
Competency Systems
Diversity
E-Learning
Coaching
Consulting
Ethics/Compliance
Meetings and Events
Management Development
Marketing
Organization Development
Orientation
Recruiting/Selection
Knowledge Management
Safety and Health Programs
Communications
Skill-Based/Compensation
Technology Implementation
Quality/Six Sigma
Wellness/Fitness Initiatives
Articles and Publicity
More than 100 articles have been published on the ROI Methodology in major publications in 30 countries
The ROI Methodology has been a cover story on at least 15 publications, magazines, and journals
At least 100 interviews in major global business and professional publications
More than 25 radio and TV interviews in different countries
The ROI Fact Sheet
: 205-678-8101 Fax: 205-678-8102
: info@roiinstitute.net
Copyright©2009, ROI Institute, Inc.
61
Books
30 books have been published on the ROI Methodology and its application (www.roiinstitute.net)
Primary reference – Return on Investment in Training and Performance Improvement Projects, 2nd Edition, Jack J. Phillips, Butterworth-Heinemann,
Woburn, MA, 2003 (originally published in 1997)
Award winning book – Bottomline on ROI, Patricia Pulliam Phillips, CEP Press, 2002 (received ISPI award)
General application – Show Me the Money, Jack J. Phillips and Patricia Pulliam Phillips, Berrett-Koehler, 2007
Most comprehensive work – Measurement and Evaluation Series, Jack J. Phillips and Patricia Pulliam Phillips, Pfieffer, 2008
Case Studies
More than 200 case studies published in books, journals, and industry publications
Four-volume set published by ASTD in 1994, 1997, 2001, and 2005
First public sector case book – 2002, published jointly by the International Personnel and Management Association and the American Society for
Training and Development
First International case book – 2005, Ireland published by Skillnets
International case studies under development in 12 countries
Workshops (One-Day, Two-Day, and Three-Day)
Approximately 200 one-day workshops conducted with more than 8,000 participants
Approximately 500 two-day workshops conducted with more than 15,000 specialists and managers attending (offered in almost every major
international city)
Routine schedules of one-day, two-day, and three-day workshops offered in the USA by ASTD (www.astd.org) and through partners around the world
ROI Certification™
Five-day workshop plus two work products lead to certification for ROI implementation
More than 4,000 professionals have attended certification, representing more than 3,000 organizations in at least 50 countries
Certifications offered routinely about 25 times per year both internally and publicly by the ROI Institute (www.roiinstitute.net)
On-line certification begins every month-six months duration (www.roiinstituteonline.com)
Global Implementation
First implementation of the ROI Methodology outside the USA – 1992, South Africa
First certification in non-English language – 1995, Italy
Implementation is accomplished through partners in various countries
Implementation is currently occurring in 51 countries, with additional implementations planned in other countries
Books published in 33 languages
12 international case study books in development or in the planning stages
The ROI Fact Sheet
: 205-678-8101 Fax: 205-678-8102
: info@roiinstitute.net
Copyright©2009, ROI Institute, Inc.
62
Jack J. Phillips, Ph.D.
Jack J. Phillips is a world-renowned expert on accountability, measurement and evaluation. Phillips
provides consulting services for Fortune 500 companies and major global organizations. The author or
editor of more than fifty books, he conducts workshops and presents at conferences throughout the
world.
Phillips has received several awards for his books and work. On three occasions, Meeting News
named him one of the 25 Most Influential People in the Meetings and Events Industry, based on his
work on ROI. The Society for Human Resource Management presented him an award for one of his
books and honored a Phillips ROI study with its highest award for creativity. The American Society for
Training and Development gave him its highest award, Distinguished Contribution to Workplace
Learning and Development for his work on ROI.
His expertise in measurement and evaluation is based on more than 27 years of corporate experience
in the aerospace, textile, metals, construction materials and banking industries. Dr. Phillips has served
as training and development manager at two Fortune 500 firms, as senior human resource officer at
two firms, as president of a regional bank and as management professor at a major state university.
Dr. Phillips regularly consults with clients in manufacturing, service and government organizations in
52 countries in North and South America, Europe, Africa, Australia and Asia.
Phillips and his wife, Dr. Patti P. Phillips, recently served as authors and series editors for the
Measurement and Evaluation Series published by Pfeiffer (2008), which includes a six-book series on
the ROI Methodology and a companion book of 14 best-practice case studies. Other books recently
authored by Phillips include ROI for Technology Projects: Measuring and Delivering Value
(Butterworth-Heinemann, 2008); Return on Investment in Meetings and Events: Tools and Techniques
to Measure the Success of all Types of Meetings and Events (Butterworth-Heinemann, 2008); Show
Me the Money: How to Determine ROI in People, Projects, and Programs (Berrett-Koehler, 2007); The
Value of Learning (Pfeiffer, 2007); How to Build a Successful Consulting Practice (McGraw-Hill, 2006);
Investing in Your Company’s Human Capital: Strategies to Avoid Spending Too Much or Too Little
(Amacom, 2005); Proving the Value of HR: How and Why to Measure ROI (SHRM, 2005); The
Leadership Scorecard (Elsevier Butterworth-Heinemann, 2004); Managing Talent Retention (Pfeiffer,
2009); Return on Investment in Training and Performance Improvement Programs, 2nd ed. (Elsevier
Butterworth-Heinemann, 2003); The Project Management Scorecard, (Elsevier Butterworth-
Heinemann, 2002); Beyond Learning Objectives (ASTD, 2008); The Human Resources Scorecard:
Measuring the Return on Investment (Elsevier Butterworth-Heinemann, 2001); Measuring for Success
(ASRD, 2010) and The Consultant’s Scorecard (McGraw-Hill, 2000). Phillips served as series editor
for ASTD’s In Action casebook series, an ambitious publishing project featuring 30 titles. He currently
serves as series editor for Elsevier Butterworth-Heinemann’s Improving Human Performance series.
Dr. Phillips has undergraduate degrees in electrical engineering, physics and mathematics; a master’s
degree in Decision Sciences from Georgia State University; and a Ph.D. in Human Resource
Management from the University of Alabama. He has served on the boards of several private
businesses and several nonprofits and associations, including the American Society for Training and
Development and the National Management Association. He is chairman of the ROI Institute, Inc., and
can be reached at (205) 678-8101, or by e-mail at jack@roiinstitute.net.
63
Patti P. Phillips, Ph.D.
Dr. Patti P. Phillips is president and CEO of the ROI Institute, Inc., the leading source of ROI
competency building, implementation support, networking, and research. A renowned expert in
measurement and evaluation, she helps organizations implement the ROI Methodology in
countries around the world, including India, Indonesia, South Africa, Australia, Chile, Brazil,
Romania, Ireland, Canada, and the United States.
Since 1997, following a 13-year career in the electric utility industry, Phillips has embraced the ROI
Methodology by committing herself to ongoing research and practice. Dr. Phillips has implemented
ROI in private sector and public sector organizations. She has conducted ROI impact studies on
programs such as leadership development, sales, new-hire orientation, human performance
improvement, K-12 educator development, and educators’ National Board Certification mentoring.
Her current work includes research and application of the ROI Methodology in workforce
development, community development, and social sector programs as well as corporate initiatives
such as learning and development, human resources, and meetings and events.
Dr. Phillips teaches others to implement the ROI Methodology through the ROI Certification
process, as a facilitator for ASTD’s ROI and Measuring and Evaluating Learning Workshops, and
as visiting professor at The University of Southern Mississippi for graduate-level evaluation
courses. She also serves as faculty for the United Nations System Staff College Evaluation and
Impact Assessment course and their Results-Based Measurement Course. She serves on
numerous doctoral dissertation committees, assisting students as they develop their own research
on measurement, evaluation, and ROI.
Phillips speaks on the topic of ROI and accountability at conferences and symposia in countries
around the world. She is often heard over the internet as she presents the ROI Methodology to a
wide variety of audiences via webcasts.
Dr. Phillips’s academic accomplishments include a Ph.D. in International Development and a
master’s degree in Public and Private Management. She is a certified in ROI evaluation and has
been awarded the designations of Certified Professional in Learning and Performance and
Certified Performance Technologist. She contributes to a variety of journals and has authored a
number of books on the subject of accountability and ROI, including Data Conversion (Pfeiffer,
2008); ROI Fundamentals (Pfeiffer, 2008); Return on Investment in Meetings and Events: Tools
and Techniques to Measure the Success of All Types of Meetings and Events (Elsevier, 2008);
Show Me the Money: How to Determine ROI in People, Projects, and Programs (Berrett-Koehler,
2007); The Value of Learning (Pfeiffer, 2007); Return on Investment Basics (ASTD, 2005); Proving
the Value of HR: How and Why to Measure ROI (SHRM, 2005); Make Training Evaluation Work
(ASTD, 2004); The Bottom Line on ROI (Center for Effective Performance, 2002), which won the
2003 ISPI Award of Excellence; ROI at Work (ASTD, 2005); the ASTD In Action casebooks
Measuring ROI in the Public Sector (2002), Retaining Your Best Employees (2002), and
Measuring Return on Investment Vol. III (2001); the ASTD Infoline series, including Planning and
Using Evaluation Data (2003), Managing Evaluation Shortcuts (2001), and Mastering ROI (1998).
No comments:
Post a Comment