ROI Is Like A Box Of Chocolates

Your significant other surprises you with a gift: a box of chocolates. Opening the box is a pleasurable experience. You anticipate the luscious taste, but looks can be deceiving. The first chocolate you select with the tempting creamy icing actually conceals bitter ingredients. Your mouthwatering anticipation ends in suppressed disappointment.

The ROI of training has a lot in common with a box of chocolates: What you see is not necessarily what you get. This is not a big issue with chocolates, but it is serious business with ROI. Crucial business decisions are often made based on ROI calculations, but below its surface may lie distortions of the truth.

Since ROI is often misused and misapplied, it's important to understand that ROI is not the results. ROI is a single metric that expresses the financial relationship between an expenditure to create and deliver a solution — course, project or change initiative — and the business benefit resulting from that expenditure. ROI simply answers the question: Did the expenditure pay for itself? This is an important metric when properly analyzed and reported, but one that should never be allowed to stand alone as an indicator of success.

Just as the quality of a box of chocolates cannot be determined by examining the box, the quality of performance results cannot be determined by the magnitude of the ROI. The quality of any ROI is determined by executing an ROI quality analysis. The true results are represented by the improvement in one or more key business outcome measures. Effective readiness solutions should enable the performers to be performance ready so they can execute in the work setting and influence the desired business outcomes. Figure 1 illustrates this desirable linked relationship.

In order to calculate ROI, a monetary value must be assigned to a business outcome improvement. The preferred source to determine business outcome and monetary value is the business records of the organization as these often include the status of a specific measure and a known monetary value. When not available from business records, estimates or other methods may be used when properly analyzed and reported by credible sources. Figure 2 illustrates the two channels for monetary value.

The ROI formula appears below along with an evaluation scenario to illustrate how the process works. The evaluation scenario is called the "misguided ROI" because it includes several questionable evaluation decisions — some obvious and some not so obvious. Following the scenario, these decisions are addressed as the ROI quality analysis is applied to determine the quality of the ROI.

Misguided ROI
At Mountain Media Cable & Communications Co., call center representatives are deficient in helping customers troubleshoot their high-speed Internet service. The training department partnered with the call center manager to conduct a needs assessment. Skill deficiencies were identified as the root cause of a significant increase in customer complaints at three call centers. To improve troubleshooting skills and reduce complaints, a six-hour training and performance solution is delivered to 180 representatives at these centers, followed by 30 days of extensive personal coaching and reinforcement from supervisors.

The training staff evaluated the solution to determine the business outcome and ROI. The fully loaded cost of the training is $110,000. Recorded calls and observation by supervisors are used to determine improvement in troubleshooting skills. Two months after solution delivery to two of the centers, the business records from these centers show that customer complaints are reduced by 30 per month. To calculate ROI, a monetary value for customer complaints is needed.

The training staff applied the value from a recent government study, which shows a typical customer complaint costs $595 to process and resolve. Their analysis calculates the improvement benefit as $17,850 monthly ($595 x 30 fewer complaints). Assuming the improvement will continue for at least a year, the annualized monthly improvement is $214,200 ($17,850 x 12 months). The ROI is calculated and rounded up to 95 percent.

Since the break-even point for ROI is zero — recovering the cost — 95 percent seems impressive. Let's melt away the chocolate icing to determine the quality of the ROI by examining the evaluation strategy and analysis more closely. It's not the ROI but the process of arriving at the ROI that should interest our inquiring minds.

The Quality of the ROI: Know What You're Getting
When analyzing results and calculating ROI, no matter how good one is at developing and following strict standards, there's room for inconsistency and error. Since numerous key decisions and variables can be misapplied or even manipulated, the quality of any ROI should be questioned.

While space limits our ability to address all types of misrepresentation and key variables, we will address four factors: causal influence, sustained impact, monetary value and cost of the solution. Each factor includes a brief explanation and addresses how its misapplication results in a significantly overstated ROI in the Mountain Media evaluation.

Causal influence: Consider the factors that influence satisfaction from the gift of chocolates. The recipient gets satisfaction from the entire experience, not just the attractive box and the taste of the chocolates. Other factors influencing satisfaction are the sentiment, the ambience and the special occasion.

Two crucial issues must be considered regarding the causal influence of training solutions.

1. Solution design: If the chocolate analogy was a training solution, all four factors (taste, sentiment, occasion and ambience) would be included in the performance design. When determining the influence of a solution, it's important to view the complete design, not just the single training component — as was done in Mountain Media. Including the training, coaching and reinforcement as one solution is practical and represents the design the partners created to influence performance.

2. Other factors: When a business measure improves, it's possible that factors external to the solution design may influence its improvement. The Mountain Media evaluation completely ignored this possibility. One of three practical methods can be used to determine the causal influence of the solution design relative to other influencing factors:

• Comparison group arrangement — sometimes called control group arrangement.
• Conditional trend analysis.
• Estimates from credible sources.

The Mountain Media solution was delivered during a two-month period for two call centers and during the third month for the remaining call center. This makes it practical to use the third call center, which has similar characteristics, as a comparison to determine the causal influence of the solution design. After two months, organization records revealed a reduction of 30 monthly complaints in the two experimental call centers.

The comparison call center had a monthly reduction of 10 complaints during the same time period, which suggests factors other than the solution caused 10 reductions in the comparison group. Since this is a reliable comparison group, it must be concluded that these external factors were also present in the experimental groups. The ROI quality analysis reveals a difference in reduction of 20 complaints, not 30.

Sustained impact: The gratifying taste of chocolates doesn't last long, but the satisfaction from the entire experience may continue for months. The length of time results are sustained has a tremendous impact on the magnitude of the ROI. In the Mountain Media evaluation, the monthly improvement was automatically annualized. It's faulty analysis to routinely assume that results influenced by solutions will be sustained beyond the initial data collection period. Annualizing is acceptable only when credible sources analyze the situation and conclude that results will be sustained for at least one year. Even when appropriate, due to many uncontrollable variables in a business environment, short-term solutions should rarely be annualized beyond one year. Conservative decisions should be made throughout the evaluation process.

By asking the right questions, our ROI quality analysis reveals the reduction in complaints resulting from the solution would likely be sustained for only nine months, not 12 as was assumed. A key factor in the analysis was a recent decision by operations personnel to provide customers with new technology approaching the last quarter of the year. The equipment includes an enhanced automated troubleshooting feature that internal experts conclude will minimize trouble calls.

Monetary value: Chocolates do not all have the same value. Some are expensive because of the brand, while others are low-cost bargains. Like chocolates, the monetary value of business measures also varies. Values can vary considerably from one organization to another, even for the same measure.

The profit margin from revenue and numerous quality measures have known values that are easily obtained from sources within the organization. Known values are viewed as credible because they are often developed internally and specific to a unique situation. When a measure does not have a known value, a value may be determined by:

1. Linking it to another measure with a known value.

2. Associating it with a credible related study.

3. Using estimations from credible sources.

At Mountain Media, the staff used "value by association" when choosing the $595 value from the government study. Asking the proper questions from credible sources, the ROI quality analysis revealed that customer service management had a specific known value of $400 for customer complaints. Data from the government study is not specific — it is a consolidation of all industry types. The overstated value seriously inflated the ROI.

Cost of the solution: The cost of chocolates and the value they represent involves much more than the purchase price. For example, spending considerable personal time to select the chocolates and ensure the proper ambience for presenting the gift has a higher cost. Just as with chocolates, a training solution usually includes additional costs.

The Mountain Media evaluation included the fully loaded cost of the training, but ignored the cost of supervisors' time for coaching. Although coaching is a normal duty of supervisors, the time beyond these routine duties should be added to the cost. Extensive coaching for a 30-day period increased the cost from $110,000 to $125,000.

Discovering the Truth
Figure 3 includes a summary of the misguided ROI and the corrected ROI as determined by the ROI quality analysis.

After the ROI quality analysis, analyzing only four of the key decision points that influence the magnitude of ROI, you may draw your own conclusions. Know what you're getting. A business or personal decision should never be made based on the ROI calculation alone; yet managers and consultants, among others, do so often. They even benchmark the ROI from one program to another as if this is an objective thing to do. Don't be misled by the number; instead, discover the truth about the ROI.

Ron Drew Stone is president of the Center for Performance and ROI and author of The Real Value of Training: Measuring and Analyzing Business Outcomes and the Quality of ROI, to be published during the first quarter of 2011.

Thanks to Ron Drew Stone / Clomedia / Chief Learning Officer Magazine / Mediatec Publishing Inc.
http://clomedia.com/articles/view/roi-is-like-a-box-of-chocolates/print:1