Continuing the blogging ROI discussion…..there was an interesting interchange yesterday between my self and Jane Bozarth on twitter about the topic of ROI.
The scenario
Jane and I talked about proving the ROI of a mythical (I think) training course called “Ethics for Firefighters”. She said to prove the ROI of such a course that first the dollar value of the public perception of ethical behavior needed to be determined. I disagreed with this, asking if the real measurement was public perception, a decrease in legal actions (both the action and the cost of defending/prosecuting the action)?
Then I think Jane thought that meant I was turning into a cold corporate numbers person because she asked “So ROI for ethical behavior of public servants is based entirely on legal costs?” And that isn’t where I was going at all.
This exchange made me think that maybe there are translation problems between the terms used in Performance Improvement and the terms used by MBAs.
Lost in Translation
I’m not an MBA. But most of the executives I know are. They use MBA terms to manage their people and environments. In one more semester I’ll have an MS in Instructional Systems. I have been trained to look at organizations as systems, and to troubleshoot what is causing performance problems in these systems. I have been taught a completely different set of terms than the terms the MBAs I work for use and understand.
This could be a problem.
MBAs love to talk about ROI. ROI literally means “return on investment”. Now, just like there are good IDs and bad IDs, there are good MBAs and bad MBAs. The good MBAs also see the organization as a system. The bad MBAs only care about quarter-to-quarter hard numbers. Unfortunately, we need to translate performance terms for good and bad MBAs.
Let’s define the term. If an MBA wants to correct me, please feel free! I got this definition from the business dictionary.com:
Earning power of assets measured as the ratio of the net income (profit less depreciation) to the average capital employed (or equity capital) in a firm or project. Expressed usually as a percentage, it is a measure of the profitability which (while not taking the time value of money into account) indicates whether or not a firm is using its resources in an efficient manner. For example, if the ROI of a firm (in the long run) is lower than its cost-of-capital then the firm will be better off by liquidating its assets and depositing the proceeds in a bank. Also called rate of return, or yield.
Sigh. That is why I didn’t get my MBA, that definition just seems so void of human concern. My gut feeling is this is where the translation issues start to happen. So what would be comparable from a systems (or human performance technology) point of view?
The first letter of ADDIE stands for Analysis. In Pershing’s Handbook of Human Performance Technology, there is a chapter written by Dr. Allison Rossett titled Analysis and More. Rosset says “analysis is the process used to figure out what to do”. She provides a this view of what analysis is:
Organizations provide the results of their learning analytics for many reasons. The most popular reason is to showcase the training’s value to the organization. Another common reason is to indicate the quality of the training services provided. Additional reasons are because stakeholders request it or to justify large expenditures.
The way I see it the analysis step, the “A” that is the key repeatable step in every instructional design theory, is central to the MBA concept of ROI. We’re all trying to get at the same thing. MBAs want to prove their organization is using their assets in the most efficient way possible. The problem is that more and more in this age of information, those “assets” are human.
This is where we as performance and learning specialists can come to the aid of our MBA friends. Proving that your human assets, the information workers that hold the means of production inside their heads, are being utilized the best way possible is very different than proving your datacenter or factory equipment is optimally configured.
A potential blogging ROI example
Here’s a real example of how hard this is. Our guys out in the field who install and configure EMC products for customers are very very busy. They usually have much more to do than there are hours in a day. And since they are not machines, there is just no way to configure them to miss sleep and work 24 hours a day so that they can catch up. 🙂
I’m going to concentrate specifically on blogging in this example. Our field guys are super smart, and are very quick learners. They have to be, or they would not be able to do the jobs put in front of them.
Now I’ve blogged before about blogging being an excellent way to reflect on new concepts. If I think what our field guys do, I would imagine that if they were able to blog about their activities they could reflect on lessons learned, what would have been helpful to know going into an engagement, advice they would give to colleagues in case they faced the same situations, etc. These sort of blog posts would be useful for other field guys, support guys, engineers, training folks, etc. The blogs may also give management a better idea about how much time certain types of engagements take.
The ROI question is which is better for the company: book these guys out for 100% of their time in customer sites to plow through the workload, or give them some time to blog about what they have been through in an engagement, providing them reflection time and providing insight into the experience for other information workers in the company?
Lets say I proposed this, and everyone bought into the plan. How would I prove hard dollar numbers that the time given to these field folks to blog benefits the company? I think this would take planning and analysis throughout the project lifecycle, here are some guesses on what could be measured:
- Measure time it takes to blog
- Measure traffic to particular blog posts
- Measure time it takes for individual blogging to perform similar tasks after the blog was written (after time for reflection)
- Observe case deflection in the call center (number of calls on topic before and after post is written)
- Measure engineering enhancements made because of post
- Measure training enhancements made because of post
- Measure impact on planning (actual time it takes to do the things talked about in the post vs what was originally planned, utilization rate improvements based on adjustments)
I would think creating a community of support larger than field support colleagues would also have some sort of impact, not sure on how to begin to measure that though.
In conclusion….
I think we have a translation problem between MBAs and ID/performance support folks. ROI is just the end result of analysis. The wrong measurement will always give an incorrect ROI, which is not good for business (is there an MBA term for that?). As education folks, we understand what interventions motivate and support performance for “human assets”. We need to figure out how to communicate this to our MBA colleagues.
I think the problem for the MBAs is getting numbers to prove the ROI of the work information workers are now expected to perform. Those numbers don’t exist in many cases because the analysis is not typically done on the human impact of information work. Its an infinite loop: we don’t do analysis on that sort of thing because we don’t do it, we don’t do it because we don’t have hard numbers, and since we don’t have hard numberswe can’t prove ROI.
What do you think? Am I totally off-base here? Is it wrong to want to collaborate with MBAs to prove ROI?
We need to coordinate with bean-counters if they’re asking us to count beans. To me there’s nothing controversial there.
My big fear with this kind of discussion is that it’s probably impossible to actually measure the things in question in any way which could yield statistical information about the impact of blogging. Too much changes week to week, never mind quarter to quarter or year to year.
I think you could survey people involved in all your bullet items, and ask them how helpful they _thought_ the blog posts were … but I don’t think anyone could actually measure it scientifically. And in some cases bad data is worse than no data. This just screams “bad data” to me.
Taking another approach, what’s the ROI of a company making luxury box seats available to high-profile customers? Is there any way to measure whether those people would have bought more/less with or without the boxes? Or is the assumption that it’s impossible to compete for those customers without some minimal investment, and that luxury box seats are part of that investment?
I think the argument we need to make is that the minimal investment for doing certain kinds business in this day and age involves a social media component. I’m not sure what kinds of business that is, of course.
Great post! I am a long-time ID and educator, now a doc student in Training & Performance Improvement. I’ve already taken the ROI class in my program. It’s based on Phillips’ model. So I am biased.
I totally think there are ways to communicate with the MBAs at work about the ROI of a class. For one thing, a full ROI includes intangible benefits…whether or not the MBA likes to think about the humans. (Side note: I say run if the MBA doesn’t.)
My example of ROI of a training class is somewhat incomplete. See, we have the Hardware Boot Camp for new hire techs…we have a great revised curriculum…metrics to measure all 5 levels from Phillips (Kirkpatrick’s 4 + ROI)…and then the economy tanked and only 2 guys have been hired in more than a year. So the metrics are weak. At any rate, we have all levels built-in…pre & post survey rating skill, written test and demonstration of skills, an in-field mentor to monitor application on the job, and metrics for ROI. Those are (in simple terms) customer satisfaction survey results, time to solo, number of calls to senior tech, number of do-overs. Not everything is monetized, but at my company customer satisfaction survey results RULE. Therefore I don’t have to monetize. We do have a fully loaded hourly salary cost so I can easily compute the cost of the time spent by a senior tech re-mediating a new hire. We also have the intangible benefits to measure like confidence, ability to problem solve, employee satisfaction based on successful learning.
Someday, the economy will come back, we will hire again, and then I’ll really get to see this in full action.
I’m liking this post! As an L&D person who is getting their Masters in the field and who was recently laid-off, I’m a big believer in ROI. Imagine the conversation my boss could have had if I had been doing full ROIs on my major courses all along, “Tricia’s courses have an ROI of 200% and a BCR of 3:1…as you know that means she’s responsible for bringing in $400,000 to this company through the courses that she’s measured alone. She’s a valued asset, and one of our largest revenue producers!” Instead, I think it went something like this, “Tricia’s salary is XXXX. People like her courses, but we have base-line new hire stuff and I think we can hopefully squeak through without an L&D person.”
In other words, I think ROI on courses that are tied to measurable business objectives can help move L&D from a cost center to a profit center. And that is language any MBA can understand!
As far as your example of the blogging techs, don’t forget to include Customer Satisfaction. You can convert this to money…even if your company doesn’t have the research, I’ve seen tons of it out on the internet. Survey b4 the initiative and again about 6-12 months after. Convert the change to dollars and prove its a good thing from both internal and external sides. You can also analyze ramp-up time of new hires – if they can work independently quicker, convert that to money too. Analyze the turnover rates and see if they go down after the initiave. With a lot of research, you may be able to isolate the blogging as the cause. Lower turnover means HUGE savings to the bean-counters!
Finally, I completely agree with you about the “language” barriers we face with MBAs. I feel, however, that we are the ones that will probably need to move closer towards MBA-ese. After all, that is the language that is spoken in the boardrooms, and ideally that is where I’d like to see this profession end up!
Sigh. Will the myth of ROI ever go away? Where to begin working through the flawed thinking here? First is the typical behavior of picking an item, any item, that can be tied to a dollar amount, relevant or not. Nothing in the ethics training scenario indicated there were concerns with legal costs. There do appear, however, to be issues with public perception, which because it is not associated with a dollar figure has been disregarded for an irrelevant, equally nebulous measure.
Then there is the tendency of the academic amateur to confuse causation with correlation. If there had been concerns with legal costs, then any number of interventions and outcomes could account for a decrease, such as settlement of legal cases, delays in settlements, removal of offending parties, and post-scandal heightened awareness of and attention to ethical concerns. The wheels of justice grind exceedingly slow but exceedingly fine, making it hard to isolate costs from one fiscal year to another. Training may or may not play a role in any of this: A decrease in legal costs could be brought about through unethical behavior, such as behind-the-scenes retaliation for whistleblowing,hiding evidence, or buddying up to the police force for support in discouragingvictims from testifying, filing charges, or bringing issues to the public attention. Steak dinners for the press might help as well. Overall, the “legal cost” measure is just another imaginary number. But it sure looks good in justifying one’s job!
Identifying an artificial measure and pretending it is legitimate does not “prove” ROI. Fine, if you can show management some decrease in legal costs and tap dance your way into showing a link to the training, and more power to you if you can pull it off. Management gets what it deserves in that case. Kudos to the MBAs for pulling off this shell game and selling it to those in the training/instruction/performance improvement business.
Pingback: Please vote for our #E20 submission on convincing execs to use social media | Adventures in Corporate Education
Pingback: Transform or Die | On the dangers of a personal brand