志愿者以声为眼 带盲孩子感触春天

May 2024 issue

I grew up in Los Angeles, with MGM Studios (now Sony Picture Studios) at one end of my street and 20th Century Fox at the other end. My grandfather worked in the movie industry, and I love watching Old Hollywood movies. Movies, of course, are big business and headlines are made whenever movies earn huge amounts of money. At this moment, the highest earning movie ever made (without adjusting for inflation) is the original Avatar from 2009, which has earned nearly 3 billion dollars. Looking at that outcome alone, Avatar has been massively successful. But, as you can guess, there’s more to understanding things than simply looking at impressive outcomes. Avatar was also one of the most expensive films ever made in terms of production – as high as $310 million – and marketing – as high as $150 million – for a combined cost as high as $460 million dollars. Those numbers can be combined with the earnings to get the “return-on-investment,” or ROI.

ROI is not a familiar topic for many researchers, who have traditionally focused on "outcomes" research, or evaluating the effects of some intervention or program without necessarily looking at what it took to get to that outcome. In the organizational world, however, no analysis is complete without an explicit consideration of ROI. Fortunately, ROI is, at least in theory, relatively easy to calculate. I'll walk through a few examples and explain how the results can be used in getting actionable insights from data.

Calculating ROI

ROI is typically calculated by first getting the returns – in this case, the profits, or total earnings minus production and marketing costs – and dividing them the investment – again, the production and marketing costs – as shown in this equation:

Let's start by calculating ROI for Avatar.

Here's how that formula can be used to calculate the ROI for Avatar:

That means that for every dollar invested in the production and marketing of Avatar, $5.52 of profit resulted. As such, Avatar seems like a great investment – and it is, given that the majority of movies either earn little money or actually lose money. (According to a 2019 article by Schuyler Moore in Forbes, 80% of movies lose money. Ouch.)

But there’s another way to put this number into perspective. In 1992, Robert Rodriguez created the film El Mariachi. In the 30 years since its release, it has earned 2 million dollars, which is literally within the rounding error for Avatar’s earnings. However, El Mariachi is the least expensive film ever to make over a million dollars: its original, Spanish-language release cost only $7,000 to make. (That’s seven thousand dollars, not seven hundred thousand dollars or seven million dollars. You can spend more than that on a bicycle; sometimes a lot more.)

Here’s the ROI equation for El Mariachi:

In this case, each dollar invested in the production of El Mariachi resulted in $284.71 of profits. That’s over 50 times more than Avatar’s ROI of $5.52.?

So, if you’re looking for excellent returns on investment, it’s not enough to know what the outcome or return was, it’s critical to know what the investment or cost was as well.

Generalizing ROI

The same principle applies to non-financial examples, as well. Before an organization implements a new program for its employees, with the idea that the program will “improve” some outcome of concern, it’s important to look at the investment required to implement the program, as some programs will have a better ROI, and some will even have a negative ROI if their costs outweigh their benefits. In this case, the investment or costs can include:

• Money for materials, training, and software
• Time for training and for any extra work that the program requires
• Productivity lost to mental switching and concerns about evaluation
• Productivity lost to new, unanticipated difficulties with the program
• Related decreases in morale and increases in absenteeism and turnover

While it may be possible to assign a dollar value to each of these outcomes, that’s not always necessary. (It may be strange to hear a data evangelist say that not everything needs to be turned into numbers, but I say that because I am a pragmatist. Like you, I have other things to do with my time, so the “eyeball” approach will often be sufficient.) It may be sufficient to simply list the potential returns or benefits in the 1st column on a whiteboard, then list the potential investment or costs in the 2nd column, and compare them informally. This is useful because, even if there is no direct financial cost, per se, the investment can still include the other elements of lost time, productivity, and so on. This is not to say that new programs should never be implemented; rather, my point is that full costs should be included to allow a rough estimation of holistic return on investment.

Include variability and probability in ROI estimates

One of the great challenges of calculating return-on-investment is that, in data as in life, nothing is certain and everything is variable. As such, there is always some uncertainty in estimates like ROI, and it is always a good idea to be clear about that uncertainty when making recommendations.

One way to do this is to create a short list of possible outcomes and provide rough estimates for the probability of each outcome, with the requirement that the probabilities sum to 100%. (If you have access to extensive datasets, such as those found in e-commerce, insurance databases, and educational records, then you can make much more precise statements about possible outcomes and possibilities. However, most people don’t have access to that kind of data, so I’m using an informal example here.) For example, a small business that is considering several new products to sell might make a list like the one shown in the table below. Please note that the profits and probabilities are fictional.

It’s possible to combine all these numbers and come up with a single expected profit or “utility,” by multiplying and summing, like this:

Generalizing expected outcomes

This process can be done with non-financial values, too. For example, a therapist might rate a client’s symptom on a -10 (worst case) to +10 (best case) scale. If they were considering a new approach for working with clients, they could make a table similar to the one below.

As with the financial example, if desired, these scores can be combined into a single expected outcome score by multiplying and summing:

What this table shows is an expected improvement (or utility) of 2.1 points on the symptom severity scale for the proposed treatment method. This expected value can then be compared with other possible treatments, which, when combined with ROI estimates, can be used to inform decisions.

[As a note, while it is possible to calculate a single value to summarize the data in the tables, it’s not always necessary. In particular, because people vary in their comfort with gains and losses, it may be better to present the overall table and have a discussion about possible choices. See the Wikipedia entries on “Expected utility hypothesis,” “Prospect theory,” and “Loss aversion.”]

What does it all mean?

Data can help guide decisions and lead to actionable insights, but it's important to know more than that treatment X has a higher average outcome than treatment Y. The tables of expected outcomes can paint a more nuanced picture of the likely outcomes, but no decision is complete without a consideration of the investment (in terms of money, time, effort, etc.) required to reach those outcomes. Having that more complete picture can give you, your clients, and your stakeholders greater confidence in their decisions, which is one of the most important things that your data work can accomplish.