Avoid faulty "insights."
Time to Act

February 2024, Vol. 13 No. 2

charlie goodrich


Every well-run company relies on data to operate the business. But using the wrong data or looking at it in the wrong way can lead to faulty “insights.”

This month, I take a look at how data should (and should not) be used by businesses when attempting to draw conclusions and make decisions.

As always, please reply with your thoughts and comments.

Charlie Goodrich
Founder and Principal
Goodrich & Associates
In this issue…

The Critical Importance of Accurate Data Use and Analysis

Heard on the Street

About Us

The Critical Importance of Accurate Data Use and Analysis

During the 2000s, the United States experienced an unprecedented decline in manufacturing employment. Roughly six million jobs were lost.

Most economists looked at the data and determined that all these jobs disappeared due to productivity improvements. But, when Susan Houseman , an economist with the Upjohn Institute for Employment Research looked into that same data, she came to a very different conclusion.

Houseman found that there had been little productivity improvement in the manufacturing sector during that time. Rather, nearly all the improvement in productivity came from the electronics sector, a relatively small part of the US economy. Further, all the productivity in the electronics sector was driven by product design or R&D, not by automation (i.e., making things faster and cheaper). Besides, most electronics manufacturing had moved to Asia by then anyway.

Businesses use data every day to draw conclusions and make decisions. But as this example highlights, using the wrong data or looking at it in the wrong way can lead to faulty “insights.”

With that in mind, and while there are all sorts of data, today we focus on numerical data of things that have already happened — not nonnumerical data (e.g., documents) or data projections regarding what may or may not happen in the future.

Disaggregate the Data

The problem with the initial analysis in the example above was that while the total indicated one thing, once the data was broken down into its components a different picture emerged. 

This type of disaggregation is also important when analyzing financial data. For example, if a company is profitable on an annual basis but only makes money a few months of the year, that is very different than a business that makes money every month. 

Similarly, while a company may look profitable in the aggregate, it may be that it is only profitable with some customers or some products.

Understand How the Data is Put Together

When looking at customer or product profitability, it is important to understand how revenue and costs are divvied up between products or customers. 

In the case of one of my manufacturing clients, its measures of customer-specific product profitably showed that nearly all customer pricing for nearly all products was under water — unprofitable. Yet the company was profitable overall. The data problem? One commodity raw material representing 50% of revenue in aggregate was measured in a horribly inaccurate way. Other key costs were allocated based on “run time” and measured erroneously as well.

It is also important to understand how government statistics and other data external to the company are put together. For example, many statistics are seasonally adjusted, a correction that can mask what is really happening. 

Surveys in particular can be misleading. An academic study made headlines by claiming that 3.6 million American households live on two dollars a day or less. However, an analysis by Bruce Meyer at the University of Chicago Harris School of Public Policy discovered that claim came from looking at a Census Bureau survey that collects data by interviewing households. When other anonymized government data was compared to the survey, it found that 90% of those whom the study said lived on less than two dollars per day were far better off. 

Finally, it is important to understand measurement accuracy. Even small errors can quickly compound when combined with other inaccurate data. Error multiplied by error, etc., always yields a number that increases exponentially.

Sample Size Matters

There’s a reason lenders don’t just look at one year of profitability — what if the prior four years were not as profitable? They know that a sample size of one is insufficient to make a lending decision. 

The manufacturing client mentioned earlier upgraded to a metric based on running a bag of raw materials through the manufacturing process a couple of times a year and then counting how many bags of finished product resulted. This sample size was way too small to be helpful. 

There are also mathematical problems with small sample sizes. Statistically, for any result to cross the 95% threshold and be significant, the finding itself must also be significant. For example, a 60+% difference may be required and numbers like that grab our attention. But the finding may not be real; it’s just a mathematical necessity given the small size of the sample used. 

Use the Proper Data Set

Speaking of statistical significance, you may (vaguely) remember this from statistics class: "It" is statistically significant if there is a greater than 95% chance that a finding is not random or the result of just noise.

Well, not quite. What the test really says is that given the data sample examined, there is a 95% chance that a finding is not random or the result of just noise. 

Data is often chosen because it is easy to get. Or, if we are trying to test a particular theory, we look for data that we think will apply. But the data set may not be sufficiently representative of the world at large. 

Other times, we look at — and take action based upon — a given study or analysis that was statistically significant… but we ignore similar studies that found nothing.

A Few Tips

Does the data look reasonable? Apply common sense. Pay particular attention and dig in if the data is accompanied by an attention-grabbing headline. 

Put controls on data collection in place. For financial data, there is usually a system of controls in place to make sure it is accurate. Do the same for other important data that you routinely collect.

Invest in better data when that data is important. The manufacturing client mentioned is investing in computerized scales and other computerized data collection tools for accuracy and continuous measurement. They are coupling this investment in equipment with an ERP system to process and house the data.

Beware of confirmation bias. That is, focusing on data that supports an existing point of view while subconsciously ignoring data that doesn’t. 


Every well-run company relies on data to operate the business. Keep in mind, however, that information is only as good as the assumptions, techniques, and circumstances through which it is generated.

When using data, always understand how it was obtained and calculated.

Please share with your colleagues:
Facebook Share This Email
Twitter Share This Email
LinkedIn Share This Email
Heard on the Street

For a short, fun synopsis of the economic history of the NFL, the Super Bowl, and now the Taylor Swift effect, read this short post by Doug French, President Emeritus of the Mises Institute.

About Us

Goodrich & Associates is a management consulting firm. We specialize in restructuring and insolvency problems. Our Founder and Principal, Charlie Goodrich, holds an MBA in Finance from the University of Chicago and a Bachelor's Degree in Economics from the University of Virginia, and has over 30 years' experience in this area.

To ensure you continue to receive emails from us, please add

[email protected] to your address book today.

Goodrich & Associates respects your privacy.

We do not sell, rent, or share your information with anybody.

Copyright © 2024 Goodrich & Associates LLC. All rights reserved.

For more on Goodrich & Associates and the services we offer, click here .

Newsletter developed by Blue Penguin Development

Goodrich & Associates