Savvy Decision-Makers Question Data - Part 1
Ever nod at a report you didn’t fully understand, but felt uncomfortable asking questions? You are not alone and this post is for you.
We use information every day to run our work - manage programs, allocate resources, pitch new ideas, and track progress. But how often do we pause and ask: Do I really understand the data in front of me? And is it the right data?
While we don’t need to know everything, as managers or decision-makers, we are accountable for understanding the data we act on.
This two-part series offers some easy-to-adopt habits to help avoid painful, common mistakes.
This first post covers where to start and common pitfalls. The goal is not to become a data analyst. It’s to feel confident asking the right questions so you can make better decisions.
Habit #1: What is the Data?
This sounds obvious, but it’s where many misunderstandings begin. We all assume we understand familiar metrics—until someone asks us to explain them. That’s when we realize: “I’m not actually sure what this number includes.” Let’s look at a few common examples:
Similar but different terms: Sales and revenue are often used interchangeably, but they are not the same and the data can show up differently. Let’s say you sell a $1200 annual subscription:
One word, many definitions: Web traffic could mean page views, sessions, or users. And if it’s users, how are repeat visits from one user counted? Metrics like “traffic” sound intuitive but can have different meanings.
Conversion rates: A 10% conversion from what to what? A conversion from a marketing email to a click is not the same as a conversion from a shopping cart to a purchase.
Vague classification criteria: A report measures "Adverse Reactions" for a new drug. What does that mean? A mild rash? Hospitalizations? Death? Classifications can vary by industry or even by department.
Pro Tip: Recurring reports sometimes change behind the scenes. Methodologies shift, systems are updated, or someone new pulls the data and forgets a step. If you see data shift without a good explanation, start looking here.
Habit #2: What is the Source?
Where did this data come from? Context shapes how you interpret the data and how much trust you put into its accuracy and reliability.
For external data sources, ask:
Who published the data? Is it from an established, credible source, like census data or a reputable research firm? These organizations specialize in data and have a track record of reliability. In contrast, vendors or lobbyists may provide research, but they also have incentives that could bias results.
Where and how do they get their data? Where did they get their data? What is the sample size? Did they do the research and analysis, or did someone else?
Is the methodology available? Transparency is key. Credible sources will usually explain how they collect their data and what limitations apply.
Internal data needs context, too. Ask:
What is the data source? Some systems cover only part of the business when you want a view of the total business. Other systems may be notorious for unclean or "bad" data when you need greater accuracy.
Is it a recurring report or a one-off request? Ad-hoc reports require more scrutiny for quality and assumptions. Regular reports typically have been improved over time.
Consider who built the report. Different departments bring different lenses: Finance may be more conservative than Sales. Marketing may count ‘customers’ as anyone who has ever engaged whereas Operations may focus only on recent ‘users.’
Each team has different data access, expertise, and perspectives. These factors influence what gets measured and how it gets presented.
Transparency is Key
Credible sources will explain how they collect their data and what assumptions or limitations apply. Data-savvy professionals ask these basic questions because they have learned (generally the hard way) how easy it is to make a false assumption that leads to faulty decision-making. If someone dodges your questions, be cautious. No data is perfect, but solid data can withstand scrutiny.
Go Deeper with 3 More Habits
In Part 2, we walk through how to:
Spot baked-in assumptions
Assess whether the data is reliable enough
Evaluate for bias
Remember:
You are accountable for understanding the data you act on.
You don’t need to know everything. But you do need to ask.
You’ve got this.