Why Focus Is the Hardest Part of Data Engineering Now

Quick answer

Connecting tools and storing data cheaply is mostly a solved problem. What companies struggle with now is focus. Every tool produces data about everything, so it's easy to build a lot and use very little. The fix is to start from the business goals, prioritize roughly, pick one North Star, and build only what supports it. The strategy itself stays with your team.

Key takeaways

• Integration and storage cost are rarely the bottleneck anymore. Deciding what to look at is.
• We start every project by asking what the business sells and what it's trying to achieve, before anyone mentions a tool.
• Hundreds of metrics is close to having no metrics. A North Star gives everyone the same thing to look at.
• A data platform built without business focus doesn't get used. People go back to spreadsheets.
• Focus takes experience, which is why "build us a data lake" is a hard brief to hand a junior hire.

The problem moved

A few years ago, the hard parts of data were getting it out of your tools, deciding where to store it, and keeping the bill down. Today most integrations are close to plug and play, and storage is cheap.

So the question changed. Out of everything your tools generate, what's actually worth looking at? Most teams we meet have already tried to answer that on their own and got stuck. That's usually the gap we help close, and honestly, it's a pretty interesting problem to have.

Start with the business, then the stack

When we start with a client, the first questions aren't about tools. What does the business sell? What are the goals this year? Do you want to keep more of the customers you have, or bring in new ones? What is the strategy telling us?

Only after that do we talk about stack. If a warehouse or a data lake isn't built around the business, people won't use it. They'll find their way around it with spreadsheets, which is slower and way more inconsistent. People use what's aligned with the business, because that's what they need to do their jobs.

We prioritize roughly at the start. It doesn't need to be perfect, it needs to be clear. Then we put in front of the team what lines up with their goals, not everything interesting we found along the way. Finding interesting things is easy. Showing all of them is what overwhelms data teams. We've been there ourselves, which is how we know there's a better way. For how this plays out over time, see 5 returns on data infrastructure that compound over time.

Hundreds of metrics is the same as none

Here's a pattern we see a lot. Each department has at least three metrics. Then each person has their own metric to take care of. Before long the company has hundreds, and from a data perspective, that's the same as having none. Nobody knows which number matters, and nobody is sure what each one means.

Picking a North Star helps. One number everyone agrees reflects whether the business is moving the right way. Choosing it is a strategy call. Making it reliable, defined once and calculated once, is data work. The same goes for reports: if you have too much to focus on, you end up focusing on nothing. This is a big part of why your Monday reports never match.

Why "build us a data lake" is a hard brief

Building a data platform from scratch is hard today, and not because of the technology. It's hard because there's so much data and so little clarity on what matters. Telling a junior data engineer "build us a data lake that makes us data-driven" leaves them with no place to start.

Clients also tell us hiring has gotten harder. You open a role and get hundreds of AI-written applications, so you use AI to review them, and you end up with candidates who are great at applying but not necessarily at the job. Working with a team that has done this many times means you skip that, plus the onboarding, because the ways of working already exist. 6 ways to get data engineering done compares the options honestly.

We stop where strategy starts

We know the vocabulary: retention, churn, conversion, click-through rate. We know how to calculate all of it, and that's our job, to make it organized and reachable. What to do with those numbers is your team's call. Your team knows the business best. We make sure they have the right data, in a structure that holds up, so their decisions are actually informed.

This is what we build

We start from your business goals, agree on what matters most, and build the data foundation around that, not around every source you happen to have. You get a short list of numbers your team trusts, instead of a long list nobody opens. See how it works.