Data Engineering Is More Than Building Pipelines: Why Data Modelling Matters
Data engineering is often reduced to building pipelines, but the real challenge lies in understanding and structuring the data behind the business.
Data Engineering Is More Than Pipelines: Why Data Modelling Matters
My career in technology has increasingly been shaped by data. My experience in data engineering has exposed me to the realities of enterprise data, from building pipelines and modernising platforms to working with legacy processes, reporting environments and complex business requirements.
When I started in data engineering, I focused heavily on the engineering itself: extracting data, transforming it according to business rules, loading it into the target environment and ensuring that the process worked reliably.
Over time, I realised that the pipeline was rarely the hardest part.
The harder challenge was understanding the data well enough to know what the pipeline should actually be doing.
The Hardest Lesson I Learned in Data Engineering
A pipeline can be technically correct and still produce an incorrect business result.
It can process millions of records, complete without errors and populate the expected tables while producing information that is misleading or unsuitable for decision-making.
This becomes particularly evident in large organisations where data has accumulated over many years. Data is usually spread across multiple systems, legacy applications, spreadsheets, reporting processes and manually maintained business rules.
During data engineering and migration work, I have encountered business logic embedded in legacy programs, SQL transformations, spreadsheets and reporting processes. Some of this logic was documented, while some existed mainly as institutional knowledge.
This creates a major challenge during modernisation. Simply reproducing an existing process on a new platform may make the migration technically successful, but it can also carry duplicated logic, outdated assumptions and inconsistent definitions into the new environment.
That changed how I viewed data engineering. I realised that the role is not simply about moving data. It is about understanding the structure and meaning of data well enough to build a reliable representation of the business.
The Difference Between Moving Data and Engineering Data
Moving data is about getting information from one location to another. Data engineering goes further.
It requires understanding where the data comes from, what it represents, how it relates to other information, what transformations are required, how historical changes should be handled and how the resulting data will be consumed.
This becomes especially important during platform migrations.
For example, an organisation may have a business definition implemented in a legacy SAS program, another version implemented in SQL and another interpretation implemented in Power BI. Each may have been created for a legitimate reason.
If these processes are migrated without examining the underlying definitions, the new platform can end up with several versions of what should be one business concept.
The technology becomes modern, but the information architecture remains fragmented.
This is why I believe data modernisation should include an examination of the underlying data model.
Why Data Modelling Matters
Data modelling provides the structure required to represent the business accurately.
One of the most important concepts is data grain: what exactly does one record represent?
A sales dataset could contain one record per transaction, invoice line, customer per month, or product and customer combination. These datasets may contain similar information while representing completely different business processes.
If the grain is not understood before joining datasets or calculating measures, incorrect results can easily be produced. A join can multiply records, revenue can become overstated and customer counts can be duplicated.
What makes this dangerous is that there may be no technical error.
The pipeline runs successfully. The database contains valid records. The dashboard loads correctly.
The problem only becomes visible when someone uses the information to make a business decision.
Data modelling is therefore not simply about deciding how tables should be structured. It is about ensuring that the structure accurately represents the business processes behind the data.
Poor Data Models Create Downstream Complexity
When the underlying data model is unclear, complexity moves downstream.
Analysts write their own SQL. BI developers create their own calculations. Data scientists apply their own filters. Business users maintain spreadsheets to reconcile different reports.
Over time, the organisation can end up with multiple definitions of the same metric.
For example, if five dashboards independently calculate a business metric, the organisation is effectively maintaining five versions of that metric.
A well-designed data model provides a foundation for common business concepts to be defined consistently. It reduces the amount of business logic that has to be recreated independently by every data consumer.
As organisations grow, this becomes increasingly important.
Data Modelling in Modern Data Platforms
Cloud platforms have changed how data architectures are implemented, but they have not changed the importance of understanding the data.
Whether an organisation uses a traditional data warehouse, cloud data warehouse, lakehouse architecture or a combination of platforms, it still needs to understand how its data represents the business.
A modern platform can ingest data from hundreds of sources. But if the relationships between those sources are poorly understood, the organisation can simply create a larger and faster version of the same problem.
I have seen this become particularly relevant when migrating workloads from legacy technologies into platforms such as Databricks or Snowflake.
The temptation is often to reproduce the existing technical functionality as quickly as possible.
A better approach is to use the migration to understand the existing architecture, identify the business logic embedded in legacy processes, determine which data is authoritative and redesign the model where necessary.
The goal should not simply be technical equivalence. The goal should be a better data foundation.
Data Modelling and Governance
Data modelling is also closely connected to data governance.
It is difficult to govern data that has not been properly understood.
When an organisation defines a data model, it is also defining important concepts about its business: what entities matter, how they relate, what they mean and how they should be represented.
Consider something as simple as the definition of a customer. If different departments have different definitions, it becomes difficult to establish ownership, measure data quality or determine which system should be considered authoritative.
Governance cannot solve this through policy alone. The underlying data architecture and business definitions must also be addressed.
For this reason, I believe governance should be designed into data engineering rather than treated as something that happens after the platform has been built.
Why Data Modelling Matters for AI
The importance of data modelling becomes even greater as organisations adopt Artificial Intelligence.
Many organisations approach AI primarily from the technology side. They evaluate models, platforms and use cases without first asking whether their data environment can reliably support them.
AI systems depend on the information available to them. Inconsistent customer definitions, fragmented product information, undocumented business rules and unreliable historical data can all create problems when building trustworthy AI solutions.
AI does not eliminate the need for strong data foundations.
In fact, it makes them more important.
For me, AI readiness depends heavily on data readiness, and data readiness requires more than data availability. It requires data that is understood, structured, governed and trusted.
The Changing Role of the Data Engineer
These experiences have changed how I think about the future of data engineering.
Technical skills remain essential. Data engineers need to understand SQL, Python, distributed processing, cloud architecture, orchestration, storage, security and performance.
However, technical capability alone is not enough.
Data engineers increasingly need to understand the business processes behind the data: why the data exists, how it is used, what decisions depend on it and what happens when it is transformed incorrectly.
The strongest data engineers will therefore not only build pipelines. They will be able to understand business processes, identify weaknesses in existing data environments and design better representations of the information.
That requires engineering discipline combined with business understanding.
What This Means for African Organisations
I believe this is particularly important as African organisations accelerate their adoption of modern data platforms and Artificial Intelligence.
There is an opportunity to build stronger data foundations rather than simply reproducing legacy environments on newer technologies.
Modernisation should not be measured only by whether an organisation has moved to the cloud or implemented an AI solution. It should also be measured by whether the organisation has improved how it understands, manages and uses its information.
We need data professionals who can bridge technology, business, governance and strategy. We also need to develop local capability to design and manage our critical information environments.
Data engineering, data governance and AI should therefore not be treated as completely separate disciplines. Together, they contribute to an organisation's ability to make better decisions from information it can trust.
From Data Engineering to Institutional Intelligence
My experience with data engineering has influenced the work I am building through Morero Intelligence Group.
My interest in Artificial Intelligence has grown directly from working with data. I have seen that intelligent systems ultimately depend on the quality of the information environment in which they operate.
For me, institutional intelligence is not simply about introducing AI into an organisation. It is about building the capability to understand information, engineer reliable data foundations, establish appropriate governance and use technology to improve decision-making.
Organisations should not only ask whether they have data. They should ask whether they understand it.
They should not only ask whether they have an AI strategy. They should ask whether their data and governance foundations can support it.
And they should not only ask whether their platform is modern. They should ask whether it has improved their ability to make reliable decisions.
Conclusion
Data engineering will continue to evolve as technologies change. Platforms will become more distributed and intelligent, while AI will become increasingly integrated into business processes.
But one challenge will remain: organisations need to understand and trust the information on which their decisions depend.
That is why data modelling matters.
It provides structure to complex information, establishes consistent definitions, makes relationships explicit, supports governance and creates a stronger foundation for analytics and AI.
My experience has taught me that building a pipeline is only one part of data engineering. The greater responsibility is understanding what the data represents and engineering a platform that preserves that meaning reliably at scale.
That is where data engineering becomes more than moving data. It becomes the engineering of an organisation's information foundation.
Mpheya Partlet Mphateng
Founder & Chief Executive Officer
Morero Intelligence Group