Data Quality Management and Profiling
Find where trust breaks across the data lifecycle.
Profile, validate, reconcile and improve the quality controls between source systems and business reporting.
Discuss this requirement →Start with the environment, risk and required outcome.
Profile, validate, reconcile and improve the quality controls between source systems and business reporting. Namandla Consultants begins with evidence and recommends a proportionate engagement rather than a generic technology package.
Connected expertise for the wider environment.
Clear steps. Visible decisions.
The six dimensions worth measuring
Data quality is vague until it is broken into things you can test.
Completeness
Are the fields that should be populated actually populated. A customer record with no region cannot be reported on by region and it silently disappears from the analysis.
Accuracy
Does the value reflect reality. The hardest to test automatically, because it needs a reference to compare against.
Consistency
Does the same fact agree across systems. Where the CRM and the ERP disagree on a customer’s status, one of them is wrong and reporting from either is a coin toss.
Validity
Does the value conform to its rules. A date in the future for a delivery that already happened. A VAT number with the wrong number of digits. Cheap to test and frequently untested.
Uniqueness
Are there duplicates. Two customer records for the same customer split their revenue in half and both halves look unremarkable.
Timeliness
Is it current enough for the decision. Data that is correct and three days stale is wrong for an operational decision and fine for a monthly review.
Measuring these gives you a number to move. “Data quality is poor” produces no action. “Eleven percent of customer records have no region, up from four percent in March” produces a conversation with a specific owner.
Data profiling
Profiling is looking at what the data actually contains rather than what the schema says it should.
Column statistics
Distinct values, null rate, minimum and maximum, distribution, length. This alone finds a great deal: a field with two hundred distinct values that was meant to hold five, a date column with entries in 1900 where a default was applied, a numeric field stored as text.
Pattern analysis
Do values match their expected shape. Identity numbers, VAT numbers, phone numbers, postal codes and email addresses all have testable formats and all of them contain surprises.
Relationship checks
Orphaned records, broken references, cardinality that does not match the documented model.
Cross-system comparison
The same entity in two systems, compared. This is where the awkward findings sit and it is the profiling most often skipped.
Profile before designing anything
A warehouse or a report designed against the schema rather than the data hits every one of these surprises during build, and each one becomes a change request. Two days of profiling at the start removes most of that.
Rules, monitoring and where to catch problems
In the pipeline, not in the report
A quality rule that runs during the load and quarantines or flags failures catches problems before anyone acts on them. A quality problem discovered in a board pack has already cost credibility.
Thresholds rather than absolutes
A rule that fails the load on any single bad record stops everything for one malformed row. A rule that flags when the failure rate exceeds a threshold, or when it moves sharply against the historical norm, is more useful and less disruptive.
Trend, not just state
A field at ninety-four percent complete is a number. A field that has moved from ninety-nine to ninety-four percent over six weeks is a signal that something changed upstream, and that is the more actionable finding.
Quarantine with a correction path
Failed records go somewhere visible, with enough detail to correct them, and there is a defined person and process for doing so. A quarantine table nobody looks at is a slow deletion.
Report quality alongside the data
A dashboard that shows completeness and freshness next to the numbers builds more confidence than one that presents figures without qualification. Being visible about known gaps is what makes the rest credible.
Ownership and data governance
The part that is not technical and decides whether any of it lasts.
A data owner per subject area, in the business rather than in IT. The person accountable for customer data being right is the person who runs the customer relationship, not the person who runs the database.
Definitions written down and agreed
Most data quality disputes turn out to be definition disputes. Two departments are both correct under their own definition and neither has written it down.
A correction process
Who fixes a bad record, in which system, within what time. Without this, quarantine queues grow and quality reports become something people stop opening.
Fix at source where possible
Cleansing in the pipeline treats the symptom and it has to run forever. Correcting the capture process, adding validation at entry or fixing the integration that introduced the problem addresses the cause. Both have a place and only one of them ends.
This is data governance in its useful sense: owners, definitions and a correction process. It does not require a framework, a committee structure or a policy document nobody reads.
What Namandla delivers
Data profiling across source systems. Quality dimension measurement with a baseline. Rule design and implementation in the pipeline. Quarantine and correction workflow. Quality monitoring and trend reporting. Owner and definition workshops. Root cause work on the capture processes that generate the problems. Ongoing monitoring under managed data services.
Remote delivery nationally. Onsite in Gauteng for the ownership and definition workshops.
Common questions about Data Quality
How do we know if our data quality is bad?
Measure it. Completeness, accuracy, consistency, validity, uniqueness and timeliness, expressed as numbers against the fields that matter rather than as a general impression.
The practical signal that it is worth doing is people not trusting reports, or reconciling by hand before they believe a number. Both mean the quality issue is already costing time, it is just being paid in effort rather than being visible as a problem.
What is data profiling?
Profiling examines what data actually contains rather than what the schema says it should: distinct values, null rates, ranges, distributions, patterns and relationship integrity.
It is the first step in any data project and it is worth two days on its own. It reliably finds fields holding values nobody expected, default dates that were never replaced, numerics stored as text and relationships that do not hold. Designing a warehouse without profiling means finding all of that during build, one change request at a time.
Where should data quality checks run?
In the pipeline, during the load, rather than in the reporting layer. A problem caught during the load can be quarantined and corrected. A problem discovered in a board pack has already cost credibility.
Use thresholds rather than absolute rules, so a single malformed record does not stop the whole load, and alert on movement against the historical norm as well as on breaching a fixed level. A field dropping from ninety-nine to ninety-four percent complete is more informative than the absolute number.
Should we cleanse the data or fix the source?
Both, in that order of urgency and the reverse order of priority. Cleansing in the pipeline gets your reporting usable now. Fixing the source stops the problem being generated.
Cleansing alone means running that correction forever and it hides the underlying cause. The source fix is usually a capture process change, a validation rule at entry or correcting an integration that introduced the defect. That work is slower and it is the one that ends the problem.
Who should own data quality?
A named owner per subject area, in the business. The person accountable for customer data being correct is whoever runs the customer relationship, not whoever runs the database.
This is the single strongest predictor of whether a data quality effort lasts. IT can measure, report and correct in the pipeline. It cannot decide what a customer status means or make a sales team capture a field properly, and treating quality as an IT problem is why most quality initiatives fade after a year.
Our two systems disagree. Which one is right?
That is a system-of-record decision rather than a technical question. One system is authoritative for each entity and the others receive.
Making that decision explicitly, per entity, is the foundation for both integration and quality work. Covered on data integration. Until it is made, every reconciliation is an argument, and reporting from either side is defensible and unreliable.
When is a data quality project more than we need?
Where you have one system, its data is captured by people who use it daily and nobody is disputing the numbers. Data quality problems concentrate where data crosses systems or where the person capturing it does not use it.
That second point is the useful predictor. A field that is captured by one team and only used by another is the field that will be wrong, every time. If your data does not have that shape, spend the money elsewhere.