SunX Research Data, A Division of SunX LLC
Data Validation

Imagine a report showing that sales in one market rose sharply. Leaders are pleased, and plans are adjusted. Weeks later, someone discovers that a data feed was loaded twice, and the rise was never real. The decisions had been made on a number that was simply wrong.
Mistakes like this are rarely dramatic. A duplicated file, a changed column order, a currency applied twice, or a missing month can slip through unnoticed and produce results that look perfectly reasonable. The risk grows when data comes from many countries, systems and teams, each with its own habits.
Data validation is the discipline of checking data before relying on it. It is the least glamorous stage of analytics, and it is the one that decides whether everything else can be believed. This guide explains how we approach it.
Validation means testing whether data is complete, accurate, consistent and suitable for the purpose you have in mind. It is different from analysis, which asks what the data shows, and from cleaning, which corrects problems once they are found. Validation is how you find them.
Think of it as quality control. You would not ship a product without inspecting it, and the same standard should apply to data.
Good validation looks at several aspects of quality.
Completeness. Is everything that should be there present? Are there missing records, blank fields or absent periods?
Accuracy. Do the values reflect reality? Are they recorded correctly at source?
Consistency. Do the same things look the same everywhere? Is a country, product or status recorded one way or five?
Uniqueness. Does each record appear once, or have duplicates crept in?
Validity. Do values fall within allowed ranges and formats? Is a date really a date?
Timeliness. Is the data current enough for the decision it supports?
Integrity. Do relationships between tables hold? Does every order link to a real customer?
Checking each dimension in turn gives structure to what can otherwise feel like an endless task.
1. Check what arrived. Confirm that the files or records you expected have been delivered, with the right names, formats and number of rows. Compare record counts with the source. A feed that quietly delivers half its usual volume should raise an alarm immediately.
2. Check the structure. Verify that columns are present, in the expected order and with the expected types. Sources change without warning, and a renamed column can silently break everything downstream.
3. Check individual values. Look for impossible or suspicious values: negative quantities, dates in the future, percentages above one hundred, ages that cannot be right, and placeholders such as zero or “unknown” standing in for missing information.
4. Check for duplicates. Test whether records that should be unique really are, and decide how repeated entries will be handled.
5. Check relationships. Make sure items link correctly across tables. Orphaned records, such as transactions with no matching customer, point to deeper problems.
6. Check totals. Reconcile aggregated figures against the source system or an independent figure. If they differ, you need to know exactly why.
7. Check against expectations. Compare the data with what you would reasonably expect. If this month’s volume is ten times last month’s, investigate before accepting it.
8. Check for change. Compare against earlier loads for shifts in distribution, new categories or altered definitions. Gradual drift is easy to miss without a deliberate comparison.
Across EMEA, validation has extra considerations.
Different formats. Date conventions, decimal separators, address structures and character sets vary between countries.
Different definitions. A measure may be calculated differently under local rules or practices.
Different currencies and units. Conversions must be consistent and documented.
Different languages. Names and labels may appear in several languages and scripts, which complicates matching and grouping.
Different data protection requirements. What can be stored, shared and processed may vary, and validation must respect those limits.
Agreeing standards and definitions early, and recording how local variations are handled, prevents many of these problems from reaching the analysis.
Checks are only useful if there is a plan for failures. For each rule, decide in advance:
Should the process stop, or continue with a warning?
Can the issue be corrected automatically, or does it need a person to decide?
Who is notified, and how quickly?
Where are failures recorded, so patterns can be spotted?
A common error is to let questionable data flow through "just this once". Over time, those exceptions become the norm, and trust erodes.
Manual checking does not scale and depends on memory. Validation rules written in Python and SQL, run automatically each time data is loaded, apply the same standards every time and leave a record of the results. Managed with Git and GitHub, they can be reviewed, improved and rolled back.
Document what each rule tests and why it exists. When a rule fires, whoever responds should understand what it means.
Validation results are valuable information in their own right. A simple data quality report, showing which checks passed, which failed and what was done, tells users how far they can rely on the data. It also shows data owners where their sources need attention, which drives improvement at the origin.
Sometimes validation shows that the data cannot support the question being asked. That is a valuable finding, not a failure. It is far better to learn that before a model is built than after a decision has been made. Honest reporting of what the data can and cannot support protects everyone involved.
Consider an organisation consolidating monthly sales data from several countries into one regional report. A validation process would confirm each country's file arrived with the expected record count, check that column structures match, test values for impossible entries, remove or flag duplicates, verify that product and customer links hold, reconcile each country's total with its source system, and compare the month with previous ones for unexplained shifts. Any failure would be logged and sent to the right owner. Only validated data would reach the report.
Analysis gets the attention, but validation earns the trust. A model is only as good as the data beneath it, and the effort spent checking that data is repaid many times over in confidence, fewer corrections and decisions that hold up.
We build validation into every data project, so the numbers behind your decisions have been tested before they are used.