---
title: "What is Data Cleaning in Machine Learning?"
description: "A model that scores 95% accuracy in testing and then misfires in production usually isn't broken — its training data was. Missing values, duplicate rows,…"
url: "https://unidata.pro/data-cleaning-in-ml/"
date_modified: "2026-09-17T10:52:03+03:00"
language: "en-US"
---
![What is Data Cleaning in Machine Learning?](https://unidata.pro/wp-content/uploads/2026/09/data-cleaning-cover-scaled.webp)A model that scores 95% accuracy in testing and then misfires in production usually isn't broken — its training data was. [Missing values](https://unidata.pro/blog/missing-values-in-data/), duplicate rows, and inconsistent formats teach an algorithm the wrong pattern long before anyone touches the model architecture.

Data cleaning is the process that catches these problems before they reach the model: finding and fixing missing values, [outliers](https://unidata.pro/blog/outliers-and-how-to-spot-them/), duplicates, and formatting errors so the data going in actually represents the problem being solved. This guide covers the core techniques, a repeatable workflow, and the leakage mistakes that quietly undo the work.

Understanding data cleaning fundamentals
----------------------------------------

A spreadsheet full of customer records can look complete and still be unusable — ages stored as text in one column and integers in another, country names spelled three different ways, the same purchase logged twice under two order IDs. Data cleaning is the work of finding and correcting these issues: missing entries, duplicate records, outliers, wrong data types, and inconsistent formatting.

None of this is optional groundwork that happens once. Real [datasets](https://unidata.pro/blog/what-is-dataset/) accumulate errors continuously — from manual entry, merged systems, sensor drift, or schema changes — so cleaning runs every time a new batch of data joins the training set. The output of a cleaning pass is not a polished dataset for its own sake; it's a dataset where every column means what the schema says it means, and where missingness, duplication, and outliers have been deliberately handled rather than silently ignored.

### Data cleaning vs data preprocessing vs data wrangling

These three terms get used interchangeably in job postings and loosely in conversation, but they describe different scopes of work, and mixing them up causes teams to skip steps.

Data cleaning is the narrowest: correcting or removing bad values — missing data, duplicates, outliers, typos — without changing the data's structure or units. Data preprocessing is broader. It includes cleaning, plus the transformations that prepare clean data for a specific algorithm: encoding categories as numbers, scaling features to a common range, and engineering new features. Data wrangling (also called data munging) is broader still — it covers reshaping, joining, and restructuring raw data from its original form (nested JSON, multiple source tables, log files) into the flat, tabular structure cleaning and preprocessing assume already exists.

| **Term** | **Scope** | **Typical task** |
|---|---|---|
| Data cleaning | Fix bad values | Impute a missing age, drop a duplicate row |
| Data preprocessing | Clean + transform for modeling | One-hot encode a category, scale a feature |
| Data wrangling | Restructure raw data into usable form | Flatten nested JSON, join three source tables |

In practice, a single pipeline run usually does all three in sequence: wrangle the raw sources into one table, clean the values inside it, then preprocess for the model that will consume it.

![Data Cleaning vs Data Preprocessing vs Data Wrangling](https://unidata.pro/wp-content/uploads/2026/09/data-cleaning-data-preprocessing-scaled.webp)Why data cleaning determines model performance
----------------------------------------------

A team that sees accuracy plateau often starts tuning hyperparameters first — when the ceiling is sitting in the training data, not the model. Andrew Ng's data-centric AI framing makes this the default diagnosis to check before the model: holding the architecture fixed and improving dataset quality is frequently what moves the accuracy number, because a model can only learn patterns that are actually present and correctly represented in its training data \[2\].

The scale of the underlying work is well documented. A widely cited CrowdFlower survey found data scientists spend about 60% of their time cleaning and organizing data — more than double the time spent on any other single task, including model building — and 57% rated it the least enjoyable part of the job \[1\]. Combined with data collection, data preparation overall accounts for roughly 80% of a typical project's time \[1\].

![Why Data Cleaning Determines Model Performance](https://unidata.pro/wp-content/uploads/2026/09/model-performance-scaled.webp)Cleaning doesn't always help, though, and treating it as a universal fix is its own mistake. Over-aggressive outlier removal can strip out the rare cases a fraud or anomaly model exists to catch. Imputing missing values with column averages can flatten variance a model needed to learn from. The right amount of cleaning depends on what the model is meant to detect — not on getting the dataset as "clean" as possible.

![ Model Performance](https://unidata.pro/wp-content/uploads/2026/09/model-performance2-scaled.webp)Core data cleaning techniques
-----------------------------

Three categories of problems show up in almost every raw dataset: values that are missing, values that are statistically extreme, and records that duplicate or contradict each other. Each has a different fix, and applying the wrong one introduces new bias rather than removing it.

### Handling missing values

The right fix depends on why the data is missing, not just how much is missing. The standard taxonomy, which builds on Rubin's 1976 work, splits missingness into three mechanisms: missing completely at random (MCAR), where the gap has nothing to do with any variable; missing at random (MAR), where it correlates with other observed variables; and missing not at random (MNAR), where the missingness itself depends on the unobserved value — a patient skipping a survey question because of the condition the question asks about \[3\]. Deleting rows is only safe under MCAR with a small affected share; under MAR or MNAR, deletion biases the remaining sample.

![Handling Missing Values](https://unidata.pro/wp-content/uploads/2026/09/handling-missing-values-scaled.webp)Imputation methods range from simple to context-aware. Mean, median, or mode substitution is fast but compresses variance and can distort relationships between columns. K-nearest-neighbors imputation fills a gap using the values of the most similar rows on other features, which preserves more of the original structure at a higher computational cost \[4\]. Regression imputation predicts the missing value from other columns directly, which works well when those columns are genuinely correlated with the missing one and poorly otherwise.

![Handling Missing Values](https://unidata.pro/wp-content/uploads/2026/09/cleaning-data-model-performance-scaled.webp)### Detecting and treating outliers

Two thresholds dominate outlier detection in practice. Tukey's rule flags any value beyond 1.5 times the interquartile range (IQR) above the third quartile or below the first as a potential [outlier](https://unidata.pro/blog/outliers-and-how-to-spot-them/), and beyond 3 times the IQR as a probable one \[5\]. The z-score method flags values more than 3 standard deviations from the mean, which works well for roughly normal distributions but loses reliability on skewed data.

A flagged value isn't automatically a removal candidate. The first step is always to check whether it's a genuine data-entry error (an age of 999, a negative price) versus a real, rare event the model needs to see — fraud cases and equipment failures are outliers by definition, and removing them removes the signal the model exists to learn. Where the value is legitimate but distorts a sensitive algorithm, capping it at a threshold (winsorizing) or applying a log transform preserves the data point while controlling its influence.

![Detecting and Treating Outliers](https://unidata.pro/wp-content/uploads/2026/09/detecting-treating-outliers-scaled.webp)### Removing duplicates and fixing inconsistencies

Exact duplicates — identical rows from a double-submitted form or a re-run import job — are the easy case; pandas' drop\_duplicates() removes them in one call once the relevant columns are identified \[6\]. Near-duplicates are harder: the same customer entered as "Jon Smith" and "Jonathan Smith," or an address with and without an apartment number. Catching these usually requires fuzzy string matching or record-linkage techniques that score similarity rather than checking for exact equality, since a straight equality check misses every formatting variant.

Inconsistency goes beyond duplicate rows into formatting itself: a status column with "Active," "active," and "ACTIVE" as three distinct values, or dates stored as both MM/DD/YYYY and DD-MM-YYYY in the same column. Standardizing casing, units, and date formats before any aggregation or join runs is what prevents a GROUP BY from silently splitting one category into three.

![Removing Duplicates and Fixing Inconsistencies](https://unidata.pro/wp-content/uploads/2026/09/removing-duplicated-scaled.webp)Transforming and standardizing data
-----------------------------------

A dataset can be fully clean — no missing values, no duplicates, no unresolved outliers — and still be unusable by most algorithms, because clean isn't the same as numeric and scaled. Categorical text and features on wildly different ranges both need a transformation step before training.

### Encoding categorical variables

The right encoding depends on whether categories have an order. One-hot [encoding](https://unidata.pro/blog/encoding-categorical-variables-one-hot-vs-label/) creates a separate binary column per category \[7\] — correct for unordered values like "country" or "payment method," since it doesn't impose a false ranking. Label encoding assigns each category an integer, which is appropriate only when the categories have a genuine order (small/medium/large); applied to unordered data, it teaches the model a ranking that doesn't exist. Ordinal encoding is the deliberate version of label encoding, where the integer order is explicitly chosen to match a known hierarchy rather than assigned arbitrarily.

![Encoding Categorical Variables](https://unidata.pro/wp-content/uploads/2026/09/encoding-categorical-variables-scaled.webp)### **Scaling and norma**l**ization**

Distance-based and gradient-based algorithms — k-nearest neighbors, support vector machines, neural networks — are sensitive to feature scale; a column ranging 0–1,000,000 will dominate one ranging 0–1 regardless of which actually matters more. Min-max scaling rescales features to a fixed range, typically 0–1. Standardization (z-score scaling) centers features at a mean of 0 with a standard deviation of 1, which suits algorithms that assume roughly normal data. Scaling by median and IQR — RobustScaler in scikit-learn — uses the median and interquartile range instead of the mean and standard deviation, which keeps a handful of extreme outliers from compressing the rest of the data into a narrow band \[8\]. Tree-based models (random forests, gradient boosting) split on thresholds rather than distances, so they're largely insensitive to scale and this step can often be skipped for them.

![Scaling and Normalization](https://unidata.pro/wp-content/uploads/2026/09/scaling-normalization-scaled.webp)Building a repeatable data cleaning workflow
--------------------------------------------

A cleaning pass that lives only in a notebook's run history isn't reproducible — the next person, or the same person three months later, can't tell what was done or why. Treating data cleaning as code, not a one-off manual edit, is what makes it auditable.

Exploratory data analysis (EDA) is step zero, not an afterthought: profiling column types, null rates, value distributions, and obvious anomalies before deciding which cleaning steps a dataset actually needs. Skipping straight to cleaning without this step means guessing at problems instead of measuring them.

Pandas covers most of the cleaning operations directly — isnull(), fillna(), drop\_duplicates() — and scikit-learn covers the preprocessing transformations that follow, with its Pipeline object chaining imputers, encoders, and scalers into a single fit/transform unit that runs identically on training and new data \[9\]. Version-controlling the cleaning script (not just the cleaned output file) and logging which rows were dropped, imputed, or capped — and why — turns a one-time fix into something the next person can rerun and check.

![Building a Repeatable Data Cleaning Workflow](https://unidata.pro/wp-content/uploads/2026/09/data-cleaning-workflow-scaled.webp)Avoiding data leakage and common pitfalls
-----------------------------------------

Data leakage happens when information that wouldn't be available at prediction time leaks into training, and the most common cause is a cleaning or scaling step run before the train/test split instead of after. Fitting a scaler or an imputer on the full dataset means the [test set](https://unidata.pro/blog/training-validation-test-datasets/)'s statistics — its mean, its range, its missing-value pattern — quietly influence how the training data gets transformed.

The fix is mechanical: fit every cleaning and preprocessing step on the training data only, then apply that same fitted transformation to validation and test data without refitting \[9\]. A scikit-learn Pipeline enforces this by construction, since each step's fit() only ever sees the data it's called on. The symptom of skipped leakage prevention is a model that posts strong validation metrics and then underperforms in production — the model never learned to generalize, because evaluation never tested it on truly unseen statistics in the first place. Target leakage is a related but distinct mistake: a feature that encodes the outcome itself, such as a "cancellation date" column present only for customers who already churned.

![Avoiding Data Leakage and Common Pitfalls](https://unidata.pro/wp-content/uploads/2026/09/data-leakage-1-scaled.webp)Real-world applications across industries
-----------------------------------------

A retailer building customer segments needs purchase and demographic records cleaned first — duplicate customer IDs from a merged loyalty database, missing income fields, and inconsistent product category labels all distort which cluster a customer lands in before any clustering algorithm runs.

In manufacturing, predictive maintenance models depend on sensor data that arrives with dropped readings, transmission noise, and occasional out-of-range spikes from a faulty sensor rather than a real fault; a 2025 study on IoT-based predictive maintenance at container terminals applied cleaning and structuring to condition-monitoring data before training models to flag equipment failures ahead of breakdown \[10\].

In finance, fraud detection models face a structural cleaning challenge: fraud cases are rare and statistically extreme by definition, so the outlier-removal habits that help other models can strip out exactly the cases the model needs to learn from. A 2025 study on credit card fraud detection addressed this by combining feature selection methods to retain the signal in a small set of genuinely informative, cleaned features rather than removing the rare-event rows entirely \[11\].

**Conclusion and next steps**
-----------------------------

Data cleaning is the step that decides whether a model learns the pattern in the business problem or the pattern in the data's mistakes. Missing-value handling, outlier treatment, deduplication, encoding, and scaling each correct a different failure mode, and skipping the workflow and version-control habits around them turns a one-time fix into a problem that resurfaces with every new data batch.

If the raw data behind a model still needs structured labeling rather than just cleaning — [start here](https://unidata.pro/data-annotation/).

## Additional Modules

### references

**List of Links:**

- **Link:** [[1] Press, G. "Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says" — Forbes — 2016](https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/) — **Active link:** active
- **Link:** [[2] Ng, A. "Unbiggen AI" — IEEE Spectrum — 2022](https://spectrum.ieee.org/andrew-ng-data-centric-ai) — **Active link:** active
- **Link:** [[3] Rubin, D. B. "Inference and Missing Data" — Biometrika, Vol. 63, No. 3 — 1976 — pp. 581–592; Little, R. J. A. and Rubin, D. B. Statistical Analysis with Missing Data — Wiley](https://www.academia.edu/145171692/Inference_and_Missing_Data) — **Active link:** active
- **Link:** [[4] Scikit-learn developers. "sklearn.impute.KNNImputer" — scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html) — **Active link:** active
- **Link:** [[5] NIST/SEMATECH. "7.1.6. What are outliers in the data?" and "1.3.5.17. Detection of Outliers" — e-Handbook of Statistical Methods, National Institute of Standards and Technology ](https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm) — **Active link:** active
- **Link:** [[6] The pandas development team. "pandas.DataFrame.drop_duplicates" — pandas documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop_duplicates.html) — **Active link:** active
- **Link:** [[7] Scikit-learn developers. "sklearn.preprocessing.OneHotEncoder" — scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) — **Active link:** active
- **Link:** [[8] Scikit-learn developers. "6.3. Preprocessing data" — scikit-learn User Guide](https://scikit-learn.org/stable/modules/preprocessing.html) — **Active link:** active
- **Link:** [[9] Scikit-learn developers. "Common Pitfalls and Recommended Practices" — scikit-learn documentation](https://scikit-learn.org/stable/common_pitfalls.html) — **Active link:** active
- **Link:** [[10] Aslam, S. et al. "Machine Learning-Based Predictive Maintenance at Smart Ports Using IoT Sensor Data" — Sensors, 25(13), 3923 — 2025](https://www.mdpi.com/1424-8220/25/13/3923) — **Active link:** active
- **Link:** [[11] Siam, A. M., Bhowmik, P., Uddin, M. P. "Hybrid feature selection framework for enhanced credit card fraud detection using machine learning models" — PLOS ONE, 20(7), e0326975 — 2025](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0326975) — **Active link:** active

[Full list of this site's AI-readable pages](https://unidata.pro/llms.txt)