Data Preprocessing in Data Mining

Data preprocessing in data mining is a must step. but before learning about data preprocessing we need to learn what is process.

What is a process?

A process is something that takes in an input (data) and produces an output (information). Consider the below image.

The main job of data processing is to simply enter the data as input into a system and analyze it, summarise it, and then convert this data into usable information. Today databases are highly susceptible to noisy, missing, and inconsistent data due to their typically huge size. Remember quality data will lead to low-quality mining results. Now the question arises, “How can the data be preprocessed to help the quality of the data?“

Data preprocessing is an activity that is used to cater to the needs of the above tasks. The steps of activities that data preprocessing includes are:

1. Data cleaning
2. Data integration
3. Data transformation
4. Data reduction

When these data processing techniques, are applied before mining can substantially improve the overall data mining results. These data mining techniques are not mutually inclusive, they may work together.

The data preprocessing methods are organized into the above categories: Concept hierarchies can be used in the alternative form of data reduction where we replace low-level data with higher-level concepts.

Data Preprocessing in Data Mining

In the data mining process, data preprocessing is the main step. We must know that practically, the data we wish to analyze by the data mining techniques are incomplete or dirty. This data could be noisy or inconsistent too.

Now, the point of concern is that real-world databases and data warehouses could accommodate this kind of noisy, incomplete data for sure, and perhaps, sometimes even relevant data may not be retrieved due to any misunderstanding. The reason for this misunderstanding could be malfunctioning of equipment This results in poor quality data which in turn results in poor-quality mining results.

Let us discuss what could be the reasons for this kind of noisy data

1. There must be many possible reasons for noisy data, for example, the instruments that have been used to correct the data could be faulty
2. Computers or manual entry of data could be faulty.
3. Error in data transmission may also occur.
4. There may be technology limitations such as limited buffer size for coordinating synchronized data transfer and consumption.
5. Data could be missing, particularly for the tuples which are having missing values.
6. Data redundancy or duplication could also be considered as a discrepancy Remember, that the data preprocessing step begins with a general overview of the structure of the data and quality assurance This is performed using sampling and visualizing these techniques.

We need to make quality decisions in such a way, that the data warehouse needs to have quality data and also there should be no missing or noisy data at all. Spurious data or poise can be identified by quantitative techniques such as minima and maxima analysis or even by distribution pattern analysis.