For simple data processing , We are basically dealing with complete data sets , But in practical problems, we often encounter data with missing values , It is very important to deal with this kind of data .

General steps for handling missing values

First, we list the general steps to deal with missing values , Have a general understanding of the whole process .

* Identify missing data ;
* Check the cause of missing data ;
* Delete instances containing missing values or interpolate missing values with reasonable values .
Types of missing data

* Complete random deletion (MCAR)
* Random deletion (MAR)
* Nonrandom deletion (NMAR)
Complete random deletion : If the missing data of a variable is not related to any other observed and unobserved variables , The data is completely random missing .

Random deletion : If the missing data on a variable is related to other observed variables , Not related to his own unobserved values , The data is missing randomly .

Nonrandom deletion : If the missing data does not belong to the above two types, it is non random missing .

Identify missing values

To handle missing values , First, we need to identify which data are missing values ,R In language ,NA Represents missing value ,NaN Represents an impossible value ,Inf and -Inf It represents positive infinity and negative infinity . There are corresponding functions,is.nan(),is.infinite() It can be used to identify missing values , Impossible value and infinite value , The result is TRUE/FALSE.

To count the number of missing values , We can go through it directly sum() Function TRUE/FALSE Make statistics , among TRUE The logical value of is 1,FALSE The logical value of is 0, Similarly, impossible values and infinite values can also be judged by this method .

Explore missing values

For missing values , It is not advisable for us to count him only , This section gives several ways to explore missing values .

one , The chart shows missing values

We can use an icon to show the missing values , stay R In language mice In the bag md.pttern() Function provides a table that can generate a matrix to show missing values , Examples are as follows :
library(lattice) library(mice) data(sleep,package="VIM") md.pattern(sleep)
The results of the chart and graph are as follows :

two , Graphic display missing values

md.pattren() Function has given us a clear list of each missing value , But the graph is a more clear way to express the missing value ,VIM A large number of visualization functions are provided in the package , Let's take a look at some of these functions .

aggr() function
library(VIM) aggr(sleep,prop=FALSE,numbers=TRUE)

matrixplot() function

marginplot() function

Treatment of missing values

one , delete

For the missing value processing, we first use the first simplest method —— Delete the row with the missing value ,R There are two functions to delete missing values , namely complete.cases() Function sum na.omit() function .

For the processing of deletion, you can use the data directly , There is no demonstration here .

two , multiple imputation

multiple imputation (MI) It is a method to deal with missing values based on repeated simulation , In the face of complex missing value problem ,MI It's a common method , It will generate a complete set of data sets from a data set containing missing values . In this section we will use R In mice The package interpolates the data set .

mice The workflow of the multiple interpolation method in the package is as follows :


be based on mice Package analysis usually follows the following procedure :
library(mice) imp <- mice(mydata,m) fit <- with(imp, analysis) pooled <-
pool(fit) summary(pooled)
Process description :

* mydata Is a matrix or data frame containing missing values
* imp It's an inclusion m A list object of an interpolation dataset , At the same time, it also contains the information to complete the interpolation process . default m The value of is 5.
* analysis Is an expression object , Used to set the m A statistical analysis method of interpolation data sets .
* fit It's an inclusion m A list object of individual statistical analysis results .
* pooled It's one that includes this m A list object of statistical average analysis results .
three , Simple interpolation method

Simple interpolation method , Use a value ( Such as mean value , median , Mode ) To replace missing values in variables . One thing to note is that , These substitutions are random , This means that random errors are not introduced .

four , Other ways to deal with missing values

R The language supports other processing methods for missing values .

Package description
Hmisc Contains multiple functions , Simple interpolation is supported , Multiple interpolation and typical variable interpolation .
mvnmle Maximum likelihood estimation of missing values in multivariate orthonormal distribution data .
cat Multiple imputation of multivariate categorical variables in linear model .
arrayInpute,Seqknn Real time function for processing missing data of microarray .
longitudinalData List of related functions , For example, a series of functions for interpolating missing values of time series .
kmi Methods of dealing with missing values in survival analysis Kaplan-Meier multiple imputation .
mix Multiple imputation of mixed categorical and continuous data in general location model .
pan Multiple interpolation of multivariate panel data or clustering data .