Handling Missing Values
Missing values are difficult to deal with, and there is no better way to deal with them. It depends on different scenarios that are random and in some cases nonrandom to the data. First, let us have a clear understanding of what we mean by missingness in data.
Missing Data Mechanisms
Missing values are difficult to deal with, and there is no better way to deal with them. It depends on different scenarios that are random and in some cases nonrandom to the data. First, let us have a clear understanding of what we mean by missingness in data.
Missing values can be indicated in different ways. For instance, indicating a missing integer with -9999
, -999
, -99
or -9
, 999
, ???
, ?
, np.inf
and N/A
. Unknown
is common or sometimes a conventional method is used for indicating floating-point missing value with NaN (Not a Number) in python and NA (Not Available) in R programming. Sometimes you may see zero on some variables, therefore, it is a good idea to take a closer look at those variables as it might be invalid or a missing value.
It is imperative to think about missing data during our analysis. Because, if we fail to take into account our analysis might be futile and if the data is fed to an algorithm, the bias will result in biased algorithms leading to biased decisions.
Before we start imputing missing values it is a good idea if we can visualize the data, so that we may be able to detect the missing values like -9999 which may appear as outliers compared to other values depending on the dataset variables domain.
The following general missingness mechanisms will give us some idea:
-
Missing completely at random (MCAR): This can be summarized as if your value is completely missing a variable at random, then by throwing out the cases of missing data, you will not miss much information or bias your inferences. A safe maximum threshold for missing data is 5% of the total. If a certain feature is missing more than 5% then you should probably discard that feature or collect more information. You can confirm this case by partitioning the data into two parts: one set containing the missing values and the other set containing non-missing values. After partitioning use the t-test of mean difference to check whether there exist any difference in the sample between the two datasets.
-
Missing at random (MAR): Here missingness is not completely random.This means that the probability that a variable is missing depends only on the available full information. In such cases, it is acceptable to exclude the missing value (i.e to treat them as NA or NaN). [expand on this explanation] These are not randomly missing across observations but are distributed within one or more sub-samples. If the missing values are not randomly distributed across the observation, it is problematic. Therefore, they can be ignored by performing data imputation to replace them.
-
Missing that depends on unobserved predictors: This arises when the missing data depends on some unrecorded information. For example, suppose there is a pattern in the missingnes that correlates to another predictor, it would reveal that relationship and the variable and the other unknown predictor. Therefore, the missingness of data is not at random, therefore, it has to be explicitly addressed in modeling by choosing robust algorithms to missing values.
-
Missing that depends on the missing value itself: This is a difficult situation where the probability of missingness depends on the probability of the variable itself. For instance, if you are dealing with a time series data and you see a missing value on the data that might depend on the previous time, that might mispredict seasonality and pattern on the data.
Deciding on best analysis strategy to yield the least biased estimates 1. Deletion Methods + Listwise Deletion, Pairwise Deletion 2. Single Imputation Methods + Mean/Median/Mode substitution, dummy variable (constant) method, single regression 3. Model-Based Methods + Maximum Likelihood, Multiple Imputation
Handling Datasets Containing Missing Values:
1. Listwise Deletion (AKA Complete Case Analysis)
Involves removal of all events (rows) with incomplete data (NA’s, NaN’s) for any feature. This reduces the effective sample size, which in turn, reduces the statistical power for hypothesis testing to unacceptably low values that lead to biased inference. Listwise deletion assumes the data are MCAR, which is not often the case in the real world. So we should be very careful in dealing with listwise deletion.
# removing all the missing values
my_data.dropna(inplace = True)
new_data <- na.omit(my_data)
Advantages:
- Simplicity
- Comparability across analysis
Disadvantages:
- Reduces statistical power (because it lowers the number of records)
- Doesn’t use all information
- Estimates may be biased if data not MCAR
2. Pairwise Deletion (AKA Available Case Analysis):
Pairwise attempts to minimize the loss of information that occurs in listwise deletion. Think of it as a correlation matrix where the correlation measures the strength of the relationship between the available data of two pairs of variables.
Advantages:
- Increases statistical power because it keeps as many records as possible
Disadvantages:
- Similarly to listwise deletion estimates may be biased if data not MCAR
- Can’t compare analysis because of sampling different variables each time
3. Zero/Constant Imputation
Involves substituting the missing values with zero or a constant value such as mean.
Advantages:
- Uses all available information about the missing observation
Disadvantages:
- Leads to biased estimates
- Not theoretically driven
# Replace missing values with a constant number
my_data['col_name'].fillna(10, inplace=True)
# missing values col_1 will be filled with 10
my_data$col_1[is.na(my_data$col_1)] = 10
4. Mean/Median/Mode Imputation:
In mean, median or mode imputation, the mean, median or mode of all values for the same feature is calculated and then imputed to the missing values. Mode imputation is appropriate to nominal and categorical values.
Advantages:
- Uses the whole data
Disadvantages:
- Reduces variability
- Ignores relationship between variables which weakens covariance and correlation estimates
# For python dataframe
# Impute the mean/median/mode for each variable
my_data.fillna(mydata.mean())
my_data.fillna(mydata.median())
my_data.fillna(mydata.mode())
#For Python array data structure
# get the mean of columns you need
cols_mean = np.nanmean(my_array, axis = 0)
# Find the indices where you want to replace
indices = np.where(np.isnan(my_array))
# Replace the column means in the indices
My_array[indices] = np.take (cols_mean, indices[1])
na_mean(my_data, option = 'mean')
na_mean(my_data, option = 'median')
na_mean(my_data, option = 'mode')
impute(data, method = "mean") # method = either "mean/median/mode"
5. Multiple Imputation
One motivation of multiple imputation is to increase statistical power by increasing the number of observations used. Multiple imputation incorporates information from all the variables in the dataset to derive values that would be imputed in the place of the missing values. Instead of filling in a single value for each missing value, a set of plausible values that closely represent the uncertainty of each of the imputed missing values. The general, oversimplified, procedure can be summed in three steps: Impute, Analyze and Pool.
- Impute: Missing values are imputed using a specified regression model. It is repeated m times, resulting in m separate dataset
- Analyze: Analysis is performed within each dataset
- Pool: The results are pooled into one estimate
Advantages:
- Variability due to sampling and due to imputation are more accurate
- Increases precision
- Resistant to outliers
Disadvantages:
- Tedious coding
- Liable to making errors when specifying models
6. Regression Imputation:
A regression model is fitted for the variables with missing values. Based on the resulting model, a new regression model is then fitted and is used to impute the missing values for each variable.
Advantages:
- Uses all information
Disadvantages:
- Overestimates model fit and correlation estimates
- Weakens variance
- May impute values which are out of the possible range for a variable
This can further be classified into two: Deterministic and stochastic regression imputation.
6.1 Deterministic Regression imputation:
Here a linear regression function is computed in which the feature with missing value is dependent variable and the other features would be independent variables. The residual (i.e. error term) The estimate of the dependent variable will be imputed in the missing value.
install.packages("mice") # installing the mice package
library("mice") # load mice package
# Multiple imputation using deterministic regression
my_data_imputed<- mice(my_data, method = "norm.predict", m = 5) #
my_data_imputed <- complete(my_data_imputed) # store data
For Linear regression we use ‘norm.predic’ for the method argument. The default number of imputation is 5, however, you are free to change the value.
6.2 Stochastic Regression Imputation:
This is a two step process which includes regressing from the other variables and adding an extra step of augmenting each predicted score with residual term. The residual term is normally distributed with a mean of zero and variance equal to the residual variance of the predictor on the outcome.
Cons:
- Stochastic regression may lead to implausible values. That is the variable to be imputed may be restricted to certain values (i.e only positive or negative value)
- Stochastic regression imputation leads to poor results when data is heteroscedastic. This assumes that the random error has on average the same size for all parts of the distribution.
install.packages("mice") # installing the mice package
library("mice") # load mice package
# Multiple imputation using deterministic regression
my_data_imputed<- mice(my_data, method = "norm.nob", m = 5) #
my_data_imputed <- complete(my_data_imputed) # store data
import fancyimpute
mice = fancyimpute.MICE(verbose=0)
X_train_fancy_mice = mice.complete(X_train)
We have changed the method argument from ‘norm.predict’ to ‘norm.nob’ for stochastic regression imputation.
7. Hot-Deck:
This method is one of the simplest single-imputation methods used to replace missing values randomly from background attributes which are selected by the user and are maximally similar to the missing one.
Here are some steps to follow when dealing with multiple variables containing missing values in a dataset:
- First rank the variables by percent of missing information.
- Begin imputing one variable at a time on those that has least MV
- Retain and use the newly substituted values in the model-building process for each subsequent variable
- Use every variable in building imputation models and discard the lower ranked variables with MVs.
- In evaluating model based imputation, compare the pre-imputed means and standard deviation (based on the full information after deleting the MV’s) with the post-imputation values. If imputation is successful, there will be little or no significant difference in these mariginals.
Sources:
Python Data Science Handbook by Jake Vanderplas
Missing Values in Data Statistics Solutions
Missing Data Imputation Data Analysis Using Regression and Multilevel/Hierarchical Models by Andrew Gelman et. al
Missing Data & How to Deal: An overview of missing data by Melissa Humphries
Regression Imputation (Stochastic vs. Deterministic & R Example) by Joachim Schork
A data set with missing values in multiple variables Stackexchange


Iam a cohort 2 graduate at LakeHub academy . A full stack software developer specialize in front-end and back-end web development software programmes. My tech journey started at Lakehub academy ,i acquired specific skills including JavaScript, ,React , Express , MongoDB. I successfully completed career coaching training for job readiness an opportunity supported by Lakehub. Am a graduate of the pilot women wentorship program by Microsoft. I am social science passionate about engaging young people .I am currently advancing my skills and open for any internship opportunities to sharpen my skills


| | |
- Checker
class Checker |
| |
This class is passed to a handshake function to check the other
party's certificate chain.
If a handshake function completes successfully, but the Checker
judges the other party's certificate chain to be missing or
inadequate, a subclass of
L{tlslite.errors.TLSAuthenticationError} will be raised.
Currently, the Checker can check either an X.509 or a cryptoID
chain (for the latter, cryptoIDlib must be installed). |
| |
Methods defined here:
- __call__(self, connection)
- Check a TLSConnection.
When a Checker is passed to a handshake function, this will
be called at the end of the function.
@type connection: L{tlslite.TLSConnection.TLSConnection}
@param connection: The TLSConnection to examine.
@raise tlslite.errors.TLSAuthenticationError: If the other
party's certificate chain is missing or bad.
- __init__(self, cryptoID=None, protocol=None, x509Fingerprint=None, x509TrustList=None, x509CommonName=None, checkResumedSession=False)
- Create a new Checker instance.
You must pass in one of these argument combinations:
- cryptoID[, protocol] (requires cryptoIDlib)
- x509Fingerprint
- x509TrustList[, x509CommonName] (requires cryptlib_py)
@type cryptoID: str
@param cryptoID: A cryptoID which the other party's certificate
chain must match. The cryptoIDlib module must be installed.
Mutually exclusive with all of the 'x509...' arguments.
@type protocol: str
@param protocol: A cryptoID protocol URI which the other
party's certificate chain must match. Requires the 'cryptoID'
argument.
@type x509Fingerprint: str
@param x509Fingerprint: A hex-encoded X.509 end-entity
fingerprint which the other party's end-entity certificate must
match. Mutually exclusive with the 'cryptoID' and
'x509TrustList' arguments.
@type x509TrustList: list of L{tlslite.X509.X509}
@param x509TrustList: A list of trusted root certificates. The
other party must present a certificate chain which extends to
one of these root certificates. The cryptlib_py module must be
installed. Mutually exclusive with the 'cryptoID' and
'x509Fingerprint' arguments.
@type x509CommonName: str
@param x509CommonName: The end-entity certificate's 'CN' field
must match this value. For a web server, this is typically a
server name such as 'www.amazon.com'. Mutually exclusive with
the 'cryptoID' and 'x509Fingerprint' arguments. Requires the
'x509TrustList' argument.
@type checkResumedSession: bool
@param checkResumedSession: If resumed sessions should be
checked. This defaults to False, on the theory that if the
session was checked once, we don't need to bother
re-checking it.
| |


{% extends "workshops/object_diff.html" %} {% load diff %} {% block title %}
| slug: | {% semantic_diff previous_version current_version 'slug' %} |
|---|---|
| start date: | {% semantic_diff previous_version current_version 'start' %} |
| end date: | {% semantic_diff previous_version current_version 'end' %} |
| host: | {% semantic_diff previous_version current_version 'host' %} |
| administrator: | {% semantic_diff previous_version current_version 'administrator' %} |
| tags: | {% semantic_diff previous_version current_version 'tags' %} |
| url: | {% semantic_diff previous_version current_version 'url' %} |
| eventbrite key: | {% semantic_diff previous_version current_version 'reg_key' %} |
| admin fee: | {% semantic_diff previous_version current_version 'admin_fee' %} |
| invoice: | {% semantic_diff previous_version current_version 'invoice_status' %} |
| attendance: | {% semantic_diff previous_version current_version 'attendance' %} |
| contact: | {% semantic_diff previous_version current_version 'contact' %} |
| location country: | {% semantic_diff previous_version current_version 'country' %} |
| location venue: | {% semantic_diff previous_version current_version 'venue' %} |
| location address: | {% semantic_diff previous_version current_version 'address' %} |
| location latitude: | {% semantic_diff previous_version current_version 'latitude' %} |
| location longitude: | {% semantic_diff previous_version current_version 'longitude' %} |
| notes: | {% semantic_diff previous_version current_version 'notes' %} |