How to remove NaN values in Python

You will often face NaN or 'Not a Number' values in data analysis. These missing data points can skew results and cause errors, so their proper removal is crucial.

In this article, you'll learn several techniques to remove NaN values. You'll also get practical tips, real-world applications, and debugging advice to clean your datasets effectively.

Using dropna() to remove NaN values

import pandas as pd
import numpy as np

df = pd.DataFrame({'A': [1, 2, np.nan, 4], 'B': [5, np.nan, 7, 8]})
clean_df = df.dropna()
print(df, "\n\nAfter dropna():\n", clean_df)--OUTPUT--A B
0 1.0 5.0
1 2.0 NaN
2 NaN 7.0
3 4.0 8.0

After dropna():
A B
0 1.0 5.0
3 4.0 8.0

The dropna() method is your go-to for quickly removing rows or columns with NaN values. By default, it operates on rows. Notice how calling df.dropna() removes any row that has even one missing value.

• Row 1 is dropped because of NaN in column 'B'.
• Row 2 is dropped because of NaN in column 'A'.

The resulting clean_df contains only the rows that were fully complete. It's a powerful but blunt approach, ideal when you can afford to discard entire records without compromising your analysis.

Basic techniques for handling NaN values

When dropna() is too blunt, you can gain finer control by filtering arrays with numpy.isnan(), using list comprehensions, or replacing values with fillna().

Using numpy.isnan() to filter arrays

import numpy as np

arr = np.array([1.0, 2.0, np.nan, 4.0, 5.0, np.nan])
clean_arr = arr[~np.isnan(arr)]
print("Original array:", arr)
print("Cleaned array:", clean_arr)--OUTPUT--Original array: [ 1. 2. nan 4. 5. nan]
Cleaned array: [1. 2. 4. 5.]

When working with NumPy arrays, numpy.isnan() is a great tool for targeted filtering. It generates a boolean mask that flags the position of each NaN value in your array.

• The function returns True for every NaN and False for all other values.
• You can then invert this mask using the tilde operator (~) to select only the valid numbers.

This approach gives you a clean array with just the non-NaN elements, making it ideal for precise data filtering without dropping entire rows or columns.

Using list comprehension with math.isnan()

import math

my_list = [1.0, 2.0, float('nan'), 4.0, 5.0, float('nan')]
clean_list = [x for x in my_list if not math.isnan(x)]
print("Original list:", my_list)
print("Cleaned list:", clean_list)--OUTPUT--Original list: [1.0, 2.0, nan, 4.0, 5.0, nan]
Cleaned list: [1.0, 2.0, 4.0, 5.0]

For standard Python lists, a list comprehension offers a concise and readable way to filter out NaN values. It’s a Pythonic one-liner that builds a new list based on a condition you set, making your code clean and expressive.

• The math.isnan() function checks each element to see if it's a NaN value.
• By including if not math.isnan(x) in the comprehension, you're telling Python to only keep the elements that are not NaN.

This method gives you a new, clean list and is often more efficient for this task than a traditional for loop.

Replacing NaN values with fillna()

import pandas as pd
import numpy as np

series = pd.Series([1.0, 2.0, np.nan, 4.0, np.nan])
filled_series = series.fillna(0)
print("Original series:\n", series)
print("After filling NaN values:\n", filled_series)--OUTPUT--Original series:
0 1.0
1 2.0
2 NaN
3 4.0
4 NaN
dtype: float64
After filling NaN values:
0 1.0
1 2.0
2 0.0
3 4.0
4 0.0
dtype: float64

Sometimes, you can't afford to lose data points. The fillna() method offers a great alternative to dropping them, letting you replace NaN values instead. This approach keeps your dataset's structure intact while handling missing data.

• The method takes an argument that specifies the replacement value. In the example, series.fillna(0) swaps every NaN with 0.
• This is useful when a missing value can be logically treated as zero, preserving your data for further analysis.

Advanced techniques for handling NaN values

When the bluntness of dropna() or the simplicity of fillna() won't cut it, more advanced techniques can offer a smarter, more nuanced approach.

Using interpolate() to estimate missing values

import pandas as pd
import numpy as np

series = pd.Series([1.0, 2.0, np.nan, 4.0, np.nan, 6.0])
interpolated = series.interpolate(method='linear')
print("Original series:\n", series)
print("After interpolation:\n", interpolated)--OUTPUT--Original series:
0 1.0
1 2.0
2 NaN
3 4.0
4 NaN
5 6.0
dtype: float64
After interpolation:
0 1.0
1 2.0
2 3.0
3 4.0
4 5.0
5 6.0
dtype: float64

The interpolate() method offers a smart way to fill NaN values by estimating them from the data around them. It's particularly powerful when your data follows a logical sequence, like in time-series analysis. Using method='linear' tells pandas to treat the missing point as the value that falls directly between its neighbors.

• The first NaN is replaced with 3.0, the midpoint between 2.0 and 4.0.
• Similarly, the second NaN becomes 5.0, the value between 4.0 and 6.0.

Replacing NaN with statistical values

import pandas as pd
import numpy as np

df = pd.DataFrame({'A': [1, 2, np.nan, 4], 'B': [5, np.nan, 7, 8]})
filled_df = df.fillna(df.mean())
print("Original DataFrame:\n", df)
print("\nAfter filling with mean values:\n", filled_df)--OUTPUT--Original DataFrame:
A B
0 1.0 5.0
1 2.0 NaN
2 NaN 7.0
3 4.0 8.0

After filling with mean values:
A B
0 1.0 5.0
1 2.0 6.7
2 2.3 7.0
3 4.0 8.0

Replacing NaNs with a column's mean or median is a common strategy to preserve data integrity. This technique is less disruptive than dropping rows, as it fills gaps with a statistically sound estimate, keeping your dataset's size consistent.

• The df.mean() method first calculates the average for each column, skipping over any NaN values.
• Then, fillna() uses these column-specific averages to replace the corresponding NaNs. For example, the missing value in column 'A' is filled with its mean (2.3), and the one in 'B' gets its mean (6.7).

Working with masked arrays

import numpy as np
import numpy.ma as ma

arr = np.array([1.0, 2.0, np.nan, 4.0, 5.0, np.nan])
masked_arr = ma.masked_array(arr, np.isnan(arr))
print("Original array:", arr)
print("Masked array:", masked_arr)
print("Mean of values (ignoring NaN):", masked_arr.mean())--OUTPUT--Original array: [ 1. 2. nan 4. 5. nan]
Masked array: [1.0 2.0 -- 4.0 5.0 --]
Mean of values (ignoring NaN): 3.0

NumPy's masked arrays let you ignore specific elements during calculations without actually removing them. You create one with ma.masked_array(), passing your data and a boolean mask. Here, np.isnan(arr) generates that mask, flagging all NaN values to be ignored.

• The resulting array displays these masked values as --.
• The real advantage is that when you call a method like mean(), it automatically skips the masked elements, giving you a clean calculation based only on the valid data.

Move faster with Replit

Replit is an AI-powered development platform that transforms natural language into working applications. You can describe what you want to build, and Replit Agent creates it—complete with databases, APIs, and deployment.

For the NaN handling techniques we've explored, Replit Agent can turn them into production-ready tools. You could build:

• A data cleaning utility that lets users upload a dataset and choose a method like dropna() or fillna() to process missing values.
• An interactive time-series dashboard that uses interpolate() to fill gaps in financial or sensor data before visualization.
• A scientific calculator that processes experimental results by using masked arrays to compute statistics while ignoring invalid NaN entries.

Describe your app idea, and Replit Agent writes the code, tests it, and fixes issues automatically, all in your browser.

Common errors and challenges

Even with the right tools, handling NaN values can sometimes lead to unexpected errors and tricky situations.

Dealing with errors when comparing to NaN directly

A classic mistake is trying to find NaN values using the equality operator, like my_value == np.nan. This comparison will always return False because, by definition, NaN is not equal to anything, including itself. This unique property is designed to prevent silent errors in calculations.

• To correctly identify NaN values, you must use dedicated functions.
• For pandas objects, use pd.isna(). For NumPy arrays or standard Python numbers, use np.isnan() or math.isnan().

Handling type errors when using dropna() with mixed data types

You might find that dropna() doesn't behave as expected when your columns contain mixed data types. Pandas often converts integer columns to float types to accommodate np.nan, which can be surprising. This automatic type casting is necessary because integers don't have a native NaN representation.

Furthermore, if your data contains Python's built-in None value in an object-type column, your results might vary. While pandas generally treats None and np.nan similarly, subtle differences in how methods handle them can lead to incomplete cleaning.

Troubleshooting unexpected results when using fillna() with method parameters

Using fillna() with a method like 'ffill' (forward fill) or 'bfill' (backward fill) is a powerful way to propagate values, but it has limitations. These methods can fail to fill NaNs that appear at the boundaries of your dataset.

• If the first value in a Series is NaN, 'ffill' has no preceding value to carry forward, so the NaN remains.
• Likewise, if the last value is NaN, 'bfill' has no subsequent value to pull back, leaving the gap unfilled.

Always inspect the head and tail of your data after using these methods to ensure no NaN values were unintentionally left behind.

Dealing with errors when comparing to NaN directly

Directly comparing values to np.nan using the == operator is a classic trap. Because of how NaN is defined in computing standards, this check will always return False—even when comparing NaN to itself—which can cause unexpected behavior in your code.

The following code demonstrates this pitfall. Observe how the boolean mask intended to find NaN values comes up empty, failing to filter the array as intended.

import numpy as np

data = np.array([1.0, 2.0, np.nan, 4.0])
nan_mask = (data == np.nan) # This is wrong!
print("NaN found at positions:", nan_mask)
print("NaN values found:", data[nan_mask].size) # Will be 0

The == operator fails to identify NaN, so the nan_mask becomes an array of all False values. This results in an empty selection, leaving the NaN value untouched. The correct approach is shown in the code below.

import numpy as np

data = np.array([1.0, 2.0, np.nan, 4.0])
nan_mask = np.isnan(data)
print("NaN found at positions:", nan_mask)
print("NaN values found:", data[nan_mask].size) # Will be 1

The correct way to find NaN values is with a dedicated function like np.isnan(). This approach works because NaN has the unique property of never being equal to itself, which makes the == operator useless for this task.

• The np.isnan() function generates a boolean mask, returning True for each NaN's position.
• You can then use this mask to reliably filter your array or count the missing values, ensuring your data cleaning is accurate.

Always use this function for NumPy arrays.

Handling type errors when using dropna() with mixed data types

Using dropna() on a DataFrame with mixed data types can lead to unexpected outcomes. If your data contains both NumPy's np.nan and Python's built-in None, you might not get the clean dataset you wanted. The following code illustrates this scenario.

import pandas as pd
import numpy as np

df = pd.DataFrame({
'A': [1, 2, np.nan, 4],
'B': ['apple', None, 'orange', 'banana']
})
clean_df = df.dropna(subset=['A'])
print("After dropping NaN in column A:\n", clean_df)

Because dropna() is restricted to column 'A', it only removes the row with np.nan. The None value in column 'B' remains untouched, resulting in an incompletely cleaned DataFrame. The following code demonstrates the proper way to handle this.

import pandas as pd
import numpy as np

df = pd.DataFrame({
'A': [1, 2, np.nan, 4],
'B': ['apple', None, 'orange', 'banana']
})
clean_df = df.dropna(how='any')
print("After dropping both NaN and None:\n", clean_df)

The dropna() method handles both np.nan and None by default. The solution works by applying it across the entire DataFrame instead of just a subset of columns.

• Using how='any' instructs pandas to drop a row if it contains a missing value in any column.
• This removes both the row with np.nan and the one with None, giving you a completely clean dataset.

Watch for this when your columns contain mixed data types.

Troubleshooting unexpected results when using fillna() with method parameters

Using fillna() with a method like 'ffill' (forward fill) is a great way to propagate the last valid observation forward. However, it's not a silver bullet. If your data starts with a NaN value, there's nothing to propagate, leaving the gap unfilled.

The following code demonstrates this issue. Notice how the NaN at the beginning of the Series remains even after applying the forward fill.

import pandas as pd
import numpy as np

data = pd.Series([np.nan, 2, np.nan, np.nan, 5])
filled_data = data.fillna(method='ffill')
print("Original data:", data.values)
print("After forward fill:", filled_data.values)

The forward fill works as expected for the middle NaN values, but it can't resolve the first one because no valid data comes before it. The next example shows how to handle this common edge case.

import pandas as pd
import numpy as np

data = pd.Series([np.nan, 2, np.nan, np.nan, 5])
filled_data = data.fillna(method='ffill').fillna(method='bfill')
print("Original data:", data.values)
print("After complete fill:", filled_data.values)

The solution is to chain two fillna() calls. First, fillna(method='ffill') propagates values forward. Then, a second call with method='bfill' fills any remaining NaNs at the beginning by using the next valid observation.

• This two-step process ensures that even leading NaNs are handled correctly.
• It's a robust approach for cleaning datasets where initial entries might be missing, which is common in time-series data.

Real-world applications

Beyond troubleshooting, methods like fillna() and interpolate() are essential for cleaning real-world sensor data and analyzing financial records.

Cleaning sensor data with fillna() methods

Inconsistent sensor readings often leave gaps in time-series data, but you can create a complete and logical dataset by using fillna() to propagate values both forward and backward.

import pandas as pd
import numpy as np

# Sample temperature sensor data with missing values
dates = pd.date_range('2023-01-01', periods=10, freq='D')
temp_data = pd.Series([20.5, 21.2, np.nan, 19.8, 20.1, np.nan, np.nan, 22.3, 21.7, 20.9], index=dates)
print("Original sensor data:")
print(temp_data)

# Strategy: Fill NaN with forward fill, then backward fill for any remaining
clean_data = temp_data.fillna(method='ffill').fillna(method='bfill')
print("\nCleaned sensor data:")
print(clean_data)

This example shows how to reliably fill gaps in time-series data, like from a temperature sensor. It’s a common scenario where chaining fillna() methods provides a complete solution.

• The first step, fillna(method='ffill'), propagates the last known temperature reading forward into any subsequent NaN slots.
• If any NaNs remain at the beginning, the second step, fillna(method='bfill'), fills them by using the next available temperature reading.

This two-pass approach ensures the entire series is filled in.

Handling missing values in financial data with interpolate()

The interpolate() method is especially useful for financial data, where you can fill gaps from non-trading days by estimating values based on their position in time.

import pandas as pd
import numpy as np

# Sample stock price data with missing values (e.g., market holidays)
dates = pd.date_range('2023-01-01', periods=10, freq='B') # Business days
stock_prices = pd.DataFrame({
'AAPL': [150.2, 152.3, np.nan, 153.7, 155.2, np.nan, 154.1, 156.8, np.nan, 157.3],
'MSFT': [242.5, np.nan, 245.2, 244.8, np.nan, 247.9, 248.3, np.nan, 250.2, 252.1]
}, index=dates)

print("Original stock price data:")
print(stock_prices)

# Fill missing values using linear interpolation for time series
filled_prices = stock_prices.interpolate(method='time')
print("\nInterpolated stock prices:")
print(filled_prices)

This example tackles missing values in time-series data. It uses the interpolate() method with method='time' to intelligently fill the NaNs in the stock price DataFrame.

• Unlike a simple linear fill, this method considers the actual timestamps in the index.
• It calculates the missing values based on how much time has passed between the known data points, making the estimates more accurate for data indexed by date or time.

Get started with Replit

Turn these techniques into a real tool. Tell Replit Agent: "Build a utility to clean CSVs with fillna()" or "Create a dashboard that interpolates missing sensor data before plotting."

Replit Agent will write the code, test for errors, and deploy your app from a simple description. Start building with Replit.