When working with DataFrames in Python, you may encounter situations where you need to filter and select specific rows based on certain conditions. In this blog post, we will explore how to create a new DataFrame that includes rows with specific criteria. We will also cover how to handle rows with missing values.

Problem Statement

Suppose you have a DataFrame with the following structure:

List Number	Political Group	Votes	Percentage	Position
23″A”	FOR CLUB, CHURCH, AND SCHOOL	1860	19.03	Cordoba
23″B”	UNITED FOR THE PEOPLE	1067	10.91	Cordoba
23″C”	CITIZENS	1130	11.56	Cordoba
23″D”	PRODUCTION AND WORK	2154	22.03	Cordoba
23″F”	POPULARS CBA	3565	36.47	Cordoba
TOTAL	TOTAL UNION POPULAR FEDERAL	9776	0.47	Cordoba

You want to create a new DataFrame that:

1. Includes rows where “List Number” starts with ’23’.
2. Includes rows where “Position” is ‘National Deputy’.
3. Includes rows with missing or empty “Votes” values.
4. Excludes certain rows based on specific criteria.

Solution

Here’s how you can achieve this using Python and the pandas library:

Step 1: Import the necessary libraries

import pandas as pd

Step 2: Create a sample DataFrame

data = {
    'List Number': ['23"A"', '23"B"', '23"C"', '23"D"', '23"F"', 'TOTAL'],
    'Political Group': ['FOR CLUB, CHURCH, AND SCHOOL', 'UNITED FOR THE PEOPLE', 'CITIZENS', 'PRODUCTION AND WORK', 'POPULARS CBA', 'TOTAL UNION POPULAR FEDERAL'],
    'Votes': [1860, 1067, 1130, 2154, 3565, 9776],
    'Percentage': [19.03, 10.91, 11.56, 22.03, 36.47, 0.47],
    'Position': ['Cordoba', 'Cordoba', 'Cordoba', 'Cordoba', 'Cordoba', 'Cordoba']
}

df = pd.DataFrame(data)

Step 3: Filter rows based on specific conditions

We will filter rows where “List Number” starts with ’23’ and “Position” is ‘National Deputy’.

df_filtered = df[df['List Number'].str.startswith('23') & (df['Position'] == 'National Deputy')]

Step 4: Handle rows with missing or empty “Votes” values

We will select rows where “Votes” is NaN or empty.

df_missing_votes = df[df["Votes"].isna() | (df["Votes"] == "")]

Step 5: Combine the filtered rows

We will append the rows with missing or empty “Votes” values to the filtered DataFrame.

df_combined = pd.concat([df_filtered, df_missing_votes], ignore_index=True)

Step 6: Display the final DataFrame

print(df_combined)

Conclusion

By following the steps outlined in this blog post, you can create a new DataFrame that includes specific rows based on certain conditions. This approach is particularly useful when working with large datasets where you need to filter and analyze specific subsets of data.