#Import libraries
import pandas as pd
import seaborn as sns
#Load dataset
penguins = sns.load_dataset("penguins")2 Data manipulation
What is dplyr? The dplyr package is designed for data manipulation, written and maintained by Hadley Wickham. It is part of the tidyverse, an ecosystem of packages designed with common APIs and a shared philosophy. Dplyr is designed to abstract common data manipulation tasks into a consistent API. However, the pandas library provides similar functions. The pandas package is a powerful data manipulation library in Python. It provides data structures and functions to manipulate data and in this chapter we will see how to use dplyr functions work in Python.
First, we need to import the necessary libraries. We will use pandas and seaborn. Furthermore, we will load the penguins dataset with load_dataset from seaborn. Let us have a look at the first five rows of the dataset with the .head function.
How does the dataset look like? The .head() function is used to show the first five rows of the dataset. The .tail() function is used to show the last five rows of the dataset. The .shape function is used to show the dimensions of the dataset. Overall, the penguins dataset contains 333 entries and 7 columns. The columns are for example species, island, and body_mass
#Inspect the first five rows of the dataset
penguins.head()#> species island bill_length_mm ... flipper_length_mm body_mass_g sex
#> 0 Adelie Torgersen 39.1 ... 181.0 3750.0 Male
#> 1 Adelie Torgersen 39.5 ... 186.0 3800.0 Female
#> 2 Adelie Torgersen 40.3 ... 195.0 3250.0 Female
#> 3 Adelie Torgersen NaN ... NaN NaN NaN
#> 4 Adelie Torgersen 36.7 ... 193.0 3450.0 Female
#>
#> [5 rows x 7 columns]Maybe you did not realize it, but the penguins dataset contains missing values. The dropna function is used to remove missing values from the dataset. And we can append the .info() function to get an overview of the dataset. The .info() function shows the number of non-null entries in each column. The dataset contains 333 entries and 7 columns.
#Drop missing values
penguins = penguins.dropna()
#Get an overview of the dataset
penguins.info()#> <class 'pandas.core.frame.DataFrame'>
#> Int64Index: 333 entries, 0 to 343
#> Data columns (total 7 columns):
#> # Column Non-Null Count Dtype
#> --- ------ -------------- -----
#> 0 species 333 non-null object
#> 1 island 333 non-null object
#> 2 bill_length_mm 333 non-null float64
#> 3 bill_depth_mm 333 non-null float64
#> 4 flipper_length_mm 333 non-null float64
#> 5 body_mass_g 333 non-null float64
#> 6 sex 333 non-null object
#> dtypes: float64(4), object(3)
#> memory usage: 20.8+ KBIf you want to know the type of the variables in the dataset, you can use the .dtypes function. The dtypes function shows the data types of the variables. In this dataset, the variables species, island, and sex are of type object. Or variable bill_length_mm is of type float64.
#data types of the variables
penguins.dtypes#> species object
#> island object
#> bill_length_mm float64
#> bill_depth_mm float64
#> flipper_length_mm float64
#> body_mass_g float64
#> sex object
#> dtype: object2.1 Filter
A common task in data manipulation is to filter the dataset. Suppose you want to filter the penguins dataset and apply an analysis only for observations for the species Adelie. Use .query() to filter the dataset. The query function is a powerful tool to filter datasets. Append .query() to the dataset and specify the condition you want to filter for.
#Filter penguis dataset where species is Adelie
penguins.query('species == "Adelie"').head()#> species island bill_length_mm ... flipper_length_mm body_mass_g sex
#> 0 Adelie Torgersen 39.1 ... 181.0 3750.0 Male
#> 1 Adelie Torgersen 39.5 ... 186.0 3800.0 Female
#> 2 Adelie Torgersen 40.3 ... 195.0 3250.0 Female
#> 4 Adelie Torgersen 36.7 ... 193.0 3450.0 Female
#> 5 Adelie Torgersen 39.3 ... 190.0 3650.0 Male
#>
#> [5 rows x 7 columns]You can also filter for multiple conditions. Say you want to filter for the species Adelie, but only if the sex of the penguins is not male.
#Filter data with mulqitple conditions
penguins.query('species == "Adelie" & sex != "Male" ').head()#> species island bill_length_mm ... flipper_length_mm body_mass_g sex
#> 1 Adelie Torgersen 39.5 ... 186.0 3800.0 Female
#> 2 Adelie Torgersen 40.3 ... 195.0 3250.0 Female
#> 4 Adelie Torgersen 36.7 ... 193.0 3450.0 Female
#> 6 Adelie Torgersen 38.9 ... 181.0 3625.0 Female
#> 12 Adelie Torgersen 41.1 ... 182.0 3200.0 Female
#>
#> [5 rows x 7 columns]2.2 Arrange
Sometimes it is handy to sort the dataset. The .sort_values() function is used to sort the dataset. Append .sort_values() to the dataset and specify the variable you want to sort by.
#Sort variables
penguins.sort_values(by="body_mass_g").head()#> species island ... body_mass_g sex
#> 190 Chinstrap Dream ... 2700.0 Female
#> 64 Adelie Biscoe ... 2850.0 Female
#> 58 Adelie Biscoe ... 2850.0 Female
#> 116 Adelie Torgersen ... 2900.0 Female
#> 98 Adelie Dream ... 2900.0 Female
#>
#> [5 rows x 7 columns]In other instances, you might want to sort the descendingly. The ascending parameter is used to sort the dataset in ascending order. By default, the ascending parameter is set to True. Set ascending to FALSE to sort the variable in descending order.
#Sort variables descendingly
penguins.sort_values(by="body_mass_g", ascending=False).head()#> species island bill_length_mm ... flipper_length_mm body_mass_g sex
#> 237 Gentoo Biscoe 49.2 ... 221.0 6300.0 Male
#> 253 Gentoo Biscoe 59.6 ... 230.0 6050.0 Male
#> 297 Gentoo Biscoe 51.1 ... 220.0 6000.0 Male
#> 337 Gentoo Biscoe 48.8 ... 222.0 6000.0 Male
#> 299 Gentoo Biscoe 45.2 ... 223.0 5950.0 Male
#>
#> [5 rows x 7 columns]2.3 Select
Large datasets often contain many variables and are difficult to work with. The select function is used to select variables. For example, put the columns species and island in brackets to select them.
#Select columns
penguins[['species', 'island']].head()#> species island
#> 0 Adelie Torgersen
#> 1 Adelie Torgersen
#> 2 Adelie Torgersen
#> 4 Adelie Torgersen
#> 5 Adelie TorgersenAnother way to select columns is to use the .loc function. The latter is used to access a group of rows and columns by labels or a boolean array. For example, select all columns from island to bill_depth_mm.
#select columns with .loc: from island to bill_depth_mm
penguins.loc[:, 'island':'bill_depth_mm'].head()#> island bill_length_mm bill_depth_mm
#> 0 Torgersen 39.1 18.7
#> 1 Torgersen 39.5 17.4
#> 2 Torgersen 40.3 18.0
#> 4 Torgersen 36.7 19.3
#> 5 Torgersen 39.3 20.6You can also select columns by their index. For example, select the first three columns of the dataset. Or suppose you want to check if a variable is in a specified set. Use the isin function to check if the variable is included in a set. For example, select the observations where the species is Adelie.
#A set with the species Adelie
myset = ['Adelie']
#Select observations where the species isin the set
adelie = penguins[penguins["species"].isin(myset)]
adelie.head()#> species island bill_length_mm ... flipper_length_mm body_mass_g sex
#> 0 Adelie Torgersen 39.1 ... 181.0 3750.0 Male
#> 1 Adelie Torgersen 39.5 ... 186.0 3800.0 Female
#> 2 Adelie Torgersen 40.3 ... 195.0 3250.0 Female
#> 4 Adelie Torgersen 36.7 ... 193.0 3450.0 Female
#> 5 Adelie Torgersen 39.3 ... 190.0 3650.0 Male
#>
#> [5 rows x 7 columns]2.4 Mutate
The mutate function is used to create new variables. We can use the assign function to create a new variable. For example, create a new variable weight which is a copy of the variable body_mass_g. Next, the .assign() function is used to create a new variable body_mass_kilo which is the body_mass_g divided by 1000. Feel free to manipulate the dataset and create new variables.
# Create a new dataset
weight = penguins[['body_mass_g']]
#Assign a new variable by common operations
weight = weight.assign(body_mass_kilo = weight['body_mass_g'] /1000)
weight.head()#> body_mass_g body_mass_kilo
#> 0 3750.0 3.75
#> 1 3800.0 3.80
#> 2 3250.0 3.25
#> 4 3450.0 3.45
#> 5 3650.0 3.652.5 Summarize
How to calculate summary statistics of the dataset? The summarize function is used to calculate summary statistics of the dataset. The .describe() function is used to calculate summary statistics of the dataset.
The summarize function is used to calculate summary statistics of the dataset. The .describe() function is used to calculate summary statistics of the dataset.
#Describe the dataset
penguins.describe()#> bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
#> count 333.000000 333.000000 333.000000 333.000000
#> mean 43.992793 17.164865 200.966967 4207.057057
#> std 5.468668 1.969235 14.015765 805.215802
#> min 32.100000 13.100000 172.000000 2700.000000
#> 25% 39.500000 15.600000 190.000000 3550.000000
#> 50% 44.500000 17.300000 197.000000 4050.000000
#> 75% 48.600000 18.700000 213.000000 4775.000000
#> max 59.600000 21.500000 231.000000 6300.000000Another way to calculate summary statistics is to use the .groupby() function. The .groupby function provides the power of the split-apply-combine pattern. The .groupby function is used to split the dataset into groups. The .apply function is used to apply a function to each group. The .combine function is used to combine the results into a new dataset. For example, calculate the mean body_mass_g of the penguins grouped by sex. Append .groupby() to the dataset and specify the variable you want to group by. In addition, use the .mean() function to calculate the mean of the grouped variable.
# .groupby() and .mean() to calculate the mean body_mass_g of the penguins
penguins[["sex", "body_mass_g"]].groupby("sex").mean()#> body_mass_g
#> sex
#> Female 3862.272727
#> Male 4545.684524The .groupby function can be used to group by multiple variables. For example, calculate the mean body_mass_g of the penguins grouped by sex and species.
#Add several variables to group by
penguins.groupby(["sex", "species"])["body_mass_g"].mean()#> sex species
#> Female Adelie 3368.835616
#> Chinstrap 3527.205882
#> Gentoo 4679.741379
#> Male Adelie 4043.493151
#> Chinstrap 3938.970588
#> Gentoo 5484.836066
#> Name: body_mass_g, dtype: float64Finally, the .count() function is used to count the number of entries in each category of a variable.
#shortr penguins["species"].value_counts()
penguins.groupby("species")["species"].count()#> species
#> Adelie 146
#> Chinstrap 68
#> Gentoo 119
#> Name: species, dtype: int64Summary
In this script, we have shown how to use dplyr functions in Python. Keep in mind that the dplyr package is designed for data manipulation in R. However, the pandas library in Python provides similar functions to dplyr. Aggregation statistics can be calculated on entire columns or rows. The groupby function provides the power of the split-apply-combine pattern. The value_counts is a convenient shortcut to count the number of entries in each category of a variable.
The .loc function is used to access a group of rows and columns by labels or a boolean array. Say we want to create a indicator variable Adelie. The .loc function is used to create a new variable Adelie. The first line of code creates a new variable Adelie with the .loc function. The second line of code fills the rest of the column with False. The third line of code selects the columns species and Adelie.
# Create a new variable with the .loc function
penguins.loc[penguins['species'] == "Adelie", 'Adelie'] = 'True'
#Fill the rest of the column with False
penguins.loc[penguins['species'] != "Adelie", 'Adelie'] = 'False'
#Select the columns species and Adelie
penguins[['species', 'Adelie']].head()#> species Adelie
#> 0 Adelie True
#> 1 Adelie True
#> 2 Adelie True
#> 4 Adelie True
#> 5 Adelie TrueSo say we want to create a new variable check_length which checks if the bill_length_mm is less than or equal to 40. The .apply() function is used to apply a function along an axis of the dataframe. For example, the lambda function is used to create a new variable check_length. It checks if the bill_length_mm is less than or equal to 40. If not, the function returns False. If the condition is met, the function returns True.
# Create a new variable check_length: check if the bill_length_mm is less than or equal to 40
penguins['check_length'] = penguins['bill_length_mm'].apply(lambda x: 'True' if x <= 40 else 'False')
#Select the columns bill_length_mm and check_length
penguins[['bill_length_mm', 'check_length']].head()#> bill_length_mm check_length
#> 0 39.1 True
#> 1 39.5 True
#> 2 40.3 False
#> 4 36.7 True
#> 5 39.3 True