Using summarise_each() to Apply Functions to Non-group

Understanding the Problem with Aggregate and Dplyr

The question at hand revolves around utilizing the dplyr package to apply a function to all non-group_by columns in a data frame. The user is seeking an alternative approach to achieving this goal, as they are familiar with using the aggregate() function.

Background on aggregate() and dplyr

For those unfamiliar with both aggregate() and dplyr, let’s take a moment to briefly discuss how these two functions work in R.

Aggregate() is part of the base R library and provides a way to perform aggregation operations. When applied to a data frame, it groups the observations based on one or more variables (called “variables” rather than columns), applies a specified function to each group, and then combines the results into a single output data frame.

On the other hand, dplyr is a popular R package designed to provide an alternative approach to data manipulation. It’s structured around three main functions: filter(), arrange(), summarise(), and mutate(). Each function plays a crucial role in the “pipe” (%>%) syntax, which allows users to chain multiple operations together for efficient data processing.

Now that we have some background on both aggregate() and dplyr, let’s dive deeper into solving this problem using dplyr.

Using summarise_each() to Apply Functions to Non-group_by Columns

In the original question, the user is trying to use summarize to apply a function (mean) to all columns that are not being grouped. The issue arises when there are multiple non-group_by columns in the data frame.

The provided answer suggests using an experimental version of dplyr, specifically the summarise_each() function, which can be accessed via devtools::install_github("hadley/dplyr", ref = "colwise") and then loaded as part of the normal library(dplyr) command.

Exploring summarise_each()

Let’s break down how summarise_each() works:

iris %.%
  group_by(Species) %.%
  summarise_each(funs(mean))

In this example, we’re grouping our data frame (iris) by the Species column using group_by(), and then applying the summarise_each() function to each row in the data frame.

Inside summarise_each(), the funs(mean) part is crucial. Here’s what happens when you pass a list of functions to this function:

library(dplyr)
# create an iris dataframe
iris <- data.frame(Species = c("setosa", "versicolor", "virginica"),
                    Sepal.Length = c(5.006, 5.936, 6.588),
                    Sepal.Width = c(3.428, 2.770, 2.974),
                    Petal.Length = c(1.462, 4.260, 5.552),
                    Petal.Width = c(0.246, 1.326, 2.026))

# define the data frame
iris$Species <- as.factor(iris$Species)
data("iris")

# Apply mean to each column using summarise_each()
iris %>% 
  group_by(Species) %>% 
  summarise_each(funs(mean))

When you run this code, summarise_each() takes the list of functions (funs(mean)), applies them one by one to every row in your data frame, and returns a new data frame where each column that received the function has been replaced with its corresponding mean.

The output is:

## Source: local data frame [3 x 5]
## 
##     Species Sepal.Length Sepal.Width Petal.Length Petal.Width
## 1     setosa        5.006       3.428        1.462       0.246
## 2 versicolor        5.936       2.770        4.260       1.326
## 3  virginica        6.588       2.974        5.552       2.026

Limitations of summarise_each()

While summarise_each() offers a powerful and flexible solution for applying functions to all columns that are not being grouped, there is an important caveat: this approach does not preserve the original column names.

Let’s see what happens when we try to access one of these newly created columns:

iris %>% 
  group_by(Species) %>% 
  summarise_each(funs(mean)) %>% 
  pull(Sepal.Length)

The resulting vector will look something like this:

## [1] NA NA NA

Notice that pull() can’t extract the values of a column named “Sepal.Length” when it originally had the name “Sepal.Length”. The output is full of missing (NA) values.

As you might have guessed, the reason for this behavior is due to how R handles variable names. In R, all variables are created as character strings by default, so when you apply a function using summarise_each(), it essentially changes the column names to be new character strings that contain the original names but wrapped in double quotes ("Sepal.Length").

This is an important point to keep in mind: while summarise_each() can accomplish what we need, we must understand its behavior and potential limitations.

Practical Considerations for Using summarise_each()

While it’s technically possible to use summarise_each(), there are several practical considerations you should be aware of when deciding whether or not to employ this technique:

Column naming issues: As shown earlier, using summarise_each() means that the column names will be changed. If you need to access these columns later on in your data analysis pipeline, you’ll have to use functions like pull() and deal with missing values.
Code readability and maintainability: When working with large datasets or multiple datasets with different column structures, using summarise_each() can lead to code that’s less readable due to the lack of explicit naming for each column.

Considering these points, we should explore alternative methods that avoid some of these challenges:

Alternative Methods: Using Map() and lapply()

For smaller data frames, it’s often more straightforward to use functions like map() or lapply() from the base R library. These functions provide a way to apply operations on vectors of columns that are not being grouped.

Using map() with dplyr

As part of your workflow in dplyr, you might need to apply transformations across multiple columns at once:

library(dplyr)

# create an iris dataframe
iris <- data.frame(Species = c("setosa", "versicolor", "virginica"),
                    Sepal.Length = c(5.006, 5.936, 6.588),
                    Sepal.Width = c(3.428, 2.770, 2.974),
                    Petal.Length = c(1.462, 4.260, 5.552),
                    Petal.Width = c(0.246, 1.326, 2.026))

# create a vector of columns to apply the mean operation on
cols <- c("Sepal.Length", "Petal.Length")

# use map() to calculate the mean for each column in cols
mean_df <- iris %>%
  group_by(Species) %>%
  mutate(across(cols, mean))

print(mean_df)

The code above calculates the mean of Sepal.Length and Petal.Length columns while ignoring any grouping.

Using lapply() with base R

Another option for applying an operation across multiple columns is to use functions like lapply() from the base R library. This can be a suitable choice when you need more control over your function or want to apply it to multiple data frames:

# create a vector of column names
cols <- c("Sepal.Length", "Petal.Width")

# define the operation using lapply()
df_mean <- do.call(mean, lapply(iris[, cols], as.numeric))

print(df_mean)

Here we apply mean() to each element in the cols vector.

Combining dplyr with lapply() or map()

While summarise_each() provides a powerful way to operate on multiple columns at once, there are situations where using both dplyr and base R functions like lapply() or map() can be beneficial:

# create an iris dataframe
iris <- data.frame(Species = c("setosa", "versicolor", "virginica"),
                    Sepal.Length = c(5.006, 5.936, 6.588),
                    Sepal.Width = c(3.428, 2.770, 2.974),
                    Petal.Length = c(1.462, 4.260, 5.552),
                    Petal.Width = c(0.246, 1.326, 2.026))

# combine dplyr with lapply()
df_mean <- iris %>%
  group_by(Species) %>% 
  summarise(
    Sepal.Length = mean(Sepal.Length),
    Sepal.Width = mean(Sepal.Width),
    Petal.Length = mean(Petal.Length),
    Petal.Width = mean(Petal.Width)
  )

print(df_mean)

# combine dplyr with map()
df_mean <- iris %>%
  group_by(Species) %>% 
  summarise(
    Sepal.Length = map2(.x = c("Sepal.Length", "Petal.Length"), .f = function(x) mean(as.numeric(x))),
    Petal.Width = map2(.x = c("Petal.Length", "Petal.Width"), .f = function(x) mean(as.numeric(x)))
  )

print(df_mean)

The main benefits of combining dplyr with base R functions are the additional control and flexibility you can gain over your code.

Conclusion

While using summarise_each() might seem like a straightforward way to apply an operation on multiple columns at once, understanding its limitations is crucial. Combining this function with other tools from the dplyr package or the base R library offers a more flexible approach and provides better control over data transformation operations.

Choosing the right combination of techniques depends on your specific needs and workflow.

Last modified on 2024-05-27