So far, we used functions that come with R itself. This is called base R. In this chapter you learn a second way to work with tables: the tidyverse. The tidyverse is a collection of packages that share one idea. Each function does one job, and you combine these small functions step by step into a longer analysis.
Before you start, create a new Quarto document for this chapter in your project folder, for example tidyverse.qmd.
5.1 Why learn a second way?
Let’s start with a question about our growth data: what is the mean root length for each condition, in the plants grown with low phosphate?
Here is one way to answer it in base R. Don’t worry if you don’t understand every detail:
# keep the low phosphate plants that have a root lengthlow_P <- growth_data[growth_data$Nutrient =="low_P"&!is.na(growth_data$Rootlength), ]# calculate the mean root length for each conditiontapply(low_P$Rootlength, low_P$Condition, mean)
129E 13C mock
69.91808 64.19465 47.06848
This works, but it is hard to read. The first line does three things at once: it compares the nutrient, checks for missing values and selects rows, all inside square brackets. You have to read it from the inside out.
Here is the same question answered with the tidyverse:
We get the same numbers. But now the code reads from top to bottom, almost like a sentence: take the growth data, then keep the low phosphate plants, then make groups by condition, then calculate the mean root length for each group. The |> at the end of each line means “and then”. It is called the pipe.
Because each step is one line that does one job, it is easy to change the analysis. Do you want the mean per condition and per experiment? Add one column name:
`summarise()` has grouped output by 'Condition'. You can override using the
`.groups` argument.
You can ignore the message about “grouped output” for now. We explain it later in this chapter.
Do you want the high phosphate plants instead? Change one word in the filter() line. This is what makes the tidyverse so handy: you build an analysis from small blocks, and you can add, remove or swap blocks without rewriting everything.
So why did we learn base R first? Because you will see it everywhere: in older scripts, in other people’s code, in answers on the internet and in the help pages. The things you learned in the previous chapters, like $, [rows, columns], vectors and data types, also stay useful in the tidyverse. From now on, this book uses the tidyverse for working with tables.
5.2 The tidyverse packages
The tidyverse contains several packages, each for one kind of task:
ggplot2 for plotting
dplyr for changing tables: filtering, sorting, adding columns, summarizing
tidyr for tidying tables, for example changing their shape
readr for reading data
tibble for tibbles, a modern version of data frames (you met them in the chapter “First steps in R”)
stringr for working with text
forcats for working with factors
purrr for repeating a task many times
Figure 5.1 shows where these tasks fit in a typical data analysis. You import your data, tidy it, and then go back and forth between transforming, plotting (visualise) and statistics (model) until you understand your data. In this chapter we focus on transforming: selecting, changing and summarizing the data in a table. The plotting chapter covers ggplot2.
Figure 5.1: The steps of a typical data analysis. Source: Wickham and Grolemund, R for Data Science.
You installed the tidyverse in the chapter First steps in R. If you skipped that, type this into the Console:
install.packages("tidyverse")
Then load the tidyverse at the top of your document:
# load the tidyverselibrary(tidyverse)
The tidyverse also has its own functions to read data, in the package readr. Our growth data is a tab-separated file (tsv, for tab-separated values), so we read it with read_tsv():
# read in the growth datagrowth_data <-read_tsv("data/Growth_Data.txt")
Rows: 594 Columns: 5
── Column specification ────────────────────────────────────────────────────────
Delimiter: "\t"
chr (3): Nutrient, Condition, Experiment
dbl (2): Rootlength, FW_shoot_mg
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
read_tsv() tells you which data type it picked for each column: chr (character) for Nutrient, Condition and Experiment, and dbl (double, a number) for the two measurements. Check this message every time you read data. If a column of numbers is read as chr, there is probably some text in it, for example a typo or a comment.
Compared with read.table(), you need fewer arguments: read_tsv() knows that the columns are separated by tabs and that the first line holds the column names. It also reads empty cells as missing values (NA).
read_tsv() gives you a tibble, like read_excel() in the chapter “First steps in R”. When you print a tibble, you only see the first 10 rows, and R tells you how many rows the table has:
growth_data
In this book and in your Quarto document, the tibble is shown as a table with pages. The number of rows is below the table, for example “1-10 of 594 rows”, and you can click through the pages. The data type of each column is shown under its name. In the Console, the first line tells you the number of rows and columns instead, for example # A tibble: 594 × 5.
This is very handy in this chapter. After each step, you see how many rows are left, without printing hundreds of rows.
5.3 Pipes and shortcuts
The pipe |> takes what is on its left and puts it into the function on its right, as the first argument. These two lines do the same:
head(growth_data)growth_data |>head()
With one function, the pipe does not help much. It gets useful when you chain several functions, as in the example above. We will write each step on its own line, with the pipe at the end of the line.
You will also see a second pipe, %>%, in a lot of code. It is the older version, from before R had its own pipe. For what we do in this book, both work the same. We use |>, which needs R 4.1 or newer.
Two keyboard shortcuts make your life easier:
Ctrl+Shift+M (Cmd+Shift+M on Mac) types a pipe. By default, RStudio types %>%. To get |>, go to Tools > Global Options > Code and tick “Use native pipe operator”.
Tab completes what you are typing. Type filt and press Tab to see all functions that start with these letters.
5.4 A first look at the data
You already know head() and str(). A third function, summary(), gives you a quick overview of each column:
summary(growth_data)
Nutrient Condition Experiment Rootlength
Length:594 Length:594 Length:594 Min. : 28.73
Class :character Class :character Class :character 1st Qu.: 56.60
Mode :character Mode :character Mode :character Median : 75.64
Mean : 75.19
3rd Qu.: 92.25
Max. :135.49
NA's :15
FW_shoot_mg
Min. : 1.56
1st Qu.:12.10
Median :15.20
Mean :16.58
3rd Qu.:19.63
Max. :47.60
NA's :8
For the text columns, summary() only tells you how many values there are. For the numeric columns, it shows the smallest and largest value, the mean, the median and the quartiles. It also counts the missing values (NA's): 15 plants have no root length, and 8 plants have no shoot weight. We come back to these later.
5.5 Filter rows with filter()
filter() keeps the rows that meet a condition (Figure 5.2).
Figure 5.2: filter() keeps the rows that meet a condition. Source: RStudio cheatsheet Data transformation with dplyr.
For example, to keep only the plants grown with low phosphate:
growth_data |>filter(Nutrient =="low_P")
== means “is equal to”. Note that it has two equal signs: a single = is used for arguments. Text, like "low_P", always goes in quotes.
The result has 300 rows instead of 594. But growth_data itself has not changed: filter() only shows the result. If you want to keep it, store it in a new object:
# store the low phosphate plants in a new objectlow_P_data <- growth_data |>filter(Nutrient =="low_P")# check the number of rowsnrow(low_P_data)
[1] 300
Here is the same filter in base R, which you will see in other people’s code:
growth_data[growth_data$Nutrient =="low_P", ]
You need to write growth_data twice, plus the brackets and the comma. In filter(), you just write the column name.
These are the comparison operators you can use in filter():
Operator
Meaning
==
equal to
!=
not equal to
<
less than
<=
less than or equal to
>
greater than
>=
greater than or equal to
!
not
&
and
|
or
For example, by using != "mock" we can remove all mock plants, so we keep only the plants with bacteria:
growth_data |>filter(Condition !="mock")
And > works on numbers. We could for example only list plants that have roots longer than 100 mm:
growth_data |>filter(Rootlength >100)
5.5.1 Filter on more than one column
You can give filter() several conditions, separated by a comma. A row is kept only if it meets all conditions:
# plants grown with low phosphate and with strain 129Egrowth_data |>filter(Nutrient =="low_P", Condition =="129E")
5.5.2 Filter for several values with %in%
What if you want the plants of both strains, 129E and 13C? One way is the “or” operator |:
Here we added nrow() at the end of the pipe, to count the rows instead of printing the whole table.
With many values, this means a lot of typing. %in% is shorter. It checks whether a value is in a vector:
# the conditions we want to keepstrains <-c("129E", "13C")growth_data |>filter(Condition %in% strains) |>nrow()
[1] 385
A vector like strains is easy to read, easy to extend and you can use it again later.
5.5.3 Exercise
How many plants were grown in experiment ex64?
How many plants grown with high phosphate have a shoot weight of 20 mg or more?
TipSolution
# 1. plants in experiment ex64growth_data |>filter(Experiment =="ex64") |>nrow()
[1] 205
# 2. high phosphate plants with a shoot weight of at least 20 mggrowth_data |>filter(Nutrient =="high_P", FW_shoot_mg >=20) |>nrow()
[1] 137
There are 205 plants from experiment ex64, and 137 high phosphate plants with a shoot weight of at least 20 mg.
5.6 Select columns with select()
filter() chooses rows, select() chooses columns. You list the columns you want to keep:
growth_data |>select(Condition, Rootlength)
With ! you remove columns instead. To remove more than one, put them in c():
# keep all columns except Experiment and FW_shoot_mggrowth_data |>select(!c(Experiment, FW_shoot_mg))
Selection helpers select columns by a pattern in their name:
starts_with(): the name starts with some text
ends_with(): the name ends with some text
contains(): the name contains some text
# all columns whose name starts with FWgrowth_data |>select(starts_with("FW"))
In our small table, this does not save much typing. But many tables in biology have hundreds of columns, for example one per sample or per gene. Then helpers like these are very useful.
5.6.1 Exercise
Select the columns Nutrient and Condition and all columns whose name contains length.
distinct() keeps each value, or each combination of values, only once. It is the tidyverse version of unique(), which you used in “First steps in R”:
# each combination of nutrient and condition only oncegrowth_data |>distinct(Nutrient, Condition)
5.8 Count rows with count()
count() counts the rows for each value, or each combination of values. How many plants do we have per treatment?
growth_data |>count(Nutrient, Condition)
The column n holds the number of plants. Not all groups have the same size: we have between 87 and 107 plants per group.
5.8.1 Exercise
How many plants were measured in each experiment?
TipSolution
growth_data |>count(Experiment)
The three experiments have 194, 205 and 195 plants.
5.9 Sort rows with arrange()
arrange() sorts the rows by the values in a column (Figure 5.3). By default, it starts with the smallest value:
Figure 5.3: arrange() sorts the rows by the values in a column. Source: RStudio cheatsheet Data transformation with dplyr.
growth_data |>arrange(Rootlength)
With desc() (short for descending) you start with the largest value. This is a quick way to find extreme values, for example to spot typos in your data:
growth_data |>arrange(desc(Rootlength))
The three plants with the longest roots were all grown with high phosphate and with one of the two strains.
5.9.1 Exercise
Which plant has the highest shoot weight? In which nutrient, condition and experiment was it grown?
TipSolution
growth_data |>arrange(desc(FW_shoot_mg))
The heaviest shoot weighs 47.6 mg. It is a mock plant, grown with high phosphate in experiment ex64.
5.10 Add or change columns with mutate()
mutate() adds new columns, or changes existing ones (Figure 5.4).
Figure 5.4: mutate() adds new columns or changes existing ones. Source: RStudio cheatsheet Data transformation with dplyr.
For example, we can convert the shoot weight from mg to g. Before the =, you write the name of the new column. After the =, you say how to calculate it. The new column is added at the end of the table:
# convert mg to g (1 g = 1000 mg)growth_data |>mutate(FW_shoot_g = FW_shoot_mg /1000)
In base R, you would write growth_data$FW_shoot_g <- growth_data$FW_shoot_mg / 1000, as in the chapter “Data objects”.
You can also calculate with several columns, and make several new columns in one mutate(), separated by commas:
Any calculation with a missing value gives a missing value. Rows 32 and 33 have no shoot weight, so their ratio is NA too. slice() shows rows by their number:
# 1 cm = 10 mmgrowth_data |>mutate(Rootlength_cm = Rootlength /10)
5.11 Combine several steps
The real strength of the tidyverse shows when you chain several steps. Let’s find the three plants with the longest roots among the low phosphate plants with strain 129E, and only show the columns we need:
Read it line by line: take the growth data, keep the low phosphate plants with 129E, sort them from the longest to the shortest root, keep two columns, show the first three rows.
When you build a pipe like this, add one step at a time and run the code after each step. Then you see what each step does, and you find mistakes quickly.
5.11.1 Exercise
What is the heaviest shoot weight, in g, of a mock plant grown with high phosphate? From which experiment does that plant come?
The heaviest shoot weighs 0.0476 g and comes from experiment ex64.
WarningFiltering and factor levels
Remember factors from the chapter “Data objects”? A factor keeps all its levels, even when you filter away all rows of a level. Let’s turn Condition into a factor with mutate(), and then keep only the strain 129E:
13C and mock are still levels, although no row has them anymore. In plots and statistics, these empty levels can show up as empty groups. droplevels() removes them:
summary() showed us that our data has missing values: 15 in Rootlength and 8 in FW_shoot_mg. Many calculations give NA if one of the values is missing. What you do about it depends on your question. Here we look at two ways to remove rows with missing values.
drop_na() removes every row that has a missing value in any column:
growth_data |>drop_na() |>nrow()
[1] 571
filter() together with is.na() removes only the rows with a missing value in one column. is.na() is TRUE for a missing value, and ! turns it around, so we keep the rows that are not missing:
growth_data |>filter(!is.na(Rootlength)) |>nrow()
[1] 579
drop_na() removes 23 rows, filter() only 15. None of our plants misses both measurements. So drop_na() also throws away 8 plants that do have a root length, only because their shoot weight is missing. If you analyse the root length, this is good data lost. Only remove the missing values you need to remove.
You can also give drop_na() the name of a column. Then it only looks at that column, and drop_na(Rootlength) does the same as our filter().
5.13 Summarize data with group_by() and summarize()
summarize() turns many rows into one row, for example by calculating a mean (Figure 5.5).
Figure 5.5: summarize() turns many rows into one. Source: RStudio cheatsheet Data transformation with dplyr.
na.rm = TRUE tells mean() to ignore the missing values. Without it, the mean would be NA.
One mean for all plants is not very useful, because our plants got different treatments. This is where group_by() comes in. It splits the table into groups, and summarize() then calculates one row for each group (Figure 5.6).
Figure 5.6: group_by() before summarize() turns each group into one row. Source: RStudio cheatsheet Data transformation with dplyr.
Let’s calculate the mean and the standard deviation of the root length for each treatment, and count the plants. You can calculate several things in one summarize(), separated by commas:
`summarise()` has grouped output by 'Nutrient'. You can override using the
`.groups` argument.
To count the plants, we use sum(!is.na(Rootlength)). !is.na() is TRUE for each plant that has a root length, and sum() counts the TRUE values. This way, we only count the plants that were used for the mean. Compare the numbers with the output of count() above: in some groups, a few plants have no root length.
Functions that are useful inside summarize():
Function
Calculates
mean(), median()
the center of the data
sd(), IQR()
the spread of the data
min(), max()
the smallest and largest value
n()
the number of rows in the group
NoteWhy ungroup()?
group_by() does not change how the table looks, but it remembers the groups. After a summarize() with two grouping columns, the result is still grouped by the first one. R tells you this with a message like “summarise() has grouped output by ‘Nutrient’”. All functions that come after it in the pipe then work per group, which can give surprising results. ungroup() removes the groups. Make it a habit to end with ungroup(), especially when you store the result in an object.
5.13.1 Exercise
Calculate the mean and the median shoot weight for each combination of nutrient and condition.
`summarise()` has grouped output by 'Nutrient'. You can override using the
`.groups` argument.
TipOptional: the same calculation for several columns with across()
If you want the mean of both the root length and the shoot weight, you can write two lines in summarize(). With many columns, across() saves typing. It applies the same function to several columns:
`summarise()` has grouped output by 'Nutrient'. You can override using the
`.groups` argument.
The ~ starts a small function, and .x stands for each column in turn. So ~ mean(.x, na.rm = TRUE) means “calculate the mean of this column, without the missing values”. The new columns keep the names of the old ones.
5.14 Combine tables with left_join()
In “First steps in R”, you combined two tables with merge(). The tidyverse version is left_join(). Let’s make a small table that says, for each condition, whether the plants got bacteria:
Now we add this information to our growth data. The argument by says which column to use for matching the rows:
growth_with_info <- growth_data |>left_join(inoculation, by ="Condition")head(growth_with_info)
As with merge(), always check the number of rows after a join. We still want 594 rows, one per plant:
nrow(growth_data)
[1] 594
nrow(growth_with_info)
[1] 594
Two differences to merge() with its default settings:
left_join() keeps all rows of the first table (the one on the left), also when they have no match in the second table. The new columns are then NA. merge() drops these rows, unless you add all.x = TRUE.
left_join() keeps the order of the rows of the first table. merge() sorts the rows by the by columns.
WarningMatching on too few columns
The mistake from “First steps in R” can happen here too. Let’s make a mapping file with distinct() and join on Condition only, although each condition appears twice in the mapping file (once per nutrient):
Warning in left_join(growth_data, mapping_file, by = "Condition"): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 1 of `x` matches multiple rows in `y`.
ℹ Row 1 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
"many-to-many"` to silence this warning.
nrow(wrong_join)
[1] 1188
Again, each plant matches two rows and is copied. Unlike merge(), left_join() warns you about this: it reports a “many-to-many relationship”. Read warnings like this carefully. The fix is the same as before: match on both columns with by = c("Nutrient", "Condition").
5.15 Other useful functions
Some more tidyverse functions that you will need sooner or later. The links go to their help pages, with examples:
pivot_longer() and pivot_wider() change the shape of a table, from wide (one column per sample) to long (one row per measurement) and back
separate_wider_delim() splits one column into several, for example low_P_129E into low_P and 129E
unite() does the opposite and pastes several columns into one
5.16 Summary
In this chapter you learned how to:
combine steps with the pipe |>
get an overview of a table with summary()
keep rows with filter(), using comparison operators and %in%
keep or remove columns with select()
remove duplicates with distinct() and count rows with count()
sort rows with arrange() and desc()
add or change columns with mutate()
remove missing values with drop_na() or filter(!is.na())
summarize groups with group_by(), summarize() and ungroup()
combine tables with left_join()
5.17 Further reading
An introduction into the tidyverse: a tutorial with more examples and exercises for the functions in this chapter. It uses a different data set, measurements of penguins, which is good practice for using the same functions on new data.