The commands you need

This page covers only the few commands you need for this tutorial. For most of them, you do not need to understand every detail. Copy the commands, and check that your output matches what the page tells you to expect. The exception is moving around: pwd, ls, cd and paths. Take the time to understand these. You use them all the time, and they help you find your way when a command fails. If you want to learn the command line properly, for example to explore your own data in more depth, follow the CLI workshop. Each command on this page also links to the matching section of the workshop, where you find more explanation and exercises.

On this page, you work on your own computer. You download a small set of practice data and try every command on it. On the next page, A first look at your data, you copy the same practice data to Crunchomics and do the full check. After that, you repeat the check with your own data.

The practice data

The practice data comes from an ITS (a DNA region used to identify fungi) amplicon sequencing project from different soil samples. It comes from two real sequencing runs, Run1 and Run2, and it has a real problem that you explore on the next page. The sample IDs are randomized, and every file was reduced to 2% of its reads to keep it small.

You will work with 32 samples that were sequenced with Illumina, a widely used type of sequencing machine. The data is paired-end: the sequencer reads each DNA fragment from both ends. So every sample has two files:

  • ..._R1.fastq.gz: the reads from one end of the fragments (the forward reads).
  • ..._R2.fastq.gz: the reads from the other end (the reverse reads).

In total, that makes 64 files. A .fastq file is a text file that holds every read, with a quality score for each base. The .gz at the end means that the file is compressed to save space.

The practice data also contains seqkit_stats_practice.tsv. This is a table with the number of reads per file, for all samples of the original two runs. You look at it further down this page.

Setting up a terminal

For this tutorial you need set up a terminal: a window in which you type commands (instructions to the computer) as text, instead of clicking on buttons and menus. If you have never opened a terminal, first follow Before you start of the CLI workshop:

  • On Windows, use a local terminal in MobaXterm.
  • On a Mac, use the Terminal app, which you find in Applications, in the Utilities folder. The Mac terminal uses a slightly different shell (zsh), the program that reads your commands. So a few error messages may look a bit different from the ones on this page.

If something goes wrong, see Getting unstuck at the bottom of this page.

The terminal and the prompt

The terminal passes what you type to a shell, the program that reads your commands and tells the computer what to do. When the computer is ready for the next command, it shows the prompt, a line that usually ends with $.

A few things that make life easier:

  • Copy and paste. In many terminals, Ctrl+C does not copy, because Ctrl+C stops a command (see below). Use the right mouse button instead, or Ctrl+Shift+C and Ctrl+Shift+V. This differs between terminals, so try it once.
  • Tab completion. Type the first letters of a file or folder name and press Tab. The terminal completes the name for you. This saves typing and avoids typos.
  • Arrow up. Press the up arrow to get your previous commands back.
  • Comments. Everything after a # is a comment. The computer ignores it. In this tutorial, every command comes with a comment that says what it does. You can paste the comments along with the commands.

Now you are ready to type your first command.

pwd: where am I?

# On your computer: print the folder you are in right now
pwd

When you open a new terminal, you start in your home folder. The path that pwd prints differs between computers. A path is the address of a folder, with folder names separated by /, something like /home/mobaxterm or /Users/yourname. The CLI workshop explains where your home folder is.

ls: what is in this folder?

# On your computer: list the files and folders in the current folder
ls

# The same, with more details: size, date, owner
# The options change how ls behaves:
# -l long list with details, -h sizes in K/M/G
ls -lh

More: ls in the workshop

The parts of a command

Most commands on this page have up to three parts. Take ls -lh data as an example, a command like the ones you use later on this page:

  • The command name is the program you want to run: ls.
  • An option (or flag) changes how the command behaves: -lh. Options usually start with a -. Several one-letter options can be written together, so -lh is the same as -l -h.
  • An argument is what the command works on, often a file or a folder: data. Some commands work without an argument. For example, ls without an argument lists the folder you are in now.

Spaces separate the parts. So always put a space between the command name, the options and the arguments.

When you see a new command on this page, try to find its three parts. The comment above the command tells you what the command does.

More: the structure of a command in the workshop

mkdir: make a folder

ImportantMobaXterm users: did you choose a permanent folder?

MobaXterm can use a temporary home folder. Then it deletes your files when you close it. Before you continue, check this with the steps in Windows + MobaXterm: choose a permanent folder on the “Before you start” page of the CLI workshop.

# On your computer, in your home folder: make a folder for the practice data, name it seq_qc_practice
mkdir seq_qc_practice

# Check if the new folder exists: 
# you should see seq_qc_practice in the list
ls -l

More: mkdir in the workshop

cd: move into a folder

# Move into the seq_qc_practice folder
cd seq_qc_practice

# Check: the path should now end in seq_qc_practice
pwd

A few more ways to use cd. Try them, and then go back into seq_qc_practice:

# Move one folder up
cd ..

# Check where you are now 
pwd

# Move back to your home folder, from anywhere
cd

# Check where you are now 
pwd

# Go back into the practice folder: ~ is short for your home folder
cd ~/seq_qc_practice
pwd

In the beginning it is a good habit to check where you are with pwd after every cd. Most errors in this tutorial come from running a command in the wrong folder.

More: cd in the workshop

Get the practice data: curl and tar

The practice data can be downloaded in one single file, practice_data.tar.gz. A .tar.gz file packs many files into one and compresses them, similar to a .zip file.

# Inside seq_qc_practice: download the practice data
# -L: if the link sends you on to another address, follow it
# -O: save the file under its own name, practice_data.tar.gz
curl -L -O "https://raw.githubusercontent.com/ndombrowski/sequencing_qc/main/data/practice_data.tar.gz"

# Check: you should see practice_data.tar.gz, with a size of about 11M (11 megabytes)
ls -lh

# Unpack the compressed folder. Used options: 
# -x extract, -z it is compressed (.gz), -f the file name follows
tar -xzf practice_data.tar.gz

# Check what changed
# you should see data, practice_data.tar.gz and seqkit_stats_practice.tsv
ls

# Check the inside of the newly generated data folder 
# you should see one folder per sample, with names like Run1_S137 
# (you learn how to count the number of files later)
ls data 

# Look inside one sample folder: you should see the R1 and R2 file of that sample
ls data/Run1_S137

The data folder contains several folders, one per sample. Each sample folder holds the sequencing files of that sample. Many sequencing centers deliver data like this. One good first sanity check is to count the number of folders or files you have per sample. You learn how to do this further down.

Keep practice_data.tar.gz. You copy it to Crunchomics on the next page. Also for your own data it is a good practice to keep a copy of your data in a safe location as a backup.

Paths: going deeper into folders

In the last command, data/Run1_S137 means: the folder Run1_S137 inside the folder data. Each / takes you one folder deeper. A path that does not start with / or ~ begins in the folder you are in now. So ls data/Run1_S137 only works when you are inside seq_qc_practice.

Sequencing centers often deliver data in several layers of folders. To get a feeling for these layers, move into one sample folder and back out again:

# On your computer, inside seq_qc_practice
# Move two folders deeper, into the folder of one sample
cd data/Run1_S137

# Check: the path should now end in seq_qc_practice/data/Run1_S137
pwd

# Check: you should see Run1_S137_R1.fastq.gz and Run1_S137_R2.fastq.gz
ls

# Move two folders up again: each .. is one folder up
cd ../..

# Check: the path should end in seq_qc_practice again
pwd

You do not need to cd into a folder to look at it. From seq_qc_practice, ls data/Run1_S137 shows the same files. In this tutorial, you stay in one folder and write the path to the files instead.

Tab completion helps with long paths. Type ls data/ and press Tab twice. The terminal shows what is inside data. Type the first letters of a folder name, for example Run1_S13, and press Tab again to complete it. On a Mac, one Tab can be enough to show the list. If several names match, Tab completes as far as it can. Press Tab twice to see the options, then type one more letter.

Some folders from a sequencing center have a space in the name, for example Project 1234. The terminal reads a space as the end of a name, because spaces separate the parts of a command. So ls data/Project 1234 looks for two things, data/Project and 1234, and you get No such file or directory.

Put the path in quotes, or use Tab completion. Tab completion adds a \ before the space, which also works:

# Both commands list the folder "Project 1234" inside data
ls "data/Project 1234"
ls data/Project\ 1234

# With a wildcard, keep the * outside the quotes
ls "data/Project 1234"/*.fastq.gz

There is no folder with a space in the practice data, so you cannot try this here.

Wildcards: *

A * in a file name means “any text”. It is called a wildcard. This way of writing file names lets you select many files at once.

# Look for R1 files directly in data: data/*_R1.fastq.gz matches every file in data that ends in _R1.fastq.gz
# You get "No such file or directory": 
# that is because the files are not in the data folder but one level deeper, in the sample folders
ls data/*_R1.fastq.gz

# Look one level deeper: the first * stands for any folder name in data
ls data/*/*_R1.fastq.gz

The last command lists all files in the folder, from data/Run1_S137/Run1_S137_R1.fastq.gz to data/Run2_ITS100/Run2_ITS100_R1.fastq.gz.

More: wildcards in the workshop

Counting: wc -l and the pipe |

The next commands help you check quickly how many files a folder holds.

wc -l counts lines. The pipe | sends the output of one command into the next command. Together, they let you count things, for example the number of files:

# On your computer, inside seq_qc_practice
# Count the fastq.gz files in the sample folders: 
# when ls sends its output into a pipe, it prints one name per line, and wc -l counts the lines
# Expected: 64
ls data/*/*.fastq.gz | wc -l

# Count only the R1 files. Expected: 32
ls data/*/*_R1.fastq.gz | wc -l

You will use this a lot. After each step, you count what you have and compare it with what you expect. For now, the command tells you that you work with 64 files from 32 samples. That is what you expect, so you can continue.

If you ever see fewer samples than what you expect then this often means an incomplete download. To check this, download the file again and repeat the steps from above to see whether this recovers your files.

More: wc and pipes in the workshop

Looking at files: head and less

seqkit_stats_practice.tsv is a table with the number of reads in every file of the original sequencing runs. A .tsv file is a table where tabs separate the columns. You make a table like this yourself on the next page. For now, you learn how to look inside a file.

# On your computer, inside seq_qc_practice
# Show the first 10 lines of the table
head seqkit_stats_practice.tsv

# Count the lines. Expected: 729 (one header line and 728 files)
wc -l seqkit_stats_practice.tsv

# Scroll through the whole table. Use the arrow keys, and press q to quit
# -S keeps long lines on one line, so tables stay readable
less -S seqkit_stats_practice.tsv

In this file you work with 728 files, not 64. That is because the table covers all 364 samples of the original runs, and the practice data is 32 of them.

The first line of seqkit_stats_practice.tsv is the header, with the column names. The fourth column, num_seqs, is the number of reads in each file.

More: head and less in the workshop

Searching and selecting: grep and cut

Sometimes you want to search a file for a word, for example to count how many samples each run has. grep does this: it keeps only the lines that contain a word or a piece of text.

Another useful command is cut, which keeps only some columns of a table. This command can be useful to extract only the relevant columns of a larger table.

Try both on seqkit_stats_practice.tsv:

# On your computer, inside seq_qc_practice
# Show only the lines for sample Run1_S137. Expected: 2 lines, one for R1 and one for R2
grep "Run1_S137" seqkit_stats_practice.tsv

# Count the lines of run 2. Expected: 248 (124 samples, with an R1 and an R2 file each)
grep "Run2_" seqkit_stats_practice.tsv | wc -l

# Keep only column 1 (the file name) and column 4 (the number of reads)
# -f: the columns to keep, separated by a comma
cut -f1,4 seqkit_stats_practice.tsv | head

These commands become very handy once you combine them with a pipe to answer a question about your data. For example: how many reads do the blanks have?

# The lines with Blank in the name, and only the file name and number of reads
# Expected: 8 lines (4 blanks, R1 and R2), with between 41 and 4561 reads
grep "Blank" seqkit_stats_practice.tsv | cut -f1,4

Good to know: grep is case-sensitive: If you had used grep "blank", you would have found nothing, because the file names use Blank. With grep -i, upper and lower case count as the same. You use this on the next page.

More: grep and cut in the workshop

Saving output to a file: >

So far, every command printed its result to the screen. With >, you save the result in a file instead. On the next page, you use this to save your read counts.

ImportantImportant: > overwrites without asking

If the file after > already exists, > replaces its content without a warning. Check the file name before you press Enter.

# On your computer, inside seq_qc_practice
# Save the file names and read numbers of the blanks in a new file, blanks.tsv
# Nothing is printed to the screen: the output goes into the file
grep "Blank" seqkit_stats_practice.tsv | cut -f1,4 > blanks.tsv

# Check: blanks.tsv is there
ls

# Check: 8 lines, the same as on the screen before
wc -l blanks.tsv
head blanks.tsv

More: redirection in the workshop

Tidy up: mv and rmdir

mv moves files. Here, you move all fastq.gz files out of the sample folders, directly into data. The mv command is handy whenever you want to re-organize data for a cleaner folder structure.

ImportantImportant: mv also renames

If the last name in an mv command is not an existing folder, mv renames your file to that name. For example, with a typo, mv sample_R1.fastq.gz dta renames the file to dta. To prevent this, end a folder name with /, as in data/. Then mv stops with an error if the folder does not exist.

# On your computer, inside seq_qc_practice
# Move all fastq.gz files from the sample folders up into the data folder
# -n: never overwrite a file that already exists in data
mv -n data/*/*.fastq.gz data/

# Check: count the fastq.gz files directly in data. Expected: 64
ls data/*.fastq.gz | wc -l

# Check: count the fastq.gz files left in the sample folders. Expected: 0
# You also see "ls: cannot access ... No such file or directory". That is fine: there are no files left to list
ls data/*/*.fastq.gz | wc -l

More: mv in the workshop

The sample folders are now empty. rmdir removes empty folders. It never removes a folder that still contains files, so it is a safe way to clean up:

# On your computer, inside seq_qc_practice
# Remove the empty sample folders: data/*/ matches every folder in data
rmdir data/*/

# Check: look at the first names, and count everything in data
# Expected: only fastq.gz files, 64 in total
ls data | head
ls data | wc -l

If a sample folder still contains a file, rmdir says Directory not empty and keeps that folder. Look inside with ls to see what is left.

Getting unstuck

  • A command does not finish, or the prompt does not come back: press Ctrl+C to stop it.
  • The prompt changes to > (or dquote> on a Mac) and nothing happens: the terminal waits for the rest of a command. This often happens when a quote " is missing, for example when you copied only part of a line. Press Ctrl+C, then copy and paste the whole command again.
  • command not found: the computer does not know this command. Check the spelling. On your own computer, it can also mean that the tool is not installed.
  • Nothing is printed: for many commands, such as mkdir, cd and mv, no output means that the command worked. Use ls or pwd to check the result.
  • An error message: read it. It usually says what went wrong, for example No such file or directory means the file name is wrong, or you are in the wrong folder (check with pwd and ls).

More: error messages and Ctrl+C in the workshop

Using an AI chat tool? You do not need one for this tutorial. Every command is tested, and the boxes tell you what to expect. If you are stuck, use the tips above, or ask Nina. If you still want to ask an AI chat tool, for example to explain an error message, read Using LLMs to understand and check code in the CLI workshop first. At the UvA, use UvA AI Chat, and never paste your sequencing data into it.

Next

Go to A first look at your data. Keep your terminal open, or remember that your practice data is in ~/seq_qc_practice.