Welcome page

Research projects often generate data that is too big for a spreadsheet, too slow for a laptop, or spread over hundreds of files you don’t want to handle one by one. The command line lets you work with large files and automate repetitive steps, and an HPC lets you run analyses that are too slow for a laptop. This hands-on workshop teaches you how to use these tools and is aimed at researchers who have never typed a command before. We use DNA sequencing data as the running example, but the skills apply to any project with many files or large text-based data.

You will first learn how to set up a terminal, a tool that lets you type commands for your computer to run. Then the actual tutorial begins. In it, you take the role of a researcher who received data from a sequencing center, and who wants to start analysing this data on the command line. While you work, you keep a notebook in which you document what you did and why. The tutorial has two main parts:

1. In the first part, you will learn how to use bash on your own computer: you learn how to organize, inspect and extract information from your data.
2. In the second part, you will learn how to work with your data on an HPC: you learn how to connect to the HPC, copy your sequencing data to it, install software and run jobs that check and clean your sequencing data.

I wrote this tutorial for researchers at the Institute for Biodiversity and Ecosystem Dynamics (IBED) at the University of Amsterdam, who can get access to Crunchomics, the Genomics Compute Environment for SILS and IBED. If you don’t have access to Crunchomics or another HPC that uses SLURM, you can still follow most of the tutorial on your own computer.

Since AI chat tools can write bash code for you, you might wonder if it is still worth learning how to use the command line. I strongly believe it is, because the analysis stays yours, whoever writes the code. You need to read the code, understand what it does to your data, and look at your files to see whether the result makes sense. This tutorial teaches you how. At the end, the page Using LLMs describes how I suggest you use these tools once you know the basics.

For topics that this tutorial does not cover, have a look at IBED’s bioinformatics guidance page. To work with tables and sequence files after this workshop, you could continue with my AWK tutorial, which teaches awk and sed.