Welcome page

Research projects often generate data that is too big for a spreadsheet, too slow for a laptop, or spread over hundreds of files you don’t want to handle one by one. The command line lets you work with large files and automate repetitive steps, and an HPC lets you run analyses that are too slow for a laptop. This hands-on workshop teaches you how to use these tools and is aimed at researchers who have never typed a command before. We use DNA sequencing data as the running example, but the skills apply to any project with many files or large text-based data.

You will first learn how to set up a terminal, a tool that lets you type commands for your computer to run. Then the actual tutorial begins. In it, you take the role of a researcher who received data from a sequencing center, and who wants to start analysing this data on the command line. While you work, you keep a notebook in which you document what you did and why. The tutorial has two main parts:

  1. In the first part, you will learn how to use bash on your own computer: you learn how to organize, inspect and extract information from your data.
  2. In the second part, you will learn how to work with your data on an HPC: you learn how to connect to the HPC, copy your sequencing data to it, install software and run jobs that check and clean your sequencing data.

I wrote this tutorial for researchers at the Institute for Biodiversity and Ecosystem Dynamics (IBED) at the University of Amsterdam, who can get access to Crunchomics, the Genomics Compute Environment for SILS and IBED. If you don’t have access to Crunchomics or another HPC that uses SLURM, you can still follow most of the tutorial on your own computer.

AI chat tools such as ChatGPT can write bash code for you, so why learn it yourself? Someone still has to read the code, understand what the code does for your data, run the code, and verify the output, and that is what this tutorial teaches. At the end, the page Using LLMs describes how I suggest you use these tools once you know the basics.

For topics that this tutorial does not cover, have a look at IBED’s bioinformatics guidance page.