sed basics

On this page, you use sed to change text: you replace one piece of text with another, delete lines you do not need, and turn a file with another column separator into the tab-separated table. At the end you will use these commands to clean up a file that was saved on Windows and that contains a problem that can quietly break your AWK commands.

Why would you change text with a command, and not by hand? Two jobs come up again and again in biology:

sed does both jobs with one short command, and like AWK, it never changes the file it reads unless you ask for that. You use the same folder as on the previous pages, with the data in awk_data/.

The shape of a sed command

The command you use most often in sed is s, for substitute (replace):

sed 's/old/new/' file
  • s tells sed to replace text.
  • old is the text to look for, and new the text to put in its place. The three slashes / separate the parts.
  • The part between single quotes is the sed command. Like in AWK, it always goes between single quotes.

sed reads the file one line at a time, makes the change on each line where it finds old, and prints every line, changed or not. For example, to shorten low_P to low:

sed 's/low_P/low/' awk_data/Growth_Data.txt | head -3

This prints:

Nutrient    Condition   Experiment  Rootlength  FW_shoot_mg
low mock    ex61    40.4    6.6
low mock    ex61    38  6.8

The file itself is not changed. sed only printed a changed version on the screen:

head -3 awk_data/Growth_Data.txt

This prints:

Nutrient    Condition   Experiment  Rootlength  FW_shoot_mg
low_P   mock    ex61    40.4    6.6
low_P   mock    ex61    38  6.8

The g flag: change every match in a line

Say you want a comma-separated table instead of a tab-separated one. In sed, \t stands for a tab, and you can use this to replace \t with ,:

sed 's/\t/,/' awk_data/Growth_Data.txt | head -3

This prints:

Nutrient,Condition  Experiment  Rootlength  FW_shoot_mg
low_P,mock  ex61    40.4    6.6
low_P,mock  ex61    38  6.8

You see that this command didn’t exactly do what you wanted: only the first tab of each line became a comma. That is because by default, s changes only the first match in a line. To change every match, add the flag g (for “global”) after the last slash. A flag is a letter that changes how a command works:

sed 's/\t/,/g' awk_data/Growth_Data.txt | head -3

This prints:

Nutrient,Condition,Experiment,Rootlength,FW_shoot_mg
low_P,mock,ex61,40.4,6.6
low_P,mock,ex61,38,6.8

To check that this worked on the whole file, not only the first lines, count the fields per line with AWK, as on AWK on a table. Since this time the fields are separated by commas, you use -F',':

sed 's/\t/,/g' awk_data/Growth_Data.txt | awk -F',' '{ print NF }' | sort | uniq -c

This prints:

    595 5

All 595 lines have 5 fields, so every tab was replaced.

QuestionExercise
  1. Replace mock with control, and look at the first 3 lines of the result.
  2. How many lines contain control after the change (use grep for this task)? Before you count, predict roughly: the table has 594 plants and 3 conditions. Afterwards, confirm the number with AWK: how many lines have mock in column 2 in the original file?
# 1. Replace and look
sed 's/mock/control/' awk_data/Growth_Data.txt | head -3

# 2. Count after the change, and check with awk on the original file
sed 's/mock/control/' awk_data/Growth_Data.txt | grep -c 'control'
awk -F'\t' '$2 == "mock"' awk_data/Growth_Data.txt | wc -l

This prints:

Nutrient    Condition   Experiment  Rootlength  FW_shoot_mg
low_P   control ex61    40.4    6.6
low_P   control ex61    38  6.8

209

209

594 / 3 gives about 198, so 209 is close to what you expect. Both commands give 209, so sed changed exactly the 209 mock plants. mock appears at most once per line, so you do not need the g flag here.

Special characters: the dot

The text you look for in sed is not plain text, but a regular expression: a pattern in which some characters have a special meaning. Most letters and numbers just mean themselves. But a dot . means “any one character”. See what happens when you try to replace the decimal points with commas:

sed 's/./,/g' awk_data/Growth_Data.txt | head -2

This prints:

,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,

You see that every character, letters and tabs included, became a comma. To look for a real dot, you need to put a backslash \ before it in order to take away the special meaning from .:

sed 's/\./,/g' awk_data/Growth_Data.txt | head -3

This prints:

Nutrient    Condition   Experiment  Rootlength  FW_shoot_mg
low_P   mock    ex61    40,4    6,6
low_P   mock    ex61    38  6,8

You can do a quick sanity check: the number of lines with a comma after the change should equal the number of lines with a dot before. In grep, the dot is special too, so you need the backslash there as well:

sed 's/\./,/g' awk_data/Growth_Data.txt | grep -c ','
grep -c '\.' awk_data/Growth_Data.txt

This prints:

590
590

Other characters with a special meaning include *, [, ], ^ and $. You meet $ at the end of this page. If the text you want to replace contains one of them, put a backslash before it.

sed sees the whole line, not columns

On AWK on a table, grep '13' found 36 more lines than the plants with bacterium 13C, because 13 also appears in numbers such as 13.3. sed has the same problem: it looks for patterns in the whole line, not only in a single column. To test this, replace 13 with X, then count and look at the lines with an X:

sed 's/13/X/g' awk_data/Growth_Data.txt | grep -c 'X'
sed 's/13/X/g' awk_data/Growth_Data.txt | grep 'X' | head -3

This prints:

211

low_P   mock    ex61    55.3    X.3
low_P   mock    ex61    48.7    X.9
low_P   mock    ex64    46.93   X.52

211 lines changed, but only 175 plants have bacterium 13C. The first lines with an X are not even 13C plants: they are mock plants whose shoot weight 13.3 became X.3, without any warning. Two ways to avoid this:

  • Make the text you look for more specific: 13C appears only in column 2.
  • If the change must happen in one column only, use AWK, which can compare and change a single field.
# More specific text
sed 's/13C/strain13C/' awk_data/Growth_Data.txt | grep -c 'strain13C'

# Only column 2, with AWK
awk -F'\t' -v OFS='\t' '$2 == "13C" { $2 = "strain13C" } { print }' awk_data/Growth_Data.txt | grep -c 'strain13C'

This prints:

175
175

Both change exactly the 175 lines of the 13C plants.

Several changes in one command

To make more than one change, separate the s commands with a semicolon ;. sed runs them one after the other, on each line:

sed 's/low_P/low/; s/high_P/high/' awk_data/Growth_Data.txt | cut -f1 | sort | uniq -c

This prints:

      1 Nutrient
    294 high
    300 low

A file with another column separator

Not every table separates its columns with tabs. The file Growth_Data_dutch.csv holds the same growth data, but with semicolons between the columns and commas as decimal symbol. You may get files like this from a spreadsheet program. That is a key reason to always look at your data first before you do anything with it:

head -3 awk_data/Growth_Data_dutch.csv | cat -A

This prints:

Nutrient;Condition;Experiment;Rootlength;FW_shoot_mg$
low_P;mock;ex61;40,4;6,6$
low_P;mock;ex61;38;6,8$

There is no ^I, so there are no tabs. A file ending in .csv (“comma-separated values”) does not always use commas: always check which character separates the columns before you use the file. Here it is ;, so in AWK you would use -F';':

awk -F';' '{ print NF }' awk_data/Growth_Data_dutch.csv | sort | uniq -c

This prints:

    595 5

If you wanted to turn this file into a tab-separated table with decimal points, you need two changes: commas to dots, and semicolons to tabs.

In the code below, you may wonder why s/,/./g needs no backslash before the dot. The dot is special only in the part you look for, the old part. In the new part, it is just a dot. Here you look for a comma, which has no special meaning, and put a dot in its place.

Save the result in a new file, and check it:

sed 's/,/./g; s/;/\t/g' awk_data/Growth_Data_dutch.csv > Growth_Data_from_csv.txt

# Check: look at it, count the lines and the fields per line
head -3 Growth_Data_from_csv.txt
wc -l Growth_Data_from_csv.txt
awk -F'\t' '{ print NF }' Growth_Data_from_csv.txt | sort | uniq -c

This prints:

Nutrient    Condition   Experiment  Rootlength  FW_shoot_mg
low_P   mock    ex61    40.4    6.6
low_P   mock    ex61    38  6.8

595 Growth_Data_from_csv.txt

    595 5

In this tutorial, you can even check that the result is exactly the original table. cmp compares two files byte by byte. It prints nothing if they are identical, so && echo 'identical' prints a message in that case:

cmp Growth_Data_from_csv.txt awk_data/Growth_Data.txt && echo 'identical'

This prints:

identical

One line of sed turned a file from a Dutch spreadsheet into the table you know, byte for byte. If you get 20 such files, a for loop does them all with the same line.

QuestionExercise
  1. Turn Growth_Data_dutch.csv into a real comma-separated file: commas between the columns, and dots as decimal symbol. Check the result with head and with the number of fields per line.
  2. Now swap the order of the two changes in your command. Before you run it, predict: what will the first line look like, and how many fields will each line have? Why? Then run it and compare.
# 1. First the decimal commas to dots, then the semicolons to commas
sed 's/,/./g; s/;/,/g' awk_data/Growth_Data_dutch.csv | head -3
sed 's/,/./g; s/;/,/g' awk_data/Growth_Data_dutch.csv | awk -F',' '{ print NF }' | sort | uniq -c

# 2. The other order
sed 's/;/,/g; s/,/./g' awk_data/Growth_Data_dutch.csv | head -3
sed 's/;/,/g; s/,/./g' awk_data/Growth_Data_dutch.csv | awk -F',' '{ print NF }' | sort | uniq -c

This prints:

Nutrient,Condition,Experiment,Rootlength,FW_shoot_mg
low_P,mock,ex61,40.4,6.6
low_P,mock,ex61,38,6.8
    595 5

Nutrient.Condition.Experiment.Rootlength.FW_shoot_mg
low_P.mock.ex61.40.4.6.6
low_P.mock.ex61.38.6.8
    595 1

In the second command, sed first turns the semicolons into commas. The second change then turns all commas into dots, the new column separators included. Each line is now one single field. sed runs the changes in the order you write them, and each change works on the result of the one before.

ImportantA comma inside a value breaks -F’,’

A comma-separated file can have a comma inside a value, for example a place name such as Texel, NL. To show that this comma belongs to the value, the program that wrote the file puts the value between double quotes. AWK and sed do not know this rule. They split at every comma. Make a small file and see what happens:

# printf writes the text into a file, \n starts a new line
printf 'sample,location,reads\nS01,Amsterdam,12000\nS02,"Texel, NL",8500\nS03,Utrecht,9100\n' > samples.csv

cat samples.csv
awk -F',' '{ print NF }' samples.csv | sort | uniq -c
awk -F',' 'NR > 1 { sum += $3 } END { print sum }' samples.csv

This prints:

sample,location,reads
S01,Amsterdam,12000
S02,"Texel, NL",8500
S03,Utrecht,9100

      3 3
      1 4

21100

The line of S02 has 4 fields instead of 3. For AWK, its column 3 is now NL", which is not a number and counts as 0. So the total number of reads comes out as 21100 instead of 29600, without an error. sed 's/,/\t/g' has the same problem: it also turns the comma inside the quotes into a tab. The NF check catches it: one line has more fields than the header.

For a file like this, use a tool that knows the quote rule, and turn the file into a tab-separated table first. After that, AWK works as usual. One such tool is csvtk:

csvtk csv2tab samples.csv > samples.tsv

# Check
cat samples.tsv
awk -F'\t' '{ print NF }' samples.tsv | sort | uniq -c
awk -F'\t' 'NR > 1 { sum += $3 } END { print sum }' samples.tsv

This prints:

sample  location    reads
S01 Amsterdam   12000
S02 Texel, NL   8500
S03 Utrecht 9100

      4 3

29600

If you continue in R anyway, read.csv("samples.csv") also reads the quoted value correctly. In code from an LLM, you may see gawk --csv or gawk -v FPAT=... for this job. Not every AWK version knows these options: in WSL with Ubuntu 24.04 (gawk 5.2.1), gawk --csv gave an error (tested November 2026).

Delete lines

Besides s, sed has a command d to delete lines. In front of d, you say which lines you want to delete and you can use a line number, or a pattern between slashes. To delete the header, line 1:

sed '1d' awk_data/Growth_Data.txt | head -2
sed '1d' awk_data/Growth_Data.txt | wc -l

This prints:

low_P   mock    ex61    40.4    6.6
low_P   mock    ex61    38  6.8

594

To delete every line that contains high_P, use a pattern (and compare with AWK):

sed '/high_P/d' awk_data/Growth_Data.txt | wc -l
awk -F'\t' '$1 != "high_P"' awk_data/Growth_Data.txt | wc -l

This prints:

301
301

301 lines: the header and the 300 low_P plants. As with s, sed looks for the pattern in the whole line. Here that is fine, because high_P appears only in column 1.

sed prints every line by default. With -n, it prints nothing, unless you ask for it with the command p (print). This is a quick way to look at one line, or a range of lines, by number:

# Line 123, the first plant without a root length
sed -n '123p' awk_data/Growth_Data.txt

# Lines 2 to 4
sed -n '2,4p' awk_data/Growth_Data.txt

This prints:

high_P  mock    ex61        18.2
low_P   mock    ex61    40.4    6.6
low_P   mock    ex61    38  6.8
low_P   mock    ex61    74.7    8.1

awk 'NR == 123' from AWK on a table does the same.

QuestionExercise
  1. Delete the header and all plants from experiment ex65, with one sed command. How many lines are left? Check the number with AWK.
# Two commands, separated by ;
sed '1d; /ex65/d' awk_data/Growth_Data.txt | wc -l

# Check: rows after the header, where column 3 is not ex65
awk -F'\t' 'NR > 1 && $3 != "ex65"' awk_data/Growth_Data.txt | wc -l

This prints:

399
399

Change a file safely

So far, sed only printed its result, or you saved it in a new file. Sometimes you want to change a file itself, for example to correct a name in a table you keep using. Do this on a copy, never on your original data.

For the rest of this page, you use a second table, mmc5_metadata.txt. It describes 27 bacterial strains, for example their taxonomy and the plant they were isolated from. It comes from a published study, and two of its strains are the bacteria 129E and 13C from the growth data. Later, on the join page, you combine the two tables.

Make a copy, so the file in awk_data/ stays as it is, and look at the strains isolated from legumes. The table has 12 columns, and with cut -f1,6 you only look at the strain name (column 1) and the host plant (column 6):

cp awk_data/mmc5_metadata.txt metadata.txt
grep 'Legume' metadata.txt | cut -f1,6

This prints:

USDA61  Legume
R7A Legume
CIAT899 Legume
NGR234  Legume
C58 Legume

The safe way to change a file has three steps: write the result to a new file, check it, and only then replace the old file with the new one. Say you want Legume_host instead of Legume:

# 1. Write the result to a new file
sed 's/Legume/Legume_host/' metadata.txt > metadata_new.txt

# 2. Check: the same number of lines, and only the lines you wanted changed
wc -l metadata.txt metadata_new.txt
diff metadata.txt metadata_new.txt | cut -f1,6

This prints:

  28 metadata.txt
  28 metadata_new.txt
  56 total

4c4
< USDA61    Legume
---
> USDA61    Legume_host
7c7
< R7A   Legume
---
> R7A   Legume_host
10c10
< CIAT899   Legume
---
> CIAT899   Legume_host
17c17
< NGR234    Legume
---
> NGR234    Legume_host
20c20
< C58   Legume
---
> C58   Legume_host

diff compares two files line by line and prints only the lines that differ. 4c4 means: line 4 was changed. Lines starting with < come from the first file, lines starting with > from the second. diff prints whole lines, which are long in this table, so | cut -f1,6 keeps only the strain and the host. Lines such as 4c4 have no tab, and cut prints them as they are. So exactly five lines changed, the way you wanted. Now replace the old file:

# 3. Replace the old file with the new one
mv metadata_new.txt metadata.txt
grep 'Legume' metadata.txt | cut -f1,6

This prints:

USDA61  Legume_host
R7A Legume_host
CIAT899 Legume_host
NGR234  Legume_host
C58 Legume_host

sed -i: change the file itself

sed can also change a file directly, with the option -i (for “in place”). It does the three steps above in one go, but without the check. This is often where mistakes happen. Say you forgot that you already made the change, and run the command again, this time with -i:

sed -i 's/Legume/Legume_host/' metadata.txt
grep 'Legume' metadata.txt | cut -f1,6

This prints:

USDA61  Legume_host_host
R7A Legume_host_host
CIAT899 Legume_host_host
NGR234  Legume_host_host
C58 Legume_host_host

Legume_host still contains Legume, so sed added _host a second time. sed gives no warning, and the old version of the file is gone. This happens easily when you run a script or a notebook a second time.

If you use -i, let sed keep a backup copy. With -i.bak, sed saves the old version as metadata.txt.bak before it changes the file:

sed -i.bak 's/Legume_host_host/Legume_host/' metadata.txt
ls metadata*
grep 'Legume' metadata.txt | cut -f1,6
grep 'Legume' metadata.txt.bak | cut -f1,6

This prints:

metadata.txt
metadata.txt.bak

USDA61  Legume_host
R7A Legume_host
CIAT899 Legume_host
NGR234  Legume_host
C58 Legume_host

USDA61  Legume_host_host
R7A Legume_host_host
CIAT899 Legume_host_host
NGR234  Legume_host_host
C58 Legume_host_host
ImportantBefore you use sed -i
  • Work on a copy, not on your original data.
  • If you are not sure, write to a new file first and check it with wc -l and diff.
  • Ask yourself what happens if the command runs twice. A change like s/Legume/Legume_host/ grows each time, s/Legume_host_host/Legume_host/ does not.
QuestionExercise
  1. Add _host to every host in column 6 of awk_data/mmc5_metadata.txt, and save the result in a new file. First look at which hosts there are. Do it once with sed, and once with AWK. Tip: on Calculate with AWK, you glued text together with $1"-"$2.
  2. Check both results: count the values in column 6, and compare the two files.
# Which hosts are there?
cut -f6 awk_data/mmc5_metadata.txt | sort | uniq -c

# 1a. sed: one change per host
sed 's/Legume/Legume_host/; s/Arabidopsis/Arabidopsis_host/' awk_data/mmc5_metadata.txt > hosts_sed.txt

# 1b. AWK: glue _host to column 6, on every line after the header
awk -F'\t' -v OFS='\t' 'NR > 1 { $6 = $6 "_host" } { print }' awk_data/mmc5_metadata.txt > hosts_awk.txt

# 2. Check
cut -f6 hosts_sed.txt | sort | uniq -c
cut -f6 hosts_awk.txt | sort | uniq -c
cmp hosts_sed.txt hosts_awk.txt && echo 'identical'

This prints:

     22 Arabidopsis
      5 Legume
      1 host

     22 Arabidopsis_host
      5 Legume_host
      1 host

     22 Arabidopsis_host
      5 Legume_host
      1 host

identical

Both commands give the same file, and most strains turn out to come from Arabidopsis, like the plants in the growth data. The sed version works here because you looked first: there are only two hosts, and these two words appear nowhere else in the table. With many different hosts, you would need one change for each.

The AWK version changes column 6 on every line, whatever is in it. $6 "_host" glues the text _host to the value in column 6. Three mistakes give a wrong result without an error:

  • Without NR > 1, the header becomes host_host.
  • Run the command a second time, on hosts_awk.txt, and you get Arabidopsis_host_host, as with sed -i above.
  • Without the double quotes around _host, AWK reads _host as the name of an empty variable, and nothing changes.

The sed on a Mac is a different version (BSD sed). Two differences matter for this page: sed -i needs a backup ending after it (sed -i '' ... for no backup, or sed -i.bak ...), and \t may not mean a tab in the text you look for. If a command from this page does not work on your Mac, this is a likely reason.

Windows line endings

Every line in a text file ends with an invisible character that says “a new line starts here”. Linux and Mac use one character for this, written as \n. Windows uses two: \r\n. A table saved on Windows, for example from Excel, may therefore have a hidden \r at the end of every line. You do not see it on screen, but AWK and grep on Linux see it as part of the last column and this can create silent errors that you will only catch if you closely inspect the output of your commands.

The metadata table you use below was saved from Excel on a Windows PC. file tells you what kind of text a file holds. Compare it with the growth data:

file awk_data/Growth_Data.txt awk_data/mmc5_metadata.txt

This prints:

awk_data/Growth_Data.txt:   ASCII text
awk_data/mmc5_metadata.txt: ASCII text, with CRLF line terminators

CRLF is the name for the Windows line ending \r\n. Now see what it does. Column 12, the last column, holds TRUE, FALSE or NA. Count these values with the tools from the CLI workshop, and then let AWK count the rows with TRUE:

cut -f12 awk_data/mmc5_metadata.txt | sort | uniq -c
awk -F'\t' '$12 == "TRUE"' awk_data/mmc5_metadata.txt | wc -l

This prints:

      1 Colonization
      3 FALSE
     13 NA
     11 TRUE

0

cut shows 11 rows with TRUE, but AWK finds 0, and there is no error. That happens because column 12 is TRUE\r, not TRUE and therefore the AWK pattern matches nothing. The problem for you is that \r is invisible on screen, also in the output of cut above. You can use cat -A, which shows the Windows line ending as ^M. Since column 12 is the last column you can use cut -f1,12 to have a short view of the strain name and the line end:

head -3 awk_data/mmc5_metadata.txt | cut -f1,12
head -3 awk_data/mmc5_metadata.txt | cut -f1,12 | cat -A

This prints:

strain  Colonization
Root123D2   NA
Root381 NA

strain^IColonization^M$
Root123D2^INA^M$
Root381^INA^M$

Remove the \r with sed

To remove the \r, replace it with nothing. In sed, \r stands for this character, and $ means “the end of the line”. So s/\r$// removes a \r only where it sits at the end of a line:

sed 's/\r$//' awk_data/mmc5_metadata.txt > mmc5_metadata_fixed.txt

# Check: file type, the hidden characters, the number of lines and characters,
# and the count from before
file mmc5_metadata_fixed.txt
head -3 mmc5_metadata_fixed.txt | cut -f1,12 | cat -A
wc -l mmc5_metadata_fixed.txt
wc -c awk_data/mmc5_metadata.txt mmc5_metadata_fixed.txt
awk -F'\t' '$12 == "TRUE"' mmc5_metadata_fixed.txt | wc -l

This prints:

mmc5_metadata_fixed.txt: ASCII text

strain^IColonization$
Root123D2^INA$
Root381^INA$

28 mmc5_metadata_fixed.txt

2808 awk_data/mmc5_metadata.txt
2780 mmc5_metadata_fixed.txt
5588 total

11

The ^M is gone, and AWK finds the 11 rows. wc -c counts characters: the fixed file has 28 lines and 28 characters fewer, one \r per line. Keep mmc5_metadata_fixed.txt, you use it on the next page.

You will often see another command for the same job: tr -d '\r' deletes every \r in a file. tr reads only from a pipe or with <, not from a file name:

tr -d '\r' < awk_data/mmc5_metadata.txt > mmc5_metadata_fixed_tr.txt
cmp mmc5_metadata_fixed_tr.txt mmc5_metadata_fixed.txt && echo 'identical'

This prints:

identical

Both commands give the same result here.

What you learned

  • sed 's/old/new/' file replaces the first match in each line, s/old/new/g every match.
  • The text you look for is a regular expression: . means any character, \. a real dot. \t is a tab, \r the hidden Windows character, $ the end of a line.
  • sed looks at the whole line, not at columns. Make the text you look for specific, or use AWK to change one column.
  • Separate several changes with ;. They run in the order you write them.
  • sed '1d' deletes line 1, sed '/text/d' every line that contains text.
  • Check which character separates the columns before you use a file, also when it ends in .csv.
  • Change a file safely: work on a copy, write to a new file, check with wc -l and diff, then replace. With sed -i, keep a backup (-i.bak), and think about what happens if the command runs twice.
  • Windows line endings (\r, ^M in cat -A) make text matches on the last column fail without an error. Find them with file or cat -A, and remove them with sed 's/\r$//'.

On the next page, More useful AWK, you learn a few more AWK forms that are handy in your own work and common in code from an LLM.