sed 's/low_P/low/' awk_data/Growth_Data.txt | head -3sed basics
On this page, you use sed to change text: you replace one piece of text with another, delete lines you do not need, and turn a file with another column separator into the tab-separated table. At the end you will use these commands to clean up a file that was saved on Windows and that contains a problem that can quietly break your AWK commands.
Why would you change text with a command, and not by hand? Two jobs come up again and again in biology:
- Making names match. One table calls a sample
Sample-01, anotherSample_01. Before you can combine the two tables, in AWK or in R, the names must be written the same way. - Cleaning a file before another tool reads it. A table may use commas as decimal symbol, have extra lines at the top, or come with Windows line endings. Some tools stop with an unclear error message on such a file. Others read it without complaint, but wrongly.
sed does both jobs with one short command, and like AWK, it never changes the file it reads unless you ask for that. You use the same folder as on the previous pages, with the data in awk_data/.
The shape of a sed command
The command you use most often in sed is s, for substitute (replace):
sed 's/old/new/' file
stells sed to replace text.oldis the text to look for, andnewthe text to put in its place. The three slashes/separate the parts.- The part between single quotes is the sed command. Like in AWK, it always goes between single quotes.
sed reads the file one line at a time, makes the change on each line where it finds old, and prints every line, changed or not. For example, to shorten low_P to low:
This prints:
Nutrient Condition Experiment Rootlength FW_shoot_mg
low mock ex61 40.4 6.6
low mock ex61 38 6.8
The file itself is not changed. sed only printed a changed version on the screen:
head -3 awk_data/Growth_Data.txtThis prints:
Nutrient Condition Experiment Rootlength FW_shoot_mg
low_P mock ex61 40.4 6.6
low_P mock ex61 38 6.8
The g flag: change every match in a line
Say you want a comma-separated table instead of a tab-separated one. In sed, \t stands for a tab, and you can use this to replace \t with ,:
sed 's/\t/,/' awk_data/Growth_Data.txt | head -3This prints:
Nutrient,Condition Experiment Rootlength FW_shoot_mg
low_P,mock ex61 40.4 6.6
low_P,mock ex61 38 6.8
You see that this command didn’t exactly do what you wanted: only the first tab of each line became a comma. That is because by default, s changes only the first match in a line. To change every match, add the flag g (for “global”) after the last slash. A flag is a letter that changes how a command works:
sed 's/\t/,/g' awk_data/Growth_Data.txt | head -3This prints:
Nutrient,Condition,Experiment,Rootlength,FW_shoot_mg
low_P,mock,ex61,40.4,6.6
low_P,mock,ex61,38,6.8
To check that this worked on the whole file, not only the first lines, count the fields per line with AWK, as on AWK on a table. Since this time the fields are separated by commas, you use -F',':
sed 's/\t/,/g' awk_data/Growth_Data.txt | awk -F',' '{ print NF }' | sort | uniq -cThis prints:
595 5
All 595 lines have 5 fields, so every tab was replaced.
- Replace
mockwithcontrol, and look at the first 3 lines of the result. - How many lines contain
controlafter the change (usegrepfor this task)? Before you count, predict roughly: the table has 594 plants and 3 conditions. Afterwards, confirm the number with AWK: how many lines havemockin column 2 in the original file?
# 1. Replace and look
sed 's/mock/control/' awk_data/Growth_Data.txt | head -3
# 2. Count after the change, and check with awk on the original file
sed 's/mock/control/' awk_data/Growth_Data.txt | grep -c 'control'
awk -F'\t' '$2 == "mock"' awk_data/Growth_Data.txt | wc -lThis prints:
Nutrient Condition Experiment Rootlength FW_shoot_mg
low_P control ex61 40.4 6.6
low_P control ex61 38 6.8
209
209
594 / 3 gives about 198, so 209 is close to what you expect. Both commands give 209, so sed changed exactly the 209 mock plants. mock appears at most once per line, so you do not need the g flag here.
Special characters: the dot
The text you look for in sed is not plain text, but a regular expression: a pattern in which some characters have a special meaning. Most letters and numbers just mean themselves. But a dot . means “any one character”. See what happens when you try to replace the decimal points with commas:
sed 's/./,/g' awk_data/Growth_Data.txt | head -2This prints:
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,
You see that every character, letters and tabs included, became a comma. To look for a real dot, you need to put a backslash \ before it in order to take away the special meaning from .:
sed 's/\./,/g' awk_data/Growth_Data.txt | head -3This prints:
Nutrient Condition Experiment Rootlength FW_shoot_mg
low_P mock ex61 40,4 6,6
low_P mock ex61 38 6,8
You can do a quick sanity check: the number of lines with a comma after the change should equal the number of lines with a dot before. In grep, the dot is special too, so you need the backslash there as well:
sed 's/\./,/g' awk_data/Growth_Data.txt | grep -c ','
grep -c '\.' awk_data/Growth_Data.txtThis prints:
590
590
Other characters with a special meaning include *, [, ], ^ and $. You meet $ at the end of this page. If the text you want to replace contains one of them, put a backslash before it.
sed sees the whole line, not columns
On AWK on a table, grep '13' found 36 more lines than the plants with bacterium 13C, because 13 also appears in numbers such as 13.3. sed has the same problem: it looks for patterns in the whole line, not only in a single column. To test this, replace 13 with X, then count and look at the lines with an X:
sed 's/13/X/g' awk_data/Growth_Data.txt | grep -c 'X'
sed 's/13/X/g' awk_data/Growth_Data.txt | grep 'X' | head -3This prints:
211
low_P mock ex61 55.3 X.3
low_P mock ex61 48.7 X.9
low_P mock ex64 46.93 X.52
211 lines changed, but only 175 plants have bacterium 13C. The first lines with an X are not even 13C plants: they are mock plants whose shoot weight 13.3 became X.3, without any warning. Two ways to avoid this:
- Make the text you look for more specific:
13Cappears only in column 2. - If the change must happen in one column only, use AWK, which can compare and change a single field.
# More specific text
sed 's/13C/strain13C/' awk_data/Growth_Data.txt | grep -c 'strain13C'
# Only column 2, with AWK
awk -F'\t' -v OFS='\t' '$2 == "13C" { $2 = "strain13C" } { print }' awk_data/Growth_Data.txt | grep -c 'strain13C'This prints:
175
175
Both change exactly the 175 lines of the 13C plants.
Several changes in one command
To make more than one change, separate the s commands with a semicolon ;. sed runs them one after the other, on each line:
sed 's/low_P/low/; s/high_P/high/' awk_data/Growth_Data.txt | cut -f1 | sort | uniq -cThis prints:
1 Nutrient
294 high
300 low
A file with another column separator
Not every table separates its columns with tabs. The file Growth_Data_dutch.csv holds the same growth data, but with semicolons between the columns and commas as decimal symbol. You may get files like this from a spreadsheet program. That is a key reason to always look at your data first before you do anything with it:
head -3 awk_data/Growth_Data_dutch.csv | cat -AThis prints:
Nutrient;Condition;Experiment;Rootlength;FW_shoot_mg$
low_P;mock;ex61;40,4;6,6$
low_P;mock;ex61;38;6,8$
There is no ^I, so there are no tabs. A file ending in .csv (“comma-separated values”) does not always use commas: always check which character separates the columns before you use the file. Here it is ;, so in AWK you would use -F';':
awk -F';' '{ print NF }' awk_data/Growth_Data_dutch.csv | sort | uniq -cThis prints:
595 5
If you wanted to turn this file into a tab-separated table with decimal points, you need two changes: commas to dots, and semicolons to tabs.
In the code below, you may wonder why s/,/./g needs no backslash before the dot. The dot is special only in the part you look for, the old part. In the new part, it is just a dot. Here you look for a comma, which has no special meaning, and put a dot in its place.
Save the result in a new file, and check it:
sed 's/,/./g; s/;/\t/g' awk_data/Growth_Data_dutch.csv > Growth_Data_from_csv.txt
# Check: look at it, count the lines and the fields per line
head -3 Growth_Data_from_csv.txt
wc -l Growth_Data_from_csv.txt
awk -F'\t' '{ print NF }' Growth_Data_from_csv.txt | sort | uniq -cThis prints:
Nutrient Condition Experiment Rootlength FW_shoot_mg
low_P mock ex61 40.4 6.6
low_P mock ex61 38 6.8
595 Growth_Data_from_csv.txt
595 5
In this tutorial, you can even check that the result is exactly the original table. cmp compares two files byte by byte. It prints nothing if they are identical, so && echo 'identical' prints a message in that case:
cmp Growth_Data_from_csv.txt awk_data/Growth_Data.txt && echo 'identical'This prints:
identical
One line of sed turned a file from a Dutch spreadsheet into the table you know, byte for byte. If you get 20 such files, a for loop does them all with the same line.
- Turn
Growth_Data_dutch.csvinto a real comma-separated file: commas between the columns, and dots as decimal symbol. Check the result withheadand with the number of fields per line. - Now swap the order of the two changes in your command. Before you run it, predict: what will the first line look like, and how many fields will each line have? Why? Then run it and compare.
# 1. First the decimal commas to dots, then the semicolons to commas
sed 's/,/./g; s/;/,/g' awk_data/Growth_Data_dutch.csv | head -3
sed 's/,/./g; s/;/,/g' awk_data/Growth_Data_dutch.csv | awk -F',' '{ print NF }' | sort | uniq -c
# 2. The other order
sed 's/;/,/g; s/,/./g' awk_data/Growth_Data_dutch.csv | head -3
sed 's/;/,/g; s/,/./g' awk_data/Growth_Data_dutch.csv | awk -F',' '{ print NF }' | sort | uniq -cThis prints:
Nutrient,Condition,Experiment,Rootlength,FW_shoot_mg
low_P,mock,ex61,40.4,6.6
low_P,mock,ex61,38,6.8
595 5
Nutrient.Condition.Experiment.Rootlength.FW_shoot_mg
low_P.mock.ex61.40.4.6.6
low_P.mock.ex61.38.6.8
595 1
In the second command, sed first turns the semicolons into commas. The second change then turns all commas into dots, the new column separators included. Each line is now one single field. sed runs the changes in the order you write them, and each change works on the result of the one before.
A comma-separated file can have a comma inside a value, for example a place name such as Texel, NL. To show that this comma belongs to the value, the program that wrote the file puts the value between double quotes. AWK and sed do not know this rule. They split at every comma. Make a small file and see what happens:
# printf writes the text into a file, \n starts a new line
printf 'sample,location,reads\nS01,Amsterdam,12000\nS02,"Texel, NL",8500\nS03,Utrecht,9100\n' > samples.csv
cat samples.csv
awk -F',' '{ print NF }' samples.csv | sort | uniq -c
awk -F',' 'NR > 1 { sum += $3 } END { print sum }' samples.csvThis prints:
sample,location,reads
S01,Amsterdam,12000
S02,"Texel, NL",8500
S03,Utrecht,9100
3 3
1 4
21100
The line of S02 has 4 fields instead of 3. For AWK, its column 3 is now NL", which is not a number and counts as 0. So the total number of reads comes out as 21100 instead of 29600, without an error. sed 's/,/\t/g' has the same problem: it also turns the comma inside the quotes into a tab. The NF check catches it: one line has more fields than the header.
For a file like this, use a tool that knows the quote rule, and turn the file into a tab-separated table first. After that, AWK works as usual. One such tool is csvtk:
csvtk csv2tab samples.csv > samples.tsv
# Check
cat samples.tsv
awk -F'\t' '{ print NF }' samples.tsv | sort | uniq -c
awk -F'\t' 'NR > 1 { sum += $3 } END { print sum }' samples.tsvThis prints:
sample location reads
S01 Amsterdam 12000
S02 Texel, NL 8500
S03 Utrecht 9100
4 3
29600
If you continue in R anyway, read.csv("samples.csv") also reads the quoted value correctly. In code from an LLM, you may see gawk --csv or gawk -v FPAT=... for this job. Not every AWK version knows these options: in WSL with Ubuntu 24.04 (gawk 5.2.1), gawk --csv gave an error (tested November 2026).
Delete lines
Besides s, sed has a command d to delete lines. In front of d, you say which lines you want to delete and you can use a line number, or a pattern between slashes. To delete the header, line 1:
sed '1d' awk_data/Growth_Data.txt | head -2
sed '1d' awk_data/Growth_Data.txt | wc -lThis prints:
low_P mock ex61 40.4 6.6
low_P mock ex61 38 6.8
594
To delete every line that contains high_P, use a pattern (and compare with AWK):
sed '/high_P/d' awk_data/Growth_Data.txt | wc -l
awk -F'\t' '$1 != "high_P"' awk_data/Growth_Data.txt | wc -lThis prints:
301
301
301 lines: the header and the 300 low_P plants. As with s, sed looks for the pattern in the whole line. Here that is fine, because high_P appears only in column 1.
sed prints every line by default. With -n, it prints nothing, unless you ask for it with the command p (print). This is a quick way to look at one line, or a range of lines, by number:
# Line 123, the first plant without a root length
sed -n '123p' awk_data/Growth_Data.txt
# Lines 2 to 4
sed -n '2,4p' awk_data/Growth_Data.txtThis prints:
high_P mock ex61 18.2
low_P mock ex61 40.4 6.6
low_P mock ex61 38 6.8
low_P mock ex61 74.7 8.1
awk 'NR == 123' from AWK on a table does the same.
- Delete the header and all plants from experiment
ex65, with one sed command. How many lines are left? Check the number with AWK.
# Two commands, separated by ;
sed '1d; /ex65/d' awk_data/Growth_Data.txt | wc -l
# Check: rows after the header, where column 3 is not ex65
awk -F'\t' 'NR > 1 && $3 != "ex65"' awk_data/Growth_Data.txt | wc -lThis prints:
399
399
Change a file safely
So far, sed only printed its result, or you saved it in a new file. Sometimes you want to change a file itself, for example to correct a name in a table you keep using. Do this on a copy, never on your original data.
For the rest of this page, you use a second table, mmc5_metadata.txt. It describes 27 bacterial strains, for example their taxonomy and the plant they were isolated from. It comes from a published study, and two of its strains are the bacteria 129E and 13C from the growth data. Later, on the join page, you combine the two tables.
Make a copy, so the file in awk_data/ stays as it is, and look at the strains isolated from legumes. The table has 12 columns, and with cut -f1,6 you only look at the strain name (column 1) and the host plant (column 6):
cp awk_data/mmc5_metadata.txt metadata.txt
grep 'Legume' metadata.txt | cut -f1,6This prints:
USDA61 Legume
R7A Legume
CIAT899 Legume
NGR234 Legume
C58 Legume
The safe way to change a file has three steps: write the result to a new file, check it, and only then replace the old file with the new one. Say you want Legume_host instead of Legume:
# 1. Write the result to a new file
sed 's/Legume/Legume_host/' metadata.txt > metadata_new.txt
# 2. Check: the same number of lines, and only the lines you wanted changed
wc -l metadata.txt metadata_new.txt
diff metadata.txt metadata_new.txt | cut -f1,6This prints:
28 metadata.txt
28 metadata_new.txt
56 total
4c4
< USDA61 Legume
---
> USDA61 Legume_host
7c7
< R7A Legume
---
> R7A Legume_host
10c10
< CIAT899 Legume
---
> CIAT899 Legume_host
17c17
< NGR234 Legume
---
> NGR234 Legume_host
20c20
< C58 Legume
---
> C58 Legume_host
diff compares two files line by line and prints only the lines that differ. 4c4 means: line 4 was changed. Lines starting with < come from the first file, lines starting with > from the second. diff prints whole lines, which are long in this table, so | cut -f1,6 keeps only the strain and the host. Lines such as 4c4 have no tab, and cut prints them as they are. So exactly five lines changed, the way you wanted. Now replace the old file:
# 3. Replace the old file with the new one
mv metadata_new.txt metadata.txt
grep 'Legume' metadata.txt | cut -f1,6This prints:
USDA61 Legume_host
R7A Legume_host
CIAT899 Legume_host
NGR234 Legume_host
C58 Legume_host
sed -i: change the file itself
sed can also change a file directly, with the option -i (for “in place”). It does the three steps above in one go, but without the check. This is often where mistakes happen. Say you forgot that you already made the change, and run the command again, this time with -i:
sed -i 's/Legume/Legume_host/' metadata.txt
grep 'Legume' metadata.txt | cut -f1,6This prints:
USDA61 Legume_host_host
R7A Legume_host_host
CIAT899 Legume_host_host
NGR234 Legume_host_host
C58 Legume_host_host
Legume_host still contains Legume, so sed added _host a second time. sed gives no warning, and the old version of the file is gone. This happens easily when you run a script or a notebook a second time.
If you use -i, let sed keep a backup copy. With -i.bak, sed saves the old version as metadata.txt.bak before it changes the file:
sed -i.bak 's/Legume_host_host/Legume_host/' metadata.txt
ls metadata*
grep 'Legume' metadata.txt | cut -f1,6
grep 'Legume' metadata.txt.bak | cut -f1,6This prints:
metadata.txt
metadata.txt.bak
USDA61 Legume_host
R7A Legume_host
CIAT899 Legume_host
NGR234 Legume_host
C58 Legume_host
USDA61 Legume_host_host
R7A Legume_host_host
CIAT899 Legume_host_host
NGR234 Legume_host_host
C58 Legume_host_host
- Work on a copy, not on your original data.
- If you are not sure, write to a new file first and check it with
wc -landdiff. - Ask yourself what happens if the command runs twice. A change like
s/Legume/Legume_host/grows each time,s/Legume_host_host/Legume_host/does not.
- Add
_hostto every host in column 6 ofawk_data/mmc5_metadata.txt, and save the result in a new file. First look at which hosts there are. Do it once with sed, and once with AWK. Tip: on Calculate with AWK, you glued text together with$1"-"$2. - Check both results: count the values in column 6, and compare the two files.
# Which hosts are there?
cut -f6 awk_data/mmc5_metadata.txt | sort | uniq -c
# 1a. sed: one change per host
sed 's/Legume/Legume_host/; s/Arabidopsis/Arabidopsis_host/' awk_data/mmc5_metadata.txt > hosts_sed.txt
# 1b. AWK: glue _host to column 6, on every line after the header
awk -F'\t' -v OFS='\t' 'NR > 1 { $6 = $6 "_host" } { print }' awk_data/mmc5_metadata.txt > hosts_awk.txt
# 2. Check
cut -f6 hosts_sed.txt | sort | uniq -c
cut -f6 hosts_awk.txt | sort | uniq -c
cmp hosts_sed.txt hosts_awk.txt && echo 'identical'This prints:
22 Arabidopsis
5 Legume
1 host
22 Arabidopsis_host
5 Legume_host
1 host
22 Arabidopsis_host
5 Legume_host
1 host
identical
Both commands give the same file, and most strains turn out to come from Arabidopsis, like the plants in the growth data. The sed version works here because you looked first: there are only two hosts, and these two words appear nowhere else in the table. With many different hosts, you would need one change for each.
The AWK version changes column 6 on every line, whatever is in it. $6 "_host" glues the text _host to the value in column 6. Three mistakes give a wrong result without an error:
- Without
NR > 1, the header becomeshost_host. - Run the command a second time, on
hosts_awk.txt, and you getArabidopsis_host_host, as withsed -iabove. - Without the double quotes around
_host, AWK reads_hostas the name of an empty variable, and nothing changes.
The sed on a Mac is a different version (BSD sed). Two differences matter for this page: sed -i needs a backup ending after it (sed -i '' ... for no backup, or sed -i.bak ...), and \t may not mean a tab in the text you look for. If a command from this page does not work on your Mac, this is a likely reason.
Windows line endings
Every line in a text file ends with an invisible character that says “a new line starts here”. Linux and Mac use one character for this, written as \n. Windows uses two: \r\n. A table saved on Windows, for example from Excel, may therefore have a hidden \r at the end of every line. You do not see it on screen, but AWK and grep on Linux see it as part of the last column and this can create silent errors that you will only catch if you closely inspect the output of your commands.
The metadata table you use below was saved from Excel on a Windows PC. file tells you what kind of text a file holds. Compare it with the growth data:
file awk_data/Growth_Data.txt awk_data/mmc5_metadata.txtThis prints:
awk_data/Growth_Data.txt: ASCII text
awk_data/mmc5_metadata.txt: ASCII text, with CRLF line terminators
CRLF is the name for the Windows line ending \r\n. Now see what it does. Column 12, the last column, holds TRUE, FALSE or NA. Count these values with the tools from the CLI workshop, and then let AWK count the rows with TRUE:
cut -f12 awk_data/mmc5_metadata.txt | sort | uniq -c
awk -F'\t' '$12 == "TRUE"' awk_data/mmc5_metadata.txt | wc -lThis prints:
1 Colonization
3 FALSE
13 NA
11 TRUE
0
cut shows 11 rows with TRUE, but AWK finds 0, and there is no error. That happens because column 12 is TRUE\r, not TRUE and therefore the AWK pattern matches nothing. The problem for you is that \r is invisible on screen, also in the output of cut above. You can use cat -A, which shows the Windows line ending as ^M. Since column 12 is the last column you can use cut -f1,12 to have a short view of the strain name and the line end:
head -3 awk_data/mmc5_metadata.txt | cut -f1,12
head -3 awk_data/mmc5_metadata.txt | cut -f1,12 | cat -AThis prints:
strain Colonization
Root123D2 NA
Root381 NA
strain^IColonization^M$
Root123D2^INA^M$
Root381^INA^M$
Remove the \r with sed
To remove the \r, replace it with nothing. In sed, \r stands for this character, and $ means “the end of the line”. So s/\r$// removes a \r only where it sits at the end of a line:
sed 's/\r$//' awk_data/mmc5_metadata.txt > mmc5_metadata_fixed.txt
# Check: file type, the hidden characters, the number of lines and characters,
# and the count from before
file mmc5_metadata_fixed.txt
head -3 mmc5_metadata_fixed.txt | cut -f1,12 | cat -A
wc -l mmc5_metadata_fixed.txt
wc -c awk_data/mmc5_metadata.txt mmc5_metadata_fixed.txt
awk -F'\t' '$12 == "TRUE"' mmc5_metadata_fixed.txt | wc -lThis prints:
mmc5_metadata_fixed.txt: ASCII text
strain^IColonization$
Root123D2^INA$
Root381^INA$
28 mmc5_metadata_fixed.txt
2808 awk_data/mmc5_metadata.txt
2780 mmc5_metadata_fixed.txt
5588 total
11
The ^M is gone, and AWK finds the 11 rows. wc -c counts characters: the fixed file has 28 lines and 28 characters fewer, one \r per line. Keep mmc5_metadata_fixed.txt, you use it on the next page.
\r with tr (optional)
You will often see another command for the same job: tr -d '\r' deletes every \r in a file. tr reads only from a pipe or with <, not from a file name:
tr -d '\r' < awk_data/mmc5_metadata.txt > mmc5_metadata_fixed_tr.txt
cmp mmc5_metadata_fixed_tr.txt mmc5_metadata_fixed.txt && echo 'identical'This prints:
identical
Both commands give the same result here.
What you learned
sed 's/old/new/' filereplaces the first match in each line,s/old/new/gevery match.- The text you look for is a regular expression:
.means any character,\.a real dot.\tis a tab,\rthe hidden Windows character,$the end of a line. - sed looks at the whole line, not at columns. Make the text you look for specific, or use AWK to change one column.
- Separate several changes with
;. They run in the order you write them. sed '1d'deletes line 1,sed '/text/d'every line that containstext.- Check which character separates the columns before you use a file, also when it ends in
.csv. - Change a file safely: work on a copy, write to a new file, check with
wc -landdiff, then replace. Withsed -i, keep a backup (-i.bak), and think about what happens if the command runs twice. - Windows line endings (
\r,^Mincat -A) make text matches on the last column fail without an error. Find them withfileorcat -A, and remove them withsed 's/\r$//'.
On the next page, More useful AWK, you learn a few more AWK forms that are handy in your own work and common in code from an LLM.