keepcalculator

06 Numbers and Powers

Bash Remove Empty Lines: 5 Ways

Card No. 06·65433 · Updated · Standard formula, shown below

To remove empty lines from a file or command output in Bash, you can use grep, sed, awk, or a simple while loop. The most common and fastest method is grep -v '^$' file, which prints every line that is not empty.

What Counts as an Empty Line?

Before choosing a command, decide what you mean by “empty.” A truly empty line has zero characters and only a newline at the end. However, many files contain lines that look empty but actually have spaces or tabs. Those are not empty—they contain whitespace. The commands below treat these two cases differently:

  • Zero-length line: ^$ in regex matches a line with nothing between start and end.
  • Whitespace-only line: ^[[:space:]]*$ matches lines that have only spaces/tabs (or nothing).

Using grep to Remove Empty Lines

The grep command is the simplest tool for this task. Use the -v flag to invert the match, meaning it prints lines that do not match the pattern.

grep -v '^$' file.txt

This prints the file without empty lines to standard output. To save the result back to the same file, you need a temporary file:

grep -v '^$' file.txt > tmp && mv tmp file.txt

If you also want to remove lines that contain only spaces or tabs, use:

grep -v '^[[:space:]]*$' file.txt

Using sed to Delete Empty Lines

sed (stream editor) can delete lines that match a pattern. The command /^$/d deletes every empty line.

sed '/^$/d' file.txt

To edit the file in-place (on GNU sed, which is standard on Linux), add the -i flag:

sed -i '/^$/d' file.txt

On macOS (BSD sed), use sed -i '' '/^$/d' file.txt.

To also remove whitespace-only lines, use sed -i '/^[[:space:]]*$/d' file.txt.

Using awk to Filter Non-Empty Lines

awk is a text-processing tool that can filter lines based on the number of fields. An empty line has zero fields, so the condition NF > 0 selects only lines with at least one field (i.e., non-empty).

awk 'NF > 0' file.txt

This automatically treats lines with only spaces as empty because NF counts fields separated by whitespace. To write back to the file, use a temporary file or sponge from moreutils:

awk 'NF > 0' file.txt > tmp && mv tmp file.txt

Or use sponge if installed: awk 'NF > 0' file.txt | sponge file.txt.

Using a While Loop (Portable but Slow)

For small files or when you need to process each line individually, a while read loop works in any POSIX shell:

while IFS= read -r line; do
    [ -n "$line" ] && printf '%s\n' "$line"
done < file.txt > tmp && mv tmp file.txt

The -n test checks if the string is non-empty. This method is slower than grep or sed for large files but gives you full control if you need to modify lines while filtering.

Removing Trailing Blank Lines Only

Sometimes you only want to remove blank lines at the end of a file (e.g., trailing newlines). sed can do this with a range address:

sed -i -e :a -e '/^\n*$/{$d;N;ba' -e '}' file.txt

This is a classic idiom. A simpler approach is to use perl:

perl -0pi -e 's/\n+\z/\n/' file.txt

This removes all trailing newlines and adds exactly one back. It does not affect blank lines in the middle of the file.

Common Mistakes and Pitfalls

  • Using ^\s*$ with grep: GNU grep supports \s as a shorthand for [[:space:]], but it is not POSIX. Use [[:space:]] for portability.
  • Overwriting the input file: Redirecting output to the same file you are reading (grep ... file > file) truncates the file before reading, resulting in an empty file. Always use a temporary file or sponge.
  • Windows line endings: If your file has CRLF (\r\n), a line may appear empty but contain a carriage return. Use dos2unix first or adjust the pattern: grep -v '^\r$'.

Frequently Asked Questions

How do I remove empty lines from a variable in Bash?

Use grep with a here-string: cleaned=$(grep -v '^$' <<< "$var"). This works for small strings. For large text, write the variable to a file first.

Does sed -i work on macOS?

Yes, but the syntax differs. macOS uses BSD sed, which requires an empty argument after -i: sed -i '' '/^$/d' file. On Linux, sed -i works without the extra argument.

What is the fastest way to remove empty lines from a huge file?

grep -v '^$' is typically the fastest because it is optimized in C and does not buffer the entire file. sed is slightly slower but still fast. Avoid shell loops for files larger than a few megabytes.

How do I remove empty lines and also trim leading/trailing spaces?

Use awk '{gsub(/^[ \t]+|[ \t]+$/, ""); if (NF) print}' file. This strips spaces and tabs from each line, then prints only non-empty lines.

Can I remove empty lines from command output without a file?

Yes, pipe the output directly: command | grep -v '^$'. This works for any command that produces text on stdout.

Why does my file still have blank lines after using grep -v '^$'?

Your blank lines likely contain spaces or tabs. Use grep -v '^[[:space:]]*$' instead. Also check for CRLF line endings as mentioned above.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →