Heredocuments in Bash: Multi-Line Strings and Input Redirection

Bash provides a powerful mechanism called heredocuments (often shortened to "heredocs") to define multi-line strings and efficiently redirect them as input to commands. This tutorial will explain how heredocs work, their syntax, and practical applications.

What are Heredocuments?

Heredocuments allow you to embed multi-line strings directly within a script or on the command line. These strings are then treated as standard input for a command, simplifying tasks that require complex, multi-line input. They’re particularly useful when dealing with commands like psql, sed, awk, or any other tool that reads from standard input.

Syntax of Heredocuments

The basic syntax of a heredoc is as follows:

command <<HEREDOC_DELIMITER
line 1 of the multi-line string
line 2 of the multi-line string
...
last line of the multi-line string
HEREDOC_DELIMITER

Let’s break down this syntax:

• command: This is the command that will receive the multi-line string as its standard input.
• <<: This is the redirection operator that signals the start of the heredoc.
• HEREDOC_DELIMITER: This is a unique string that marks the end of the multi-line input. It’s commonly (but not necessarily) EOF (End Of File), but you can choose any string as long as it doesn’t appear within the multi-line input itself. The delimiter must appear on a line by itself, with no leading or trailing whitespace.
• The lines between <<HEREDOC_DELIMITER and HEREDOC_DELIMITER constitute the multi-line string.
• HEREDOC_DELIMITER: The delimiter, on a line by itself, signals the end of the heredoc.

Example: Sending Multi-Line SQL to psql

Suppose you want to execute a series of SQL commands using psql. Instead of manually typing them or reading them from a file, you can use a heredoc:

psql -d your_database <<EOF
BEGIN;

SELECT * FROM your_table WHERE condition;

UPDATE another_table SET column = value WHERE id = 123;

COMMIT;
EOF

In this example, the SQL commands between <<EOF and EOF are sent as input to psql. The -d your_database option specifies the database to connect to.

Variable Expansion and Quoting

By default, Bash performs variable expansion, command substitution, and arithmetic expansion within a heredoc. This means any variables or commands within the multi-line string will be evaluated before the string is passed to the command.

To prevent variable expansion, you can quote the heredoc delimiter:

psql -d your_database <<'EOF'
BEGIN;

SELECT * FROM your_table WHERE condition;

UPDATE another_table SET column = value WHERE id = 123;

COMMIT;
EOF

Using single quotes around the delimiter prevents any expansion, treating the entire string literally. Double quotes would allow expansion.

Stripping Leading Tabs with <<-

Sometimes, you might want to indent your heredoc within a script for readability. The <<- operator allows you to strip leading tab characters from each line of the heredoc, making the code more visually appealing while preserving the intended input.

psql -d your_database <<-EOF
	BEGIN;

	SELECT * FROM your_table WHERE condition;

	UPDATE another_table SET column = value WHERE id = 123;

	COMMIT;
EOF

In this case, the leading tabs will be removed before the input is sent to psql. Note that only tabs are stripped; spaces are preserved.

Important Considerations:

• Delimiter Uniqueness: Choose a delimiter that is unlikely to appear within the multi-line input. EOF is a common choice, but if your input might contain EOF, use a less common string.
• Delimiter Placement: The delimiter must be on a line by itself, with no leading or trailing whitespace.
• Expansion Control: Be mindful of variable expansion. Use quotes around the delimiter if you need to prevent expansion.
• Readability: Use indentation to make your heredocs more readable, especially when they are embedded within scripts.