sed Scripting


sed

According to the IBM site:
The sed command modifies lines from the specified File parameter according to an edit script and writes them to standard output.

sed, like awk, is a *nix command-line tool designed for editing text (ASCII) files. The macOS is based on Unix, so it's readily available in Terminal. Yes, you should be comfortable in that environment. If not, you could get introduced here.

This is based on macOS Sonoma running GNU sed 4.9
or if you have Homebrew: brew install gnu-sed

If you are using a different version of sed, your mileage my vary.

How it works

Since sed is all about finding patterns, it relies on regular expressions to locate them. As you get more involved with sed, you should become familiar and comfortable with regexes. A good place to start is the GNU sed Manual. We will discuss some of them as we move on.

If you've done some programming you might be familiar with the process of a loop, where a series of commands is executed over-and-over until a condition is met, then quits.

sed works in a similar manner, except the loop and the counting variable are INTERNAL and IMPLICIT. They are held in 2 special variables called the hold space and the pattern space, so a programmer does not need to worry about them.

The loop begins with the first line of the input stream, executes the commands (what pattern to match, and what action to perform), and quits when the input stream ends (end of file).

A command is made of 2 editing actions within front slashes (/) but other characters could be used to make legibility clearer. Just be sure they are NOT in the text being replaced.

Some very complex programs and games(!) have been written just using sed.

Show Me

There are a few reasons you may want to use sed instead of an editor:

  • it is very fast
  • several files can be edited with a single command
  • it can be used in scripts

    Those alone represent a large time-saving advantage if you need to do this kind of thing on a regular basis.

    The kinds of files you can work with may be log files, configuration files, or any that you or your system creates - as long as they are text (ASCII) files.

    One fear that you may have as a new user, is that your script or 1-liner will mangle your file and won't be undoable.

    Have no fear.
    The cool thing about sed is that it DOES NOT alter the file, only the OUTPUT.

    In order to save your changes, you have to create a new file with your intended changes, by redirecting the output yourself, or in your script.

    Substitute

    The most common function of sed is to substitute or swap 1 set of characters for another using s.

    The syntax of a sed command is simple: 

    sed 's/change this/to this/' somefile.txt

    The s stands for substitute or swap.
    The text we want to swap OUT follows the first front slash,
    then another front slash,
    followed by the text we want to swap IN,
    ending with another front slash,
    followed by the name of the file or input stream.

    POC

    As a Proof Of Concept do this: save the following text to a text file on your Desktop: (no word processors allowed here)

    The quick brown fox slept until noon
Our Scaly Starfish Ran Partly
This Fidgety Heron Seamlessly Complied

    The 2nd & 3rd lines were generated by a program I wrote to create pass phrases.

    I've called it sedit.txt but you can name it whatever you like.

    Now open Terminal, and 'cd ~/Desktop'...
    that will now allow you to work with files on your Desktop.

    To make sure, enter 'cat sedit.txt' and you should see the contents of that file.

    Now enter the following sed command: 

    sed 's/noon/3pm/' sedit.txt
    and you should see ...
    The quick brown fox slept until 3pm
Our Scaly Starfish Ran Partly
This Fidgety Heron Seamlessly Complied

    Now, enter 'cat sedit.txt'

    Excellent! You can see that your file WAS NOT changed - only the output.

    So how do you now make the changes permanent?

    sed 's/noon/3pm/' sedit.txt > didit.txt

    Again, your original file has NOT been altered. You have redirected the output of the sed command to a new file instead.

    You may be thinking 'but that only changed the first match'. But of course we can change ALL the words or letters that match our pattern very simply - just add a g for global. 

    sed 's/o/0/g' sedit.txt

    The quick br0wn f0x slept until n00n
Our Scaly Starfish Ran Partly
This Fidgety Her0n Seamlessly C0mplied

    Note that Our was not altered, since we were looking for a lower case o. Here's how we would look for and swap both upper and lower case: 

    sed 's/[oO]/0/g' sedit.txt

    Now we see 

    The quick br0wn f0x slept until n00n
0ur Scaly Starfish Ran Partly
This Fidgety Her0n Seamlessly C0mplied

    Not only can we swap a set of characters, we can replace a whole line: 

    sed 's/^Our.*/Flippantly My Dearest Bear Apprehended/g' sedit.txt

    A couple of new things in there:

    Now we get 

    The quick brown fox slept until noon
Flippantly My Dearest Bear Apprehended
This Fidgety Heron Seamlessly Complied

    That was also an introduction to regular expressions, as mentioned above.

    What else can you do?

    There are other actions that can be performed:

    These actions are usually inherent in an interactive editor, not a non-interactive one like sed. Welcome to the *nix world!

    Delete

    We'll use the file we created earlier (sedit.txt) to work with:

    Enter the following into a shell window: 

    sed '/quick/d' sedit.txt

    You should see that the first line 'The quick brown fox slept until noon' is now missing, because it had the characters 'quick' in it: 

    Our Scaly Starfish Ran Partly
This Fidgety Heron Seamlessly Complied

    But again, sed does NOT alter the original file - only the output.

    Append, Insert