Sed patterns

There really many sed tutorials out there, so I do not think there is a point in me writing another one. The one of the bests is at grymoire and I totally recommend it. In this blog I would rather talk about my patterns in sed that I use.

The biggest breakthrough I had when understanding sed is what actually is pattern space and hold space. Once it “clicked”, the rest of the commands came naturally.

Do not be mistaken – this is sed. It’s a really simple tool for simple stuff. For anything more complicated, get a real programming language. AWK is always available at hand and GNU AWK is a full-blown programming language with libraries support. Nowadays, Python is the top one to choose for anything more complicated.

This post is based on my experience on stack overflow where I had answered many sed questions multiple times. My sed skills became refined to adjust for the most common problems other programmers have. This has prompted me to write this post.

“Be conservative in what you do, be liberal in what you accept from others”.

When writing a regex you have two choices in how to accept the input. You can twiddle with the regex to make it as strict as possible, matching the single line as exact as possible to the regex, or write a “just works” regex to match anything.

More often than not, sed script is going to be a small ad hoc regex that is used in a terminal to do small transformation. Something I learned over the years of my programming journey, react proactively. Most often than not, you’ll fix the regex when it breaks. Bottom line, my advice is to consider being liberal in what you accept in regexes in sed. Matching input strictly takes more time and more often than not excludes too much than intended.

Load everything, parse later.

Sed typically works on a line per line basis. It loads one line into the pattern space and then parses the pattern space and then prints the pattern space. The end.

Sometimes you want to do operations on multiple lines, like replace something only if previous line ends with something s/something\nelse/new/. There are two patterns for that.

Firstly, you can load everything into sed and parse later:

sed ':a;N;$!ba;s/somthing\nelse/new/'

The :a;N;$!ba creates a label a, appends new line to pattern space, if not last line branches to label a. After that you have everything in pattern space.

However, most of the time we have GNU sed available. In that case, just parse a newline separated file as zero separated file.

sed -z 's/something\nelse/new/'

Escape everything.

Sometimes you have a string in Bash that you want to put to sed to replace or substitute literally. How to escape that string, so it’s not interpreted as a regex in the first part of s command, and as a special replacement characters & \ in the replacement part of s command?

The answer comes from the amazing https://stackoverflow.com/questions/407523/escape-a-string-for-a-sed-replace-pattern :

replace="stuff to replace"
keyword="The Keyword You Need";
escaped_keyword=$(printf '%s\n' "$keyword" | sed -e 's/[]\/$*.^[]/\\&/g'
escaped_replace=$(printf '%s\n' "$replace" | sed -e 's/[\/&]/\\&/g')
sed "s/$escaped_keyword/$escaped_replace/g"

Substitute part of line.

Typical when working with a CSV file. For example, you want to replace all spaces in the 3rd field of a CSV file. However, parsing CSV files with sed is not a good idea – it is not too much work to much 1,"string\"escapes" inside a CSV file.

Consider the following imaginary scenario: in a space separated file you want to replace , comma by a space only in the second column.

Tim,26 makes,good,money.
Thomas,25 does,marry, Susanne,37.

Running global regex over only part of the line consists of steps:

  • splitting the line into interesting part and not interesting part,
  • typically for representing list I use \n newline,
  • note technically, \n is not allowed in replacement list in POSIX sed,
  • putting the not interesting part(-s) into hold space,
  • putting the interesting part into pattern space,
  • running the regex,
  • getting hold space,
  • shuffling the pattern space with hold space for output.

In a script it looks like the following:

sed '
  s/\([^ ]*\) \([^ ]*\)\(.*\)/\1\n\2\n\3/
  h
  s/[^\n]*\n\([^\n*]\).*/\1/
  s/,/ /g
  G
  s/\([^\n]*\)\n[^\n]*\n\([^\n]*\)\n\(.*\)/\1\3\2/
'

Join or remove line only if previous line.

99% of the time the file will fit in RAM. First, load it fully into sed. See above. Then run a single regex. To match a line, match [^\n]*\n, not just .*.

Edit file in ??? format with sed.

If the format of the file is anything more than a newline separated columns, consider using a better programming language. Sed is great, and it’s great for running regex over one line. When working with specifically formatted input in predefined complicated formats, do not use sed. Use a tool for that that handles that format. It will save you time from coming up with clever sed solutions that do not work and will save future yourself from the work you need to put when your clever sed breaks.

Edit JSON files with sed.

DO NOT edit JSON files with sed. It’s not a tool for that. Sed is not a JSON parser. Use jq or write a Python program.

Edit XML files with sed.

DO NOT edit XML files with sed. It’s not a tool for that. Use Python. If you are writing simple extraction script “get value of node a with class that”, then consider xmllint and xmlstarlet. However, I find xmllint and xmlstarlet scripts extremely complicated. I can’t understand them.

There are times when the XML files are formatted correctly. The most important are newlines, as in XML you can put a newline inside tags. If so, sed is just acceptable, but do know that it will break once the XML format changes. That’s why I sometimes reformat XML (HTTP) files with xmllint and then put it into sed. But for most cases, xmllint is enough. For anything complicated, like editing XML files, use a real programming language like Python.

End

Sed is a small and fun language, but bottom line it is very limited, with no function and returns and “manual” stack management in hold space. Keep it simple. Run line replacements in sed when working with text and enjoy it.

Leave a Comment

Scroll to Top