Friday, December 3, 2010

Bash named pipes

Pipes are a unix staple. They're core to the unix philosophy of modularity. Bashes named pipes just knock this up a notch.

Lets say we have two directories that should have contain the same files, but one is on a case insensitive file system, so some files might differ in case. We could do this, using normal pipes to process the directory entries.

ls dir1 | tr "[:lower:]" "[:upper:]" | sort > dir1_contents
ls dir2 | tr "[:lower:]" "[:upper:]" | sort > dir2_contents
diff dir1_contents dir2_contents

But this is kind of hacky and creates two temporary files.

Bash named pipes lets you use the output of a pipe system in place of a filename. Its almost like the pipes were written into temporary files and the files deleted afterwards, but no disk space is consumed by the files.

diff <( ls dir1 | tr "[:lower:]" "[:upper:]" | sort ) <( ls dir2 | tr "[:lower:]" "[:upper:]" | sort )

Much neater.

Mind you I haven't used these often, but if you know they exist they can make some painful multistep processes much simpler.


3 comments:

  1. I find it interesting that the bash name pipes actually have no names!

    Effectively you are pushing in the output of two subshells.

    ReplyDelete
  2. Hi Wally, It does seem odd at first doesn't it.

    However the underlying OS feature used to implement this is called "named pipes".

    What bash does is actually create a named temporary file/pipe using the mkfifo command to do everything and hides the magic from you. An equivalent but more verbose way of doing this is

    mkfifo pipeA ; mkfifo pipeB ; diff pipeA pipeB ; ls dir1 | sort > pipeA ; ls dir2 | sort > pipeB

    But the bash way use using those named pipes is much neater IMO.

    ReplyDelete
  3. Named pipes are magically named under the hood. It's much more helpful to have your explanation for that magic, and I certainly agree that the "bash-way" is the more elegant.

    (Its still 2 sub-shells throwing stuff into those pipes).

    Now, I wonder if I can tart up my Solaris backup scripts to do things this way and make them smaller and simpler.

    Then again they have run OK every day for the last 5 years w/o trouble so perhaps I'll just leave them alone.

    ReplyDelete