Bash debugger

This is for all of you hardcore bash scripters that sometimes want better debugging than placing "echo" statements in your scripts.

A co-worker sent this out to our internal sysadmin mail alias today.

http://www.linux.com/feature/153383

It references the bashdb project (bash debugger). I didn't know one existed.

Careful where you copy/paste

I got an IM from an old coworker discussing that he was trying to answer someone's question about an in-house app and which shared libraries it was using. (NOTE: some of the info has been sanitized to protect the "innocent")

He ran the following command, and then copy/pasted the output

[root@customftp root]# ldd /nfs/customapp/bin/daemon
libkstat.so.1 => /usr/lib/libkstat.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libsched.so.1 => /usr/lib/libsched.so.1
libaio.so.1 => /usr/lib/libaio.so.1
librt.so.1 => /usr/lib/librt.so.1
libthread.so.1 => /usr/lib/libthread.so.1
libstdc++.so.6 => /usr/lib/libstdc++.so.6
libm.so.1 => /usr/lib/libm.so.1
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1
libc.so.1 => /usr/lib/libc.so.1
libmp.so.2 => /usr/lib/libmp.so.2
libmd5.so.1 => /usr/lib/libmd5.so.1
/usr/platform/SUNW,Sun-Fire-V210/lib/libc_psr.so.1
/usr/platform/SUNW,Sun-Fire-V210/lib/libmd5_psr.so.1

He then tried to copy paste it into an email, but didn't realize his SSH session was still the active window.

[root@customftp root]# nfs/customapp/bin/daemon
bash: nfs/customapp/bin/daemon: No such file or directory
[root@customftp root]# libkstat.so.1 => /usr/lib/libkstat.so.1
bash: libkstat.so.1: command not found
[root@customftp root]# libnsl.so.1 => /usr/lib/libnsl.so.1
bash: libnsl.so.1: command not found
Connection to customftp closed by remote host.
Connection to customftp closed.

It took me a few minutes to see why he was sending me this IM. My initial reaction was "OK, so you accidently copy/pasted into the wrong window, and somehow you got kicked out." Before reading further, take a look back to see what was so bad about it. Maybe you'll pick it up faster than I did.

Ready?

The output of ldd in a copy/paste was interpretted as piping the output of bad commands to his library files. He basically nuked files he needed in /usr/lib. I initally laughed out loud, but then realized that it was a very unfortunate situation. He said it took him about 2 hours to recover from this mistake.

Cron entry mistakes

This mistake came at the expense of one of my old co-workers, who decided to share this with me over Instant Messenger. I got a good laugh at his misfortune. He wanted to set up a cron job that would restart a poorly performing service every 6 hours. This is the cron entry he added.

* */6 * * * service_restart.sh

The lead developer, whose app depends on that service, came up to him wondering why his app was having outages. What my colleague should have configured was

0 */6 * * * service_restart.sh

His cron entry was restarting the service every 6 hours, and then restarting it every minute for an hour during that sixth hour. Brilliant! Although I guess I shouldn't throw stones in glass houses.