Quick and Dirty NetBackup Reporting

I needed a quick way to get the statuses of my NetBackup runs on a daily basis. This works for me, but maybe because there aren't that many servers (~30) that we are backing up. I'm sure there's nicer ways to do this, but if you want something quick and dirty, I have this line in my crontab.

/usr/openv/netbackup/bin/admincmd/bperror -U -backstat -s info -hoursago 24 | mail -s "Netbackup Report for `date +%D`" backup-admins@example.com

Man page reading tip

I usually keep my ssh windows pretty small, but it makes them a pain to read man pages. The way I read man pages now is using Google and finding man pages posted online. That way, it's as easy as reading any other web page.

For Linux, I use the search string (minus the quotes) "man linux command", and it usually pulls up the appropriate man page on http://linux.die.net.

For Solaris, I'll use the search string (minus the quotes) "man sunos command", since the syntax or switches of the Solaris command may be slightly different than the Linux one.

Another bonus is that commands and configurations in the "SEE ALSO" section are usually hyperlinked to the corresponding web entry. There are probably browser plugins or toolbars that will accomplish the same thing, but this is universal and lightweight.

Deleting a Solaris RAID created with Volume Manager

It seems most of my "howto" posts recently have been Solaris related. The main reason for that is that my Solaris admin knowledge is weak, requiring me to look up Solaris-specifc tasks. And when I learn how to do something, I like to share with others the answer, just in case they didn't either.

We have a Solaris 10 host in the lab that we do software tests on, and I somehow have been tapped to admin and be in charge of its OS. The production hosts we create for clients use RAID-1 with two disks, so the orignal install steps specified using the Sun Volume Manager using the different "meta" commands (metadb, metainit, metattach). Without much warning, they've gone and changed the install steps to use the simpler "raidctl" command. My problem now is that I have to reinstall the lab test host, and setting up the RAID with raidctl. However, I have no idea how to delete the previous RAID configuration. Thankfully, I came across this. It appears all I had to do was run "metaclear -a".

My take on "Which Unix to learn"

I came across this post on TaoSecurity today, with Richard Bejtlich's suggestions for an "Introduction to Unix." Like text editors, the discussion about which distribution of Unix to learn can be considered a "religious" argument. If you're a regular reader of his blog, it's no surprise he suggests FreeBSD. He does clarify and say if you're running a server, he suggests FreeBSD; and if you're running a desktop, he now prefers Ubuntu. And if you're still adamant about running Linux as a server, he suggests Debian.

My opinion differs, depending mainly on "why do you want to learn Unix?" If the answer is related to increasing your work/resume skill sets, I would have to disagree about using FreeBSD or any of its other BSD derivatives (Net, Open, etc.). In my experiences of being an admin or some other support role, I have yet to encounter a *BSD server. I'd encourage someone to use a distribution that they would encounter in a corporate environment. The Unix server OS's I've had to support have been Red Hat (now Fedora), Red Hat Enterprise Linux (also known as RHEL), and Sun Solaris. To avoid having to pay licensing fees, you could substitute CentOS for RHEL.

If I were asked the question, I would suggest a "major" Linux distribution or Solaris/Open Solaris instead. I would think their device names, software packages, and file system organization would help with familiarity when trying to translate the knowledge learned to a corporate environment. Although this may be an unfounded opinion, I also think that there are more support options and supported software using them, rather than using *BSD. I do agree with Bejtlich that if you want to run a Unix distribution on the desktop, to stick with Ubuntu, since it seems to "just work" when installed and there are less configuration headaches. Or you can just use a Mac if you want a Unix desktop (troll... and yes Mac could be considered a BSD variant).

Linux Mag's Top 10 Sys Admin Articles of 2008

With 2008 coming to an end, everyone is starting to put out their "Top" lists. I saw the announcement today in my Inbox that Linux Magazine posted its Top 10 Sys Admin Articles of 2008. Unfortuantely, the links to articles requires registration, but it's free. The two articles on ZFS interest me the most, and I want to refresh myself on the "port knocking" article. I forget where I've heard about port knocking before. I'm pretty sure it was a couple years ago though.

It also made me realize I miss reading physical magazines over online copies. Then again, I rarely read the ones I get in the mail now. It's amazing how technology changes things.

On a totally unrelated note, I just wanted to share a good laugh about Matt Simmons' comment on a blog post about having multiple superusers on the VMware ESX server.

"Once, many many moons ago, probably near the peak of my danger curve, I got tired of su’ing all the time, so I just changed my uid to 0 in the passwd file. That was fun."

I can't explain why it struck me as funny as it did, but that was probably the funniest thing I've read today.

"find" and "Permission denied"

I was trying to run "find" on a directory, looking for a perl script. Because of which user I was running it as, I was getting a bunch of "Permission denied" errors.

I thought I could just tack on "| grep -v "Permission denied" at the end of my command, but no luck. I fruitlessly checked to see if there would be something in the find man page as well.

I came across this site: http://www.hypexr.org/linux_find_help.php

The correct answer to what I was trying to accomplish was

$ find / -name foo.bar -print 2>/dev/null

Are new releases of Slackware newsworthy?

From the Slashdot RSS feed, I saw "Slackware 12.2 Released", and it made me wonder if this is actually newsworthy. Granted, Slackware was technically the first Linux distribution I ever installed. It came with a "Linux for Dummies" book I bought in 1997, when I was starting to become more interested in doing more with computers than basic every day tasks (email, web surfing, word processing, games). However, my experience with Slackware was short lived because I was still living at home, and my parents used AOL for Internet access.

Nowadays, it seems Slackware is revered by older users, but isn't used. To me, it falls into the same area as using the Eudora mail client, or until recently, Netscape web browser. It seems Ubuntu (and its variants) and Fedora dominate the Linux desktop, with a smattering of Debian users; and CentOS, Red Hat Enterprise Linux, and Ubuntu (somewhat surprising to me), with some Gentoo diehards, leading the Linux servers. I think this may be the least used "mainstream" *NIX distribution available. Except for nostalgia, what reasons do users install Slackware for?

Not that I'm discouraging further development of Slackware, or trying to insult their users. I'm just questioning their relevance in today's IT news, unless it's just a slow news day :)

Should you learn vi?

There are so many arguments about which editor to use. Some people are quite passionate about their editor of choice. I came across this blog post today through Planet SysAdmin discussing why one person uses vi for system administration.

http://utcc.utoronto.ca/~cks/space/blog/sysadmin/WhyViForSysadmins

It pretty much sums up why it was stronly suggested to me to use vi at the beginning of my career. At that time, I only knew pico and a few commands in emacs; although now, I couldn't tell you how to edit a file in either. Over the years I've noticed the author's point is true. Almost any unix type system you encounter, it will have at least vi installed. It's definitely handy to have a basic proficiency with vi if you do any command line *NIX administration.

So, while you get your caffeine fix at work, this could come in handy as well.

http://www.thinkgeek.com/homeoffice/mugs/7bbe/

Howto Quickly wipe a disk in Solaris

I've had to wipe disks in Solaris because of re-installs or other reconfiguration. I was disappointed that using "format" would take hours. Obviously I'm using the wrong tool.

I came across this thread.

http://www.linuxquestions.org/questions/solaris-opensolaris-20/quick-format-a-hard-drive-496294/

It's strange, but I'm noticing people are posting Solaris questions on a Linux website. I wonder how Solaris admins would've felt 10 years ago if people were posting Linux questions on a Solaris site or Usenet group. But, I digress.

The response is to use "newfs" instead of "format", and it seems to work so far. Thankfully, I learn something new every day.

Careful where you copy/paste

I got an IM from an old coworker discussing that he was trying to answer someone's question about an in-house app and which shared libraries it was using. (NOTE: some of the info has been sanitized to protect the "innocent")

He ran the following command, and then copy/pasted the output

[root@customftp root]# ldd /nfs/customapp/bin/daemon
libkstat.so.1 => /usr/lib/libkstat.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libsched.so.1 => /usr/lib/libsched.so.1
libaio.so.1 => /usr/lib/libaio.so.1
librt.so.1 => /usr/lib/librt.so.1
libthread.so.1 => /usr/lib/libthread.so.1
libstdc++.so.6 => /usr/lib/libstdc++.so.6
libm.so.1 => /usr/lib/libm.so.1
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1
libc.so.1 => /usr/lib/libc.so.1
libmp.so.2 => /usr/lib/libmp.so.2
libmd5.so.1 => /usr/lib/libmd5.so.1
/usr/platform/SUNW,Sun-Fire-V210/lib/libc_psr.so.1
/usr/platform/SUNW,Sun-Fire-V210/lib/libmd5_psr.so.1

He then tried to copy paste it into an email, but didn't realize his SSH session was still the active window.

[root@customftp root]# nfs/customapp/bin/daemon
bash: nfs/customapp/bin/daemon: No such file or directory
[root@customftp root]# libkstat.so.1 => /usr/lib/libkstat.so.1
bash: libkstat.so.1: command not found
[root@customftp root]# libnsl.so.1 => /usr/lib/libnsl.so.1
bash: libnsl.so.1: command not found
Connection to customftp closed by remote host.
Connection to customftp closed.

It took me a few minutes to see why he was sending me this IM. My initial reaction was "OK, so you accidently copy/pasted into the wrong window, and somehow you got kicked out." Before reading further, take a look back to see what was so bad about it. Maybe you'll pick it up faster than I did.

Ready?

The output of ldd in a copy/paste was interpretted as piping the output of bad commands to his library files. He basically nuked files he needed in /usr/lib. I initally laughed out loud, but then realized that it was a very unfortunate situation. He said it took him about 2 hours to recover from this mistake.

Using fvwm2 on Solaris 10

Since I don't like CDE, and the Java Desktop runs painfully slow and looks ugly on my Ultra5; I decided to install fvwm2. I found this site that makes adding fvwm2 fairly painless.

http://www.tiem.utk.edu/~peek/solaris/

Firefox still runs in it, but looks ugly because of the lack of colors. I thought I could get 16bit colors, but I haven't figured it out yet.

I'm debating making it a lab Jumpstart server since it has a 40GB drive. You could argue that I should junk my Ultra5 and it's just a waste of time. It's definitely not worth running as a workstation, since I can easily run VMware Server on my laptop if I want a Unix OS.

Segmentation faults when formatting a disk in Solaris

I've had an old Sun Ultra 5 under my desk since I've started at the current job. Instead of keeping it as my footstool, I decided to fire it up and see what happens. Needless to say, this has been a timesuck. I could go into all the details and other mistakes on my part, but it isn't very interesting.

What I didn't know would be a problem was trying to install Solaris 10 after I had impatiently installed NetBSD. Apparently the Solaris installer doesn't know how to recognize the disk formatted by NetBSD. When running "format", it just segfaults and dumps core.

After some Google searches, I came up on this thread.

http://groups.google.com/group/comp.unix.solaris/browse_thread/thread/8107078ff8e2174d?pli=1

after running

dd if=/dev/zero of=/dev/rdsk/c0t0d0s0 bs=512 count=4

I was able to run format on the disk, and run the Solaris 10 install.

Solaris Jumpstart server interference

I've been working on trying to install Solaris 10 via DVD and "boot cdrom" on a host in the lab. I've been wondering why the install keeps assigning it an IP address and hostname without me supplying one. Even after running "sys-unconfig", I was never prompted to put in a new IP address or hostname.

The problem was our Jumpstart server. We had jumpstarted this host in the past, testing some automation installs, and the server still had an entry in the ethers file. Apparently when the server starts up, it tries to be slick and configure its interfaces for you when it boots. It looks like the Jumpstart server was picking up the requests and saw it had a matching MAC address in its configuration. The Jumpstart server then passed the hostname and IP address to the server I was trying to install. After we shut down the Jumpstart server, the sys-unconfig worked as expected. I wish the "discovery of network card settings" was not the default behavior.

Sudo not included in default Solaris 10 installs

To my surprise, my "default installation" of Solaris 10 that seems to install every known package in the world (who needs Firefox on a server?!?!) does not include "sudo." Apparently, it's on the Companion DVD or available for download from

http://www.sun.com/software/solaris/freeware/

There are also a couple interesting twists with the default package install. One is that the package is installed in /opt/sfw. This includes the sudoers file, which is located and read from /opt/sfw/etc/. In my opinion, this is a pain, because it doesn't put sudo or visudo in the normal default $PATH=/usr/bin:/usr/sbin:/bin:/sbin. I ended up creating symlinks for /etc/sudoers, /usr/bin/sudo, and /usr/sbin/visudo to the appropriate locations in /opt/sfw.

Another strange twist is that after the install, you need to chmod the sudo command with "chmod 4111 sudo" because Solaris doesn't inherently trust the package and it's files. This is another annoyance, since obviously anyone installing sudo had to take extra steps to do so in the first place and obviously already trusts its integrity.

Chalk up another reason why I like Linux over Solaris.

Update on useful UNIX tools for Win32

A co-worker sent out this link that has ways to "turbocharge" PuTTY.

Finally looks like I can get my tabbed SSH/terminal sessions. It also mentions puttycyg (affirmation, baby!).

UPDATE (8/29/2008 10:33 EDT):
My comments so far trying out tabbed putty/ssh sessions.
- It doesn't look like PuttyCM supports Puttycyg, which is disappointing. If you use the original putty.exe, it does seem to work how I'd want it to.
- The Putty sessions aren't rolled up into the Putty Tabs program (they still show individually in my taskbar). This doesn't improve my workspace much.

Cron entry mistakes

This mistake came at the expense of one of my old co-workers, who decided to share this with me over Instant Messenger. I got a good laugh at his misfortune. He wanted to set up a cron job that would restart a poorly performing service every 6 hours. This is the cron entry he added.

* */6 * * * service_restart.sh

The lead developer, whose app depends on that service, came up to him wondering why his app was having outages. What my colleague should have configured was

0 */6 * * * service_restart.sh

His cron entry was restarting the service every 6 hours, and then restarting it every minute for an hour during that sixth hour. Brilliant! Although I guess I shouldn't throw stones in glass houses.

Useful UNIX tools for Win32

OK, it's been a while since I've posted. Being laid off and unemployed kind of hurts writing about work experiences. However, don't worry. I'm now at a new job in a new town.

Obtaining work responsibilities have been slow, so I've been reading and playing with software that I hope will help me in making my job easier later. One area I've been reading and trying to increase my knowledge in is remote access, predominantly SSH. I've found the following to be useful.

SSH Agent
I've known about shared-key authentication with SSH for a while, but I've been lame and not applied a passphrase to my key. Honestly, I wanted a quick way to log in, and skip the password. However, I finally was exposed to ssh-agent, which allows you to cache your key's passphrase so you only have to enter it once. After that, when you ssh into a host, ssh-agent will take care of your passphrase for you.

I'm using Cygwin for my shell on my PC, so I'm using the ssh-agent that comes with Cygwin. I hear you can use a program called Pageant to perform this for you PuTTY users.

• Link to Cygwin and ssh-agent
• Link to HOWTO for Pageant with PuTTY
  

Puttycyg
I hate the command window that Cygwin defaults to. The copy/paste is terrible, as I think it borrows from cmd.exe. So, I found Puttycyg that gives the Cygwin command prompt a better look and feel. Now I actually get a copy/paste behavior that I like.

Poderosa
I've been spoiled with Mac OSX's iTerm for having tabbed terminal windows. Unfortunately, PuTTY and Cygwin spawn new windows, and their management gets cluttered. Right now, I'm trying Poderosa, mainly because it appears to be open source and managed by the Apache license. It tabs your Cygwin sessions, and apparently other SSH/Telnet sessions as well (but I haven't tried the latter). I found another alternative in Wintabber, but I haven't tried it yet. It appears to be free and have more features I'd be interested in, but I believe is closed source.

I'd be interested in your experiences with tabbed command/shell windows and other useful remote access tricks.

Problem with slave BIND server

After one of the sysadmins at work patched our RHEL4 DNS servers, I noticed our slave server was not successfully receiving updated zone files from the master.

Looking at the logs, I kept seeing the following error

named[5182]: dumping master file: tmp-XXXXTCPn1l: open: permission denied

I installed strace on the server and tried to find where it was trying to write to and see if somehow the directory permissions were incorrect. Unfortunately, strace just showed the "tmp-XXXXX" file failing, but not the directory location.

After some investigation on Google, I found this site. It explained that an updated version of BIND required changes to the named.conf file for slave DNS servers. I then modified our named.conf so that the setting for the zone file was

file "slaves/example.com.zone";

instead of the

file "example.com.zone";

I'm not sure how it worked in the past. Perhaps Red Hat backported an update in BIND that was not initially in RHEL4? I don't know what the Update version was before the patching, else I could probably dig through release notes. However, if you're seeing strangeness with your slave BIND servers, I'd check to make sure your named.conf isn't out of date.

Creating ISO images with Mac OS X

Since moving .iso files is easier than CD's when performing remote installs, I was trying to find out how I could without paying for software. Apparently, you can do this natively with Mac OS X.

Giving credit where credit is due, I found this link very useful.

http://www.slashdotdash.net/articles/2006/08/14/create-iso-cd-dvd-image-with-mac-os-x-tiger-10-4

And then, if your kernel supports it, you can mount it like this.

# mount -o loop -t iso9660 file.iso

Cfengine File Permission Blunder

While testing some changes to our sshd_config, I imported a copy of it from another host to our cfengine repository. After making adjustments to our cfagent.conf so that the test machine would be the only host to get this copy, I saved the file and ran cfagent.

Much to my surprise, the test server's sshd_config file would not update. I double checked my cfagent.conf, made sure the file was correct in the cfengine repository, etc. Looking at /var/log/messages, I saw

Sep 4 12:52:50 testhost cfengine:testhost[4333]: Network access to cfserver:/repository/cfengine/test/etc/ssh/sshd_config denied

This was confusing, because the testhost:/var/cfengine/cfagent.conf was getting the new configuration file. After banging my head for a while, I realized that the permissions of cfserver:/repository/cfengine/test/etc/ssh/sshd_config were incorrect.

-r-------- 1 root root 3050 Sep 4 10:53 sshd_config

Apparently, when importing the file over to my repository, I kept its original permissions. After chown'ing the file to cfengine, everything worked. Unfortunately, "Network access denied" is misleading. So, in the future, check you file perms to make sure cfengine can read the file.