logrotate with date extensions

My recent problem?  The version of logrotate (logrotate 3.7.1) I'm running on an old web server doesn't support the "dateext" option.  I wanted to have my Apache logs rotate with a date as its extension instead of ".#".  Here's a sample of my /etc/logrotate.d/httpd that does this, if you have the same problem.

# cat /etc/logrotate.d/httpd
/var/log/httpd/*log {
    rotate 5
    daily
    create
    missingok
    ifempty
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    lastaction
        DATE=`date +%F`;
        for i in `ls /var/log/httpd/*.1`
          do
            mv $i ${i%.1}-$DATE
          done
    endscript
}

Info on manipulating shell variable strings can be found via http://tldp.org/LDP/abs/html/string-manipulation.html.  You can see I'm removing the ".1" substring in the mv step so that my logs do not look like log.1-$DATE.

Lose your ssh key pub file?

I somehow lost my ssh key pub file, which is inconvenient when I want to copy it out to other hosts.  I could grab it from another host's authorized_key file, but wouldn't it be easier just to recreate it?  Here's how to do it.

[reamer77@myhost .ssh]$ ssh-keygen -y > id_rsa.pub
Enter file in which the key is (/home/reamer77/.ssh/id_rsa):
Enter passphrase:

Viola.  You should now have your pub file back without having to generate a new private and pub key file.

Quick and Dirty NetBackup Reporting

I needed a quick way to get the statuses of my NetBackup runs on a daily basis. This works for me, but maybe because there aren't that many servers (~30) that we are backing up. I'm sure there's nicer ways to do this, but if you want something quick and dirty, I have this line in my crontab.

/usr/openv/netbackup/bin/admincmd/bperror -U -backstat -s info -hoursago 24 | mail -s "Netbackup Report for `date +%D`" backup-admins@example.com

Man page reading tip

I usually keep my ssh windows pretty small, but it makes them a pain to read man pages. The way I read man pages now is using Google and finding man pages posted online. That way, it's as easy as reading any other web page.

For Linux, I use the search string (minus the quotes) "man linux command", and it usually pulls up the appropriate man page on http://linux.die.net.

For Solaris, I'll use the search string (minus the quotes) "man sunos command", since the syntax or switches of the Solaris command may be slightly different than the Linux one.

Another bonus is that commands and configurations in the "SEE ALSO" section are usually hyperlinked to the corresponding web entry. There are probably browser plugins or toolbars that will accomplish the same thing, but this is universal and lightweight.

Modifications to stock CUPS server

I've been tasked with setting up a Unix print server, since the current one runs unmanaged on a PC beneath someone's desk. Since we use RHEL4 for our servers, obviously this is going to be using CUPS. Setting up CUPS isn't too painful. The web interface is pretty easy to use. My concern though is when you click on the Administration link and log in, it continues to use plain-text HTTP to pass the credentials.

Here are the few things I've changed to make me feel a little bit less uneasy.

First, I created a self-signed SSL certificate and copied the key and crt to /etc/cups/ssl.

Then, I enabled the following in cupsd.conf

ServerCertificate /etc/cups/ssl/server.crt
ServerKey /etc/cups/ssl/server.key
SSLPort 443

I still have the stock port 631 listening as well.

Finally, I modified the index HTML page for the CUPS service. I found this located in /usr/share/cups/doc/index.html. I edited the two hyperlinks for administrator so that it pointed to "https://print.example.com/admin". I know this won't stop people from using "http://print.example.com:631/admin", but at least if they are just clicky-clicky, it will have them log into the administrative interface using HTTPS. I'm not sure how to force users to use HTTPS when accessing the administrative page in CUPS, but at least this is a start.

Now, I don't feel like my usernames and passwords are floating around in the clear when it comes to CUPS.

My take on "Which Unix to learn"

I came across this post on TaoSecurity today, with Richard Bejtlich's suggestions for an "Introduction to Unix." Like text editors, the discussion about which distribution of Unix to learn can be considered a "religious" argument. If you're a regular reader of his blog, it's no surprise he suggests FreeBSD. He does clarify and say if you're running a server, he suggests FreeBSD; and if you're running a desktop, he now prefers Ubuntu. And if you're still adamant about running Linux as a server, he suggests Debian.

My opinion differs, depending mainly on "why do you want to learn Unix?" If the answer is related to increasing your work/resume skill sets, I would have to disagree about using FreeBSD or any of its other BSD derivatives (Net, Open, etc.). In my experiences of being an admin or some other support role, I have yet to encounter a *BSD server. I'd encourage someone to use a distribution that they would encounter in a corporate environment. The Unix server OS's I've had to support have been Red Hat (now Fedora), Red Hat Enterprise Linux (also known as RHEL), and Sun Solaris. To avoid having to pay licensing fees, you could substitute CentOS for RHEL.

If I were asked the question, I would suggest a "major" Linux distribution or Solaris/Open Solaris instead. I would think their device names, software packages, and file system organization would help with familiarity when trying to translate the knowledge learned to a corporate environment. Although this may be an unfounded opinion, I also think that there are more support options and supported software using them, rather than using *BSD. I do agree with Bejtlich that if you want to run a Unix distribution on the desktop, to stick with Ubuntu, since it seems to "just work" when installed and there are less configuration headaches. Or you can just use a Mac if you want a Unix desktop (troll... and yes Mac could be considered a BSD variant).

Linux Mag's Top 10 Sys Admin Articles of 2008

With 2008 coming to an end, everyone is starting to put out their "Top" lists. I saw the announcement today in my Inbox that Linux Magazine posted its Top 10 Sys Admin Articles of 2008. Unfortuantely, the links to articles requires registration, but it's free. The two articles on ZFS interest me the most, and I want to refresh myself on the "port knocking" article. I forget where I've heard about port knocking before. I'm pretty sure it was a couple years ago though.

It also made me realize I miss reading physical magazines over online copies. Then again, I rarely read the ones I get in the mail now. It's amazing how technology changes things.

On a totally unrelated note, I just wanted to share a good laugh about Matt Simmons' comment on a blog post about having multiple superusers on the VMware ESX server.

"Once, many many moons ago, probably near the peak of my danger curve, I got tired of su’ing all the time, so I just changed my uid to 0 in the passwd file. That was fun."

I can't explain why it struck me as funny as it did, but that was probably the funniest thing I've read today.

"find" and "Permission denied"

I was trying to run "find" on a directory, looking for a perl script. Because of which user I was running it as, I was getting a bunch of "Permission denied" errors.

I thought I could just tack on "| grep -v "Permission denied" at the end of my command, but no luck. I fruitlessly checked to see if there would be something in the find man page as well.

I came across this site: http://www.hypexr.org/linux_find_help.php

The correct answer to what I was trying to accomplish was

$ find / -name foo.bar -print 2>/dev/null

Are new releases of Slackware newsworthy?

From the Slashdot RSS feed, I saw "Slackware 12.2 Released", and it made me wonder if this is actually newsworthy. Granted, Slackware was technically the first Linux distribution I ever installed. It came with a "Linux for Dummies" book I bought in 1997, when I was starting to become more interested in doing more with computers than basic every day tasks (email, web surfing, word processing, games). However, my experience with Slackware was short lived because I was still living at home, and my parents used AOL for Internet access.

Nowadays, it seems Slackware is revered by older users, but isn't used. To me, it falls into the same area as using the Eudora mail client, or until recently, Netscape web browser. It seems Ubuntu (and its variants) and Fedora dominate the Linux desktop, with a smattering of Debian users; and CentOS, Red Hat Enterprise Linux, and Ubuntu (somewhat surprising to me), with some Gentoo diehards, leading the Linux servers. I think this may be the least used "mainstream" *NIX distribution available. Except for nostalgia, what reasons do users install Slackware for?

Not that I'm discouraging further development of Slackware, or trying to insult their users. I'm just questioning their relevance in today's IT news, unless it's just a slow news day :)

Red Hat Satellite 5.2 released

In an earlier post, I mentioned I was unable to fit Red Hat Satellite or Fedora Spacewalk into our infrastructure. The previous version of Red Hat Satellite required Oracle 9i (we run 10g), and Spacewalk requires a derivative of RHEL/CentOS 5 (our standard is version 4). I saw this release today by Red Hat announcing Satellite version 5.2. It now supports Oracle 10g, and still runs on RHEL4 or RHEL5. Too bad everyone is hurting now and pulling the purse strings on their budgets. It would be tough for me to ask for a RHEL AS4 and a Satellite 5.2 license now.

I wonder how hard it would be to backport Spacewalk to RHEL4?

Should you learn vi?

There are so many arguments about which editor to use. Some people are quite passionate about their editor of choice. I came across this blog post today through Planet SysAdmin discussing why one person uses vi for system administration.

http://utcc.utoronto.ca/~cks/space/blog/sysadmin/WhyViForSysadmins

It pretty much sums up why it was stronly suggested to me to use vi at the beginning of my career. At that time, I only knew pico and a few commands in emacs; although now, I couldn't tell you how to edit a file in either. Over the years I've noticed the author's point is true. Almost any unix type system you encounter, it will have at least vi installed. It's definitely handy to have a basic proficiency with vi if you do any command line *NIX administration.

So, while you get your caffeine fix at work, this could come in handy as well.

http://www.thinkgeek.com/homeoffice/mugs/7bbe/

Update on useful UNIX tools for Win32

A co-worker sent out this link that has ways to "turbocharge" PuTTY.

Finally looks like I can get my tabbed SSH/terminal sessions. It also mentions puttycyg (affirmation, baby!).

UPDATE (8/29/2008 10:33 EDT):
My comments so far trying out tabbed putty/ssh sessions.
- It doesn't look like PuttyCM supports Puttycyg, which is disappointing. If you use the original putty.exe, it does seem to work how I'd want it to.
- The Putty sessions aren't rolled up into the Putty Tabs program (they still show individually in my taskbar). This doesn't improve my workspace much.

Unhappy with Red Hat

Right now, I'm not happy with Red Hat. At least not with their sales department.

My group is interested in the capabilities of Red Hat Satellite Server, so I was going to try to obtain a trial license and download it. As far as I know, we don't have any RHEL licenses in our department. I'm sure somewhere in the company we do, since we have thousands of employees, but I sincerely doubt I'll be able to track that account down, and even then, doubt I can use it for access to the Red Hat Network. So, I applied for a trial license of RHEL and was hoping through that entry into RHN, I could also get a trial of Satellite Server. I still couldn't find that as a download, so I've made a couple of attempts to submit requests through their website on how I could evaluate their Satellite Server. No response. I would think sales people would jump at the chance to talk to a warm body that offered up their contact information.

I know there's Spacewalk, and I am interested in that. Unfortunately, it's only supported on RHEL5 compatible OS's. Our standard hasn't included RHEL5 yet; and they aren't open to "exceptions".

Any suggestions? Are there other resale vendors I could talk to that would let me have a 30 day evaluation to download and run Satellite for RHEL4?

Cron entry mistakes

This mistake came at the expense of one of my old co-workers, who decided to share this with me over Instant Messenger. I got a good laugh at his misfortune. He wanted to set up a cron job that would restart a poorly performing service every 6 hours. This is the cron entry he added.

* */6 * * * service_restart.sh

The lead developer, whose app depends on that service, came up to him wondering why his app was having outages. What my colleague should have configured was

0 */6 * * * service_restart.sh

His cron entry was restarting the service every 6 hours, and then restarting it every minute for an hour during that sixth hour. Brilliant! Although I guess I shouldn't throw stones in glass houses.

Problem with slave BIND server

After one of the sysadmins at work patched our RHEL4 DNS servers, I noticed our slave server was not successfully receiving updated zone files from the master.

Looking at the logs, I kept seeing the following error

named[5182]: dumping master file: tmp-XXXXTCPn1l: open: permission denied

I installed strace on the server and tried to find where it was trying to write to and see if somehow the directory permissions were incorrect. Unfortunately, strace just showed the "tmp-XXXXX" file failing, but not the directory location.

After some investigation on Google, I found this site. It explained that an updated version of BIND required changes to the named.conf file for slave DNS servers. I then modified our named.conf so that the setting for the zone file was

file "slaves/example.com.zone";

instead of the

file "example.com.zone";

I'm not sure how it worked in the past. Perhaps Red Hat backported an update in BIND that was not initially in RHEL4? I don't know what the Update version was before the patching, else I could probably dig through release notes. However, if you're seeing strangeness with your slave BIND servers, I'd check to make sure your named.conf isn't out of date.

Dell Server OMSA Reporting

I used to think that Dell OpenManage Server Administrator (OMSA) was worthless. I was wrong. I apologize.

I've found for me it's the best way to do actual hardware monitoring. Although there are probably "Official Dell Best Practices" on using and implementing OMSA, I've just gone and installed it and then accessed the machine via my web browser (https://hostname:1311). However, today I found you can actually get some good stuff via the command line using "omreport". Dell's command line documentation for it is located at

http://support.dell.com/support/edocs/software/svradmin/5.2/en/cli/html/report.htm#wp1068065

For instance, I can find out really quick what the status is of my hardware, except for disk/storage related hardware.

# /opt/dell/srvadmin/oma/bin/omreport chassis
Health

Main System Chassis

SEVERITY : COMPONENT
Ok : Fans
Ok : Intrusion
Ok : Memory
Ok : Power Supplies
Ok : Processors
Ok : Temperatures
Ok : Voltages
Ok : Hardware Log

Also, someone wrote a nagios plugin that executes this as well.

http://www.nagiosexchange.org/DELL_Server.61.0.html?&tx_netnagext_pi1%5Bp_view%5D=432

To check storage related, you can run the following command.

# /opt/dell/srvadmin/oma/bin/omreport storage pdisk controller=0
List of Physical Disks on Controller PERC 4e/Di (Embedded)

Controller PERC 4e/Di (Embedded)
ID : 0:0
Status : Ok
Name : Physical Disk 0:0
State : Online
Failure Predicted : No
Progress : Not Applicable
Type : SCSI
Capacity : 68.24 GB (73274490880 bytes)
Used RAID Disk Space : 68.24 GB (73274490880 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare : No
Vendor ID : FUJITSU
Product ID : MAW3073NC
Revision : 5803
Serial No. : DAL3P6200PR8
Negotiated Speed : 320
Capable Speed : 320
Manufacture Day : Not Available
Manufacture Week : Not Available
Manufacture Year : Not Available
SAS Address : Not Available

ID : 0:1
Status : Ok
Name : Physical Disk 0:1
State : Online
Failure Predicted : No
Progress : Not Applicable
Type : SCSI
Capacity : 68.24 GB (73274490880 bytes)
Used RAID Disk Space : 68.24 GB (73274490880 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare : No
Vendor ID : FUJITSU
Product ID : MAW3073NC
Revision : 5803
Serial No. : DAL3P6200PK3
Negotiated Speed : Not Available
Capable Speed : Not Available
Manufacture Day : Not Available
Manufacture Week : Not Available
Manufacture Year : Not Available
SAS Address : Not Available

My Controller ID is 0. This can be found by running "omreport storage controller". Storage commands can be found on

http://support.dell.com/support/edocs/software/svradmin/5.2/en/cli/html/storage.htm#wp1082304

Couldn't Start Apache On Non-standard Ports

I was trying to configure Apache 2.x to listen on non-standard ports (ex. 8001) and was getting "permission denied ... unable to open logs" errors. This was on a Red Hat Enterprise Linux 4 ES server that I had just built, and a fresh install of httpd using up2date. Strangely, if Apache was configured to listen on port 80, it would start with no problems.

Apparently, if you do not disable SELinux in your kickstart file, it is enabled by default. SELinux will not allow httpd to start on the high ports I was specifying. Since I currently do not use SELinux in my environment, I have no qualms disabling it. To disable it (RHEL4), I modified /etc/sysconfig/selinux and configured "SELINUX=disabled". This requires a reboot to take effect.

Setting up LVM in RHEL kickstart file

Another piece of fun today...

Task:
Set up and configure a linux box with the following partition information.

Non-LVM
/boot - 100MB

LVM
/ - 9GB
/opt - 12GB
/var - 8GB
swap - 8GB

Kickstart Entries:
clearpart --all
part /boot --fstype ext3 --size=100 --ondisk=sda
part pv.4 --size=0 --grow --ondisk=sda
volgroup VolGroup00 pv.4
logvol / --fstype ext3 --name=root --vgname=VolGroup00 --size=9216
logvol /opt --fstype ext3 --name=usr --vgname=VolGroup00 --size=12228
logvol /var --fstype ext3 --name=var --vgname=VolGroup00 --size=8192
logvol swap --fstype swap --name=swap --vgname=VolGroup00 --size=8192

My big mistake was trying to set the "pesize" in the volgroup line. Apparently, it will assume to use all if not set, which is what I wanted. RedHat documentation about kickstart options can be found here.