Cron entry mistakes

This mistake came at the expense of one of my old co-workers, who decided to share this with me over Instant Messenger. I got a good laugh at his misfortune. He wanted to set up a cron job that would restart a poorly performing service every 6 hours. This is the cron entry he added.

* */6 * * * service_restart.sh

The lead developer, whose app depends on that service, came up to him wondering why his app was having outages. What my colleague should have configured was

0 */6 * * * service_restart.sh

His cron entry was restarting the service every 6 hours, and then restarting it every minute for an hour during that sixth hour. Brilliant! Although I guess I shouldn't throw stones in glass houses.

Useful UNIX tools for Win32

OK, it's been a while since I've posted. Being laid off and unemployed kind of hurts writing about work experiences. However, don't worry. I'm now at a new job in a new town.

Obtaining work responsibilities have been slow, so I've been reading and playing with software that I hope will help me in making my job easier later. One area I've been reading and trying to increase my knowledge in is remote access, predominantly SSH. I've found the following to be useful.

SSH Agent
I've known about shared-key authentication with SSH for a while, but I've been lame and not applied a passphrase to my key. Honestly, I wanted a quick way to log in, and skip the password. However, I finally was exposed to ssh-agent, which allows you to cache your key's passphrase so you only have to enter it once. After that, when you ssh into a host, ssh-agent will take care of your passphrase for you.

I'm using Cygwin for my shell on my PC, so I'm using the ssh-agent that comes with Cygwin. I hear you can use a program called Pageant to perform this for you PuTTY users.

• Link to Cygwin and ssh-agent
• Link to HOWTO for Pageant with PuTTY
  

Puttycyg
I hate the command window that Cygwin defaults to. The copy/paste is terrible, as I think it borrows from cmd.exe. So, I found Puttycyg that gives the Cygwin command prompt a better look and feel. Now I actually get a copy/paste behavior that I like.

Poderosa
I've been spoiled with Mac OSX's iTerm for having tabbed terminal windows. Unfortunately, PuTTY and Cygwin spawn new windows, and their management gets cluttered. Right now, I'm trying Poderosa, mainly because it appears to be open source and managed by the Apache license. It tabs your Cygwin sessions, and apparently other SSH/Telnet sessions as well (but I haven't tried the latter). I found another alternative in Wintabber, but I haven't tried it yet. It appears to be free and have more features I'd be interested in, but I believe is closed source.

I'd be interested in your experiences with tabbed command/shell windows and other useful remote access tricks.

Problem with slave BIND server

After one of the sysadmins at work patched our RHEL4 DNS servers, I noticed our slave server was not successfully receiving updated zone files from the master.

Looking at the logs, I kept seeing the following error

named[5182]: dumping master file: tmp-XXXXTCPn1l: open: permission denied

I installed strace on the server and tried to find where it was trying to write to and see if somehow the directory permissions were incorrect. Unfortunately, strace just showed the "tmp-XXXXX" file failing, but not the directory location.

After some investigation on Google, I found this site. It explained that an updated version of BIND required changes to the named.conf file for slave DNS servers. I then modified our named.conf so that the setting for the zone file was

file "slaves/example.com.zone";

instead of the

file "example.com.zone";

I'm not sure how it worked in the past. Perhaps Red Hat backported an update in BIND that was not initially in RHEL4? I don't know what the Update version was before the patching, else I could probably dig through release notes. However, if you're seeing strangeness with your slave BIND servers, I'd check to make sure your named.conf isn't out of date.

What We Should Strive Toward In Operations

I generally keep up with the O'Reilly Sysadmin blog (http://www.oreillynet.com/sysadmin/blog/), although I sometimes find the content and the frequency of updates disappointing. To be fair, you would probably say the same thing about this blog . However, today, they had a blog entry that pointed to another blog entry about "Operations Mantras" that I found interesting.

O'Reilly Link:
http://www.oreillynet.com/sysadmin/blog/2008/02/operations_mantras.html

Operations Mantra Link:
http://dormando.livejournal.com/484577.html

It's a long read, and I'm not totally finished with it, but I found some useful thoughts and some points that give me self-affirmation that I'm not a total screw-up when it comes to system administration.

Some examples of points that I'm interested in following up on are

- In "Understand your data storage and databases", it suggests to investigate starling and Gearman
- The topics brought up in "Asynchronous Jobs"
- In "Use source control", avoid SVN and use Git or Mercurial instead (I'm assuming because these use distributed repositories for version control instead of a central one)

There are also some interesting technical and non-technical theories, practices, and procedures mentioned as well.

Dell Server OMSA Reporting

I used to think that Dell OpenManage Server Administrator (OMSA) was worthless. I was wrong. I apologize.

I've found for me it's the best way to do actual hardware monitoring. Although there are probably "Official Dell Best Practices" on using and implementing OMSA, I've just gone and installed it and then accessed the machine via my web browser (https://hostname:1311). However, today I found you can actually get some good stuff via the command line using "omreport". Dell's command line documentation for it is located at

http://support.dell.com/support/edocs/software/svradmin/5.2/en/cli/html/report.htm#wp1068065

For instance, I can find out really quick what the status is of my hardware, except for disk/storage related hardware.

# /opt/dell/srvadmin/oma/bin/omreport chassis
Health

Main System Chassis

SEVERITY : COMPONENT
Ok : Fans
Ok : Intrusion
Ok : Memory
Ok : Power Supplies
Ok : Processors
Ok : Temperatures
Ok : Voltages
Ok : Hardware Log

Also, someone wrote a nagios plugin that executes this as well.

http://www.nagiosexchange.org/DELL_Server.61.0.html?&tx_netnagext_pi1%5Bp_view%5D=432

To check storage related, you can run the following command.

# /opt/dell/srvadmin/oma/bin/omreport storage pdisk controller=0
List of Physical Disks on Controller PERC 4e/Di (Embedded)

Controller PERC 4e/Di (Embedded)
ID : 0:0
Status : Ok
Name : Physical Disk 0:0
State : Online
Failure Predicted : No
Progress : Not Applicable
Type : SCSI
Capacity : 68.24 GB (73274490880 bytes)
Used RAID Disk Space : 68.24 GB (73274490880 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare : No
Vendor ID : FUJITSU
Product ID : MAW3073NC
Revision : 5803
Serial No. : DAL3P6200PR8
Negotiated Speed : 320
Capable Speed : 320
Manufacture Day : Not Available
Manufacture Week : Not Available
Manufacture Year : Not Available
SAS Address : Not Available

ID : 0:1
Status : Ok
Name : Physical Disk 0:1
State : Online
Failure Predicted : No
Progress : Not Applicable
Type : SCSI
Capacity : 68.24 GB (73274490880 bytes)
Used RAID Disk Space : 68.24 GB (73274490880 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare : No
Vendor ID : FUJITSU
Product ID : MAW3073NC
Revision : 5803
Serial No. : DAL3P6200PK3
Negotiated Speed : Not Available
Capable Speed : Not Available
Manufacture Day : Not Available
Manufacture Week : Not Available
Manufacture Year : Not Available
SAS Address : Not Available

My Controller ID is 0. This can be found by running "omreport storage controller". Storage commands can be found on

http://support.dell.com/support/edocs/software/svradmin/5.2/en/cli/html/storage.htm#wp1082304

Clean cabling in the datacenter

Often, clean cabling in the datacenter is neglected; especially by yours truly. It's a hassle, it takes time, and usually you cut corners because you're trying to get other tasks done. However, I was sent this blog post that shows good cabling. I may have re-think about investing the time to do cabling right.

http://royal.pingdom.com/?p=240

RE: iPhone MAC address post

I ran into this post today from TUAW, and there is now an iPhone app that can switch your MAC address.

http://www.tuaw.com/2008/01/14/tuaw-responds-mac-addresses-on-the-iphone/

I guess that would make it more difficult to keep users from using the corporate wireless network with iPhones, unless you can implement a 802.1 solution.

What I've done at work is only allow wireless into our Guest VLAN. If they need to access corporate assets (printers, shared drives, etc.), then they need to use VPN after connecting to the
wireless network.