Showing posts with label sysadmin. Show all posts
Showing posts with label sysadmin. Show all posts

Monday, January 7, 2013

logrotate with date extensions

My recent problem?  The version of logrotate (logrotate 3.7.1) I'm running on an old web server doesn't support the "dateext" option.  I wanted to have my Apache logs rotate with a date as its extension instead of ".#".  Here's a sample of my /etc/logrotate.d/httpd that does this, if you have the same problem.

# cat /etc/logrotate.d/httpd
/var/log/httpd/*log {
    rotate 5
    daily
    create
    missingok
    ifempty
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    lastaction
        DATE=`date +%F`;
        for i in `ls /var/log/httpd/*.1`
          do
            mv $i ${i%.1}-$DATE
          done
    endscript
}


Info on manipulating shell variable strings can be found via http://tldp.org/LDP/abs/html/string-manipulation.html.  You can see I'm removing the ".1" substring in the mv step so that my logs do not look like log.1-$DATE.

Friday, September 21, 2012

Lose your ssh key pub file?

I somehow lost my ssh key pub file, which is inconvenient when I want to copy it out to other hosts.  I could grab it from another host's authorized_key file, but wouldn't it be easier just to recreate it?  Here's how to do it.

[reamer77@myhost .ssh]$ ssh-keygen -y > id_rsa.pub
Enter file in which the key is (/home/reamer77/.ssh/id_rsa):
Enter passphrase:


Viola.  You should now have your pub file back without having to generate a new private and pub key file.

Wednesday, February 3, 2010

Quick and Dirty NetBackup Reporting

I needed a quick way to get the statuses of my NetBackup runs on a daily basis. This works for me, but maybe because there aren't that many servers (~30) that we are backing up. I'm sure there's nicer ways to do this, but if you want something quick and dirty, I have this line in my crontab.

/usr/openv/netbackup/bin/admincmd/bperror -U -backstat -s info -hoursago 24 | mail -s "Netbackup Report for `date +%D`" backup-admins@example.com

Wednesday, July 8, 2009

For DBA's, new blog to follow

An old co-worker of mine has started The Bungling DBA blog. Even though we now live on separate coasts, he and I are forever bound together as Lunch Twins. I'm not sure if this blog is also a bit of friendly ribbing at my expense or not, but I applaud him for starting a tech blog and look forward to his updates.

Friday, June 26, 2009

Server "uptime" bragging

I recently saw a blog post about someone showing their server having an uptime of over 400 days, and wanting other readers to reply with some of their larger uptimes. Quite a few people obliged, and the numbers were in the hundreds of days. This made me think, "Is this really a "good thing" anymore?"

Some questions that come to my mind when I see servers with long uptimes are

1.) Are patches being applied? There are a lot of security and performance updates that are released within a year. Some may not be critical, but are you being responsible and diligent in keeping your server up to date and secure?

2.) Does the server need to be up for so long because it is a single point of failure for a critical service? Hardware gets cheaper and cheaper, and many services can be loadbalanced or clustered. With the popularity of virtual machines, even more so. If this service experiences a failure, will your customers or users notice? How long will it take to restore its functionality?

3.) Do you know if the server will restart correctly in the event something causes a reboot? This could be unexpected, like a hardware or power failure; or expected, like applying kernel updates. Over a long period of time, a lot of small changes can happen that could cause startup scripts to break, but would go undetected until you have to restart. Or, your hardware just might not want to go through a restart for whatever whacky reason.

I guess what I'm saying is, having regular maintenance reboots aren't a "bad thing." Yeah, it used to look cool to have a server up for 600 days, but I don't think it's really worth it now.

Monday, January 12, 2009

Deleting a Solaris RAID created with Volume Manager

It seems most of my "howto" posts recently have been Solaris related. The main reason for that is that my Solaris admin knowledge is weak, requiring me to look up Solaris-specifc tasks. And when I learn how to do something, I like to share with others the answer, just in case they didn't either.

We have a Solaris 10 host in the lab that we do software tests on, and I somehow have been tapped to admin and be in charge of its OS. The production hosts we create for clients use RAID-1 with two disks, so the orignal install steps specified using the Sun Volume Manager using the different "meta" commands (metadb, metainit, metattach). Without much warning, they've gone and changed the install steps to use the simpler "raidctl" command. My problem now is that I have to reinstall the lab test host, and setting up the RAID with raidctl. However, I have no idea how to delete the previous RAID configuration. Thankfully, I came across this. It appears all I had to do was run "metaclear -a".

Tuesday, January 6, 2009

My take on "Which Unix to learn"

I came across this post on TaoSecurity today, with Richard Bejtlich's suggestions for an "Introduction to Unix." Like text editors, the discussion about which distribution of Unix to learn can be considered a "religious" argument. If you're a regular reader of his blog, it's no surprise he suggests FreeBSD. He does clarify and say if you're running a server, he suggests FreeBSD; and if you're running a desktop, he now prefers Ubuntu. And if you're still adamant about running Linux as a server, he suggests Debian.

My opinion differs, depending mainly on "why do you want to learn Unix?" If the answer is related to increasing your work/resume skill sets, I would have to disagree about using FreeBSD or any of its other BSD derivatives (Net, Open, etc.). In my experiences of being an admin or some other support role, I have yet to encounter a *BSD server. I'd encourage someone to use a distribution that they would encounter in a corporate environment. The Unix server OS's I've had to support have been Red Hat (now Fedora), Red Hat Enterprise Linux (also known as RHEL), and Sun Solaris. To avoid having to pay licensing fees, you could substitute CentOS for RHEL.

If I were asked the question, I would suggest a "major" Linux distribution or Solaris/Open Solaris instead. I would think their device names, software packages, and file system organization would help with familiarity when trying to translate the knowledge learned to a corporate environment. Although this may be an unfounded opinion, I also think that there are more support options and supported software using them, rather than using *BSD. I do agree with Bejtlich that if you want to run a Unix distribution on the desktop, to stick with Ubuntu, since it seems to "just work" when installed and there are less configuration headaches. Or you can just use a Mac if you want a Unix desktop (troll... and yes Mac could be considered a BSD variant).

Monday, January 5, 2009

Managing "To Do" items for work

I'm trying to decide the best way to manage my tasks, projects, and "mini-projects". I define mini-project as something larger than a task, but I'm my own manager and don't need to submit a project plan. Maybe I should've asked for Tim Limoncelli's "Time Management for System Administrators" as a Christmas gift.

The options I know of are:

"Remember the Milk" - Pros: Seems very extensible, geeky, multiple ways to manage, etc. Cons: I doubt my company would like me storing information with a third party.

Using "Tasks" in Outlook - Pros: This would be stored and backed up at work. Cons: I like to keep my interaction with Outlook at a minimum. I just don't like the interface for it, anyways.

Creating tickets in a case management system - Pros: I've done this at previous jobs creating cases assigned to myself for tasks and mini-projects. Cons: My employer takes its case tracking statistics seriously.

Personal Wiki - Pros: Uses a web browser to edit. Some Wiki's have version control and search capabilities. Cons: Some wiki's require running on a server and using heavyweight services (is that overkill?). Is it really the best tool for the job?

Right now, I'm using TiddlyWiki. I feel the interface is a little clunky, but does a lot of what I want it to do. For instance; it's lightweight (does not require a server or a database), can be portable (it's just files), only requires a web browser and access to its files, and has some searching capability.

I'd be interested to know what other people are using to manage their tasks and other assignments that are not necessarily part of the "everyday routine."

Tuesday, December 23, 2008

Linux Mag's Top 10 Sys Admin Articles of 2008

With 2008 coming to an end, everyone is starting to put out their "Top" lists. I saw the announcement today in my Inbox that Linux Magazine posted its Top 10 Sys Admin Articles of 2008. Unfortuantely, the links to articles requires registration, but it's free. The two articles on ZFS interest me the most, and I want to refresh myself on the "port knocking" article. I forget where I've heard about port knocking before. I'm pretty sure it was a couple years ago though.

It also made me realize I miss reading physical magazines over online copies. Then again, I rarely read the ones I get in the mail now. It's amazing how technology changes things.

On a totally unrelated note, I just wanted to share a good laugh about Matt Simmons' comment on a blog post about having multiple superusers on the VMware ESX server.

"Once, many many moons ago, probably near the peak of my danger curve, I got tired of su’ing all the time, so I just changed my uid to 0 in the passwd file. That was fun."

I can't explain why it struck me as funny as it did, but that was probably the funniest thing I've read today.

Friday, December 12, 2008

"find" and "Permission denied"

I was trying to run "find" on a directory, looking for a perl script. Because of which user I was running it as, I was getting a bunch of "Permission denied" errors.

I thought I could just tack on "| grep -v "Permission denied" at the end of my command, but no luck. I fruitlessly checked to see if there would be something in the find man page as well.

I came across this site: http://www.hypexr.org/linux_find_help.php

The correct answer to what I was trying to accomplish was

$ find / -name foo.bar -print 2>/dev/null

Monday, December 1, 2008

TaoSecurity's thoughts on the future

REF: http://taosecurity.blogspot.com/2008/11/dont-fight-future.html

The points he raises are interesting, and some seem very probable. The ones I think are interesting are

Prediction 1: VPN connections will disappear.
"Some of you might limit the type of connection to certain protocols, but why not just expose those protocols directly to the outside world and avoid the VPN altogether?"

This may have some merits, but it makes me uncomfortable. It's quite possible I may be looking at it from a narrow point of view of performing my job remotely, and not looking at normal day-to-day work. Some internal assets such as email, collaborative web tools (ex. wikis, Microsoft Share Point), and code repositories (ex. CVS, SVN) might be able to to be exposed with proper strength of authentication and encryption. However, I would think it would be difficult to do many remote administration tasks securely. Maybe the title implies that the VPN connections dramatically decrease, but not totally disappear?

Prediction 2: Intranets will disappear.

This prediction builds off the previous one. I suppose this is a semantics argument. The assets are still going to exist, but since they are exposed to the internet, they would no longer be considered intranet?

The other predictions are definitely plausible to happen in the future. It's a fairly quick and interesting read, so I'd recommend clicking the link above. He has other good posts too, so you may want to add him to your RSS feed if you haven't already.

Monday, November 24, 2008

Red Hat Satellite 5.2 released

In an earlier post, I mentioned I was unable to fit Red Hat Satellite or Fedora Spacewalk into our infrastructure. The previous version of Red Hat Satellite required Oracle 9i (we run 10g), and Spacewalk requires a derivative of RHEL/CentOS 5 (our standard is version 4). I saw this release today by Red Hat announcing Satellite version 5.2. It now supports Oracle 10g, and still runs on RHEL4 or RHEL5. Too bad everyone is hurting now and pulling the purse strings on their budgets. It would be tough for me to ask for a RHEL AS4 and a Satellite 5.2 license now.

I wonder how hard it would be to backport Spacewalk to RHEL4?

Monday, November 3, 2008

Should you learn vi?

There are so many arguments about which editor to use. Some people are quite passionate about their editor of choice. I came across this blog post today through Planet SysAdmin discussing why one person uses vi for system administration.

http://utcc.utoronto.ca/~cks/space/blog/sysadmin/WhyViForSysadmins

It pretty much sums up why it was stronly suggested to me to use vi at the beginning of my career. At that time, I only knew pico and a few commands in emacs; although now, I couldn't tell you how to edit a file in either. Over the years I've noticed the author's point is true. Almost any unix type system you encounter, it will have at least vi installed. It's definitely handy to have a basic proficiency with vi if you do any command line *NIX administration.

So, while you get your caffeine fix at work, this could come in handy as well.

http://www.thinkgeek.com/homeoffice/mugs/7bbe/

Friday, October 31, 2008

Howto Quickly wipe a disk in Solaris

I've had to wipe disks in Solaris because of re-installs or other reconfiguration. I was disappointed that using "format" would take hours. Obviously I'm using the wrong tool.

I came across this thread.

http://www.linuxquestions.org/questions/solaris-opensolaris-20/quick-format-a-hard-drive-496294/

It's strange, but I'm noticing people are posting Solaris questions on a Linux website. I wonder how Solaris admins would've felt 10 years ago if people were posting Linux questions on a Solaris site or Usenet group. But, I digress.

The response is to use "newfs" instead of "format", and it seems to work so far. Thankfully, I learn something new every day.

Monday, October 27, 2008

Careful where you copy/paste

I got an IM from an old coworker discussing that he was trying to answer someone's question about an in-house app and which shared libraries it was using. (NOTE: some of the info has been sanitized to protect the "innocent")

He ran the following command, and then copy/pasted the output

[root@customftp root]# ldd /nfs/customapp/bin/daemon
libkstat.so.1 => /usr/lib/libkstat.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libsched.so.1 => /usr/lib/libsched.so.1
libaio.so.1 => /usr/lib/libaio.so.1
librt.so.1 => /usr/lib/librt.so.1
libthread.so.1 => /usr/lib/libthread.so.1
libstdc++.so.6 => /usr/lib/libstdc++.so.6
libm.so.1 => /usr/lib/libm.so.1
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1
libc.so.1 => /usr/lib/libc.so.1
libmp.so.2 => /usr/lib/libmp.so.2
libmd5.so.1 => /usr/lib/libmd5.so.1
/usr/platform/SUNW,Sun-Fire-V210/lib/libc_psr.so.1
/usr/platform/SUNW,Sun-Fire-V210/lib/libmd5_psr.so.1


He then tried to copy paste it into an email, but didn't realize his SSH session was still the active window.

[root@customftp root]# nfs/customapp/bin/daemon
bash: nfs/customapp/bin/daemon: No such file or directory
[root@customftp root]# libkstat.so.1 => /usr/lib/libkstat.so.1
bash: libkstat.so.1: command not found
[root@customftp root]# libnsl.so.1 => /usr/lib/libnsl.so.1
bash: libnsl.so.1: command not found
Connection to customftp closed by remote host.
Connection to customftp closed.


It took me a few minutes to see why he was sending me this IM. My initial reaction was "OK, so you accidently copy/pasted into the wrong window, and somehow you got kicked out." Before reading further, take a look back to see what was so bad about it. Maybe you'll pick it up faster than I did.

Ready?

The output of ldd in a copy/paste was interpretted as piping the output of bad commands to his library files. He basically nuked files he needed in /usr/lib. I initally laughed out loud, but then realized that it was a very unfortunate situation. He said it took him about 2 hours to recover from this mistake.

Thursday, October 23, 2008

Administering Windows from the command line

I sometimes forget that you can do a lot via the command line with Windows. When Unix Admins complain about having to Windows administration, my usual response is "Come on, it's clicky-clicky. Just click around and you'll figure it out." In the past two days, I came across two blog posts that reminded me that you're not necessarily stuck with having to use GUI programs to administer Windows.

The first blog post I came across was posted today about an alternative to running programs remotely on Windows hosts.

http://standalone-sysadmin.blogspot.com/2008/10/issue-remote-commands-to-windows.html

Apparently SysInternals distributed a group of tools called PsTools, which included a utility called PsExec. PsExec allows a user to remotely execute commands on a Windows host. I'm not sure what the security implications are using these utilities. Any time someone uses "telnet" in their description of remote administration, it makes me a bit nervous.

Less than an hour later, I came across this blog post that was written yesterday that makes for a nice supplement to the Standalone Sysadmin post.

http://taosecurity.blogspot.com/2008/10/what-to-do-on-windows.html

It points to another blog discussing useful Windows commands that can be running from the DOS command prompt. I knew all about "net start", "net stop", and some other useful commands to use, but it definitely covers some I didn't know (ex. netsh).

Tuesday, October 7, 2008

Segmentation faults when formatting a disk in Solaris

I've had an old Sun Ultra 5 under my desk since I've started at the current job. Instead of keeping it as my footstool, I decided to fire it up and see what happens. Needless to say, this has been a timesuck. I could go into all the details and other mistakes on my part, but it isn't very interesting.

What I didn't know would be a problem was trying to install Solaris 10 after I had impatiently installed NetBSD. Apparently the Solaris installer doesn't know how to recognize the disk formatted by NetBSD. When running "format", it just segfaults and dumps core.

After some Google searches, I came up on this thread.

http://groups.google.com/group/comp.unix.solaris/browse_thread/thread/8107078ff8e2174d?pli=1

after running

dd if=/dev/zero of=/dev/rdsk/c0t0d0s0 bs=512 count=4

I was able to run format on the disk, and run the Solaris 10 install.

Friday, October 3, 2008

Confused by an SSL related blog post

I subscribe to the Planet SysAdmin, an aggregation of Sys Admin related blogs via RSS (http://planetsysadmin.com/). Today, I came across a post that really confused me on what the author was trying to accomplish or explain.

http://utcc.utoronto.ca/~cks/space/blog/web/RevocableSSLNames

I'm hoping that one of you that is smarter than me gets it and maybe can explain it to me. Using his example of online banking, I look at my bank and it looks like the cert is probably handled by a load balancer; or maybe explained in his second case, an HTTPS Proxy server. He discusses revoking the cert by changing the host name.

A) How would that help when I have a bookmark to https://onlinebanking.example.com?
B) and wouldn't you revoke the compromised cert with the certificate issuer anyways?

The only thing I could think of is he's discussing SSL on both ends of his proxy (web client to proxy, proxy to application server). But in that scenario, the only thing that is apparent to the end user is the SSL cert to the proxy which would have to be revoked by the certificate issuer if compromised.

Or, I'm missing something entirely...

Friday, August 29, 2008

Update on useful UNIX tools for Win32

A co-worker sent out this link that has ways to "turbocharge" PuTTY.

Finally looks like I can get my tabbed SSH/terminal sessions. It also mentions puttycyg (affirmation, baby!).

UPDATE (8/29/2008 10:33 EDT):
My comments so far trying out tabbed putty/ssh sessions.
- It doesn't look like PuttyCM supports Puttycyg, which is disappointing. If you use the original putty.exe, it does seem to work how I'd want it to.
- The Putty sessions aren't rolled up into the Putty Tabs program (they still show individually in my taskbar). This doesn't improve my workspace much.

Monday, May 5, 2008

Problem with slave BIND server

After one of the sysadmins at work patched our RHEL4 DNS servers, I noticed our slave server was not successfully receiving updated zone files from the master.

Looking at the logs, I kept seeing the following error

named[5182]: dumping master file: tmp-XXXXTCPn1l: open: permission denied

I installed strace on the server and tried to find where it was trying to write to and see if somehow the directory permissions were incorrect. Unfortunately, strace just showed the "tmp-XXXXX" file failing, but not the directory location.

After some investigation on Google, I found this site. It explained that an updated version of BIND required changes to the named.conf file for slave DNS servers. I then modified our named.conf so that the setting for the zone file was

file "slaves/example.com.zone";

instead of the

file "example.com.zone";

I'm not sure how it worked in the past. Perhaps Red Hat backported an update in BIND that was not initially in RHEL4? I don't know what the Update version was before the patching, else I could probably dig through release notes. However, if you're seeing strangeness with your slave BIND servers, I'd check to make sure your named.conf isn't out of date.