Rants, raves and stuff from this dude's brain

The idea for having an encrypted drive is to secure information. What information is more important that email? Really, I don’t think much other information is quite as personal and important. So, with that I guess I’ve given away the reason I started looking at encryption for drives. I want to encrypt my emails in case some baddy takes my laptop. One negative that’s come of this is I the ability to search through the body of the email in Mail.app has stopped working. This is very disappointing and quite crippling for someone like me that believes in using search instead of infinite number of folder-inside-folder hierarchy to keep things organized. After much work I found a solution that works for me. I can’t promise it’ll work for you.

Let me start at the beginning and we’ll work our way to the actual fix. Make you\’re you are not running Mail.app. The first part is to move your Mail.app directory into an encrypted drive. From the previous post we will use the hdiutil but in a slightly different way. It has a flag to take an existing directory and create an image out of it. We will take our mail directory ~/Library/Mail and create a 4GB image out of it on to our Documents directory called MailDrive.sparseimage.

~]# cd ~/Documents
~]# hdiutil create -size 4G -srcfolder ~/Library/Mail -format SPARSE -encryption -stdinpass MailDrive

You should be prompted for a password. Every-time you mount this drive you will be prompted for this password.

Now move the old Mail directory out of the way. Don’t delete until after everything is working. Then mount the image you just created in it’s place. Again we will use the hdiutil command to mount the image where we want it to be mounted. You will be prompted for the password you set in the previous step.

~]# cd ~/Library/Mail
~]# mv Mail Mail.bak
~]# mkdir Mail
~]# hdiutil attach -mountpoint ~/Library/Mail ~/Documents/MailDrive.sparseimage

With this done, start up Mail.app. Everything should looks like it did before but, under the covers, your email is now encrypted! The problem is, most likely, your search function for “Entire Message” is probably broken now. So we can get to the crux of the matter. Mail.app relied on Spotlight for it’s search. Spotlight does not scan thought mounted drives. I don’t know the reasoning as I don’t work at Apple but my guess is they don’t want the index to glow to infinity every time you connect a little USB device or other things of the sort. But this is your email! You want to search though it! We can ask Spotlight to do exactly that. The mditul command seems to be the the way to plug into Spotlight, at least for drives.

The following first command will tell Spotlight to index the drive. While this is running you will see the Spotlight magnifying glass on the top right of your screen pulsating and thinking. Once this is done you will be able to search though your emails. The second command will just confirm the indexing is on. Maill.app does not need to be restarted for this to work which I think is pretty cool.

~]# mdutil ~/Library/Mail -i on
~]# mdutil ~/Library/Mail -s

Most FASTA files I come across with have multiple lines of sequence. Something like this from the yeast Amino Acid file from NCBI:

>gi|6381997|ref|NP_009302.2| Q0010p

MYYIMFLYNMLLIIILIFYSIVGVPIIIFNNNYYWDPDIFLFIIYYFIKFIIIFNLYLYYMINYIVYTPSGSPPGRGTYI

LLYNMLYSYNMFIDYVMKFITCVTYMYLMFWLLSPTPSPYYVSEVPVS

>gi|6226517|ref|NP_009303.1| Q0017p

MCATYMFNITVIITHPTPTLRTRGPGFVRNRDLYIYKYKSNLINNLNNMTYIL

>gi|6381998|ref|NP_009304.2| Q0032p

MLMMYMLFIMMKTYPMLSYHMMSYHIMLYTIMWYMKYSTYMRLWLLYKSYFIFIFIWTNNNYNNNYWYVTMLMNTYLYYN

MNIHFLTINKKFLYSL

Some sequences, specially the nucleotide ones, are way more than a couple of lines.  Being that I never know how long the sequence is going to be I’d much rather have a file with one line of FASTA description followed by a line of the sequence.  This way I know that all odd lines are comments and all even lines are sequence.  Makes my life much easier.

To do this I wrote a little PERL script that takes a file as a parameter and spits out the newly formatted string to standard out.

$info = "" ;
$sequence = "" ;

open (FILE, "<", $ARGV[0]) or die $! ;

while () {
        if ( $_ =~ /^>.*/ ) { # Found the start of an entry
                $info = $& ;
                print "$sequence\n" ;
                print "$info\n" ;
                $sequence = "" ;
        } elsif ( $_ =~ /^[[:alpha:]]*/ ) { # Sequence data
                $sequence = $& . $sequence;
        }
}
print "$sequence\n"

The output is something like this:

>gi|6381997|ref|NP_009302.2| Q0010p
LLYNMLYSYNMFIDYVMKFITCVTYMYLMFWLLSPTPSPYYVSEVPVSMYYIMFLYNMLLIIILIFYSIVGVPIIIFNNNYYWDPDIFLFIIYYFIKFIIIFNLYLYYMINYIVYTPSGSPPGRGTYI
>gi|6226517|ref|NP_009303.1| Q0017p
MCATYMFNITVIITHPTPTLRTRGPGFVRNRDLYIYKYKSNLINNLNNMTYIL
>gi|6381998|ref|NP_009304.2| Q0032p
MNIHFLTINKKFLYSLMLMMYMLFIMMKTYPMLSYHMMSYHIMLYTIMWYMKYSTYMRLWLLYKSYFIFIFIWTNNNYNNNYWYVTMLMNTYLYYN

In my opinion this file format is way more useful.

I’ve been hitting my head against a wall for the last couple of days .  My Cacti install just stopped working.  The UI was fine but no graphs were being created.  For a couple of days I thought it was the machine’s fault.  The hardware is old and I thought it was just time for the machine to kick the can.  I looked though systems logs and rebooted the machine a couple of times.  Ran less services on it thinking it might be there’s not enough RAM but nothing worked.  It was still broken.  I started to look into the cacti logs and found the following:

03/08/2009 04:10:03 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (10, 'traffic_out', '2009-03-08 04:10:03', '3677235031')'

I went online and looked for information on this error. Most posts I found just said to run a cacti script to repair the table. The scrip it cacti/cli/repair_database.php.  This started to get me worried as these posts implied something had happened to the database.  Looking back though the logs I went to check the database and, most importantly the poller_output table.  I manually checked the poller_output table and it gave me a nasty error.  Looking at the filesystem I saw that table was missing some files.  Never a good thing.
I couldn’t think of any good way to fix that table and from a quick search it seems there’s no really imporant information in there anyway. The best thing was to recreate the table. I looked at the cacti install and found the DB creation file and just copied the table I needed into a file called fixme.txt.

DROP TABLE IF EXISTS `poller_output`;
CREATE TABLE `poller_output` (
`local_data_id` mediumint(8) unsigned NOT NULL default '0',
`rrd_name` varchar(19) NOT NULL default '',
`time` datetime NOT NULL default '0000-00-00 00:00:00',
`output` text NOT NULL,
PRIMARY KEY  (`local_data_id`,`rrd_name`,`time`)
) TYPE=MyISAM;

With that file I just fed it to mysql.

mysql cacti < fixme.txt

Just to be on the safe side I rebooted the server.  Presto!  Now it works!

UPDATE: Just fixed a couple of typos.

Trying to find a way to make awstats and newsyslog work well together has been a problem.

newsyslog is a great way to take care of rotating files. It’s got some great knobs that can be tweaked. Logs can be rotated by date (different ways of determining date as well) or by size. There’s a great explanation of what can be done with newsyslog here.

awstats is a great way to get some very useful information from log files.  I mostly use it on Apache log files but it can also many more.  If you haven’t checked it out, you should.

The problem with trying to get these two programs to work is I haven’t been able to find a way to not lose data when a log gets rotated.  If a log gets rotated before awstats gets a chance to parse it that information is lost.  As far as I know newsyslog doesn’t have the capability of calling  script before it does the log rotation.  This is something that was trivial to do in Linux with logrotate.  But I don’t have it so I can’t use it.  The line in /etc/newsyslog.conf for me looks like this:

/var/log/http/apache.log 644     6       100     *       B       /var/run/httpd.pid      30

The first parameter is the log file to be rotated.  The second is the permissions of the file to be created (r-wr–r–).  The third is the number of copies to keep around (6).  The forth is the size in KB before the file gets rotated (100).  The fifth is a date or time for the file to be rotated.  In this case I just care about the size so no time is given.  The sixth is a set of predefiend letters that can do different things.  In this case the “B” is there so no comments are introduced into the log file.  The seventh and eith field are the path to the PID file for Apache and the signal to send to it to rotate the logs.  A much better explanation than I will ever be able to give on why the signal is 30 can be found here.

I did find that awstats has a very cool program called logresolvemerge.pl that can merge a bunch of files together.  Even better is the fact that it can be called in-line from awstats configuration in the LogFile variable.  My awstats config now looks like this:

LogFile="/var/www/awstats/tools/logresolvemerge.pl /var/log/http/apache.log* |"

This now uses the star as a wildcard and logresolvemerge.pl will just merge all the files it can find and return them to awstats.  The pipe at the end of the string is very important.  It doesn’t work without it.  Now, even if a file got rotated between runs of awstats, they still will be parsed!