Archive for December 2009

Fetching and (semi) FW’ing OWA Mail to Gmail Using msfetch

In the world of web, sometimes the web just isn’t convenient!  This is true when you are *forced* to use the web to check email.  My latest job only allows me to check email using Outlook Web Access or OWA.  Even worse, they removed the ability to forward emails and IMAP/POP access.

There are a couple of solutions out there but I couldn’t get any of them to work … in comes msfetch ( http://www.isrcomputing.com/msfetch/).

When I initially installed the tool for (outlook 2007) on my linux machine, I had issues getting it working.  Huge thanks to Paul (owner) for working with me.

In these examples I will be using the variables:

  • username: myusername
  • email: myuser.name@mydomain.com

Take note that the username and email is different!  This will be the root of our issues below.

First thing I had to do was download and install the scripts (took all of 5 seconds).  Then we need to edit the .msfetchrc file.  Here is an example of a WORKING copy … not the one that gave me problems.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
MAILSERVER=email.mydomain.com
MAILPROT=https
MAILUSER=myusername@mydomain.com
MAILDOMAIN=
MAILCRED=mypassword123
MAILMETHOD=owa2007
MAILBOXFILE=/tmp/msfetch.mbox
MAILOUTBOX=/tmp/msfetch.outbox
MAILHISTORY=/tmp/msfetch.history

#Exclude e-mail collection from the folders below
EXCLUDE=RSS%20Feeds
EXCLUDE=Tasks
EXCLUDE=Calendar
EXCLUDE=Deleted%20Items
EXCLUDE=Outbox
EXCLUDE=Sent%20Items

turn on DEBUG within the msfetchrc script.  This was easy to do.  Simply uncomment out the DEBUG line near the top.

Once this was done I executed and got these errors near the bottom:

1
2
3
4
5
6
7
8
9
10
11
12
Login successful!
-------------------------------------------------------------
Listing E-Mail Folders...
Query: SELECT "DAV:id", "DAV:href" FROM
"https://email.mydomain.com/exchange/myusername/" WHERE "DAV:ishidden" = false
AND "DAV:isfolder" = true
Fetch folder listing failed (https://email.mydomain.com/exchange/myusername/)...
440 Login Timeout
Number of folders: 0
There are no folders to fetch.
Started:   2009/12/28 17:06:58
Completed: 2009/12/28 17:06:59

Again, working with the author, he was able to try a couple of tricks and it turns out that my login name “myusername” in this example is different than my email “myemail.name”.  The real example is dmauro -vs- douglas.mauro.  This was screwing up the WebDav and some changes needed to be made in the msfetchrc script

Line:61 – I commented out the original and inserted the new line above

1
2
3
Line :61
my $strUrl = $strProt."://" . $strServer . "/exchange/myemail.name/";
#my $strUrl = $strProt."://" . $strServer . "/exchange/" . $strAlias . "/";

You could also modify the line 62 to get Sent items (I didn’t in this example).

It works! Here is a run (no emails):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@localhost ~]# ./msfetch

msfetch v0.4 - download e-mail messages from Exchange OWA 2000/2003/2007 to mbox

Login successful!
Listing E-Mail Folders...
processing folder (https://email.mydomain.com/exchange/myuser.name/Inbox/)
skipping folder (https://email.mydomain.com/exchange/myuser.name/Outbox/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Sent%20Items/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Deleted%20Items/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Calendar/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Contacts/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Drafts/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Journal/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Notes/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Tasks/), exclusion list...
skipping folder (https://email.mydomain.com/exchange/myuser.name/Junk%20E-Mail/), exclusion list...
Number of folders: 1
Fetching https://email.mydomain.com/exchange/myuser.name/Inbox/
Completed https://email.mydomain.com/exchange/myuser.name/Inbox/
Started:   2009/12/29 11:33:47
Completed: 2009/12/29 11:33:48

Of course I couldn’t leave well enough alone. I wanted a way to notify me via pop/imap when I get email on this account. I turned to setting up mutt to use gmail and ran a *crude* 1 liner to send emails to my account

Line:463

1
2
3
4
5
6
print "Message ($web_id) fetch successful...\n";

# New line below
  `su - douglasmauro -c 'mutt -s \"FUN-EMAIL-MSG: $web_header \" douglasmauro\@mygmailaccount.com < /dev/null'`;

 return 0;

Going into my douglasmauro account on the system reveals a .muttrc of:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
set realname = "Douglas R. Mauro"
set spoolfile = imaps://imap.gmail.com:993/INBOX
set imap_user = 'username@mygmaildomain.com'
set imap_pass = 'mypassword'
set smtp_url="smtp://username@mygmaildomain.com@smtp.gmail.com:587/"
set smtp_pass = 'mypassword'
set folder = imaps://imap.gmail.com:993
set postponed="imaps://imap.gmail.com/[Gmail]/Drafts"
set record=""
set imap_check_subscribed="yes"
set imap_list_subscribed="yes"
set header_cache="~msgcache"
set certificate_file="~.mutt_certificates"
set mbox="imaps://imap.gmail.com/[Gmail]/All Mail"

and a simple entry into cron

[cce]
[root@localhost ~]# crontab -l
*/5 * * * * /root/msfetch 2>&1 > /dev/null # Fetch OWA Mail
[root@localhost ~]#
[\cce]

Now every 5 minutes, it’ll check this OWA account and forward an EMPTY msg to the email I listed with the subject = $web_header.

yes, yes, I could find a way to forward the whole msg but this is soo much easier ;)

More thanks to Paul for helping me get this working!

Just a few files please….

Simple way to grab some files out of a RPM

1
$ rpm2cpio php-5.1.4-1.esp1.x86_64.rpm | cpio -idmv

Git Videos…

watch Command

While researching some issues I’m having with iostat, I found an interesting command called ‘watch’

Instead of running the usual command (up arrow, return, up arrow, return)

1
2
3
4
5
6
[root@nms-tpt-1 ~]# cat /proc/diskstats | grep "sda "
   8    0 sda 42113 18004 1395004 80723 18000492 45624354 508998694 7650613 0 4953776 7730941
[root@nms-tpt-1 ~]# cat /proc/diskstats | grep "sda "
   8    0 sda 42113 18004 1395004 80723 18000492 45624354 508998694 7650613 0 4953776 7730941
[root@nms-tpt-1 ~]# cat /proc/diskstats | grep "sda "
   8    0 sda 42113 18004 1395004 80723 18000497 45624372 508998878 7650614 0 4953777 7730942

…or writing a never ending while loop. You can issue “watch” which will run your program (clearing the screen each time) and even show you (highlighted) the diffs between each call … pretty nice:

1
[root@nms-tpt-1 ~]# watch -n 1 -d 'cat /proc/diskstats | grep "sda "'

would show in a new screen:

1
2
3
Every 1.0s: cat /proc/diskstats | grep "sda "                                                                                                            Mon Dec  7 09:42:18 2009

   8    0 sda 42113 18004 1395004 80723 18000540 45624456 508999894 7650638 0 4953787 7730966

Adding Perf Data (which op5 will auto graph!) w/NRPE

The NRPE command for checking procs simply returns

PROCS OK: 112 processes
1
PROCS OK: 112 processes

Using some code off the net, we can create a little wrapper to print the Nagios Performance Data way (if that’s even a sentence)

1
2
3
4
5
6
7
8
[root@op5 op5_custom_plugins]# cat /opt/local/tpt/op5_custom_plugins/check_procs_wp.sh
#!/bin/bash
LINE=`/opt/plugins/check_procs $*`
RC=$?
COUNT=`echo $LINE | awk '{print $3}'`
echo $LINE \| procs=$COUNT
exit $RC
[root@op5 op5_custom_plugins]#

Running this by hand gives us:

1
2
3
[root@op5 op5_custom_plugins]#  /opt/local/tpt/op5_custom_plugins/check_procs_wp.sh
PROCS OK: 114 processes | procs=114
[root@op5 op5_custom_plugins]#

Create a newline in the config file

1
2
3
4
[root@op5 op5_custom_plugins]# cat /etc/nrpe.d/op5_commands.cfg | grep total_procs
command[total_procs]=/opt/plugins/check_procs -w 150 -c 200
command[total_procs_wp]=/opt/local/tpt/op5_custom_plugins/check_procs_wp.sh -w 150 -c 200
[root@op5 op5_custom_plugins]#

Using the name “total_procs_wp” in the nrpe call, we’ll get the total procs running AND with op5, it’ll start auto graphing that for us.

Clock Drift Kills…

Well … not kills but many systems and applications go crazy when the system clock drifts too far.

In this post we’ll focus on adding a new service check or plugin to the op5 Monitor software.  If you’re using Nagios, this will work for you as well but you’ll have to make all the changes in the txt files by hand … sorry.

Be sure to read the op5 walk-through of writing your own plugin for more.

Let’s first talk about time .. NTP is used by many sites and seems to work well.  The issue is that if the clock becomes WAY out of wack, it will not sync the clock (this is by design).  This means if your clock jumps 1 day off (as an example), it will never get sync’ed by NTP without someone stepping in.

One way we can combat this and proactively check the drift is by installing the check_time plugin.  I’ve used Nagios for many years (and love it) but you’ll see below how much easier it is (in a few mouse clicks) to add a new check and automagically graph  the perf data!

Be sure to install and run NTP on your op5 server.

There’s a time service installed (but not usually activated) on *nix boxes called daytime.  Check to see if yours is running by issuing the command:

(BTW you’ll want to install this service on the nodes you’re checking.  We set ours up on the op5 server / localhost as an example)

1
2
3
4
5
6
[root@op5 op5_custom_plugins]# chkconfig --list | grep -i time

time-stream:    off
daytime-dgram:  off
daytime-stream: off
time-dgram:     off

You can turn these daytime ones on with the following commands:

1
2
[root@op5 op5_custom_plugins]# chkconfig daytime-dgram on
[root@op5 op5_custom_plugins]# chkconfig daytime-stream on

Once that is done, you should be able to telnet to localhost port 13

1
2
3
4
5
6
7
[root@op5 op5_custom_plugins]# telnet localhost 13
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
02 DEC 2009 09:50:32 MST
Connection closed by foreign host.
[root@op5 op5_custom_plugins]#

Now that we have the time service up and running, let’s move onto the check program….

There are 1000s of plugins avail to download and use for free.  We’ll be using the check_time plugin.  Go find and download this onto your op5 Monitor server:

http://www.nagioswiki.org/wiki/Plugin:check_time

http://www.nagioswiki.org/wiki/Plugin:check_time

Install the perl program in a safe place.  We picked:

1
/opt/local/tpt/op5_custom_plugins/check_time.pl

As with any program you download from the net … be sure to look it over before running it on your prod boxes!

Run the program by hand to see if you have everything you need

1
2
[root@op5 op5_custom_plugins]# ./check_time.pl -H localhost -Pdaytime -w 5 -c 60 -f
TIME OK - localhost time is Wed Dec  2 10:01:07 2009 (0 seconds variance) | time=1259773267 diff=0

You might have to install a few perl modules to get this rolling.  In our case, we had to install the TimeDate:

1
[root@op5 op5_custom_plugins]# yum install perl-TimeDate

As with most (well written) plugins, there’s a useful help when run with -h

1
2
3
4
5
6
7
8
[root@op5 op5_custom_plugins]# ./check_date.pl -h

check_daytime plugin for nagios version 0.2
by William Leibzon - william(at)leibzon.org

Usage: ./check_time.pl -v | -h | -P daytime|daytime-udp|time|time-udp [-p &lt;port&gt;] -H &lt;hostname&gt; [-w &lt;warning variance&gt;] [-c &lt;critical variance&gt;] [-t &lt;timeout seconds&gt;] [-o str|usec] [-f]

&lt;...snipped...&gt;

With the check command and time service working, let’s setup op5 now.  Click on ‘Configure’ and then ‘Commands’

op5-plugin-001

Now enter in the following information info the correct lines:

  • command_name: check_time (you can pick any name here you like)
  • command_line: /opt/local/tpt/op5_custom_plugins/check_time.pl -H $HOSTADDRESS$ -Pdaytime -w $ARG1$ -c $ARG2$ -f
  • FILE: etc/checkcommand.cfg (DEFAULT DONT CHANGE)

op5-plugin-002

  • $HOSTADDRESS$ will get replaced with the address of the device you are checking.
  • -p specifies the protocol to use.  In this case, daytime.
  • -w number of seconds we will WARN if time drifts beyond (1st passed argument)
  • -c number of seconds we will CRITICAL if time drifts beyond (2nd passed argument)
  • -f spits out some performance numbers that op5 will auto graph on (nice!)

Once you hit ‘Apply Changes’ we can go into a host and add this new check as a new service.

If not already there, click ‘Configure’ -> Select host and then click Go.  Top right click ‘Services for host XYZ’

Here’s where we’ll add the new service.

op5-plugin-003

  • service_description: We picked Check Clock Time (this appears in the Service Details section)
  • check_command: here’s where we want to select the new check command we created above
  • check_command_args: This specifies $ARG1$ (no spaces) !  (no spaces) $ARG2$ that we want to pass.  5!60
  • Once you have this click on ‘Test this service’ to see if you got it right

Click ‘Apply Changes’ when you are finished.

op5-plugin-004