Tag Archives: Tools

Extract All Email Addresses from Outlook

I was looking for a solution to extract a list of all emails I have ever used or emailed through outlook. It might be helpful for you too. This what I ended up doing:

a) Export email and contacts into an olm file (Mac). I was using Outlook 2011. This is a binary compressed format. I selected Email and Contacts.

b) Use StuffItExpander to extract the olm to a readable XML structure. Just install StuffItExpander and drag the olm file on to it. You end up with a parsable directory structure including all files in xml format.

c) I could not find a solution to recursively parse the mess, so I decided to merge all xml files form all subdirectories into one big file:

$ find /path/to/directory/ -name *.xml -print0 | xargs -0 -I file cat file > merged.file

d) Extract all email addresses from the merged file into a file:

$ perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}' merged.file | sort -u > output.txt

e) You will be surprised how many lines this file will have. Check the output.txt containing all unique extracted email addresses from outlook. The list needs to be cleaned. There will be a lot of invalid or temporary emails you need to go through manually.

Have fun with your list.

Emulate sendmail on your Dev Machine

In order to prevent email delivery during development and log all email messages that would have been delivered, you can actually do a simple trick: Replace the file /usr/sbin/sendmail (on Ubuntu, use ‘locate sendmail’ to find it if it lies elsewhere) with this little shell-script, or rather make a _bak of the original and save the following instead of the sendmail binary:

#!/bin/bash

LOGDIR="/tmp"
PREFIX="sendmail"
NOW=$(date +%Y-%m-%dT%H.%M.%S)
CNT=1
PRIVATELOG="$LOGDIR/$PREFIX-$NOW.$CNT.log"
COMBINEDLOG="$LOGDIR/$PREFIX-combined.log"

# If privatelogs are being used...
if [ ! -z "$PRIVATELOG" ]; then
# ...make sure the filename is unique and create the file
while [ -f $PRIVATELOG ]; do
CNT=$(($CNT + 1))
PRIVATELOG="$LOGDIR/$PREFIX-$NOW.$CNT.log"
done

echo "$0 $*" > $PRIVATELOG
else
# ...otherwise swap filenames
PRIVATELOG=$COMBINEDLOG
COMBINEDLOG=''
fi

echo "[$NOW]" >> $PRIVATELOG
while read BUF
do
echo $BUF >> $PRIVATELOG
done

# Append privatelog to combinedlog when both logs are used
if [ ! -z "$COMBINEDLOG" ]; then
echo "[$NOW]" >> $COMBINEDLOG
cat $PRIVATELOG >> $COMBINEDLOG
fi

exit 0

When your application now sends mail, these things happen:

  • No email is actually sent.
  • The message gets appended to the file /tmp/sendmail-combined.log, on which you could set a ‘tail -f’ in order to see which emails would have been sent and what contet they would have.
  • One new file (e.g. /tmp/sendmail-2011-02-08T08.02.48.1.log) gets written for every email sent. I personally only use the combined file.

Inspired by http://stackoverflow.com/questions/3710864/simulating-sendmail-with-dummy-script

Drawing Lines Using Google Static Maps API

If you are looking for a simple way to display lines like a roundtrip on a map, Google’s Static Maps API might be for you.

Documentation and examples: http://code.google.com/intl/en-US/apis/maps/documentation/staticmaps/

Limits:
- There is a query limit of 1000 unique (different) image requests per viewer per day.
- URLs are restricted to 2048 characters.


Here are some self-explanatory examples:

Using geo-coordinates:
http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|49.584486,8.462219|49.441343,8.462219|49.384161,8.654230&size=400x400&sensor=false

Using city names:
http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|Stuttgart|Mannheim|Bad Homburg|Frankfurt|Würzburg|Stuttgart&size=500x200&sensor=false

This is how you would use it on your website just use it like a local image:
<img src="http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|Stuttgart|Mannheim|Bad Homburg|Frankfurt|Würzburg|Stuttgart&size=500x200&sensor=false" alt="my roundtrip" />
my roundtrip

Pretty slick!

In case you need to extract geocoordinates, there are lots of online browser tools available, just click on a spot and copy-paste the coordinates into your static maps-URL:

Clean your PHP4 Legacies using sed

If you have to deal with very old PHP4 legacy code containing every syntax crime you may know from the early years, how would you handle it? Give it to your junior people to fix it manually? I like to have at least some handy helpers for the first rough corrections. I found sed to be a very powerful helper here.

Code you might encounter - associative array elements without quotes.

I spent quite some time to find useful regular expressions to help me. This is how I did this:

A way to test your regular-expression is to echo a sample and apply your regex to test the results:

echo '$_REQUEST[action] reise_l_topic_ids[] $dat[ticket_order]' | sed "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g"

Once it works you can apply your regex to one file:

sed -i "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g" my_old_file.php

Or apply your regex to all *.php-files recursively below the current directory to a whole project:

find . -name "*.php" -exec sed -i "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g" '{}' ;

By the way: The usage of sed works fine on your linux command line, but not on OSX. The syntax is slightly different here (sed -i “” -e “s/blah/blubb/” file). This is of course only a start to automate otherwise painfull and boring corrections down to just a few seconds. It will not save you from special manual work and break syntax at some points. But it weeds out 90% and leaves you with the other 10% acutal manual work.

You could imagine many more sed regexes e.g. to replace short tags <?=$my_var?> to a proper <?php echo $my_var; ?> etc.

echo '<?=$out?> sakdhs sakdhas k <?php echo $xyz; ?> ddd' | sed "s/<?=/<?php echo /g";

I will collect more regexes as I need and find them. If you have ideas please add them in the comment section.

Sync Your Stuff to S3

This is a receipe how I save stuff to S3 from my Mac:

1.) Signup with S3: http://aws.amazon.com/s3/ (check pricing!). This will give you access to the AWS Management Console.

2.) Create a Bucket: This can be done via the AWS Management console. If you are not familiar with the concept of ‘buckets’ check-out the S3 documentation. Simply put, it is a virtual storage device that has a fixed geographical location.

3.) Go to ‘Security Credentials’ in your account settings in the AWS Management console and create an accesskey.

4.) Download JetS3t. You then have the following directory on your Mac:
/Applications/jets3t-0.7.3

Open the Terminal and change into the bin directory:
$ cd bin;

Create a file named synchronize.properties there:
$ nano synchronize.properties;
and save the following content using your keys from step 3:

accesskey=XXXXXXXXXXXXXXXXXXXX
secretkey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5.) To sync the contents in the path /Users/marco/MySyncStuff with the bucket myBucketName use this command:
$ ./synchronize.sh UP myBucketName /Users/marco/MySyncStuff/ –properties synchronize.properties;

Of course you can create as many buckets as you like and script and schedule your data syncs now from here as you wish. Use the command
$ ./synchronize.sh –help
to see what the synchronize.sh else has on offer.

6.) Browsing buckets: JetS3t has its own S3 browser. To start and use it do the following:

$ cp cockpitlite.sh cockpitlite.command;
$ ./cockpitlite.command &;

You should see the Java coffee cup on your task bar. Use your keys to log in and browse your buckets.

You can also use the free S3 Browser for Mac.

Consistent Development Environments using VirtualMachines

As a development team we always run into situations where we have trouble setting up a proper development environment for each of the team members to get going or add new staff on the go. It annoyed me every time since it causes a lot of unnecessary communication and friction.

I often heard of virtualization but never actually played seriously with it. The idea is:

If we could have a virtual machine for every project that contains an equivalent environment like the production system, everybody working on it…

  • … could just rely on his development environment by just starting the VM without having to set up anything half-baked themselves.
  • … could use his favourite working environment OS, IDE and tools on which they are most comfortable and thus happy and productive.
  • … could work on their own checked out working copy using version control.
  • … could immedately see what they built refreshing the local browser or starting Unittests on the VM via ssh to check their dev increments.

We used http://www.virtualbox.org. A good starting point to get to know VirtualBox better and learn how to start your first virtual machine: https://help.ubuntu.com/community/VirtualBox

Our target was to be able to startup the development VM as guest system on any developers development machine being the host system, open a browser on the host (!) and call for example http://develop/ to see the webroot of the VM. Additionally we set up samba and ssh on the VM in order to have the webserver’s webroot on the VM available via the filesystem. In order to do that you need to…

  • …start your VM with networking set to ‘Host interface’ instead of the default NAT. This is explained in detail on this page (sorry German) http://www.nwlab.net/tutorials/virtualbox/virtual-networking.html – for me it was tricky to get the guest machine available on the host and have internet access at the same time.
  • …edit the hosts file (on OSX ‘sudo nano /private/etc/hosts’ and reboot) on the development machines and add something like the following line: ’192.168.56.101 develop’. To find out the IP enter ‘sudo ifconfig’ (OSX/Linux) on your host system after you have started the VM. You will see aditional adapters set by virtualbox and the IP address.
  • …configure /etc/samba/smb.conf on the VM, restart samba and connect (e.g. smb://develop/webroot). We check out a working copy of the applicaton under development directly onto the webroot and create a new PHP-Project there in our IDE. Update and commit directly from there.

If you search the web for ready-made VMs you find mostly VMware images. You can not run them directly in VirtualBox and need to convert them.

Under Linux you can convert the VMware image (.vmdk) into a VirtualBox image (.vdi) like this:

sudo apt-get install virtualbox-ose virtualbox-ose-guest-utils;
sudo apt-get install qemu;

qemu-img convert xxx.vmdk xxx.bin;
VBoxManage convertdd xxx.bin xxx.vdi

Install the required packages with apt-get install once. VBoxManage is part of VirtualBox. The last two lines do the conversion.


We ended up creating a fresh install from a Debian 5 netinstall iso. The iso-file can be mounted as CDROM on the creation of the new VM with VirtualBox. Receipes for setting up the appropriate LAMP environment with apt-get install can be found on the web. You only have to do it once. Save the state of your VM afterwards.


There are ways to generate a virtual machine from a physical server. Use Google to find receipes. I used http://www.partimage.org on a Debian Etch system with the live CDROM from http://www.sysresccd.org. This requires that you are able to umount your filesystem or rather boot into the live cd on the production/staging machine in order to generate the partimage.

You mount an external drive over the network or a usb harddrive. The partition (e.g. /dev/sda1) you would like to backup must be umounted. From the live cd you can see your partitions, including attached usb drives, with the ‘fdisk -l‘ command. Just mount the target (e.g. /mnt/usbdrive) and start partimage from the commandline. Dialogues guide you through the image creation.


In case you wonder what is meant by the ‘Host key’ to enter or leave a running VM with your mouse… it is the right (!) Strg-Button on your Keyboard.


I just installed Ubuntu Server on a VM from my MacBook. To have a usable keyboard once you logged on to the new VM, you must do the following in order to have a keymap including the pipe symbol, braces etc.:

  • sudo apt-get install console-data; #to install the keymaps
  • sudo loadkeys mac-macbook-de; #to set the keymap for German MacBook

Once you have done that, you can use your right Command-Key as ‘Alt-Gr-Key’ like on a PC keyboard. The pipe symbol is then typable with ‘Alt-Gr + >’, Braces and Brackets are typable via ‘Alt-Gr + 6,7,8,9′.


This is how you copy a virtual machine using VirtualBox tools:

$ VBoxManage clonevdi /Users/marco/MyMachine.vdi /Users/marco/MyMachine_copy.vdi
$ VBoxManage internalcommands setvdiuuid /Users/marco/MyMachine_copy.vdi

Frameworks…

Find out interesting thoughts and experiences about the selection and usage of a frameworks for your projects in this set of slides ‘Living with Frameworks‘ by Stuart Herbert, Technical Manager at www.gradwell.com.

You will learn about:

  • How frameworks save you time (=money) and ensure quality but can also waste resources if applied in another way the framework was intended to be used.
  • The importance of the framework guru role.
  • That a chosen framework and architecture should be strategically introduced (top-down).
  • Introduction and proper use of a new framework has a steep learning curve.
  • Legacy code and the parallel maintenance hassle.
  • Refactor early, refactor often, perform regular code reviews with the framework guru.
  • Frameworks will not fix bad practice (specification, quality, no training).
  • Your framework should fit your overall development plan and practice.
  • Training your staff in your framework is a means of building your team.
  • Headcount on projects has increased, means more teamwork, organisation around needs to mature too.
  • Importance of upgrades and backward compatibility.

A Simple PHP Google Sitemap Generator

I just hacked together a very simple class and CLI script to help me generate Google sitemap XML on a daily basis using PHP. If you have not yet heard of it: Google sitemaps help you promote your content if you have certain deep links that only show up if a visitor for example performs a search and clicks a result.

Check the documentation of the Google Sitemap Format

Here is the code: Google Sitemap Generator

It works like this:

$ php generate_gsitemaps.php; #will generate your standard sitemap
$
php generate_gsitemaps.php -e > my_sitemap.xml; #echo sitemap xml to another file

Example: generated_example_sitemap.xml

Have fun being found!

Etherpad

EtherPad is a free webbsased editor which lets people remotely and simultaneously edit the same text document. Just click ‘Create new Pad‘ and you see the URL which you can send to your remote friend(s).

etherpad_screen

Use it for group development on code snippets, catch realtime meeting minutes, brainstorming ideas and many things more…

Thanks AppJet for this nice tool!