Sync Your Stuff to S3

Mac, Snippets, Tutorials June 24th, 2010

This is a receipe how I save stuff to S3 from my Mac:

1.) Signup with S3: http://aws.amazon.com/s3/ (check pricing!). This will give you access to the AWS Management Console.

2.) Create a Bucket: This can be done via the AWS Management console. If you are not familiar with the concept of ‘buckets’ check-out the S3 documentation. Simply put, it is a virtual storage device that has a fixed geographical location.

3.) Go to ‘Security Credentials’ in your account settings in the AWS Management console and create an accesskey.

4.) Download JetS3t. You then have the following directory on your Mac:
/Applications/jets3t-0.7.3

Open the Terminal and change into the bin directory:
$ cd bin;

Create a file named synchronize.properties there:
$ nano synchronize.properties;
and save the following content using your keys from step 3:

accesskey=XXXXXXXXXXXXXXXXXXXX
secretkey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5.) To sync the contents in the path /Users/marco/MySyncStuff with the bucket myBucketName use this command:
$ ./synchronize.sh UP myBucketName /Users/marco/MySyncStuff/ –properties synchronize.properties;

Of course you can create as many buckets as you like and script and schedule your data syncs now from here as you wish. Use the command
$ ./synchronize.sh –help
to see what the synchronize.sh else has on offer.

6.) Browsing buckets: JetS3t has its own S3 browser. To start and use it do the following:

$ cp cockpitlite.sh cockpitlite.command;
$ ./cockpitlite.command &;

You should see the Java coffee cup on your task bar. Use your keys to log in and browse your buckets.

You can also use the free S3 Browser for Mac.

Useful Linux Commands 12/2009

Linux December 29th, 2009

Recursively remove all .svn directories from a working copy:

find . -name .svn -exec rm -rf {} \;

Recursively remove all ._xyz-files (OSX meta file info) from your WebDav-Drive, set via hourly cron:

find /var/data/ -name "._*" -exec rm {} \;

Do not forget to set your path ;).

Add a PHP Post-Commit-Hook to your SVN

Snippets, Software development, Tools, Tutorials November 10th, 2009

This is just a link to a very good tutorial on how to make your own SVN post-commit hook using a PHP script. It sends the following information via email:

  • Committer name
  • Commit message
  • List of files edited
  • Diff of changes made

http://techchorus.net/writing-php-script-send-svn-commit-changeset-email-notification

Quick PHP Hack to Tidy up Trashed HTML

Snippets, Tools September 21st, 2009

I just hacked together the following quick and dirty PHP-script to use the tidy-extension from the command line. Maybe somebody else needs something like this somewhere. Check the comment for details:

<?php

//
// (a) Save this as tidy.php
// (b) Call it from commandline like this: $ php tidy.php trashed.html > tidy.html
//     to tidy the file trashed.html to a new file tidy.html.
//

// Installation and configuration see http://php.net/tidy
$config = array('indent'=> true, 'output-xhtml' => true, 'wrap'=> 200);

$tidy = new tidy;
$tidy->parseString(file_get_contents($_SERVER['argv'][1]), $config, 'utf8');
$tidy->cleanRepair();

echo $tidy;

Useful Linux Commands 04/2009

Linux, Productivity April 12th, 2009

I had a list of files from a large file structure as a result from a maintenance script run with lines like this:

/home/web/.../sources/.../2008/12/25/4f1feabbd76f79ecab150bdee3f6ae4d.xml
/home/web/.../sources/.../2008/12/25/e506e433a2d87f0275c7641da59bbf7f.xml
/home/web/.../sources/.../2008/12/28/901c4f081645b986e9b1377d3f586b8e.xml
/home/web/.../sources/.../2008/12/28/6bec4d4bbcf8f596c40694210d220a3b.xml
/home/web/.../sources/.../2008/12/24/477c535d6111605c8f6020a959f32fde.xml
/home/web/.../sources/.../2008/12/24/9f253a96fc26d8f6d9e61b8f1bdb3453.xml

Each line represented a document path to a file which was supposed to be removed from the filesystem. You can do that with the following simple oneliner:

for LINE in $( cat ../log/my_empty_files.txt ) ; do rm $LINE ; done

You can try it with ‘echo’ instead of ‘rm’ first to see if it would work:

for LINE in $( cat ../log/my_empty_files.txt ) ; do echo " # $LINE" ; done

Bulk Image Resize using Conditional Width

Linux, Productivity, Snippets, Tools April 4th, 2009

I am currently working on a project in which we have lots of images from an old CMS waiting to be migrated into a new layout. Of course there are restrictions so it should not happen that certain image types exceed a certain max. width.

OK, we have many many images… So I took a closer look at ImageMagick (also take a look at the usage examples). And I have to say: Awsome!

You can install ImageMagick on Ubuntu or Debian with a simple
# apt-get install imagemagick

In combination with a bit conditional scripting I came up with the following solution:

Console doing bulk resize.

Console doing bulk resize.

I wanted to have a shell script that, given a directory containing all our images, checks the width of each image and resizes it if it exceeded a certain width. Simple, but powerful.

Usage:

$ ./resize_image_dir.sh ../../brand_logos

And you are done with thousands of images in a minute. Do not forget to make a backup if designers change the desired width later…

You can download the shell scripts with example images ready to test:
!resize_conditional_images_bulk2

Useful Linux Commands 10/2008

Linux October 10th, 2008

(1) Finds files in generated documentation, containing <span class=”field”>webservice:</span> and writes a file containing a clickable list of links to those pages:
~/sites/html/phpdoc_all_global$ find -type f -print0 | xargs -0 grep -li ‘<span class=”field”>webservice:</span>’ | while read in; do echo “<a href=\”$in\”>$in</a>”; done > clickable_list_page.html

(2) A for-loop on the commandline. The * equals all files and dirs in current directory (try “echo *”). Of course you can replace the ‘echo $dir’ with whatever command you like using $dir as loop-variable.:
for dir in *; do echo $dir; done

(Thanks Alexander)

(3) A simlpe piped grep, which displays the files (-l) containing the string “<FormularObjectGenerator”. And since I was searching recureively (-r) in a svn working copy, I did not want to see all the double filepaths containing “.svn” I filtered by excluding (-v) this pattern from the output.

grep -lr “<FormularObjectGenerator” ./ | grep -v “\.svn”

Useful Linux Commands 08/2008

Linux August 20th, 2008

Inspired by the book “The Productive Programmer” I was paying attention more and more on how I actually get things done and how I could better exploit existing shortcuts or faster paths.

Please do not ask why, but I had the following problem: One my PHP-Apps had scattered syntax errors. The question was, how I could locate them in the application. The first attempt was to let Zend Studio analyse the project… This was very slow and contained a lot of information which I actually did not want to see.

So I came up with the following solution:

find -name ‘*.php’ | xargs -i php -l {} | grep ‘Parser error:’

There are 3 parts piped together:

  1. find: Lists all .php files in your project.
  2. xargs: Executes one command for each of the lines coming from standard input. The -i flag replaces one line from STDIN with the {}. The php-executalbe has a flag -l (=lowercase L), which lets php check for correct syntax only.
  3. grep: Filters the output (lots of output like ‘No syntax errors detected in ./includes/show.inc.php’) to the actual information I liked to see: The files which had syntax errors.

From this experience I learned the following:

  • Pay attention to flags of commandline tools, even if they do not appear useful at first sight - like the ‘php -l script.php’. They might be very useful when combined via pipe with other commands.
  • I understood one more time the power of the commandline and will more often visit man pages of commands.