Sync Your Stuff to S3

Mac, Snippets, Tutorials June 24th, 2010

This is a receipe how I save stuff to S3 from my Mac:

1.) Signup with S3: http://aws.amazon.com/s3/ (check pricing!). This will give you access to the AWS Management Console.

2.) Create a Bucket: This can be done via the AWS Management console. If you are not familiar with the concept of ‘buckets’ check-out the S3 documentation. Simply put, it is a virtual storage device that has a fixed geographical location.

3.) Go to ‘Security Credentials’ in your account settings in the AWS Management console and create an accesskey.

4.) Download JetS3t. You then have the following directory on your Mac:
/Applications/jets3t-0.7.3

Open the Terminal and change into the bin directory:
$ cd bin;

Create a file named synchronize.properties there:
$ nano synchronize.properties;
and save the following content using your keys from step 3:

accesskey=XXXXXXXXXXXXXXXXXXXX
secretkey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5.) To sync the contents in the path /Users/marco/MySyncStuff with the bucket myBucketName use this command:
$ ./synchronize.sh UP myBucketName /Users/marco/MySyncStuff/ –properties synchronize.properties;

Of course you can create as many buckets as you like and script and schedule your data syncs now from here as you wish. Use the command
$ ./synchronize.sh –help
to see what the synchronize.sh else has on offer.

6.) Browsing buckets: JetS3t has its own S3 browser. To start and use it do the following:

$ cp cockpitlite.sh cockpitlite.command;
$ ./cockpitlite.command &;

You should see the Java coffee cup on your task bar. Use your keys to log in and browse your buckets.

You can also use the free S3 Browser for Mac.

Add a PHP Post-Commit-Hook to your SVN

Snippets, Software development, Tools, Tutorials November 10th, 2009

This is just a link to a very good tutorial on how to make your own SVN post-commit hook using a PHP script. It sends the following information via email:

  • Committer name
  • Commit message
  • List of files edited
  • Diff of changes made

http://techchorus.net/writing-php-script-send-svn-commit-changeset-email-notification

Free XSD Editor - Generate XSDs from XML

Software development, Tools, Tutorials June 28th, 2009

From an old post: To start out with XML-Schema this might be of interest to you:

XSD-Tutorial: http://www.liquid-technologies.com/XsdTutorial_01.aspx

Free graphical Tool: http://www.liquid-technologies.com/LiquidXMLStudio.aspx

[2009-06-27] Update: A very cool feature is the generation of a Schema, based on example-XML files you give Liquid XML Studio! I discovered this when I built a Schema that would not validate against my desired XML structure and I could not figure out why. I generated the XSD like this:

  • Open one of your XML examples with Liquid XML Studio.
  • Select from the menu “Tools / Infer XSD Schema”.
  • You are asked for more examples. I did it with just one XML file.
  • And bingo: You have your XSD.

As I did a diff of my handcraftet version of the XSD and the generated one, it revealed the reason for not validating nicely for me ;).

Cool tool, you can use with the 30-day trial licence.

Start longer running Jobs with screen

Linux, Serverstuff, Tutorials June 19th, 2008

On the command line if you close a console with a running job, you kill the job. This is different with the tool ’screen’, where you can attach and detach from a ’screen’ without terminating it. You can even start a job in a screen on another machine, detatch, travel somewhere else and re-attach to it on another machine.

If you do not have screen yet, install it on your Debian box with: apt-get install screen

Commands:

  • screen -S indexing - Create a screen with name ‘indexing’.
  • screen -ls - Show available screens.
  • screen -r indexing - Re-attach to the screen ‘indexing’.
  • Strg-A, Strg-D - Detatch from a current screen (without terminating it).
  • exit OR Strg-D - Exit from a current screen. This terminates the screen session.

Trigger and Organize Timed Tasks with ‘at’

Linux, Serverstuff, Tools, Tutorials May 29th, 2008

Since I am doing more and more stuff on the commandline, I noticed that sometimes I just wait for some task to finish to do a next step in a sequence to accomplish a certain goal. What if a running task would take an estimated 3-8 hours and it is Friday afternoon?

In this case you can use the commandline-tool ‘at’ and schedule something like ‘at 1am tomorrow do xyz’.

If you do not have at yet, install it on your Debian box with: apt-get install at

There are 3 commands available:

  • at <datetime> - Starts the scheduling-dialog for a specific date or/and time to execute certain commands.
  • atq - Shows a list of already scheduled and pending jobs.
  • atrm <job-id> - Deletes pending jobs you would like to remove from the job-queue.

Case: What do you have to do in order to set a sequence of commands as root at a specific time?

Let’s say it is Friday 15:00h (and the server-clock says that too) and you would like to fire the command

  • echo “Good morning!” > hello.txt @ 08:00h tomorrow

You would do the following:

  • Log in as root.
  • Type ‘at 8.00′ <enter>.
  • Since for today 08.00 is already in the past, at assumes that you mean tomorrow. But you can use all sorts of time- and date signatures, for example you could use ‘at 08:00 01.06.2008′.
  • Now the prompt changes to at> and waits for your commands to be executed sequentially at the time you specified.
  • Now you type your first command to be executet: ‘cd ~/stuff’ <enter> - to make sure where the next command is executed, since it writes a file.
  • Now type your 2nd command: ‘echo “Hello 8.00″ > hello.txt’ <enter>
  • The prompt shows another at>. If you have more commands add them. In our example we have only two. To finish we press STRG-D and you have your normal prompt back.
  • Type the command ‘atq’ to see the pending jobs in the queue. Your job, you just added has a job-ID in the first column.
  • To delete the job from the queue type ‘atrm <job-id>’.

Good timing!

Tutorials and Articles

Links, Tutorials May 8th, 2008

In search of an example how to implement a top-level exception handler in PHP I came across this nice site containing lots of well explained tutorials and articles:

http://www.phpro.org

Use svn:externals to integrate external libraries

Tools, Tutorials April 25th, 2008

As projects get more complex, I came to use Subversion’s externals. This SVN property enables you to use more than one ‘checkout’ inside your architecture. For example if you are using Zend Framework as an external library, why make a ‘hard’ copy of it? It gets out of date very quickly, right? Externals allows you and your co-workers to check out a working-copy of a particular revision of it (if you want) and maintain it as ‘external’. This means you can use ’svn update’ not only on your code but also on Zend Framework or any other external libraries you are using in your project architecture. Precondition is, you have access to a Subversion repository that contains the libraries. Externals are very well explained at:


Here is a command-line example to setup and define an external:

Let’s say you would like to use the external library ‘external-lib’ in your project. Change dir to a directory, where you have your libraries and enter the following command:

  1. Create a new directory inside your working copy and change dir into that directory:
    /home/web/projectname/htdocs/libs$ mkdir external-lib;
    /home/web/projectname/htdocs/libs$ cd external-lib;
  2. Check out a working copy of the desired external library:
    /home/web/projectname/htdocs/libs/external-lib$ svn co https://svn.external-projectrepository.com/svn/external-lib/tags/1.1 .;
  3. Change dir back again and declare your checkout external:
    /home/web/projectname/htdocs/libs/external-lib$ cd ..;
    /home/web/projectname/htdocs/libs$ svn propset svn:externals “external-lib https://svn.external-projectrepository.com/svn/external-lib/tags/1.1″ “external-lib”
  4. Commit your changes:
    /home/web/projectname/htdocs/libs$ svn commit -m “External for external-lib set.”

Now every time you update your working copy, updates are fetched from your external(s) too.

Profiling your App with XDebug and KCacheGrind

Software development, Tools, Tutorials March 21st, 2008

If profiling of applications is something you already heard of but never played with, here is what I learned from starting to dig into this topic:

(1) Install XDebug (if dou do not already have it on your Dev-System). If not, please refert to the XDebug-Docs. The procedure is very well documented there. I used the pecl installer.

(2) I added this content to my php.ini files (for Apache2 and CLI). In the “Switch profiling”- and “Switch tracing”-sections you can comment the lines “xdebug.profiler_enable = 1/0″ and “xdebug.auto_trace = 1/0″ in or out. If you leave profiling and/or tracing on, it will significantly slow down all PHP execution on your server and generate many large cachegrind-files. It is not wise to leave it on if you do not need to profile or trace!

If I decide to do profiling, I switch it on, do my script executions to be profiled, which produces the required cachegrind-files and then I switch it off again. Remember you have to restart Apache2 after each change to your php.ini in order to take effect.

Set the path to your directory where you like to get the cachegrind-files written to. I set it to ‘/var/www/__profiling_data’.

(3) Now you need a viewer. I started under Windows with the free Tool WinCacheGrind. But to be honest, I did expect something cooler… As I tried KCacheGrind, which is a free Tool for Linux I was pleased to see informative screens like the following:

KCacheGrind Screen 1 KCacheGrind Screen 2

For those of you who do not know how to use Linux Apps… On Debian do “apt-get install kcachegrind” and once it is installed, type “kcachegrind &” on a console after logging in in the GUI-interface of Linux.

I was surprised at which points my scripts spent their time. Check the XDebug-Docs for more information on what you acually can read from the profiledata.

Happy profiling!

Build Functional Test-Suites with Selenium

Software development, Tutorials February 6th, 2008

I am currently writing lots of Selenium tests for a website with lots of form based enquiries. On my way I came accross some questions and obstacles to be overcome. This is a wrap-up of stuff I discovered:

Tools you need:

Links you might check out for background information and examples:

I did it this way:

  • Use Selenium IDE to record my tests roughly.
  • Install/upload Selenium Core onto the target system (this is necessary since Selenium Core is based on JavaScript and thus must be loaded from the same domain - keyword: XXS security in browsers, it would otherwise prevent access of Selenium Core to the page content) .
  • I organized my Tests in a directory structure like Testarea/Tests.html.
  • I manually updated all single tests using xpath-syntax to check/select items or click buttons or links. As default Selenium IDE took id-attributes which are in my case generated at random (dont ask me why) and did not work on repeated tests when pages were reloaded. Doing this the Web Developer-toolbar is very helpful. Click the button ‘Forms’ and select ‘Display form details…’ and you see additional information right on your page under test, which helps you do the xpath stuff. Here are some examples:
    • Push a submit button
      • Command: clickAndWait
      • Target: //input[@type='submit' and @name='send']
      • Value:
    • Make a selection on a dropdown list:
      • Command: select
      • Target: //select[@name='item9']
      • Value: value=1
    • Check a radio button:
      • Command: check
      • Target: //input[@type='radio' and @name='field3' and @value='1']
      • Value:
    • Click a link:
      • Command: clickAndWait
      • Target: //a[@title='Link zur Seite Erholungsstrategie']
      • Value:
    • Enter a value into an inputbox:
      • Command: type
      • Target: //input[@type='text' and @name='age']
      • Value: 32
    • After you opened and closed a popup, return to main window:
      • Command: selectWindow
      • Target: null
      • Value:

Organize all tests in a test-suite and execute them with Selenium Core:

  • Build a file like this one here with relative links to your test files: TestSuite.html
  • To execute a test-suite open Selenium Core in a Browser, enter the path to yourtest-suite in the upper left frame. A list of all your tests in the test-suite should appear. You are ready for executing them now.

My personal tips:

  • Take a look at the tab ‘Reference’ and play around with different commands.
  • When rewriting xpath expressions, occasionally click on the find-button next to it. It will give you an error or mark a found target with a green frame on the page for a second.
  • If you execute tests on a slow server it sometimes happens that an xpath coordinate will not find its target (since the page has not yet been completely loaded). Try to execute your test step by step at a slower pace.
  • In my case of testing enquiry forms I had to test several succeeding, session-based forms with error alerts on false entries etc. I had to delete session cookies very often via the Web Developer-toolbar (Cookies/Clear Session Cookies).

Further things to try:

  • Get Selenium RemoteControl, install and run it on your server and generate PHPUnit test code with Selenium IDE from your existing tests. As far as I know, this way you can even test for compatibility of different browsers and automate these tests for your builds.

Another good start with Solr

Links, Tools, Tutorials February 5th, 2008

Here is another good page to get started with Solr:

PHP for a client:

[25.04.2008] I found another helpful tutorial: http://www.xml.com/lpt/a/1668

Get started with Solr

Articles, Tools, Tutorials December 30th, 2007

I am currently playing around with Solr. Here is a nice tutorial to get started:
http://www.xml.com/pub/a/2006/08/09/solr-indexing-xml-with-lucene-andrest.html

A more indepth article from IBM:

Design Patterns in PHP5

Articles, Resources, Software development, Tutorials December 28th, 2007

The last resource also contains lots of interesting chapters from current web development books of various topics.

Webstuff in plain English

Tools, Tutorials, Videos December 21st, 2007

I came across some nice little explanations of the following web-topics:

[23.03.2008] New ones added:

Simple Captcha with PHP

Tools, Tutorials December 14th, 2007

In search of a simple CAPTCHA solution I came across the following tutorial (German). I modified it slightly and packaged it into a neat little demo including PHP code for the form and the images. It creates CAPTCHA images using GD like this:

Example CAPTCHA

Download: Simple CAPTCHA demo for PHP

Original Tutorial: http://www.stoppt-den-spam.info/webmaster/captcha-tutorial/index.html

Audio for your page made simple

Add-ons, Snippets, Tutorials December 3rd, 2007

One of my customers wanted to have spoken text on his webpage. Having no idea about Flash & Co. I found this neat solution: Use a Wordpress-Plugin to do the job.

Doors open

Other stuff, Tutorials September 2nd, 2007

If you ever lose the keys or lock yourself out of your house have a ‘bump key’ ready:
Watch: http://www.youtube.com/watch?v=pwTVBWCijEQ

You lost your car keys? Have a punched tennis ball ready:
Watch: http://www.youtube.com/watch?v=PnvI-3BTRcg

Run out of small change? Hack a vending machine:
Watch: http://www.youtube.com/watch?v=Rkq_Y0tZD1M

Change the password on a Win XP machine:
Watch: http://www.youtube.com/watch?v=oC8c7zKLnU4

Slides: Slides of Jeff Moore’s session at php|tek

Software development, Tutorials July 29th, 2007

Here are some nice slides covering the following topics:

Goto: Jeff Moore’s blog

Rasmus 30 Second AJAX Tutorial

Tutorials July 12th, 2007

This is something I came across browsing a mailing list:

Goto: http://marc.info/?l=php-general&m=112198633625636

Up to now for me AJAX was a topic which I had on my educational todo-list for ages - of course knowing roughly what it is all about in theory and waiting for the first project to get ‘paid practice’ ;).

Rasmus’ simplification was so inspiring, that I prepared 2 ready to click and easy to reverse-engineer examples from it. All you need is PHP and a browser with JavaScript.

Thanks Rasmus!

XPath - Windows Testing Setup and Cheatsheet

Software development, Tools, Tutorials, XML July 10th, 2007

When starting to use XML-related technologies like XSL and XPath, the details and concepts can be intimidating in the first place. One question I had to solve was: How can I create a testing setup on my windows workstation that takes a XML source file and a ‘console’ from which I could execute XPath queries against that XML and see the result. This is comparable to having a mySQL table or database (your XML content) and shooting SQL queries to it via phpMyAdmin.

Here is the setup to get started:

(1) Preparations - do the following:

(2) Get your hands dirty:

  • Start Cooktop and paste the XML Example into the XML-tab.
  • If you have ugly XML or a machine generated oneline XML -> Menu ‘Tools’ / ‘Format current XML Tidy’.
  • Be aware of namespaces! They have to be registered and used throughout your queries. It always confused me when testing, because the expected result is just empty nothing if you get that wrong. Solution: To simplify your test setup, make sure to delete all namespace info in the source XML. The example does not use a namespace, so we can forget about that for the moment.
  • Go to the XPath-tab in Cooktop, read the instructions comment and delete it.
  • All your XPath queries must be preceded by ‘nodes: ‘ or ‘values: ‘! Otherwise pressing the enter-key will not execute your query and just return the cursor to a new line.
  • Write to a new line: ‘nodes: //dealnews_issues/name_de’ and press enter.
  • You see the resulting nodes from your query. Cool!
  • That’s what it is all about with XPath: Isolate a particular XML chunk to further process it in your software or use it in a XSL file for transformation into something different or check for existence or count elements etc. etc.
  • Change your query to ‘values: //dealnews_issues/name_de’ and notice the difference.
  • Now you can get a more complex XML content and play with it. Some good introductory examples of queries can be taken from my XPath Cheatsheet. You will soon discover that XPath can be very powerful and also fun to use.

Additional information:

Software development mechanics (1)

Books, Resources, Softskills, Software development, Tutorials June 28th, 2007

I have often wondered why it is so difficult for non-developers (management, sales people, customers, inexperienced developers etc.) to understand the mechanics of software development, its life-cycle and the related elements like for example the importance of

  • code-base quality,
  • refactoring,
  • versioning,
  • unit testing,
  • agile development practices,
  • requirements management,
  • project-wide standards,
  • creating a productive, open and creative atmosphere around development,
  • permanent curiosity for new tech-stuff and the freedom to play with it,
  • and many more…

Some individuals of those groups don’t even at least try to find out more about it. Obviously they have not realized yet that when building complex websites, maintaining and extending them for many years they are already developing software - and that this is different from building a toaster once and selling it. So, I decided to post some related articles, books, podcasts etc. whenever I get across them which convey the nature of software development in understandable words.

I hope this selection motivates people to read/listen an maybe even understand one day and thus reduces unnecessary friction.

Here are the first links: