Useful Linux Commands 12/2009

Linux December 29th, 2009

Recursively remove all .svn directories from a working copy:

find . -name .svn -exec rm -rf {} \;

Recursively remove all ._xyz-files (OSX meta file info) from your WebDav-Drive, set via hourly cron:

find /var/data/ -name "._*" -exec rm {} \;

Do not forget to set your path ;).

Useful Linux Commands 04/2009

Linux, Productivity April 12th, 2009

I had a list of files from a large file structure as a result from a maintenance script run with lines like this:

/home/web/.../sources/.../2008/12/25/4f1feabbd76f79ecab150bdee3f6ae4d.xml
/home/web/.../sources/.../2008/12/25/e506e433a2d87f0275c7641da59bbf7f.xml
/home/web/.../sources/.../2008/12/28/901c4f081645b986e9b1377d3f586b8e.xml
/home/web/.../sources/.../2008/12/28/6bec4d4bbcf8f596c40694210d220a3b.xml
/home/web/.../sources/.../2008/12/24/477c535d6111605c8f6020a959f32fde.xml
/home/web/.../sources/.../2008/12/24/9f253a96fc26d8f6d9e61b8f1bdb3453.xml

Each line represented a document path to a file which was supposed to be removed from the filesystem. You can do that with the following simple oneliner:

for LINE in $( cat ../log/my_empty_files.txt ) ; do rm $LINE ; done

You can try it with ‘echo’ instead of ‘rm’ first to see if it would work:

for LINE in $( cat ../log/my_empty_files.txt ) ; do echo " # $LINE" ; done

Bulk Image Resize using Conditional Width

Linux, Productivity, Snippets, Tools April 4th, 2009

I am currently working on a project in which we have lots of images from an old CMS waiting to be migrated into a new layout. Of course there are restrictions so it should not happen that certain image types exceed a certain max. width.

OK, we have many many images… So I took a closer look at ImageMagick (also take a look at the usage examples). And I have to say: Awsome!

You can install ImageMagick on Ubuntu or Debian with a simple
# apt-get install imagemagick

In combination with a bit conditional scripting I came up with the following solution:

Console doing bulk resize.

Console doing bulk resize.

I wanted to have a shell script that, given a directory containing all our images, checks the width of each image and resizes it if it exceeded a certain width. Simple, but powerful.

Usage:

$ ./resize_image_dir.sh ../../brand_logos

And you are done with thousands of images in a minute. Do not forget to make a backup if designers change the desired width later…

You can download the shell scripts with example images ready to test:
!resize_conditional_images_bulk2

Upgrade PHP5 with an alternative sources.list on Debian etch

Linux, Problems, Serverstuff February 20th, 2009

I was having trouble with a server running Debian 4.0 (etch). Using the standard sources in the /etc/apt/sources.list the supported PHP5 version was 5.2.0-8+etch13 which contained a very annoying bug for my application.

A daily running script - let’s call it the Importer - regularly exited randomly with a “Fatal error: Out of memory (allocated 12320768) (tried to allocate 2851436 bytes) in …” and I had to restart it manually nearly every morning. I had…

  • …checked my application for memory wasting operations and loops and fixed them,
  • …used ini_set(’memory_limit’, ‘64M’); at runtime, and
  • …finally increased memory_limit = 64M in my php.ini.

But all this did not change the bahaviour of the Importer!

So I took a look at the PHP5 Changelog to find potentially fixed bugs in newer releases. Bug #39438 described exactly my problem. So a simple upgrade would help me. But it did not work with ‘apt-get upgrade’ or ‘apt-get install php5=5.2.8′ since the highest version in the apt source I used was the one that I already had: 5.2.0-8+etch13, issued in November 2006… (pretty ancient)

Finally it was this page that had the information we needed: an alternative apt source

deb http://packages.dotdeb.org etch all
deb-src http://packages.dotdeb.org etch all

After getting an impression whether dotdeb was a trustworthy source, we first tried it on our dev-system with ‘apt-get update; apt-get upgrade;’. At this point I was once more glad to have written so many UnitTests. They all passed and everything looked good.

Thanks Kim for your help!

Subversion Repository on Samba Share

Linux, Tools December 22nd, 2008

I use a Subversion repository which sits on a Samba share in our LAN. Most of the times I used it from Windows machines where I simply assigned a drive letter (N:\) to the filesystem of the network share (//TERASTATION2/share) hosting the repository. A SVN checkout including URL looked like this:

  • svn co file://N:/_repos/projectname/trunk .

Now I have my new Ubuntu/Linux notebook and asked myself how I could use the repository from there too. I tried using smb://terastation2/share/_repos/myproject/trunk - but this did not work in combination with file://.

The solution looks like this:

  • Create a dir to mount the share to: $ sudo mkdir /mnt/terastation2;
  • Mount the filesystem: $ sudo smbmount //TERASTATION2/share /mnt/terastation2 -o lfs;
    (You can create a launcher to do that for you in the future)
  • Then you can do your desired operations: $ svn co file:///mnt/terastation2/_repos/projectname/trunk .;

Useful Linux Commands 10/2008

Linux October 10th, 2008

(1) Finds files in generated documentation, containing <span class=”field”>webservice:</span> and writes a file containing a clickable list of links to those pages:
~/sites/html/phpdoc_all_global$ find -type f -print0 | xargs -0 grep -li ‘<span class=”field”>webservice:</span>’ | while read in; do echo “<a href=\”$in\”>$in</a>”; done > clickable_list_page.html

(2) A for-loop on the commandline. The * equals all files and dirs in current directory (try “echo *”). Of course you can replace the ‘echo $dir’ with whatever command you like using $dir as loop-variable.:
for dir in *; do echo $dir; done

(Thanks Alexander)

(3) A simlpe piped grep, which displays the files (-l) containing the string “<FormularObjectGenerator”. And since I was searching recureively (-r) in a svn working copy, I did not want to see all the double filepaths containing “.svn” I filtered by excluding (-v) this pattern from the output.

grep -lr “<FormularObjectGenerator” ./ | grep -v “\.svn”

Useful Linux Commands 08/2008

Linux August 20th, 2008

Inspired by the book “The Productive Programmer” I was paying attention more and more on how I actually get things done and how I could better exploit existing shortcuts or faster paths.

Please do not ask why, but I had the following problem: One my PHP-Apps had scattered syntax errors. The question was, how I could locate them in the application. The first attempt was to let Zend Studio analyse the project… This was very slow and contained a lot of information which I actually did not want to see.

So I came up with the following solution:

find -name ‘*.php’ | xargs -i php -l {} | grep ‘Parser error:’

There are 3 parts piped together:

  1. find: Lists all .php files in your project.
  2. xargs: Executes one command for each of the lines coming from standard input. The -i flag replaces one line from STDIN with the {}. The php-executalbe has a flag -l (=lowercase L), which lets php check for correct syntax only.
  3. grep: Filters the output (lots of output like ‘No syntax errors detected in ./includes/show.inc.php’) to the actual information I liked to see: The files which had syntax errors.

From this experience I learned the following:

  • Pay attention to flags of commandline tools, even if they do not appear useful at first sight - like the ‘php -l script.php’. They might be very useful when combined via pipe with other commands.
  • I understood one more time the power of the commandline and will more often visit man pages of commands.