OCI Interface Integration into your eShop via Punch-Out from SAP/SRM

April 23, 2013 by marco · Leave a Comment
Filed under: Articles, Projects, Tools 
Social Tagging: • • • •

If it comes to integration of eShops like Magento, OXID or even your cusom made eShops into a corporate buying infrastructure using SAP, you need to deal with the SRM module (Supplier Relationship Management) using the OCI interface.

On the shop side we had a Gambio eShop in this case, which is a commercial fork of XT-Commerce. At first we checked if there where any affordable ready-to-use modules available. We found some for around 1500-2000EUR with additional (unclear) service cost. To be honest, what we found looked a bit outdated and after some contacts with the responsible sales people not even very trust worthy. After skipping over the documentation we decided to integrate OCI on our own – the custom hacked way. After all, it is just simple HTTP.

If you check the documentation and your first thought is, you could easily do it using cURL at the checkout, be warned – OCI does NOT work using a cURL call from a remote server! We did it this way and failed the first try as soon as we discovered how OCI mechanics really worked (see below). The documentation is a bit ambiguous in that respect.

The main problem is, that only your customers customer (!) with whom you have to co-operate closely and directly can help you through testing. People there are not that technical, even though their job title says so. They own the SRM testsystem (you don’t) and you always have to ask for a punch-out-click in order to find new insights integrating OCI into your eShop. It took us many weeks instead of a couple of minutes due to vacation and corporate politics and overhead. Finally we integrated an all-encompassing logging for each minor step and response to reduce the communication to ‘please click again’ – and bamm, 3 days later we had a bunch of logging information powering the next development step.

Here is a SVN diff showing all files we added or modified in the gambio release with a description what we modified:

Changed files in original Gambio distribution:
M  checkout_shipping.php (added optical info, we're in OCI-mode)
M  includes/configure.php (added a switch to work on live and development server)
A  includes/modules/oci/oci.php (Main functionality wrapped in a class)
A  includes/modules/oci/ociParameters.php (Configuration with additional parameters for SRM)
A  includes/modules/oci/logger.php (Own simple logger to see what happens during test calls)
A  oci_out.php (exit sending real hidden html form onLoad via JavaScript, redirecting to SRM)
M  admin/includes/configure.php (added a switch to work on live and development server)
A  oci.php (entry point for calling Punch-Out incl authentication)
M  checkout_confirmation.php (added optical info, we're in OCI-mode)
M  checkout_process.php (redirect to exit oci_out.php instead to successpage after checkout)
M  templates/EyeCandy/module/checkout_shipping.html (added optical info, we're in OCI-mode)
M  templates/EyeCandy/module/checkout_confirmation.html (added optical info, we're in OCI-mode)
M  templates/EyeCandy/module/checkout_success.html (added optical info, we're in OCI-mode)


OCI mechanics work like this:

At first the purchase department selects a supplier in their SRM. As soon as the order agent starts shopping, SRM performs a ‘Punch-Out’, which opens a browser window with a URL like this one:

http://your-php-eshop.com/oci.php?Z_EMAIL=ebp%40xxx%2ede&password=password&userid=8&BYPASS_INB_HANDLER=X&BYPASS_OUTB_HANDLER=X&OCI_VERSION=4%2e0&OPI_VERSION=1%2e0&returntarget=_top&HOOK_URL=https%3a%2f%2ftestsystem7%2esrm%2ede%2fsap%2fbc%2fnwbc%2fsrm%2f%3fsap-nwbc-action_url%3d%252fsap%252fbc%252fwebdynpro%252fsapsrm%252fwda_l_fpm_gaf%253bsap-ext-sid%253dKMDRyFbbH6pX0000mAWtxW–KMDRyVbbH6pX0000mAWtxW–%253fsap-ep-tstamp%253d20130419102708%2526sap-wd-tstamp%253d20130419102708%2526sap-ep-version%253d7%26NavigationTarget%3dportal_content%2fnwbc_back%26NavMode%3d3%26UsePost%3dTrue%26SAPSRM_RESUME_ID%3dSAPSRM_OCI%26sap-nwbc-has_post_params%3dX%26sap-client%3d010%26sap-language%3dDE%26sap-nwbc-node%3dresume_appl

This call goes to your /oci.php. It needs to do the following:

  • It checks if there is a customer/user in your eShop that can be authenticated with the userid, password and Z_EMAIL parameters. If so /oci.php logs the user in to the eShop. Of course you have created the customer maually before. Send the SRM owner the values of the parameters for their punch-out and test if auto-login works via punch-out.
  • /oci.php must remember the value of the parameter HOOK_URL in the session. It is only valid for this purchase and the contents of the shopping cart must be posted back to this URL – it is your value for your form’s action attribute. Remember: It can NOT be done via cURL.

The user is now navigating your eShop and fills his shopping cart and proceeds to the checkout and clicks the final checkout button. The purchase is saved in the shopping application by your eShop system. Instead of redirection to the shop-success page a hidden form is created (see OCI documentation for fieldnames and fixed field values necessary) with the shopping information for the SRM and it is sumbitted immedately by javascript. You need to intercept the redirection to the success page and redirect to your /oci_out.php instead providing this form. The action url has been handed over via the HOOK_URL parameter on punch-out.

The hidden form looks like this:

<!– OCI autosubmit –>
<script type=”text/javascript” language=”javascript”>
document.getElementById(“srm_form_submit”).click();
</script>

<form id=”srm_form” action=”https://testsystem.srm.customer.com/sap/bc/nwbc/srm/?sap-nwbc-action_url=%2fsap%2fbc%2fwebdynpro%2fsapsrm%2fwda_l_fpm_gaf%3bsap-ext-sid%3dKN2O27IP8fdX0000mAWtxW–KN2O2NIP8fdX0000mAWtxW–%3fsap-ep-tstamp%3d20130419104715%26sap-wd-tstamp%3d20130419104715%26sap-ep-version%3d7&NavigationTarget=portal_content/nwbc_back&NavMode=3&UsePost=True&SAPSRM_RESUME_ID=SAPSRM_OCI&sap-nwbc-has_post_params=X&sap-client=010&sap-language=DE&sap-nwbc-node=resume_appl” method=”POST”><code> <input name=”NEW_ITEM-LINE[1]” type=”hidden” value=”00001″ />
<input name=”NEW_ITEM-DESCRIPTION[1]” type=”hidden” value=”Herren Fleece-Jacke Langley Rot” />
<input name=”NEW_ITEM-LONGTEXT_1:132[]” type=”hidden” value=”Herren Fleece-Jacke Langley Rot” />
<input name=”NEW_ITEM-QUANTITY[1]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-UNIT[1]” type=”hidden” value=”C62″ />
<input name=”NEW_ITEM-PRICE[1]” type=”hidden” value=”41.87″ />
<input name=”NEW_ITEM-CURRENCY[1]” type=”hidden” value=”EUR” />
<input name=”NEW_ITEM-PRICEUNIT[1]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-LEADTIME[1]” type=”hidden” value=”21″ />
<input name=”NEW_ITEM-VENDOR[1]” type=”hidden” value=”16222″ />
<input name=”NEW_ITEM-VENDORMAT[1]” type=”hidden” value=”287{2}5{1}23{5}17″ />
<input name=”NEW_ITEM-MATGROUP[1]” type=”hidden” value=”41010000″ />
<input name=”NEW_ITEM-LINE[2]” type=”hidden” value=”00002″ />
<input name=”NEW_ITEM-DESCRIPTION[2]” type=”hidden” value=”Pique Polo Safran” />
<input name=”NEW_ITEM-LONGTEXT_2:132[]” type=”hidden” value=”Pique Polo Safran” />
<input name=”NEW_ITEM-QUANTITY[2]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-UNIT[2]” type=”hidden” value=”C62″ />
<input name=”NEW_ITEM-PRICE[2]” type=”hidden” value=”16.95″ />
<input name=”NEW_ITEM-CURRENCY[2]” type=”hidden” value=”EUR” />
<input name=”NEW_ITEM-PRICEUNIT[2]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-LEADTIME[2]” type=”hidden” value=”21″ />
<input name=”NEW_ITEM-VENDOR[2]” type=”hidden” value=”16222″ />
<input name=”NEW_ITEM-VENDORMAT[2]” type=”hidden” value=”112{2}4{1}1{5}16″ />
<input name=”NEW_ITEM-MATGROUP[2]” type=”hidden” value=”41010000″ />
<input id=”srm_form_submit” style=”display: none;” type=”submit” value=”Go” />
</code></form>

After this form has been submitted, SRM generates another page also including a hidden form, additional internal information and lots of JavaScript logic also with auto-submit. Please note: a) The array counter must start at 1 not at 0. And b) note the syntax of the longtext field.

The page is generated and ‘executed’ on the server side of the SRM system! This is the reason why it could not be done via cURL: The action to the HOOK_URL opens a page on the SRM server submitting another (!) form which finally writes the purchase to SRM – not your initial POST. It also closes the browser window for the order agent. If SRM can not close the browser window something went wrong.

Migrate Email via IMAP using imapsync

February 25, 2013 by marco · Leave a Comment
Filed under: Linux, Productivity, Tools 
Social Tagging: • •

This time I had to migrate not just a domain, but a whole bunch of existing imap-email accounts with it.

To do that, just create all email accounts on the new machine and have a list with all credentials ready. For the hard work I used a very good Perl tool – imapsync.

Very cool with imapsync is, that it actually syncs from the old server. So you do not have to deal with double email or lost mail. See also imapsync description here.

Receipe for Debian Linux:

Download imapsync from https://fedorahosted.org/released/imapsync/:

$ wget https://fedorahosted.org/released/imapsync/imapsync-1.525.tgz

Unzip and cdir into the unzipped directory.

Check your perl installation with:

$ perl -c imapsync

If you see a “Can’t locate Mail/IMAPClient.pm in @INC…” you need to install the IMAPClient:

# cpan Mail::IMAPClient

If you see “imapsync syntax OK”, you already have the required modules.

Migrate one emailbox:

$ ./imapsync --host1 imap.1und1.de --host2 imap.man.ticore.it --port1 993 --port2 993 --ssl1 --ssl2 --user1 testuser@migration-domain.xx --user2 new_username_on_target_system --password1 xxx --password2 yyy

If you get a “Can’t locate IO/Socket/SSL.pm in @INC…” error, this might fix the problem:

# apt-get install libio-socket-ssl-perl

Retry migration:

$ ./imapsync --host1 imap.1und1.de --host2 imap.man.ticore.it --port1 993 --port2 993 --ssl1 --ssl2 --user1 testuser@migration-domain.xx --user2 new_username_on_target_system --password1 xxx --password2 yyy

If output says “Info: host imap.man.ticore.it says it has NO CAPABILITY for AUTHENTICATE LOGIN” just add ‘–authmech2 PLAIN’

Retry migration:

$ ./imapsync --host1 imap.1und1.de --host2 imap.man.ticore.it --port1 993 --port2 993 --ssl1 --ssl2 --user1 testuser@migration-domain.xx --user2 new_username_on_target_system --password1 xxx --password2 yyy --authmech2 PLAIN

Now you have test-migrated one email account. If you have more, which is likely, takle a look at imapsync-1.525/examples. You find 2 important files:

  • file.txt – contains a list of accounts with the self-explanatory line-format user001_1;password001_1;user001_2;password001_2
  • sync_loop_unix.sh – mass-migrates the accounts from the list. Pleas note: You must set the two imap-hostnames manually in this file! Check the directions in the file. Use via
    $ ./sync_loop_unix.sh

Have fun syncing your IMAP stuff around!

Migrate arbitrary Content to WordPress – e.g. from typo3 tt_content

September 28, 2012 by marco · Leave a Comment
Filed under: Problems, Snippets 
Social Tagging: • • •

If you see yourself confronted with the following task, here is a quick receipe: Setup a wordpress instance for us and import all our content from system x. In this case we focus on the import part. ‘System x’ was a somewhat customized typo3 setup. I got myself an ordinatry editors login to identify the articles which had to be migrated. After some fulltext searching and access to phpMyAdmin I identified the area, where I hat to extract the content from. It boiled down to the table tt_content. At this point you can forget about typo3. All articles could be extracted with this SQL query:

SELECT uid, pid, header, bodytext FROM tt_content WHERE header<>'' AND CType='editorialcontent';

Export the content using phpMyAdmin’s export to PHP array functionality. You end up with a file like shown in the screenshot. If you have HTML-content and doubt, if this could be valid php, save the file under export.php and do ‘$ php -l export.php’ – it says ‘No syntax errors.’ – Good.

OK, next you need to get familiar with wp_insert_post(). You need to loop your exported content over it. And that is it. This is how I did it:

  • Create a directory in your wordpress setup ‘wp-import’.
  • Copy/move your export.php array into this directory.
  • And execute the following self-explanatory file located also in your wp-import directory: import.php

That’s it.

Use GROUP BY to concatenate strings in MySQL?

September 22, 2012 by marco · Leave a Comment
Filed under: Snippets, mySQL 

I just had the case with mySQL where I needed to aggregate via GROUP BY but at the same time concatenate the content in the body-field.

The test result (see image) came using:

SELECT m.template_id, m.site, m.path, m.name, tf.body FROM menu AS m
LEFT JOIN template_fields AS tf
ON tf.template_id=m.template_id
WHERE m.type='template' AND m.inactive=0

The solution to get the concatenated content:

SELECT m.template_id, m.site, m.path, m.name, GROUP_CONCAT(tf.body SEPARATOR ' ') AS body FROM menu AS m
LEFT JOIN template_fields AS tf
ON tf.template_id=m.template_id
WHERE m.type='template' AND m.inactive=0 GROUP BY m.template_id

Fraudulent emails trying to get IPhone without paying

September 11, 2012 by marco · Leave a Comment
Filed under: Other stuff 

It seems, that being upset is a good motivator wrinting blogposts… Sorry for being lazy for such a long time.

Currently I am selling lots of ‘old stuff’ that has not being used for 2 years, among other things also my iPhone 3GS advertising it to local boards such as www.quoka.de. In short it goes like this:

  • Viallain askes if the IPhone is still available. I answer ‘yes, come bring cash and pick it up’.
  • Villain: Can you send it, I pay for the shipping. Me: OK that’s x bucks extra. Fine.
  • Villain: Please send me your interantional bank-account number, which I did, because it is as safe as giving out your local bank account number and any debit actions can be objected to for 6 weeks at your bank.
  • And now it gets weird: I receive a fake bank email (see below) saying, the money is in transit. By the way it had a BIC from another account – copy&paste error. Villain is also lazy and probably a bit disorganized and stupid too. I can get the money as soon as I send the package to an address in Italy (see map and Streetviev of the house below – probably the villain’s house, not sure but likely) and provide tracing number of the package. At this point it would be fun to return a fake one or a ticking package or some of the other 100 jokes jumping to my mind at this moment ;), but I have currently no time to do that.
  • If you disagree with this procedure, villain gets angry. Maybe this works in some % of the cases.
  • End of communication.

Here are some screens of emails. Have fun:

First contact email

Not from the bank

Note from the bank 2


Big view

This is the address in Streetview.

Email addreses I got spammed so far:

  • talk2peace2016@yahoo.com
  • gina_sigrid@yahoo.de
  • bostinsley@yahoo.de
  • bostinslley@yahoo.de
  • surely more to come…

Update: There is a webinterface, see http://abuse.yahoo.com or forward phishing-email to phishing@cc.yahoo-inc.com.

Attachment: mail-content-for-search-engines

Next steps: Informed local commissario di San Martino, Yahoo abuse dept. and Nordea Bank – have fun getting busted, loser!

Update: Received a cheque for my pinball table (Dirty Harry, abvertised for 1500EURs). Rest was meant to send container with ‘some other things’ to Urkraine, I should pay for the shipping with the cheque. This is it. Find the faults:

Cheque sent via mail

Awesomely entertaining shit!

Some more useful linux commands

July 10, 2012 by marco · Leave a Comment
Filed under: Linux 

Check machine for open ports, running services, guess OS:
$ sudo nmap -O man.ticore.it

Check applictaion running:
$ nc 123.123.123.123 8983
And the type GET (+enter) to get a response.

Unzip multiple gzipped SQL-files and stream them directly into your mysql database:
$ gzip -cd db_123.sql.gz  db_123_[234567].sql.gz | mysql -u<db_user> -p<db_pass> <db_name>

List each dir-size in human readable format to find the big files:
$ du -hsc my-packed-webspace.com/

List each file greater 10MB recursively:
$ find . -size +10000k -exec du -h {} \;

Set a yourself a userfriendly commandline editor before you do things like $ corontab -e:
$ export EDITOR=/usr/bin/nano

When copying or (un)zipping large files, instead of re-emitting commands to see progress, use watch and relax:
$ watch -d -n 5 ls -lh

List files without user:group and size:
$ find lsis/sources/bionity/2013/03/28/ -type f -print

Stuff I need to look-up every time

November 7, 2011 by marco · Comments Off
Filed under: Snippets, mySQL 
Social Tagging: •

You lost your leading zeros of your varchars importing/exporting your data. You have this example data in your table storagebins:

area = L, row = 1, rack = 2, shelf = 3
and it is supposed to look like this
area = L, row = 001, rack = 02, shelf = 03

Use this SQL to correct this:

UPDATE storagebins SET row=LPAD(row, 3, '0');
UPDATE storagebins SET rack=LPAD(rack, 2, '0');
UPDATE storagebins SET shelf=LPAD(shelf, 2, '0');

VBA Macro to Convert Dates in a Range – the Dirty Way

November 1, 2011 by marco · Comments Off
Filed under: Snippets, VBA, mySQL 
Social Tagging: • •

I used this hacked Excel-Macro to convert all dates in a selected range from a manual data import using Excel to generate all required INSERTs for our MySql database

  • from 03.08.2011 16:51:01
  • to 2011-08-03 16:51:01

In case you wonder, where you can find your Visual-Basic editor and execute your VBA-code… follow these instructions. Here is the code:

Sub PrefixSingleQuoteToRange()
Dim cell As Range
Dim year As String
Dim month As String
Dim day As String

For Each cell In Selection.Cells
' Convert 03.08.2011 16:51:01 to 2011-08-03 16:51:01
year = Mid(DateValue(cell), 7, 4)
month = Mid(DateValue(cell), 4, 2)
day = Mid(DateValue(cell), 1, 2)
cell = "'" & year & "-" & month & "-" & day & " " & TimeValue(cell)
Next

End Sub

If you scratch your head… This is how I ‘generated’ my INSERT-statements:

  • One sheet equals one table.
  • Each line represents a record.
  • For example you have A to L fields paste the following into your cell M:

="INSERT INTO orders (order_id, processing_start_datetime, processing_end_datetime, delivery_datetime, staff_id, store_id, name, phone, comment_order, comment_procession, created_at, updated_at) VALUES ('"&A2&"', '"&B1&"' ,'"&C1&"' ,'"&D1&"' ,'"&E1&"' ,'"&F1&"' ,'"&G1&"' ,'"&H1&"' ,'"&I1&"','"&J1&"','"&K1&"','"&L1&"');"

  • It is a lot of copy and pasting. But in the end all your INSERTS can be copied and pasted into a txt-file to be dumped into your database.

If you know a smarter way to ‘import’ MSSQL data into MySql, please let me know. And I guess there must be many of them ;).

More Efficient Running

June 11, 2011 by marco · Comments Off
Filed under: Running, Sports 
Social Tagging: • •

I hope you geeks out there do any sports regularly despite the common prejudices. I do and I am getting a lot in return. Moving your body in a skilled way running, cycling, swimming, playing whatever is a form of intelligence, gives you new energy for your work, clears your mind and sometimes even delivers unexpected inspiration too.

So, I would like to share a very interesting ‘life hack’ with you in order to have more fun and less pain running and improve distance, endurance and speed at the same time.

What happened: One year ago I simply started running, bought myself a basic set of running gear and exercised once or twice a week – without any training or professional advice. My distances increased during the first months from 2km up to the point where I could run around 10km. My average pace was 6:30min per km at that time (very slow like a power walker). I thought I needed to be as slow as that in order to make the 10km.

Then I registered for a half marathon and tried to increase the distance to 20km – I thought this would just be as simple as getting from zero to 10km. But I couldn’t. It was so exhausting to run 10km or maybe sometimes even 13km. The complete crash came, when I bought myself new running shoes – you know, not the ugliest and not the cheapest ones I could jog around the local running store without thinking that they do not fit well…

Honestly, I frequently read some running magazines and books but never ever came across the things I really needed to know to be able to run a half-marathon without hurting myself.

After the first exercises with the new shoes distances decreased and I felt more and more pain the day after the exercise in my left knee. This must have been the point where most people stop running and do something else! Despite the pain I forced it and got very frustrated. Sometimes it happened, that after 5/8km I made a clumsy step on uneven ground and BANG I had to stop and sometimes take the tram or a taxi or at least take a shortcut and walk back home. Distances I could run decreased rapidly. The last run was just some 100 meters and BOOM. What a mess!

So I went to a doctor, seriously concerned. 5 weeks before my first half-marathon event. This was the first time I heard about pronation and supination and about how this incredibly built foot absorbes the shock from each of your steps when running. The doctor asked me that type of runner I was and I had no f#$%&ng clue what he was talking about. It turned out, that I bought just the wrong shoes for my anatomy. Shocks went directly into my knee causing direct damage there and in my lower back too. Diagnose: irritated knee. Prescription: 2 weeks pause and painkillers. Advice: Beginner, buy yourself the right shoes and make your homework before you ruin your health! The doctor gave me a contact to a good physical therapist and urged me to do some training sessions with him.

A long story, but important to understand where I came from. I would like to share with you geeks what I learned about running and running equipment and I hope to save you from experiencing a similar disaster:

  1. Buy yourself the right running shoes: Make your homework and understand the terms pronation and supination. This describes if your foot tilts to the inside or the outside when it hits the ground while running. Check youtube.com (search for ‘running pronation supination’) for some very good explanations. Go to a good store that can make a video analysis of you running along a track wearing your new shoes. Watch your steps in slow motion with the sales assistant and let him make a definitive recommendation for a pair of shoes. Ensure that the store takes the shoes back if you have any problems in your knee or back, even after some weeks of training. The color of the shoes is absolutely no factor.
  2. Learn how to do forefoot running: Watch yourself running or videotape yourself. Make your homework what a good posture looks like and how your feet push the road behind you with every step without braking when you step ahead of your point of balance. The whole mystery boils down to two points: Avoid heel strike, practice forefoot running. Again check youtube.com (search for ‘forefoot running’ and ‘running heel strike’) for the details. It will take some time to understand the real difference between a good running technique and your old, intuitive bad one. Be aware that this changes a lot: You need to have stronger calves and do regular ham string exercises. Otherwise they will be the new parts that hurt. Start to increase distances and speeds again from zero. Do not expect to extend your distances like in my case from 10km. Start over when you change your running technique. Forefoot running increases the speed and step length and you will get tired more quickly. But this is just when you start. So, start with shorter distances and do not overpace – like I did, since I had to be ready for the halfe marathon in a couple of weeks.
  3. Join a running group: This is an optional part but can help a lot if your group or class consists of some experienced runners with a proven track record – multiple iron (wo-)men or the like. There you can learn a lot and the stories you hear may also push you forward.

I really hope, to save you from the pain and frustrations I experienced. In the end I finished with the official time of 2:20h on my first half marathon and it will not have been the last one.

Update: Finished 5th www.lucernemarathon.ch (half-distance) on 30.10.2011 in 1:58:40.

Have fun running!

Emulate sendmail on your Dev Machine

February 8, 2011 by marco · Comments Off
Filed under: Linux, Serverstuff, Tools 
Social Tagging: • • • •

In order to prevent email delivery during development and log all email messages that would have been delivered, you can actually do a simple trick: Replace the file /usr/sbin/sendmail (on Ubuntu, use ‘locate sendmail’ to find it if it lies elsewhere) with this little shell-script, or rather make a _bak of the original and save the following instead of the sendmail binary:

#!/bin/bash

LOGDIR="/tmp"
PREFIX="sendmail"
NOW=$(date +%Y-%m-%dT%H.%M.%S)
CNT=1
PRIVATELOG="$LOGDIR/$PREFIX-$NOW.$CNT.log"
COMBINEDLOG="$LOGDIR/$PREFIX-combined.log"

# If privatelogs are being used...
if [ ! -z "$PRIVATELOG" ]; then
# ...make sure the filename is unique and create the file
while [ -f $PRIVATELOG ]; do
CNT=$(($CNT + 1))
PRIVATELOG="$LOGDIR/$PREFIX-$NOW.$CNT.log"
done

echo "$0 $*" > $PRIVATELOG
else
# ...otherwise swap filenames
PRIVATELOG=$COMBINEDLOG
COMBINEDLOG=''
fi

echo "[$NOW]" >> $PRIVATELOG
while read BUF
do
echo $BUF >> $PRIVATELOG
done

# Append privatelog to combinedlog when both logs are used
if [ ! -z "$COMBINEDLOG" ]; then
echo "[$NOW]" >> $COMBINEDLOG
cat $PRIVATELOG >> $COMBINEDLOG
fi

exit 0

When your application now sends mail, these things happen:

  • No email is actually sent.
  • The message gets appended to the file /tmp/sendmail-combined.log, on which you could set a ‘tail -f’ in order to see which emails would have been sent and what contet they would have.
  • One new file (e.g. /tmp/sendmail-2011-02-08T08.02.48.1.log) gets written for every email sent. I personally only use the combined file.

Inspired by http://stackoverflow.com/questions/3710864/simulating-sendmail-with-dummy-script

Drawing Lines Using Google Static Maps API

November 2, 2010 by marco · Comments Off
Filed under: Links, Read-Write-Web, Snippets, Tools 
Social Tagging: •

If you are looking for a simple way to display lines like a roundtrip on a map, Google’s Static Maps API might be for you.

Documentation and examples: http://code.google.com/intl/en-US/apis/maps/documentation/staticmaps/

Limits:
- There is a query limit of 1000 unique (different) image requests per viewer per day.
- URLs are restricted to 2048 characters.


Here are some self-explanatory examples:

Using geo-coordinates:
http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|49.584486,8.462219|49.441343,8.462219|49.384161,8.654230&size=400x400&sensor=false

Using city names:
http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|Stuttgart|Mannheim|Bad Homburg|Frankfurt|Würzburg|Stuttgart&size=500x200&sensor=false

This is how you would use it on your website just use it like a local image:
<img src="http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|Stuttgart|Mannheim|Bad Homburg|Frankfurt|Würzburg|Stuttgart&size=500x200&sensor=false" alt="my roundtrip" />
my roundtrip

Pretty slick!

In case you need to extract geocoordinates, there are lots of online browser tools available, just click on a spot and copy-paste the coordinates into your static maps-URL:

Clean your PHP4 Legacies using sed

October 19, 2010 by marco · Comments Off
Filed under: Linux, Snippets, Tools 
Social Tagging: • • •

If you have to deal with very old PHP4 legacy code containing every syntax crime you may know from the early years, how would you handle it? Give it to your junior people to fix it manually? I like to have at least some handy helpers for the first rough corrections. I found sed to be a very powerful helper here.

Code you might encounter - associative array elements without quotes.

I spent quite some time to find useful regular expressions to help me. This is how I did this:

A way to test your regular-expression is to echo a sample and apply your regex to test the results:

echo '$_REQUEST[action] reise_l_topic_ids[] $dat[ticket_order]' | sed "s/\$\([a-zA-Z0-9_]\+\)\[\([a-zA-Z0-9_]\+\)\]/$\1['\2']/g"

Once it works you can apply your regex to one file:

sed -i "s/\$\([a-zA-Z0-9_]\+\)\[\([a-zA-Z0-9_]\+\)\]/$\1['\2']/g" my_old_file.php

Or apply your regex to all *.php-files recursively below the current directory to a whole project:

find . -name "*.php" -exec sed -i "s/\$\([a-zA-Z0-9_]\+\)\[\([a-zA-Z0-9_]\+\)\]/$\1['\2']/g" '{}' \;

By the way: The usage of sed works fine on your linux command line, but not on OSX. The syntax is slightly different here (sed -i “” -e “s/blah/blubb/” file). This is of course only a start to automate otherwise painfull and boring corrections down to just a few seconds. It will not save you from special manual work and break syntax at some points. But it weeds out 90% and leaves you with the other 10% acutal manual work.

You could imagine many more sed regexes e.g. to replace short tags <?=$my_var?> to a proper <?php echo $my_var; ?> etc.

echo '<?=$out?> sakdhs sakdhas k <?php echo $xyz; ?> ddd' | sed "s/<?=/<?php echo /g";

I will collect more regexes as I need and find them. If you have ideas please add them in the comment section.

Stuff I need to lookup every time

September 30, 2010 by marco · Comments Off
Filed under: Linux, Productivity 
Social Tagging: • •

Set ignore to all files of a directory with subversion:
$ cd cache
$ svn propset svn:ignore '*' .
$ svn ci . -m 'Ignore set on cache dir.'

Show changed files between two revisions, overview
$ svn diff -r 300:HEAD --summarize

Show changed files between two revisions, for each revision:
$ svn log -v -r 300:304

See overall latest 20 commit-messages:
$ svn log -l 20

Branching and merging:
See: http://blog.evanweaver.com/2007/08/15/svn-branching-best-practices-in-practice/

Only grep in php source files, not jpgs, movies etc.:
$ grep -i 'whatever' `find . -name '*.php' -print`

Add all new files in a large filestructure to subversion, like after an update of vendors in Symfony2
svn st | grep "^\?" | awk "{print \$2}" | xargs svn add $1

Remove all deleted files from a large filestructure from subversion, like after a vendors update in Symfony2
svn st | grep '^!' | awk '{print $2}' | xargs svn delete --force

Setup external libraries in subversion:
svn mkdir ZFVersions;
svn add ZFVersions;
svn ci ZFVersions -m 'Added dir for all versions.';
cd ZFVersions;
svn mkdir 1.11;
svn add 1.11;
svn ci 1.11 -m 'Added version subdir.'
cd 1.11;
svn propset svn:externals 'Zend http://framework.zend.com/svn/framework/standard/tags/release-1.11.0/library/Zend' .; # This will checkout in another dir Zend into your dir 1.11. You need this since autoloading is using paths like this require_once(Zend/Feed/Rss.class.php)!!
svn commit -m 'Set external.';
svn up .; # Loads external lib.

Correct date problems in a mysql database – 2020 instead of 2011 in YYYY-MM-DD dates:
UPDATE accounts SET member_startdate = CONCAT('2010', '-', MONTH(member_startdate), '-', DAYOFMONTH(member_startdate)) WHERE YEAR(member_startdate) > 2011;

What is it you guys always look up?

Sync Your Stuff to S3

June 24, 2010 by marco · Comments Off
Filed under: Mac, Snippets, Tutorials 
Social Tagging: • •

This is a receipe how I save stuff to S3 from my Mac:

1.) Signup with S3: http://aws.amazon.com/s3/ (check pricing!). This will give you access to the AWS Management Console.

2.) Create a Bucket: This can be done via the AWS Management console. If you are not familiar with the concept of ‘buckets’ check-out the S3 documentation. Simply put, it is a virtual storage device that has a fixed geographical location.

3.) Go to ‘Security Credentials’ in your account settings in the AWS Management console and create an accesskey.

4.) Download JetS3t. You then have the following directory on your Mac:
/Applications/jets3t-0.7.3

Open the Terminal and change into the bin directory:
$ cd bin;

Create a file named synchronize.properties there:
$ nano synchronize.properties;
and save the following content using your keys from step 3:

accesskey=XXXXXXXXXXXXXXXXXXXX
secretkey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5.) To sync the contents in the path /Users/marco/MySyncStuff with the bucket myBucketName use this command:
$ ./synchronize.sh UP myBucketName /Users/marco/MySyncStuff/ –properties synchronize.properties;

Of course you can create as many buckets as you like and script and schedule your data syncs now from here as you wish. Use the command
$ ./synchronize.sh –help
to see what the synchronize.sh else has on offer.

6.) Browsing buckets: JetS3t has its own S3 browser. To start and use it do the following:

$ cp cockpitlite.sh cockpitlite.command;
$ ./cockpitlite.command &;

You should see the Java coffee cup on your task bar. Use your keys to log in and browse your buckets.

You can also use the free S3 Browser for Mac.

Entrepreneurial maxims

May 27, 2010 by marco · Comments Off
Filed under: Entrepreneurship 
Social Tagging: •

I recently came across an interesting table comparing typical mindsets of freelancers to those of entrepreneurs, which I immedately printed out and sticked to my refrigerator as a reminder. Since we currently have startup conditions too in our company, I know that it takes a lot of discipline to make adjustments to a mindset that served you well for the last 10 years but now more often gets in your way as daily tasks, priorities and responsibilities (must) change.

Area Freelancer / skilled employee
 Entrepreneur
Employees, colleagues I can get stuff done best on my own. My employees are better trained, specialized and have more skills.
Complexity More complexity makes things more interesting. Simple things make you more successful.
Value in own work I cost less than my employees, since they get a fixed salary. My work is the most expensive and should not be wasted.
Time Money has more value to me than time. I spend time to save money. Time is more valuable than money. I spend money to save time.
Money Money means security. Money yields opportunity for investments and ventures.
Education, learning My professional development is important to me. My personal development is most important.
Risk I try to avoid risk whenever I can. Steps to success are naturally accompanied by risk-taking. Being employed is for example more risky than being an entrepreneur.
Mission My profession is my mission. My mission is to build a great company and actively build my environment with it.
Comfort zone I am best at, where I feel save and comfortable. My success and growth both lie exactly where I feel uncomfortable.

Thanks to Google I also found the source of the table: It is taken from a German blog post titled ‘Warum manche Selbständige nicht zu Unternehmern werden‘ (Why some freelancers never become entrepreneurs).

Hope it helps some of you too.

MODx Revolution Links

April 6, 2010 by marco · Comments Off
Filed under: Links, Tools 
Social Tagging: •

I have recently been playing with the new MODx (‘Revolution’) and I must say, I love the concept and it’s flexibility. Just to make things easier for folks starting out with MODx or who migrate from MODx Evolution here are some recommended links.

General:

Focus topics:

  • Tag format changes from Evolution to Revolution: The formats for content tags have changed. They now can be nested, they can carry parameters and look a lot cleaner.
  • Use the Package Manager to install your snippets: There are still some packages to migrate (e.g. AjaxSearch). And if you install Login and wonder why you can not logout, just check the naming of the parameters. I am sure these issues will soon be solved.
  • Internationalization: The new Lexicon facility makes the creation of multilingual pages easier, especially central shared Pages or Chunks used in all available languages like the login, password reminder, 404 etc. You can even use Placeholders in lexicon entries and fill them dynamically with Parameters. On bigger sites I would prefer to have a home page for each language and maintain each language in separate menu trees. This is explained in this forum entry. Notice, the tutorial is based on Evolution, but works the same way in Revolution. You only need to use the new tag syntax.
  • New access to MODx objects: Most of the methods from Evolution to interact with $modx will be deprecated soon in Revolution. If you are writing Snippets or Plugins you need to get familiar with $modx->getObject() and Co.
  • New permission system: Instead of the distinction of Web Users and Manager Users in Evolution, Revolution switched to ACL (Acesss Control Lists) and Context based permissions.
  • Make Resources external to make use of version control: Required resources can also be loaded from files instead of saving everything to the MODx database.
  • Friendly URLs: How to use the enclosed .htaccess and what MODx settings to use.

What I dont like with MODx Revolution is the slow Manager interface based on ExtJS. If you have to go through a longer configuration or exploration session, waiting for the frontend to finish loading can be a drag. But the ajaxy Manager helps you to perform some actions in parallel without you losing your form content, e.g. if you need a href and need to lookup the document-ID you just click on the Resource tab and look it up in the tree.

Overall I must say MODx Revolution makes the expression to me like a very well crafted, cutting-edge and thought through CMS. Congratulations and many thanks to all contributors! You are awesome!

MSSQL CAST() and GROUP BY

March 29, 2010 by marco · Comments Off
Filed under: Problems, Snippets 
Social Tagging:

Just in case somebody has a similar problem… Coming mainly from MySql I had to deal with some MsSql/SQL Server specialties. Consider the following working SQL for MsSql:

SELECT AK.AK_TEXT, BU.BUDGET_BEREICH ,TY.AKTIONSTYP_TEXT ,AK.AK_STARTDATUM ,AK.AK_ENDEDATUM
FROM AKTIONEN AS AK WITH (nolock)
LEFT JOIN BUDGET_TRAEGER AS BU
ON AK.AK_BUDGET_NUMMER=BU.BUDGET_NUMMER
LEFT JOIN AKTIONSTYP AS TY
ON AK.AK_TYP=TY.AKTIONSTYP_TYP
LEFT JOIN AKTIONEN_FILIALE AS AF
ON AK.AK_LFDNR=AF.AF_LFDNR
WHERE (
  ( AK.AK_ENDEDATUM<=CAST('2010-06-30' AS smalldatetime)
   AND AK.AK_ENDEDATUM>=CAST('2010-01-01' AS smalldatetime)
  )
  OR ( AK.AK_STARTDATUM<=CAST('2010-06-30' AS smalldatetime)
    AND AK.AK_STARTDATUM>=CAST('2010-01-01' AS smalldatetime)
  )
)
GROUP BY AK.AK_TEXT, BU.BUDGET_BEREICH ,TY.AKTIONSTYP_TEXT ,AK.AK_STARTDATUM ,AK.AK_ENDEDATUM;

Interesting parts explained:

(a) The addition ‘WITH (nolock)’ prevents my application from locking the table during select queries.

(b) The fields from which I liked to select a bunch of records touching a daterange between AK_STARTDATUM and AK_ENDEDATUM were of the fieldtype ‘smalldatetime’. This is some strage SQL Server format. It needs to be converted to with the CAST() function in order to return the expected resultsets. It understands several input formats. You can test the correct conversion with a simplified SQL statement like this: SELECT CAST(’2010-03-29′ AS smalldatetime);

(c) The statement will not work and return an error if you do not include ALL your selected fields also in the GROUP BY clause.

Feedbackloops in Software Development

February 28, 2010 by marco · Comments Off
Filed under: Productivity, Software development 
Social Tagging: •

Since feedback is a form of communication, excessive feedback will have the effect like an explosion in communication paths and will eventually produce more noise than signal.

Any feedback loop in the system must help facilitate the management of change and efficiently adapt requirements along the the way. If this is not the case for a feedbackloop, folks will try to avoid it.

As developers we also experience a decrease in quality and value of feedback if it is not immediate enough and/or requires too much effort from a team member to receive it. So most of the feedbackloops are built right into modern software development. Take a look at the following list of areas where a software development team or team member receives feedback in different forms:

  • Your IDE checks syntax and indicates errors before you hit ‘save’.
  • A release demo for your customer generates prose user feedback.
  • Commit hooks refuse your changes.
  • Deployment on development and staging systems works or doesn’t.
  • The Compiler compiles or refuses to compile and returns errors.
  • Daily standup meetings serve as publishing institution of individual statuses, schedules and problems. Team members get updated, help each other out or problems get escalated up the command chain.
  • Pair programming for direct feedback from colleagues on development skills, technical decisions, used tools etc.
  • Sprint reviews reflect development processes, collect team feedback and improve them.
  • Test driven development repeatedly asserts requirements and APIs of existing and future components and their interdependencies.
  • The continuous integration server runs all test suites, generates documentation, runs the build, code sniffer etc. and informs the whole development team if something goes wrong.

I hope this article serves you as a short reminder for where and how feedback can create value in your software projects. And remember: Software is written by humans and feedback is the opposite of thought-reading, which often leads to unnecessary friction and pain.

Interesting Links:

Software Craftsmanship Manifesto

January 22, 2010 by marco · Comments Off
Filed under: Books, Software development 
Social Tagging: • • •

I have just received a new book: “Apprenticeship Patterns“: Skimming through the first pages I noticed the words in the 3rd column extending from the points in the 2nd column and thought, ‘Hmm, I have heard about these 2nd column points’…

The following table popped up in my mind, which I would like to share with you:

Software development practices:

Amazon

Traditional
<2000		 Agile Manifesto
2001		 Craftsman Manifesto
2010
processes and tools Individuals and interactions but also a community of professionals
comprehensive documentation Working software but also well-crafted software
contract negotiation Customer collaboration but also productive partnerships
following a plan Responding to change but also steadily adding value

Any ideas for a 4th column? Looking forward to actually reading the book…

Backup MySql-Database using only FTP

January 18, 2010 by marco · Comments Off
Filed under: Problems, Snippets 
Social Tagging: • •

We have just had the case of having ftp access to a site on a shared hosting LAMP webspace but needed also the database and mainly the database scheme to setup our own development-system for the app. This is actually very straight forward using the following snippet uploaded and executed on the webserver:

<?php
exec( 'mysqldump -h dbXXX.1und1.de -u dboYYY -pZZZ dbYYY > dump.sql', $ret );
var_dump($ret);

This is what you do:

  • Find the configuration with the db-credentials in the source of the application and replace
    • dbXXX.1und1.de with the hostname,
    • dboYYY with the username,
    • ZZZ with the password (notice there is NO blank between the option -p and the password!),
    • dbYYY with the database you would like to backup.
  • Save the snippet as file with extension .php and copy it via ftp to the webspace.
  • Find the URL to execute the script.
  • If it worked, there should be a new file dump.sql in the same directory, download it, it contains the db-schema and all data.
  • Delete both files to wipe your traces.

If it does not work try something like “exec(‘ls -la’);”. If you see the contents of the directory you can use the exec-function and the problem is something else.

Happy hacking!

« Older Entries