Category Archives: Tools

OCI Interface Integration into your eShop via Punch-Out from SAP/SRM

If it comes to integration of eShops like Magento, OXID or even your cusom made eShops into a corporate buying infrastructure using SAP, you need to deal with the SRM module (Supplier Relationship Management) using the OCI interface.

On the shop side we had a Gambio eShop in this case, which is a commercial fork of XT-Commerce. At first we checked if there where any affordable ready-to-use modules available. We found some for around 1500-2000EUR with additional (unclear) service cost. To be honest, what we found looked a bit outdated and after some contacts with the responsible sales people not even very trust worthy. After skipping over the documentation we decided to integrate OCI on our own – the custom hacked way. After all, it is just simple HTTP.

If you check the documentation and your first thought is, you could easily do it using cURL at the checkout, be warned – OCI does NOT work using a cURL call from a remote server! We did it this way and failed the first try as soon as we discovered how OCI mechanics really worked (see below). The documentation is a bit ambiguous in that respect.

The main problem is, that only your customers customer (!) with whom you have to co-operate closely and directly can help you through testing. People there are not that technical, even though their job title says so. They own the SRM testsystem (you don’t) and you always have to ask for a punch-out-click in order to find new insights integrating OCI into your eShop. It took us many weeks instead of a couple of minutes due to vacation and corporate politics and overhead. Finally we integrated an all-encompassing logging for each minor step and response to reduce the communication to ‘please click again’ – and bamm, 3 days later we had a bunch of logging information powering the next development step.

Here is a SVN diff showing all files we added or modified in the gambio release with a description what we modified:

Changed files in original Gambio distribution:
M  checkout_shipping.php (added optical info, we're in OCI-mode)
M  includes/configure.php (added a switch to work on live and development server)
A  includes/modules/oci/oci.php (Main functionality wrapped in a class)
A  includes/modules/oci/ociParameters.php (Configuration with additional parameters for SRM)
A  includes/modules/oci/logger.php (Own simple logger to see what happens during test calls)
A  oci_out.php (exit sending real hidden html form onLoad via JavaScript, redirecting to SRM)
M  admin/includes/configure.php (added a switch to work on live and development server)
A  oci.php (entry point for calling Punch-Out incl authentication)
M  checkout_confirmation.php (added optical info, we're in OCI-mode)
M  checkout_process.php (redirect to exit oci_out.php instead to successpage after checkout)
M  templates/EyeCandy/module/checkout_shipping.html (added optical info, we're in OCI-mode)
M  templates/EyeCandy/module/checkout_confirmation.html (added optical info, we're in OCI-mode)
M  templates/EyeCandy/module/checkout_success.html (added optical info, we're in OCI-mode)


OCI mechanics work like this:

At first the purchase department selects a supplier in their SRM. As soon as the order agent starts shopping, SRM performs a ‘Punch-Out’, which opens a browser window with a URL like this one:

http://your-php-eshop.com/oci.php?Z_EMAIL=ebp%40xxx%2ede&password=password&userid=8&BYPASS_INB_HANDLER=X&BYPASS_OUTB_HANDLER=X&OCI_VERSION=4%2e0&OPI_VERSION=1%2e0&returntarget=_top&HOOK_URL=https%3a%2f%2ftestsystem7%2esrm%2ede%2fsap%2fbc%2fnwbc%2fsrm%2f%3fsap-nwbc-action_url%3d%252fsap%252fbc%252fwebdynpro%252fsapsrm%252fwda_l_fpm_gaf%253bsap-ext-sid%253dKMDRyFbbH6pX0000mAWtxW–KMDRyVbbH6pX0000mAWtxW–%253fsap-ep-tstamp%253d20130419102708%2526sap-wd-tstamp%253d20130419102708%2526sap-ep-version%253d7%26NavigationTarget%3dportal_content%2fnwbc_back%26NavMode%3d3%26UsePost%3dTrue%26SAPSRM_RESUME_ID%3dSAPSRM_OCI%26sap-nwbc-has_post_params%3dX%26sap-client%3d010%26sap-language%3dDE%26sap-nwbc-node%3dresume_appl

This call goes to your /oci.php. It needs to do the following:

  • It checks if there is a customer/user in your eShop that can be authenticated with the userid, password and Z_EMAIL parameters. If so /oci.php logs the user in to the eShop. Of course you have created the customer maually before. Send the SRM owner the values of the parameters for their punch-out and test if auto-login works via punch-out.
  • /oci.php must remember the value of the parameter HOOK_URL in the session. It is only valid for this purchase and the contents of the shopping cart must be posted back to this URL – it is your value for your form’s action attribute. Remember: It can NOT be done via cURL.

The user is now navigating your eShop and fills his shopping cart and proceeds to the checkout and clicks the final checkout button. The purchase is saved in the shopping application by your eShop system. Instead of redirection to the shop-success page a hidden form is created (see OCI documentation for fieldnames and fixed field values necessary) with the shopping information for the SRM and it is sumbitted immedately by javascript. You need to intercept the redirection to the success page and redirect to your /oci_out.php instead providing this form. The action url has been handed over via the HOOK_URL parameter on punch-out.

The hidden form looks like this:

<!– OCI autosubmit –>
<script type=”text/javascript” language=”javascript”>
document.getElementById(“srm_form_submit”).click();
</script>

<form id=”srm_form” action=”https://testsystem.srm.customer.com/sap/bc/nwbc/srm/?sap-nwbc-action_url=%2fsap%2fbc%2fwebdynpro%2fsapsrm%2fwda_l_fpm_gaf%3bsap-ext-sid%3dKN2O27IP8fdX0000mAWtxW–KN2O2NIP8fdX0000mAWtxW–%3fsap-ep-tstamp%3d20130419104715%26sap-wd-tstamp%3d20130419104715%26sap-ep-version%3d7&NavigationTarget=portal_content/nwbc_back&NavMode=3&UsePost=True&SAPSRM_RESUME_ID=SAPSRM_OCI&sap-nwbc-has_post_params=X&sap-client=010&sap-language=DE&sap-nwbc-node=resume_appl” method=”POST”><code> <input name=”NEW_ITEM-LINE[1]” type=”hidden” value=”00001″ />
<input name=”NEW_ITEM-DESCRIPTION[1]” type=”hidden” value=”Herren Fleece-Jacke Langley Rot” />
<input name=”NEW_ITEM-LONGTEXT_1:132[]” type=”hidden” value=”Herren Fleece-Jacke Langley Rot” />
<input name=”NEW_ITEM-QUANTITY[1]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-UNIT[1]” type=”hidden” value=”C62″ />
<input name=”NEW_ITEM-PRICE[1]” type=”hidden” value=”41.87″ />
<input name=”NEW_ITEM-CURRENCY[1]” type=”hidden” value=”EUR” />
<input name=”NEW_ITEM-PRICEUNIT[1]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-LEADTIME[1]” type=”hidden” value=”21″ />
<input name=”NEW_ITEM-VENDOR[1]” type=”hidden” value=”16222″ />
<input name=”NEW_ITEM-VENDORMAT[1]” type=”hidden” value=”287{2}5{1}23{5}17″ />
<input name=”NEW_ITEM-MATGROUP[1]” type=”hidden” value=”41010000″ />
<input name=”NEW_ITEM-LINE[2]” type=”hidden” value=”00002″ />
<input name=”NEW_ITEM-DESCRIPTION[2]” type=”hidden” value=”Pique Polo Safran” />
<input name=”NEW_ITEM-LONGTEXT_2:132[]” type=”hidden” value=”Pique Polo Safran” />
<input name=”NEW_ITEM-QUANTITY[2]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-UNIT[2]” type=”hidden” value=”C62″ />
<input name=”NEW_ITEM-PRICE[2]” type=”hidden” value=”16.95″ />
<input name=”NEW_ITEM-CURRENCY[2]” type=”hidden” value=”EUR” />
<input name=”NEW_ITEM-PRICEUNIT[2]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-LEADTIME[2]” type=”hidden” value=”21″ />
<input name=”NEW_ITEM-VENDOR[2]” type=”hidden” value=”16222″ />
<input name=”NEW_ITEM-VENDORMAT[2]” type=”hidden” value=”112{2}4{1}1{5}16″ />
<input name=”NEW_ITEM-MATGROUP[2]” type=”hidden” value=”41010000″ />
<input id=”srm_form_submit” style=”display: none;” type=”submit” value=”Go” />
</code></form>

After this form has been submitted, SRM generates another page also including a hidden form, additional internal information and lots of JavaScript logic also with auto-submit. Please note: a) The array counter must start at 1 not at 0. And b) note the syntax of the longtext field.

The page is generated and ‘executed’ on the server side of the SRM system! This is the reason why it could not be done via cURL: The action to the HOOK_URL opens a page on the SRM server submitting another (!) form which finally writes the purchase to SRM – not your initial POST. It also closes the browser window for the order agent. If SRM can not close the browser window something went wrong.

Migrate Email via IMAP using imapsync

This time I had to migrate not just a domain, but a whole bunch of existing imap-email accounts with it.

To do that, just create all email accounts on the new machine and have a list with all credentials ready. For the hard work I used a very good Perl tool – imapsync.

Very cool with imapsync is, that it actually syncs from the old server. So you do not have to deal with double email or lost mail. See also imapsync description here.

Receipe for Debian Linux:

Download imapsync from https://fedorahosted.org/released/imapsync/:

$ wget https://fedorahosted.org/released/imapsync/imapsync-1.525.tgz

Unzip and cdir into the unzipped directory.

Check your perl installation with:

$ perl -c imapsync

If you see a “Can’t locate Mail/IMAPClient.pm in @INC…” you need to install the IMAPClient:

# cpan Mail::IMAPClient

If you see “imapsync syntax OK”, you already have the required modules.

Migrate one emailbox:

$ ./imapsync --host1 imap.1und1.de --host2 imap.man.ticore.it --port1 993 --port2 993 --ssl1 --ssl2 --user1 testuser@migration-domain.xx --user2 new_username_on_target_system --password1 xxx --password2 yyy

If you get a “Can’t locate IO/Socket/SSL.pm in @INC…” error, this might fix the problem:

# apt-get install libio-socket-ssl-perl

Retry migration:

$ ./imapsync --host1 imap.1und1.de --host2 imap.man.ticore.it --port1 993 --port2 993 --ssl1 --ssl2 --user1 testuser@migration-domain.xx --user2 new_username_on_target_system --password1 xxx --password2 yyy

If output says “Info: host imap.man.ticore.it says it has NO CAPABILITY for AUTHENTICATE LOGIN” just add ‘–authmech2 PLAIN’

Retry migration:

$ ./imapsync --host1 imap.1und1.de --host2 imap.man.ticore.it --port1 993 --port2 993 --ssl1 --ssl2 --user1 testuser@migration-domain.xx --user2 new_username_on_target_system --password1 xxx --password2 yyy --authmech2 PLAIN

Now you have test-migrated one email account. If you have more, which is likely, takle a look at imapsync-1.525/examples. You find 2 important files:

  • file.txt – contains a list of accounts with the self-explanatory line-format user001_1;password001_1;user001_2;password001_2
  • sync_loop_unix.sh – mass-migrates the accounts from the list. Pleas note: You must set the two imap-hostnames manually in this file! Check the directions in the file. Use via
    $ ./sync_loop_unix.sh

Have fun syncing your IMAP stuff around!

Emulate sendmail on your Dev Machine

In order to prevent email delivery during development and log all email messages that would have been delivered, you can actually do a simple trick: Replace the file /usr/sbin/sendmail (on Ubuntu, use ‘locate sendmail’ to find it if it lies elsewhere) with this little shell-script, or rather make a _bak of the original and save the following instead of the sendmail binary:

#!/bin/bash

LOGDIR="/tmp"
PREFIX="sendmail"
NOW=$(date +%Y-%m-%dT%H.%M.%S)
CNT=1
PRIVATELOG="$LOGDIR/$PREFIX-$NOW.$CNT.log"
COMBINEDLOG="$LOGDIR/$PREFIX-combined.log"

# If privatelogs are being used...
if [ ! -z "$PRIVATELOG" ]; then
# ...make sure the filename is unique and create the file
while [ -f $PRIVATELOG ]; do
CNT=$(($CNT + 1))
PRIVATELOG="$LOGDIR/$PREFIX-$NOW.$CNT.log"
done

echo "$0 $*" > $PRIVATELOG
else
# ...otherwise swap filenames
PRIVATELOG=$COMBINEDLOG
COMBINEDLOG=''
fi

echo "[$NOW]" >> $PRIVATELOG
while read BUF
do
echo $BUF >> $PRIVATELOG
done

# Append privatelog to combinedlog when both logs are used
if [ ! -z "$COMBINEDLOG" ]; then
echo "[$NOW]" >> $COMBINEDLOG
cat $PRIVATELOG >> $COMBINEDLOG
fi

exit 0

When your application now sends mail, these things happen:

  • No email is actually sent.
  • The message gets appended to the file /tmp/sendmail-combined.log, on which you could set a ‘tail -f’ in order to see which emails would have been sent and what contet they would have.
  • One new file (e.g. /tmp/sendmail-2011-02-08T08.02.48.1.log) gets written for every email sent. I personally only use the combined file.

Inspired by http://stackoverflow.com/questions/3710864/simulating-sendmail-with-dummy-script

Drawing Lines Using Google Static Maps API

If you are looking for a simple way to display lines like a roundtrip on a map, Google’s Static Maps API might be for you.

Documentation and examples: http://code.google.com/intl/en-US/apis/maps/documentation/staticmaps/

Limits:
- There is a query limit of 1000 unique (different) image requests per viewer per day.
- URLs are restricted to 2048 characters.


Here are some self-explanatory examples:

Using geo-coordinates:
http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|49.584486,8.462219|49.441343,8.462219|49.384161,8.654230&size=400x400&sensor=false

Using city names:
http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|Stuttgart|Mannheim|Bad Homburg|Frankfurt|Würzburg|Stuttgart&size=500x200&sensor=false

This is how you would use it on your website just use it like a local image:
<img src="http://maps.google.com/maps/api/staticmap?path=color:0x0000ff|weight:5|Stuttgart|Mannheim|Bad Homburg|Frankfurt|Würzburg|Stuttgart&size=500x200&sensor=false" alt="my roundtrip" />
my roundtrip

Pretty slick!

In case you need to extract geocoordinates, there are lots of online browser tools available, just click on a spot and copy-paste the coordinates into your static maps-URL:

Clean your PHP4 Legacies using sed

If you have to deal with very old PHP4 legacy code containing every syntax crime you may know from the early years, how would you handle it? Give it to your junior people to fix it manually? I like to have at least some handy helpers for the first rough corrections. I found sed to be a very powerful helper here.

Code you might encounter - associative array elements without quotes.

I spent quite some time to find useful regular expressions to help me. This is how I did this:

A way to test your regular-expression is to echo a sample and apply your regex to test the results:

echo '$_REQUEST[action] reise_l_topic_ids[] $dat[ticket_order]' | sed "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g"

Once it works you can apply your regex to one file:

sed -i "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g" my_old_file.php

Or apply your regex to all *.php-files recursively below the current directory to a whole project:

find . -name "*.php" -exec sed -i "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g" '{}' ;

By the way: The usage of sed works fine on your linux command line, but not on OSX. The syntax is slightly different here (sed -i “” -e “s/blah/blubb/” file). This is of course only a start to automate otherwise painfull and boring corrections down to just a few seconds. It will not save you from special manual work and break syntax at some points. But it weeds out 90% and leaves you with the other 10% acutal manual work.

You could imagine many more sed regexes e.g. to replace short tags <?=$my_var?> to a proper <?php echo $my_var; ?> etc.

echo '<?=$out?> sakdhs sakdhas k <?php echo $xyz; ?> ddd' | sed "s/<?=/<?php echo /g";

I will collect more regexes as I need and find them. If you have ideas please add them in the comment section.

MODx Revolution Links

I have recently been playing with the new MODx (‘Revolution’) and I must say, I love the concept and it’s flexibility. Just to make things easier for folks starting out with MODx or who migrate from MODx Evolution here are some recommended links.

General:

Focus topics:

  • Tag format changes from Evolution to Revolution: The formats for content tags have changed. They now can be nested, they can carry parameters and look a lot cleaner.
  • Use the Package Manager to install your snippets: There are still some packages to migrate (e.g. AjaxSearch). And if you install Login and wonder why you can not logout, just check the naming of the parameters. I am sure these issues will soon be solved.
  • Internationalization: The new Lexicon facility makes the creation of multilingual pages easier, especially central shared Pages or Chunks used in all available languages like the login, password reminder, 404 etc. You can even use Placeholders in lexicon entries and fill them dynamically with Parameters. On bigger sites I would prefer to have a home page for each language and maintain each language in separate menu trees. This is explained in this forum entry. Notice, the tutorial is based on Evolution, but works the same way in Revolution. You only need to use the new tag syntax.
  • New access to MODx objects: Most of the methods from Evolution to interact with $modx will be deprecated soon in Revolution. If you are writing Snippets or Plugins you need to get familiar with $modx->getObject() and Co.
  • New permission system: Instead of the distinction of Web Users and Manager Users in Evolution, Revolution switched to ACL (Acesss Control Lists) and Context based permissions.
  • Make Resources external to make use of version control: Required resources can also be loaded from files instead of saving everything to the MODx database.
  • Friendly URLs: How to use the enclosed .htaccess and what MODx settings to use.

What I dont like with MODx Revolution is the slow Manager interface based on ExtJS. If you have to go through a longer configuration or exploration session, waiting for the frontend to finish loading can be a drag. But the ajaxy Manager helps you to perform some actions in parallel without you losing your form content, e.g. if you need a href and need to lookup the document-ID you just click on the Resource tab and look it up in the tree.

Overall I must say MODx Revolution makes the expression to me like a very well crafted, cutting-edge and thought through CMS. Congratulations and many thanks to all contributors! You are awesome!

Quick PHP Hack to Tidy up Trashed HTML

I just hacked together the following quick and dirty PHP-script to use the tidy-extension from the command line. Maybe somebody else needs something like this somewhere. Check the comment for details:

<?php

//
// (a) Save this as tidy.php
// (b) Call it from commandline like this: $ php tidy.php trashed.html > tidy.html
//     to tidy the file trashed.html to a new file tidy.html.
//

// Installation and configuration see http://php.net/tidy
$config = array('indent'=> true, 'output-xhtml' => true, 'wrap'=> 200);

$tidy = new tidy;
$tidy->parseString(file_get_contents($_SERVER['argv'][1]), $config, 'utf8');
$tidy->cleanRepair();

echo $tidy;

Install PEAR on OSX

Here is how you install PEAR on Mac OSX:

sudo mkdir /usr/local/temp;
sudo chmod -R 777 /usr/local/temp;
sudo mkdir /usr/local/share/pear;
sudo chmod -R 777 /usr/local/share/pear;
curl http://pear.php.net/go-pear | sudo php;

Set /usr/local as path prefix and install.

You should then be able to use pear, e.g.: pear help.

In order to install modules you also need a temp dir:

sudo mkdir /usr/local/temp;
sudo chmod 0777 /usr/local/temp/;

You can now install PHPUnit:

sudo pear channel-discover pear.phpunit.de;
sudo pear install phpunit/PHPUnit;

Have Fun!

Consistent Development Environments using VirtualMachines

As a development team we always run into situations where we have trouble setting up a proper development environment for each of the team members to get going or add new staff on the go. It annoyed me every time since it causes a lot of unnecessary communication and friction.

I often heard of virtualization but never actually played seriously with it. The idea is:

If we could have a virtual machine for every project that contains an equivalent environment like the production system, everybody working on it…

  • … could just rely on his development environment by just starting the VM without having to set up anything half-baked themselves.
  • … could use his favourite working environment OS, IDE and tools on which they are most comfortable and thus happy and productive.
  • … could work on their own checked out working copy using version control.
  • … could immedately see what they built refreshing the local browser or starting Unittests on the VM via ssh to check their dev increments.

We used http://www.virtualbox.org. A good starting point to get to know VirtualBox better and learn how to start your first virtual machine: https://help.ubuntu.com/community/VirtualBox

Our target was to be able to startup the development VM as guest system on any developers development machine being the host system, open a browser on the host (!) and call for example http://develop/ to see the webroot of the VM. Additionally we set up samba and ssh on the VM in order to have the webserver’s webroot on the VM available via the filesystem. In order to do that you need to…

  • …start your VM with networking set to ‘Host interface’ instead of the default NAT. This is explained in detail on this page (sorry German) http://www.nwlab.net/tutorials/virtualbox/virtual-networking.html – for me it was tricky to get the guest machine available on the host and have internet access at the same time.
  • …edit the hosts file (on OSX ‘sudo nano /private/etc/hosts’ and reboot) on the development machines and add something like the following line: ’192.168.56.101 develop’. To find out the IP enter ‘sudo ifconfig’ (OSX/Linux) on your host system after you have started the VM. You will see aditional adapters set by virtualbox and the IP address.
  • …configure /etc/samba/smb.conf on the VM, restart samba and connect (e.g. smb://develop/webroot). We check out a working copy of the applicaton under development directly onto the webroot and create a new PHP-Project there in our IDE. Update and commit directly from there.

If you search the web for ready-made VMs you find mostly VMware images. You can not run them directly in VirtualBox and need to convert them.

Under Linux you can convert the VMware image (.vmdk) into a VirtualBox image (.vdi) like this:

sudo apt-get install virtualbox-ose virtualbox-ose-guest-utils;
sudo apt-get install qemu;

qemu-img convert xxx.vmdk xxx.bin;
VBoxManage convertdd xxx.bin xxx.vdi

Install the required packages with apt-get install once. VBoxManage is part of VirtualBox. The last two lines do the conversion.


We ended up creating a fresh install from a Debian 5 netinstall iso. The iso-file can be mounted as CDROM on the creation of the new VM with VirtualBox. Receipes for setting up the appropriate LAMP environment with apt-get install can be found on the web. You only have to do it once. Save the state of your VM afterwards.


There are ways to generate a virtual machine from a physical server. Use Google to find receipes. I used http://www.partimage.org on a Debian Etch system with the live CDROM from http://www.sysresccd.org. This requires that you are able to umount your filesystem or rather boot into the live cd on the production/staging machine in order to generate the partimage.

You mount an external drive over the network or a usb harddrive. The partition (e.g. /dev/sda1) you would like to backup must be umounted. From the live cd you can see your partitions, including attached usb drives, with the ‘fdisk -l‘ command. Just mount the target (e.g. /mnt/usbdrive) and start partimage from the commandline. Dialogues guide you through the image creation.


In case you wonder what is meant by the ‘Host key’ to enter or leave a running VM with your mouse… it is the right (!) Strg-Button on your Keyboard.


I just installed Ubuntu Server on a VM from my MacBook. To have a usable keyboard once you logged on to the new VM, you must do the following in order to have a keymap including the pipe symbol, braces etc.:

  • sudo apt-get install console-data; #to install the keymaps
  • sudo loadkeys mac-macbook-de; #to set the keymap for German MacBook

Once you have done that, you can use your right Command-Key as ‘Alt-Gr-Key’ like on a PC keyboard. The pipe symbol is then typable with ‘Alt-Gr + >’, Braces and Brackets are typable via ‘Alt-Gr + 6,7,8,9′.


This is how you copy a virtual machine using VirtualBox tools:

$ VBoxManage clonevdi /Users/marco/MyMachine.vdi /Users/marco/MyMachine_copy.vdi
$ VBoxManage internalcommands setvdiuuid /Users/marco/MyMachine_copy.vdi