Tag Archives: php

Eternal Image Cache Based on Demandware eShop via Simple API for External Image Usage

Here is another pragmatic web-technology based solution I would like to share with you. We had a given Demandware eShop with 35k variants at the time (in 2014). Where a variant represents a ‘product’ in a unique color and ‘articles’ being the children of the variants as real orderable items – a variant in a certain size. A big eShopping site.

 

Call of an image URL in the browser.

Call of an image URL in the browser.

Our goal was to have any image of articles in the eShop under a very simple URL that can be generated only by knowing a ‘number’ (SKU or EAN if available) leading to an existing article like this: http://imagex.xxxxx.de/<sku>. The response would be a html image header and the streamed content of the image itself or if not available a placeholder image.

Fortunately Demandware offers a comprehensive, well documented REST based API (we used  the Open Commerce API 13.6) to interact with the eShop returning results in pure JSON.

The initial use case was to add pictures of the articles to help logistics people picking the purchased items from the warehouse. They use mobile devices with a screen and item lists on it. Another use case was to add article images on various internal reports based on the business intelligence system informing management of what is currently being sold.

Article images used in a report.

Article images used in a report.

The workflow of the application works like this:

  1. The image is stored in an ‘eternal cache’ in a directory path that is generated from the ‘number’ – e.g. 4054781891957 converts to this path data/400/405/478/189/195/7/4054781891957.jpg (400 being the width of the cached image version).
    This way we did not have to use a database. We use just use is_file($path) to find out if we have a valid cache hit.
  2. If a cached file does not yet exist (first call on a ‘number’), depending on the type of the given ‘number’ we check a bunch things via the API in the background. This is like a regular search on the eShop, fetch the URL of the article’s main image, download it and save it to the local cache converted to 400px, 200px and 100px width.

Drawback: OCAPI only offers information about articles that are ‘online’ at the time of the API call.

Optional parameters:
Example-URL: http://imagex.xxxxx.de/30281349?debug&nocache&w=400

  • debug: Outputs the differnets actions along the way to the image.
  • nocache: Forces re-fetch from Demandware OCAPI.
  • w: Default width is 200px. Possible Values 400, 200, 100

The application is hosted in our own infrastructure for the customer. The application is based on Symfony2.

wsdl with remote methods

Update: The application now has also a SOAP/WSDL API to a) integrate images into other enterprise applications via base64-stream and b) force cache refresh of an image (happens on recycled article numbers).

OCI Interface Integration into your eShop via Punch-Out from SAP/SRM

If it comes to integration of eShops like Magento, OXID or even your cusom made eShops into a corporate buying infrastructure using SAP, you need to deal with the SRM module (Supplier Relationship Management) using the OCI interface.

On the shop side we had a Gambio eShop in this case, which is a commercial fork of XT-Commerce. At first we checked if there where any affordable ready-to-use modules available. We found some for around 1500-2000EUR with additional (unclear) service cost. To be honest, what we found looked a bit outdated and after some contacts with the responsible sales people not even very trust worthy. After skipping over the documentation we decided to integrate OCI on our own – the custom hacked way. After all, it is just simple HTTP.

If you check the documentation and your first thought is, you could easily do it using cURL at the checkout, be warned – OCI does NOT work using a cURL call from a remote server! We did it this way and failed the first try as soon as we discovered how OCI mechanics really worked (see below). The documentation is a bit ambiguous in that respect.

The main problem is, that only your customers customer (!) with whom you have to co-operate closely and directly can help you through testing. People there are not that technical, even though their job title says so. They own the SRM testsystem (you don’t) and you always have to ask for a punch-out-click in order to find new insights integrating OCI into your eShop. It took us many weeks instead of a couple of minutes due to vacation and corporate politics and overhead. Finally we integrated an all-encompassing logging for each minor step and response to reduce the communication to ‘please click again’ – and bamm, 3 days later we had a bunch of logging information powering the next development step.

Here is a SVN diff showing all files we added or modified in the gambio release with a description what we modified:

Changed files in original Gambio distribution:
M  checkout_shipping.php (added optical info, we're in OCI-mode)
M  includes/configure.php (added a switch to work on live and development server)
A  includes/modules/oci/oci.php (Main functionality wrapped in a class)
A  includes/modules/oci/ociParameters.php (Configuration with additional parameters for SRM)
A  includes/modules/oci/logger.php (Own simple logger to see what happens during test calls)
A  oci_out.php (exit sending real hidden html form onLoad via JavaScript, redirecting to SRM)
M  admin/includes/configure.php (added a switch to work on live and development server)
A  oci.php (entry point for calling Punch-Out incl authentication)
M  checkout_confirmation.php (added optical info, we're in OCI-mode)
M  checkout_process.php (redirect to exit oci_out.php instead to successpage after checkout)
M  templates/EyeCandy/module/checkout_shipping.html (added optical info, we're in OCI-mode)
M  templates/EyeCandy/module/checkout_confirmation.html (added optical info, we're in OCI-mode)
M  templates/EyeCandy/module/checkout_success.html (added optical info, we're in OCI-mode)


OCI mechanics work like this:

At first the purchase department selects a supplier in their SRM. As soon as the order agent starts shopping, SRM performs a ‘Punch-Out’, which opens a browser window with a URL like this one:

http://your-php-eshop.com/oci.php?Z_EMAIL=ebp%40xxx%2ede&password=password&userid=8&BYPASS_INB_HANDLER=X&BYPASS_OUTB_HANDLER=X&OCI_VERSION=4%2e0&OPI_VERSION=1%2e0&returntarget=_top&HOOK_URL=https%3a%2f%2ftestsystem7%2esrm%2ede%2fsap%2fbc%2fnwbc%2fsrm%2f%3fsap-nwbc-action_url%3d%252fsap%252fbc%252fwebdynpro%252fsapsrm%252fwda_l_fpm_gaf%253bsap-ext-sid%253dKMDRyFbbH6pX0000mAWtxW–KMDRyVbbH6pX0000mAWtxW–%253fsap-ep-tstamp%253d20130419102708%2526sap-wd-tstamp%253d20130419102708%2526sap-ep-version%253d7%26NavigationTarget%3dportal_content%2fnwbc_back%26NavMode%3d3%26UsePost%3dTrue%26SAPSRM_RESUME_ID%3dSAPSRM_OCI%26sap-nwbc-has_post_params%3dX%26sap-client%3d010%26sap-language%3dDE%26sap-nwbc-node%3dresume_appl

This call goes to your /oci.php. It needs to do the following:

  • It checks if there is a customer/user in your eShop that can be authenticated with the userid, password and Z_EMAIL parameters. If so /oci.php logs the user in to the eShop. Of course you have created the customer maually before. Send the SRM owner the values of the parameters for their punch-out and test if auto-login works via punch-out.
  • /oci.php must remember the value of the parameter HOOK_URL in the session. It is only valid for this purchase and the contents of the shopping cart must be posted back to this URL – it is your value for your form’s action attribute. Remember: It can NOT be done via cURL.

The user is now navigating your eShop and fills his shopping cart and proceeds to the checkout and clicks the final checkout button. The purchase is saved in the shopping application by your eShop system. Instead of redirection to the shop-success page a hidden form is created (see OCI documentation for fieldnames and fixed field values necessary) with the shopping information for the SRM and it is sumbitted immedately by javascript. You need to intercept the redirection to the success page and redirect to your /oci_out.php instead providing this form. The action url has been handed over via the HOOK_URL parameter on punch-out.

The hidden form looks like this:

<!– OCI autosubmit –>
<script type=”text/javascript” language=”javascript”>
document.getElementById(“srm_form_submit”).click();
</script>

<form id=”srm_form” action=”https://testsystem.srm.customer.com/sap/bc/nwbc/srm/?sap-nwbc-action_url=%2fsap%2fbc%2fwebdynpro%2fsapsrm%2fwda_l_fpm_gaf%3bsap-ext-sid%3dKN2O27IP8fdX0000mAWtxW–KN2O2NIP8fdX0000mAWtxW–%3fsap-ep-tstamp%3d20130419104715%26sap-wd-tstamp%3d20130419104715%26sap-ep-version%3d7&NavigationTarget=portal_content/nwbc_back&NavMode=3&UsePost=True&SAPSRM_RESUME_ID=SAPSRM_OCI&sap-nwbc-has_post_params=X&sap-client=010&sap-language=DE&sap-nwbc-node=resume_appl” method=”POST”><code> <input name=”NEW_ITEM-LINE[1]” type=”hidden” value=”00001″ />
<input name=”NEW_ITEM-DESCRIPTION[1]” type=”hidden” value=”Herren Fleece-Jacke Langley Rot” />
<input name=”NEW_ITEM-LONGTEXT_1:132[]” type=”hidden” value=”Herren Fleece-Jacke Langley Rot” />
<input name=”NEW_ITEM-QUANTITY[1]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-UNIT[1]” type=”hidden” value=”C62″ />
<input name=”NEW_ITEM-PRICE[1]” type=”hidden” value=”41.87″ />
<input name=”NEW_ITEM-CURRENCY[1]” type=”hidden” value=”EUR” />
<input name=”NEW_ITEM-PRICEUNIT[1]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-LEADTIME[1]” type=”hidden” value=”21″ />
<input name=”NEW_ITEM-VENDOR[1]” type=”hidden” value=”16222″ />
<input name=”NEW_ITEM-VENDORMAT[1]” type=”hidden” value=”287{2}5{1}23{5}17″ />
<input name=”NEW_ITEM-MATGROUP[1]” type=”hidden” value=”41010000″ />
<input name=”NEW_ITEM-LINE[2]” type=”hidden” value=”00002″ />
<input name=”NEW_ITEM-DESCRIPTION[2]” type=”hidden” value=”Pique Polo Safran” />
<input name=”NEW_ITEM-LONGTEXT_2:132[]” type=”hidden” value=”Pique Polo Safran” />
<input name=”NEW_ITEM-QUANTITY[2]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-UNIT[2]” type=”hidden” value=”C62″ />
<input name=”NEW_ITEM-PRICE[2]” type=”hidden” value=”16.95″ />
<input name=”NEW_ITEM-CURRENCY[2]” type=”hidden” value=”EUR” />
<input name=”NEW_ITEM-PRICEUNIT[2]” type=”hidden” value=”1″ />
<input name=”NEW_ITEM-LEADTIME[2]” type=”hidden” value=”21″ />
<input name=”NEW_ITEM-VENDOR[2]” type=”hidden” value=”16222″ />
<input name=”NEW_ITEM-VENDORMAT[2]” type=”hidden” value=”112{2}4{1}1{5}16″ />
<input name=”NEW_ITEM-MATGROUP[2]” type=”hidden” value=”41010000″ />
<input id=”srm_form_submit” style=”display: none;” type=”submit” value=”Go” />
</code></form>

After this form has been submitted, SRM generates another page also including a hidden form, additional internal information and lots of JavaScript logic also with auto-submit. Please note: a) The array counter must start at 1 not at 0. And b) note the syntax of the longtext field.

The page is generated and ‘executed’ on the server side of the SRM system! This is the reason why it could not be done via cURL: The action to the HOOK_URL opens a page on the SRM server submitting another (!) form which finally writes the purchase to SRM – not your initial POST. It also closes the browser window for the order agent. If SRM can not close the browser window something went wrong.

Migrate arbitrary Content to WordPress – e.g. from typo3 tt_content

If you see yourself confronted with the following task, here is a quick receipe: Setup a wordpress instance for us and import all our content from system x. In this case we focus on the import part. ‘System x’ was a somewhat customized typo3 setup. I got myself an ordinatry editors login to identify the articles which had to be migrated. After some fulltext searching and access to phpMyAdmin I identified the area, where I hat to extract the content from. It boiled down to the table tt_content. At this point you can forget about typo3. All articles could be extracted with this SQL query:

SELECT uid, pid, header, bodytext FROM tt_content WHERE header<>'' AND CType='editorialcontent';

Export the content using phpMyAdmin’s export to PHP array functionality. You end up with a file like shown in the screenshot. If you have HTML-content and doubt, if this could be valid php, save the file under export.php and do ‘$ php -l export.php’ – it says ‘No syntax errors.’ – Good.

OK, next you need to get familiar with wp_insert_post(). You need to loop your exported content over it. And that is it. This is how I did it:

  • Create a directory in your wordpress setup ‘wp-import’.
  • Copy/move your export.php array into this directory.
  • And execute the following self-explanatory file located also in your wp-import directory: import.php

That’s it.

Emulate sendmail on your Dev Machine

In order to prevent email delivery during development and log all email messages that would have been delivered, you can actually do a simple trick: Replace the file /usr/sbin/sendmail (on Ubuntu, use ‘locate sendmail’ to find it if it lies elsewhere) with this little shell-script, or rather make a _bak of the original and save the following instead of the sendmail binary:

#!/bin/bash

LOGDIR="/tmp"
PREFIX="sendmail"
NOW=$(date +%Y-%m-%dT%H.%M.%S)
CNT=1
PRIVATELOG="$LOGDIR/$PREFIX-$NOW.$CNT.log"
COMBINEDLOG="$LOGDIR/$PREFIX-combined.log"

# If privatelogs are being used...
if [ ! -z "$PRIVATELOG" ]; then
# ...make sure the filename is unique and create the file
while [ -f $PRIVATELOG ]; do
CNT=$(($CNT + 1))
PRIVATELOG="$LOGDIR/$PREFIX-$NOW.$CNT.log"
done

echo "$0 $*" > $PRIVATELOG
else
# ...otherwise swap filenames
PRIVATELOG=$COMBINEDLOG
COMBINEDLOG=''
fi

echo "[$NOW]" >> $PRIVATELOG
while read BUF
do
echo $BUF >> $PRIVATELOG
done

# Append privatelog to combinedlog when both logs are used
if [ ! -z "$COMBINEDLOG" ]; then
echo "[$NOW]" >> $COMBINEDLOG
cat $PRIVATELOG >> $COMBINEDLOG
fi

exit 0

When your application now sends mail, these things happen:

  • No email is actually sent.
  • The message gets appended to the file /tmp/sendmail-combined.log, on which you could set a ‘tail -f’ in order to see which emails would have been sent and what contet they would have.
  • One new file (e.g. /tmp/sendmail-2011-02-08T08.02.48.1.log) gets written for every email sent. I personally only use the combined file.

Inspired by http://stackoverflow.com/questions/3710864/simulating-sendmail-with-dummy-script

Clean your PHP4 Legacies using sed

If you have to deal with very old PHP4 legacy code containing every syntax crime you may know from the early years, how would you handle it? Give it to your junior people to fix it manually? I like to have at least some handy helpers for the first rough corrections. I found sed to be a very powerful helper here.

Code you might encounter - associative array elements without quotes.

I spent quite some time to find useful regular expressions to help me. This is how I did this:

A way to test your regular-expression is to echo a sample and apply your regex to test the results:

echo '$_REQUEST[action] reise_l_topic_ids[] $dat[ticket_order]' | sed "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g"

Once it works you can apply your regex to one file:

sed -i "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g" my_old_file.php

Or apply your regex to all *.php-files recursively below the current directory to a whole project:

find . -name "*.php" -exec sed -i "s/$([a-zA-Z0-9_]+)[([a-zA-Z0-9_]+)]/$1['2']/g" '{}' ;

By the way: The usage of sed works fine on your linux command line, but not on OSX. The syntax is slightly different here (sed -i “” -e “s/blah/blubb/” file). This is of course only a start to automate otherwise painfull and boring corrections down to just a few seconds. It will not save you from special manual work and break syntax at some points. But it weeds out 90% and leaves you with the other 10% acutal manual work.

You could imagine many more sed regexes e.g. to replace short tags <?=$my_var?> to a proper <?php echo $my_var; ?> etc.

echo '<?=$out?> sakdhs sakdhas k <?php echo $xyz; ?> ddd' | sed "s/<?=/<?php echo /g";

I will collect more regexes as I need and find them. If you have ideas please add them in the comment section.

Backup MySql-Database using only FTP

We have just had the case of having ftp access to a site on a shared hosting LAMP webspace but needed also the database and mainly the database scheme to setup our own development-system for the app. This is actually very straight forward using the following snippet uploaded and executed on the webserver:

<?php
exec( 'mysqldump -h dbXXX.1und1.de -u dboYYY -pZZZ dbYYY > dump.sql', $ret );
var_dump($ret);

This is what you do:

  • Find the configuration with the db-credentials in the source of the application and replace
    • dbXXX.1und1.de with the hostname,
    • dboYYY with the username,
    • ZZZ with the password (notice there is NO blank between the option -p and the password!),
    • dbYYY with the database you would like to backup.
  • Save the snippet as file with extension .php and copy it via ftp to the webspace.
  • Find the URL to execute the script.
  • If it worked, there should be a new file dump.sql in the same directory, download it, it contains the db-schema and all data.
  • Delete both files to wipe your traces.

If it does not work try something like “exec(‘ls -la’);”. If you see the contents of the directory you can use the exec-function and the problem is something else.

Happy hacking!

Quick PHP Hack to Tidy up Trashed HTML

I just hacked together the following quick and dirty PHP-script to use the tidy-extension from the command line. Maybe somebody else needs something like this somewhere. Check the comment for details:

<?php

//
// (a) Save this as tidy.php
// (b) Call it from commandline like this: $ php tidy.php trashed.html > tidy.html
//     to tidy the file trashed.html to a new file tidy.html.
//

// Installation and configuration see http://php.net/tidy
$config = array('indent'=> true, 'output-xhtml' => true, 'wrap'=> 200);

$tidy = new tidy;
$tidy->parseString(file_get_contents($_SERVER['argv'][1]), $config, 'utf8');
$tidy->cleanRepair();

echo $tidy;

Install PEAR on OSX

Here is how you install PEAR on Mac OSX:

sudo mkdir /usr/local/temp;
sudo chmod -R 777 /usr/local/temp;
sudo mkdir /usr/local/share/pear;
sudo chmod -R 777 /usr/local/share/pear;
curl http://pear.php.net/go-pear | sudo php;

Set /usr/local as path prefix and install.

You should then be able to use pear, e.g.: pear help.

In order to install modules you also need a temp dir:

sudo mkdir /usr/local/temp;
sudo chmod 0777 /usr/local/temp/;

You can now install PHPUnit:

sudo pear channel-discover pear.phpunit.de;
sudo pear install phpunit/PHPUnit;

Have Fun!

Upgrade PHP5 with an alternative sources.list on Debian etch

I was having trouble with a server running Debian 4.0 (etch). Using the standard sources in the /etc/apt/sources.list the supported PHP5 version was 5.2.0-8+etch13 which contained a very annoying bug for my application.

A daily running script – let’s call it the Importer – regularly exited randomly with a “Fatal error: Out of memory (allocated 12320768) (tried to allocate 2851436 bytes) in …” and I had to restart it manually nearly every morning. I had…

  • …checked my application for memory wasting operations and loops and fixed them,
  • …used ini_set(‘memory_limit’, ’64M’); at runtime, and
  • …finally increased memory_limit = 64M in my php.ini.

But all this did not change the bahaviour of the Importer!

So I took a look at the PHP5 Changelog to find potentially fixed bugs in newer releases. Bug #39438 described exactly my problem. So a simple upgrade would help me. But it did not work with ‘apt-get upgrade’ or ‘apt-get install php5=5.2.8′ since the highest version in the apt source I used was the one that I already had: 5.2.0-8+etch13, issued in November 2006… (pretty ancient)

Finally it was this page that had the information we needed: an alternative apt source

deb http://packages.dotdeb.org etch all
deb-src http://packages.dotdeb.org etch all

After getting an impression whether dotdeb was a trustworthy source, we first tried it on our dev-system with ‘apt-get update; apt-get upgrade;’. At this point I was once more glad to have written so many UnitTests. They all passed and everything looked good.

Thanks Kim for your help!

Use of MySql Variables

Case: I was building a CLI script to aggregate data from one MySql database and write results to another MySql database using 2 open db-connections. This way the script’s execution time was in the some-minutes-range… not good! I tried a more effective attempt: I let the CLI script output all necessary statements to drop and create all needed tables and fill them with insert-statements. I called it like this from the command line and diverted output to a file:

$ php aggregatData.php > all_sql_i_need.sql

I could then import the whole thing in one go into my target database like this:

$ mysql the_dbname -h localhost -uMarco -pPass < all_sql_i_need.sql

Insight: The execution-time was way better than my first attempt and took only seconds. I could also have combined the two lines in a shell-script.

The problem with the generated SQL statements was, that the data also contained relations between primary and foreign keys of records and that these IDs were generated at import-time by auto_increment fields. I solved this issue by using MySql variables which can be set and use d again like this:

INSERT INTO gruppen (name) VALUES ('Gruppe 1');
SET @temp_gruppe_id = LAST_INSERT_ID();
INSERT INTO gruppe_eigenschaften (id_gruppe, name, wert) VALUES
     (@temp_gruppe_id, 'email1', 'email1@emailxxx.com');

The variable @temp_gruppe_id is set and can be used in the subsequent SQL statements. It also worked in MySql4.

Here is an example of SQL statements illustrating this mechanism in the above mentioned script. You can cut-and-paste it into your PhpMyAdmin.