# app.psgi
use Plack::App::JSP;
Plack::App::JSP->new( js => q{
[ 200, [ 'Content-type', 'text/html' ], [ 'Hello, World!' ] ]
});

# displays: Hello, World!

Given the similarities between Perl and JSON you can’t actually tell if I’m cheating or not from the above snippet.

Let’s try something more convincing:

Plack::App::JSP->new( js => q{
function respond(body) {
    return [ 200, [ 'Content-type', 'text/html' ], [ body ] ]
}

respond("Five factorial is " +
    (function(x) {
      if ( x<2 ) return 1;
      return x * arguments.callee(x - 1);
    })(5)
);
});

# displays: Five factorial is 120

I just pushed Plack::App::JSP to the CPAN.
Thanks to Salvador Ortiz and Miguel Ibarra whose recently released JSP module makes this possible.

Recently Padre::Plugin::Plack, my Plack (the Perl Web Server) plugin for Padre (the Perl IDE) was translated into Dutch, Brazilian Portuguese and French.

I always get such a buzz when I find out that something I’ve worked on has been translated into another language. It sends a clear message that someone found your work useful enough to go to the effort of translating it so that a new community of people could use it. That’s an awesome form of positive feedback, and a great motivator for being involved in open source software.

Thanks translators!

Plack has a really neat solution to this problem.

Following a tip from day #5 of Miyagawa’s Plack advent calendar, serving up the current directory over http is as simple as this Perl one-liner:

$ plackup -MPlack::App::Directory -e 'Plack::App::Directory->new

Plack::Server::Standalone: Accepting connections at http://0:5000/

Or if you need the files served over port 80:

$ sudo plackup -MPlack::App::Directory -e 'Plack::App::Directory->new' -p 80

Plack::Server::Standalone: Accepting connections at http://0:80/

At about 11pm on the night before I headed off for the States, Ted the Vet and I sat down to talk Tarpo.

In case you’re wondering, Tarpo is an open source, cross platform Data Management desktop application for Dog Health programs in rural and remote Indigenous Australian communities, written entirely in Javascript.

The last time I blogged about Tarpo was way back in April 2008, and since then I haven’t really touched the code, but in the meantime Ted and his team of helpers have used it to record several thousand House Visits, Medical and Surgical Cases as part of the Maningrida Dog Health program.

Several other Veterinarians have started expressing interest in using Tarpo for their own Dog Health programs, and needless to say, we’ve accumulated a long list of bugs and feature requests over the past 18 months. So this afternoon, with a bit of spare time up my sleeve, I decided to get the ball rolling again.

Tarpo now has a proper project page, which you can visit at: http://pdonelan.github.com/tarpo

As you can see from the url, the code now lives on GitHub. Apart from the front page which has the all-important “Install Tarpo” button, the most important page on the GitHub project page is the Issue Tracker, which I’m hoping Ted and other Vets will use to report all Bugs and Feature requests. With a bit of luck, that will also make it easier for other developers to get involved too.

With these things in place, I started attacking the bit rot. Firstly I had to install Aptana and the Adobe Air plugin, which took several hours and reminded me how much I prefer doing development in Padre! On a side note, I spent a few hours trying to think if there was some way I could rewrite the entire app in Perl (I like Javascript in a twisted sort of way, but I’d rather ditch Adobe Air since it’s a proprietary framework) – but in the end I decided to stuck with the existing architecture, at least for now.

I though about doing it as a wxPerl app (like Padre), but I’d hate to lose the HTML widget set that ExtJS gives me.. hmm that gets me thinking (help me out if you’re reading Gábor or any of the other Padre gurus..) does wxPerl have a modern embedded browser that I could use? Maybe webkit-based one..? Then the backend could be rewritten as a standalone Plack-powered webapp and I’d have all of CPAN at my disposal.. hmm maybe that’s too ambitious, but it does feel like the sort of thing you *should* be able to do with Perl..

I didn’t have much time left to do any real coding after that, except for updating a few dependencies so that Tarpo at least starts up ok on the latest version of Adobe Air (a small victory). But at least now in a position to start attacking the ticket list, which means that real progress can’t be too far off..

There’s a new wobject coming in the latest WebGUI beta (version 7.7.3), called Carousel. On the surface, it’s a nice little integration between WebGUI and YUI, but below the surface it’s a great multi-purpose tool for rendering content dynamically in Javascript or Flash.

It’s a common requirement to want to display rich text via Javascript or Flash. For example you might have a clickable diagram that shows different text depending on what part of the diagram a user clicks on.

Giving content managers access to the dynamic text can be rather cumbersome. In the worst case scenario the text is hard-coded in the Flash or Javascript source, out of reach of the content manager. Better solutions are where you have a bunch of editable Articles or Snippets that the Flash or Javascript reads from.

None of these solutions are very satisfactory though, because as far as the content manager is concerned the dynamic object is a single asset, and they really want a simple way to edit all of the text elements from the same place.

We designed the new Wobject specifically for this scenario. The Edit Screen of the wobject allows content managers to add as many rich text editor (RTE) boxes as they like (dynamically, a la FilePile). These RTEs are displayed one after the other on a single screen, allowing the content manager to edit all of the text in a single place. For each RTE, the content manager can also specify a unique ID.

By default, the template for the Carousel wobject renders your RTE items into an attractive YUI Carousel widget:

The Wobject template does this by looping over all of the RTEs and generating markup expected by the YUI Carousel widget. This has the nice side-effect of allowing your widget to gracefully degrade in non-js browsers.

Taking a step back, we have a WebGUI wobject that lets us manipulate a collection of RTEs. The only part of the wobject specific to the YUI Carousel is the template. Let’s have a look at it:

<div class="yui-skin-sam">

...

<tmpl_if item_loop>
<div id="<tmpl_var assetId>">
<ol>
 <tmpl_loop item_loop>
	<li class="item" id="<tmpl_var itemId>">
 <tmpl_var text></li>
</tmpl_loop></ol>
</div>
<script>
 YAHOO.util.Event.onDOMReady(function (ev) {
 var carousel = new YAHOO.widget.Carousel("<tmpl_var assetId>", {
 isCircular: true,
 numVisible: 1,
 animation: { speed: 0.5 }
 });
 carousel.render(); // get ready for rendering the widget
 carousel.show();   // display the widget
 });
 </script>

</tmpl_if></div>
</script>

We can just as easily create our own template to do something else with the data. Returning to our original example, say you want to use the RTE text in a Flash diagram. All you have to do is change the template to output the contents of each RTE into a textarea dom element, styled to be hidden from view, with the id attribute set to the RTE unique id. Then, your Flash (or Javascript for that matter) can easily read in the rich text via DOM methods (e.g. document.getElementById(RTE_unique_id).value) and display it dynamically and/or do whatever it likes with it.

For example, we might use a template such as:

<tmpl_loop item_loop>
 <textarea id="<tmpl_var itemId>" style="visibility: hidden;"><tmpl_var text></textarea>
 </tmpl_loop>

The only coordination required is that the person deploying the wobject would need to know how many text blocks the flash/javascript expects, and what unique ids to use.  For some flash/javascript widgets you could ignore unique ids altogether by adding a special class attribute to the generated textareas and then using a class selector to dynamically select all matching textareas on the page.

This should make it easier to use dynamic flash/javascript widgets in WebGUI whilst still retaining the power and ease of use of a content management system.

Later that day you get home and on a hunch run a quick search through your source tree:

> ack  ‘console.log’
www/extras/wobject/Survey/dd.js
96:console.log(destEl);

The culprit is Firebug’s console.log() function. The damn thing is just too handy. So handy that a few years ago the collective web development community abandoned alert() as the #1 debugging weapon of choice. Now we all use console.log(). Forget the firebug debugger, nobody can be bothered bringing up their js source in the script tab and clicking the break button or setting a watch variable, that’s like, 4 clicks too many man! Whack in a of console.log(a, b, c)s and it’s all there, plain as day in the firebug console, ready to be clicked on and examined further  in the DOM tab.

Until you forget that not everyone in the world runs firebug. Hard to believe, I know. But actually, the problem occurs for anyone who hasn’t got the firebug console turned on for YOUR SITE. The problem being of course that you probably always have the firebug console turned on for your site, but no-one else in the world does. So the likelihood of being bitten are pretty high.

Of course, there are lots of workarounds. One approach is to just define console.log as an empty function (and while you’re at it, you may as well define all the other Firebug functions too):

if (!("console" in window) || !("firebug" in console)) {
 var names = ["log", "debug", "info", "warn", "error", "assert", "dir", "dirxml", "group", "groupEnd", "time", "timeEnd", "count", "trace", "profile", "profileEnd"];
 window.console = {};
 for (var i = 0, len = names.length; i < len; ++i) {
 window.console[names[i]] = function(){};
 }
}

In my case the code was part of a drag and drop method used in WebGUI. And like all sane web projects, WebGUI oursources its cross-browser Javascript concerns to a library (in our case YUI). And it turns out YUI has a perfectly good logger of its own, guaranteed to be available on any page that YUI is loaded onto. Hello YAHOO.log();

Now, “YAHOO.log” is 1 letter shorter than “console.log”, so why does everyone end up using console.log() all the same? Maybe they don’t like typing in upper-case. More probably they noticed that when they use YAHOO.log(‘blah’); nothing appears in the firebug console. What gives? Well, you need to tell YAHOO to pass on log  messages to the console, that’s what. Type the following line first, and everything will work out just fine:

YAHOO.widget.Logger.enableBrowserConsole();

Of course, that’s a lot more letters than just typing console.log(), and what’s more you need to use YAHOO.lang.dump() to serialise objects passed to log() (no nice firebug object examination), which admittedly is a lot less fun. But it does mean a little less broken code in production.

Recently one of my clients who was using the Survey wobject in WebGUI noticed their modperl server grinding to a halt. Funny how one little decision can trigger such a catastrophic failure. Funny in the sense that it takes you a few days to repair the damage and track down the root cause of the problem.

The reason modperl was grinding to a halt was easy enough to find – child processes chewing up as much memory as they could whenever a web request touched a Survey instance. The cause of that was also quickly apparent.. one of the text fields in a Survey instance was filled with ~150Mb of rubbish data. The smoking gun looked like this:

In contrast to obsessions, your worries don\x{c3}\x{83}\x{c2}\x{83}\x{c3}\x{82}\x{c2}…

An apostrophe in the text and then BAM! everything goes haywire (that \x{blah} unicode text goes on for 150Mb..)

Turns out the user who triggered the problem was cutting and pasting text from Microsoft Word. As anyone who has worked with people who cut and paste text from Word into content management systems knows, Word uses non-ASCII characters for “smart quotes”, apostrophes, dashes, triple periods, etc.. Those non-ASCII characters usually show up in HTML as garbage characters. In this case it looked like those characters were causing the Survey internals to break.

On closer inspection Survey was breaking as follows: any non-ASCII text was converted to a few garbage characters. On the next view/edit, each of these garbage characters were turned into a few more garbage characters, and so on.. which meant that in no time at all the web server had ground to a halt and the Survey table was weighing in at hundreds of Mb. Fun!

Now, I knew that JSON was supposed to be UTF8 encoded, and Survey serialises its data to JSON, so somewhere along the way the JSON string/Perl String/MySQL string encoding was breaking.

Best practices in WebGUI dictate that you work with JSON as non-encoded, non-binary, Perl UTF8 strings and then let wg modules such as WebGUI::Session::Form do the encoding for you (via the Encode module). Most of the time you don’t need to care if a string in Perl is UTF8 or not, and doing things this way means that most of the time you can remain blissfully unaware of what’s happening under the hood.

In accordance with this, Survey uses JSON::to_json/from_json everywhere. On a hunch I switched all the from_json calls to decode_json, which expects a binary, UTF8-encoded string instead of a non-binary string, and voila, the problem disappeared. So somewhere along the way our JSON-encoded string was being converted to a binary string. The question was: where?

The plot thickened when I noticed that calling Encode::decode_utf8() immediately after reading the string from mysql also removed the issue (decode_utf8 turns a binary UTF8 string into a non-binary utf8 string).

With the help of perlmonkey2 and perlDreamer on #webgui, we found the root cause.. the json-encoded string was being stored in a mysql field of type longblob as opposed to a non-binary type (such as text, mediumtext, etc..). Something was transparently converting it to a binary UTF-encoded string (perhaps DBD::mysql?), and when  Survey then pulled this binary string out the database, it was trying to use from_json() to decode it (which expects a non-binary string).

A quick change to the field type:

alter table Survey modify surveyJSON long

and all was well again with the universe, but if you’d asked me 4 hours ago what mysql field type should be used for json my first thought would probably have been a binary type, because all I knew was the JSON is UTF8 encoded. A little bit of knowledge can be a dangerous thing..

Some functions always return a value. For example, the following simple function always returns the value of PI:

function pi(){
    return Math.PI;
}

Some functions go into infinite loops and never halt. For example, the following function enters a while loop and never comes out:

function eternity(){
    while(true) {
        // goodbye, cruel world
    }
}

Wouldn’t it be nice to be able to tell ahead of time whether a function is going to halt or go into an infinite loop? (cue evil laughter from the computer scientists in the back row)

Let’s write a function that tells us whether another function halts. (It’s easy to write functions that take other functions as arguments in Javascript because it’s a functional programming language). What would such a function look like?

function halts(fn){ // takes one argument, fn (a function)
    // code that determines if fn halts goes here
}

We’re not sure yet what code to put into the function body, but let’s start by looking at what we know about this function. It has to return True or False. True if the argument fn halts and returns a value, False if not. Simple right?

Let’s try out halts() on some functions we’ve already talked about to get a feel for how it’s supposed to work.

halts(pi); // returns true because pi() halts
halts(eternity); // returns false because eternity() doesn't halt

So far so good. Ok, let’s create one last function just for fun and see how halts() goes.

function maybe() {
    if (halts(maybe)) {
        eternity(); // never ends..
    } else {
        return "ok I halted!";
    }
}

The question is, does maybe() halt? Well let’s see..

• Let’s assume that maybe() halts. Therefore the if clause is true and so eternity() gets called.. meaning that maybe() never halts. Wait! That can’t be right! We just contradicted ourselves. We must have guessed wrong.
• Ok, in that case, let’s say that maybe() doesn’t halt. Therefore the else clause gets called instead, and we return a value. Meaning that we halted. But that’s a contradiction too!

What this means is that it’s impossible to write a function like halts() that determines whether another arbitrary function halts or not, because the very definition of such a function leads to a contradiction. Sucks to be a programmer huh?

Thanks Alan M. Turing (1912-1954), and Kurt Gödel (1906-1978).

What happens, however, when your intended users have no internet access? I was recently faced with that exact problem: My Dad (Dr Ted Donelan) runs a pro bono dog health program in Maningrida, a self-governing aboriginal community in Arnhem Land (500km east of Darwin) where, unless you want to pay through the teeth for that Telstra NextG internet rubbish, there’s no easy access to the internet. And certainly no way of logging into a website while you’re riding around from house to house in a ute or carrying out surgery on an outdoor trestle table underneath the local grog-shelter. Dr Ted wanted a way to record the statistical information that he collects (dog body conditions, mange scores, fleas, ticks, etc..) along with his formal veterinary records of surgical cases, medical cases, etc.. All of which can be neatly captured in a handful of relational tables (attested by the fact that previously he had been recording his data in hand-written tables, later to be transferred to Excel). I suppose I could have stepped back into the C#/Java world and implemented a stand-alone app to solve the problem, but I’m not familiar enough with any frameworks in those languages to whip up something quickly. And that was the point, I only had a small amount of time to come up with a solution. And what I really wanted was a way to build him a stand-alone app using my existing HTML/Javascript/CSS-fu..

Cue Adobe AIR.. a desktop framework that lets you build traditional dektop apps using web technologies. Now I’m not the biggest fan of proprietary technology (Adobe Flash being a prime example) since it goes against the grain of the Open Web, but there’s really no open-source alternatives at the moment.

The first really awesome AIR app I saw was ExtJS creator Jack Slockum’s “Simple Tasks“. Check out the screen-shots and you’ll see why I was excited: here was a desktop application with an embedded (SQLite) database running pure HTML/CSS/Javascript. And not just any Javascript either. ExtJS shines most when used to create windows application-style interfaces, and as such it’s the perfect library to use in an AIR app when you really are building a desktop application.

Simple Tasks is released under GPL v3, so I decided to base my code on Jack’s. It was the right decision. With no prior experience with AIR it took less than a day to have a working prototype, and by working I mean a full database implementation, an editable data grid and an Add/Edit form for individual rows, all with the trademark ExtJS polish. That’s ridiculously fast.

After about 4 days with lots of feedback from Dr. Ted we had a finished app, named Tarpo, complete with polished input forms, reporting, backup, CSV/XML export, …

I’ll be releasing Tarpo under GPL v3. Until then I’ve got a little page where you can install it straight off the web and have a play. Dr. Ted is heading up to Maningrida next week so I’m excited to see how Tarpo goes on her first test-run. If everything works well we’ll make her available to other Vets running similar programs in other indigenous communities. One cool follow-on from this will be that vets running AMRRIC-endorsed programs could use Tarpo to submit their activity reports direct to the AMRRIC website (which I recently developed).

Andy (also of SDH) is whipping up a Tarpo logo for me, which I’ll post up here as soon as he’s finished inking it!

How to use it the Multiple Columns Bookmarklet:
This is the link to my Multiple Columns Bookmarklet.

For starters, try it out yourself on this page! Select a few words in the first paragraph of this page and click the bookmarklet link. You should see it magically reformat into multiple columns 200px wide.

Now select two continuous paragraphs and click the bookmarklet link. You should see the entire article reformatted into easy to read columns. Cool huh?

If you want to use the bookmarklet on other pages, bookmark it now. When you’re on a page that you want to reformat select the chunk of text and click on the bookmarklet. voila!

If you select text a single paragraph, only that paragraph will be turned into columns. If you select multiple paragraphs, the containing DOM element will be reformated (this isn’t perfect but it usually does what you want). If no text is selected when you launch the bookmarklet, the entire page will be turned into multiple columns (that won’t look great on my blog, but it’s handy on pages with minimal styling)

If you add a keyword such as col to your bookmarklet link (Right Click > Properties), then you can launch the bookmarklet from the location bar by typing col rather than clicking on the link.

Finally, if you provide an argument of max, eg. col max then the maximum height of the columns will be set to the size of your web browser window. This is really handy if you’re viewing a long document and you don’t want the columns to scroll vertically for miles and miles (by setting the column height the columns will instead continue horizonally, just like a large-format newspaper).

In case you’re wondering, the bookmarklet is only intended for Firefox because none of the other browsers have implemented any column-related CSS features.