Latest Web::Simple master (git://git.shadowcat.co.uk/catagits/Web-Simple.git) is now using Plack, the perl web server directly for CGI and FastCGI and I hope do do another release shortly with notes on deploying as both on shared hosts (more specifically Dreamhost but with an invitation to bitch if they don’t work on other budget hosts).

mst,  ”Oh Subdispatch, Oh Subdispatch“

Every time someone mentions a cool new web framework like Web::Simple, Dancer or Tatsumaki, my immediate reaction is to start thinking up cool little niche web apps I could build with it. You know, the kind you imagine yourself whipping up during your lunch break, that will probably never get more than 10 curious users but just might turn into the next big thing if only you got it up and running on a public server. And that’s when the second thought immediately arrives: where am I going to run this thing? Do I really think the idea has enough legs to justify paying for a virtual server plan? Can I be bothered going through the pain of figuring out how to deploy it on a shared hosting provider? And normally that’s the point where I sigh wistfully and go back to reading my RSS feeds.

The thing is, with the advent of things like local::lib it’s getting a lot easier to deploy Perl web apps on shared hosting. And with most web frameworks adopting Plack/PSGI, the work required to deploy Perl web apps on budget hosts is converging into a similar sequence of steps.

So this time during my lunch break I started envisaging a centralised, SEO-friendly information source (hello perlsharedhosting.com) that cobbles together all of the currently available information into an easy to digest form to make it ridiculously easy to choose a shared hosting provider, deploy your web app and troubleshoot common problems.

A site that works like this:

• The front page contains a list of Perl-friendly hosting providers, with a meta-score based on how many features they support (+) and how many unresolved issues they have (-), and maybe user review scores thrown into the mix too
• Each hosting provider has a page of its own, listing in full which features are supported (used to compute the meta-score). Users can post comments on each of these pages, to leave testimonials (“I am currently running a Web::Simple site that gets x hits per hour on this host”) and note unresolved issues (“module X::Y fails to install on this host”). As issues get solved these become tips.
• Each technology/feature/technique also gets a page of its own, with links to the official man pages, shared-hosting specific notes and again user comments
  • dependency related: local::lib, cpan, cpanm, ..
  • environment related: locating and running perl, paths, daemons, viewing error output, ..
  • web deployment related: CGI, FastCGI, ..
  • framework / web app related: Web::Simple, Dancer, MojoMojo, Catalyst, Tatsumaki, ..

I haven’t decided yet what engine would fit best.. MojoMojo is a front-runner, and in fact the Catalyst Friendly Hosting page runs on MojoMojo does a large chunk of what I’ve described above.

The site would be very “cookbook” oriented, since we’re specifically targeting people who can’t be bothered learning the ins and outs of 10 different technologies (not to mention figuring out how to get them to play nicely together) for the sake of deploying toy web applications on cheap shared hosting. Just the essentials: this is what you need, this is how you achieve it. And if you want to learn more, go here.

With that in place, we’d end up with a single, visible, place to research and document the complications of running Perl web apps on shared hosting. And if the site became popular, hosting providers might even take notice and start offering more Perl-friendly shared environments. We could have live demo pages running on different hosts, possibly even donated by hosting providers if they see it as a way of showcasing their Perl-friendliness (mojomojo.dreamhost.perlsharedhosting.com, dancer.resellerzoom.perlsharedhosting.com, etc..).

I got as far as registering the domain name; I was planning on getting a simple MojoMojo prototype up and running, but I got side-tracked researching how to deploy it on my shared hosting account…

As the name suggests, Padre::Plugin::Plack adds Plack awareness to Padre. With the plugin installed, opening *.psgi files causes some special things to happen. PSGI files are really just ordinary Perl files, so Padre does its normal Perl lexing/syntax highlighting magic on them, but the real fun starts with the Plack-specific features that appear in the per-file graphical plackup control panel that shows up. The panel lets you run your web app in a Plack server at the click of a button, view server output, configure plackup options and launch a web browser on the appropriate port.

The great thing about Plack/PSGI is that unlike my previous plugin (Padre::Plugin::WebGUI) which was specific to a single web app (albeit a big one), this plugin can be used for any web app built in a web framework that supports Plack (Catalyst, CGI::Application, HTTP::Engine, etc..). This potential for cross-framework application is one of the motivating factors that makes developing Plack modules (Middleware etc..) so much fun.

The plugin turns on plackup’s “–reload” option by default, which conveniently causes the plack server to reload every time you modify your source files in Padre. This makes for quite a nice, if somewhat minimal “Plack IDE” experience (this is version 0.01 after all).

The plugin integrates all of the Plack example “dot-psgi” files as templates that can be used to create different types of Plack apps straight from the GUI menu.

The pre-populated list of Plack servers and the simple start/stop button makes for a nice way of exploring the Plack server ecosystem. You can use the other panel options to enter a specific port to run on, toggle auto-start mode and pass additional options to plackup (options that start with “–” are passed through to the backend server).

The output panel is similar to the output panel that Padre normally displays when you execute Perl files, except that you get one panel per .psgi file meaning that you can run multiple plack servers simultaneously and independently view their output. The appropriate panel is automatically selected when you click on the corresponding file tab, and running processes are stopped when you close the tab.

It should be really easy to turn Padre::Plugin::Plack into new plugins that involve the same basic ingredients, namely  a file extension and an external command for running those files, with a per-file panel for command options and output. So I encourage anyone who has a similar plugin in mind to steal liberally from Padre::Plugin::Plack (as I did from Padre::Plugin::Catalyst – thanks garu++). Ruby Rack support comes to mind as a trivial example.

Make Padre your domain-specific IDE today

Plack has a really neat solution to this problem.

Following a tip from day #5 of Miyagawa’s Plack advent calendar, serving up the current directory over http is as simple as this Perl one-liner:

$ plackup -MPlack::App::Directory -e 'Plack::App::Directory->new

Plack::Server::Standalone: Accepting connections at http://0:5000/

Or if you need the files served over port 80:

$ sudo plackup -MPlack::App::Directory -e 'Plack::App::Directory->new' -p 80

Plack::Server::Standalone: Accepting connections at http://0:80/

At about 11pm on the night before I headed off for the States, Ted the Vet and I sat down to talk Tarpo.

In case you’re wondering, Tarpo is an open source, cross platform Data Management desktop application for Dog Health programs in rural and remote Indigenous Australian communities, written entirely in Javascript.

The last time I blogged about Tarpo was way back in April 2008, and since then I haven’t really touched the code, but in the meantime Ted and his team of helpers have used it to record several thousand House Visits, Medical and Surgical Cases as part of the Maningrida Dog Health program.

Several other Veterinarians have started expressing interest in using Tarpo for their own Dog Health programs, and needless to say, we’ve accumulated a long list of bugs and feature requests over the past 18 months. So this afternoon, with a bit of spare time up my sleeve, I decided to get the ball rolling again.

Tarpo now has a proper project page, which you can visit at: http://pdonelan.github.com/tarpo

As you can see from the url, the code now lives on GitHub. Apart from the front page which has the all-important “Install Tarpo” button, the most important page on the GitHub project page is the Issue Tracker, which I’m hoping Ted and other Vets will use to report all Bugs and Feature requests. With a bit of luck, that will also make it easier for other developers to get involved too.

With these things in place, I started attacking the bit rot. Firstly I had to install Aptana and the Adobe Air plugin, which took several hours and reminded me how much I prefer doing development in Padre! On a side note, I spent a few hours trying to think if there was some way I could rewrite the entire app in Perl (I like Javascript in a twisted sort of way, but I’d rather ditch Adobe Air since it’s a proprietary framework) – but in the end I decided to stuck with the existing architecture, at least for now.

I though about doing it as a wxPerl app (like Padre), but I’d hate to lose the HTML widget set that ExtJS gives me.. hmm that gets me thinking (help me out if you’re reading Gábor or any of the other Padre gurus..) does wxPerl have a modern embedded browser that I could use? Maybe webkit-based one..? Then the backend could be rewritten as a standalone Plack-powered webapp and I’d have all of CPAN at my disposal.. hmm maybe that’s too ambitious, but it does feel like the sort of thing you *should* be able to do with Perl..

I didn’t have much time left to do any real coding after that, except for updating a few dependencies so that Tarpo at least starts up ok on the latest version of Adobe Air (a small victory). But at least now in a position to start attacking the ticket list, which means that real progress can’t be too far off..

I got a kick out of seeing Miyagawa pull up my blog post on PlebGUI in response to someone’s question about porting mod_perl applications (e.g. apps that pass around an Apache $r object) to Plack. For the sake of the guy who asked the question, the way I did it (at least for the first pass) should be equally viable for other mod_perl applications – just create a fake Apache $r object that delegates everything to Plack::Request and Plack::Response (your mileage will obviously vary depending on how much of the mod_perl API your app uses).

1. To sync music to my iPhone
2. To use the Australian Tax Office website

I would dearly love to expunge windows from my life entirely, but so far I haven’t been able to find suitable workarounds. For the iPhone, the obvious solution is to follow chromatic’s advice and buy hardware that supports linux, such as an Android phone – and now that Android 2 is out that is looking increasingly viable.

The ATO issue is rather more intractable. Any linux user in Australia who run a business and/or submits their own tax return via the ATO website will be painfully aware that the site doesn’t work on non-proprietary operating systems; specifically, anything other than Windows or Mac. The culprit is the “Common use Signing Interface (CSI)” that the site uses to “allow businesses to securely transact online with Government agencies using digital certificates”. CSI is written in Java, which despite being a horribly verbose enterprise-friendly language, is at least supposed to be a horribly verbose cross-platform enterprise-friendly language (remember WORA?). But in spite of this, the developers did their bit to prove Kirrily Robert right and made the application only work under Windows and Mac.

For fun, I decided to submit this as a bug. Subsequent correspondence with the ATO follows for your enjoyment.

Mike@ATO:

I spoke to you yesterday in relation to the feedback you gave the ATO in relation to our Business Portal.  As I said on the phone  here is an email with some information I have been able to find.

I have received some advice from our Portal Support area.  Their response to the issues you raised were:

ATO Portal Support: (my emphasis)

Stated here is that the client cannot log into the Business Portal as he is running Ubuntu and Firefox. In this situation Firefox is not the issue as Firefox is supported for the Portal and testing has been completed around this.

The issue for this client is to do with CSI (the digital certificate software) and Ubuntu. CSI would not be able to be installed on Ubuntu and is not supported for CSI software as per the link below:

http://www.ato.gov.au/onlineservices/content.asp?doc=/content/36220.htm

If the user was using Firefox on a windows based system this would work correctly.

The difficulty with Linux based operating systems is that the percentage of users is quite low for Linux operating systems and also there is a large quantity of Linux distributions (or types) making it very difficult to support.

Hope this answers the queries set out below, if you need any further information please don’t hesitate to contact me.

Mike@ATO: (my emphasis)

Probably not what you wanted to hear and not sure how much this helps you.  It is unlikely that the ATO would be doing any work to address the compatability issues for Ubuntu due to the costs involved and the anticipated small number of users. The ATO believes that it is complying with the Australian Government’s guidelines and recommendations for government websites.

If you believe that the Business Portal is not supporting Firefox and were able to provide the specific circumstances it would give us an pportunity to do further investigations and address those concerns.  If you have any further issues or comments feel free to contact me.

Me:

Thanks Mike, I’m really impressed with the way you’ve followed up on this.

That’s great news re: Firefox.

If it’s not imposing too much, would you be able to ask the Portal Support person what percentage of Portal users are Linux users and what percentage of Portal users have Javascript turned off?

For example, on an internet-wide level, the latest W3C stats show Linux usage at 4.1% vs 5% for browsers with Javascript turned off. This is likely to vary wildly on a per-site basis, which is why I’m interested to find out what the specific ATO Portal numbers are.

I’m wondering what percentage is required before the Linux user group becomes worth supporting, and seeing how this compares to the current percentage of disabled web users which the ATO website currently supports via Australian Governement Accessibility standards (e.g. http://australia.gov.au/about/accessibility).

Mike@ATO: (my emphasis)

The following information is provided in relation to your latest enquiries.

In relation to the Business Portal we do not gather specific statistics regarding specific operating system usage or if a user has Java script turned off.  In saying this though I could assume that there would be 0% Portal users utilising a Linux Distro, reasoning behind this response is where by CSI (the software that allows login with a digital certificate) cannot be loaded onto or used on a Linux operating system and is not supported by the ATO.  So basically there wouldn’t be any users as they wouldn’t be able to access via Linux.  Regarding the Java script query, this is not something we would be able to answer as we do not gather those statistics.

The statistics you have quoted are an internet wide statistic as stated and cannot be related to the Portal as in the first response there would not be any users as CSI is not supported for Linux.

It should be noted that as the stats for usage for Linux is quite low this is not just an issue affecting the Portal and the ATO. There are a plethora of systems and software that does not support Linux as the user base is just not there to support. This position also aligns to that of the majority of the software developers producing the accounting and practice management software used by our clients.

The link you provided to the Australian Government Accessibility Standards goes to a page discussing the commitments by the Australian Government to ensure access to online information for people with disabilities.  There is no corollary between our obligations to support users with disabilities and users of specific operating software.  Government agencies are bound by specific legislation (Disabilities Discrimination Act) on the issue of accessibility of Government services.  This does not apply in the case of operating software choice.

The final point to note is a Catch 22 situation ie Linux users aren’t clients because the systems don’t support them and because they can’t use our systems they’ll never make up a large enough percentage of our clients to warrant changing our position.  I think the answer to this is that the Linux users of our website as opposed to our installable applications (e-tax, ECI, CSI, eSAT) should not be overly impacted by compatibility issues.  Statistics collected by the ATO show that only 0.3 percent of visitors to ato.gov.au were identified as Linux users.

In summary:

* We do not routinely test our online applications against Linux;
* Consequently, we do not support Linux through our technical support areas;
* Given the very low verifiable client base percentage (<1%) that are Linux users, there are no current plans to change this approach;
* There is no defined threshold % at which this position would change.

Realistically, in the absence of any substantial research or statistics confirming a much greater Linux client base then we are currently aware of, the position stated in the dot points above is unlikely to change.

I also appreciate that this situation is less than satisfactory to the Linux clients, however, we are bound under our policies and financial legistlation to ensure that expenditure of public monies promotes the efficient, effective and ethical use of all Commonwealth resources and provides value for money.  Any stance other than that above would not be in keeping with our obligations under the Financial Management and Accountability Act.

Hope this further information answers your enquiries.

So there you have it. End of story. No ATO for linux users. Ever.

..

heh yeah right.

Where the Australian Government fails you, the linux community saves you. Someone slap me for wasting 3 weeks on an email exchange instead of just asking google.

Whirlpool.net.au to the rescue:

sudo apt-get install sun-java6-bin sun-java6-jre sun-java6-plugin
export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre
wget http://pki.ato.gov.au/atocsiInstall/CSIinstall.dmg
sudo mkdir /tmp/csi
sudo mount -t hfs -o loop CSIinstall.dmg /tmp/csi
sudo cp -p /tmp/csi/CsiInstaller.pkg/Contents/Resources/jarFiles/csi.jar $JAVA_HOME/lib/ext
sudo cp -p /tmp/csi/CsiInstaller.pkg/Contents/Resources/jarFiles13/jce1_2_2.jar $JAVA_HOME/lib/security
sudo umount /tmp/csi
sudo rmdir /tmp/csi
java au.gov.bafcsi.clapi.crypto.CsiManager # run CSI certificate manager program

You can even drop your .csi directory from your windows home dir into your linux home dir and your existing certificates will be appear without any need to manually export/import them. After completing the above steps, completely disable AdBlock, restart Firefox and you can log in to the ATO website!
(tested on Ubuntu 9.10).

Now all I have to do is ditch my iPhone..

But you see mod_perl is an overly zealous lover. Every intimate phase of the Apache request cycle is offered up to your eager Perl embrace. Sure, you have to learn a few new tricks to get decent performance (such as two-tier mod_proxy/mod_perl) but hey it’s 2001 and mod_perl is SO much better than CGI. Without hesitation you commit to a life of PerlResponseHandlers and Apache2::Const::OKs.

Years pass. Life is good.

Your user base had become very adept at deploying Apache. In fact you make life easier for them by distributing the complete Perl/Apache/MySQL stack as a simple installer, pre-configured for optimum performance.

Every now and then someone appears on the mailing list asking questions about WebGUI 5, a throw-back to the days when it used to be possible to deploy WebGUI on cheap shared hosting cPanel servers in CGI mode. You want to help, but WebGUI has become so powerful that CGI mode isn’t feasible anymore. That’s the price that was paid for evolving into an Enterprise-grade system. Developers lament the fact that small-time users can’t take advantage of all the awesome things WebGUI can do, not to mention all the word-of-mouth promotion WebGUI is missing out on, but them’s the brakes. WebGUI continues to grow. “Carrier-grade” is the new black. And across town, the non-enterprise crowd is left to content themselves with Wordpress, Joomla and Drupal.

And then a guy called Tatsuhiko Miyagawa comes along.

He says, gee, look at these wonderful server abstractions that Python (WSGI) and Ruby (Rack) have. The Perl world might have moved on from CGI to things like Catalyst::Engine::* and HTTP::Engine, but there’s still duplicated effort everywhere. We can do better.

So he sits down and writes PSGI, an absurdly simple, manifestly beautiful specification for an interface between Perl web apps and web servers (drawing heavily on WSGI and Rack for inspiration).

PSGI

The idea goes like this: Web applications, when all is said and done, are really just on about sending three pieces of information to web browsers: a HTTP status code, a list of HTTP headers, and some content (a file or some text, normally HTML).

This is the specification:

[                                           # an array ref, containing..
    200,                                    # a HTTP status code
    [ 'Content-Type' => 'text/html', .. ],  # an array of HTTP headers
    [ '<html>...</html>' ],                 # the text content (or a $filehandle)
]

And that’s really all there is to it.

Some people write web apps according to the interface. Other people write server backends according to the interface. Web app developers use whatever whiz-bang technology they like, and as long as they return an array that complies with the spec their web app can run on any server. And server backend developers can do lots of clever things to get that information back to web browsers really fast, and have the results of their work benefit all PSGI/Plack consumers. No more duplicated effort. No more server-specific love lock-in.

And folks took notice. Not just any folks either. Really, really smart people like Yuval Kogman, Stevan Little, Shawn Moore, Matt Trout, Jesse Vincent, Chia-liang Kao, Dave Rolsky, Simon Cozens, ..  (They’re some of the well-known names that jumped out at me from the PSGI.pod spec doc. The others are probably even smarter, stealth hackers. I know, it’s scary).

Plack

Before you could say Plack there was a reference implementation, server backend support for CGI, FastCGI, mod_perl (welcome back!), AnyEvent, Coro, Perlbal, Nginx, .. framework support for Catalyst, CGI-Application, Mason, Continuity, Maypole, Mojo .. and a whole suite of Middleware and Utilities.

Meaning that all of a sudden any web app written in one of those frameworks can now be deployed on any of those servers. Or on the pure-perl standalone server that runs from the command line. Or on one of the more experimental/exotic servers I haven’t listed. If you’re not excited yet, just wait until Google AppEngine appears in that list.

WebGUI

But where is WebGUI! The problem is that WebGUI is both a framework and a web app. A really big web app. With a mod_perl addiction. Frameworks are built with multiple servers in mind, so they generally already have an in-built server abstraction layer. Which makes adding PSGI support relatively simple. WebGUI, on the other hand, deliberately eschews an abstraction layer so that it can fully embrace mod_perl and eke out every last ounce of power and performance it can from Apache.

So, faced with extreme framework envy, I did what any reasonable person would do.

I built a PSGI/Plack layer for WebGUI.

PlebGUI

I’ve codenamed the project “PlebGUI“, which I think aptly describes the way it makes it possible for the little people to run WebGUI on low-cost shared hosting.

And it actually works. Take for instance plebgui.patspam.com, a demo PlebGUI site site running in FastCGI mode on HostMonster (the prototypical low-cost shared webhost).

app.psgi

The second wonderously simple idea in the PSGI spec is that a web app is just a plain old perl subroutine. Here’s one I prepared earlier:

sub { [ 200, [ 'Content-Type' => 'text/html' ], [ 'Hello World' ] ] }

I know, it’s almost insulting. I’m a web developer man! I do sophisticated things! But try putting that single-line sub into a test file called app.psgi. And then after you’ve installed Plack (it’s not on CPAN yet so you have to install it from miyagawa++’s git repo) try running this:

$ plackup
Accepting connections at http://0:8080/

Go on, visit that url in your web browser. Hello to you too!

Middleware

Ok that was a cute trick, let’s try something more exciting:

use Plack::Builder;
builder {
    add "Plack::Middleware::Static", path => qr/./, root => '/var/www';
    sub { [ 404, [ "Content-Type" => "text/plain" ], [ "Not Found" ] ] };
};

Congratulations, you just added your first Middleware. Assuming you have some static files located at /var/www, you’ll get the static files returned to your browser with the correct mimetype (thanks to Plack::Middleware::Static). Middleware just wraps your web app (a plain old Perl sub) with another plain old Perl sub. Middleware can do logging, pretty HTML stack traces, pre/post processing, or anything else you like. Simple, but immensely powerful.

Plackup

Plackup is a simple utility script that launches your web app with a specified server backend. By default it runs the pure-perl standalone development server. It expects your webapp to live in a file, similar to the ones we just created. Want to run your web app on another server backend? Try one of these:

plackup                                         # dev server with StackTrace and AccessLog enabled
plackup -s CGI                                  # remember how slow web apps used to be?
plackup -s AnyEvent                             # nonblocking
plackup -s Coro --port 9090                     # coroutines
plackup -s Standalone::Prefork --max-workers 20 # blazingly fast preforking ftw!

dev.localhost.localdomain.psgi

Here’s what the current per-site .psgi file looks like for PlebGUI:

use Plack::Builder;
use lib '/data/WebGUI/lib';
use WebGUI;

builder {

 # Populate $env from site.conf
 add 'Plack::Middleware::WebGUI',
   root => '/data/WebGUI',
   config => 'dev.localhost.localdomain.conf';

 # Handle /extras via Plack::Middleware::Static
 add 'Plack::Middleware::Static',
   path => qr{^/extras/},
   root => '/data/WebGUI/www';

 # Handle /uploads via Plack::Middleware::WGAccess (including .wgaccess)
 add 'Plack::Middleware::WGAccess',
   path     => qr{^/uploads/},
   root => '/data/domains/dev.localhost.localdomain/public';

 sub { WebGUI::handle_psgi(shift) };
}

What you can see there are 3 Middleware layers added in, one to set up the WebGUI site-specific environment, one to handle /extras static content, and one to handle /uploads static content (taking into account .wgaccess file permissions).

All of those plackup command variations above can be used to launch WebGUI outside of mod_perl. Prefer running inside of Apache? How about one of these:

<VirtualHost *:80>
 PerlOptions +Parent
 PerlSwitches -I/data/WebGUI/lib

 # CGI
 #AddHandler cgi-script cgi
 #ScriptAlias / /data/WebGUI/etc/dev.localhost.localdomain.cgi/
 #<Directory /data/WebGUI/etc>
 #   Options +ExecCGI
 #</Directory>

 # mod_perl
 #SetHandler perl-script
 #PerlHandler Plack::Server::Apache2
 #PerlSetVar psgi_app /data/WebGUI/etc/dev.localhost.localdomain.psgi

 # FastCGI
 FastCgiServer /data/WebGUI/etc/dev.localhost.localdomain.fcgi
 ScriptAlias / /data/WebGUI/etc/dev.localhost.localdomain.fcgi/

 # mod_psgi
 #<Location />
 #    SetHandler psgi
 #    PSGIApp /data/WebGUI/etc/dev.localhost.localdomain.psgi
 #</Location>

</VirtualHost>

Using those directives you can run WebGUI in CGI, mod_perl, FastCGI mode, or even the in-development mod_psgi Apache module.

Benchmarks

Ok so how fast are these different backends? Let’s use ApacheBench to do some simple, unscientific tests of how many requests per second we can squeeze out of WebGUI.

First we’ll start with WebGUI in its original, un-plebified form, running on the WRE (more is better):

$ ab -n 1000 -c 10 -k http://dev.localhost.localdomain:8081/ | grep ‘Requests per’
Requests per second:    122.77 [#/sec] (mean)

The result is of course completely dependent on your Apache configuration – in this case I have (StartServers, MinSpareServers, MaxSpareServers, MaxClients) = (5,5,10,20).

Ok, now have a look at these numbers:

$ ./ab.pl –app /data/WebGUI/etc/dev.localhost.localdomain.psgi
Testing implementations: AnyEvent, Standalone, Standalone::Prefork, ServerSimple, Coro
app: /data/WebGUI/etc/dev.localhost.localdomain.psgi
ab:  ab -n 1000 -c 10 -k
URL: http://127.0.0.1/

– server: AnyEvent
Accepting requests at http://0.0.0.0:10001/
Requests per second:    68.06 [#/sec] (mean)

– server: Standalone
Accepting connections at http://0:10001/
Requests per second:    64.92 [#/sec] (mean)

– server: Standalone::Prefork
Accepting connections at http://0:10001/
Requests per second:    214.54 [#/sec] (mean)

– server: ServerSimple
Plack::Server::ServerSimple: You can connect to your server at http://localhost:10001/
Requests per second:    66.43 [#/sec] (mean)

– server: Coro
2009/10/11-23:17:32 Plack::Server::Coro::Server (type Net::Server::Coro) starting! pid(1581)
Requests per second:    67.55 [#/sec] (mean)

Did you see that? Standalone::Prefork is almost twice as fast as WebGUI in the WRE! The pre-forking server’s max-workers setting defaults to 10, so the comparison might actually be fair too.

Holy Shit

I think that’s worth repeating. PlebGUI, which is currently less than one week old and running on a compatibility layer optimised for “let’s just get this thing working and worry about performance later” is already capable of out-performing the WRE in terms of raw speed by 200%. (If you believe the benchmarks).

But Plack/PSGI is not just about speed. It’s about flexibility. Think Koen de Jonge, WebGUI hosting extraordinaire at Procolix, deploying a high availability “follow the sun” WebGUI cluster on Nginx servers. Think Colin Kuskie, WebGUI test suite overlord, probing the dark corners of the WebGUI API using standard HTTP::Request and HTTP::Response pairs through mocked HTTP and live HTTP servers. Think web developers working on WebGUI client code, with a web server fully integrated into Padre. Think WebGUI entirely deployable from the CPAN.

Think of every new project added to the PSGI/Plack ecosystem as a potential new PlebGUI feature.

Where to Now

The approach I took in turning WebGUI into PlebGUI was to create a fake Apache2::Request object, since that’s the closest thing WebGUI has to a server abstraction layer. Plack contains two helper classes Plack::Request and Plack::Response that make this really easy. Currently though, that leaves us in the curious situation where WebGUI does all of its work thinking it’s talking to mod_perl, only to have its real output re-routed through the PSGI-compatibility layer, to be subsequently processed by a specific server backend. If you don’t get the joke yet, just think about what happens when the PSGI server backend happens to be mod_perl.

The benchmarks would clearly improve if we ripped out mod_perl altogether. Lots of code would simplify too. Certain parts of WebGUI could disappear altogether, such as URL Handlers which could be entirely replaced with Middleware.

The only feature that PlebGUI currently lacks is content streaming. I deliberately left that out since it will be a lot easier to achieve once mod_perl disappears. In Plack land the way to drip-feed browsers with “chunked” content is to return an IO::Handle-like object that responds to getline() and close(). The Plack folks are planning a fancy new module called IO::Writer that will Do The Right Thing under both blocking and non-blocking servers. Expect awesome things.

The following screenshot shows a nice example of several tabs open at once with different WebGUI Asset types displaying their own icons.

I’m hoping to change that. As of today I’ve stripped out all the experimental features and concentrated on the single feature that seems to have the widest possible appeal, namely, remote editing.

Version 0.02 of Padre::Plugin::WebGUI, which is on its way to the CPAN, allows you to connect over HTTP to your webgui site, view you asset tree and edit your assets from inside the Padre text editor. That hopefully makes it as useful for designers and content managers as it is for developers.

Previously the plugin got most of its power from the WGDev library. However this meant that you had to install WebGUI and WGDev as a prerequisite. Given that the WRE ships with non-threaded perl, this meant either installing these prereqs into your system perl or building another perl specifically to run them.  Not an overly arduous task, but enough of a hurdle to dissuade most WebGUI developers from playing with the plugin. That obstacle has now been removed. You do not need to have WebGUI or WGDev installed to run the plugin.

Using WGDev also meant that you could only connect locally to your site. That restriction is now gone.

Installation

Firstly, install Padre.

To allow your WebGUI site to talk to the plugin, add this content handler (WebGUI::Content::Padre) to your site and restart modperl  (the easiest way to do this is to drop the module into your /data/WebGUI/lib/WebGUI/Content dir and add the new content handler to your site config file. There is also an install script that will do the site config editing for you).

Next install the plugin from the CPAN

cpan Padre::Plugin::WebGUI

Usage

Inside Padre, go to the Plugin Manager and enable the WebGUI plugin. You should then see a WebGUI item appear in your Plugins menu.

From the WebGUI menu, tick “Show Asset Tree” and double-click on “Connect” in the new panel that appears.

You will be prompted for the URL of the site you want to connect to. Currently you need to enter you the site in the form: http://username:password@site.com (later we will add a site manager to make this nicer). Remember that this can be any webgui site that you have access to, not just a local site.

You will then see the Asset Tree dynamically populated with information from your site.

Try double-clicking on an article.

A new file tab should open with the name set to the name of your article (with [wg] prepended). The article content will be loaded into to tab, with Padre’s HTML syntax highlighting in effect. If you have Padre::Plugin::HTML installed, you can use the Tidy/Lint/Validate the HTML to your heart’s content. And every time you hit Save, your changes will be pushed up to the site as a new Version Tag.

Try double-clicking on another sort of asset, for instance a Template. Once again, the template html will be loaded into a new tab with nice syntax highlighting.

And for fun, try opening up a CSS Snippet. The plugin will detect that your snippet is set to the ‘text/css’ mime type and use CSS code highlighting instead of HTML.

And similarly for JavaScript Snippets.

Design and Extensions

As far as Padre is concerned the file you are editing is just a regular file on your file system. For example when you edit the text, an asterisk (*) appears next to the name of the file indicating that you haven’t save it yet.  The integration between the webgui content handler and the plugin is minimal, meaning that you could conceivably use this plugin to do remote editing on non-webgui sites too.

Padre has a nice architectural design, which makes it really easy for us to associate different WebGUI::Asset types with Padre::Document types. Currently most assets delegate to the default Padre::Document::WebGUI::Asset module which does HTML syntax highlighting and expects only generic content from the content handler running on your site. Padre::Document::WebGUI::Snippet has a slightly higher IQ level in that it asks the server to tell it what mimetype the asset is set to, so that it can use the appropriate syntax highlighter.

We could do some really interesting things by building asset-specific Padre::Document modules. For instance you could build a document type that uses fancy wxWidgets controls to do things that aren’t possible on the web (desktop integration etc..) And it’s just as easy to build these for your own custom asset types as it is for core ones. Given that Padre is cross-platform, you’re basically looking at a way of extending WebGUI interactions out into desktop-land.

If you’re a WebGUI designer/developer/content manager and you want to start playing with Padre::Plugin::WebGUI as a novel way of editing content, come find me on #webgui or #padre.

One problem that content managers run into is that after deleting a template in WebGUI, any assets that still refer to the template will die a horrible death when users try to view them. What would be nice is an automated way of finding all such assets that refer to missing template, so that you can update them to refer to a different template.

For starters, WGDev can give you a list of all available templates on your site via:

wgd ls root -r --format %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq > available

You can also ask for a list of all in-use templateIds, via:

wgd ls root -r --format %templateId% --filter '%templateId% ~~ /.+/' | sort | uniq > in_use

All you need to do then is ask the comm command to tell you which templateIds are in_use but not available:

comm -23 in_use available

Putting it all together on one line (without the temporary files), that becomes:

comm -23 <(wgd ls root -rf %templateId% --filter '%templateId% ~~ /.+/' | sort | uniq) <(wgd ls root -rf %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq)

The output of this command is a nice list of missing templateIds.

Of course, what you probably then want to do is ask WGDev to give you a list of asset urls that are trying to use these missing Templates, for example, if one item in your list is eHMsLCMqoXYpYrsZ7CWUtg, you would do:

wgd ls root -rf %url% --filter '%templateId% ~~ eHMsLCMqoXYpYrsZ7CWUtg'

Which would print something like

/my/broken/asset/url

Combining this with the previous command to do it all on one line is left as an exercise for the reader

Now, templateId isn’t the only Asset field that templates can be referenced under, for example, if you want to also find missing Style and Printable Style templates, you’d want to do something like:

for tmpl in templateId styleTemplateId printableStyleTemplateId
do
comm -23 <(wgd ls root -rf %${tmpl}% --filter "%${tmpl}% ~~ /.+/" | sort | uniq) <(wgd ls root -rf %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq)
done

It’s not optimised for speed, but it sure is handy. And you could apply the same approach to search for invalid group permissions, etc.. Of course if you found yourself doing this often you’re probably better off writing a dedicated WGDev plugin to wrap up the functionality.

There’s a WebGUI branch on GitHub containing some extra Template Usage-related functions – the plan is to add the ability to find all the different ways that templates are referenced (in Asset definitions, Account module definitions, …) so perhaps we will end up adding a “find missing Templates” button to the Edit Template page.

[ perl ironman ]