patspam » WebGUI

PlebGUI: WebGUI Meets Plack

Patrick — Mon, 12 Oct 2009 13:20:02 +0000

WebGUI is an Apache mod_perl application. Not just any mod_perl application; reputedly the most deployed mod_perl application on the planet. You’d be forgiven for thinking that we love Apache. And we do. Mostly.

But you see mod_perl is an overly zealous lover. Every intimate phase of the Apache request cycle is offered up to your eager Perl embrace. Sure, you have to learn a few new tricks to get decent performance (such as two-tier mod_proxy/mod_perl) but hey it’s 2001 and mod_perl is SO much better than CGI. Without hesitation you commit to a life of PerlResponseHandlers and Apache2::Const::OKs.

Years pass. Life is good.

Your user base had become very adept at deploying Apache. In fact you make life easier for them by distributing the complete Perl/Apache/MySQL stack as a simple installer, pre-configured for optimum performance.

Every now and then someone appears on the mailing list asking questions about WebGUI 5, a throw-back to the days when it used to be possible to deploy WebGUI on cheap shared hosting cPanel servers in CGI mode. You want to help, but WebGUI has become so powerful that CGI mode isn’t feasible anymore. That’s the price that was paid for evolving into an Enterprise-grade system. Developers lament the fact that small-time users can’t take advantage of all the awesome things WebGUI can do, not to mention all the word-of-mouth promotion WebGUI is missing out on, but them’s the brakes. WebGUI continues to grow. “Carrier-grade” is the new black. And across town, the non-enterprise crowd is left to content themselves with WordPress, Joomla and Drupal.

And then a guy called Tatsuhiko Miyagawa comes along.

He says, gee, look at these wonderful server abstractions that Python (WSGI) and Ruby (Rack) have. The Perl world might have moved on from CGI to things like Catalyst::Engine::* and HTTP::Engine, but there’s still duplicated effort everywhere. We can do better.

So he sits down and writes PSGI, an absurdly simple, manifestly beautiful specification for an interface between Perl web apps and web servers (drawing heavily on WSGI and Rack for inspiration).

PSGI

The idea goes like this: Web applications, when all is said and done, are really just on about sending three pieces of information to web browsers: a HTTP status code, a list of HTTP headers, and some content (a file or some text, normally HTML).

This is the specification:

[                                           # an array ref, containing..
    200,                                    # a HTTP status code
    [ 'Content-Type' => 'text/html', .. ],  # an array of HTTP headers
    [ '...' ],                 # the text content (or a $filehandle)
]

And that’s really all there is to it.

Some people write web apps according to the interface. Other people write server backends according to the interface. Web app developers use whatever whiz-bang technology they like, and as long as they return an array that complies with the spec their web app can run on any server. And server backend developers can do lots of clever things to get that information back to web browsers really fast, and have the results of their work benefit all PSGI/Plack consumers. No more duplicated effort. No more server-specific love lock-in.

And folks took notice. Not just any folks either. Really, really smart people like Yuval Kogman, Stevan Little, Shawn Moore, Matt Trout, Jesse Vincent, Chia-liang Kao, Dave Rolsky, Simon Cozens, .. (They’re some of the well-known names that jumped out at me from the PSGI.pod spec doc. The others are probably even smarter, stealth hackers. I know, it’s scary).

Plack

Before you could say Plack there was a reference implementation, server backend support for CGI, FastCGI, mod_perl (welcome back!), AnyEvent, Coro, Perlbal, Nginx, .. framework support for Catalyst, CGI-Application, Mason, Continuity, Maypole, Mojo .. and a whole suite of Middleware and Utilities.

Meaning that all of a sudden any web app written in one of those frameworks can now be deployed on any of those servers. Or on the pure-perl standalone server that runs from the command line. Or on one of the more experimental/exotic servers I haven’t listed. If you’re not excited yet, just wait until Google AppEngine appears in that list.

WebGUI

But where is WebGUI! The problem is that WebGUI is both a framework and a web app. A really big web app. With a mod_perl addiction. Frameworks are built with multiple servers in mind, so they generally already have an in-built server abstraction layer. Which makes adding PSGI support relatively simple. WebGUI, on the other hand, deliberately eschews an abstraction layer so that it can fully embrace mod_perl and eke out every last ounce of power and performance it can from Apache.

So, faced with extreme framework envy, I did what any reasonable person would do.

I built a PSGI/Plack layer for WebGUI.

PlebGUI

I’ve codenamed the project “PlebGUI“, which I think aptly describes the way it makes it possible for the little people to run WebGUI on low-cost shared hosting.

And it actually works. Take for instance plebgui.patspam.com, a demo PlebGUI site site running in FastCGI mode on HostMonster (the prototypical low-cost shared webhost).

app.psgi

The second wonderously simple idea in the PSGI spec is that a web app is just a plain old perl subroutine. Here’s one I prepared earlier:


sub { [ 200, [ 'Content-Type' => 'text/html' ], [ 'Hello World' ] ] }

I know, it’s almost insulting. I’m a web developer man! I do sophisticated things! But try putting that single-line sub into a test file called app.psgi. And then after you’ve installed Plack (it’s not on CPAN yet so you have to install it from miyagawa++’s git repo) try running this:


$ plackup
Accepting connections at http://0:8080/

Go on, visit that url in your web browser. Hello to you too!

Middleware

Ok that was a cute trick, let’s try something more exciting:


use Plack::Builder;
builder {
    add "Plack::Middleware::Static", path => qr/./, root => '/var/www';
    sub { [ 404, [ "Content-Type" => "text/plain" ], [ "Not Found" ] ] };
};

Congratulations, you just added your first Middleware. Assuming you have some static files located at /var/www, you’ll get the static files returned to your browser with the correct mimetype (thanks to Plack::Middleware::Static). Middleware just wraps your web app (a plain old Perl sub) with another plain old Perl sub. Middleware can do logging, pretty HTML stack traces, pre/post processing, or anything else you like. Simple, but immensely powerful.

Plackup

Plackup is a simple utility script that launches your web app with a specified server backend. By default it runs the pure-perl standalone development server. It expects your webapp to live in a file, similar to the ones we just created. Want to run your web app on another server backend? Try one of these:


plackup                                         # dev server with StackTrace and AccessLog enabled
plackup -s CGI                                  # remember how slow web apps used to be?
plackup -s AnyEvent                             # nonblocking
plackup -s Coro --port 9090                     # coroutines
plackup -s Standalone::Prefork --max-workers 20 # blazingly fast preforking ftw!

dev.localhost.localdomain.psgi

Here’s what the current per-site .psgi file looks like for PlebGUI:


use Plack::Builder;
use lib '/data/WebGUI/lib';
use WebGUI;

builder {

 # Populate $env from site.conf
 add 'Plack::Middleware::WebGUI',
   root => '/data/WebGUI',
   config => 'dev.localhost.localdomain.conf';

 # Handle /extras via Plack::Middleware::Static
 add 'Plack::Middleware::Static',
   path => qr{^/extras/},
   root => '/data/WebGUI/www';

 # Handle /uploads via Plack::Middleware::WGAccess (including .wgaccess)
 add 'Plack::Middleware::WGAccess',
   path     => qr{^/uploads/},
   root => '/data/domains/dev.localhost.localdomain/public';

 sub { WebGUI::handle_psgi(shift) };
}

What you can see there are 3 Middleware layers added in, one to set up the WebGUI site-specific environment, one to handle /extras static content, and one to handle /uploads static content (taking into account .wgaccess file permissions).

All of those plackup command variations above can be used to launch WebGUI outside of mod_perl. Prefer running inside of Apache? How about one of these:



 PerlOptions +Parent
 PerlSwitches -I/data/WebGUI/lib

 # CGI
 #AddHandler cgi-script cgi
 #ScriptAlias / /data/WebGUI/etc/dev.localhost.localdomain.cgi/
 #
 #   Options +ExecCGI
 #

 # mod_perl
 #SetHandler perl-script
 #PerlHandler Plack::Server::Apache2
 #PerlSetVar psgi_app /data/WebGUI/etc/dev.localhost.localdomain.psgi

 # FastCGI
 FastCgiServer /data/WebGUI/etc/dev.localhost.localdomain.fcgi
 ScriptAlias / /data/WebGUI/etc/dev.localhost.localdomain.fcgi/

 # mod_psgi
 #
 #    SetHandler psgi
 #    PSGIApp /data/WebGUI/etc/dev.localhost.localdomain.psgi
 #

Using those directives you can run WebGUI in CGI, mod_perl, FastCGI mode, or even the in-development mod_psgi Apache module.

Benchmarks

Ok so how fast are these different backends? Let’s use ApacheBench to do some simple, unscientific tests of how many requests per second we can squeeze out of WebGUI.

First we’ll start with WebGUI in its original, un-plebified form, running on the WRE (more is better):

$ ab -n 1000 -c 10 -k http://dev.localhost.localdomain:8081/ | grep ‘Requests per’
Requests per second: 122.77 [#/sec] (mean)

The result is of course completely dependent on your Apache configuration – in this case I have (StartServers, MinSpareServers, MaxSpareServers, MaxClients) = (5,5,10,20).

Ok, now have a look at these numbers:

$ ./ab.pl –app /data/WebGUI/etc/dev.localhost.localdomain.psgi
Testing implementations: AnyEvent, Standalone, Standalone::Prefork, ServerSimple, Coro
app: /data/WebGUI/etc/dev.localhost.localdomain.psgi
ab: ab -n 1000 -c 10 -k
URL: http://127.0.0.1/

– server: AnyEvent
Accepting requests at http://0.0.0.0:10001/
Requests per second: 68.06 [#/sec] (mean)

– server: Standalone
Accepting connections at http://0:10001/
Requests per second: 64.92 [#/sec] (mean)

– server: Standalone::Prefork
Accepting connections at http://0:10001/
Requests per second: 214.54 [#/sec] (mean)

– server: ServerSimple
Plack::Server::ServerSimple: You can connect to your server at http://localhost:10001/
Requests per second: 66.43 [#/sec] (mean)

– server: Coro
2009/10/11-23:17:32 Plack::Server::Coro::Server (type Net::Server::Coro) starting! pid(1581)
Requests per second: 67.55 [#/sec] (mean)

Did you see that? Standalone::Prefork is almost twice as fast as WebGUI in the WRE! The pre-forking server’s max-workers setting defaults to 10, so the comparison might actually be fair too.

Holy Shit

I think that’s worth repeating. PlebGUI, which is currently less than one week old and running on a compatibility layer optimised for “let’s just get this thing working and worry about performance later” is already capable of out-performing the WRE in terms of raw speed by 200%. (If you believe the benchmarks).

But Plack/PSGI is not just about speed. It’s about flexibility. Think Koen de Jonge, WebGUI hosting extraordinaire at Procolix, deploying a high availability “follow the sun” WebGUI cluster on Nginx servers. Think Colin Kuskie, WebGUI test suite overlord, probing the dark corners of the WebGUI API using standard HTTP::Request and HTTP::Response pairs through mocked HTTP and live HTTP servers. Think web developers working on WebGUI client code, with a web server fully integrated into Padre. Think WebGUI entirely deployable from the CPAN.

Think of every new project added to the PSGI/Plack ecosystem as a potential new PlebGUI feature.

Where to Now

The approach I took in turning WebGUI into PlebGUI was to create a fake Apache2::Request object, since that’s the closest thing WebGUI has to a server abstraction layer. Plack contains two helper classes Plack::Request and Plack::Response that make this really easy. Currently though, that leaves us in the curious situation where WebGUI does all of its work thinking it’s talking to mod_perl, only to have its real output re-routed through the PSGI-compatibility layer, to be subsequently processed by a specific server backend. If you don’t get the joke yet, just think about what happens when the PSGI server backend happens to be mod_perl.

The benchmarks would clearly improve if we ripped out mod_perl altogether. Lots of code would simplify too. Certain parts of WebGUI could disappear altogether, such as URL Handlers which could be entirely replaced with Middleware.

The only feature that PlebGUI currently lacks is content streaming. I deliberately left that out since it will be a lot easier to achieve once mod_perl disappears. In Plack land the way to drip-feed browsers with “chunked” content is to return an IO::Handle-like object that responds to getline() and close(). The Plack folks are planning a fancy new module called IO::Writer that will Do The Right Thing under both blocking and non-blocking servers. Expect awesome things.

Padre::Plugin::WebGUI Tab Icons

Patrick — Thu, 01 Oct 2009 09:49:25 +0000

Version 0.03 of Padre::Plugin::WebGUI (my WebGUI plugin for the Padre Perl IDE) is on its way to the CPAN, with one small but rather nice feature: tab icons. Just as the plugin uses WebGUI’s native icons to display the Asset Tree for your site, you now get the same attractive icon in the tab that opens up when you double-click to edit something. Apart from the aesthetics, it’s handy to know at a glance what sort of asset you’re editing (WebGUI uses the same trick for the edit icons on your site when you turn admin mode on). I was hoping to be able to add hover-text too (in case you don’t recognise the Asset type from its icon) but thus far I haven’t figured out if wxPerl and/or Padre supports it.

The following screenshot shows a nice example of several tabs open at once with different WebGUI Asset types displaying their own icons.

Padre::Plugin::WebGUI now with remote editing

Patrick — Sat, 26 Sep 2009 14:26:20 +0000

Up until now the WebGUI Padre plugin I’ve been working on has been more or less a proof-of-concept / toy / test-bed for experimental ideas. Whilst that has been fun (and made for good conference slides), the reality is that I haven’t actually been using the plugin for my daily WebGUI development, so how could I expect other WebGUI developers to do so?

I’m hoping to change that. As of today I’ve stripped out all the experimental features and concentrated on the single feature that seems to have the widest possible appeal, namely, remote editing.

Version 0.02 of Padre::Plugin::WebGUI, which is on its way to the CPAN, allows you to connect over HTTP to your webgui site, view you asset tree and edit your assets from inside the Padre text editor. That hopefully makes it as useful for designers and content managers as it is for developers.

Previously the plugin got most of its power from the WGDev library. However this meant that you had to install WebGUI and WGDev as a prerequisite. Given that the WRE ships with non-threaded perl, this meant either installing these prereqs into your system perl or building another perl specifically to run them. Not an overly arduous task, but enough of a hurdle to dissuade most WebGUI developers from playing with the plugin. That obstacle has now been removed. You do not need to have WebGUI or WGDev installed to run the plugin.

Using WGDev also meant that you could only connect locally to your site. That restriction is now gone.

Installation

Firstly, install Padre.

To allow your WebGUI site to talk to the plugin, add this content handler (WebGUI::Content::Padre) to your site and restart modperl (the easiest way to do this is to drop the module into your /data/WebGUI/lib/WebGUI/Content dir and add the new content handler to your site config file. There is also an install script that will do the site config editing for you).

Next install the plugin from the CPAN


cpan Padre::Plugin::WebGUI

Usage

Inside Padre, go to the Plugin Manager and enable the WebGUI plugin. You should then see a WebGUI item appear in your Plugins menu.

From the WebGUI menu, tick “Show Asset Tree” and double-click on “Connect” in the new panel that appears.

You will be prompted for the URL of the site you want to connect to. Currently you need to enter you the site in the form: http://username:password@site.com (later we will add a site manager to make this nicer). Remember that this can be any webgui site that you have access to, not just a local site.

You will then see the Asset Tree dynamically populated with information from your site.

Try double-clicking on an article.

A new file tab should open with the name set to the name of your article (with [wg] prepended). The article content will be loaded into to tab, with Padre’s HTML syntax highlighting in effect. If you have Padre::Plugin::HTML installed, you can use the Tidy/Lint/Validate the HTML to your heart’s content. And every time you hit Save, your changes will be pushed up to the site as a new Version Tag.

Try double-clicking on another sort of asset, for instance a Template. Once again, the template html will be loaded into a new tab with nice syntax highlighting.

And for fun, try opening up a CSS Snippet. The plugin will detect that your snippet is set to the ‘text/css’ mime type and use CSS code highlighting instead of HTML.

And similarly for JavaScript Snippets.

Design and Extensions

As far as Padre is concerned the file you are editing is just a regular file on your file system. For example when you edit the text, an asterisk (*) appears next to the name of the file indicating that you haven’t save it yet. The integration between the webgui content handler and the plugin is minimal, meaning that you could conceivably use this plugin to do remote editing on non-webgui sites too.

Padre has a nice architectural design, which makes it really easy for us to associate different WebGUI::Asset types with Padre::Document types. Currently most assets delegate to the default Padre::Document::WebGUI::Asset module which does HTML syntax highlighting and expects only generic content from the content handler running on your site. Padre::Document::WebGUI::Snippet has a slightly higher IQ level in that it asks the server to tell it what mimetype the asset is set to, so that it can use the appropriate syntax highlighter.

We could do some really interesting things by building asset-specific Padre::Document modules. For instance you could build a document type that uses fancy wxWidgets controls to do things that aren’t possible on the web (desktop integration etc..) And it’s just as easy to build these for your own custom asset types as it is for core ones. Given that Padre is cross-platform, you’re basically looking at a way of extending WebGUI interactions out into desktop-land.

If you’re a WebGUI designer/developer/content manager and you want to start playing with Padre::Plugin::WebGUI as a novel way of editing content, come find me on #webgui or #padre.

Finding missing WebGUI Templates

Patrick — Thu, 30 Jul 2009 09:06:59 +0000

Following on from my previous WGDev post, here’s another example of wgd ls that lets you find missing Templates on a WebGUI site.

One problem that content managers run into is that after deleting a template in WebGUI, any assets that still refer to the template will die a horrible death when users try to view them. What would be nice is an automated way of finding all such assets that refer to missing template, so that you can update them to refer to a different template.

For starters, WGDev can give you a list of all available templates on your site via:


wgd ls root -r --format %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq > available

You can also ask for a list of all in-use templateIds, via:


wgd ls root -r --format %templateId% --filter '%templateId% ~~ /.+/' | sort | uniq > in_use

All you need to do then is ask the comm command to tell you which templateIds are in_use but not available:


comm -23 in_use available

Putting it all together on one line (without the temporary files), that becomes:


comm -23 <(wgd ls root -rf %templateId% --filter '%templateId% ~~ /.+/' | sort | uniq) <(wgd ls root -rf %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq)

The output of this command is a nice list of missing templateIds.

Of course, what you probably then want to do is ask WGDev to give you a list of asset urls that are trying to use these missing Templates, for example, if one item in your list is eHMsLCMqoXYpYrsZ7CWUtg, you would do:


wgd ls root -rf %url% --filter '%templateId% ~~ eHMsLCMqoXYpYrsZ7CWUtg'

Which would print something like


/my/broken/asset/url

Combining this with the previous command to do it all on one line is left as an exercise for the reader

Now, templateId isn’t the only Asset field that templates can be referenced under, for example, if you want to also find missing Style and Printable Style templates, you’d want to do something like:


for tmpl in templateId styleTemplateId printableStyleTemplateId
do
comm -23 <(wgd ls root -rf %${tmpl}% --filter "%${tmpl}% ~~ /.+/" | sort | uniq) <(wgd ls root -rf %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq)
done

It’s not optimised for speed, but it sure is handy. And you could apply the same approach to search for invalid group permissions, etc.. Of course if you found yourself doing this often you’re probably better off writing a dedicated WGDev plugin to wrap up the functionality.

There’s a WebGUI branch on GitHub containing some extra Template Usage-related functions – the plan is to add the ability to find all the different ways that templates are referenced (in Asset definitions, Account module definitions, …) so perhaps we will end up adding a “find missing Templates” button to the Edit Template page.

[ perl ironman ]

Ode to wgd and the command line

Patrick — Tue, 14 Jul 2009 08:45:41 +0000

Doug just posted an ode to the Linux command line over at the Plain Black staff blog.

I’ve been getting a lot of joy out of the command line today too thanks to Graham’s ridiculously useful WGDev. One wgd command that I only started using in earnest today is the ls command. In its most basic form, it lets you print out a list of assets on your site:


$ wgd ls /home

getting_started
your_next_step
the_latest_news
tell_a_friend
documentation
site_map

I added a bunch of extensions to ls today, the first one being a -r option that tells ls to recursively list assets. Given that this prints to STDOUT, you can pipe this into any other command line tool to manipulate the output. For example, to list all assets that have the word “you” in their menu title:


$ wgd ls /root -f "%menuTitle%" -r | grep -i you
Profile Account Layout
Inbox Account Layout
Friends Layout Template
Account Layout
Shop Account Layout
Contributions Layout
Default Account Layout
Layout
Your Next Step

(the -f “%menuTitle%” option above tells the command to output only the menuTitle property of each asset).

Under the hood ls is calling asset->getLineage(), so I exposed a few of the getLineage API options to ls.

For example, this is one way of getting a sorted list of Survey templates on your site:


$ wgd ls root -r --includeOnlyClass WebGUI::Asset::Template -f "%url% %namespace%" | perl -ane 'print "$F[0] ($F[1])\n" if $F[1] =~ m/survey/i' | sort
root/import/expireincompletesurveyresponses/expireincompletesurveyresponses (ExpireIncompleteSurveyResponses)
root/import/survey/default-answer-edit (Survey/Edit)
root/import/survey/default-feedback (Survey/Feedback)
root/import/survey/default-gradebook-report (Survey/Gradebook)
root/import/survey/default-overview-report (Survey/Overview)
root/import/survey/default-question-edit (Survey/Edit)
root/import/survey/default-questions (Survey/Take)
root/import/survey/default-section-edit (Survey/Edit)
root/import/survey/default-survey-edit (Survey/Edit)
root/import/survey/default-survey-summary (Survey/Summary)
root/import/survey/default-survey (Survey)
root/import/survey/default-survey-take (Survey/Take)
root/import/survey/default-test-results (Survey/TestResults)

While I was writing this post, Andy asked me to provide him with a list of all Carousel wobjects on a site we’re working on. That’s as easy as:


$ wgd ls -r root --includeOnlyClass WebGUI::Asset::Wobject::Carousel

The most useful option I added to ls is the –filter smartmatch filter option. This option filters the results using an asset property of your choosing. Format looks like “%url% ~~ smartmatch”, where “url” is the field to filter against, and “smartmatch” is either a Perl regular expression such as “/(?i:partial_match)/” or a string such as “my_exact_match”.

For example, the following command lists all templates that include absolute urls. We generate the output to an html file so that someone can easily open all the links and manually review/remove the absolute urls:


$ wgd ls -r /root --filter "%template% ~~ /http:///" --includeOnlyClass WebGUI::Asset::Template --format "%url%
" > links.html

links.html then contains:

style_01

style_02

style_03


...

One type of command that I can see myself using a lot in the future is the following, which displays all assets that are viewable to a specific group (in this case the “Turn Admin On” group, which has an id of “12″):


$ wgd ls -r root --format "%groupIdView% %url%" --filter "%groupIdView% ~~ 12"
12 root/import/newsletter
12 root/import/projectmanager/resource
12 root/import/wiki

Alternatively, you can use a variation of the above to list all assets that are NOT viewable to a certain group (or groups). This is great for finding assets that have accidentally been set to say, Admins, when you want them viewable by Registered Users. For example, the following command lists assets that are not viewable to the default “Visitors”, “Everyone”, or “Registered Users” groups:


$ wgd ls -r root --format "%groupIdView% %url%" | perl -ane 'print "$F[0] $F[1]\n" if !grep {$F[0] eq $_} qw(1 2 7)'
12 root/import/newsletter
12 root/import/projectmanager/resource
12 root/import/wiki

All of these commands take in the order of 1 second to run, which is a lot less time consuming than manually reviewing your site asset-by-asset. And if you incorporate them into your test suite, you can re-verify their output against your site policy/expectations whenever you like.

Padre::Plugin::WebGUI

Patrick — Fri, 03 Jul 2009 08:42:06 +0000

For a little while now I’ve been working on a WebGUI plugin for Padre (the Perl Application Development and Refactoring Environment). Progress has been pretty slow, partly due to lack of time but also because often the things I want to do are at the edge of what Padre currently supports (or just that there’s no existing examples of how to do something easily).

I’ve released the code onto CPAN, and the current dev version is available via github, but since it’s all a bit hacky and alpha at the moment I thought it worthwhile doing a little tour of what currently works (on my dev box at least).

Installation

Installation is the same as any other Padre plugin, either install via cpan


cpan install Padre::Plugin::WebGUI

or from git:


git clone git://github.com/pdonelan/Padre-Plugin-WebGUI.git

perl Makefile.PL

make && make test && make install

Since this is a WebGUI plugin, you need to first install WebGUI, and also WGDev.

Once you’ve done that, you will see the WebGUI plugin listed in the Padre Plugin Manager:

Click the enable button, and you’ll see a WebGUI item appear in the Plugin menu.

Controlling WRE Services

The first thing you can do with the plugin is control your WRE services (mysql, modperl, modproxy and spectre).

Click on: WebGUI > WRE Services > Modperl > Restart and you’ll see a restart status message appear in the Padre output panel.

Currently this is just a simple wrapper around the wreservice.pl script that ships with WebGUI. A nice enhancement would be to turn the menu item into a services dashboard with lights telling you which services are currently running, and buttons to start/stop/restart.

Online Resources

Another item in the menu called Online Resources gives you access to a bunch of handy WebGUI links that open in your browser – including one that takes you straight to a web-based IRC client connected to #webgui.

WebGUI Log

If you click the “Logview” checkbox on the WebGUI menu, a WebGUI Log panel will open. This panel will start monitoring all of your WRE log files (webgui.log, modperl.log, modproxy.log, etc..) and display changes in real-time. WebGUI uses Log4perl as its logging engine, but it uses the sitename as the Log4perl category, which means that you can’t easily e.g. turn on DEBUG output and filter out everything except the module you’re working on (unless you use my Log4perl::Filter::CallerMatch module). Once we add in the ability to easily customise what files are being logged and apply filters to what gets displayed, this will make the above scenario a breeze.

The logview panel uses a long-running Padre task and File::Tail::select to continuous monitor a bunch of files in realtime. The main thread writes commands to a temp file to send commands to the monitoring thread (the most important command being “exit” which tells the thread to break out of its monitoring loop!). Later I plan on adding support for other commands such as “add/remove this file from the list of files you’re watching”, “add/remove filter”, etc.. Since this seems like a generic sort of feature I’m hoping to bundle Padre::Plugin::WebGUI::Logview and Padre::Plugin::WebGUI::Task::Logview as a stand-alone module for other Padre plugin developers to use (and improve).

Asset Tree

And finally, the coolect feature the WebGUI plugin has is the Asset Tree.

If you turn on the Asset Tree from the plugin menu, a new panel appears with a “Connect” button on it. Currently this is hardwired to the default WebGUI dev site at dev.localhost.localdomain but later I plan on adding a Site Manager to allow you to define sites here.

Double-clicking on the Connect button connects to the site, and pulls down the asset tree:

This is the same Asset Tree you see when you log in to your WebGUI site and turn admin mode on. As you can see from the screenshot, right-clicking on a node brings up a context menu, from which you can do asset-specific actions. The plugin already gives you access to all of the WGDev commands from the main plugin menu (with a dialog box asking you what arguments you want to pass to the command), but it becomes a lot more useful once you hook these WGDev commands to the Asset Tree context menu. Then instead of having to remember what url or assetId your asset has, you can just click through the tree to find the node you want, right-click and off you go.

For example, WGDev has a package command that lets you package up node and all of its children into a tarball that can be imported into other sites, used in upgrade scripts, etc.. By integrating this command into the context menu, you can have Padre prompt you you for the folder that you want to save the WebGUI package in (via your native Save File dialog box).

Similarly, WGDev has an edit command that lets you edit an asset (for example a template file) via your favourite editor and then have the changes pushed back into the database after you save and close the file. By integrating this command into Padre, you could leave the file open, and push the changes up every time you hit save(allowing for more convenient continuous editing).

Survey vs SurveyMonkey

Patrick — Mon, 04 May 2009 09:35:09 +0000

For my first post in the Enlightened Perl “Iron Man” get-more-people-to blog-outside-of-the-Perl-echo-chamber initiative, I thought it’d be fun to do a follow-up to my recent WebGUI Survey Goodies post, comparing our humble Perl powered Survey tool to the industry giant: SurveyMonkey.

SurveyMonkey

According to their official website, SurveyMonkey is “the leading survey tool on the web (ranked by Alexa), with over 80% of the Fortune 100 currently using SurveyMonkey”. Impressive stuff! They must have an awesome tool. Let’s see how Survey stacks up. I’m comparing Survey to the free version of SurveyMonkey, so if someone has a paid account and wants to let us know what extra features we’re missing out on that’d be great.

Let’s start with the SurveyMonkey features page.

Design

Select From Over 20 Types of Questions – We support everything from multiple choice to rating scales to open-ended text. You can customize the layout of every question type for the ultimate in design flexibility.

20 question types sounds impressive. I don’t think we have that many. Looking closer though, their list actually only numbers 15 in total, and some of them are redundant items like “Multiple Choice (only one answer)”, “Multiple Choice (Multiple Answers)”. Ok, first win to SurveyMonkey! – we don’t have a drop-down list question type. But that’s only because no-one has asked for one yet – Kaleb could probably commit one faster than it took me to write this blog post. Digging deeper, most of the other question types could be recreated by playing with Survey’s per-question/answer settings such as horizontal/vertical display, etc.. Preconfigured bundles of questions are an interesting concept.. Survey has the concept of a multi-choice question bundle, which is a question plus all of its answers – these can be copied, and also saved so that you can deploy them at will anywhere in your survey, but it might be interesting to apply that concept to the Section level (SurveyMonkey doesn’t let you create your own bundles by the way).

Missing from SurveyMonkey’s arsenal are “Date Range”, draggable sliders (and dual-sliders), and a calendar-style date picker. So let’s call it even for now.

Use Our Survey Templates, Create custom themes

Yup, we’re fully templated, using the same template system (HTML::Template) that powers the rest of WebGUI. There appears to be some skinning you can do in SurveyMonkey, but the “Survey Template Library” is only listed for the pro version on their Pricing Page. I think we’re edging ahead at this point.

Add a Logo to Your Survey

Um..?? Well let’s see.. You can run Survey on your own domain, within a WebGUI site, however you please. And yes, that includes putting your logo wherever you like. (Looks like the logo feature is disabled in the free version of SurveyMonkey. bummer.).

Validate Your Survey Responses

SurveyMonkey has some nice per-question validation options that go beyond what Survey is currently capable of. They definitely beat us on this front. However that will change when we make the planned switch to inputEx for rendering html form elements – inputEx has a huge array of sexy looking form elements that can be easily wired up with validation logic.

Randomize or Sort Your Question Choices

Can do.

Save Your Survey as a PDF

Nice! That’d be a great feature to have. No doubt a solution is only one CPAN module away, but right now SurveyMonkey has it and we don’t.

Collection Features

I won’t list all the features here but they mostly pertain to controlling who can access your questionnaires. Survey runs inside of WebGUI, so you have the full power of WebGUI Authentication/Authorization at your disposal. You have fine-grained control over who can edit surveys, take surveys and view reports. Want to use LDAP users? Easy. WebGUI Auth is fully pluggable too, so there’s nothing stopping you from integrating your site with whatever other infrastructure you already have. Survey also lets you set time windows on how long users have to complete a survey, which can be useful if data validity is important to you. I’m confident we win hands down here, although of course you need to be competent enough to set it up.

Analysis Features

I think SurveyMonkey has more web-based features here, since our current focus is to do only basic online reporting and let users export their data to something more powerful like SPSS for proper statistical analysis. There’s definitely a lot we can improve on here, although we’re completely driven by the features that end-users want, so only time will tell what we decide to implement on this front.

OK, my turn

Off the top of my head, here are some features that Survey has but appear to be missing from the free version of SurveyMonkey:

Jump/Branching expression. The front page mentions “control the flow with custom skip logic”. Presumably that’s similar to our Jump Expressions (see my previous post). I’d love to do a detailed comparison of the relative power of their expression engine vs ours, but sadly it’s only available in the paid version. Ours gives you a very large subset of the Perl language to play with, so we’re probably winning here.
I hardly consider this a “feature”, but it looks like the free version of SurveyMonkey only lets you create 10 questions per survey, and you can only have 100 responses. And the free version doesn’t let you export your data to a spreadsheet program. Ouch!
Scoring, and branching based on scores
No sign of quiz mode with tabular and graphical feedback for the user to let them know how they scored (see my previous post)
Visualisation (see my previous post)
Drag and drop section/question/answer re-ordering. I can’t imagine see myself being very productive if I was trying to work on the 500+ question survey I’m building inside of the menu-driven SurveyMonkey interface.
The ability to modify the survey engine to do whatever custom magic you like
Complete control over your data ffs!This is the biggie as far as I’m concerned. Surveys built with SurveyMonkey store their data on the SurveyMonkey servers. Australia’s privacy laws are very strict about sending private data overseas, and I assume the same is true for other countries. You could land yourself in a world of trouble if you created supposedly “private” surveys where people were asked to enter personally identifying information (particularly if the data was medical related) and that data was stored overseas. You can pay to add SSL to your SurveyMonkey account, but there is no mention of database encryption, which is also something to ponder. At the end of the day, if you’re doing anything remotely private you should be self-hosting the data.
Have access to the code, which, among other things, means that you can build a comprehensive test suite for your complex 500+ question survey. For any complex Survey involving non-trivial branching logic, it’s invaluable to be able to build an automated test suite that independently verifies your branching and scoring logic (and makes sure that any future changes don’t break things). In Survey you can write this in Perl at the code level, interacting with Survey and the database via the WebGUI API, or using any programming language to create an automated “virtual user” that interacts with the Survey via the web interface (simulating page clicks etc..).

So…

Overall that was a bit disappointing. I mean, there’s are definitely areas we can improve on, and I’m sure that the SurveyMonkey documentation and support is great, but for something that purports to be the best of breed, … well let’s just that that as far as the free version goes you’re pretty limited in what you can do. No doubt the paid version is better, and of course most people out there just want something simple that they can jump right in and start using with minimal fuss. But it seems to me that pretty soon you start running into limitations, especially with the free version.

Ok, enough of that, let’s get back to what’s new in Survey. Here are a couple of new features that have been added in the past week:

Firstly, Kaleb committed some code a few days ago that allows you to flag Sections as “logical”. What this means is that the section is hidden from the user, but appears in the admin screen and can have branching expressions defined for it. This is really handy if you have sophisticated branching logic and you run into a situation where you need an intermediate jump target. For example, it’s a common scenario to have a bunch of “pre-section” rules that you want to evaluate before letting the user start the section. Rather than duplicating these rules on every other node that jumps to the section, you can have those nodes jump to a pre-section node that evaluates the rules in one place.

Another awesome feature that was recently committed to the Survey repo (and that won’t cost you a banana) is tagging. The Survey expression engine now allows you to tag data as you go, which allows you to do all sorts of interesting things. For instance, say you’re doing one of those magazine pop-quizzes where you (very scientifically) classify people into personality types. After asking a bunch of dog-related questions, you can now say something like this in your branch expression:

tag(“personality type”, “dog lover”) if score(“dog questions section”) > 5

That creates a new data field called “personality type” that is stored against the user’s response (in this case the stored data is the string “dog lover”). Later, you can actually use that tag inside another branch expression to only show a bunch of questions to the dog lovers:

jump { tag(“personality type”) eq “dog lover” } “more dog questions section”

Or you could use that tag inside the text of another question as templated text:

Clearly you’re a [[personality type]]!

..which would appear to the user as:

Clearly you’re a dog lover!

And finally, the tagged data is stored as part of the user’s response data, meaning that you can pull it out of the database later (using another WebGUI asset like SQLReport or your own custom code) and do whatever you like with it.

WebGUI Survey goodies

Patrick — Thu, 09 Apr 2009 08:12:20 +0000

Kaleb (aka perlmonkey2) and I have been rounding out WebGUI‘s Survey wobject with a bunch of features over the past couple of weeks. Here’s a run-down of some of the highlights:

Improved Visual Feedback

The Survey Edit page now uses “Loading…” progress masks whenever it’s busy doing something. When you’re dealing with large data sets (and slow servers) it’s really handy to be able to see when the page is busy doing loading/saving/etc..

Multi-choice Bundles

Survey ships with a bunch of useful multiple-choice question bundles to get you going. Some of these are very generic such as Gender (Male/Female), Yes/No, etc.. Others are more specific to single domain, such as Confidence (an 11 answer scale from “0 – Not at all confident” to “10 – Extremely confident”). Of course, every site is going to want a different set of re-usable bundles. You can now create your own bundles, and add/edit/delete the default ones.

Validation Warnings

Now that Survey is a complex beast, allowing you to do all sorts of fancy things, it pays to have some automated checking of various things that could go wrong. The Edit Survey page constantly checks your Survey instance for warnings, and alerts you if anything smells bad.. everything from using Jump Expressions when enableSurveyExpressionEngine is disabled for your site, to duplicate Section/Question variable names, circular Jump targets, all the way up to full-blown Expression Engine compilation & pseudo-execution checking.

Survey Summaries

This is a long-awaited feature curtesy of Kaleb. You can now define which answers are correct, and assign arbitraty numeric scores to them. Then, when the user reaches the end of the Survey, they get shown their quiz results – both numerically (in a YUI DataTable) and graphically (with YUI Charts).

Coming soon to this department will be the ability to display quiz summaries at the end of individual sections rather than just all at the end.

Visualisation

Meanwhile, I’ve been building a tool to allow you to visualise the branching structure of a Survey using GraphViz. Here is an example of a real-world Survey instance that has ~500 questions. Turning this into a button that allows you to generate visualisations on the fly from the Edit Survey page on your own site is just a matter of sorting out the last remaining issues with GraphViz/ImageMagick support in the wre.

Branch Expressions

This has been the main focus of my work, and probably deserves a post all to itself. Survey now has a dedicated expression engine, which allows you to control branching using a large sub-set of the Perl language. Expressions are evaluated using Safe module, which limits what opcodes the expression can generate and limits access to only the variables that we chose to expose. Of course, even Safe.pm has caveats about what it can and can’t protect you from (for example infinite loops), so it’s important to note that the expression engine is disabled by default, and can only be enabled by site administrators in the WebGUI site config file.

When it comes down to it, branch expressions are just perl code, so you have a whole lot of power at your disposal. You can access recorded responses for the user by supplying the Section/Question variable name to the value() sub:


value(variable_name);

To actually trigger a branch, you call the jump() sub:


jump { value(q1) == 1 } section3

That says, if the user’s recorded response value to q1 has a numeric value of 1, jump to section 3.

Here’s a more advanced one:


jump { score(q1) > 5 and value(q2) =~ m/textmatch/ } section3;
jump { avg(value(q1), value(q2), value(home/anotherSurvey, q3)) > 10 } section4;

The above expression demonstrates some of the cooler branch expression features:

Two alternative jump clauses
Branching based of Question score (if you ask for score(Section) you get the total score for that section)
Text matching (you have the power of Perl at your disposal after all)
Calling a utility function, in this case avg(). The expression engine makes available a bunch of handy utility functions, such as all of the subs from List::Util (min, max, sum, etc..)
Resolving the recorded response value from another survey instance altogether (the first argument to value() and score() can be an assetId or an asset url.)

Behind the scenes, the ExpressionEngine is quite nice too, telling you what it’s doing along the way, which makes debugging complex expressions quite painless.

Along with all these changes, the test suite continues to grow. Check it out in the next WebGUI beta and let us know what you think.

WebGUI Carousel Wobject

Patrick — Mon, 06 Apr 2009 06:41:32 +0000

There’s a new wobject coming in the latest WebGUI beta (version 7.7.3), called Carousel. On the surface, it’s a nice little integration between WebGUI and YUI, but below the surface it’s a great multi-purpose tool for rendering content dynamically in Javascript or Flash.

It’s a common requirement to want to display rich text via Javascript or Flash. For example you might have a clickable diagram that shows different text depending on what part of the diagram a user clicks on.

Giving content managers access to the dynamic text can be rather cumbersome. In the worst case scenario the text is hard-coded in the Flash or Javascript source, out of reach of the content manager. Better solutions are where you have a bunch of editable Articles or Snippets that the Flash or Javascript reads from.

None of these solutions are very satisfactory though, because as far as the content manager is concerned the dynamic object is a single asset, and they really want a simple way to edit all of the text elements from the same place.

We designed the new Wobject specifically for this scenario. The Edit Screen of the wobject allows content managers to add as many rich text editor (RTE) boxes as they like (dynamically, a la FilePile). These RTEs are displayed one after the other on a single screen, allowing the content manager to edit all of the text in a single place. For each RTE, the content manager can also specify a unique ID.

By default, the template for the Carousel wobject renders your RTE items into an attractive YUI Carousel widget:

The Wobject template does this by looping over all of the RTEs and generating markup expected by the YUI Carousel widget. This has the nice side-effect of allowing your widget to gracefully degrade in non-js browsers.

Taking a step back, we have a WebGUI wobject that lets us manipulate a collection of RTEs. The only part of the wobject specific to the YUI Carousel is the template. Let’s have a look at it:

...

We can just as easily create our own template to do something else with the data. Returning to our original example, say you want to use the RTE text in a Flash diagram. All you have to do is change the template to output the contents of each RTE into a textarea dom element, styled to be hidden from view, with the id attribute set to the RTE unique id. Then, your Flash (or Javascript for that matter) can easily read in the rich text via DOM methods (e.g. document.getElementById(RTE_unique_id).value) and display it dynamically and/or do whatever it likes with it.

For example, we might use a template such as:



 <tmpl_var text>

The only coordination required is that the person deploying the wobject would need to know how many text blocks the flash/javascript expects, and what unique ids to use. For some flash/javascript widgets you could ignore unique ids altogether by adding a special class attribute to the generated textareas and then using a class selector to dynamically select all matching textareas on the page.

This should make it easier to use dynamic flash/javascript widgets in WebGUI whilst still retaining the power and ease of use of a content management system.