Latest Web::Simple master (git://git.shadowcat.co.uk/catagits/Web-Simple.git) is now using Plack, the perl web server directly for CGI and FastCGI and I hope do do another release shortly with notes on deploying as both on shared hosts (more specifically Dreamhost but with an invitation to bitch if they don’t work on other budget hosts).

mst,  ”Oh Subdispatch, Oh Subdispatch“

Every time someone mentions a cool new web framework like Web::Simple, Dancer or Tatsumaki, my immediate reaction is to start thinking up cool little niche web apps I could build with it. You know, the kind you imagine yourself whipping up during your lunch break, that will probably never get more than 10 curious users but just might turn into the next big thing if only you got it up and running on a public server. And that’s when the second thought immediately arrives: where am I going to run this thing? Do I really think the idea has enough legs to justify paying for a virtual server plan? Can I be bothered going through the pain of figuring out how to deploy it on a shared hosting provider? And normally that’s the point where I sigh wistfully and go back to reading my RSS feeds.

The thing is, with the advent of things like local::lib it’s getting a lot easier to deploy Perl web apps on shared hosting. And with most web frameworks adopting Plack/PSGI, the work required to deploy Perl web apps on budget hosts is converging into a similar sequence of steps.

So this time during my lunch break I started envisaging a centralised, SEO-friendly information source (hello perlsharedhosting.com) that cobbles together all of the currently available information into an easy to digest form to make it ridiculously easy to choose a shared hosting provider, deploy your web app and troubleshoot common problems.

A site that works like this:

• The front page contains a list of Perl-friendly hosting providers, with a meta-score based on how many features they support (+) and how many unresolved issues they have (-), and maybe user review scores thrown into the mix too
• Each hosting provider has a page of its own, listing in full which features are supported (used to compute the meta-score). Users can post comments on each of these pages, to leave testimonials (“I am currently running a Web::Simple site that gets x hits per hour on this host”) and note unresolved issues (“module X::Y fails to install on this host”). As issues get solved these become tips.
• Each technology/feature/technique also gets a page of its own, with links to the official man pages, shared-hosting specific notes and again user comments
  • dependency related: local::lib, cpan, cpanm, ..
  • environment related: locating and running perl, paths, daemons, viewing error output, ..
  • web deployment related: CGI, FastCGI, ..
  • framework / web app related: Web::Simple, Dancer, MojoMojo, Catalyst, Tatsumaki, ..

I haven’t decided yet what engine would fit best.. MojoMojo is a front-runner, and in fact the Catalyst Friendly Hosting page runs on MojoMojo does a large chunk of what I’ve described above.

The site would be very “cookbook” oriented, since we’re specifically targeting people who can’t be bothered learning the ins and outs of 10 different technologies (not to mention figuring out how to get them to play nicely together) for the sake of deploying toy web applications on cheap shared hosting. Just the essentials: this is what you need, this is how you achieve it. And if you want to learn more, go here.

With that in place, we’d end up with a single, visible, place to research and document the complications of running Perl web apps on shared hosting. And if the site became popular, hosting providers might even take notice and start offering more Perl-friendly shared environments. We could have live demo pages running on different hosts, possibly even donated by hosting providers if they see it as a way of showcasing their Perl-friendliness (mojomojo.dreamhost.perlsharedhosting.com, dancer.resellerzoom.perlsharedhosting.com, etc..).

I got as far as registering the domain name; I was planning on getting a simple MojoMojo prototype up and running, but I got side-tracked researching how to deploy it on my shared hosting account…

As the name suggests, Padre::Plugin::Plack adds Plack awareness to Padre. With the plugin installed, opening *.psgi files causes some special things to happen. PSGI files are really just ordinary Perl files, so Padre does its normal Perl lexing/syntax highlighting magic on them, but the real fun starts with the Plack-specific features that appear in the per-file graphical plackup control panel that shows up. The panel lets you run your web app in a Plack server at the click of a button, view server output, configure plackup options and launch a web browser on the appropriate port.

The great thing about Plack/PSGI is that unlike my previous plugin (Padre::Plugin::WebGUI) which was specific to a single web app (albeit a big one), this plugin can be used for any web app built in a web framework that supports Plack (Catalyst, CGI::Application, HTTP::Engine, etc..). This potential for cross-framework application is one of the motivating factors that makes developing Plack modules (Middleware etc..) so much fun.

The plugin turns on plackup’s “–reload” option by default, which conveniently causes the plack server to reload every time you modify your source files in Padre. This makes for quite a nice, if somewhat minimal “Plack IDE” experience (this is version 0.01 after all).

The plugin integrates all of the Plack example “dot-psgi” files as templates that can be used to create different types of Plack apps straight from the GUI menu.

The pre-populated list of Plack servers and the simple start/stop button makes for a nice way of exploring the Plack server ecosystem. You can use the other panel options to enter a specific port to run on, toggle auto-start mode and pass additional options to plackup (options that start with “–” are passed through to the backend server).

The output panel is similar to the output panel that Padre normally displays when you execute Perl files, except that you get one panel per .psgi file meaning that you can run multiple plack servers simultaneously and independently view their output. The appropriate panel is automatically selected when you click on the corresponding file tab, and running processes are stopped when you close the tab.

It should be really easy to turn Padre::Plugin::Plack into new plugins that involve the same basic ingredients, namely  a file extension and an external command for running those files, with a per-file panel for command options and output. So I encourage anyone who has a similar plugin in mind to steal liberally from Padre::Plugin::Plack (as I did from Padre::Plugin::Catalyst – thanks garu++). Ruby Rack support comes to mind as a trivial example.

Make Padre your domain-specific IDE today

Plack has a really neat solution to this problem.

Following a tip from day #5 of Miyagawa’s Plack advent calendar, serving up the current directory over http is as simple as this Perl one-liner:

$ plackup -MPlack::App::Directory -e 'Plack::App::Directory->new

Plack::Server::Standalone: Accepting connections at http://0:5000/

Or if you need the files served over port 80:

$ sudo plackup -MPlack::App::Directory -e 'Plack::App::Directory->new' -p 80

Plack::Server::Standalone: Accepting connections at http://0:80/

I got a kick out of seeing Miyagawa pull up my blog post on PlebGUI in response to someone’s question about porting mod_perl applications (e.g. apps that pass around an Apache $r object) to Plack. For the sake of the guy who asked the question, the way I did it (at least for the first pass) should be equally viable for other mod_perl applications – just create a fake Apache $r object that delegates everything to Plack::Request and Plack::Response (your mileage will obviously vary depending on how much of the mod_perl API your app uses).

But you see mod_perl is an overly zealous lover. Every intimate phase of the Apache request cycle is offered up to your eager Perl embrace. Sure, you have to learn a few new tricks to get decent performance (such as two-tier mod_proxy/mod_perl) but hey it’s 2001 and mod_perl is SO much better than CGI. Without hesitation you commit to a life of PerlResponseHandlers and Apache2::Const::OKs.

Years pass. Life is good.

Your user base had become very adept at deploying Apache. In fact you make life easier for them by distributing the complete Perl/Apache/MySQL stack as a simple installer, pre-configured for optimum performance.

Every now and then someone appears on the mailing list asking questions about WebGUI 5, a throw-back to the days when it used to be possible to deploy WebGUI on cheap shared hosting cPanel servers in CGI mode. You want to help, but WebGUI has become so powerful that CGI mode isn’t feasible anymore. That’s the price that was paid for evolving into an Enterprise-grade system. Developers lament the fact that small-time users can’t take advantage of all the awesome things WebGUI can do, not to mention all the word-of-mouth promotion WebGUI is missing out on, but them’s the brakes. WebGUI continues to grow. “Carrier-grade” is the new black. And across town, the non-enterprise crowd is left to content themselves with Wordpress, Joomla and Drupal.

And then a guy called Tatsuhiko Miyagawa comes along.

He says, gee, look at these wonderful server abstractions that Python (WSGI) and Ruby (Rack) have. The Perl world might have moved on from CGI to things like Catalyst::Engine::* and HTTP::Engine, but there’s still duplicated effort everywhere. We can do better.

So he sits down and writes PSGI, an absurdly simple, manifestly beautiful specification for an interface between Perl web apps and web servers (drawing heavily on WSGI and Rack for inspiration).

PSGI

The idea goes like this: Web applications, when all is said and done, are really just on about sending three pieces of information to web browsers: a HTTP status code, a list of HTTP headers, and some content (a file or some text, normally HTML).

This is the specification:

[                                           # an array ref, containing..
    200,                                    # a HTTP status code
    [ 'Content-Type' => 'text/html', .. ],  # an array of HTTP headers
    [ '<html>...</html>' ],                 # the text content (or a $filehandle)
]

And that’s really all there is to it.

Some people write web apps according to the interface. Other people write server backends according to the interface. Web app developers use whatever whiz-bang technology they like, and as long as they return an array that complies with the spec their web app can run on any server. And server backend developers can do lots of clever things to get that information back to web browsers really fast, and have the results of their work benefit all PSGI/Plack consumers. No more duplicated effort. No more server-specific love lock-in.

And folks took notice. Not just any folks either. Really, really smart people like Yuval Kogman, Stevan Little, Shawn Moore, Matt Trout, Jesse Vincent, Chia-liang Kao, Dave Rolsky, Simon Cozens, ..  (They’re some of the well-known names that jumped out at me from the PSGI.pod spec doc. The others are probably even smarter, stealth hackers. I know, it’s scary).

Plack

Before you could say Plack there was a reference implementation, server backend support for CGI, FastCGI, mod_perl (welcome back!), AnyEvent, Coro, Perlbal, Nginx, .. framework support for Catalyst, CGI-Application, Mason, Continuity, Maypole, Mojo .. and a whole suite of Middleware and Utilities.

Meaning that all of a sudden any web app written in one of those frameworks can now be deployed on any of those servers. Or on the pure-perl standalone server that runs from the command line. Or on one of the more experimental/exotic servers I haven’t listed. If you’re not excited yet, just wait until Google AppEngine appears in that list.

WebGUI

But where is WebGUI! The problem is that WebGUI is both a framework and a web app. A really big web app. With a mod_perl addiction. Frameworks are built with multiple servers in mind, so they generally already have an in-built server abstraction layer. Which makes adding PSGI support relatively simple. WebGUI, on the other hand, deliberately eschews an abstraction layer so that it can fully embrace mod_perl and eke out every last ounce of power and performance it can from Apache.

So, faced with extreme framework envy, I did what any reasonable person would do.

I built a PSGI/Plack layer for WebGUI.

PlebGUI

I’ve codenamed the project “PlebGUI“, which I think aptly describes the way it makes it possible for the little people to run WebGUI on low-cost shared hosting.

And it actually works. Take for instance plebgui.patspam.com, a demo PlebGUI site site running in FastCGI mode on HostMonster (the prototypical low-cost shared webhost).

app.psgi

The second wonderously simple idea in the PSGI spec is that a web app is just a plain old perl subroutine. Here’s one I prepared earlier:

sub { [ 200, [ 'Content-Type' => 'text/html' ], [ 'Hello World' ] ] }

I know, it’s almost insulting. I’m a web developer man! I do sophisticated things! But try putting that single-line sub into a test file called app.psgi. And then after you’ve installed Plack (it’s not on CPAN yet so you have to install it from miyagawa++’s git repo) try running this:

$ plackup
Accepting connections at http://0:8080/

Go on, visit that url in your web browser. Hello to you too!

Middleware

Ok that was a cute trick, let’s try something more exciting:

use Plack::Builder;
builder {
    add "Plack::Middleware::Static", path => qr/./, root => '/var/www';
    sub { [ 404, [ "Content-Type" => "text/plain" ], [ "Not Found" ] ] };
};

Congratulations, you just added your first Middleware. Assuming you have some static files located at /var/www, you’ll get the static files returned to your browser with the correct mimetype (thanks to Plack::Middleware::Static). Middleware just wraps your web app (a plain old Perl sub) with another plain old Perl sub. Middleware can do logging, pretty HTML stack traces, pre/post processing, or anything else you like. Simple, but immensely powerful.

Plackup

Plackup is a simple utility script that launches your web app with a specified server backend. By default it runs the pure-perl standalone development server. It expects your webapp to live in a file, similar to the ones we just created. Want to run your web app on another server backend? Try one of these:

plackup                                         # dev server with StackTrace and AccessLog enabled
plackup -s CGI                                  # remember how slow web apps used to be?
plackup -s AnyEvent                             # nonblocking
plackup -s Coro --port 9090                     # coroutines
plackup -s Standalone::Prefork --max-workers 20 # blazingly fast preforking ftw!

dev.localhost.localdomain.psgi

Here’s what the current per-site .psgi file looks like for PlebGUI:

use Plack::Builder;
use lib '/data/WebGUI/lib';
use WebGUI;

builder {

 # Populate $env from site.conf
 add 'Plack::Middleware::WebGUI',
   root => '/data/WebGUI',
   config => 'dev.localhost.localdomain.conf';

 # Handle /extras via Plack::Middleware::Static
 add 'Plack::Middleware::Static',
   path => qr{^/extras/},
   root => '/data/WebGUI/www';

 # Handle /uploads via Plack::Middleware::WGAccess (including .wgaccess)
 add 'Plack::Middleware::WGAccess',
   path     => qr{^/uploads/},
   root => '/data/domains/dev.localhost.localdomain/public';

 sub { WebGUI::handle_psgi(shift) };
}

What you can see there are 3 Middleware layers added in, one to set up the WebGUI site-specific environment, one to handle /extras static content, and one to handle /uploads static content (taking into account .wgaccess file permissions).

All of those plackup command variations above can be used to launch WebGUI outside of mod_perl. Prefer running inside of Apache? How about one of these:

<VirtualHost *:80>
 PerlOptions +Parent
 PerlSwitches -I/data/WebGUI/lib

 # CGI
 #AddHandler cgi-script cgi
 #ScriptAlias / /data/WebGUI/etc/dev.localhost.localdomain.cgi/
 #<Directory /data/WebGUI/etc>
 #   Options +ExecCGI
 #</Directory>

 # mod_perl
 #SetHandler perl-script
 #PerlHandler Plack::Server::Apache2
 #PerlSetVar psgi_app /data/WebGUI/etc/dev.localhost.localdomain.psgi

 # FastCGI
 FastCgiServer /data/WebGUI/etc/dev.localhost.localdomain.fcgi
 ScriptAlias / /data/WebGUI/etc/dev.localhost.localdomain.fcgi/

 # mod_psgi
 #<Location />
 #    SetHandler psgi
 #    PSGIApp /data/WebGUI/etc/dev.localhost.localdomain.psgi
 #</Location>

</VirtualHost>

Using those directives you can run WebGUI in CGI, mod_perl, FastCGI mode, or even the in-development mod_psgi Apache module.

Benchmarks

Ok so how fast are these different backends? Let’s use ApacheBench to do some simple, unscientific tests of how many requests per second we can squeeze out of WebGUI.

First we’ll start with WebGUI in its original, un-plebified form, running on the WRE (more is better):

$ ab -n 1000 -c 10 -k http://dev.localhost.localdomain:8081/ | grep ‘Requests per’
Requests per second:    122.77 [#/sec] (mean)

The result is of course completely dependent on your Apache configuration – in this case I have (StartServers, MinSpareServers, MaxSpareServers, MaxClients) = (5,5,10,20).

Ok, now have a look at these numbers:

$ ./ab.pl –app /data/WebGUI/etc/dev.localhost.localdomain.psgi
Testing implementations: AnyEvent, Standalone, Standalone::Prefork, ServerSimple, Coro
app: /data/WebGUI/etc/dev.localhost.localdomain.psgi
ab:  ab -n 1000 -c 10 -k
URL: http://127.0.0.1/

– server: AnyEvent
Accepting requests at http://0.0.0.0:10001/
Requests per second:    68.06 [#/sec] (mean)

– server: Standalone
Accepting connections at http://0:10001/
Requests per second:    64.92 [#/sec] (mean)

– server: Standalone::Prefork
Accepting connections at http://0:10001/
Requests per second:    214.54 [#/sec] (mean)

– server: ServerSimple
Plack::Server::ServerSimple: You can connect to your server at http://localhost:10001/
Requests per second:    66.43 [#/sec] (mean)

– server: Coro
2009/10/11-23:17:32 Plack::Server::Coro::Server (type Net::Server::Coro) starting! pid(1581)
Requests per second:    67.55 [#/sec] (mean)

Did you see that? Standalone::Prefork is almost twice as fast as WebGUI in the WRE! The pre-forking server’s max-workers setting defaults to 10, so the comparison might actually be fair too.

Holy Shit

I think that’s worth repeating. PlebGUI, which is currently less than one week old and running on a compatibility layer optimised for “let’s just get this thing working and worry about performance later” is already capable of out-performing the WRE in terms of raw speed by 200%. (If you believe the benchmarks).

But Plack/PSGI is not just about speed. It’s about flexibility. Think Koen de Jonge, WebGUI hosting extraordinaire at Procolix, deploying a high availability “follow the sun” WebGUI cluster on Nginx servers. Think Colin Kuskie, WebGUI test suite overlord, probing the dark corners of the WebGUI API using standard HTTP::Request and HTTP::Response pairs through mocked HTTP and live HTTP servers. Think web developers working on WebGUI client code, with a web server fully integrated into Padre. Think WebGUI entirely deployable from the CPAN.

Think of every new project added to the PSGI/Plack ecosystem as a potential new PlebGUI feature.

Where to Now

The approach I took in turning WebGUI into PlebGUI was to create a fake Apache2::Request object, since that’s the closest thing WebGUI has to a server abstraction layer. Plack contains two helper classes Plack::Request and Plack::Response that make this really easy. Currently though, that leaves us in the curious situation where WebGUI does all of its work thinking it’s talking to mod_perl, only to have its real output re-routed through the PSGI-compatibility layer, to be subsequently processed by a specific server backend. If you don’t get the joke yet, just think about what happens when the PSGI server backend happens to be mod_perl.

The benchmarks would clearly improve if we ripped out mod_perl altogether. Lots of code would simplify too. Certain parts of WebGUI could disappear altogether, such as URL Handlers which could be entirely replaced with Middleware.

The only feature that PlebGUI currently lacks is content streaming. I deliberately left that out since it will be a lot easier to achieve once mod_perl disappears. In Plack land the way to drip-feed browsers with “chunked” content is to return an IO::Handle-like object that responds to getline() and close(). The Plack folks are planning a fancy new module called IO::Writer that will Do The Right Thing under both blocking and non-blocking servers. Expect awesome things.

The following screenshot shows a nice example of several tabs open at once with different WebGUI Asset types displaying their own icons.

I’m hoping to change that. As of today I’ve stripped out all the experimental features and concentrated on the single feature that seems to have the widest possible appeal, namely, remote editing.

Version 0.02 of Padre::Plugin::WebGUI, which is on its way to the CPAN, allows you to connect over HTTP to your webgui site, view you asset tree and edit your assets from inside the Padre text editor. That hopefully makes it as useful for designers and content managers as it is for developers.

Previously the plugin got most of its power from the WGDev library. However this meant that you had to install WebGUI and WGDev as a prerequisite. Given that the WRE ships with non-threaded perl, this meant either installing these prereqs into your system perl or building another perl specifically to run them.  Not an overly arduous task, but enough of a hurdle to dissuade most WebGUI developers from playing with the plugin. That obstacle has now been removed. You do not need to have WebGUI or WGDev installed to run the plugin.

Using WGDev also meant that you could only connect locally to your site. That restriction is now gone.

Installation

Firstly, install Padre.

To allow your WebGUI site to talk to the plugin, add this content handler (WebGUI::Content::Padre) to your site and restart modperl  (the easiest way to do this is to drop the module into your /data/WebGUI/lib/WebGUI/Content dir and add the new content handler to your site config file. There is also an install script that will do the site config editing for you).

Next install the plugin from the CPAN

cpan Padre::Plugin::WebGUI

Usage

Inside Padre, go to the Plugin Manager and enable the WebGUI plugin. You should then see a WebGUI item appear in your Plugins menu.

From the WebGUI menu, tick “Show Asset Tree” and double-click on “Connect” in the new panel that appears.

You will be prompted for the URL of the site you want to connect to. Currently you need to enter you the site in the form: http://username:password@site.com (later we will add a site manager to make this nicer). Remember that this can be any webgui site that you have access to, not just a local site.

You will then see the Asset Tree dynamically populated with information from your site.

Try double-clicking on an article.

A new file tab should open with the name set to the name of your article (with [wg] prepended). The article content will be loaded into to tab, with Padre’s HTML syntax highlighting in effect. If you have Padre::Plugin::HTML installed, you can use the Tidy/Lint/Validate the HTML to your heart’s content. And every time you hit Save, your changes will be pushed up to the site as a new Version Tag.

Try double-clicking on another sort of asset, for instance a Template. Once again, the template html will be loaded into a new tab with nice syntax highlighting.

And for fun, try opening up a CSS Snippet. The plugin will detect that your snippet is set to the ‘text/css’ mime type and use CSS code highlighting instead of HTML.

And similarly for JavaScript Snippets.

Design and Extensions

As far as Padre is concerned the file you are editing is just a regular file on your file system. For example when you edit the text, an asterisk (*) appears next to the name of the file indicating that you haven’t save it yet.  The integration between the webgui content handler and the plugin is minimal, meaning that you could conceivably use this plugin to do remote editing on non-webgui sites too.

Padre has a nice architectural design, which makes it really easy for us to associate different WebGUI::Asset types with Padre::Document types. Currently most assets delegate to the default Padre::Document::WebGUI::Asset module which does HTML syntax highlighting and expects only generic content from the content handler running on your site. Padre::Document::WebGUI::Snippet has a slightly higher IQ level in that it asks the server to tell it what mimetype the asset is set to, so that it can use the appropriate syntax highlighter.

We could do some really interesting things by building asset-specific Padre::Document modules. For instance you could build a document type that uses fancy wxWidgets controls to do things that aren’t possible on the web (desktop integration etc..) And it’s just as easy to build these for your own custom asset types as it is for core ones. Given that Padre is cross-platform, you’re basically looking at a way of extending WebGUI interactions out into desktop-land.

If you’re a WebGUI designer/developer/content manager and you want to start playing with Padre::Plugin::WebGUI as a novel way of editing content, come find me on #webgui or #padre.

One problem that content managers run into is that after deleting a template in WebGUI, any assets that still refer to the template will die a horrible death when users try to view them. What would be nice is an automated way of finding all such assets that refer to missing template, so that you can update them to refer to a different template.

For starters, WGDev can give you a list of all available templates on your site via:

wgd ls root -r --format %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq > available

You can also ask for a list of all in-use templateIds, via:

wgd ls root -r --format %templateId% --filter '%templateId% ~~ /.+/' | sort | uniq > in_use

All you need to do then is ask the comm command to tell you which templateIds are in_use but not available:

comm -23 in_use available

Putting it all together on one line (without the temporary files), that becomes:

comm -23 <(wgd ls root -rf %templateId% --filter '%templateId% ~~ /.+/' | sort | uniq) <(wgd ls root -rf %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq)

The output of this command is a nice list of missing templateIds.

Of course, what you probably then want to do is ask WGDev to give you a list of asset urls that are trying to use these missing Templates, for example, if one item in your list is eHMsLCMqoXYpYrsZ7CWUtg, you would do:

wgd ls root -rf %url% --filter '%templateId% ~~ eHMsLCMqoXYpYrsZ7CWUtg'

Which would print something like

/my/broken/asset/url

Combining this with the previous command to do it all on one line is left as an exercise for the reader

Now, templateId isn’t the only Asset field that templates can be referenced under, for example, if you want to also find missing Style and Printable Style templates, you’d want to do something like:

for tmpl in templateId styleTemplateId printableStyleTemplateId
do
comm -23 <(wgd ls root -rf %${tmpl}% --filter "%${tmpl}% ~~ /.+/" | sort | uniq) <(wgd ls root -rf %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq)
done

It’s not optimised for speed, but it sure is handy. And you could apply the same approach to search for invalid group permissions, etc.. Of course if you found yourself doing this often you’re probably better off writing a dedicated WGDev plugin to wrap up the functionality.

There’s a WebGUI branch on GitHub containing some extra Template Usage-related functions – the plan is to add the ability to find all the different ways that templates are referenced (in Asset definitions, Account module definitions, …) so perhaps we will end up adding a “find missing Templates” button to the Edit Template page.

[ perl ironman ]

I’ve been getting a lot of joy out of the command line today too thanks to Graham’s ridiculously useful WGDev. One wgd command that I only started using in earnest today is the ls command. In its most basic form, it lets you print out a list of assets on your site:

$ wgd ls /home

getting_started
your_next_step
the_latest_news
tell_a_friend
documentation
site_map

I added a bunch of extensions to ls today, the first one being a -r option that tells ls to recursively list assets. Given that this prints to STDOUT, you can pipe this into any other command line tool to manipulate the output. For example, to list all assets that have the word “you” in their menu title:

$ wgd ls /root -f "%menuTitle%" -r | grep -i you
Profile Account Layout
Inbox Account Layout
Friends Layout Template
Account Layout
Shop Account Layout
Contributions Layout
Default Account Layout
Layout
Your Next Step

(the -f “%menuTitle%” option above tells the command to output only the menuTitle property of each asset).

Under the hood ls is calling asset->getLineage(), so I exposed a few of the getLineage API options to ls.

For example, this is one way of getting a sorted list of Survey templates on your site:

$ wgd ls root -r --includeOnlyClass WebGUI::Asset::Template -f "%url% %namespace%" | perl -ane 'print "$F[0] ($F[1])\n" if $F[1] =~ m/survey/i' | sort
root/import/expireincompletesurveyresponses/expireincompletesurveyresponses (ExpireIncompleteSurveyResponses)
root/import/survey/default-answer-edit (Survey/Edit)
root/import/survey/default-feedback (Survey/Feedback)
root/import/survey/default-gradebook-report (Survey/Gradebook)
root/import/survey/default-overview-report (Survey/Overview)
root/import/survey/default-question-edit (Survey/Edit)
root/import/survey/default-questions (Survey/Take)
root/import/survey/default-section-edit (Survey/Edit)
root/import/survey/default-survey-edit (Survey/Edit)
root/import/survey/default-survey-summary (Survey/Summary)
root/import/survey/default-survey (Survey)
root/import/survey/default-survey-take (Survey/Take)
root/import/survey/default-test-results (Survey/TestResults)

While I was writing this post, Andy asked me to provide him with a list of all Carousel wobjects on a site we’re working on. That’s as easy as:

$ wgd ls -r root --includeOnlyClass WebGUI::Asset::Wobject::Carousel

The most useful option I added to ls is the –filter smartmatch filter option. This option filters the results using an asset property of your choosing. Format looks like “%url% ~~ smartmatch”, where “url” is the field to filter against, and “smartmatch” is either a Perl regular expression such as “/(?i:partial_match)/” or a string such as “my_exact_match”.

For example, the following command lists all templates that include absolute urls. We generate the output to an html file so that someone can easily open all the links and manually review/remove the absolute urls:

$ wgd ls -r /root --filter "%template% ~~ /http:///" --includeOnlyClass WebGUI::Asset::Template --format "<a href=http://dev.localhost.localdomain/%url%?func=edit>%url%</a><br>" > links.html

links.html then contains:

<a href=http://dev.localhost.localdomain/style_01?func=edit>style_01</a><br>
<a href=http://dev.localhost.localdomain/style_02?func=edit>style_02</a><br>
<a href=http://dev.localhost.localdomain/style_03?func=edit>style_03</a><br>

...

One type of command that I can see myself using a lot in the future is the following, which displays all assets that are viewable to a specific group (in this case the “Turn Admin On” group, which has an id of “12″):

$ wgd ls -r root --format "%groupIdView% %url%" --filter "%groupIdView% ~~ 12"
12 root/import/newsletter
12 root/import/projectmanager/resource
12 root/import/wiki

Alternatively, you can use a variation of the above to list all assets that are NOT viewable to a certain group (or groups). This is great for finding assets that have accidentally been set to say, Admins, when you want them viewable by Registered Users. For example, the following command lists assets that are not viewable to the default “Visitors”, “Everyone”, or “Registered Users” groups:

$ wgd ls -r root --format "%groupIdView% %url%" | perl -ane 'print "$F[0] $F[1]\n" if !grep {$F[0] eq $_} qw(1 2 7)'
12 root/import/newsletter
12 root/import/projectmanager/resource
12 root/import/wiki

All of these commands take in the order of 1 second to run, which is a lot less time consuming than manually reviewing your site asset-by-asset. And if you incorporate them into your test suite, you can re-verify their output against your site policy/expectations whenever you like.

I’ve released the code onto CPAN, and the current dev version is available via github, but since it’s all a bit hacky and alpha at the moment I thought it worthwhile doing a little tour of what currently works (on my dev box at least).

Installation

Installation is the same as any other Padre plugin, either install via cpan

cpan install Padre::Plugin::WebGUI

or from git:

git clone git://github.com/pdonelan/Padre-Plugin-WebGUI.git

perl Makefile.PL

make && make test && make install

Since this is a WebGUI plugin, you need to first install WebGUI, and also WGDev.

Once you’ve done that, you will see the WebGUI plugin listed in the Padre Plugin Manager:

Click the enable button, and you’ll see a WebGUI item appear in the Plugin menu.

Controlling WRE Services

The first thing you can do with the plugin is control your WRE services (mysql, modperl, modproxy and spectre).

Click on: WebGUI > WRE Services > Modperl > Restart and you’ll see a restart status message appear in the Padre output panel.

Currently this is just a simple wrapper around the wreservice.pl script that ships with WebGUI. A nice enhancement would be to turn the menu item into a services dashboard with lights telling you which services are currently running, and buttons to start/stop/restart.

Online Resources

Another item in the menu called Online Resources gives you access to a bunch of handy WebGUI links that open in your browser – including one that takes you straight to a web-based IRC client connected to #webgui.

WebGUI Log

If you click the “Logview” checkbox on the WebGUI menu, a WebGUI Log panel will open. This panel will start monitoring all of your WRE log files (webgui.log, modperl.log, modproxy.log, etc..) and display changes in real-time. WebGUI uses Log4perl as its logging engine, but it uses the sitename as the Log4perl category, which means that you can’t easily e.g. turn on DEBUG output and filter out everything except the module you’re working on (unless you use my Log4perl::Filter::CallerMatch module). Once we add in the ability to easily customise what files are being logged and apply filters to what gets displayed, this will make the above scenario a breeze.

The logview panel uses a long-running Padre task and File::Tail::select to continuous monitor a bunch of files in realtime. The main thread writes commands to a temp file to send commands to the monitoring thread (the most important command being “exit” which tells the thread to break out of its monitoring loop!). Later I plan on adding support for other commands such as “add/remove this file from the list of files you’re watching”, “add/remove filter”, etc.. Since this seems like a generic sort of feature I’m hoping to bundle  Padre::Plugin::WebGUI::Logview and Padre::Plugin::WebGUI::Task::Logview as a stand-alone module for other Padre plugin developers to use (and improve).

Asset Tree

And finally, the coolect feature the WebGUI plugin has is the Asset Tree.

If you turn on the Asset Tree from the plugin menu, a new panel appears with a “Connect” button on it. Currently this is hardwired to the default WebGUI dev site at dev.localhost.localdomain but later I plan on adding a Site Manager to allow you to define sites here.

Double-clicking on the Connect button connects to the site, and pulls down the asset tree:

This is the same Asset Tree you see when you log in to your WebGUI site and turn admin mode on. As you can see from the screenshot, right-clicking on a node brings up a context menu, from which you can do asset-specific actions. The plugin already gives you access to all of the WGDev commands from the main plugin menu (with a dialog box asking you what arguments you want to pass to the command), but it becomes a lot more useful once you hook these WGDev commands to the Asset Tree context menu. Then instead of having to remember what url or assetId your asset has, you can just click through the tree to find the node you want, right-click and off you go.

For example, WGDev has a package command that lets you package up node and all of its children into a tarball that can be imported into other sites, used in upgrade scripts, etc.. By integrating this command into the context menu, you can have Padre prompt you you for the folder that you want to save the WebGUI package in (via your native Save File dialog box).

Similarly, WGDev has an edit command that lets you edit an asset (for example a template file) via your favourite editor and then have the changes pushed back into the database after you save and close the file. By integrating this command into Padre, you could leave the file open, and push the changes up every time you hit save(allowing for more convenient continuous editing).