Archive for July, 2009

Finding missing WebGUI Templates

July 30th, 2009

Following on from my previous WGDev post, here’s another example of wgd ls that lets you find missing Templates on a WebGUI site.

One problem that content managers run into is that after deleting a template in WebGUI, any assets that still refer to the template will die a horrible death when users try to view them. What would be nice is an automated way of finding all such assets that refer to missing template, so that you can update them to refer to a different template.

For starters, WGDev can give you a list of all available templates on your site via:


wgd ls root -r --format %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq > available

You can also ask for a list of all in-use templateIds, via:


wgd ls root -r --format %templateId% --filter '%templateId% ~~ /.+/' | sort | uniq > in_use

All you need to do then is ask the comm command to tell you which templateIds are in_use but not available:


comm -23 in_use available

Putting it all together on one line (without the temporary files), that becomes:


comm -23 <(wgd ls root -rf %templateId% --filter '%templateId% ~~ /.+/' | sort | uniq) <(wgd ls root -rf %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq)

The output of this command is a nice list of missing templateIds.

Of course, what you probably then want to do is ask WGDev to give you a list of asset urls that are trying to use these missing Templates, for example, if one item in your list is eHMsLCMqoXYpYrsZ7CWUtg, you would do:


wgd ls root -rf %url% --filter '%templateId% ~~ eHMsLCMqoXYpYrsZ7CWUtg'

Which would print something like


/my/broken/asset/url

Combining this with the previous command to do it all on one line is left as an exercise for the reader ;)

Now, templateId isn’t the only Asset field that templates can be referenced under, for example, if you want to also find missing Style and Printable Style templates, you’d want to do something like:


for tmpl in templateId styleTemplateId printableStyleTemplateId
do
comm -23 <(wgd ls root -rf %${tmpl}% --filter "%${tmpl}% ~~ /.+/" | sort | uniq) <(wgd ls root -rf %assetId% --includeOnlyClass WebGUI::Asset::Template | sort | uniq)
done

It’s not optimised for speed, but it sure is handy. And you could apply the same approach to search for invalid group permissions, etc.. Of course if you found yourself doing this often you’re probably better off writing a dedicated WGDev plugin to wrap up the functionality.

There’s a WebGUI branch on GitHub containing some extra Template Usage-related functions – the plan is to add the ability to find all the different ways that templates are referenced (in Asset definitions, Account module definitions, …) so perhaps we will end up adding a “find missing Templates” button to the Edit Template page.

[ perl ironman ]

Ode to wgd and the command line

July 14th, 2009

Doug just posted an ode to the Linux command line over at the Plain Black staff blog.

I’ve been getting a lot of joy out of the command line today too thanks to Graham’s ridiculously useful WGDev. One wgd command that I only started using in earnest today is the ls command. In its most basic form, it lets you print out a list of assets on your site:


$ wgd ls /home

getting_started
your_next_step
the_latest_news
tell_a_friend
documentation
site_map

I added a bunch of extensions to ls today, the first one being a -r option that tells ls to recursively list assets. Given that this prints to STDOUT, you can pipe this into any other command line tool to manipulate the output. For example, to list all assets that have the word “you” in their menu title:


$ wgd ls /root -f "%menuTitle%" -r | grep -i you
Profile Account Layout
Inbox Account Layout
Friends Layout Template
Account Layout
Shop Account Layout
Contributions Layout
Default Account Layout
Layout
Your Next Step

(the -f “%menuTitle%” option above tells the command to output only the menuTitle property of each asset).

Under the hood ls is calling asset->getLineage(), so I exposed a few of the getLineage API options to ls.

For example, this is one way of getting a sorted list of Survey templates on your site:


$ wgd ls root -r --includeOnlyClass WebGUI::Asset::Template -f "%url% %namespace%" | perl -ane 'print "$F[0] ($F[1])\n" if $F[1] =~ m/survey/i' | sort
root/import/expireincompletesurveyresponses/expireincompletesurveyresponses (ExpireIncompleteSurveyResponses)
root/import/survey/default-answer-edit (Survey/Edit)
root/import/survey/default-feedback (Survey/Feedback)
root/import/survey/default-gradebook-report (Survey/Gradebook)
root/import/survey/default-overview-report (Survey/Overview)
root/import/survey/default-question-edit (Survey/Edit)
root/import/survey/default-questions (Survey/Take)
root/import/survey/default-section-edit (Survey/Edit)
root/import/survey/default-survey-edit (Survey/Edit)
root/import/survey/default-survey-summary (Survey/Summary)
root/import/survey/default-survey (Survey)
root/import/survey/default-survey-take (Survey/Take)
root/import/survey/default-test-results (Survey/TestResults)

While I was writing this post, Andy asked me to provide him with a list of all Carousel wobjects on a site we’re working on. That’s as easy as:


$ wgd ls -r root --includeOnlyClass WebGUI::Asset::Wobject::Carousel

The most useful option I added to ls is the –filter smartmatch filter option. This option filters the results using an asset property of your choosing. Format looks like “%url% ~~ smartmatch”, where “url” is the field to filter against, and “smartmatch” is either a Perl regular expression such as “/(?i:partial_match)/” or a string such as “my_exact_match”.

For example, the following command lists all templates that include absolute urls. We generate the output to an html file so that someone can easily open all the links and manually review/remove the absolute urls:


$ wgd ls -r /root --filter "%template% ~~ /http:///" --includeOnlyClass WebGUI::Asset::Template --format "<a href=http://dev.localhost.localdomain/%url%?func=edit>%url%</a><br>" > links.html

links.html then contains:

<a href=http://dev.localhost.localdomain/style_01?func=edit>style_01</a><br>
<a href=http://dev.localhost.localdomain/style_02?func=edit>style_02</a><br>
<a href=http://dev.localhost.localdomain/style_03?func=edit>style_03</a><br>

...

One type of command that I can see myself using a lot in the future is the following, which displays all assets that are viewable to a specific group (in this case the “Turn Admin On” group, which has an id of “12″):


$ wgd ls -r root --format "%groupIdView% %url%" --filter "%groupIdView% ~~ 12"
12 root/import/newsletter
12 root/import/projectmanager/resource
12 root/import/wiki

Alternatively, you can use a variation of the above to list all assets that are NOT viewable to a certain group (or groups). This is great for finding assets that have accidentally been set to say, Admins, when you want them viewable by Registered Users. For example, the following command lists assets that are not viewable to the default “Visitors”, “Everyone”, or “Registered Users” groups:


$ wgd ls -r root --format "%groupIdView% %url%" | perl -ane 'print "$F[0] $F[1]\n" if !grep {$F[0] eq $_} qw(1 2 7)'
12 root/import/newsletter
12 root/import/projectmanager/resource
12 root/import/wiki

All of these commands take in the order of 1 second to run, which is a lot less time consuming than manually reviewing your site asset-by-asset. And if you incorporate them into your test suite, you can re-verify their output against your site policy/expectations whenever you like.

Padre::Plugin::WebGUI

July 3rd, 2009

For a little while now I’ve been working on a WebGUI plugin for Padre (the Perl Application Development and Refactoring Environment). Progress has been pretty slow, partly due to lack of time but also because often the things I want to do are at the edge of what Padre currently supports (or just that there’s no existing examples of how to do something easily).

I’ve released the code onto CPAN, and the current dev version is available via github, but since it’s all a bit hacky and alpha at the moment I thought it worthwhile doing a little tour of what currently works (on my dev box at least).

Installation

Installation is the same as any other Padre plugin, either install via cpan


cpan install Padre::Plugin::WebGUI

or from git:


git clone git://github.com/pdonelan/Padre-Plugin-WebGUI.git

perl Makefile.PL

make && make test && make install

Since this is a WebGUI plugin, you need to first install WebGUI, and also WGDev.

Once you’ve done that, you will see the WebGUI plugin listed in the Padre Plugin Manager:

Padre Plugin Manager

Click the enable button, and you’ll see a WebGUI item appear in the Plugin menu.

Controlling WRE Services

The first thing you can do with the plugin is control your WRE services (mysql, modperl, modproxy and spectre).

Click on: WebGUI > WRE Services > Modperl > Restart and you’ll see a restart status message appear in the Padre output panel.

restart-modperl

Currently this is just a simple wrapper around the wreservice.pl script that ships with WebGUI. A nice enhancement would be to turn the menu item into a services dashboard with lights telling you which services are currently running, and buttons to start/stop/restart.

Online Resources

Another item in the menu called Online Resources gives you access to a bunch of handy WebGUI links that open in your browser – including one that takes you straight to a web-based IRC client connected to #webgui.

WebGUI Log

If you click the “Logview” checkbox on the WebGUI menu, a WebGUI Log panel will open. This panel will start monitoring all of your WRE log files (webgui.log, modperl.log, modproxy.log, etc..) and display changes in real-time. WebGUI uses Log4perl as its logging engine, but it uses the sitename as the Log4perl category, which means that you can’t easily e.g. turn on DEBUG output and filter out everything except the module you’re working on (unless you use my Log4perl::Filter::CallerMatch module). Once we add in the ability to easily customise what files are being logged and apply filters to what gets displayed, this will make the above scenario a breeze.

The logview panel uses a long-running Padre task and File::Tail::select to continuous monitor a bunch of files in realtime. The main thread writes commands to a temp file to send commands to the monitoring thread (the most important command being “exit” which tells the thread to break out of its monitoring loop!). Later I plan on adding support for other commands such as “add/remove this file from the list of files you’re watching”, “add/remove filter”, etc.. Since this seems like a generic sort of feature I’m hoping to bundle  Padre::Plugin::WebGUI::Logview and Padre::Plugin::WebGUI::Task::Logview as a stand-alone module for other Padre plugin developers to use (and improve).

logview

Asset Tree

And finally, the coolect feature the WebGUI plugin has is the Asset Tree.

If you turn on the Asset Tree from the plugin menu, a new panel appears with a “Connect” button on it. Currently this is hardwired to the default WebGUI dev site at dev.localhost.localdomain but later I plan on adding a Site Manager to allow you to define sites here.

Double-clicking on the Connect button connects to the site, and pulls down the asset tree:

asset-tree

This is the same Asset Tree you see when you log in to your WebGUI site and turn admin mode on. As you can see from the screenshot, right-clicking on a node brings up a context menu, from which you can do asset-specific actions. The plugin already gives you access to all of the WGDev commands from the main plugin menu (with a dialog box asking you what arguments you want to pass to the command), but it becomes a lot more useful once you hook these WGDev commands to the Asset Tree context menu. Then instead of having to remember what url or assetId your asset has, you can just click through the tree to find the node you want, right-click and off you go.

For example, WGDev has a package command that lets you package up node and all of its children into a tarball that can be imported into other sites, used in upgrade scripts, etc.. By integrating this command into the context menu, you can have Padre prompt you you for the folder that you want to save the WebGUI package in (via your native Save File dialog box).

Similarly, WGDev has an edit command that lets you edit an asset (for example a template file) via your favourite editor and then have the changes pushed back into the database after you save and close the file. By integrating this command into Padre, you could leave the file open, and push the changes up every time you hit save(allowing for more convenient continuous editing).