patspam.com

Recently I've been plugging away at an experimental "WebGUI_flux" branch of the awesome Perl based content management system / web application framework WebGUI that wG founder JT created for me a month or so ago. I'm happy to say that the code has progressed to the point where I'm ready to announce Flux to the world and invite anyone who is interested to have a play with the demo server, explore the API, give me feedback and if you like the look of it, contribute some plugins of your own.

Here's the original impetus for building Flux:

As per most content management systems, WebGUI's built-in authorization mechanism is based on User Group membership. This is perfectly adequate for a large class of websites: content managers define one or more User Groups and then set the "Who Can View" and "Who Can Edit" Security Tab options on their Wobjects to the appropriate group. However in one important aspect such sites are static in nature: the set of pages that a user can access is fixed according to what groups they belong to. Authorization can be made dynamic by manipulating group membership (manually, which isn't really dynamic, or via custom code) or by writing custom Wobject authorization logic. Both of these approaches work, however, custom code leaves content managers out of the loop.

You can find the latest version of the design doc in SVN in pdf and odt formats. Flux is quite large in scope and hard to condense into a single post, so please refer to the design doc if you want more details, but I'll try to give a quick summary here.

For content managers:

• Flux is a rule-based authorization layer for WebGUI content managers. What that means is that you can define Rules (using a GUI) that define the conditions that must be met for a user to be given access to an Asset. Each time a user tries to access an Asset, Flux evaluates the Rule(s) you have defined to see if the user should be granted access.
• By default Flux is disabled, meaning zero performance hit and no change to the UI.
• You can enable/disable Flux at the site-wide level (via the Flux Admin Console), and also on individual Assets (via the asset's Security Tab).
• With Flux turned on for an Asset, the Security Tab shows a combo box containing all Flux Rules that you have defined. These combo boxes appear for each action your Asset supports (e.g. "Who Can View", "Who Can Edit", etc..). If you pick a Rule for an action, authorization logic for that action is delegated to that particular Flux Rule.
• Rules are powerful, flexible, and pluggable.
• You define Rules via a simple, intuitive step-by-step wizard in the Flux Admin Console.
• Rules can depend on static information, user-specific information, time-dependent information, asset-specific information, and anything else we plug into the framework.
• Rules can also depend on other Rules, meaning that you can construct an arbitrarily complex graph of connected Rules. Flux makes sure that infinite loops don't occur, and can even dynamically generate a visualisation of your Flux Graph showing all interrelationship between Rules

• Rules are made up of one or more Expressions
• Each Expression is comprised of an Operator and two Operands.
• The UI guides content managers through the process of picking these, displaying helpful messages and prompting for any additional information required along the way
• The following Operators are built:
  • IsEqualTo, IsNotEqualTo
  • LessThan, LessThenOrEqualTo, GreaterThan, GreaterThanOrEqualTo
  • MatchesPartialText, DoesNotMatchPartialText
• The following Operands have been built:
  • TextValue - allows the content manager to enter a simple text value
  • NumericValue - allows the content manager to enter a numeric value
  • TruthValue - allows the content manager to enter yes/no
  • DateTime - allows the content manager to enter a date from a date-picker
  • Group - allows the content manager to choose a group from the wG Group combo box. The Operand evaluates to true/false depending on whether the user being tested against belongs to that group
  • UserProfileField - allows the content manager to specify the name of a User Profile field. The Operand evaluates to the value of that user profile field for the user being tested against.
  • FluxRule - allows the content manager to choose another Flux Rule from a combo box. The Operand evaluates to true/false depending on the result of testing that Flux Rule against the user.
• Operands can prompt the user for an arbitrary number of extra arguments as part of the step-by-step wizard. These arguments are stored as a JSON-encoded string.
• Modifiers can be registered against Operand return types, causing the UI to prompt the content manager for extra information when an Operand is chosen during Expression building. For example, the following two Modifiers have been built and registered against the DateTime type:
  • DateTimeFormat - prompts the content manager for date format patterns (as per strftime()) so that the DateTime object can be formatted as a string. Also prompts the content manager for a timezone to use (any valid timezone can be specified, or 'user' which indicates that the timezone of the user being tested against should be used). This is useful so that you can e.g. compare a DateTime object to a string such as "Monday".
  • DateTimeCompareToNow - prompts the content manager for a timezone and a Duration unit to use (e.g. 'hours', 'days', etc..). This allows you to construct Rules that evaluate to true a certain number of days since a particular date etc..
• By default, multiple Expressions are ANDed together. But if you like you can specify an arbitrarily complex boolean logical expression instead (using AND, OR, NOT and parenthesis).

I've included a contrived example, which might makes things clearer. It might also make things more confusing since you don't have the nice UI to guide you through the process of building the Expressions, but see how you go..

Imagine you have a WebGUI site with some content that you want to show to your French users only.

• You create a Rule called "French Only" and then add an Expression to it that has:
  • operand1: UserProfileField
  • operand1Args: {field: homeCountry} # note that this is JSON-encoded
  • operator: IsEqualTo
  • operand2: TextValue
  • operand2Args: {value: "France"}
• You then enable Flux on the Security Tab for that content and choose the "French Only" Rule from the drop-down list for "Who Can View"… voila!

Now say you've been using good old-fashioned Group membership on your site. Let's assume you have a "Premium Members" Group. On a whim, you decide your content should be restricted to French Premium Members:

• You add a second Expression to the "French Only" Rule to require the Premium Members group:
  • operand1: Group
  • operand1Args: {groupId: <id of Premium Members group as chosen from group combo box>}
  • operator: IsEqualTo
  • operand2: TruthValue
  • operand2Args: {value: 1}

A few weeks later, you decide to add an extra page that French Premium Members can see on their birthday.

• You create a new Rule called "Bon Anniversaire" and add an Expression linking to your earlier "French Only" Rule:
  • operand1: FluxRule
  • operand1Args: {fluxRuleId: <id of the "French Only" Flux Rule as chosen from a combo box>}
  • operator: IsEqualTo
  • operand2: TruthValue
  • operand2Args: {value: 1}
• ..and a second Expression requiring that today be the user's birthday (in their time zone of course):
  • operand1: UserProfileField
  • operand1Args: {field: birthdate}
  • operand1Modifier: DateTimeCompareToNow
  • operand1ModifierArgs: {units: "days", time_zone: "user"} # Modifier args are JSON-encoded too
  • operator: IsEqualTo
  • operand2: NumericValue
  • operand2Args: {value: 0}

Cool huh?

Hopefully that gives you a feel for the power of the framework. Be sure to check out the design docs for more details (did I mention it also supports Workflow triggers?). The test suite (in t/Flux) is pretty comprehensive (currently at >85% coverage) - the Operand-specific tests in particular should get you up to speed pretty quickly on how to drive Flux.

Test Coverage Report:

---------------------------- ------ ------ ------ ------ ------ ------ ------

File                           stmt   bran   cond    sub    pod   time  total

---------------------------- ------ ------ ------ ------ ------ ------ ------

lib/WebGUI/Flux.pm             88.1   71.9   50.0  100.0   75.0    2.4   82.3

...WebGUI/Flux/Expression.pm   93.3   85.7   60.0   89.5  100.0   17.2   89.1

...lux/Expression/Builder.pm  100.0    n/a    n/a  100.0    n/a    0.1  100.0

lib/WebGUI/Flux/Modifier.pm    84.9   61.9   44.4  100.0  100.0    2.9   77.0

...r/DateTimeCompareToNow.pm  100.0  100.0    n/a  100.0    0.0    0.2   92.6

...odifier/DateTimeFormat.pm  100.0   50.0    n/a  100.0    0.0    0.3   87.5

lib/WebGUI/Flux/Operand.pm     85.9   66.7   44.4  100.0  100.0   14.7   78.9

.../Flux/Operand/DateTime.pm  100.0    n/a    n/a  100.0    0.0    0.2   91.3

.../Flux/Operand/FluxRule.pm   97.1   83.3    n/a  100.0    0.0    1.3   90.0

...GUI/Flux/Operand/Group.pm  100.0    n/a    n/a  100.0    0.0    0.1   90.5

...x/Operand/NumericValue.pm  100.0    n/a    n/a  100.0    0.0    0.1   89.5

...Flux/Operand/TextValue.pm  100.0    n/a    n/a  100.0    0.0    0.6   89.5

...lux/Operand/TruthValue.pm  100.0    n/a    n/a  100.0    0.0    0.1   89.5

...erand/UserProfileField.pm  100.0    n/a    n/a  100.0    0.0    0.1   90.5

lib/WebGUI/Flux/Operator.pm    84.5   53.1   33.3  100.0  100.0   14.5   75.0

...oesNotMatchPartialText.pm  100.0    n/a    n/a  100.0    0.0    0.1   95.5

...lux/Operator/IsEqualTo.pm  100.0  100.0  100.0  100.0    0.0    1.8   96.6

...Operator/IsGreaterThan.pm  100.0  100.0  100.0  100.0    0.0    0.2   96.6

...IsGreaterThanOrEqualTo.pm  100.0  100.0  100.0  100.0    0.0    0.2   96.6

...ux/Operator/IsLessThan.pm  100.0  100.0   66.7  100.0    0.0    0.2   93.1

...or/IsLessThanOrEqualTo.pm  100.0  100.0  100.0  100.0    0.0    0.2   96.6

.../Operator/IsNotEqualTo.pm  100.0  100.0  100.0  100.0    0.0    0.2   96.6

...tor/MatchesPartialText.pm  100.0    n/a    n/a  100.0    0.0    0.1   95.5

lib/WebGUI/Flux/Rule.pm        94.9   88.9   84.4   93.3   93.8   42.4   92.3

Total                          93.1   75.7   70.2   97.8   53.2  100.0   87.4

---------------------------- ------ ------ ------ ------ ------ ------ ------

Implementation status:

• You can do just about everything through the API (check out the test suite for more info)
• You can write your own Operands and Operators as plugins. To give you an idea of how easy it is to add your own plugins, most of the current Operands require 1-3 lines of unique code. Operators are simple too.
• The Flux Admin Console is working, although not ajaxified yet
• Per-asset Flux authorization UI options are available on some Assets, although only tested on Article and PageLayout thus far and not pretty.
• I've only implemented a simple CRUD interface for manipulating Rules and Expressions through the Admin console. The current interface is really just a simple slap-dash job put together so that people can play with the underlying framework without needing to write code. The Expression builder in particular is quite clunky to use - you need to pass in fully-formed JSON Operand arguments. The UI has inline documentation to help you do this. The design doc has mock-ups of what the finished UI will look like.
• I haven't implemented Wobject-bound Rules, although I will soon - this is where Flux teams up with next generation WebGUI Wobjects such as Thingy and Survey 2.0 to do some really cool things (check the design doc).

I've set up a demo server where you can play with Flux. The demo sites come pre-bundled with some Rules to get you started: go the the Flux item in the Admin Console and click on Flux Graph to generate your version of the image attached to this email. As mentioned, you'll probably find the Expression builder pretty awkward since it nice step-by-step wizard isn't built yet.

You can also download the latest version of the code from SVN and run it yourself - be aware that visualisation of the Flux Graph is currently done using GraphViz so you will need to install it on your system using something similar to "apt-get install graphviz gsfonts" and "cpan GraphViz".

The big caveat for getting Flux included in a future version of WebGUI is performance. Obviously checking Flux Rules is more expensive than doing a single call to $user->isInGroup(). I've done quite a bit of work on benchmarking, profiling and optimisation, and so far things look good but there's still work to be done. I'll save that discussion for a later post.

• Leave a comment... (0)

A huge class of applications are nothing more than a bunch of relational database tables with a simple Create Read Update Delete (CRUD) user interface. More and more these applications are being built as web-apps, eg. the user interface is HTML/CSS/Javascript running inside a users' browser, and the backend is a simple server-side layer (Perl/PHP/Ruby/J2EE/..) controlling access to an online database. This architecture is becoming increasingly easy to implement with the advent of server-side web/application frameworks (Catalyst/Rails/Spring) and database Object Relational Mappers (DBIx::Class/ActiveRecord/Hibernate,..) that remove most of the tedium, along with awesome Javascript libraries in the front-end (ExtJS/jQuery/Dojo/..) that let you make the user interface much more engaging and adaptable.

What happens, however, when your intended users have no internet access? I was recently faced with that exact problem: My Dad (Dr Ted Donelan) runs a pro bono dog health program in Maningrida, a self-governing aboriginal community in Arnhem Land (500km east of Darwin) where, unless you want to pay through the teeth for that Telstra NextG internet rubbish, there's no easy access to the internet. And certainly no way of logging into a website while you're riding around from house to house in a ute or carrying out surgery on an outdoor trestle table underneath the local grog-shelter. Dr Ted wanted a way to record the statistical information that he collects (dog body conditions, mange scores, fleas, ticks, etc..) along with his formal veterinary records of surgical cases, medical cases, etc.. All of which can be neatly captured in a handful of relational tables (attested by the fact that previously he had been recording his data in hand-written tables, later to be transferred to Excel). I suppose I could have stepped back into the C#/Java world and implemented a stand-alone app to solve the problem, but I'm not familiar enough with any frameworks in those languages to whip up something quickly. And that was the point, I only had a small amount of time to come up with a solution. And what I really wanted was a way to build him a stand-alone app using my existing HTML/Javascript/CSS-fu..

Cue Adobe AIR.. a desktop framework that lets you build traditional dektop apps using web technologies. Now I'm not the biggest fan of proprietary technology (Adobe Flash being a prime example) since it goes against the grain of the Open Web, but there's really no open-source alternatives at the moment.

The first really awesome AIR app I saw was ExtJS creator Jack Slockum's "Simple Tasks". Check out the screen-shots and you'll see why I was excited: here was a desktop application with an embedded (SQLite) database running pure HTML/CSS/Javascript. And not just any Javascript either. ExtJS shines most when used to create windows application-style interfaces, and as such it's the perfect library to use in an AIR app when you really are building a desktop application.

Simple Tasks is released under GPL v3, so I decided to base my code on Jack's. It was the right decision. With no prior experience with AIR it took less than a day to have a working prototype, and by working I mean a full database implementation, an editable data grid and an Add/Edit form for individual rows, all with the trademark ExtJS polish. That's ridiculously fast.

After about 4 days with lots of feedback from Dr. Ted we had a finished app, named Tarpo, complete with polished input forms, reporting, backup, CSV/XML export, …

I'll be releasing Tarpo under GPL v3. Until then I've got a little page where you can install it straight off the web and have a play. Dr. Ted is heading up to Maningrida next week so I'm excited to see how Tarpo goes on her first test-run. If everything works well we'll make her available to other Vets running similar programs in other indigenous communities. One cool follow-on from this will be that vets running AMRRIC-endorsed programs could use Tarpo to submit their activity reports direct to the AMRRIC website (which I recently developed).

Andy (also of SDH) is whipping up a Tarpo logo for me, which I'll post up here as soon as he's finished inking it!

• Read comments... (2)

I just scratched an itch I've had for a long time - a proper way to backup cPanel accounts from a reseller account. Up until now I've been scripting w get to grab the daily-generated backup of specific accounts every week (imitating what you can do manually by logging into cPanel and navigating to the Backup tab). It was annoying because I had to manually put in the username and password of each account I wanted to backup. What I really wanted was a solution to backup all accounts with minimal fuss.

The solution is a cron job that runs from my reseller account (you don't need SSH access to do this as cPanel gives you access to your crontab). The solution looks like this:

My reseller server puts the daily-generated backups into ~/backups. So, all we have to do is tar this up and copy it over to a backup server.

Notes:

• We give tar the h option so that it stores the contents of symlinked directories in the archive, not real symlinks (the /backups directory in the primary reseller account contains symlinks inside sub-account subdirectories to the real location of eg. mysql snapshots etc..)
• We compress the entire directory and pipe it across the wire via ssh
• The backup is stored with the current day of the week in the name, effectively giving us 7-day rotation
• You should set up password-less SSH access using public/private keys so that the command isn't prompted for a password
• The above command should be run from cron
• The backslash escaping of the percentage sign in the date command is necessary when the command is run from cron
• UPDATE: nice'd the command so that it doesn't consume too many resources and get killed by my provider

As usual, I've documented this for future reference in Paddypedia.

• Read comments... (2)

Reading large chunks of text on a widescreen monitor can be a real pain if your eyes have to read across long horizontal lines. That's why newspapers print articles in columns! Ideally, websites should too, at least when you're trying to view them on a large monitor. Obviously you can't expect webpage designers to predict the size of your screen, but if you're using Firefox you can use this simple bookmarklet I wrote to reformat a selection of text (or the whole page) into multiple columns yourself.

How to use it the Multiple Columns Bookmarklet:
This is the link to my Multiple Columns Bookmarklet.

For starters, try it out yourself on this page! Select a few words in the first paragraph of this page and click the bookmarklet link. You should see it magically reformat into multiple columns 200px wide.

Now select two continuous paragraphs and click the bookmarklet link. You should see the entire article reformatted into easy to read columns. Cool huh?

If you want to use the bookmarklet on other pages, bookmark it now. When you're on a page that you want to reformat select the chunk of text and click on the bookmarklet. voila!

If you select text a single paragraph, only that paragraph will be turned into columns. If you select multiple paragraphs, the containing DOM element will be reformated (this isn't perfect but it usually does what you want). If no text is selected when you launch the bookmarklet, the entire page will be turned into multiple columns (that won't look great on my blog, but it's handy on pages with minimal styling)

If you add a keyword such as col to your bookmarklet link (Right Click > Properties), then you can launch the bookmarklet from the location bar by typing col rather than clicking on the link.

Finally, if you provide an argument of max, eg. col max then the maximum height of the columns will be set to the size of your web browser window. This is really handy if you're viewing a long document and you don't want the columns to scroll vertically for miles and miles (by setting the column height the columns will instead continue horizonally, just like a large-format newspaper).

In case you're wondering, the bookmarklet is only intended for Firefox because none of the other browsers have implemented any column-related CSS features.

• Read comments... (2)