December 19, 2011

Virtual Solari Board Improved

As part of the series of posts about displaying data, I've updated the virtual Solari board project again.

It now uses a plugin framework to handle customized look and feel. You create a javascript object with properties and methods specific to your data source, like so:

sf.display.ImageDrum = function() {
  this.order = [' ','AFL','AAL','BAW','DAL','UAE','KLM','DLH','RYR','UAL','AWE'];
};

sf.plugins.airport =  {
  
  // input: map of parameters
  // output: url
  url: function(params){
    var base_url = "data/airport_schedule.php";
    return base_url + "?" + params.serialize();
  },

  formatData: function(data){
    return data.response.results[0].data;
  }

};

sf.display.ImageDrum()'s order is now a series of strings instead of integers, which should be more readable. So in the above example, the CSS looks like:

.splitflap .image span.ctrn { background-position: 0px 0px; } /* transition */
.splitflap .image span.csp { background-position: 0px -40px; } /* blank */
.splitflap .image span.AFL { background-position: 0px -80px; } /* aeroflot */
.splitflap .image span.AAL { background-position: 0px -120px; } /* american */
.splitflap .image span.BAW { background-position: 0px -160px; } /* british airways */
.splitflap .image span.DAL { background-position: 0px -200px; } /* delta */
.splitflap .image span.UAE { background-position: 0px -240px; } /* emirates */
.splitflap .image span.KLM { background-position: 0px -280px; } /* klm */
.splitflap .image span.DLH { background-position: 0px -320px; } /* lufthansa */
.splitflap .image span.RYR { background-position: 0px -360px; } /* ryanair */
.splitflap .image span.UAL { background-position: 0px -400px; } /* united */
.splitflap .image span.AWE { background-position: 0px -440px; } /* us airways */

But the big change here is that the project now uses backbone.js as its MVC framework. That means that the list of rows is a Collection and each row is a Model. This allows us to use Backbone's comparator function to handle row sorts. The Collection is always maintained in the correct order, and you have access to all the cool stuff you get in that framework.

It also means that split-flap.js used only for the display stuff. In fact, there's really only one object there--sf.display--and you really only call two methods on it: initRow() and either loadRow() or loadSequentially(). The rest of the code is outside split-flap.js.

Creating a board now looks like this:

// generate the empty rows markup (a backbone View)
var board = new Board;

// get the sort, etc. params
var container = $("#display1");
var params = container.find(".chartPrefs input");
var dataOptions = {
  "sort": container.find("input[name=sort]").val(),
  "order": container.find("input[name=order]").val()
};

// create the chart object (a backbone Collection)
var flights = new Flights;
flights.dataOptions = dataOptions;
flights.url = sf.plugins.airport.url(params);
flights.comparator = function(flight){
  if(dataOptions === "desc"){
    return -flight.get(dataOptions.sort);
  } else {
    return flight.get(dataOptions.sort);
  }
}
// update the chart (and set a refresh interval)
flights.update(container);
setInterval(function(){
  flights.update(container);
}, 30000); // refresh interval

See that "flights.comparator"? That's just an example of how you might want to handle sorting. How you structure your Collections is up to you. For example, the collection of flights looks like this:

// This Collection is used to hold the datset for this board. 
var Flights = Backbone.Collection.extend({
  update: function(container){
    this.fetch({
      success: function(response){
         sf.display.loadSequentially(response.toJSON(), container);
      }
    });
  },
  parse: function(json){
    return(sf.plugins.airport.formatData(json)); // normalize this data 
  }
});

You can see the plugin I showed earlier at work here where we call sf.plugins.airport.formatData(). This allows you to normalize your data formatting across APIs, which is something I've been pushing a lot lately.

I've also created another example board to go with the airport departures board. This one uses the Weather Underground's Weather API to display some airport weather data. This one is actually slightly more complex than the airport board because it has to make several calls to the API to build the Collection.

Anyway, check it out. As always, the project is on github here:
https://github.com/baspete/Split-Flap

and the live demo is here:
http://dev.basdesign.com/split-flap/

[edit - changed github repo]

December 16, 2011

Displaying Service-Based Data, Part 3

In my previous post I wrote about normalizing service-based data to a common format. Now I'm going to talk about getting that data onto the screen.

Continuing with our earlier example, say we've retrieved some weather data and transformed it into the following format:

{
  response: {
    meta: { ... some stuff ... },
    results: [
      { id:'KSFO', temp_f:45, wdir:170, wspd:4, pressure:1026, trend:'+', sky:'partly cloudy' },
      { id:'KHAF', temp_f:52, wdir:0, wspd:0, pressure:1025, trend:'+', sky:'clear' },
      { id:'KSQL', temp_f:45, wdir:0, wspd:0, pressure:1025, trend:'+', sky:'partly cloudy' },
      { id:'KOAK', temp_f:45, wdir:60, wspd:6, pressure:1026, trend:'+', sky:'partly cloudy' }
    ]
  }
}

Displaying this data consists of two steps:

  1. Getting the data into the right order and length
  2. Rendering the data

In some cases the data is already in the order we need because the API we used to retrieve it allowed us to specify sort criteria, max results, etc. But we may also be dealing with UI elements which allow the user to re-order this data, for example:




If the user changes something here, we shouldn't be going back to the API with a second call just to get the same data in a different order. A much better solution is to always pass the data through a sorting function which picks up those parameters on the way to the the rendering function.

In the past I've done this like so:

var render = function(data){
  ... create some markup and insert into DOM ...
};
var sort = function(data){
  var container = $("#chart"),
      sortBy = container.find('input[name=sortBy]').val(),
      sortOrder = container.find('input[name=sortOrder]').val(),
      maxResults = container.find('input[name=maxResults]').val(),
      truncate = container.find('input[name=truncate]').val();
  ... sort/truncate/etc ...
  return sortedData;
}
render(sort(data.response.results), $("#chart"));

This is a lousy idea, and I did it for years. The problem is that it requires both the sort() and render() functions to know enough about the DOM to find and read the UI elements specifying sort, maxResults and so on, and where they're supposed to render the resulting markup. A better solution is to do something like this:

var params = function(container){
  return {
    "sortBy": container.find('input[name=sortBy]').val(),
    "sortOrder": container.find('input[name=sortOrder]').val(),
    "maxResults": container.find('input[name=maxResults]').val(),
    "truncate": container.find('input[name=truncate]').val()
  }
};
var filter = function(data, filterCriteria){
  ... sort/truncate/maxResults ...
  return sortedData;
};
var render = function(data){
  ... create some markup ...
  return markup;
};

var container = $("#chart");
var chartType = container.attr("class").eq(0);
container.append(render[chartType](filter(data.response.results, params(container)));

params(), filter() and render() are all abstracted from the DOM now, and can be reused anywhere in your application, provided you keep your naming conventions the same everywhere.

Now let's talk about that chart() function.

In most web applications you're going to have to be able to handle more than one method for data visualization. You may be using flot for graphs, a jQuery plugin for sparklines, and Underscore.js for table markup. Each of these is likely to use a different data format, but because we've normalized our data we're starting in the same place for each of them.

A good way to handle this is to do something like this:

var render = {
  chart = function(data, options){
    ...format data for flot...
    var container = $("div");
    $.plot(container, formattedData, options.chartType);
    return container;
  },
  sparkline = function(data, options){
    ...format data for sparkline...
    var container = $("div");
    container.sparkline(formattedData, options);
    return container;
  },
  table = function(data, options){
    ...format data for table...
    var markup = _.template(options.template);
    return markup(data);
  }
}

Two things to note here: we're using jQuery to create a <div>, but we're not doing any DOM manipulation--we're simply returning that object to the calling function. Second in some cases we'll need to pass an "options" parameter along when calling render(). For example, when using the underscore.js "template" method, it's nice to be able to pass in your template markup from the page itself, like so:

var container = $("#chart");
var chartType = container.attr("class").eq(0);
var options = {template: $('#table_template').html()};
container.append(render[chartType](filter(data.response.results, params(container)), options);

In the next post, I'll bring this all together in a working example.

December 14, 2011

Displaying Service-Based Data, Part 2

In the first part of this series I talked about the importance of normalizing service-based data from different sources. The obvious question at this point is "normalized to what?", and that's the subject of this post.

When we talk about displaying "data", we're usually talking about the visual display of quantitative information. A table. A chart. A map. Something which makes visual comparisons between discreet items possible. When we're displaying categories (as in a table) we're often called upon to sort the data according to one or more properties. In a time-series we need to make sure that the sequence in which we display values matches their order. So, more often than not the order in which items are displayed is important, and we're talking about an array for our primary data structure.

What about the case where there's just a single item and we're displaying its properties? It's tempting to use a map, but it's almost always a bad idea. An array with one item in it works just fine and leaves you the ability to use the same structure for multiple items.

For example, using the Weather Underground data from the previous post, a request for weather data from a single station returns:

{
  response: { ... some stuff ... },
  location: { ... some stuff ... },
  current_observation: {
    ... some stuff ... 
  }
}

And the request for a list of weather stations returns:

{
  response: {... some stuff ...},
  location: {
    ... some stuff ...
    nearby_weather_stations: {
      airport: {
        station: [
          { ... result 1 stuff ... },
          { ... result 2 stuff ... }
        ]
      },
      pws:{
       station:[
          { ... result 1 stuff ... },
          { ... result 2 stuff ... }
       ]
      }
      ... some more stuff ... 
    }
  }
}

A normalized data structure for both of these cases might look like:

{
  response: {
    meta: { ... some stuff ... },
    results: [
      { ...result 1 stuff... },
      { ...result 2 stuff... },
      { ...result 3 stuff... },
      { ...result 4 stuff... }
    ]
  }
}

The normalization functions which would return the above data structure for the list of weather stations might look like:

plugins.wunderground = {

  stations: {
    formatData: function(data){
      var airports = data.location.nearby_weather_stations.airport.station,
          pws = response.location.nearby_weather_stations.pws.station,
          i = 0,
          formattedData = {
            respone: {
              meta: {whatever},
              results: []
            }
          };
      for(i=0;i<airports.length;i++){
        formattedData.response.results.push(airports[i]);
      }
      for(i=0;i<pws.length;i++){
        formattedData.response.results.push(pws[i]);
      }
    return formattedData;
    }
  },

  station: {
    formatData: function(data){
      var results = data.current_observation,
          formattedData = {
            respone: {
              meta: {whatever},
                results: []
              }
            };
      formattedData.response.push(results);
      return formattedData;
    }
  }

};

All this might seem like overkill until you realize it allows you to do something like this:

var dataSrc = "wunderground";
var id = "KSFO";

$.ajax({
  url: plugins[dataSrc].stations.url(id),
  dataType: plugins[dataSrc].dataType,
  success: function(response){
    var data = plugins[dataSrc].stations.formatData(response);
    var stations = new Backbone.Collection;
    for (var i=0;i<data.response.results.length;i++){
      var station = new Backbone.Model; 
      station.url = plugins[dataSrc].station.url(data.response.results[i].id); 
      stations.add(station);
    }
  }
});

Without worrying about the format in which data is returned from a service. If the Weather Underground changes its data format, we only need to change the formatData() method in the plugin. If we go with a different data provider, we only need to write a plugin for that provider and change dataSrc.

So what do we do once we've got this far? In the next post I'll talk about getting data from those data models onto the screen.

Displaying Service-Based Data, Part 1

Developers are frequently tasked with creating systems for the visual display of quantitative information (http://www.edwardtufte.com/tufte/books_vdqi). Over the years I've done a lot of these, and I've developed some ideas about what works and what doesn't. This is the first of several posts on the subject.

To start, I think it's important to recognize that there are three fundamentally different steps in the process of getting information displayed in a web browser:
  1.  Data retrieval
  2.  Data formatting
  3.  Data display
Historically, most information which eventually made it to a browser was either:
  •  Sent down to the browser as as part of a web page
  •  Assembled (or proxied) by the same server which served the page itself and exposed as a web service
  •  Retrieved and displayed in a separate frame
There are still good reasons to use the first two options, including bootstrapping, caching, removal of sensitive data, and complex business logic which may be unavailable to the browser. There is never a good reason to use the third option.

Prior to the widespread adoption of server-side support for JSONP the same origin policy meant that most web applications relied on their own servers for data services. This is no longer the case; the web developer is increasingly free to retrieve data from web services directly. From my point of view this is a fantastic development for two reasons:

  1. It frees companies to focus on their strengths. While Facebook, the USGS and Google all have browser interfaces, the most interesting work is being done by others who use the data made available through those organization's APIs. 
  2. It much more closely aligns the skillsets of the various software developers who work on a product with the parts of the stack where they work. Java devs are free to concentrate on ORMs without worrying about display logic. Front-end devs have access to those data models and they can use them to quickly build dynamic user interfaces.

Having spent 7 years working in a Java/Velocity framework (and JSP before that--ouch!), I'm particularly keen on the second point. I've always felt that the arguments about how much rope a templating language should give you reflect the fact that server-side templating languages are a hack in the first place. They're an unfortunate byproduct of early browsers and ECMAScript1 forcing us to move a big chunk of display logic onto the server.

In an earlier post I talked about how the big problem in front-end development is getting developers build their applications in the right order. In an environment where all the display logic is handled in the browser, creating a framework and a workflow which supports this is critical.

An Example

Say you were building an application in which you needed to display the current temperature in San Francisco. You'd have the option of using any of a number of publicly available APIs.

Here's the one from the Weather Underground's Weather API:

http://api.wunderground.com/api/d73bc72d0f231c10/conditions/q/CA/KSFO.json

Here's the same request from the Yahoo Weather API:

http://query.yahooapis.com/v1/public/yql?q=select%20item%20from%20weather.forecast%20where%20location%3D%2294107%22&format=json

Note how differently those requests are formatted.

Now take a look at the responses. The one from the Weather Underground looks like:

{
    "response": { ... some stuff ... },
    "location": { ... a whole bunch of stuff },
    "current_observation": {
        ... a bunch of stuff ...

        "temp_f": 55.8,

        ... a bunch more stuff ...
    }
}

... while the one from Yahoo! looks like:

{
   "query":{
      "count":1,
      "created":"2011-12-14T21:57:51Z",
      "lang":"en-US",
      "results":{
         "channel":{
            "item":{
               ... a bunch of stuff ...

               "condition":{
                  "code":"30",
                  "date":"Wed, 14 Dec 2011 12:55 pm PST",
                  "temp":"56",
                  "text":"Partly Cloudy"
               },

               ... a bunch more stuff ...
            }
         }
      }
   }
}

Note how differently these responses are formatted. If you were absolutely sure about which one you were going to use, you could just slam this data into a javascript object and start working on displaying the data, but that's almost never the case. Your company may decide to switch providers, or you may need to support multiple providers or combine data sources.

A much better solution is to create a plugin with two parts: a request URL formatter and a data normalization function which maps the response to common format--one that makes sense in the context of your application. Here's a simple example:

myApp.plugins.wunderground = {

    url: function(station_code){
      var base_url = "http://api.wunderground.com/api/d73bc72d0f231c10/conditions/q/";
      return base_url + station_code + ".json?" + "callback=myCallback";
    },

    formatData: function(response){
      var formattedData = {};
      formattedData["id"] = response.current_observation.station_id;
      formattedData["temp"] = response.current_observation.temp_f;
      return formattedData;
    }
};

Once you have that in place, the process of displaying data becomes independent of the data source. I'll talk about why that's important in the second part of this series.

December 8, 2011

StackOverflow Makes Me Stupid

I've been doing a fair amount of coding lately down on my boat and one thing that I've noticed is that when I forget to bring my Verizon MiFi I wind up being tremendously more productive. When I run into a problem, instead of googling something like "backbone.js model event binding" and then descending into a rabbit hole of information on StackOverflow, I'm forced to actually read and understand the code.

What a concept.

PS: an obvious shout-out to git is deserved here, which allows me to commit changes without a remote repository.

November 30, 2011

Improved Solari Board

Recently I received some interest in the virtual Solari Board project, which was the incentive I needed to take another look at it. The result is both simplified and improved:

  • Rows load sequentially now, and the end result is a much smoother animation as the number of elements changing at any one time is far fewer than before.
  • Added support for multiple different display element "drums", including letters only, numbers only, and images as well as the full set.
  • The status lights are now set as part of the same rendering process which handles all the other display elements. So no special scripting needed, and they change in concert with their row.
  • Removed a ton of extraneous code and refactored the data source to reflect a more common JSON format.
  • Updated to jQuery 1.7.1
As part of this refactor I removed most of the code associated with the other chart types. I think the right thing to do here is to add them back in a more modular way (require.js?). That'll probably be the next step on this project.

Here's the live demo and project on github.

    [edit - changed github repo]

    November 15, 2011

    An MVC approach to the Legislators App

    Recently, a couple of people pointed out to me that the architecture I was trying to explore in UI-RPC might lend itself to a more MVC approach. Since the Legislators app I created there makes a good "hello world" for testing javascript architectures, I'll use it again for this project.

    Lately the Backbone.js project is getting a lot of traction as a neat way to build an MVC javascript architecture. It provides models, views, collections, and routers (REST URI mappings) and ties them all together with object and event bindings. It also uses Underscore.js, which among other things has a built-in templating function. Since dynamically creating markup was one of the things I need to solve for UI-RPC, it makes sense to give it a test drive here.

    First things first:

    Code: https://github.com/baspete/backbone
    Live: http://dev.basdesign.com/backbone/

    The guts of the app are in legislators.js. The app is surprisingly small, consisting of a Model and View for each individual legislator, a Collection of legislators, and a Model and View for the filter widget.

    In this case, even though the models are the most fundamental objects here, they're not very interesting.

    var Legislator = Backbone.Model.extend({
      initialize: function(){
        console.log("Created Legislator",this.get("legislator"));
      }
    }); 
    

    Models fundamentally exist to hold the data model (duh) for an object. They can be bound to Views, and the magic of backbone is that changes to a Model will automatically update its View.

    Views are responsible for actually making something happen on the screen, and in this case what they mainly do is pass a JSON object to underscore.js for rendering:

    var LegislatorView = Backbone.View.extend({
      template: _.template($('#legislator_template').html()),
      initialize: function(){
        this.render();
      },
      render: function() {
        var l = this.model.toJSON().legislator;
        $("#legislators").append(this.template(l));
        return this;
      }
    });
    

    Models, Views, Collections and Routers in backbone are constructors. In our example, a new legislator is created by instantiating its View and binding it to a Model, like so:

    var view = new LegislatorView({
      model: Legislator
    });
    

    A Collection is a list of things, in our case a View for each legislator:

    var LegislatorList = Backbone.Collection.extend({
      url: sunlightBaseUrl,
      initialize: function(){
        this.fetch({
          success: function(collection) {
            collection.each(function(Legislator) {
              var view = new LegislatorView({
                model: Legislator
              });
            });
           }
        })
      },
      sync: function(method, model, options){  
        options.cache    = true; // sunlightlabs needs this to return jsonp
        options.jsonp    = "jsonp"; // sunlightlabs needs this to return jsonp
        options.dataType = "jsonp";  // by tell backbone.js to use jsonp
        return Backbone.sync(method, model, options);  
      },
      parse: function(response){
        return(response.response.legislators); // just return the array, not the whole object
      }
    });
    

    The first issue I ran into was the same one raised in UI-RPC: how do you bind an action in one view to a different view or collection? I cheated:

    var Filter = Backbone.Model.extend({
      change: function(){
        $("#legislators").empty();
        var params = this.toJSON();
        // sending empty query params breaks 
        for(var i in params){
          if(params[i] === "undefined" || params[i] === ""){
            delete params[i];
          }
        }
        // This is ghetto. How to bind here?
        legislators.url = sunlightBaseUrl + $.param(params);
        legislators.initialize();
      }
    });
    

    "legislators" is an instance of the Legislators collection, and one thing I'm certain of is that a Model shouldn't know anything about instances. Interestingly, when I researched this online I found a bunch of posts talking about using a controller to handle this event mapping. Hmmm... that's the same problem I ran into in UI-RPC.

    At any rate, even though this is an incomplete version of the Legislators app, with only one data source (the sunlight labs API), it's an interesting start. I'll get it to the same level of functionality that UI-RPC is currently, then start exploring the two architectures in parallel.

    New Editor

    Over the years, the progression of editors I've used as my primary tool has gone something like:
    • 1984 - vi
    • 1993 - TextPad
    • 1998 - BBEdit
    • 2008 - TextMate
    The other day I tried Sublime Text 2 and it only took a few minutes to decide I had found my New Very Best Friend.

    Among other great features:
    • Great UI out of the gate. I like the 10,000 ft view.
    • Multi-select. This is so cool:
      • Splat-click on as many things as you want and they're all selected and change together as you type.
      • Select a block of lines (splat-shift-l) and they're edited together.
      • With the cursor on a word, hitting splat-d selects additional occurrences of that word. Or, just hit ctrl-splat-g to select all occurrences of that word.
    • Predictive text (both inline using <tab> or in a command palette accessed by hitting shift-splat-p
    • Lightning-fast file switching/opening. Hit splat-p and start typing the name of a file. As you type, the list narrows, but all those files are actually *open*. It's hard to describe how well this works.
    • vi(m) command mode! Hit escape when editing and all those vi commands you have memorized will work.
    • Smart indenting with alignment marks. The thing that pissed me off most about TextMate was the braindead way it did indenting--if a line was half-selected it wouldn't follow any other lines you were indenting/outdenting. Sublime Text fixes this, and with the addition of the "Alignment" plugin makes it easy to align things like equal signs on multiple lines.
    Sublime Edit has an active development community and a ton of plugins for all the things you'd expect, like zen coding, GitHub integration, etc. One really cool thing about the plugin package management (which works kind of like apt-get) is that the packages are actually checked out using Git, and they're automatically updated every time you start the app.

    I really like this editor.

    November 4, 2011

    Virtual Solari Board

    I've always been intrigued by those split-flap displays you used to see clattering away in train stations and airports. Not only are they fun to watch, but I think they really symbolize the peak of 20th century electro-mechanical design and engineering. The Italian company Solari di Udine made tons of them, so they're often called Solari Boards.

    The principle on which they work is relatively straightforward. Here's a schematic of a single element:


    The display element is advanced by a stepper motor, which in turn receives pulses from a controller (or the controller may be built in) and each pulse advances the display one character. The element may contain letters, numbers, punctuation, pictures, entire words, or any combination thereof. This type of system appeals to me because it translates well into software, so a while ago I wrote a javascript application to create a virtual Solari Board. I'm bringing it up here because it's probably time to take another look at it.

    The wheel can be represented as an array, with the characters on the flaps as elements in that array, like so:

    Drum: function() {
      this.startPosition = [' ','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','0','1','2','3','4','5','6','7','8','9','.',',','?','!','/','\'','+','-','↑','↓'];
    }
    

    Changing the display to a different value is a matter of rotate()ing the array as needed, like so:

    change: function() {
      var container = arguments[0],
        c = arguments[1], // the new character
        index, i, j;
      // get the curent order of the display element's drum
      var values = container.data("order");
      // how many times do we need to increment the drum?
      index = values.indexOf(c);
      // increment the drum
      for(i=0;i<index;i++) {
        sf.chart.splitFlap.show(container, (values[i + 1]));
      }
      // rotate the dom element's stored array to the new order for next time
      container.data("order", values.rotate(index));
      // change the number in the markup
      // container.text(c);
    }
    

    Note the use of jQuery's .data() API for storing the new order of the drum array. I'm a fan of storing state information this way, particularly when it comes to UI elements--it's a perfect use of the feature. Certainly you can get into trouble if you go overboard and try and store non-ui state stuff, but it works well in this case.

    This app is based on the groundwork I did at CMSG for our charting stuff (and the complexity of the javascript reflects that broader context) using $.getJSON() to grab data in the background and load it dynamically. Looking over this project, I really like some of the ideas I used:

    • A single method for rendering multiple chart types (split flap is considered a chart type)
    • A common data format, with specs for time-series, category and other data types
    • A plug-in architecture for different chart types

    I think this project is worth revisiting. There's good work here.

    Live demo: http://dev.basdesign.com/split-flap/activity_tracker.html

    Github repo: https://github.com/baspete/Split-Flap

    [Edit: linked to Solari di Udine]
    [edit - changed github repo]

    October 12, 2011

    A First Pass at UI-RPC

    Based on the ideas I laid out the other day, I've taken a first pass at something I'm calling UI-RPC. This is a simple proof-of-concept project meant to demonstrate the use of HTML5's postMessage() API for event handling and inter-widget communication.

    Code: https://github.com/baspete/UIRPC
    Live: http://dev.basdesign.com/UIRPC/

    The single-page app displays a list of congresspersons matching some simple filter criteria. Clicking on a congressperson's photo produces some additional information about them.

    The html page itself is just a skeleton, responsible for loading CSS, javascript libraries, and laying out the basic page structure. The actual functionality is divided among two types of javascript objects: widgets and workers. The (admittedly arbitrary) distinction between the two is that widgets have a presentation component, and workers don't. Widgets are added into the markup like so:


    Note that those divs are empty. That's because, near the end of the page markup, we call a javascript function, createWidgets(), which will crawl the DOM, find the various widget containers, instantiate an appropriate widget object for each container, and fire that object's init() method:

    UIRPC.createWidgets();
    

    At the same time, a different javascript function, createWorkers() is passed an array of workers which should be instantiated on this page.

    UIRPC.createWorkers(["congress","facebook","twitter"]);
    

    Finally, an event dispatcher is initialized, which events and helpers use to communicate with each other. This is a special type of helper, and looks for a map which associates one or more procedures with an event (for convention, events are capitalized and procedures are camel-cased):

    UIRPC.events = {
        FILTER_CHANGED: ["getLegislators","showRetrievingData"],
        LEGISLATORS_CHANGED: ["showLegislators"],
        REQUEST_DETAILS: ["getCommittees","getFacebookInfo","getTwitterInfo"],
        GET_COMMITTEES_RESULTS: ["showCommittees"],
        GET_FACEBOOK_INFO_RESULTS: ["showFacebookInfo"],
        GET_TWITTER_INFO_RESULTS: ["showTwitterInfo"]
      };
    

    A widget or helper registers a procedure like so:

    pmrpc.register({
        publicProcedureName: "showLegislators",
        procedure: function (data, options) {
          showLegislators(data, options);
        },
        isAsynchronous: true
      });
    

    Widgets and helpers are written as closures, and each publicly registered procedure corresponds to one of the module's private methods (in this case, "showLegislators").

    When a widget or helper wants to fire an event to the event dispatcher it uses the pmrpc.call() method. For example:

    pmrpc.call({
          destination : window,
          publicProcedureName : "event",
          params : {
            data: {
              eventName: "FILTER_CHANGED",
              data: getFilterCriteria(),
              options: null
            }
          }
        });
    

    In this case, the FILTER_CHANGED event is called with the data returned by the widget's private "getFilterCriteria()" method.

    The easiest way to see this in action is to load the page and watch the javascript console. The event dispatcher will log to the console every time it dispatches an event to a procedure:

    dispatching  FILTER_CHANGED  to  getLegislators  with data:  Object  and options  null
    dispatching  FILTER_CHANGED  to  showRetrievingData  with data:  Object  and options  null
    dispatching  LEGISLATORS_CHANGED  to  showLegislators  with data:  [Object, Object, Object]  and options  null
    

    As you can see, the FILTER_CHANGED event is being passed to the getLegislators and showRetrievingData procedures, and then the LEGISLATORS_CHANGED event is passed to the showLegislators procedure.

    Clicking on a legislator's picture produces something like:

    dispatching  REQUEST_DETAILS  to  getCommittees  with data:  Object  and options  Object
    dispatching  REQUEST_DETAILS  to  getFacebookInfo  with data:  Object  and options  Object
    dispatching  REQUEST_DETAILS  to  getTwitterInfo  with data:  Object  and options  Object
    dispatching  GET_FACEBOOK_INFO_RESULTS  to  showFacebookInfo  with data:  Object  and options  Object
    dispatching  GET_TWITTER_INFO_RESULTS  to  showTwitterInfo  with data:  [Object, Object, Object]  and options Object
    dispatching  GET_COMMITTEES_RESULTS  to  showCommittees  with data:  [Object, Object, Object, Object, Object]  and options  Object
    

    Issues


    Clearly, there are a couple of problems with this framework as it currently exists:

    1. The event mapping is too simplistic. For real-life applications, the current method of mapping events to an array of procedures won't be sufficient--a developer will need to be able to apply conditional logic, failovers, etc. But the idea seems sound, and this issue could be addressed through the use of a real map instead of just an array.
    2. There's markup in my javascript!. Yup, and this sucks just as bad as having markup hard-coded into the page. I think this is the most important thing to fix here. There are templating systems out there, but I'm not sold on any of them. I'm thinking about the way backbone.js does it may be a good place to look for ideas.
    3. There's a lot of duplicated code, particularly the procedure registration part. But we're using the javascript module pattern here, and the next logical step seems to be creating specific modules/helpers by extend()ing generic module/helper modules.

    Overall, I think this might be a useful pattern. At this point it's hard to tell if it meets the criteria of getting front-end developers to think first about functionality. That will take another project.

    [edit - changed github repo]

    September 10, 2011

    A New Idea for Front-End Development

    I've been doing front-end web development for almost 15 years now, and for that entire time this the typical development model has looked something like this:

    1. Start with a page design and mark up a single page, using the templating language of your choice to dynamically generate content. Create some CSS to make it look the way you want.
    2. Use javascript to handle realtime interactions (hiding/showing stuff, XHR transactions, etc).


    As web applications have grown more dynamic on the front end, that second step has grown to encompass most of a developer's time, but we haven't really changed much about the first step. A perfect example is the way jQueryUI works: you create your markup in the html template as usual, but then you add one or more classes to it--for example, "ui-widget" or "ui-corners-all". When the page loads, jQueryUI's init() method crawls the DOM looking for those classes and attaches event bindings, behaviors, extra markup, etc., to them.

    In theory this is a great idea, but in practice it's a nightmare of inconsistency; every widget is marked up differently according to the needs at the time of its design. This is because jQueryUI's built-in behaviors often aren't sufficient for your needs, so you wind up writing a bunch of custom code to handle additional bindings and behaviors. That code winds up being full of widely-scoped methods and tight coupling between objects, and this quickly becomes unmaintainable. The Eos front-end certainly wound up like this.

    I don't want to develop web applications that way any more. I'd like to try something which looks more like this:

    • The purpose of a web page is to be a container which provides context (URL, session, DOM, pub/sub broker, etc) for the items within it. It has little or no built-in functionality as initially sent down to the browser, and only enough markup to establish a basic layout according to the design.
    • Based on the context provided by the page, the browser's javascript parser instantiates one or more objects. These objects are responsible for generating their own markup and inserting it into the DOM, retrieving whatever data they require from whatever service they need (page context is important here), and establishing their own event bindings. They communicate with each other (and even other resources not on the page) using a pub/sub/multicast RPC-like asynchronous messaging API (JSON-RPC is a good choice of protocol here).

    When viewed like this, it seems that the development process is very different than the one I've shown above. In particular, it forces front-end devs to think in terms of components and interfaces from the beginning. I think the process would look more like:

    1. Start with the design of an entire site and figure out what components you're going to have to build. Many of those components will exist in multiple places and will appear and/or behave differently depending on the page context (a simple example would be a navigation tool which needs to know the URL or session information so it can determine how to render itself).
    2. Define each component's interface: what messages it will send in response to what events, how it will respond to any messages it has subscribed to, and the exact format of each of those message.
    3. Code the pages for site, including their basic markup, look/feel (CSS), etc. At this point the pages will be mostly empty, with no functional components.
    4. Code the javascript objects and markup representing those components, with their event bindings, methods, properties, and insert them into a library through which they can be loaded individually into a page and instantiated via a Constructor or Prototype pattern.

    (Steps 3 and 4 could be done at the same time if you have some programmers who are better at doing layout/look/feel and others who are better at javascript)

    It seems to me this architecture of loosely-coupled components interacting through a common message-based framework has some advantages:

    • By focusing on architecture first, we're likely to spot problems with the design earlier than later. In this model a front-end architect's job will largely be to define those interfaces.
    • A message-based front-end architecture which uses well-defined interfaces and pub/sub means unit testing is not only possible, but easy. Front-end test-driven development, yo!
    • Components which are reusable are going to encourage the use of more semantic markup, which *should* lead to clearer CSS and improved maintainability.
    • This message-based model currently exists as a way to handle inter-frame browser programming. Take a look at http://code.google.com/p/pmrpc, it's basically a set of wrappers around HTML5's postMessage API which implement an RPC model using JSON-RPC. It provides a framework for callbacks, timeouts, retries, synchronous and asynchronous procedures, etc. But the thing I'm most interested in is the multicast/publish-subscribe stuff, along with service discovery. The project is designed for inter-window communication, but the ideas there make sense even within a single window context.
    • A service-oriented application architecture should be more flexible, allowing right out of the gate the development of apps on different platforms (mobile, etc). It also divides the tasks sensibly between front-end developers and application developers.


    My guess is that at this point, you're either going "welcome to 2007" or "Pete just discovered IOC." OK, but while it's true that there are some shops doing development using models very similar to this, and everyone is practicing at least some level of componentization and code reuse, the vast majority of web applications I've seen still use some version of what I described at the beginning of this note, and I can't think of any applications that use intra-window pub/sub messaging interfaces between UI components.

    Why is that? The tools for doing this have been around for a while--you don't need HTML5's publishMessage API to do pub/sub (although it certainly makes it easier). I've been dynamically inserting markup into the DOM since at least 1999, and jQuery has made event-based programming almost trivial (take a look at http://api.jquery.com/category/deferred-object/ and consider what you can do with it). Something like Google Web Toolkit allows us to take code generation to the extreme.

    I think the real reason has nothing to do with programming. If you look at how designers work, they've traditionally thought mainly in terms of pages and less in terms of holistic application behaviors. If you think about it, the fact that a designer thinks it's acceptable to deliver a single screen at a time is astounding--they're assuming that what they've designed on that screen will be appropriate for the rest of the application, a huge assumption indeed. But it's understandable given that 15 years ago designers and design firms got most of their work from print and ad design, which focus on creating standalone images. That design culture has persisted in front-end software development, where most front-end software developers (myself included) have little or no formal training in object-oriented software design, and many of them are in fact visual designers first. So they start with the visuals.

    And building things this way is tremendously appealing, because--at least initially--it seems like you're making a *ton* of progress. With a few hours work a competent front-end coder can have something on the screen which looks exactly like the most complex mock. But it's an illusion--it doesn't *do* anything yet. The real work comes when you start coding behaviors and at that point two things always happen: first, you try and reuse some component from elsewhere in the app, but find it too tightly-coupled to that other context to be useful, so you build another version just for this page; and second, as the designers and product managers create more pages they uncover invalid assumptions and fix them by changing the design. You find yourself chasing a moving target.

    So how to fix this? I honestly don't think we ever will--I've never worked on anything that wasn't a moving target, and it's probably a good thing that we have designers who want to code user interfaces. But what I've proposed here at least offers some hope of dealing with those changes in a way that makes sense. If an architecture forces us to think first about discreet, reusable browser components, we're more likely to confine our changes to those components--and perhaps stimulate some thinking about consistency of design in the product itself. By relying on well-defined interfaces and a message-based architecture we ensure that our components stay testable as we make changes.

    Are there problems? Of course. Componentization and code reuse have been fundamental tenets of software development forever and we still screw those up as often as we can. There's nothing to stop a proliferation of similar components and interfaces--and for that matter, web services--when refactoring a single component to handle several different contexts would be a better choice. But *starting* with the components and their interfaces rather than whole pages at least encourages reuse more than the way we're doing it now.

    Also, there are a number of other obvious issues:

    1. HTML5 is widely but not universally supported. I don't think this is a big deal--the postMessage API is supported in IE8+, FF3+, Opera 9 and all webkit browsers since at least 2008.
    2. Dynamically generated markup isn't currently parsed by the googlebot. I honestly don't know what to do about this one, but in an increasingly services-based-architecture world they're going to have to do something about this eventually.
    3. Figuring out how to dynamically generate and cache javascript and load it with performance in mind is non-trivial, especially when you consider that an attractive way to organize this kind of application is by separating components in the filesystem, each with its own markup and CSS. How about a javascript buffer into which objects are concatenated at runtime, and which is then linked as a single file in the page markup? Ditto with CSS.

    I'm sure there's more, but this seems like the right place to stop and start coding the "hello world" for this. If you've got this far, thanks for reading, and I'd love to get any feedback you have, pointers to projects doing this kind of thing, etc.