December 19, 2011

Virtual Solari Board Improved

As part of the series of posts about displaying data, I've updated the virtual Solari board project again.

It now uses a plugin framework to handle customized look and feel. You create a javascript object with properties and methods specific to your data source, like so:

sf.display.ImageDrum = function() {
  this.order = [' ','AFL','AAL','BAW','DAL','UAE','KLM','DLH','RYR','UAL','AWE'];
};

sf.plugins.airport =  {
  
  // input: map of parameters
  // output: url
  url: function(params){
    var base_url = "data/airport_schedule.php";
    return base_url + "?" + params.serialize();
  },

  formatData: function(data){
    return data.response.results[0].data;
  }

};

sf.display.ImageDrum()'s order is now a series of strings instead of integers, which should be more readable. So in the above example, the CSS looks like:

.splitflap .image span.ctrn { background-position: 0px 0px; } /* transition */
.splitflap .image span.csp { background-position: 0px -40px; } /* blank */
.splitflap .image span.AFL { background-position: 0px -80px; } /* aeroflot */
.splitflap .image span.AAL { background-position: 0px -120px; } /* american */
.splitflap .image span.BAW { background-position: 0px -160px; } /* british airways */
.splitflap .image span.DAL { background-position: 0px -200px; } /* delta */
.splitflap .image span.UAE { background-position: 0px -240px; } /* emirates */
.splitflap .image span.KLM { background-position: 0px -280px; } /* klm */
.splitflap .image span.DLH { background-position: 0px -320px; } /* lufthansa */
.splitflap .image span.RYR { background-position: 0px -360px; } /* ryanair */
.splitflap .image span.UAL { background-position: 0px -400px; } /* united */
.splitflap .image span.AWE { background-position: 0px -440px; } /* us airways */

But the big change here is that the project now uses backbone.js as its MVC framework. That means that the list of rows is a Collection and each row is a Model. This allows us to use Backbone's comparator function to handle row sorts. The Collection is always maintained in the correct order, and you have access to all the cool stuff you get in that framework.

It also means that split-flap.js used only for the display stuff. In fact, there's really only one object there--sf.display--and you really only call two methods on it: initRow() and either loadRow() or loadSequentially(). The rest of the code is outside split-flap.js.

Creating a board now looks like this:

// generate the empty rows markup (a backbone View)
var board = new Board;

// get the sort, etc. params
var container = $("#display1");
var params = container.find(".chartPrefs input");
var dataOptions = {
  "sort": container.find("input[name=sort]").val(),
  "order": container.find("input[name=order]").val()
};

// create the chart object (a backbone Collection)
var flights = new Flights;
flights.dataOptions = dataOptions;
flights.url = sf.plugins.airport.url(params);
flights.comparator = function(flight){
  if(dataOptions === "desc"){
    return -flight.get(dataOptions.sort);
  } else {
    return flight.get(dataOptions.sort);
  }
}
// update the chart (and set a refresh interval)
flights.update(container);
setInterval(function(){
  flights.update(container);
}, 30000); // refresh interval

See that "flights.comparator"? That's just an example of how you might want to handle sorting. How you structure your Collections is up to you. For example, the collection of flights looks like this:

// This Collection is used to hold the datset for this board. 
var Flights = Backbone.Collection.extend({
  update: function(container){
    this.fetch({
      success: function(response){
         sf.display.loadSequentially(response.toJSON(), container);
      }
    });
  },
  parse: function(json){
    return(sf.plugins.airport.formatData(json)); // normalize this data 
  }
});

You can see the plugin I showed earlier at work here where we call sf.plugins.airport.formatData(). This allows you to normalize your data formatting across APIs, which is something I've been pushing a lot lately.

I've also created another example board to go with the airport departures board. This one uses the Weather Underground's Weather API to display some airport weather data. This one is actually slightly more complex than the airport board because it has to make several calls to the API to build the Collection.

Anyway, check it out. As always, the project is on github here:
https://github.com/baspete/Split-Flap

and the live demo is here:
http://dev.basdesign.com/split-flap/

[edit - changed github repo]

December 16, 2011

Displaying Service-Based Data, Part 3

In my previous post I wrote about normalizing service-based data to a common format. Now I'm going to talk about getting that data onto the screen.

Continuing with our earlier example, say we've retrieved some weather data and transformed it into the following format:

{
  response: {
    meta: { ... some stuff ... },
    results: [
      { id:'KSFO', temp_f:45, wdir:170, wspd:4, pressure:1026, trend:'+', sky:'partly cloudy' },
      { id:'KHAF', temp_f:52, wdir:0, wspd:0, pressure:1025, trend:'+', sky:'clear' },
      { id:'KSQL', temp_f:45, wdir:0, wspd:0, pressure:1025, trend:'+', sky:'partly cloudy' },
      { id:'KOAK', temp_f:45, wdir:60, wspd:6, pressure:1026, trend:'+', sky:'partly cloudy' }
    ]
  }
}

Displaying this data consists of two steps:

  1. Getting the data into the right order and length
  2. Rendering the data

In some cases the data is already in the order we need because the API we used to retrieve it allowed us to specify sort criteria, max results, etc. But we may also be dealing with UI elements which allow the user to re-order this data, for example:




If the user changes something here, we shouldn't be going back to the API with a second call just to get the same data in a different order. A much better solution is to always pass the data through a sorting function which picks up those parameters on the way to the the rendering function.

In the past I've done this like so:

var render = function(data){
  ... create some markup and insert into DOM ...
};
var sort = function(data){
  var container = $("#chart"),
      sortBy = container.find('input[name=sortBy]').val(),
      sortOrder = container.find('input[name=sortOrder]').val(),
      maxResults = container.find('input[name=maxResults]').val(),
      truncate = container.find('input[name=truncate]').val();
  ... sort/truncate/etc ...
  return sortedData;
}
render(sort(data.response.results), $("#chart"));

This is a lousy idea, and I did it for years. The problem is that it requires both the sort() and render() functions to know enough about the DOM to find and read the UI elements specifying sort, maxResults and so on, and where they're supposed to render the resulting markup. A better solution is to do something like this:

var params = function(container){
  return {
    "sortBy": container.find('input[name=sortBy]').val(),
    "sortOrder": container.find('input[name=sortOrder]').val(),
    "maxResults": container.find('input[name=maxResults]').val(),
    "truncate": container.find('input[name=truncate]').val()
  }
};
var filter = function(data, filterCriteria){
  ... sort/truncate/maxResults ...
  return sortedData;
};
var render = function(data){
  ... create some markup ...
  return markup;
};

var container = $("#chart");
var chartType = container.attr("class").eq(0);
container.append(render[chartType](filter(data.response.results, params(container)));

params(), filter() and render() are all abstracted from the DOM now, and can be reused anywhere in your application, provided you keep your naming conventions the same everywhere.

Now let's talk about that chart() function.

In most web applications you're going to have to be able to handle more than one method for data visualization. You may be using flot for graphs, a jQuery plugin for sparklines, and Underscore.js for table markup. Each of these is likely to use a different data format, but because we've normalized our data we're starting in the same place for each of them.

A good way to handle this is to do something like this:

var render = {
  chart = function(data, options){
    ...format data for flot...
    var container = $("div");
    $.plot(container, formattedData, options.chartType);
    return container;
  },
  sparkline = function(data, options){
    ...format data for sparkline...
    var container = $("div");
    container.sparkline(formattedData, options);
    return container;
  },
  table = function(data, options){
    ...format data for table...
    var markup = _.template(options.template);
    return markup(data);
  }
}

Two things to note here: we're using jQuery to create a <div>, but we're not doing any DOM manipulation--we're simply returning that object to the calling function. Second in some cases we'll need to pass an "options" parameter along when calling render(). For example, when using the underscore.js "template" method, it's nice to be able to pass in your template markup from the page itself, like so:

var container = $("#chart");
var chartType = container.attr("class").eq(0);
var options = {template: $('#table_template').html()};
container.append(render[chartType](filter(data.response.results, params(container)), options);

In the next post, I'll bring this all together in a working example.

December 14, 2011

Displaying Service-Based Data, Part 2

In the first part of this series I talked about the importance of normalizing service-based data from different sources. The obvious question at this point is "normalized to what?", and that's the subject of this post.

When we talk about displaying "data", we're usually talking about the visual display of quantitative information. A table. A chart. A map. Something which makes visual comparisons between discreet items possible. When we're displaying categories (as in a table) we're often called upon to sort the data according to one or more properties. In a time-series we need to make sure that the sequence in which we display values matches their order. So, more often than not the order in which items are displayed is important, and we're talking about an array for our primary data structure.

What about the case where there's just a single item and we're displaying its properties? It's tempting to use a map, but it's almost always a bad idea. An array with one item in it works just fine and leaves you the ability to use the same structure for multiple items.

For example, using the Weather Underground data from the previous post, a request for weather data from a single station returns:

{
  response: { ... some stuff ... },
  location: { ... some stuff ... },
  current_observation: {
    ... some stuff ... 
  }
}

And the request for a list of weather stations returns:

{
  response: {... some stuff ...},
  location: {
    ... some stuff ...
    nearby_weather_stations: {
      airport: {
        station: [
          { ... result 1 stuff ... },
          { ... result 2 stuff ... }
        ]
      },
      pws:{
       station:[
          { ... result 1 stuff ... },
          { ... result 2 stuff ... }
       ]
      }
      ... some more stuff ... 
    }
  }
}

A normalized data structure for both of these cases might look like:

{
  response: {
    meta: { ... some stuff ... },
    results: [
      { ...result 1 stuff... },
      { ...result 2 stuff... },
      { ...result 3 stuff... },
      { ...result 4 stuff... }
    ]
  }
}

The normalization functions which would return the above data structure for the list of weather stations might look like:

plugins.wunderground = {

  stations: {
    formatData: function(data){
      var airports = data.location.nearby_weather_stations.airport.station,
          pws = response.location.nearby_weather_stations.pws.station,
          i = 0,
          formattedData = {
            respone: {
              meta: {whatever},
              results: []
            }
          };
      for(i=0;i<airports.length;i++){
        formattedData.response.results.push(airports[i]);
      }
      for(i=0;i<pws.length;i++){
        formattedData.response.results.push(pws[i]);
      }
    return formattedData;
    }
  },

  station: {
    formatData: function(data){
      var results = data.current_observation,
          formattedData = {
            respone: {
              meta: {whatever},
                results: []
              }
            };
      formattedData.response.push(results);
      return formattedData;
    }
  }

};

All this might seem like overkill until you realize it allows you to do something like this:

var dataSrc = "wunderground";
var id = "KSFO";

$.ajax({
  url: plugins[dataSrc].stations.url(id),
  dataType: plugins[dataSrc].dataType,
  success: function(response){
    var data = plugins[dataSrc].stations.formatData(response);
    var stations = new Backbone.Collection;
    for (var i=0;i<data.response.results.length;i++){
      var station = new Backbone.Model; 
      station.url = plugins[dataSrc].station.url(data.response.results[i].id); 
      stations.add(station);
    }
  }
});

Without worrying about the format in which data is returned from a service. If the Weather Underground changes its data format, we only need to change the formatData() method in the plugin. If we go with a different data provider, we only need to write a plugin for that provider and change dataSrc.

So what do we do once we've got this far? In the next post I'll talk about getting data from those data models onto the screen.

Displaying Service-Based Data, Part 1

Developers are frequently tasked with creating systems for the visual display of quantitative information (http://www.edwardtufte.com/tufte/books_vdqi). Over the years I've done a lot of these, and I've developed some ideas about what works and what doesn't. This is the first of several posts on the subject.

To start, I think it's important to recognize that there are three fundamentally different steps in the process of getting information displayed in a web browser:
  1.  Data retrieval
  2.  Data formatting
  3.  Data display
Historically, most information which eventually made it to a browser was either:
  •  Sent down to the browser as as part of a web page
  •  Assembled (or proxied) by the same server which served the page itself and exposed as a web service
  •  Retrieved and displayed in a separate frame
There are still good reasons to use the first two options, including bootstrapping, caching, removal of sensitive data, and complex business logic which may be unavailable to the browser. There is never a good reason to use the third option.

Prior to the widespread adoption of server-side support for JSONP the same origin policy meant that most web applications relied on their own servers for data services. This is no longer the case; the web developer is increasingly free to retrieve data from web services directly. From my point of view this is a fantastic development for two reasons:

  1. It frees companies to focus on their strengths. While Facebook, the USGS and Google all have browser interfaces, the most interesting work is being done by others who use the data made available through those organization's APIs. 
  2. It much more closely aligns the skillsets of the various software developers who work on a product with the parts of the stack where they work. Java devs are free to concentrate on ORMs without worrying about display logic. Front-end devs have access to those data models and they can use them to quickly build dynamic user interfaces.

Having spent 7 years working in a Java/Velocity framework (and JSP before that--ouch!), I'm particularly keen on the second point. I've always felt that the arguments about how much rope a templating language should give you reflect the fact that server-side templating languages are a hack in the first place. They're an unfortunate byproduct of early browsers and ECMAScript1 forcing us to move a big chunk of display logic onto the server.

In an earlier post I talked about how the big problem in front-end development is getting developers build their applications in the right order. In an environment where all the display logic is handled in the browser, creating a framework and a workflow which supports this is critical.

An Example

Say you were building an application in which you needed to display the current temperature in San Francisco. You'd have the option of using any of a number of publicly available APIs.

Here's the one from the Weather Underground's Weather API:

http://api.wunderground.com/api/d73bc72d0f231c10/conditions/q/CA/KSFO.json

Here's the same request from the Yahoo Weather API:

http://query.yahooapis.com/v1/public/yql?q=select%20item%20from%20weather.forecast%20where%20location%3D%2294107%22&format=json

Note how differently those requests are formatted.

Now take a look at the responses. The one from the Weather Underground looks like:

{
    "response": { ... some stuff ... },
    "location": { ... a whole bunch of stuff },
    "current_observation": {
        ... a bunch of stuff ...

        "temp_f": 55.8,

        ... a bunch more stuff ...
    }
}

... while the one from Yahoo! looks like:

{
   "query":{
      "count":1,
      "created":"2011-12-14T21:57:51Z",
      "lang":"en-US",
      "results":{
         "channel":{
            "item":{
               ... a bunch of stuff ...

               "condition":{
                  "code":"30",
                  "date":"Wed, 14 Dec 2011 12:55 pm PST",
                  "temp":"56",
                  "text":"Partly Cloudy"
               },

               ... a bunch more stuff ...
            }
         }
      }
   }
}

Note how differently these responses are formatted. If you were absolutely sure about which one you were going to use, you could just slam this data into a javascript object and start working on displaying the data, but that's almost never the case. Your company may decide to switch providers, or you may need to support multiple providers or combine data sources.

A much better solution is to create a plugin with two parts: a request URL formatter and a data normalization function which maps the response to common format--one that makes sense in the context of your application. Here's a simple example:

myApp.plugins.wunderground = {

    url: function(station_code){
      var base_url = "http://api.wunderground.com/api/d73bc72d0f231c10/conditions/q/";
      return base_url + station_code + ".json?" + "callback=myCallback";
    },

    formatData: function(response){
      var formattedData = {};
      formattedData["id"] = response.current_observation.station_id;
      formattedData["temp"] = response.current_observation.temp_f;
      return formattedData;
    }
};

Once you have that in place, the process of displaying data becomes independent of the data source. I'll talk about why that's important in the second part of this series.

December 8, 2011

StackOverflow Makes Me Stupid

I've been doing a fair amount of coding lately down on my boat and one thing that I've noticed is that when I forget to bring my Verizon MiFi I wind up being tremendously more productive. When I run into a problem, instead of googling something like "backbone.js model event binding" and then descending into a rabbit hole of information on StackOverflow, I'm forced to actually read and understand the code.

What a concept.

PS: an obvious shout-out to git is deserved here, which allows me to commit changes without a remote repository.